Risk-sensitive control as inference with Rényi divergence
Authors: Kaito Ito
Kenji Kashima
Venue: NeurIPS 2024 poster (2024)
Keywords: risk-sensitive control
optimal control
reinforcement learning
variational inference
Rényi divergence
Primary area: reinforcement_learning
TL;DR: This study formulates risk-sensitive control as variational inference using Rényi divergence. Based on the proposed unifying framework, we reveal several equivalence results for control problems and derive risk-sensitive RL algorithms.

Abstract: This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

Risk-Sensitive Control as Inference with Rényi Divergence
Kaito Ito
Kenji Kashima Kyoto University kkθi Kroiikyytouuccm ac JP
The University of Tokyo kaito0g ecc. u-tokyo ac.
Abstract
This paper introduces the risk-sensitive control inference (RCal) that extends Cal by using Rényi divergence variational inference. RCal is shown to be equivalent log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving soft Bellman equation, which reveals several equivalences between RCal, MaxEnt control, the optimal posterior for Cal, and linearly-sole solvalle control. Moreover, based on RCal, we derive the risk sensiivve reinforcement learning (RL) methods: the policy gradient and the soft actor- crtttcciiiic. As the sesisiisit parameter vanishes, we recover the risk-neutral Cal and RL, which means that RCal is unifying framework. Furthermore, we give another risk- sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions. the optimal policies have the same structure even though the derivations are very different.
1 Introduction
Optimal control theory is powerful framework for sequential decision making [1]. In optimal control problems, one seeks find control policy that minimizes a given cost functional and typically assumes the full knowledge of the system's dynamics. Optimal control with unknown or partially known dynamics is called reinforcement learning (RL) [2], which has been successfully applied to highly complex and uncertain systems, e.g., robotics [3], self-driving vehicles [4]. However, solving optimal control and RL problems is still challenging, especially for continuous spaces.
Control Inference (Cal), which connects optimal control and Bayesian inference, is promising paradigm for overcoming the challenges of RI. [5]. In Cal, the optimality of state and control trajectory is defined by introducing optimality variables rather than explicit costs. Consequently, an optimal control problem can be formulated as probabilistic inference problem. In particular, maximum entropy (MaxEnt) control [6, 7] is equivalent to a variational inference problem using the Kullback-Leibler (KL) divergence. MaxEnt control has entropy regularization of control policy, and as a result, the optimal policy is stochastic. Several works have revealed the advantages of the regularization such robustness against disturbances [8], natural exploration induced by the stochasticity [7, 9], fast convergence the MaxEnt policy gradient method [10].
On the other hand, the KL divergence is not the only option available for variational inference. In [11], the variational inference was extended to Rényi a-divergence [12], which is a rich family of divergences including the KI. divergence. Similar to the traditional variational inference, this extension optimizes lower bound of the evidence, which is called the variational Rényí bound. The parameter of Rényi divergence controls the balance between mass-covering and zero-forcing effects for approximate inference [13]. However, if we use Rényi divergence for Cal, it remains unclear how affects the optimal policy, and a natural question arises: what objective does Cal using Rényi divergence optimize?
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Contributions The contributions of this work are as follows:
1. We reveal that Cal with Rényi divergence solves a log- seg-poobbbility (LP) regularized risk- sensitive control problem with exponential utility [14] (Theorem 2). The order parameter a of Rényi divergence plays a role of the risk-sensitivity parameter, which determines whether the resulting policy is risk-averse or risk-seeking. Based on the result, we refer to Cal using Rényi divergence as risk-sensitive Cal (RCal). Since Rényi divergence includes the KI divergence, RCal is unifying framework of Cal. Additionally, we show that the risk-sensitive optimal policy takes the form of the Gibbs distribution whose energy is given by the Q-function, which can be obtained by solving soft Bellman equation (Theorem 3). Furthermore, this reveals several equivalence results between RCal, MaxEnt control, the optimal posterior for Cal, and linearly- solvable control [15, 16].
2. Based on RCal, we derive risk sensiiveee RL methods. First, we provide a policy gradient method [17-19] for the regularized risk- sensitvvee RL (Proposition 7). Next, we derive the risk- counterpart of the soft actor-critic algorithm [7] through the maximization of the variational Rényi bound (Subsection 4.2). As the risk-sensttivity parameter vanishes, the proposed methods converge to REINFORCE [19] with entropy regularization and risk-neutral soft actor-critic [7]. respectively. One of their advantages over other risk- sensitive approaches, including distributional RI. [20, 21J, is that they require only minor modifications to the standard REINFORCE and soft actor-critic. The behavior of the risk-sensitive soft actor-critic is examined via an experiment.
3. Although the risk-sensitive control induced by RCal has LP regularization of the policy, it is not entropy, unlike the MaxEnt control with the Shannon entropy regularization. To bridge this gap, we provide another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We prove that the resulting optimal policy and the Bellman equation have the same structure as the LP regularized risk risk-sensitive control (Theorem 6). The derivation differs significantly from that for the LP regularization, and for the analysis, we establish the duality between exponential integrals and Rényi entropy (Lemma 5).
The established relations between several control problems in this paper are summarized in Fig.
Related work The duality between con- trol and inference has been extensively studied [15, 22-26]. Inspired by Cal, [27, 28] reformulated model predictive con- trol (MPC) as variational inference prob- lem. In [29], variational inference MPC us- ing Tsallis divergence, which is equivalent to Rényi divergence, was proposed. The difference between results and theirs is that variational inference MPC infers feed- forward optimal control while RCal infers feedback optimal control. Consequently, the equivalence of risk- control and Tsallis variational inference MPC is not derived, unlike RCal. The work [30] proposed an EM-style algorithm for RL based on Cal, where the resulting policy is risk-seeking. However, risk- averse policies cannot be derived from Cal by this approach. Our framework provides the equivalence between Cal and risk-sensitive control both for risk-seeking and risk-averse cases.
VI wish KL divengenee
MaxEnt control
Cal
VI with Rényi (1 I diveieereoooo (RCal)
Deterministic system
- 0
(Ineurly solwbbee
LP regularized risk-sensitive control
Policy cuuverges as + -1
Same structure
(VI: varlacional inferene))
Rényi entropy regularized risk-aensitive control
equivaleure
Figure 1: Relations of control problems.
Risk-averse policies are known to yield robust control [31, 32]. and risk-sccking policies are useful for balancing exploration and exploitation for RL [33]. Because of these merits, many efforts have been devoted to risk-sensitive RL [19, 34-36]. In [37], risk-sensitive RL with Shanmon entropy regularization was investigated. However, their theoretical results are valid only for almost resk-nl cases. Our results imply that LP and Rényi entropy regularization are suitable for the risk-sensitive RL.
In [16], risk-sensitive control whose control cost is defined by Rényi divergence was investigated, and it was shown that the associated Bellman equation can be linearized. However, it is assumed that the transition distribution can be controlled as desired, which not satisfied in general as pointed out in [38]. On the other hand, our result shows that when the dynamics is deterministic, LP

---

and Rényi entropy regularized risk-sensitive control problems are linearly solvable without the full controllability assumption of the transition distribution.
For simplicity, by abuse of notation, we write the density (or probability mass) functions of random variables 2, y as p(z), p(y), and the expectation with respect to p(z) is denoted by Ep(z)- For set S, the set of all densities on S is denoted by P(S). Rényi entropy and divergence with parameter a > 0, * are defined as (p) = a(1-a) log (up(w)=0) p(u)ddu) Da (Pr P22
Notation
For the factor a(1-a) of Ha. we follow [39, 40] because this choice is convenient for the analysis in Subsection 3.2 rather than another common choice 1/(1 a). We formally extend the definition of H.s a < O. Denote the Shannon entropy and KL divergence by H1 (p), Di (Pr (P2)). respectively because lima-11 Ha = (p),,, lima->1 Da (P: (Ipz) D1 (pillpp) PP2) For further properties of the Rényi entropy and divergence, see e.g., [41]. The set of integers (k, k 1, ,,...., 8). & < is denoted by [k; s]]. A sequence denoted by Tk:s. The set of non-negative real numbers denoted by Rz0-
- log (upp (M)))(()>)))) (1(2)" p
{2R, {0k, #kt1...,,..
2 Brief introduction to control as inference
First, briefly introduce the framework of Cal. For the de- tailed derivation, see Appendix A and [5]. Throughout the pa- per, 2t and u, denote X-valued state and U-valued control vari- ables at time f, respectively, where X C R U C R", and μz(U) O. Here, jzz denotes the Lebesgue measure """., The initial distribution is p(au), and the transition density is denoted by p(z++1 [ uz), which depends only on the current state and control input. Let T >> 0 be finite time horizon. Cal connects control and probabilistic inference problems by introducing optimality variables O2 E {0, 1} in Fig. 2. For
Figure 2: Graphical model for Cal.
q: : X x U R20, CT + Rzo, which will serve as cost functions, the distribution of O is given (Ia, u4) (c 1 E [0. T 1] and p p(Or [ O, = 1, then (Zz, 2x) at time is said to be "optimal." " The control posterior p(u, [2,, OxT = 1) is called the optimal policy. Let the prior of ue be uniform: p(u,) 1/ (U), Vu, E U. Although this choice common for Cal, the arguments in this paper may be extended to non-uniform priors. Then, for the graphical model in Fig. 2, the distribution of the optimal state and control input trajectory
by p(O, 1 Me) exp(-cc
(). If
T = (Fo.T, -1) satisfies
T-1
P(T 1) x p(ro) II p p (re, ue)
pOO p(0r=1107 1
II
p(O, = 1/ ut)
T-1
T
= p(zo)
II t=
p(
[Te, uz)
exp
(-rr(oT) - Σ Ca un)
(I)
t=
For notational simplicity, will drop 1 for O₂ in the remainder of this paper.
The optimal policy p(uelzr, O..) can be computed in a recursive manner. To this end, define
Qe(a,, uz) log p(Ourlas, Ve(x;) == log p(Or(l)))))) (U)
(2)
which play a role of value functions. Then, the following result holds.
Proposition 1. Assume that (U) < co and let c, u)) == aa ca un) + log fL (U). Assume further the existence of density functions p(zo) and p(1241/ u.) for any E [0. T 11 Then, holds that
pluekar, OCT = = exp (- (e(( u.) + Vz(x))), Va, E E Vu, E U,
(3)
When considering discrete variables the assumption (u. < is replaced by the finiteness of the set U, and the existence of the densities is not required.
3

---

where
Ve(1z) - log V ( ue de)) vt E [0, II. Vr(eT) = cπ(TT),
log Qr( ue) = ci (2 u+) - log Bp (exp(-V++1(e++)))) W |0,T 1]
(4)
(5)
4
The recursive computation (4), (5) is similar to the Bellman equation for the risk-seeking control. However, not still clear what kind performance index the optimal trajectory p(r(Orr) optimizes because (4) does not coincide with that of the conventional risk-seeking control. An indirect way to make this clear is variational inference. Let us consider finding the closest trajectory distribution (r) to the optimal distribution P(T]Oo::) The variational distribution is chosen
T-1
p" (T( p(x0)
II p( (aa, ue) (x)). 2mo
where nT(-1x,)= E P(U) is the conditional density of un given æy and corresponds to a control policy. Then, the minimization of the KL divergence D2 (p" (r)):)()(o::r) is known to be equivalent the following MaxEnt control problem:
T-1
minimize Epp (r) Cr (TT) + Σ (c((Fr Hi (m(Te( (me)
Especially when the system p(xe+1l Ue) is deterministie, the minimum value of Di(p" (r (r)llp(((oor) is O, and the posterior p(u, [re O+r)) yields the optimal control (7). As mentioned in Introduction, this work uses Rényi divergence rather than the KL divergence. Morcover, we characterize the optimal posterior p(u (x,, Ocr) more directly even for stochastic systems.
Control as Rényi divergence variational inference
In this section, we address the question of what kind of control problem is solved by Cal with Rényi divergence and characterize the optimal policy.
3.1 Equivalence between Cal with Rényi divergence and risk-sensitive control
Let > -1, 1) Then, Cal using Rényi variational inference formulated as the minimization of (F)()()(7 with respect to p" in (6). Now, we have
dd (1+v(p" (Ip(1(0:r)) = log f pT (T) Our)) "dr + log p(Oo:r).
(8)
(Variational Rényl bound)
That is, Cal with Rényi divergence equivalent to maximizing the above variational Rényi bound. Moreover, by (1), it holds that
log ʃ p (T) + "([T, (ur,) "d dd
= log /DTTT
p( I p(a++ (a,, u (()
dr
II p(20)
p(# (re, un) exp (-or(ET) - ETD ci ve)
log / (T( exp ner(ET)
(xp)((pe((())) + 1 Σ (1(2,, uz) + log कI (u)l))))) dr (Z))
Consequently, we obtain the first equivalence result in this paper.
Theorem 2. Suppose that the assumptions in Proposition hold. Then, for any -1, v + 0, the 1)) with respect to pd (6) is equivalent to
minimization of D1+0 (v [pp(-[Oo.z
T-1
minimize log (ps(r) exp (TT( + n Σ (m))
(a,, ue) + log Te m=(ur((t))
(9)

---

Problem (9) is risk-sensitive control problem with the log- propabbbbbllity regularization log (u, (2:) of the control policy. Let (7) be the exponent in (9). Then, 5 log Elexp(n)(F))]
(T))
[r) + O(n)), where Var[-] denotes the variance [42]. Hence, > 0 (resp. < 0) leads to risk-averse (resp. risk-seeking) policies. As 17 goes to zero, the objective in (9) converges to the risk-neutral MaxEnt control problem (7).
3.2 Derivation of optimal control and further equivalence results
In this subsection, we derive the optimal policy of (9) and give its characterizations. For the analysis, we do not need the non-negativity of the cost Cz. We only sketch the derivation, and the detailed proof given in Appendix B. Similar to the conventional optimal control problems, we adopt the dynamic programming. Another approach based on variational inference will be given in Subsection 4.2.
Define the optimal Sstate-yvalu function V2 X + R and the Q-function Q2 X x U + R follows:
T-1
Vi(xz) = inf 1 (v. (=. 7
log E (r(ze) exp (CT(TT)) +
Σ (co(ss, us) + log n. (w,)
(10)
1 2e(xe, ue) = ai (z1, uz) + log Eptrenl [exp(TVe=1(<<x)) E
(11)
17
and Vi == er (xT).. Then, it can be shown that the Bellman equation for Problem (9) is
Vi(11a) = log /
exp (-2 u')) du'
+ inf D (Tt (+1x:)),
(12)
(((())))
where oi (1g (1 xx) ;= exp (-Qr wz / Z (xz), and the normalizing constant is assumed to fulfill
Ze (are) == ( v')) du' < co. Since (Te (+12z)]] (-)ae) attains its minimum
value 0 if and only if TH. (12.)))) = के (+lax), the unique optimal policy that minimizes the right-hand
side of (12) is given by nī (-12x)) and
Vi(1a)) - log [[ exp exp (-2 w'))
-lg
du' du'
कī (uel(x)) exp (x ue) (z+)). (13)
Because of the softmin operation above, the left equation in (13) is called the soft Bellman equation.
Theorem 3. Assume that Ju exp (r v()) du' < oo holds for any E T -1] and X. Let n > 1, 11 + O. Then, the unique optimal policy of Problem (9) is given by (13). Especially when the dynamics is deterministic, i.e., p(x+41 | uz) 8(0141 Je(Ii, we)) for some Sa x + X and the Dirac delta function 8, it holds that
Ce(Ee, un) = ci ux) + Vi+1 (f((e, u)),
(14)
and the optimal policy of the MaxEnt control problem (7) solves the LP LP-regularized risk-sensitive control problem (9) for any 1) > -1, 17 } 0.
Assumption Ju exp ( 2a n')) du' < oo is satisfied for example when e is bounded for any t [0 and (U) co. The linear quadratic setting also fulfills this assumption; see (16).
Theorem suggests several equivalence results:
RCal and MaxEnt control for deterministic systems. First, we emphasize that even though the equivalence between unregularized risk-neutral and risk-sensitive controls for deterministic systems is already known, our equivalence result for MaxEnt and regularized risk-sensitive controls is nontrivial. This is because the regularized policy कं makes a system stochastic even though the original system is deterministic, and for stochastic systems, the unregularized risk-sensitive control does not coincide with the risk-neutral control. This implics that the optimal randomness introduced by the regularization does not affect the risk sensitivity of the policy. This provides insight into the robustness of MaxEnt control [8]. Note that [43] mentioned that the MaxEnt control objective can be reconstructed by the risk-sensitive control objective under the heuristic assumption that the cost follows uniform distribution. However, this assumption is not satisfied in general. Our equivalence result does not require such an unrealistic assumption.
RCal and optimal posterior. Although the optimal posterior p(u, (z,, OuT) yields the MaxEnt con- trol for deterministic systems as mentioned in Section 2, it is not known what objective p(u,/z,, OuTr
5

---

optimizes for stochastic systems. Theorem 3 gives a new characterization of p(ue|a,, OuT). By formally substituting 7 = into (11). the Bellman equation for computing nH becomes (4), (5) for the optimal posterior p(u,) zy. O..r). Note that even if the cost function e in (9) is replaced by in Proposition } is still optimal. Therefore, by taking the limit as 1j > the policy of (u (2+) in Theorem 3 converges to p(u, ( Our)) and in this sense, the policy p OCT) is risk-seeking.
Corollary 4. Under the assumptions in Proposition I, holds that
lim oi (uular) = =xx((-22 (x u4) + Vi(x:)) = p(ue z2, Te, OuT = =1) 1),
(15)
" n
where V, and Qr are given by (11), (13) with 1j -1
4
RCal for deterministic systems and linearly-solvable control. For deterministic systems, by the transformation Ex(xx)
(aze)) )) the Bellman equation (14) becomes linear: (xxx) J exp(-c; (#,, F (Zz u') )'))"" That is, when the system deterministic, the I.Prreularizeed risk-sensitive control, or equivalently, the MaxEnt control is linearly solvable [15, 16, 44], which enables efficient computation RL. Even for the MaxEnt control, this fact seems not to be mentioned explicitly in the literature.
RCal and unregularized risk-sensitive control in linear quadratic setting. Similar to the un- regularized and MaxEnt problems [45, 46], Problem (9) with a linear system p(z u.) = N ( A + Beu Mn) and quadratic costs C2 we) = (a) Qe + u Re er = QFQTr/22 admits an explicit form of the optimal policy:
7. (u|z)) = .N(ul - (Re B, 11 I, (1 - 111 BB) B.) "B, I4 - - A (R2 + (I - -1 (BB)))
(B)=
(16)
Here, (-Ilu, Σ) denotes the Gaussian density with mean μ and covariance Σ. The definition of II and the proof are given in Appendix C. In general, the mean of the regularized risk-sensitive control deviates from the unregularized risk-sensitive control. However, in the linear quadratic Gaussian (LQG) case, the mean of the optimal policy (16) coineides with the optimal control of risk-sensitive LQG control without the regularization [47].
3.3 Another risk-sensitive generalization of MaxEnt control via Rényi entropy
The Shannon entropy regularization (I-eH((me--lzz)) of the MaxEnt control problem (7) can be rewritten as Ellog गy (u, [x,]]. In this sense, the risk-sensitive control (9) is a natural extension of (7). Nevertheless, for the risk-sensitive case, the interpretation of log Te (ue (2xz) as entropy is longer available. In this subsection, we provide another risk-sensitive extension of the MaxEnt control. Inspired by the Rényi divergence utilized so far, we employ Rényi entropy regularization:
minimize 1 [Te) "
log (p)(() exp net (TCT(ET) +
Σ (al ue) H1 -HI-
(17)
where 7y E TR \ (0, and (e(+)x) E (U) = fo E P(U)) So ((u)) < 00), Ya, which implies | (T( < co. As tends to zero, (17) converges the MaxEnt control problem (7).
Define the value function Vi and the f-function 2, associated with (17) like (10) and (11). Then, as in Subsection 3.2, the following Bellman equation holds. The derivation is given in Appendix E.
inf TELL (L) ( log Le (mall (u') exp(1 u ddd)' - -11-n(m(((1))))
[((x))
(18)
For the minimization in (18), we establish the duality between exponential integrals and Rényi entropy like in [40] because the same procedure for (12) cannot be applied.
Lemma 5 (Informal). For B3, E IR. [0} such that B < q and for g U - IR, it holds that
{ log [/.
log B
exp(3g(u))du =
inf PEL" - (U)
exp((o(u)))((udd
7 3 - (p)
H1
(19)

---

and the unique optimal solution that minimizes the right-hand side of (19) is given by
p(u) exp (-(y- ())((() Vu E U.
(20)
exepp( -=)g(("))ddd
For the precise statement and the proof, see Appendix D. By applying Lemma 5 with , = 1, Y 17 (18), we obtain the optimal policy of (17) as follows.
Theorem 6. Assume that o, is bounded below for any t [0, []. Assume further that for any x E X
and E [0.T
T 1], it holds that fu exp (- (-2,(, ) du' < 00, exp (-(1- 2x, u') du' <
co. Then, the unique optimal policy of Problem (17) is given by
(ue((())) = ((x1) exp (-21 u)), E [0, 11, Vxz E X, Vu E EU, (21)
where 21(xz)) = fo exeppp 2, ut )dd', and it holds that
-1
log J( exp (1)))))((z,,))))) dul
vi(x1) 1 1-11 17
Vt 10, 11, Vx, E X.
(22)
Recall that the LP regularized risk-sensitive optimal control is given by (11), (13) while the Rényi entropy regularized control is determined by (21), (22), and 2,(1e, ut) = (22, u)) + log Ep(z (lr).)) (1c.:))exp((///4112 ((x4())(=((4422] Hence, the only difference between the risk- sensitisiiie con- n
trols for the LP and Rényi regularization the coefficient the soft Bellman equations (13), (22).
Risk-sensitive reinforcement learning via RCal
Standard RL methods can be derived from Cal using the KL divergence [5]. In this section, we derive risk-sensitive policy gradient and soft actor-critic methods from RCal.
4.1 Risk-sensitive policy gradient
In this subsection, we consider minimizing the cost (9) by a time-invariant policy parameterized as Te (u]ar) = x. (8) (u]x). 0 E Re= Let Co(T) == er (xT) + 2-'1 (cx ue) + log x (0)) (mx (x:)) and po be the density of the trajectory t under the policy (()), Then, Problem (9) can be reformulated as the minimization of J(0)/n where J(Ø)
To optimize J(0)/n by gradient descent, we give the gradient (f). The proof is shown in Appendix F.
(T( exp(n)C(()
Proposition 7. Assume the existence of densities p(I++1 12,, Uz), p(æo)- Assume further that - (0) differentiable in 0, and the derivative and the integral can be interchanged as VoJ(O) V Volpo(T) exp(nCo(T)))d Then, for any function b. R -> R. it holds that
VoJ(O) = (17 + P)
Vo log n (0) (ue(2)))
1=
T-1
(o( Ws) + log π. (0 (us(())))) - Morr H|
exp
noT(ET)
+ n
(23)
The function b referred as baseline function, which can be used for reducing the variance of an estimate of VoJ. The following gradient estimate of [(f)nn is unbiased:
T-1 v lo (9) (wslas)) { exp NTT((T)) + +1 Σ (ca(Ens) u.) + log (0) (u.)os))) Man))
Σ
This is almost the same as risk-sensitive REINFORCE [19] except for the additional term log (0) (ux(ar)) In the risk-neutral limit v - 0, this estimator converges to the MaxEnt policy gradient estimator [5].
7

---

4.2 Risk-vensitive soft actor-critic
In Subsection 3.2, we used dynamic programmiing to obtain the optimal policy {x;). Rather, in this section, we adopt a standard procedure of variational inference [48]. First, we find the optimal factor Te for fixed .. * as follows. The proof is deferred to Appendix G.
Proposition 8. Fort t E [0,T - 1I. let Tg. S + be fixed. Let 1) > -1, o O. Then, the optimal factor के == arg min E.P(I) D1+n (p" 1p(-100.r = 1)) is given by
-1/
1 (x(
(
s-t+1 To (u)
my (ux (Fr)
Exp (1AAnteeeeee
M! p(O, (Ies Us)
(24)
where (z4) is the normalizing constant.
By (24), the optimal factor n. is independent of the past factors T. 8 E [0, t - 1]. Therefore, the variational Rényi bound in (8) is maximized by optimizing ne in backward order from t = T to t = 0, which is consistent with the dynamic programming. Associated with (24), we define
[(100 Lont (ug(ta.)
1 1
Vi" (x:)
log E (2 uer-
p(OO((TT) II. smt ((O,/ze,, us)
].
log (o alme)
exp ner (2or)
Σ (co(2 u.) log IT (u)(()))))
(25)
which is the value function for the policy {IT IT satisfying the following Bellman equation.
1 log E (mullr))
((
(u((((17)) (p(Orlze,, uz) ) E Ep( (exp(TV71((444))
Ve (xz)
(26)
7
log E. (mr exp (n7ca ue) + 17 log Te (wal2z))) Epiress m.nu)) [exp(nV2, (1x-
By the value function, क. (wel can be written
pl p(O,)1z,,2)) ue) Ze(xt)
[
I Ta(uak(s)
"7 -1/m
ai (ur (ee)
Ep Epter--r -1
IT- p(O, 1 U.))
p(OO (10, Mt) (x()r)
(27)
Next, we define the Q-function for {T. Ma=2 as follows:
1 log Bptrens [exp(nVA
O7 (2t, uz) logp(O,(1z,, ux) + 1 Then, it follows from (26) and (27) that
(28)
ve (xx) = log (u.lze)) (uo())))))) exp(n17 (x,, uc)),
(29)
exp(- (z,, uz)), Z (zx)) / exp (-Q ( v')) du'.
n/ (ue(x:) (xe),)
(30)
Especially when Te ( which coincides with the soft Bellman equation in (13). In summary, in order to obtain the optimal factor o,, it is sufficient to compute V," and Q in a backward manner.
(2)) = n. [u (r2). it holds that V (2x) = - log [ʃ exp( (I u/ d''),
Next, we consider the situation when the policy is parameterized as (uglz)) 0 T and there (0) ... To accommodate this situation, we utilize the variational Rényi bound. One can easily see that the maximization of the Rényi bound in (8) with respect single factor T, equivalent the following problem.
no parameter / that gives the optimal factor n;
minimize me 77
log (xp( [Ere(ueled [me (x.)" exp(n (xx, u))l]
(31)
8

---

This suggests choosing 0 that minimizes (31) whose ve is replaced by (0) Note that this is further equivalent to
exp(p exp(-Q0 (1t: )
minimize (x.) Ditn Th (0) (1(z)
(32)
(z(x))
We also parameterize VT and Q7 as V(v), Q(+) and optimize v. ø so that the relations (28), (29) approximately hold. To obtain unbiased gradient estimators later, we minimize the following squared residual error based on (28), (29), and the transformation Tv)(w) == (en 1)/n, IR:
{ To (Z), un) - c(x22, un) 1p (1 (Eet1)) (2(000)))))
Jo(6)= == Ep (e))
2
2 Tr (v(() - (url0))) [n (Q (x),, ue) + log x (uel ) 27
(vb) = Ep (2r)
Using Q(+) and Tr. we replace (31) with the following equivalent objective:
(+) = Ep (ex) (o. [T, (Q(()) (z,, uz) + log (ugl)))]
(33)
Noting that lim T, (x(()) k(0) for x. IR + I. the risk sensitivity 1 goes zero, the objectives Jq. Jv, Jn converge to those used for the risk-neutral soft actor-critic [7]. Now, we have
voJa(Ø) (p
(VoQ(0) (z,, ue)) exp exp(n)(() (ze, ue) n ue)
x
ue) c(a,, u))) -F Ep |
(34)
VoJv((6) = (xr) (Vo: (xx)) (xm)))xxpli ex)))) (xz))
x (V,((( - E, [Ty (Q(6) un) + log n (0)
[
(35)
VoJ+(0) = (1+ Bp (Vo log - (9) (we (r.) ( T., (+) (Ix, we) + log ग (0) (ur [r:))] (36)
Thanks the transformation Tnp the expectations appear linearly, and an unbiased gradient estimator can be obtained by removing them. By simply replacing the gradients of the soft actor-critic [7] with (34)-(36), obtain the risk-sensitive soft actor-critic (RSAC). It is worth mentioning that since RSAC requires only minor modifications to SAC, techniques for stabilizing SAC, e.g., reparameteri- zation, minibatch sampling with replay buffer, target networks, double Q-network, can be directly used for RSAC.
5 Experiment
Unregularized risk-averse control is known be robust against perturbations in systems [32]. Since the robustness of the regularized cases has not yet been established the- oretically, we verify the robustness of policies learned by RSAC through a numerical example. The environment Pendulum-vl in OpenAl Gymnasium. We trained control policies using the hyperparameters shown in Appendix There were no significant differences in the control perfor- mance obtained or the behavior during training. On the other hand, for each 77, one control policy was selected and was applied to a slightly different environment without retrain- ing. To be more precise, the pendulum length which is 1.0 during training, is changed to 1. 1.25 and 1.5; See Fig. 3. In this example, it can be seen that the control policy obtained with larger 1 has a smaller performance degradation due to environmental changes. This robustness can be considered benefit of risk-sensitive control.
[-
4
1-
4 1-11
6:0
5:0
030
230
-0:01
0.01
Figure 3: Average episode cost for
RSAC with some and standard SAC.
In Fig. 4, empirical distributions of the costs for different risk-sensitivity parameters 1) are plotted. Only the distribution for 7 0.02 does not change so much under the system perturbations. The

---

0902
0662
1413
-0.02
9 0001
-0.02
0 0.01
-0.02
- 0.0L
0.00
1.00
101
-0.01
Densily
SAC
SAC
BAC
0.00
0083
1:00.
200
400
1000
1200
200
600
800
1000 1200
400
600
800
1000 1200
Episode cost
Episodo cost
Episodo cost
(a) Pendulum length / 1.0 during training
(b) System perturbation ( 1.25
(c) System perturbation 1.5
Figure 4: Empirical distributions of the costs for different risk-sensitivity parameters n).
distribution for SAC (1) = 0) with 1.5 deviates from the original one (1 1.0), and another peak of the distribution appears in the high-cost area. This means that there is high probability of incurring a high cost, which clarifies the advantage of RSAC. The more risk-sceking the policy becomes, the less robust it becomes against the system perturbation.
Conclusions
In this paper, we proposed a unifying framework of Cal, named RCal, using Rényi divergence variational inference. We revealed that RCal yields the LP regularized risk-sensitive control with exponential performance criteria. Moreover, we showed the equivalences for risk-sensitive control, MaxEnt control, the optimal posterior for Cal, and linearly-solvable control. In addition these connections, we derived the policy gradient method and the soft actor-critic method for the risk- sensitive RL via RCal. Interestingly, Rényi entropy regularization also results in the same form the risk-sensitive optimal policy and the soft Bellman equation as the LP regularization.
From practical point of view, major limitation of the proposed risk senssivse soft actor-critic is its numerical instability for large (y] cases. Since 1) appears, for example, exp(1)(() (xz, uz)) in the gradients (34)-(36), the magnitude of n) that does not cause the numerical instability depends on the scale of costs. Therefore, using Pendulum-vl, in| that is larger than 0.03 results in the failure of learning due to the numerical instability. Although it is an important future work to address this issue, would like to note that this issue not specific to our algorithms, but occurs in general risk-sensitive RL with exponential utility. It is also important how to choose specific value of the order parameter + 7) of Rényi divergence. Since we showed that 77 determines the risk sensitivity of the optimal policy, we can follow previous studies the choice of the sensitivity parameter of the risk-sensitive control without regularization. The properties the derived algorithms also need to be explored in future work, e.g., the compatibility of function approximator for RSAC [49].
need to choose 7) depending on environments. In the experiment
Acknowledgments
The authors thank Ran Wang for his valuable help in conducting the experiment. This work was supported in part by JSPS KAKENHI Grant Numbers JP23K19117, JP24K17297, JP21H0475..
References
[1] Onésimo Hernández-Lerma and Jean B. Lasserre, Discrete-time Markov Control Processes: Basic Optimality Criteria, vol. 30, Springer- Verlag New York, 1996.
[2] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, MIT Press, second edition, 2018
[3] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine, "Learning to walk via deep reinforcement learning", in Robotics: Science and Systems, 2019.
10

---

[4] B. Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani. and Patrick Pérez. "Deep reinforcement learning for autonomous driving: survey". IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 1909-4926, 2022.
[5] Sergey Levine, "Reinforcement learning and control as probabilistic inference: Tutorial and review", arXiv preprint arXiv: 1805. 0005,00909, 2018.
[6] Brian D. Ziebart, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy, PhD thesis, Carnegie Mellon University, 2010.
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, "Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor", International Conference on Machine Learning. PMLR, 2018, pp. 1861-1870.
[8] Benjamin Eysenbach and Sergey Levine, "Maximum entropy RL (provably) solves some robust RL problems", in International Conference on Leurning Reprexentctiinss 2022.
[9] Tuomas Haarmoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine, "Soft actor-critic algorithms and applications", arXiv preprint arXiv: /8/1:/88/22 18150090 2018.
[10] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans, "On the global con- vergence rates of softmax policy gradient methods", in International Conference on Machine Learning. PMLR, 2020, vol. 119, pp. 6820-6829.
[11] Yingzhen Li and Richard E. Turner, "Rényi divergence variational inference", in Advances in Neural Information Processing Systems, 2016, vol. 29, pp. 1073-1081.
[12] Alfréd Rényi, "On measures of entropy and information", in Proceedings of the fourth Berkeley Symposium on Mathematical Statistics and Probubility, 1961, vol. 1, pp. 547-561.
[13] Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt, "Advances in varia- tional inference", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 8. pp. 2008-2026, 2019.
[14] Peter Whittle, Risk-Sensitive Optimal Control, John Wiley & Sons, Ltd., 1990.
[15] Emanuel Todorov, "Linearly-solvable Markov decision problems". in Advances in Neural Information Processing Systems, 2006, vol. 19, pp. 1369-1376.
[16] Krishnamurthy Dvijotham and Emanuel Todorov, "A unifying framework for linearly solvable control", in 27th Conference on Uncertainty in Artificial Intelligence, 201 1, pp. 179-186.
[17] Ronald Williams, "Simple statistical gradient- following algorithms for connectionist rein- forcement learning", Machine Learning, vol. 8, pp. 229-256, 1992.
[18] David Nass, Boris Belousov, and Jan Peters, "Entropic risk measure in policy search", in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 1101-1106.
[19] Erfaun Noorani and John S. Baras, "Risk- sensitive REINFORCE: A Monte Carlo policy gradient algorithm for exponential performance criteria", in 2021 60th IEEE Conference on Decision and Control (CDC). IEEE, 2021, pp. 1522-1527.
[20] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng, "Dis- tributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors". IEEE Transactions Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6594- 2022.
[21] Jinyoung Choi, Christopher Dance, Jung-Fun Kim, Seulbin Hwang, and Kyung-sik Park, "Risk-conditionee distributional soft actor-critic for risk- navigation", in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 8337-8344.
[22] Hilbert J. Kappen, "Path integrals and symmetry breaking for optimal control theory", Journal of Statistical Mechanics: Theory and Experiment, vol. 2005, no. 11, pp. P11011, 2005.
11

---

[23] Emanuel Todorov, "General duality between optimal control and estimation", in 2008 47th IEEE Conference on Decision and Control. IEEE, 2008, pp. 4286 4292.
[24] Hilbert J. Kappen, Vicenç Gómez, and Manfred Opper, "Optimal control as a graphical model inference problem", Machine Learning, vol. 87, pp. 159-182, 2012.
[25] Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar, "On stochastic optimal control and reinforcement learning by approximate inference", in Proceedings of Robotics: Science and Systems, 2012.
[26] Marc Toussaint, "Robot trajectory optimization using approximate inference", in International Conference on Machine Learning, 2009, pp. 1056.
[27] Masashi Okada and Tadahiro Taniguchi, "Variational inference MPC for Bayesian model-based reinforcement learning", Conference on Robot Learning. PMLR, 2020, pp. 258-272.
[28] Alexander Lambert, Fabio Ramos, Byron Boots, Dieter Fox, and Adam Fishman, "Stein variational model predictive control", in Conference on Robot Learning. PMLR, 2021, vol. 155,
pp. 1278-1297,
[29] Ziyi Wang, Oswin So, Jason Gibson, Bogdan Vlahov, Manan S. Gandhi, Guan-Horng Liu, and Evangelos A. Theodorou, "Variational inference MPC using Tsallis divergence", in Robotics: Science and Systems, 2021.
[30] Yinlam Chow, Brandon Cui, MoonKyung Ryu, and Mohammad Ghavamzadeh, "Variational model-based policy optimization", arXiv preprint arXiv:2006.05443; 2020.
[31] Marco C. Campi and Matthew R. James, "Nonlinear discrete-time risk-sensitive optimal control", International Journal of Robust and Nonlinear Control, vol. 6, no. 1, pp. 1-19, 1996.
[32] Ian Petersen, Matthew James, and Paul Dupuis, "Minimax optimal control of stochastic uncertain systems with relative entropy constraints". IEEE Transactions on Automatic Control, vol. 45, no. 3, pp. 398-412, 2000.
[33] Brendan O' D'onoghue, "Variational Bayesian reinforcement learning with regret bounds", in Advances Neural Information Processing Systems, 2021, vol. 34, pp. 28208-2822 28221.
[34] Vivek S. Borkar, "Q-learning for risk-sensitive control", Mathematics of Operations Research, vol. 27, no. 2. pp. 2002.
[35] Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang, "Exponential Bellman equation and improved regret bounds for risk-sensitive reinforcement learning", in Advances in Neural Information Processing Systems, 2021, vol. 34, pp. 20436-20446.
[36] Javier Garcia and Fernando Fernández, "A comprehensive survey safe reinforcement learning". Journal of Machine Learning Research, vol. 16, no. 1, pp. 1437-1480, 2015.
[37] Tobias Enders, James Harrison, and Maximilian Schiffer, "Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts", arXiv preprint 099992. 2024.
[38] Kaito Ito and Kenji Kashima, "Kullback-Leilee control for discrete-time nonlinear systems on continuous spaces", SICE Journal of Control, Measurement, and System Integration, vol. 15, no. 2. pp. 119- 2022.
[39] Friedrich Liese and Igor Vajda, Convex Statistical Distances, Teubner, Leipzig, 1987.
[40] Rami Atar, Kenny Chowdhary, and Paul Dupuis, "Robust bounds on risk-sensitive functionals via Rényi divergence", SIAM/ASA Journal on Uncertainty Quantification, vol. 3, no. 1, pp. 18-33, 2015.
[41] Tim Van Erven and Peter Harremos, "Rényi divergence and Kullback-Leiihe divergence", IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797-3820, 2014.
12

---

[42] Oliver Mihatsch and Ralph Neuneier, "Risk-sensitive reinforcement learning", Machine Learning, vol. 49, pp. 267-290, 2002.
[43] Erfaun Noorani, Christos Mavridis, and John Baras, "Risk-sensitive reinforcement learning with exponential criteria", arXiv preprint arXiv:2212.0900 2023.
[44] Krishnamurthy Dvijotham and Emanuel Todorov, "Inverse optimal control with linearly- solvable MDPs", in Proceedings of the 27th International Conference on Machine Learning, 2010, pp. 335-342.
[45] Kaito Ito and Kenji Kashima, "Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges", IEEE Transactions on Automatic Control, vol. 69, no. pp. 1536-1551, 2023.
[46] Kaito Ito and Kenji Kashima, "Maximum entropy density control of discrete-time linear systems with quadratic cost", To appear in IEEE Transactions Automatic Control, 2025, arXiv preprint arXiv:2309.
[47] Peter Whittle, "Risk-sensitive Imear/qudrrati//Gausiin control", Advances in Applied Proba- bility, vol. 13, no. 4, pp. 764-777, 1981.
[48] Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
[49] Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour, "Policy gradient methods for reinforcement learning with function approximation", in Advances in Neural Information Processing Systems, 1999, vol. 12, pp. 1057-1063.
[50] Antonin Rafiin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann, "Stable-baselinest Reliable reinforcement learning implementations", Journal of Machine Learning Research, vol. 22, no. 268, pp. 1-8, 2021.
[51] Diederik P. Kingma and Jimmy Ba, " "Adam: A method for stochastic optimization", arXiv preprint arXiv: 2014.
13

---

A More details on Control as Inference
In this appendix, we give more details on Cal. As mentioned in (1), the distribution of the state and control imput trajectory given optimality variables satisfies
p(T(Oo:T) p(T, Oo.r)
T-1
II p(O, un)
T-1 p(rr) II pla 1l u)p(p)))
p(OrlaT) (T)
where p(u) (U) and p(T, Oo.r) is defined so that
P(T E B, Oor 00:T) / p(T, o0:r)dT
for any Ourr E 1) and any Borel set B, where IP denotes the probability. Therefore, we have
T-1
p(T 1) p(ro) II p(1 ue) t=
exp (-or(ar)
Σ calorsu))
The posteriorp(u, Oz. 1) given the optimality condition OCTT = 1 is called the optimal policy, We emphasize that the optimality p Ou.T is defined by the condition Ou. rather than by introducing a cost functional, unlike (", (zz) in (13). In the following, we drop for O,
The optimal policy can be computed follows. Define
(x ue) = p(O ((
(37)
(e(xx) = pl ((Orrla)))
(38)
Then, holds that
(((x))= = p00= p(Orrlo ()p(ul/:)du = Be (a(r,,)))(( (p)(u))du (L) S Be ue)du.
(39)
In addition, we have
Be(x = ((pr,)))p(O.r
(1, u.) p1 uc)
=p(O, u²)
p(C us d
p(O.1zt, u²)
(e+1( dx++1,
(40)
(x(T)) (x0))))))) exp(-cr(Fr)),
where we used
p(O 1 u4)
pl 0(0411:r dx)2141
| p(0 u ( ur ddr++
f p( 1004))p(24111) dx
In terms of 8 and Çe, the optimal policy can be written as
p(ue: u., O.T)) p(1e, OLT) p P(O,T Ut)
puu, Fr, O.T))
p(u))z))
p(O Belxe, un)
(41)
pr. (x))
14

---

Next, by the logarithmic transformation, we define
Qe ue) == log ( u.)
(42)
Uz(U)
Ve(zz) == log Ce (21).
(43)
Then, by (41), the optimal policy satisfies
p(uelae. Ou.)) exp Qi un) + Ve(zz))
(44)
By (39), it holds that
log 1/8 exp((=Q we)
Ve(x,) -lg
(45)
By using (40), obtain
exp( Qep(-Qu(Ter))) u)))L = exp(-cc(z;,un) / Se+1 ue dd
which yields
Qe(12, ux) (I ue) log Pl (exp(-V++1
(46)
Here, we defined (Ia, uz) == (x u4) + (U)) In summary, Proposition holds.
B Proof of Theorem 3
This appendix devoted to the analysis of the following problem:
log exp (1OT((T)) + Σ (c us) + a slog Te (wol )
T-1
1
minimize (mr)
(47)
7)
subject
1 fe(Jes ues we), ur E U, V1 E O
(48)
~ (-1x) given 30 Pro.
(49)
(50)
Here, T- is an independent sequence, Zo independent of (w:), > the regularization parameter, and 17 is the risk-sensitivity parameter satisfying 7 ---1, * 0. Note that we do assume the existence of densities p(r++1| 4t), p(#o). To perform dynamic programming for Problem (47), define the value function and the Q-function
T-1
(2,2,, u.) + +elog log n. (u, (1.) ((1))r)))
Vi(2) = inf (M,)T=
log E exp nor(IT)) +
Σ
21 = x
t [0, 1], X.
(51)
Vi((x) = (x)), x x x
Qx(x,u) = (x, u) + log E
wr)))]l te [0, - 1I, xEX, U
(22)
Then, under the assumption that f exp policy of Problem (47) is given by
(-2c 2c(mmuu)) du' oc, we prove that the unique optimal
(n(m,))
exp
n (u|z) = Ju exp
t [0, 1], a E U, x
(53)
Ce(u,ur)
du
15

---

First, by definition, we have
J Te [T((ul))B exp ne u) + en log (u(a)) + ()TT(TT)
Vi(x)
inf (Fo)ame 1)
log
T-1
Σ Ca(2s, Us) + t + ma(uslls)))
7
24 = x. the = 1 d2
1 log / (e(u(z) exp (1aa u) + ong mol
inf bys 11 1)
x E exp nor(xr) +n Σ (colzs) Us) + e (tg
xt =x, du
inf log /, Te (u(o)) (ne u) + eng (u))))) [exp(>Ve+] (xxp wr]]]] du
By the definition of the Q-function (52), we get
Vi(x)=
inf
log
exp(en (xp(t())))pp(((((( u))du
(.(E)))(())
- (,(o,,)) ))
log (ma(ulz:)) lten
inf
exp
du
m((Z))P(U)
log (foox ( Ox(x,)e)
/ Te ( +tn (ulzx)='duu
inf +.(())))(U)
du'
E
-=llog S/o0 exp
2((x, dul
+ inf (=()))))(U))
eD (Te (-1z)).
Since D1 Den (Te(- (-0([1=)]]] (+1x)) attains its minimum value 0 if and only if m((12x) m (+(rr, we
conclude that
Vi(x) = - =slog log J, exp
(-Lotr.)))
du' Vor E
(54)
and the unique optimal policy of Problem (47) is given by (53). Moreover, प. can be rewritten as
Vi(xa)
# (ular) exp
E [0, - 1], u U. xxX.
(55)
E
When considering the deterministic system Te+1 fe uz), immediately obtain the relation (x = ce(x, u) + V (x,,)))
(56)
On the other hand, the unique optimal policy of the MaxEnt control problem:
minimize
E
cT (XT) + Σ ((((), -eHa
(57)
(Te)
also given by (55) whose Q-function (52) replaced by
Q1(2, w) (x, u) + E[Ve+1 u, wx))]
Therefore, when the system is deterministic, the Q-function of the LP regularized risk-sensitive control problem (47) coincides with that of the MaxEnt control problem (57). Consequently, the
optimal policy of Problem (57) solves Problem (47) for any n > -1, 1) o for deterministic systems.
16

---

C Linear quadratic Gaussian setting
In this appendix, derive the regularized risk-sensitive optimal policy in the linear quadratic Gaussian setting.
Theorem 9. Let p(Ez+1lze,u)) = NA + B,u, Ex) and ca(x,, un) = (a Qi: + ui R cr(eT) = z QTIT where Σt Qu, and Rs are positive definite matrices for any and N (m. 2) denotes the Gaussian distribution with mean p and covariance E. Let X R",, U = RR Assume that there exists a solution [11, [1le to the following Riccati difference equation:
11, = Qi + 14+1l - + BeR, B, 11+1) Ae, Vt E [0, 1],
(58)
1 = Qr:
(59)
such that x,- n positive definite for any E [0. T - 11. Here, denotes the identity matrix of appropriate dimension. Then, the unique optimal policy of Problem (9) is given by
Ti (u|z)) = =N(ui - (Re B3 11++111 112x B,) 10--B B, 11++1 (1- -/Exlle+1) A
(R+ + B.II++1(I- - -BB)) 11)
(60)
Proof. In this proof, for notational simplicity, we often drop the time index as A, B. First, for t == T --1, the Q-function in (11) is
1r-1(2, u) |1lller + [ log E
1
exp " ("1lAr- Br-1 WT - )]
(61)
where | == x Pæ for symmetric matrix P. Here, we have
exp 1 | + Bu + wrr-1lli -11tt)
exp
|w -y-1
+ " 11Arx + Bu + wilir dw,
(62)
(2T) nt ΣT-1
R
2
where Er-1l denotes the determinant of Er-1, and
/ +
"1 A1 + Bu + w w
[10l ---11-1-111-- 27pw" II(Az + Bu) ||Ax + Bull?
By the assumption that Σ nII is positive definite and a completion of squares argument,
| 1-- 2
+ !7 11 + Bu wllf
= (Iwe- -EE-1 -I- -nn TT
Bu) 1-l [[T + Bu) B -- ||Ar Bull2n)
Thus, we obtain
Jen exp
11 - + || + Bu + wlii dw
(2T) (E-1 I|I exp
+ Bu)
|| Bullhu) (63)
Consequently, by (61)- the Q-function can be written as (x, u)
1 [lull
| er
+ 2n
| + Br- ( -1I7)
2
1 |A-
BT-iu/li + Cor
1
2
==l=1la
+
||ul + +|Ax Bulls 1-m I1+I1 + Cer-
2
1
= = 1 + |ull
+
|| + Bullie--en + Cer 1
2
17

---

where the constant Cor- is independent of (z, u). Now, we adopt completion of squares argument again:
Qr-1(2, u) ( + 2r A III nyTHE) Bu | (nt.nnnT-pppp (1
Cor-1
|uu | B - -B
-1 (I-
-B (T-n)lTs)
R+B -111-111)-2
|101 + II 11---11-11
+ Cer
||u + (R B - [I ) A 1r((-1EM)))118
+
|a
2 liri + Cer-1
Here, we used l1 - 1y 111) - (1= and
= 11-1-Q--11 - By
AT
=Q+ 1T - 1
1 B T (
-rrA
Therefore, the optimal policy at = T
पं - (u| = Null - (Rr-1++ IIr(1I -
-- Axx
(RT-1 + B 1
(64)
The value function is given by
V--1(x)- log
(o---rr-(z,
| lir CV7-11
R
where Cvy-, does not depend on
By applying the same argument as above for T' ....., O, we arrive at the optimal policy (60) and
V((+) 2
| Cv.
(65)
(x,
Ilu uu + (Ri + BT 11+1(1 -2 BBBBBBBB BB 1114111 m)lll(11)
R A
Arll
2
BB
1 | + Co.
(66)
where Cv, and Co, are independent of (x, u). This completes the proof.
By the same argument as above, the optimal policy of the Rényi entropy regularized risk-sensitive control problem (17) in the linear quadratic Gaussian setting is also given by (60).
D Proof of Lemma 5
First, we give the precise statement of Lemma 5. To this end, for b E R, define
B (U) = { 9 9: U-UR R
g is bounded below, / exp(ag(u))du co, I exp(bo(u))ddu oo).
(67)
18

---

Similarly, define (U) for upper bounded functions. For given g U 7 R. E R, and {0, 1], define
Pu) p P P(U) / ooppraglw)lllyyuukaa soo).
/ plu) < o0).
L*(U)= =
If p E (U) and a E (0, 1), then it holds that Ha(p) < co. If (-20,0) n (1, (1) we have Ha(p) > -00.
Now, we are ready to state the duality lemma.
Lemma 10. For B,7 E R {0} such that 7 and for g E ((s 813 (U), it holds that
log [ exp((3(u))dd))
{ log V exp((s(w))p(u)d
inf DEL'TTOR T
- (p) -1-
H
B
B
(68)
and the unique optimal solution that minimizes the right-hand side of (68) is given by
p(u) = exp (-(7 - 3)g(u))
u WED U.
(69)
Ju exxppp 3)d(""))du'"
In addition, for h B (9 (U), it holds that
log If [csp()b(u))du)) γ
{ log [f (ssp(hh(w))dduddd
+ 7 ,
1
sup
PE (U)
(70)
and the unique optimal solution that maximizes the right-hand side of (70) given by
expl(r - -B)h(w)
p(u) = I Texp((7 3))("'))du"* WE U.
(71)
Although the proof is similar to that of the duality between exponential integrals and Rényi diver- gence [40], requires more careful analysis because we do not assume the upper boundedness of and the lower boundedness of h, unlike in [40].
Proof. For notational simplicity, we often drop as L First, we note that it is sufficient to prove that for > 0, * 1,g E Bla-1,-1) and E B (o..... , holds that
log / (oxp((a= - 1)g(u))du
log [ eexp(ag()))(uuda - 71 N1-a601
inf
a
PEL
(72)
1 log /
/ (on((a-1)((u))((uud)) + Ha(o))
[ex((a(w))ddu
sup peta
log
(73)
and
exp(-g(u))
exp(h(u))
("(u) = J exp((ggu'))ddu) p (u)
(u) := J Texp(h(u))dd''
(74)
the unique optimal solutions to (72), (73), respectively. To this, that if (72), (73) hold for
> 0, # 1, they hold for any a E {0, 1}. Indeed, when a < 0, let ā = a > and for
E B [o.1]= let = -h. Since ğ E B ((a-1,-1) by (72), we have
1
log /
exp((( -oxp(a--1)(()ddu = inf
1)
log [
(epplo(tu)) - Hi No-alo)
-
pell
19

---

Therefore, it holds that
log { (cxp(ah(u)jau)
1 log
/ exp((a 1))(())))aa - Haaco)
-
= inf DEt=
1
1
log / eepp((a
1)((w))(()dd) (xxp((a-))(())ppaaa) + Ho (0))
sup PEL
which means that for any < O and any h E Ba,,, (73) holds. Similarly, by considering h -9 E [a..)) for E BB (o-1,-1) we can see that for any a < 0 and any E (a- (72) holds. Additionally, (72) and (73) with a 2 5-3 = (7 - B)g. h = (T - B3hh coincide with (68), (70)
where g and are replaced by g. h.
In what follows, for a > a 1. we prove (72). Note that when L' [p)) < oo holds. Hence, for the minimization of (72), is sufficient consider p Pa.g L 1--a The density p defined in (74) fulfills p² Paa L because y E B ((a-11-1)= and it can be easily seen that
1 a
log /
- 1)g(u))du (cp((a-))((u)ddu
1 log / explag(u)))
(1-a(p"). (75)
a
First, we consider the case a > Define p(u) == exp((a - 1)g(u)), p(u) = exp(-g(u)). Then, by Hölder's 's inequality, for any E Pag n L 1-a it holds that
f p(u)du= / p(u)
(plu)
(((u) (p(u)
p(u)du
(p(u)
Akuddu)
(
p((u) (p(u)
plu)da
p(u)
( (fexptaolw))owwhu
p(u)) ((w)==adu)
(76)
Noting that a - and taking the logarithm of (76), we get for any p E Pa.g O L
1 α
log [f
exp(((a- 1))(())d < log [ exp(ag(u)))(w)du
1
Hi-a(p).
Combining this with (75), the relation (72) holds, and by (75), p" in (74) is optimal solution. The equality of Hölder's inequality (76) holds if and only if there exist 01, 02 2 0, 0102 * 0 such that
p(u) a2 p(u) holds fialmoot everywhere. Here, is the measure defined by p. This condition satisfied only for p", that is, it an unique optimal solution.
(()) p(u)
a1
Next, we analyze the case (0, 1). By Hölder's inequality, for any p E Pa.
/ (((u) p(u)
a-1
p(u)du (f1Veaassda)) Aujjdaa)
" (n((
(p(())
(P(W) p(u)
a-1
1-a
p(u)du
(ʃ plw))d)) "J
p(u) plu)) plu)duy (laypla
1-a
which yields
a (exp((a-11oo0)) 1)g(4))du < 1 [ exp(ag(()))((ddd Hi-o(p), Up E Pagg
1 log
(77)
Then, similar to the case a 1, can be seen that for a (0, 1), (72) holds and p² is a unique optimal solution.
20

---

Next, we show (73) for a > 1. Since a > 1 and h is upper bounded. it holds that p E Pa-1.h. The density p** defined in (74) satisfies pa* E 1. O L because h [(..)) and one can easily see that
1 log /
1 exp(ah(u))du a-1 log / /xxp((a (sp((a-1)((u))))) 1))(w))pp"())))))) (u)d) + Ha (p**).
Define p(u) exp((a- 1))(()))((u), A(u) exp(-h(u))p(uu) Then, by Hölder's inequality, for any p E L it holds that
/ Æu)du=
1 / A(u)= " (()))~ -- Mu)du
(1 Muu)a~Jalsaal =h
((X(x) It follows from the above that for any p E Is.
= -(folujaaa " (foxp(an(u))aa)
log [ exp((a- 1)h(u))p(u)ddt
1
log [f (esplathu))dtu H. (p).
a
Hence, by the same argument as for (72), we can show that (73) holds for > 1, and pr is a unique optimal solution.
Lastly, we show (73) for a E (0, 1). For E L it holds that [a)( < co. Then, noting that 1 < O, it is sufficient to perform the maximization in (73) for P E P Pa- n L By Hölder's inequality, for any E Pa we have
S p S X(u)a-Pauudu
Mu)ddu (f1Voolojda) (( 1-a
= =(fopp((e - "(a))))))))) "(exppaah(u))d 1-a
Therefore,
log [ exxp((a-1)(w)))ouuaa
1 log [ explah(w))du - Ho (p),
and similar to the case 1, we arrive at (73) for E (0, 1), and the unique optimal solution is p** This completes the proof.
E
Proof of Theorem 6
In this appendix, we analyze the following problem:
T-1
minimize
1
log
exp nor((T) +
Σ (a(2,, u)) (z( - 1=0
(78)
17
where >>>>>>>00 E E E 11) the system is given by (48)-(50), and ne(-[x) E (U) (pe
P(U) Su
oo) for any x E X and E [0, T 1].
Define the value function and the fuunciio associated with (78) as
T-1 Σ
f((x) = inf 1 log E exp (TT((T)) + (M.)
Co us) - -EHl-en - ( (TT((1
1)
amt
t E [0, - 1], x E X,
(79)
7x(x cT (z), x X.
2,(2, u) = cz u) + log E exp (1 w,))), tE [0, T 1]. x E X, u uE U.
(80)
For the analysis, assume the following conditions.
21

---

Assumption 11. For any E [0, T), is bounded below.
Assumption 12. The Q-function 2, in (80) satisfies
J exp
2)(x, du < co, / exp (-(1- En) 21(x, u) du < 0o
(81)
for any X and E [0, 1].
D
For example, when c is bounded for any t E [0, 7[. 2, is also bounded, and in addition, if [z(( < oc, (81) holds. In the linear quadratic setting, Assumption 12 also holds without the boundedness of and U.
Now, we prove Theorem 6 by induction. First, for t T have
inf ar o1-1(12)6L)-r00)) - (
yr-1(x))
/ TT
(mer- ( + (er())) [T-1 2, ur-1 20 du }
log
xr-1(u/x))
The derivation is same as (85) and (86). By the definition of the Q-function in (80), holds that
{ log 1/ (x)p(u/)))xp(n2--aae.. u))du ee ( 1-1x))
(r-1(x)= = (-)())) (())
inf
(82)
Since er and cr-1 are bounded below, 2r-1 is also bounded below. Therefore, by Assumption 12, 2r-1 (2, B 1 -- (U) (see (67) for the definition of B.t)). and we apply Lemma 10 with 3 = - (e- 11), 7 1 to (82). As result,
E-1 - log U, exp
-1
(r-)(z)
u)) du
(83)
and the unique optimal policy that minimizes the right-hand side of (82) is
2
exp
(u
(84)
fo exp
(ar,w²)
du
Moreover, since 2T-1 is bounded below, Yr-1 is also bounded below.
Next, we assume the induction hypothesis that for some /. E [o, T 2], [m))==
T-1 the unique
optimal policy of the minimization in the definition of 741,, and %441 is bounded below. By definition.
22

---

we have
1
(((x)
inf (mo).
log exp neols, - en nOT (xT)
1)
T-1
(c)( (
+
It
#=++1
inf (5x)
-i (Te (-1x))
T-1 Σ ==
1 log E exp
n)((2, ue) + (er((T) +
(Z. up) -e
21 ===a
"
inf (Mxbame 12-1
EM (Te((())(( + log
To Te(u//)) exp n2,22, u) + (er((1))
T-1
1
Σ #=++1
(cs (2, - (o(())))))))) (o
Iz =x, = a du
- 1 (T) + log / exx(xc)) u))
1
inf =c(
T-1
(m:
exp ()T((TT) (xp((1C((TT) 1
Σ (ca(zs,, us) (o)
x x, ux
du
(85)
Moreover, noting that
T-1
exp(174+1(z)) E (1 exp not (ET) +n
Σ
(co(2s) Uz) - E (TT(((())
x++1 =
e=++1
we get
7(x)= =
inf m1(-)))L---0
- -HHI-on (olloo((T((-pp)))
f (T((ul)) exp(ncc ( [esp u, wx)))] du
+
log
"
(86)
By using 2,, the above equation can be written as
({ (e(u(x)) exp(n2. u))du
%(x)= inf a.(-)e)EL ()) 17
log
EHi (T)
(87)
Since assumed that 7/+1 is bounded below, 2, is also bounded below. By combining this with --------e--
Assumption 12, it holds that 2e(x, E B (e----- - (U). Thus, by Lemma 10, the unique optimal policy that minimizes the right-hand side of the above equation is
(x.u.)
exp
of (u[z) =
(88)
Ju exp
2,(2,u)) :
du'
and
k(x) 8-1 log / exp
w)) du
(89)
Lastly, since 2, is bounded below, V48 also bounded below. This completes the induction step, and obtain Theorem 6.
23

---

Proof of Proposition 7
By using the relation V log po()) = (opo(T)/po(T),, we obtain
VoJ(O) (ofr))nn((CC44))))//GGG) Vo Yologpo()))rr
In addition, by the expression
T-1
pT(T) = p(zo)
II
p 24, (0) (ullz))
we get
VoJ(0) = - po(t) exp((C(())))
T-1
T
Σ
vo log x (0) (uelxz)
Vo log x (9) (uslos)) dr
]
= (1 p))
Σ 0
Ve log n (0) (ueler) exp noT(xT) +
(crl2r, uz) + log x (0) (uxl (uslzze)
(90)
Note that for any h (x)++1 x (U)++1 + R, it holds that
/ h (FFoe uo:) (F())
E [h(xoee uo:)]
II p( 12, (o.
==0
f h M Me ((())
T-2
II p( [ (0.
I ((Frrer--,, u1 -x) (WT-1 ( d
T-2
/ h (aoo. Moa ((x))())))))))))))))))
p(0 1. uz) (d [u
II ==0
-/ h(oo. ( ==
Il
p(x.+ 1 (r,, (0) d.
24

---

It follows from the above that
ppo(r)
V Volog 1 (wel2s) exp
(T Us) + log n (ug (r,))
= Ve log (0) (uelzz) exp
Σ (c. us) log π (us (x:))
x (ur
p( p Mes u,)a (0) (us (dd
a=0
= = V (0) (ue exp
(o, us) + log π (0) (us (ma)) Σ
[-1
p(ao)
II p(1 [e, (1 (x
==0
/ (Vo Vo / x (ut
(usl (usl=x)due exp (7
Σ
(ca(Irs; us) ) + log π 10) (us
s=0
p(20)
II p( 1s. uz) ( )do;
0.
(91)
By combining this with (90), we get
VoJ(O)
T-1
(n + 1) (T)
Σ Vo log (u,lz) exp (eelor) + n
Σ a=0
(co( u,) + log π 10 ( (u,
(92)
Lastly, for any function b: R= + R. it holds that
E Epo(r) Vo log (0) (ue (x.))a)(())))) f polors ue)
Vo (uela))) blxs )dx,du,
nt (ue(o))
= I (ur = 0.
=
This completes the proof.
G Proof of Proposition
By definition,
1
(
T-1 (us((t))
arg min log EUE 7)
p(O ((T] (((0 11221 p(O,kIo, us)
dr
(93)
25

---

The term between the brackets is
J p" (zo.; uo.z)
(ʃ D0 (1141:T, uet1rr ue)
IT= Ta (us
dra.duc:
p(OT (TT)
IT p(O,kers us)
I p" (Torts uoa))
Is' (2g
p(O, (Tss Us)
(f | u441:1/22 ue)
IT-1
s=t Ts (us(
"
x
dr41rdu411
dro:duass
(T) IT p( Iis Us)
=.M
where
/ p² (1et1:r, Met1rr/ze, uc)
I TT (2,
"
M = =((u((z))"
[M(Or (2T))
I1= p( O 1iis us))
In addition, by the expression pd (Ioies uo:e) p( II (( (T |as uo (uo
1
/ mr(uulkrel
arg min log We
1
(p(O))err +-++1 (2g(((
drddu
x (err (et1rle)
(94)
II=2 p(O, |:c Us)
Now, define
[1100 T T (u
(e(u)(z)) = Ze(xr)
Ep
( (
plO,
IT
p(O (0l.xou Uz)
(95)
Z((x:) : / Ep (
(Ep
[ IT p(O, 10, Uho
IT=11 Tes uz(u Fis)
-1/0
du. (96)
Then, (94) can be rewritten
1 log | 2x(xz)" / Fe(mel=r))
arg min T1 17
duydes
(97)
By Jensen's inequality, for any 11 m + holds that
Zx(xz)" / (ullus)))
log
durdor
#e(ue (2x))
2x(xz)"' (Br Falluelas 8x
Te Te (2))) des (uel
1t7
>
log
dx:
7
1
log
Ze(
(98)
1
where the equality holds if and only if (-1xx,) R,(-)1r)), and ((.1=x))//{(((1zz) is constant Pa,.- almost everywhere. Here, P. is the probability distribution associated with R (-(x)))). Hence, the infimum (98) is attained only by n. = f.. This completes the proof.
26

---

H
Details of the experiment
The implementation of the risk-sensitive SAC (RSAC) algorithm follows the stable-baselines3 [50] version of the SAC algorithm, which means that the RSAC algorithm also implements some tricks including reparameterization, minibatch sampling with a replay buffer, target networks, and double Q-network. Now, we introduce series of hyperparameters listed in Table shared for both SAC and RSAC algorithms.
Table I: SAC and RSAC Hyperparameters
Parameter
Value
optimizer
Adam [51]
learning rate
10-3
discount factor regularization coefficient
0.99
0.1
target smoothing coefficient
0.005
replay buffer size
105
number of critic networks
2
number of hidden layers (all networks)
2
number of hidden units per layer number of samples per minibatch activation function
256
256
ReLU
As mentioned in Section 5, there were no significant differences in the control performance obtained or the behavior during training shown in Fig. 5 with those hyperparameters: However, when is too small or too large, the training process becomes unstable due to the gradient vanishing problem and the gradient exponential growth problem, respectively, leading to training failure. To this end, we compare the robustness of the trained policies with RSAC (17 {-0.02, -0.01, 0.01, 0.02)) and the standard SAC, which corresponds to 17 0, in the experiment. For each learned policy, do trail for 20 times. For each trail, we take 100 sampling paths to calculate the average episode cost. In Fig. 3, the error bars depict the max and min values, and the points depict the mean value among the 20 trails. We change the length of the pole / in the Pendulum-vi environment to test the robustness of the learned policies (1 1.0 m in the original environment). For the training, we used an Ubuntu 20.04 server (GPU: NVIDIA GeForce RTX 2080Ti). The code available at https:/ /ttt:://ii... wtt.//ooooaaooo wwwiiittooirrrr-i----ee www.ee--ennssiiiiiieeeee sac. git.
9 0002
9 -0011
SAC
1700
0.01
1250
1000
250
200
10000
20000
00020
20000
Learning steps
Figure 5: Training process of RSAC (with different n) and SAC in terms of average episode cost.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are made based on theoretical results (Theorems 2, 6, and Propositions 7, 8).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
Assumptions and a complete proof of all our results (Theorems 2, 3, 6, and Propositions 7, 8) are provided in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: All the information is disclosed in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
We provide open access to the code via GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ StaeedeeeCooeeSSisiiiiiiccee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cooo wttpccccpppccccceecooc.cccccccct hppppicgguudes.....ppppppp/p
htter/gguudeeeoeeeuuiuiiiiiiiiii /Codeeeooiisseeeeee cerbbbmsiiiinn seeiiissiinniiiieeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details are given in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in Fig. 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
30

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///eurripcc.. http://c/ccppppcce.tttciiiiieee.
Answer: [Yes]
Justification: This work does not involve human subjects or participants, and there are no data-related concerns such privacy issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The contribution of this paper theoretical and do not anticipate any direct societal impact of the work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
In this work, we do not need data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
For the experiment, OpenAl Gym, and it is properly mentioned.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We submit the documentation as supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "73zST21iyB", "reviewer": "Reviewer_5JeB", "summary": "In this paper, the authors consider a risk-sensitive control problem with Renyi divergence. The contributions are primarily theoretical with some rudimentary experiments. The contributions include connection to risk-sensitive control under exponential cost formulation, with Renyi divergence leading to an additional regularization term. Extensions to a RL setting with a policy gradient theorem and an actor-critic version are proposed. An alternative risk-senstive control problem with Renyi\nentropy is discussed.", "questions": "1. The policy gradient theorem in Prop 7 has an expectation over trajectories. Does this mean for one update to the policy parameter, an entire trajectory needs to be simulated? In other words, is there a REINFORCE variant that updates after every sample transition?\n2. The soft-actor critic extension is unclear to me. Does the proposed soft actor critic algorithm with gradient estimates in (34)-(36)? If yes, can you characterize the limiting policy wrt risk-sensitivity? What about compatibility issues that one usually sees with actor-critic algorithms that employ function approximation?\n3. While this may be outside the scope of this work, can you comment on extensions to a long-run average cost formulation, with Renyi divergence?", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Tackles an important risk-sensitive control problem and makes advances by employing a general divergence measure\n- Connections to exponential utility formulation", "weaknesses": "- Some bits of RL extension are unclear to me (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments. We sincerely appreciate your positive evaluation.\n\n> The policy gradient theorem in Prop 7 has an expectation over trajectories. Does this mean for one update to the policy parameter, an entire trajectory needs to be simulated? In other words, is there a REINFORCE variant that updates after every sample transition?\n\nWe are not entirely certain whether the derived risk-sensitive REINFORCE can be extended to the one that updates the policy parameter after every sample transition because we do not know of the specific literature that the reviewer has in mind.\n  Nevertheless, we expect that such extension is possible because the policy gradient (23) for the regularized risk-sensitive control is structurally the same as the standard policy gradient for the risk-neutral control.\n  That is, by replacing the exponential of the accumulated cost $ \\\\exp(\\\\eta c_T(x_T) + \\\\eta \\\\sum_{s=t}^{T-1} (c_s(x_s,u_s) + \\\\log \\\\pi^{(\\\\theta)} (u_s | x_s))) $ in (23) by $  c_T(x_T) +  \\\\sum_{s=t}^{T-1} (c_s(x_s,u_s) + \\\\log \\\\pi^{(\\\\theta)} (u_s | x_s)) $, we recover the standard REINFORCE with regularization.\n\n\n> The soft-actor critic extension is unclear to me. Does the proposed soft actor critic algorithm with gradient estimates in (34)-(36)? If yes, can you characterize the limiting policy wrt risk-sensitivity?\n\nYes, the proposed soft actor-critic estimates the gradient by the sample approximation of (34)-(36). For example, the gradient estimate of $ \\\\nabla_\\\\phi \\mathcal{J}\\_Q(\\\\phi) $ is given by $ (\\\\nabla_\\\\phi Q^{(\\\\phi)}(x_t,u_t)) \\\\exp(\\\\eta Q^{(\\\\phi)}(x_t,u_t) - \\\\eta c(x_t,u_t))  \\\\{ T_\\\\eta (Q^{(\\\\phi)}(x_t,u_t) - c(x_t,u_t))  -  T_\\\\eta( V^{(\\\\psi)}(x_{t+1}) )  \\\\}  $, where $ x_t $, $ u_t $, and $ x_{t+1} $ are samples.\n\n\n  Although we do not yet know how to analyze the limiting policy learned by the proposed soft actor-critic algorithm, it is interesting to investigate the relationship between the risk-sensitivity parameter $ \\eta $ and the robustness of the limiting policy. We appreciate your insightful comment, and we would like to study it in future work.\n\n\n\n> What about compatibility issues that one usually sees with actor-critic algorithms that employ function approximation?\n\nThe compatible function approximation theorem can be extended to the proposed soft actor-critic algorithm.\n  For the standard actor-critic algorithm, the compatibility of a function approximator $ Q^{(\\\\phi)} $ is defined by the condition $ \\\\nabla_\\\\phi Q^{(\\\\phi)} (x,u) =  \\\\nabla_\\\\theta \\\\log \\\\pi^{(\\\\theta)} (u_t|x_t) $ [R1]. For the proposed risk-sensitive soft actor-critic (RSAC), the compatibility condition is modified as $ \\\\nabla_\\\\phi Q^{(\\\\phi)} (x,u) \\\\exp(\\\\eta Q^{(\\\\phi)} (x,u) - \\\\eta c(x,u)) =  \\\\nabla_\\\\theta \\\\log \\\\pi^{(\\\\theta)} (u_t|x_t) $, where $ \\\\eta $ is the risk-sensitivity parameter. By letting $ \\\\eta = 0 $, we recover the standard compatibility condition.\n  We would like to omit the detail of the compatible function approximation theorem for RSAC because we plan to study the properties of RSAC in a forthcoming paper.\n\n[R1] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, \"Policy gradient methods for reinforcement learning with function approximation,\" Advances in Neural Information Processing Systems, vol. 12, pp. 1057-1063, 1999.\n\n\n\n\n > While this may be outside the scope of this work, can you comment on extensions to a long-run average cost formulation, with Renyi divergence?\n \nThere will be several technical difficulties in dealing with the regularized risk-sensitive control (reinforcement learning) for the long-run average cost by the same reason as for the unregularized case [29, R2]. Even so, we expect that similar results to ours will hold for the long-run average problems, that is, the regularized risk-sensitive control problem with long-run average cost will boil down to solving a soft Bellman equation, and this leads to risk-sensitive soft actor-critic for long-run average cost.\nWe would like to tackle this important and challenging extension in future work.\n\n\n[29] V. S. Borkar, \"Q-learning for risk-sensitive control,\" Mathematics of Operations Research, vol. 27, no. 2, pp. 294-311, 2002.\n\n[R2] A. Biswas and V. S. Borkar, \"Ergodic risk-sensitive control--a survey,\" Annual Reviews in Control, vol. 55, pp. 118-141, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7UELIegMa", "reviewer": "Reviewer_4yoD", "summary": "This paper considers the control as inference framework of RL. Instead of minimizing the KL which is commonly done in control as inference (and has been shown to be equivalent to MaxEnt RL), they consider minimizing the Rényi divergence. They prove that this minimization is equivalent to minimizing a functional similar to the entropic risk measure, and show that the order of the Rényi divergence controls the risk behaviour of the policy obtained (averse/neutral/seeking). They then show that this minimization over policies is equivalent to solving a soft Bellman equation. They then take another route of generalizing maximum entropy control, by replacing the standard entropy with Rényi entropy, which they prove is also optimized by the same soft Bellman equation. They then show how this can be transformed into an implementable algorithm, by introducing a variant of policy gradient and soft actor-critic for their formulation. They round out their work with an experiment section to illustrate properties of their algorithm at varying levels of risk sensitivity.", "questions": "- It is written that for large $|\\eta|$ the algorithm is unstable. In the experiments, the largest $|\\eta|$ used is .02. Is the algorithm unstable for $\\eta$ larger than this? A better understanding of what range $\\eta$ can safely take would be helpful. \n- There have been a number of risk-senstive soft actor critic algorithms proposed in recent years, none of which are referenced in this paper. Is there a reason why you haven't mentioned them, and compared/contrasted your proposed algorithm to them?", "limitations": "The authors adequately discuss the limitations, and list them as items for future work (such as numerical instability for large $|\\eta|$).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I find this paper to be clearly written, and guides the reader through their contributions. The theoretical results and their proof also flow nicely and are clearly presented. \n\nI think that taking the two paths to generalize MaxEnt RL (replacing the minimization of the KL with Rényi and replacing the entropy regularization with Rényi entropy regularization) is a very nice approach and makes a strong case for why this is a natural generalization.\n\nOverall I think this is an interesting theoretical work and extension to the MaxEnt RL framework, and I think it can inspire future work.", "weaknesses": "I believe that the major weakness of this work is the experiment section. I understand the goal of this work is mainly theoretical, however, I believe the current experiments are unmotivated, and do not complement the theoretical results. In particular, there is currently only a single experiment, which I believe to be unfit for the following reasons:\n- the experiment measures the generalization across environments of the algorithm at different levels of $\\eta$. I am quite surprised by this,  as none of the theoretical results discuss generalization across environments. \n- the theoretical results prove that the algorithm optimizes a quantity that *does not* solely depend on the expectation of the cost function, but the plot shows the average cost obtained by the algorithm. \n- touching on the previous point, I think that what the experiment section can be quite helpful for, which it currently lacks, is provide the reader with a more intuitive understanding of the level of risk sensitivity as a function of $\\eta$. For example, perhaps one could plot the empirical distributions of returns obtained for different $\\eta$ (so that one can visualize the difference between risk-seeking/neutral/averse, and how continuous this behaviour is wrt $\\eta$).\n- there is only a single environment used (Pendulum-v1), with no justification as to why this was chosen.\n\nI believe that if the experiment section can be improved in the ways I highlighted above, the paper has the potential to be more interpretable and impactful, and I would be happy to update my score to reflect this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments. \n\n\n> the experiment measures the generalization . . . discuss generalization across environments.\n\nWe apologize for the lack of explanation.\n  Through the experiment, we aim to show the robustness of policies learned by the risk-sensitive soft actor-critic (RSAC) for risk-averse cases $ \\\\eta > 0 $.\n  Indeed, for the unregularized case, it is known that risk-sensitive control with exponential utility solves a robust control problem (Petersen, James, Dupuis, (IEEE TAC, 2000)).\n  Specifically, risk-sensitive control with $ \\\\eta > 0 $ equivalently solves the following minimax control problem: $$ \\\\min_{\\\\{u_t\\\\}} \\\\max_{\\\\nu \\\\in B_d (\\\\mu)} \\mathbb{E}\\_{\\\\nu} [ c_T(x_T) +  \\\\sum_{t=0}^{T-1} c_t(x_t,u_t) ], $$ \nwhere $ \\\\mu $ is the reference distribution of the trajectory $ \\\\{x_t\\\\}\\_{t=0}^{T} $, $ \\\\nu $ is the perturbed distribution of $ \\\\{x_t\\\\}\\_{t=0}^{T} $, and \n$ B_d (\\mu) := \\\\{ \\\\nu : D_{\\\\rm KL} (\\\\nu \\\\| \\\\mu) \\\\le d \\\\} $ defines the set of all admissible perturbed distributions $ \\\\nu $. The radius $ d $ is related to the sensitivity parameter $ \\\\eta $. This minimax problem optimizes the control considering the worst-case perturbation of the state distributions, which means that the risk-sensitive control with $ \\\\eta > 0 $ is robust against the perturbation of system parameters and the noise. \n  Therefore, RSAC is expected to learn robust policies, which generalize well across environments. However, we have not yet revealed the equivalence between the \"regularized\" risk-sensitive control and minimax control.\n  The role of the experiment in this paper is to verify the robustness of RSAC, which has not yet been ensured theoretically in this work.\n  Consequently, we have observed the robustness as expected.\n\n  We will add the above explanation to the revised manuscript.\n\n\n> the theoretical results prove that  . . . one could plot the empirical distributions of returns obtained for different $ \\eta $\n\nFollowing your suggestion, we have plotted the empirical distributions of costs for different $ \\eta $ in Fig. 1 of the attached PDF. As can be seen, only the distribution for $ \\eta = 0.02 $ does not change so much under the system perturbations. The distribution for SAC ($ \\eta = 0 $) with $ l = 1.5 $ deviates from the original one ($ l = 1.0 $), and another peak of the distribution appears in the high-cost area. This means that there is a high probability of incurring a high cost, which clarifies the advantage of RSAC.\nOn the other hand, the more risk-seeking the policy becomes, the less robust against the system perturbation it becomes. However, at the expense of the robustness, for $ l = 1.25 $, the policy with $ \\eta = -0.02 $ yields a high probability in the low-cost area (from cost $ =20 $ to $ {\\rm cost} = 80 $).\nWe will add the evaluation of the empirical distributions to the revised manuscript. We appreciate your suggestion.\n \n\n\n\n> there is only a single environment used (Pendulum-v1), with no justification as to why this was chosen.\n\nWe are very sorry that we were not able to conduct an additional experiment on a different environment in time for the rebuttal.\n  As mentioned above, the purpose of the experiment is to verify the robustness of RSAC, which has not been proved theoretically in this paper. Although it would be preferable to use several environments, we believe that the experiment together with the additional evaluation of the empirical distributions plays a sufficient role in complementing the theoretical contribution of this work. \n\n> I believe that if the experiment section can be improved in the ways I highlighted above, the paper has the potential to be more interpretable and impactful, and I would be happy to update my score to reflect this.\n\nWe hope the additional evaluation and the responses have made our work more interpretable and impactful.\n\n > It is written that for large $ |\\eta| $\nthe algorithm is unstable. . . . A better understanding of what range $ \\eta $\ncan safely take would be helpful.\n \nSince $ \\eta $ appears, for example, as $ \\exp(\\eta Q^{(\\phi)}(x_t,u_t)) $ in the gradients (34)-(36), the magnitude of $ \\eta $ that does not cause the numerical instability depends on the scale of the reward (cost). \n  Therefore, we need to choose $ \\eta $ depending on environments.\n  In the experiment using Pendulum-v1, $ |\\eta| $ that is larger than $ 0.03 $ results in the failure of learning due to the numerical instability. We would like to emphasize that this issue is not specific to our algorithms, but occurs in general risk-sensitive reinforcement learning with exponential utility.\n\n  Thank you for your suggestion. We will add the above explanation to the revised version.\n\n\n> There have been a number of risk-sensitive soft actor critic algorithms . . . Is there a reason why you haven't mentioned them, and compared/contrasted your proposed algorithm to them?\n\n\n\nThis is because in this paper, we focus on the risk sensitivity induced by the exponential performance criteria.\n  To the best of our knowledge, the only work that proposes a risk-sensitive soft actor-critic type algorithm for the exponential utility is (Enders, Harrison, Schiffer, (arXiv:2402.09992, 2024)) mentioned in the manuscript.\n  If you know of any other references and could share them with us, we sincerely appreciate it.\n\n  If we should mention another type of risk-sensitive soft actor-critic, we would like to add the references (Duan, Guan, Li, Ren, Sun, Cheng, (IEEE TNNLS, 2021), (Choi, Dance, Kim, Hwang, Park, (ICRA, 2021)), which consider risk by distributional RL, to the manuscript.\n  An advantage of RSAC over other risk-sensitive approaches including the distributional RL is that we only need minor modifications to the standard SAC. Thanks to this, techniques for stabilizing SAC, e.g., reparameterization, minibatch sampling with a replay buffer, target networks, double Q-network, can be directly used for RSAC.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBMSWmH2MR", "reviewer": "Reviewer_pMhb", "summary": "The paper generalizes the control as inference framework to the risk-sensitive setting using Renyi divergence variational inference. This yields a cost function with an exponential utility and log-probability regularization, weighted by the Renyi divergence parameter $\\eta$. From the Taylor expansion of the cost, we can see that $\\eta$ controls the level to which we are risk-averse or risk-seeking. And when $\\eta$ goes to zero, we recover the traditional MaxEnt control problem. Next, the authors show that the risk-sensitive optimal policy can be obtained by solving the soft Bellman equation, which relates it to many existing methods. They also show that for deterministic dynamics, their framework, RCaI, and MaxEnt give the same optimal policy. The authors then develop RCaI versions of the policy gradient and soft actor-critic algorithms. They also show that using Renyi entropy regularization in place of the standard KL divergence in maximum entropy control yields an optimal policy with the same structure. Finally, they provide a proof-of-concept experiment on the classical Pendulum benchmark using their risk-sensitive SAC algorithm. They show that the choice of risk-sensitivity parameter can improve robustness to dynamics mismatch.", "questions": "- How does the risk-sensitive policy gradient method compare to RSAC and REINFORCE?", "limitations": "The authors discuss limitations of their method in the conclusions. Specifically, they discuss the numerical instability of their method for large $\\eta$ cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- RCaI is a novel formulation of risk-sensitive control using the control-as-inference framework with a unique choice of divergence metric. They derive two RL algorithms from this framework, a policy gradient and soft actor-critic algorithm, which are easily swapped in for their risk-neutral equivalents.\n- The authors show that RCaI actually generalizes the MaxEnt formulation to incorporate risk sensitivity. They also show many interesting properties of the optimal policy and how it relates to existing approaches.\n- They show experimentally that RCaI may yield improvements in robustness by introducing the risk-sensitivity term.\n- The paper is well organized and overall written well. It provides a thorough related work section and does a good job explaining the novelty and results.", "weaknesses": "- The experimental evaluation is sparse, with only one simple task and no baselines other than RSAC with $\\eta=0$. They also do not evaluate the policy gradient method and contrast it with RSAC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments.\n\n> The experimental evaluation is sparse, with only one simple task and no baselines other than RSAC with $ \\eta = 0 $. They also do not evaluate the policy gradient method and contrast it with RSAC.\n\nWe sincerely apologize for the lack of explanation.\nFirst, we would like to clarify the purpose of the experiment in this work.\n  The notion of risk of the risk-sensitive control with exponential utility can be described by minimax robust control, and through the experiment, we aim to show the robustness of policies learned by RSAC for risk-averse cases $ \\\\eta > 0 $.\n  Indeed, for the unregularized case, it is known that risk-sensitive control with $ \\\\eta > 0 $ equivalently solves the following minimax control problem [R1]: $$ \\\\min_{\\\\{u_t\\\\}} \\\\max_{\\nu \\\\in B_d (\\\\mu)} \\\\mathbb{E}\\_{\\\\nu} [ c_T(x_T) +  \\\\sum_{t=0}^{T-1} c_t(x_t,u_t) ], $$ \nwhere $ \\\\mu $ is the reference distribution of the trajectory $ \\\\{x_t\\\\}\\_{t=0}^{T} $, $ \\nu $ is the perturbed distribution of $ \\\\{x_t\\\\}\\_{t=0}^{T} $, and $ B_d (\\\\mu) := \\\\{ \\\\nu : D_{\\\\rm KL} (\\\\nu \\\\| \\\\mu) \\\\le d \\\\} $ defines the set of all admissible perturbed distributions $ \\\\nu $. The radius $ d $ is related to the sensitivity parameter $ \\\\eta $. This minimax problem optimizes the control considering the worst-case perturbation of the state distributions, which means that the risk-sensitive control with $ \\\\eta > 0 $ is robust against the perturbation of system parameters and the noise. \n  Therefore, RSAC is expected to learn robust policies. However, we have not yet revealed the equivalence between the \"regularized\" risk-sensitive control and minimax control. The role of the experiment in this paper is to verify the robustness of RSAC, which has not yet been ensured theoretically in this work.\n  Consequently, we have observed the robustness as expected.\n\nAs an additional evaluation, we have plotted the empirical distributions of the cost under RSAC for different $ \\\\eta $ in Fig. 1 of the attached PDF. As can be seen, only the distribution for $ \\\\eta = 0.02 $ does not change so much under the system perturbations. The distribution for SAC ($ \\\\eta = 0 $) with $ l = 1.5 $ deviates from the original one ($ l = 1.0 $), and another peak of the distribution appears in the high-cost area. This means that there is a high probability of incurring a high cost, which clarifies the advantage of RSAC.\nOn the other hand, the more risk-seeking the policy becomes, the less robust against the system perturbation it becomes. However, at the expense of the robustness, for $ l = 1.25 $, the policy with $ \\\\eta = -0.02 $ yields a high probability in the low-cost area (from cost $ =20 $ to $ {\\rm cost} = 80 $).\n\n\nWe believe that the experiment together with this additional evaluation plays a sufficient role in complementing the theoretical contribution of this work. \nEven though it would be preferable to compare RSAC with any other methods to show its advantages besides the robustness, it is beyond the scope of this work because our main motivation is to extend the framework of CaI and to provide new theoretical perspectives on CaI.\nWe plan to study the properties of RSAC in a forthcoming paper.\n\n\n[R1] I. R. Petersen, M. R. James, and P. Dupuis, \"Minimax optimal control of stochastic uncertain systems with relative entropy constraints,\" IEEE Transactions on Automatic Control, vol. 45, no. 3, 2000.\n\n\n\n> How does the risk-sensitive policy gradient method compare to RSAC and REINFORCE?\n\nWe are very sorry that we were not able to conduct the additional experiment of the derived risk-sensitive REINFORCE in time for the rebuttal.\nHowever, we would like to mention that REINFORCE suffers from high variance, delayed updates, and less sample efficiency, and SAC generally outperforms REINFORCE in a wide range of environments. Hence, it is expected that RSAC also outperforms the derived risk-sensitive REINFORCE. For the comparison between the risk-sensitive REINFORCE and the standard REINFORCE, the previous work [20] showed that by using an appropriate sensitivity parameter $ \\\\eta $, the risk-sensitive REINFORCE learns faster and has lower variance than the standard REINFORCE for the unregularized case.\nThe regularized risk-sensitive REINFORCE derived in this paper will have similar properties to the unregularized risk-sensitive REINFORCE because the only difference between the regularized and unregularized REINFORCE methods is the presence of the log-probability term in the policy gradient.\n\n[20] E. Noorani and J. S. Baras, \"Risk-sensitive REINFORCE: A Monte Carlo policy gradient algorithm for exponential performance criteria,\" in 2021 60th IEEE Conference on Decision and Control (CDC), pp. 1522-1527, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YCVYe9KKcF", "reviewer": "Reviewer_qM7b", "summary": "This paper explores the core research question, \"What kind of control problem is solved by control-as-inference (CaI) with Renyi divergence?\"  The authors characterize the CaI objective that results from replacing Kullback-Leibler (KL) divergence with the more general Renyi divergence.  They show that the Renyi order parameter $\\alpha$ controls risk-sensitivity of the learned policy and refer to the result as risk-sensitive CaI (RCaI).  The paper also proposes a policy gradient method for optimizing the resulting objective via variational inference.  An experiment is provided in the Pendulum-v1 environment from OpenAI Gymnasium.", "questions": "See \"Weaknesses\" section", "limitations": "The authors explicitly note some limitations of the work in the Conclusions section.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is relatively well-written, easy-to-read, and the ideas discussed are interesting and appear novel.  Risk-sensitive RL / control is an important problem area, and contributions made are likely to have impact.", "weaknesses": "While the work is interesting the paper lacks a strong motivation.  The research question of \"what kind of problem is solved by CaI with Renyi divergence?\" is interesting enough, but the authors propose no hypothesis or motivation for why one would want to consider Renyi divergence.  What is wrong with the existing formulation in terms of KL divergence?  I believe that there is a straightforward answer to this question, but the current manuscript does not address it.  The resulting RSAC algorithm is somewhat more complicated than the SAC counterpart, and so a strong justification for what is gained by considering RSAC in lieu of SAC, or indeed any of the other risk-sensitive RL approaches, would be necessary.\n\nOne way to justify the proposed method over existing ones is to empirically demonstrate some advantage.  However, the authors do not provide such empirical comparison.  There is no comparison of the proposed method(s) to baselines from the literature.  The authors only consider a single environment (Pendulum-v1).  Moreover, the notion of \"risk\" is not clear in the chosen environment.  Overall the experimental validation needs to be more convincing to recommend publication.  The authors should consider more environments, particularly where there is a precise notion of \"risk\", and compare to baseline methods such as SAC, MBPO, MPO, VMBPO, Mismatched no More, etc.\n\n**Detailed comments**\n* L284-286 : It is unclear what the authors intend to demonstrate to show that the proposed method \"works.\" It would be helpful to be more specific.  What are you showing?\n* Eq. (23) : The existence of the exponential function in the gradient suggests that learning may be numerically unstable.  The authors should address numerical stability of their approach beyond noting it in the conclusion.\n* L147-151 : These are known results and references should be provided for them", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments.\n\n>  While the work is interesting the paper lacks a strong motivation. . . .  risk-sensitive RL approaches, would be necessary.\n\nWe apologize for the unclear presentation, which misled the reviewer thinking that our motivation was to improve SAC.\nOur main motivation is to extend the framework of CaI and to provide new theoretical perspectives on CaI.\nAs mentioned in the literature review, there has been an attempt to use a divergence other than the KL divergence for CaI, and it was shown that model predictive control based on the Tsallis divergence outperforms the KL divergence in some situations. Despite this advantage found experimentally, no theoretical properties of CaI with divergences other than the KL divergence were known. \nIn this work, we have discovered that the Renyi divergence is a natural choice for the extension of CaI in the sense that the resulting policy solves the well-known risk-sensitive control problem with exponential utility.\nIn addition to this theoretical discovery, thanks to the proposed unifying framework, we have revealed several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control.\nIn summary, our main contribution is to establish a unifying framework of CaI based on the Renyi divergence, which improves our theoretical understanding on CaI.\nFurthermore, the fact that the theoretically established our framework (RCaI) gives risk-sensitive extension of the well-known SAC also strengthens our contribution.\n\nAdditionally, we would like to emphasize that the derived RSAC algorithm requires only a minor modification to the standard SAC. Indeed, by letting $ \\eta = 0 $ in the gradients (34)-(36), we recover SAC. This is an advantage of RSAC because techniques for stabilizing SAC, e.g., reparameterization, target networks, double Q-network, can be directly used for RSAC.\n\nWe will specify the above in the revised version.\n\n\n\n\n> One way to justify the proposed method . . . MPO, VMBPO, Mismatched no More, etc.\n\nWe sincerely apologize for the lack of explanation.\n  The notion of risk of the risk-sensitive control with exponential utility can be described by minimax robust control, and through the experiment, we aim to show the robustness of policies learned by RSAC for risk-averse cases $ \\eta > 0 $.\n  Indeed, for the unregularized case, it is known that risk-sensitive control with $ \\eta > 0 $ equivalently solves the following minimax control problem (Petersen, James, Dupuis, (IEEE TAC, 2000)): $$ \\\\min_{\\\\{u_t\\\\}} \\max_{\\\\nu \\\\in B_d (\\\\mu)} \\\\mathbb{E}\\_{\\\\nu} [ c_T(x_T) +  \\\\sum_{t=0}^{T-1} c_t(x_t,u_t) ] ,$$\n where $ \\mu $ is the reference distribution of the trajectory $ \\\\{x\\_t\\\\}\\_{t=0}^{T} $, $ \\\\nu $ is the perturbed distribution of $ \\\\{x\\_t\\\\}\\_{t=0}^{T} $, and $ B_d (\\\\mu) := \\\\{ \\\\nu : D_{\\\\rm KL} (\\\\nu \\\\| \\\\mu) \\\\le d \\\\} $ defines the set of all admissible perturbed distributions $ \\\\nu $. The radius $ d $ is related to the sensitivity parameter $ \\\\eta $. This minimax problem optimizes the control considering the worst-case perturbation of the state distributions, which means that the risk-sensitive control with $ \\eta > 0 $ is robust against the perturbation of system parameters and the noise. \n  Therefore, RSAC is expected to learn robust policies. However, we have not yet revealed the equivalence between the \"regularized\" risk-sensitive control and minimax control. The role of the experiment in this paper is to verify the robustness of RSAC, which has not yet been ensured theoretically in this work.\n  Consequently, we have observed the robustness as expected.\n  To make the robustness clearer, we have plotted the empirical distribution of the cost for different $ \\\\eta $ in Fig. 1 of the attached PDF. Please see the PDF for the discussion of this additional evaluation. It should be noted that the experiment shows RSAC with $ \\\\eta > 0 $ is more robust than the standard SAC ($ \\\\eta = 0 $).\n  \n  We believe that the experiment together with this additional evaluation plays a sufficient role in complementing the theoretical contribution of this work.\n  Even though it would be preferable to compare RSAC with methods other than SAC to show its advantages besides the robustness, it is beyond the scope of this work because the focus of this work is on a theoretically established unifying framework of CaI, not on RSAC. We plan to study the properties of RSAC including its numerical issue in a forthcoming paper; please see also the response to the comment on the numerical instability.\n\nWe will specify the above in the revised manuscript.\n\n\n> L284-286 : It is unclear what the authors intend . . .  What are you showing?\n\nWe apologize for the unclear explanation. As explained above, the experiment is conducted to verify the robustness of policies learned by RSAC.\n\n\n> The existence of the exponential function  . . . beyond noting it in the conclusion.\n\nFirst, we would like to emphasize that in the experiment, even for $ \\\\eta = 0.02 $, where the numerical instability is not problematic, the robustness of the learned policy is improved. Hence, although it is desirable to resolve the numerical instability for large $ |\\\\eta| $, we consider RSAC to be useful enough at the moment to improve the robustness of the policies.\n\nIt may be possible to alleviate this numerical issue by considering a dual problem associated with the regularized risk-sensitive control as done for resolving the numerical instability of the Sinkhorn algorithm used for solving entropic optimal transport problems.\nWe would like to try this approach in a forthcoming paper, where we aim to reveal the properties of RSAC.\nLastly, we would like to emphasize that this issue is not specific to our algorithms, but occurs in general risk-sensitive RL with exponential utility.\n\n\n> L147-151 : These are known results . . . provided for them\n\nWe would like to cite (Whittle, (John Wiley & Sons, 1990)) for them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have attached a PDF file with the experimental results of the empirical distributions of cost under the derived risk-sensitive soft actor-critic.

Meta-review: The paper investigates risk-sensitive control using Renyi divergence in the control-as-inference (CaI) framework. The authors establish a connection between Renyi divergence and risk-sensitive control, with theoretical results showing that the Renyi order parameter controls risk behavior (averse/neutral/seeking). They propose a risk-sensitive policy gradient method and a soft actor-critic (RSAC) variant based on this framework. While most of the reviewers agree with the novelty of this approach, some of the reviewers raise concerns about the experiments. The current experiment results seem a bit toyish. It would be great if the authors could include more experiments in the next version.
Decision: Accept (poster)