Title: Risk-sensitive control as inference with Rényi divergence
Venue: NeurIPS 2024
Accepted: yes
Keywords: risk-sensitive control

optimal control

reinforcement learning

variational inference

Rényi divergence
Abstract: This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

Risk-Sensitive Control as Inference with Rényi Divergence

Abstract

This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risksensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

1 Introduction

Optimal control theory is a powerful framework for sequential decision making [1]. In optimal control problems, one seeks to find a control policy that minimizes a given cost functional and typically assumes the full knowledge of the system's dynamics. Optimal control with unknown or partially known dynamics is called reinforcement learning (RL) [2], which has been successfully applied to highly complex and uncertain systems, e.g., robotics [3], self-driving vehicles [4]. However, solving optimal control and RL problems is still challenging, especially for continuous spaces.

Control as Inference (CaI), which connects optimal control and Bayesian inference, is a promising paradigm for overcoming the challenges of RL [5]. In CaI, the optimality of a state and control trajectory is defined by introducing optimality variables rather than explicit costs. Consequently, an optimal control problem can be formulated as a probabilistic inference problem. In particular, maximum entropy (MaxEnt) control [6, 7] is equivalent to a variational inference problem using the Kullback-Leibler (KL) divergence. MaxEnt control has entropy regularization of a control policy, and as a result, the optimal policy is stochastic. Several works have revealed the advantages of the regularization such as robustness against disturbances [8], natural exploration induced by the stochasticity [7, 9], fast convergence of the MaxEnt policy gradient method [10].

On the other hand, the KL divergence is not the only option available for variational inference. In [11], the variational inference was extended to Rényi α -divergence [12], which is a rich family of divergences including the KL divergence. Similar to the traditional variational inference, this extension optimizes a lower bound of the evidence, which is called the variational Rényi bound. The parameter α of Rényi divergence controls the balance between mass-covering and zero-forcing effects for approximate inference [13]. However, if we use Rényi divergence for CaI, it remains unclear how α affects the optimal policy, and a natural question arises: what objective does CaI using Rényi divergence optimize?

Kenji Kashima

Contributions The contributions of this work are as follows:

We reveal that CaI with Rényi divergence solves a log-probability (LP) regularized risksensitive control problem with exponential utility [14] (Theorem 2). The order parameter α of Rényi divergence plays a role of the risk-sensitivity parameter, which determines whether the resulting policy is risk-averse or risk-seeking. Based on the result, we refer to CaI using Rényi divergence as risk-sensitive CaI (RCaI). Since Rényi divergence includes the KL divergence, RCaI is a unifying framework of CaI. Additionally, we show that the risk-sensitive optimal policy takes the form of the Gibbs distribution whose energy is given by the Q-function, which can be obtained by solving a soft Bellman equation (Theorem 3). Furthermore, this reveals several equivalence results between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control [15, 16].

Based on RCaI, we derive risk-sensitive RL methods. First, we provide a policy gradient method [17-19] for the regularized risk-sensitive RL (Proposition 7). Next, we derive the risk-sensitive counterpart of the soft actor-critic algorithm [7] through the maximization of the variational Rényi bound (Subsection 4.2). As the risk-sensitivity parameter vanishes, the proposed methods converge to REINFORCE [19] with entropy regularization and risk-neutral soft actor-critic [7], respectively. One of their advantages over other risksensitive approaches, including distributional RL [20, 21], is that they require only minor modifications to the standard REINFORCE and soft actor-critic. The behavior of the risk-sensitive soft actor-critic is examined via an experiment.

Although the risk-sensitive control induced by RCaI has LP regularization of the policy, it is not entropy, unlike the MaxEnt control with the Shannon entropy regularization. To bridge this gap, we provide another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We prove that the resulting optimal policy and the Bellman equation have the same structure as the LP regularized risk-sensitive control (Theorem 6). The derivation differs significantly from that for the LP regularization, and for the analysis, we establish the duality between exponential integrals and Rényi entropy (Lemma 5).

The established relations between several control problems in this paper are summarized in Fig. 1.

Related work The duality between control and inference has been extensively studied [15, 22-26]. Inspired by CaI, [27, 28] reformulated model predictive control (MPC) as a variational inference problem. In [29], variational inference MPC using Tsallis divergence, which is equivalent to Rényi divergence, was proposed. The difference between our results and theirs is that variational inference MPC infers feedforward optimal control while RCaI infers feedback optimal control. Consequently, the equivalence of risk-sensitive control

CaI

MaxEnt control

LP regularized

risk-sensitive control

R´ enyi entropy regularized

risk-sensitive control

+ VI with KL divergence

Same structure

Deterministic system

(linearly solvable)

(VI: variational inference)

: equivalence

η

→

0

+ VI with R´ enyi (1 +

η

)-divergence

Policy converges

as

η

→-

1

(RCaI)

and Tsallis variational inference MPC is not derived, unlike RCaI. The work [30] proposed an EM-style algorithm for RL based on CaI, where the resulting policy is risk-seeking. However, riskaverse policies cannot be derived from CaI by this approach. Our framework provides the equivalence between CaI and risk-sensitive control both for risk-seeking and risk-averse cases.

Risk-averse policies are known to yield robust control [31, 32], and risk-seeking policies are useful for balancing exploration and exploitation for RL [33]. Because of these merits, many efforts have been devoted to risk-sensitive RL [19, 34-36]. In [37], risk-sensitive RL with Shannon entropy regularization was investigated. However, their theoretical results are valid only for almost risk-neutral cases. Our results imply that LP and Rényi entropy regularization are suitable for the risk-sensitive RL.

In [16], risk-sensitive control whose control cost is defined by Rényi divergence was investigated, and it was shown that the associated Bellman equation can be linearized. However, it is assumed that the transition distribution can be controlled as desired, which is not satisfied in general as pointed out in [38]. On the other hand, our result shows that when the dynamics is deterministic, LP

and Rényi entropy regularized risk-sensitive control problems are linearly solvable without the full controllability assumption of the transition distribution.

glyph[negationslash]

Notation For simplicity, by abuse of notation, we write the density (or probability mass) functions of random variables x, y as p ( x ) , p ( y ) , and the expectation with respect to p ( x ) is denoted by E p ( x ) . For a set S , the set of all densities on S is denoted by P ( S ) . Rényi entropy and divergence with parameter α > 0 , α = 1 are defined as H α ( p ) := 1 α (1 -α ) log [∫ { u : p ( u ) > 0 } p ( u ) α d u ] , D α ( p 1 ‖ p 2 ) := 1 α -1 log [∫ { u : p 1 ( u ) p 2 ( u ) > 0 } p 1 ( u ) α p 2 ( u ) 1 -α d u ] . For the factor 1 α (1 -α ) of H α , we follow [39, 40] because this choice is convenient for the analysis in Subsection 3.2 rather than another common choice 1 / (1 -α ) . We formally extend the definition of H α to α < 0 . Denote the Shannon entropy and KL divergence by H 1 ( p ) , D 1 ( p 1 ‖ p 2 ) , respectively because lim α → 1 H α ( p ) = H 1 ( p ) , lim α → 1 D α ( p 1 ‖ p 2 ) = D 1 ( p 1 ‖ p 2 ) . For further properties of the Rényi entropy and divergence, see e.g., [41]. The set of integers { k, k + 1 , . . . , s } , k < s is denoted by [ [ k, s ] ] . A sequence { x k , x k +1 , . . . , x s } is denoted by x k : s . The set of non-negative real numbers is denoted by R ≥ 0 .

2 Brief introduction to control as inference

First, we briefly introduce the framework of CaI. For the detailed derivation, see Appendix A and [5]. Throughout the paper, x t and u t denote X -valued state and U -valued control variables at time t , respectively, where X ⊆ R n x , U ⊆ R n u , and µ L ( U ) > 0 . Here, µ L denotes the Lebesgue measure on R n u . The initial distribution is p ( x 0 ) , and the transition density is denoted by p ( x t +1 | x t , u t ) , which depends only on the current state and control input. Let T > 0 be a finite time horizon. CaI connects control and probabilistic inference problems by introducing optimality variables O t ∈ { 0 , 1 } as in Fig. 2. For

x

0

x

1

x

2

O

0

O

1

O

2

u

0

u

1

u

2

x

T

O

T

c t : X × U → R ≥ 0 , c T : X → R ≥ 0 , which will serve as cost functions, the distribution of O t is given by p ( O t = 1 | x t , u t ) = exp( -c t ( x t , u t )) , t ∈ [ [0 , T -1] ] and p ( O T = 1 | x T ) = exp( -c T ( x T )) . If O t = 1 , then ( x t , u t ) at time t is said to be 'optimal.' The control posterior p ( u t | x t , O t : T = 1) is called the optimal policy. Let the prior of u t be uniform: p ( u t ) = 1 /µ L ( U ) , ∀ u t ∈ U . Although this choice is common for CaI, the arguments in this paper may be extended to non-uniform priors. Then, for the graphical model in Fig. 2, the distribution of the optimal state and control input trajectory τ := ( x 0: T , u 0: T -1 ) satisfies

For notational simplicity, we will drop = 1 for O t in the remainder of this paper.

The optimal policy p ( u t | x t , O t : T ) can be computed in a recursive manner. To this end, define

which play a role of value functions. Then, the following result holds.

Proposition 1. Assume that µ L ( U ) < ∞ and let c t ( x t , u t ) := c t ( x t , u t ) + log µ L ( U ) . Assume further the existence of density functions p ( x 0 ) and p ( x t +1 | x t , u t ) for any t ∈ [ [0 , T -1] ] 1 . Then, it holds that

where

The recursive computation (4), (5) is similar to the Bellman equation for the risk-seeking control. However, it is not still clear what kind of performance index the optimal trajectory p ( τ |O t : T ) optimizes because (4) does not coincide with that of the conventional risk-seeking control. An indirect way to make this clear is variational inference. Let us consider finding the closest trajectory distribution p π ( τ ) to the optimal distribution p ( τ |O 0: T ) . The variational distribution is chosen as

where π t ( ·| x t ) ∈ P ( U ) is the conditional density of u t given x t and corresponds to a control policy. Then, the minimization of the KL divergence D 1 ( p π ( τ ) ‖ p ( τ |O 0: T )) is known to be equivalent to the following MaxEnt control problem:

Especially when the system p ( x t +1 | x t , u t ) is deterministic, the minimum value of D 1 ( p π ( τ ) ‖ p ( τ |O 0: T )) is 0 , and the posterior p ( u t | x t , O t : T ) yields the optimal control of (7). As mentioned in Introduction, this work uses Rényi divergence rather than the KL divergence. Moreover, we characterize the optimal posterior p ( u t | x t , O t : T ) more directly even for stochastic systems.

3 Control as Rényi divergence variational inference

In this section, we address the question of what kind of control problem is solved by CaI with Rényi divergence and characterize the optimal policy.

3.1 Equivalence between CaI with Rényi divergence and risk-sensitive control

glyph[negationslash]

Let η > -1 , η = 0 . Then, CaI using Rényi variational inference is formulated as the minimization of D 1+ η ( p π ( τ ) ‖ p ( τ |O 0: T )) with respect to p π in (6). Now, we have

That is, CaI with Rényi divergence is equivalent to maximizing the above variational Rényi bound. Moreover, by (1), it holds that

Consequently, we obtain the first equivalence result in this paper.

Theorem 2. Suppose that the assumptions in Proposition 1 hold. Then, for any η > -1 , η = 0 , the minimization of D 1+ η ( p π ‖ p ( ·|O 0: T = 1)) with respect to p π in (6) is equivalent to glyph[negationslash]

Problem (9) is a risk-sensitive control problem with the log-probability regularization log π t ( u t | x t ) of the control policy. Let η Φ( τ ) be the exponent in (9). Then, 1 η log E [exp( η Φ( τ ))] = E [Φ( τ )] + η 2 Var[Φ( τ )] + O ( η 2 ) , where Var[ · ] denotes the variance [42]. Hence, η > 0 (resp. η < 0 ) leads to risk-averse (resp. risk-seeking) policies. As η goes to zero, the objective in (9) converges to the risk-neutral MaxEnt control problem (7).

3.2 Derivation of optimal control and further equivalence results

In this subsection, we derive the optimal policy of (9) and give its characterizations. For the analysis, we do not need the non-negativity of the cost c t . We only sketch the derivation, and the detailed proof is given in Appendix B. Similar to the conventional optimal control problems, we adopt the dynamic programming. Another approach based on variational inference will be given in Subsection 4.2. Define the optimal (state-)value function V t : X → R and the Q-function Q t : X × U → R as follows:

and V T ( x T ) := c T ( x T ) . Then, it can be shown that the Bellman equation for Problem (9) is

where π ∗ t ( u t | x t ) := exp ( -Q t ( x t , u t )) / Z t ( x t ) , and the normalizing constant is assumed to fulfill Z t ( x t ) := ∫ U exp( -Q t ( x t , u ′ )) d u ′ < ∞ . Since D 1+ η ( π t ( ·| x t ) ‖ π ∗ t ( ·| x t )) attains its minimum value 0 if and only if π t ( ·| x t ) = π ∗ t ( ·| x t ) , the unique optimal policy that minimizes the right-hand side of (12) is given by π ∗ t ( ·| x t ) and

glyph[negationslash]

Because of the softmin operation above, the left equation in (13) is called the soft Bellman equation. Theorem 3. Assume that ∫ U exp( -Q t ( x, u ′ )) d u ′ < ∞ holds for any t ∈ [ [0 , T -1] ] and x ∈ X . Let η > -1 , η = 0 . Then, the unique optimal policy of Problem (9) is given by (13) . Especially when the dynamics is deterministic, i.e., p ( x t +1 | x t , u t ) = δ ( x t +1 -¯ f t ( x t , u t )) for some ¯ f t : X × U → X and the Dirac delta function δ , it holds that

and the optimal policy of the MaxEnt control problem (7) solves the LP-regularized risk-sensitive control problem (9) for any η > -1 , η = 0 . ♦

glyph[negationslash]

Assumption ∫ U exp( -Q t ( x, u ′ )) d u ′ < ∞ is satisfied for example when c t is bounded for any t ∈ [ [0 , T ] ] and µ L ( U ) < ∞ . The linear quadratic setting also fulfills this assumption; see (16).

Theorem 3 suggests several equivalence results:

RCaI and MaxEnt control for deterministic systems. First, we emphasize that even though the equivalence between unregularized risk-neutral and risk-sensitive controls for deterministic systems is already known, our equivalence result for MaxEnt and regularized risk-sensitive controls is nontrivial. This is because the regularized policy π ∗ t makes a system stochastic even though the original system is deterministic, and for stochastic systems, the unregularized risk-sensitive control does not coincide with the risk-neutral control. This implies that the optimal randomness introduced by the regularization does not affect the risk sensitivity of the policy. This provides insight into the robustness of MaxEnt control [8]. Note that [43] mentioned that the MaxEnt control objective can be reconstructed by the risk-sensitive control objective under the heuristic assumption that the cost follows a uniform distribution. However, this assumption is not satisfied in general. Our equivalence result does not require such an unrealistic assumption.

RCaI and optimal posterior. Although the optimal posterior p ( u t | x t , O t : T ) yields the MaxEnt control for deterministic systems as mentioned in Section 2, it is not known what objective p ( u t | x t , O t : T )

optimizes for stochastic systems. Theorem 3 gives a new characterization of p ( u t | x t , O t : T ) . By formally substituting η = -1 into (11), the Bellman equation for computing π ∗ t becomes (4), (5) for the optimal posterior p ( u t | x t , O t : T ) . Note that even if the cost function c t in (9) is replaced by c t in Proposition 1, { π ∗ t } is still optimal. Therefore, by taking the limit as η ↘-1 , the policy π ∗ t ( u t | x t ) in Theorem 3 converges to p ( u t | x t , O t : T ) , and in this sense, the policy p ( u t | x t , O t : T ) is riskseeking .

Corollary 4. Under the assumptions in Proposition 1, it holds that

where V t and Q t are given by (11) , (13) with η = -1

RCaI for deterministic systems and linearly-solvable control. For deterministic systems, by the transformation E t ( x t ) := exp( -V t ( x t )) , the Bellman equation (14) becomes linear: E t ( x t ) = ∫ exp( -c t ( x t , u ′ )) E t +1 ( ¯ f t ( x t , u ′ ))d u ′ . That is, when the system is deterministic, the LP-regularized risk-sensitive control, or equivalently, the MaxEnt control is linearly solvable [15, 16, 44], which enables efficient computation of RL. Even for the MaxEnt control, this fact seems not to be mentioned explicitly in the literature.

RCaI and unregularized risk-sensitive control in linear quadratic setting. Similar to the unregularized and MaxEnt problems [45, 46], Problem (9) with a linear system p ( x t +1 | x t , u t ) = N ( x t +1 | A t x t + B t u t , Σ t ) and quadratic costs c t ( x t , u t ) = ( x glyph[latticetop] t Q t x t + u glyph[latticetop] t R t u t ) / 2 , c T ( x T ) = x glyph[latticetop] T Q T x T / 2 admits an explicit form of the optimal policy:

Here, N ( ·| µ, Σ) denotes the Gaussian density with mean µ and covariance Σ . The definition of Π t and the proof are given in Appendix C. In general, the mean of the regularized risk-sensitive control deviates from the unregularized risk-sensitive control. However, in the linear quadratic Gaussian (LQG) case, the mean of the optimal policy (16) coincides with the optimal control of risk-sensitive LQG control without the regularization [47].

3.3 Another risk-sensitive generalization of MaxEnt control via Rényi entropy

The Shannon entropy regularization E [ -H 1 ( π t ( ·| x t ))] of the MaxEnt control problem (7) can be rewritten as E [log π t ( u t | x t )] . In this sense, the risk-sensitive control (9) is a natural extension of (7). Nevertheless, for the risk-sensitive case, the interpretation of log π t ( u t | x t ) as entropy is no longer available. In this subsection, we provide another risk-sensitive extension of the MaxEnt control. Inspired by the Rényi divergence utilized so far, we employ Rényi entropy regularization:

where η ∈ R \ { 0 , 1 } , and π t ( ·| x ) ∈ L 1 -η ( U ) := { ρ ∈ P ( U ) | ∫ U ρ ( u ) 1 -η d u < ∞} , ∀ x , which implies |H 1 -η ( π t ( ·| x t )) | < ∞ . As η tends to zero, (17) converges to the MaxEnt control problem (7).

Define the value function V t and the Q-function Q t associated with (17) like (10) and (11). Then, as in Subsection 3.2, the following Bellman equation holds. The derivation is given in Appendix E.

For the minimization in (18), we establish the duality between exponential integrals and Rényi entropy like in [40] because the same procedure as for (12) cannot be applied.

Lemma 5 ( Informal). For β, γ ∈ R \ { 0 } such that β < γ and for g : U → R , it holds that

and the unique optimal solution that minimizes the right-hand side of (19) is given by

♦

For the precise statement and the proof, see Appendix D. By applying Lemma 5 with β = η -1 , γ = η to (18), we obtain the optimal policy of (17) as follows.

Theorem 6. Assume that c t is bounded below for any t ∈ [ [0 , T ] ] . Assume further that for any x ∈ X and t ∈ [ [0 , T -1] ] , it holds that ∫ U exp( -Q t ( x, u ′ )) d u ′ < ∞ , ∫ U exp( -(1 -η ) Q t ( x, u ′ )) d u ′ < ∞ . Then, the unique optimal policy of Problem (17) is given by

where Z t ( x t ) := ∫ U exp( -Q t ( x t , u ′ ))d u ′ , and it holds that

♦

Recall that the LP regularized risk-sensitive optimal control is given by (11), (13) while the Rényi entropy regularized control is determined by (21), (22), and Q t ( x t , u t ) = c t ( x t , u t ) + 1 η log E p ( x t +1 | x t ,u t ) [exp( η V t +1 ( x t +1 ))] . Hence, the only difference between the risk-sensitive controls for the LP and Rényi regularization is the coefficient in the soft Bellman equations (13), (22).

4 Risk-sensitive reinforcement learning via RCaI

Standard RL methods can be derived from CaI using the KL divergence [5]. In this section, we derive risk-sensitive policy gradient and soft actor-critic methods from RCaI.

4.1 Risk-sensitive policy gradient

In this subsection, we consider minimizing the cost (9) by a time-invariant policy parameterized as π t ( u | x ) = π ( θ ) ( u | x ) , θ ∈ R n θ . Let C θ ( τ ) := c T ( x T ) + ∑ T -1 t =0 ( c t ( x t , u t ) +log π ( θ ) ( u t | x t )) and p θ be the density of the trajectory τ under the policy π ( θ ) . Then, Problem (9) can be reformulated as the minimization of J ( θ ) /η where J ( θ ) := ∫ p θ ( τ ) exp( ηC θ ( τ ))d τ . To optimize J ( θ ) /η by gradient descent, we give the gradient ∇ θ J ( θ ) . The proof is shown in Appendix F.

Proposition 7. Assume the existence of densities p ( x t +1 | x t , u t ) , p ( x 0 ) . Assume further that π ( θ ) is differentiable in θ , and the derivative and the integral can be interchanged as ∇ θ J ( θ ) = ∫ ∇ θ [ p θ ( τ ) exp( ηC θ ( τ ))]d τ . Then, for any function b : R n x → R , it holds that

The function b is referred to as a baseline function, which can be used for reducing the variance of an estimate of ∇ θ J . The following gradient estimate of J ( θ ) /η is unbiased:

This is almost the same as risk-sensitive REINFORCE [19] except for the additional term log π ( θ ) ( u s | x s ) . In the risk-neutral limit η → 0 , this estimator converges to the MaxEnt policy gradient estimator [5].

4.2 Risk-sensitive soft actor-critic

In Subsection 3.2, we used dynamic programming to obtain the optimal policy { π ∗ t } . Rather, in this section, we adopt a standard procedure of variational inference [48]. First, we find the optimal factor π t for fixed π s , s = t as follows. The proof is deferred to Appendix G.

glyph[negationslash]

glyph[negationslash]

Proposition 8. For t ∈ [ [0 , T -1] ] , let π s , s = t be fixed. Let η > -1 , η = 0 . Then, the optimal factor π · t := arg min π t ∈P ( U ) D 1+ η ( p π ‖ p ( ·|O 0: T = 1)) is given by glyph[negationslash]

where Z t ( x t ) is the normalizing constant.

♦

By (24), the optimal factor π · t is independent of the past factors π s , s ∈ [ [0 , t -1] ] . Therefore, the variational Rényi bound in (8) is maximized by optimizing π t in backward order from t = T -1 to t = 0 , which is consistent with the dynamic programming. Associated with (24), we define

which is the value function for the policy { π s } T -1 s = t satisfying the following Bellman equation.

By the value function, π · t ( u t | x t ) can be written as

Next, we define the Q-function for { π s } T -1 s = t +1 as follows:

Then, it follows from (26) and (27) that

Especially when π t ( u t | x t ) = π · t ( u t | x t ) , it holds that V π t ( x t ) = -log [∫ exp( -Q π t ( x t , u ′ ))d u ′ ] , which coincides with the soft Bellman equation in (13). In summary, in order to obtain the optimal factor π · t , it is sufficient to compute V π t and Q π t in a backward manner.

Next, we consider the situation when the policy is parameterized as π ( θ ) t ( u t | x t ) , θ ∈ R n θ and there is no parameter θ that gives the optimal factor π ( θ ) t = π · t . To accommodate this situation, we utilize the variational Rényi bound. One can easily see that the maximization of the Rényi bound in (8) with respect to a single factor π t is equivalent to the following problem.

This suggests choosing θ that minimizes (31) whose π t is replaced by π ( θ ) t . Note that this is further equivalent to

We also parameterize V π t and Q π t as V ( ψ ) , Q ( φ ) and optimize ψ, φ so that the relations (28), (29) approximately hold. To obtain unbiased gradient estimators later, we minimize the following squared residual error based on (28), (29), and the transformation T η ( v ) := (e ηv -1) /η , v ∈ R :

Using Q ( φ ) and T η , we replace (31) with the following equivalent objective:

Noting that lim η → 0 T η ( κ ( η )) = κ (0) for κ : R → R , as the risk sensitivity η goes to zero, the objectives J Q , J V , J π converge to those used for the risk-neutral soft actor-critic [7]. Now, we have

∇

ψ

J

V

(

ψ

) =

E

p

π

(

x

t

)

∇

ψ

V

(

x

t

)

exp(

ηV

(

x

t

))

Thanks to the transformation T η , the expectations appear linearly, and an unbiased gradient estimator can be obtained by removing them. By simply replacing the gradients of the soft actor-critic [7] with (34)-(36), we obtain the risk-sensitive soft actor-critic (RSAC). It is worth mentioning that since RSAC requires only minor modifications to SAC, techniques for stabilizing SAC, e.g., reparameterization, minibatch sampling with a replay buffer, target networks, double Q-network, can be directly used for RSAC.

5 Experiment

Unregularized risk-averse control is known to be robust against perturbations in systems [32]. Since the robustness of the regularized cases has not yet been established theoretically, we verify the robustness of policies learned by RSAC through a numerical example. The environment is Pendulum-v1 in OpenAI Gymnasium. We trained control policies using the hyperparameters shown in Appendix H. There were no significant differences in the control performance obtained or the behavior during training. On the other hand, for each η , one control policy was selected and was applied to a slightly different environment without retraining . To be more precise, the pendulum length l , which is 1.0 during training, is changed to 1.25 and 1.5; See Fig. 3. In this example, it can be seen that the control policy obtained with larger η has a smaller performance degradation due to

environmental changes. This robustness can be considered a benefit of risk-sensitive control.

In Fig. 4, empirical distributions of the costs for different risk-sensitivity parameters η are plotted. Only the distribution for η = 0 . 02 does not change so much under the system perturbations. The

[ (

(

ψ

)

)

(

ψ

)

Density

(a) Pendulum length l = 1 . 0 during training

(b) System perturbation l = 1 . 25

(c) System perturbation l = 1 . 5

distribution for SAC ( η = 0 ) with l = 1 . 5 deviates from the original one ( l = 1 . 0 ), and another peak of the distribution appears in the high-cost area. This means that there is a high probability of incurring a high cost, which clarifies the advantage of RSAC. The more risk-seeking the policy becomes, the less robust it becomes against the system perturbation.

6 Conclusions

In this paper, we proposed a unifying framework of CaI, named RCaI, using Rényi divergence variational inference. We revealed that RCaI yields the LP regularized risk-sensitive control with exponential performance criteria. Moreover, we showed the equivalences for risk-sensitive control, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. In addition to these connections, we derived the policy gradient method and the soft actor-critic method for the risksensitive RL via RCaI. Interestingly, Rényi entropy regularization also results in the same form of the risk-sensitive optimal policy and the soft Bellman equation as the LP regularization.

From a practical point of view, a major limitation of the proposed risk-sensitive soft actor-critic is its numerical instability for large | η | cases. Since η appears, for example, as exp( ηQ ( φ ) ( x t , u t )) in the gradients (34)-(36), the magnitude of η that does not cause the numerical instability depends on the scale of costs. Therefore, we need to choose η depending on environments. In the experiment using Pendulum-v1, | η | that is larger than 0 . 03 results in the failure of learning due to the numerical instability. Although it is an important future work to address this issue, we would like to note that this issue is not specific to our algorithms, but occurs in general risk-sensitive RL with exponential utility. It is also important how to choose a specific value of the order parameter 1 + η of Rényi divergence. Since we showed that η determines the risk sensitivity of the optimal policy, we can follow previous studies on the choice of the sensitivity parameter of the risk-sensitive control without regularization. The properties of the derived algorithms also need to be explored in future work, e.g., the compatibility of a function approximator for RSAC [49].

Acknowledgments

The authors thank Ran Wang for his valuable help in conducting the experiment. This work was supported in part by JSPS KAKENHI Grant Numbers JP23K19117, JP24K17297, JP21H04875.

References

Onésimo Hernández-Lerma and Jean B. Lasserre, Discrete-time Markov Control Processes: Basic Optimality Criteria , vol. 30, Springer-Verlag New York, 1996.

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction , MIT Press, second edition, 2018.

Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine, 'Learning to walk via deep reinforcement learning', in Robotics: Science and Systems , 2019.

B. Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez, 'Deep reinforcement learning for autonomous driving: A survey', IEEE Transactions on Intelligent Transportation Systems , vol. 23, no. 6, pp. 4909-4926, 2022.

Sergey Levine, 'Reinforcement learning and control as probabilistic inference: Tutorial and review', arXiv preprint arXiv:1805.00909 , 2018.

Brian D. Ziebart, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy , PhD thesis, Carnegie Mellon University, 2010.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, 'Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor', in International Conference on Machine Learning . PMLR, 2018, pp. 1861-1870.

Benjamin Eysenbach and Sergey Levine, 'Maximum entropy RL (provably) solves some robust RL problems', in International Conference on Learning Representations , 2022.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine, 'Soft actor-critic algorithms and applications', arXiv preprint arXiv:1812.05905 , 2018.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans, 'On the global convergence rates of softmax policy gradient methods', in International Conference on Machine Learning . PMLR, 2020, vol. 119, pp. 6820-6829.

Yingzhen Li and Richard E. Turner, 'Rényi divergence variational inference', in Advances in Neural Information Processing Systems , 2016, vol. 29, pp. 1073-1081.

Alfréd Rényi, 'On measures of entropy and information', in Proceedings of the fourth Berkeley Symposium on Mathematical Statistics and Probability , 1961, vol. 1, pp. 547-561.

Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt, 'Advances in variational inference', IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 41, no. 8, pp. 2008-2026, 2019.

Peter Whittle, Risk-Sensitive Optimal Control , John Wiley & Sons, Ltd., 1990.

Emanuel Todorov, 'Linearly-solvable Markov decision problems', in Advances in Neural Information Processing Systems , 2006, vol. 19, pp. 1369-1376.

Krishnamurthy Dvijotham and Emanuel Todorov, 'A unifying framework for linearly solvable control', in 27th Conference on Uncertainty in Artificial Intelligence , 2011, pp. 179-186.

Ronald J. Williams, 'Simple statistical gradient-following algorithms for connectionist reinforcement learning', Machine Learning , vol. 8, pp. 229-256, 1992.

David Nass, Boris Belousov, and Jan Peters, 'Entropic risk measure in policy search', in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 1101-1106.

Erfaun Noorani and John S. Baras, 'Risk-sensitive REINFORCE: A Monte Carlo policy gradient algorithm for exponential performance criteria', in 2021 60th IEEE Conference on Decision and Control (CDC) . IEEE, 2021, pp. 1522-1527.

Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng, 'Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors', IEEE Transactions on Neural Networks and Learning Systems , vol. 33, no. 11, pp. 6584-6598, 2022.

Jinyoung Choi, Christopher Dance, Jung-Eun Kim, Seulbin Hwang, and Kyung-sik Park, 'Risk-conditioned distributional soft actor-critic for risk-sensitive navigation', in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 8337-8344.

Hilbert J. Kappen, 'Path integrals and symmetry breaking for optimal control theory', Journal of Statistical Mechanics: Theory and Experiment , vol. 2005, no. 11, pp. P11011, 2005.

Emanuel Todorov, 'General duality between optimal control and estimation', in 2008 47th IEEE Conference on Decision and Control . IEEE, 2008, pp. 4286-4292.

Hilbert J. Kappen, Vicenç Gómez, and Manfred Opper, 'Optimal control as a graphical model inference problem', Machine Learning , vol. 87, pp. 159-182, 2012.

Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar, 'On stochastic optimal control and reinforcement learning by approximate inference', in Proceedings of Robotics: Science and Systems , 2012.

Marc Toussaint, 'Robot trajectory optimization using approximate inference', in International Conference on Machine Learning , 2009, pp. 1049-1056.

Masashi Okada and Tadahiro Taniguchi, 'Variational inference MPC for Bayesian model-based reinforcement learning', in Conference on Robot Learning . PMLR, 2020, pp. 258-272.

Alexander Lambert, Fabio Ramos, Byron Boots, Dieter Fox, and Adam Fishman, 'Stein variational model predictive control', in Conference on Robot Learning . PMLR, 2021, vol. 155, pp. 1278-1297.

Ziyi Wang, Oswin So, Jason Gibson, Bogdan Vlahov, Manan S. Gandhi, Guan-Horng Liu, and Evangelos A. Theodorou, 'Variational inference MPC using Tsallis divergence', in Robotics: Science and Systems , 2021.

Yinlam Chow, Brandon Cui, MoonKyung Ryu, and Mohammad Ghavamzadeh, 'Variational model-based policy optimization', arXiv preprint arXiv:2006.05443 , 2020.

Marco C. Campi and Matthew R. James, 'Nonlinear discrete-time risk-sensitive optimal control', International Journal of Robust and Nonlinear Control , vol. 6, no. 1, pp. 1-19, 1996.

Ian R Petersen, Matthew R James, and Paul Dupuis, 'Minimax optimal control of stochastic uncertain systems with relative entropy constraints', IEEE Transactions on Automatic Control , vol. 45, no. 3, pp. 398-412, 2000.

Brendan O'Donoghue, 'Variational Bayesian reinforcement learning with regret bounds', in Advances in Neural Information Processing Systems , 2021, vol. 34, pp. 28208-28221.

Vivek S. Borkar, 'Q-learning for risk-sensitive control', Mathematics of Operations Research , vol. 27, no. 2, pp. 294-311, 2002.

Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang, 'Exponential Bellman equation and improved regret bounds for risk-sensitive reinforcement learning', in Advances in Neural Information Processing Systems , 2021, vol. 34, pp. 20436-20446.

Javier Garcıa and Fernando Fernández, 'A comprehensive survey on safe reinforcement learning', Journal of Machine Learning Research , vol. 16, no. 1, pp. 1437-1480, 2015.

Tobias Enders, James Harrison, and Maximilian Schiffer, 'Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts', arXiv preprint arXiv:2402.09992 , 2024.

Kaito Ito and Kenji Kashima, 'Kullback-Leibler control for discrete-time nonlinear systems on continuous spaces', SICE Journal of Control, Measurement, and System Integration , vol. 15, no. 2, pp. 119-129, 2022.

Friedrich Liese and Igor Vajda, Convex Statistical Distances , Teubner, Leipzig, 1987.

Rami Atar, Kenny Chowdhary, and Paul Dupuis, 'Robust bounds on risk-sensitive functionals via Rényi divergence', SIAM/ASA Journal on Uncertainty Quantification , vol. 3, no. 1, pp. 18-33, 2015.

Tim Van Erven and Peter Harremos, 'Rényi divergence and Kullback-Leibler divergence', IEEE Transactions on Information Theory , vol. 60, no. 7, pp. 3797-3820, 2014.

Oliver Mihatsch and Ralph Neuneier, 'Risk-sensitive reinforcement learning', Machine Learning , vol. 49, pp. 267-290, 2002.

Erfaun Noorani, Christos Mavridis, and John Baras, 'Risk-sensitive reinforcement learning with exponential criteria', arXiv preprint arXiv:2212.09010 , 2023.

Krishnamurthy Dvijotham and Emanuel Todorov, 'Inverse optimal control with linearlysolvable MDPs', in Proceedings of the 27th International Conference on Machine Learning , 2010, pp. 335-342.

Kaito Ito and Kenji Kashima, 'Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges', IEEE Transactions on Automatic Control , vol. 69, no. 3, pp. 1536-1551, 2023.

Kaito Ito and Kenji Kashima, 'Maximum entropy density control of discrete-time linear systems with quadratic cost', To appear in IEEE Transactions on Automatic Control , 2025, arXiv preprint arXiv:2309.10662.

Peter Whittle, 'Risk-sensitive linear/quadratic/Gaussian control', Advances in Applied Probability , vol. 13, no. 4, pp. 764-777, 1981.

Christopher M. Bishop, Pattern Recognition and Machine Learning , Springer, 2006.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour, 'Policy gradient methods for reinforcement learning with function approximation', in Advances in Neural Information Processing Systems , 1999, vol. 12, pp. 1057-1063.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann, 'Stable-baselines3: Reliable reinforcement learning implementations', Journal of Machine Learning Research , vol. 22, no. 268, pp. 1-8, 2021.

Diederik P. Kingma and Jimmy Ba, 'Adam: A method for stochastic optimization', arXiv preprint arXiv:1412.6980 , 2014.

A More details on Control as Inference

In this appendix, we give more details on CaI. As mentioned in (1), the distribution of the state and control input trajectory given optimality variables satisfies

where p ( u t ) = 1 /µ L ( U ) and p ( τ, O 0: T ) is defined so that

for any o 0: T ∈ { 0 , 1 } T +1 and any Borel set B , where P denotes the probability. Therefore, we have

The posterior p ( u t | x t , O t : T = 1) given the optimality condition O t : T = 1 is called the optimal policy. We emphasize that the optimality of p ( u t | x t , O t : T = 1) is defined by the condition O t : T = 1 rather than by introducing a cost functional, unlike π ∗ ( u t | x t ) in (13). In the following, we drop = 1 for O t .

The optimal policy can be computed as follows. Define

Then, it holds that

In addition, we have

where we used

In terms of β t and ζ t , the optimal policy can be written as

Next, by the logarithmic transformation, we define

Then, by (41), the optimal policy satisfies

By (39), it holds that

By using (40), we obtain

which yields

Here, we defined c t ( x t , u t ) := c t ( x t , u t ) + log µ L ( U ) . In summary, Proposition 1 holds.

B Proof of Theorem 3

This appendix is devoted to the analysis of the following problem:

subject to

U

,

t

[ [0

, T

1] ]

,

(48)

glyph[negationslash]

Here, { w t } T -1 t =0 is an independent sequence, x 0 is independent of { w t } , ε > 0 is the regularization parameter, and η is the risk-sensitivity parameter satisfying η > -ε -1 , η = 0 . Note that we do not assume the existence of densities p ( x t +1 | x t , u t ) , p ( x 0 ) . To perform dynamic programming for Problem (47), define the value function and the Q-function as

Then, under the assumption that ∫ U exp ( -Q t ( x,u ′ ) ε ) d u ′ < ∞ , we prove that the unique optimal policy of Problem (47) is given by

First, by definition, we have

By the definition of the Q-function (52), we get

Since D 1+ εη ( π t ( ·| x ) ‖ π ∗ t ( ·| x )) attains its minimum value 0 if and only if π t ( ·| x ) = π ∗ t ( ·| x ) , we conclude that

and the unique optimal policy of Problem (47) is given by (53). Moreover, π ∗ t can be rewritten as

When considering the deterministic system x t +1 = ¯ f t ( x t , u t ) , we immediately obtain the relation

On the other hand, the unique optimal policy of the MaxEnt control problem:

is also given by (55) whose Q-function (52) is replaced by

Therefore, when the system is deterministic, the Q-function of the LP regularized risk-sensitive control problem (47) coincides with that of the MaxEnt control problem (57). Consequently, the optimal policy of Problem (57) solves Problem (47) for any η > -ε -1 , η = 0 for deterministic systems.

glyph[negationslash]

C Linear quadratic Gaussian setting

In this appendix, we derive the regularized risk-sensitive optimal policy in the linear quadratic Gaussian setting.

Theorem 9. Let p ( x t +1 | x t , u t ) = N ( A t x t + B t u t , Σ t ) and c t ( x t , u t ) = ( x glyph[latticetop] t Q t x t + u glyph[latticetop] t R t u t ) / 2 , c T ( x T ) = x glyph[latticetop] T Q T x T / 2 , where Σ t , Q t , and R t are positive definite matrices for any t , and N ( µ, Σ) denotes the Gaussian distribution with mean µ and covariance Σ . Let X = R n x , U = R n u . Assume that there exists a solution { Π t } T t =0 to the following Riccati difference equation:

such that Σ -1 t -η Π t +1 is positive definite for any t ∈ [ [0 , T -1] ] . Here, I denotes the identity matrix of appropriate dimension. Then, the unique optimal policy of Problem (9) is given by

Proof. In this proof, for notational simplicity, we often drop the time index t as A,B . First, for t = T -1 , the Q-function in (11) is

where ‖ x ‖ 2 P := x glyph[latticetop] Px for a symmetric matrix P . Here, we have where | Σ T -1 | denotes the determinant of Σ T -1 , and

By the assumption that Σ -1 T -1 -η Π T is positive definite and a completion of squares argument,

Thus, we obtain

Consequently, by (61)-(63), the Q-function can be written as

where the constant C Q T -1 is independent of ( x, u ) . Now, we adopt a completion of squares argument again:

Here, we used Π T ( I -η Σ T -1 Π T ) -1 = ( I -η Π T Σ T -1 ) -1 Π T and

Therefore, the optimal policy at t = T -1 is

The value function is given by

where C V T -1 does not depend on x .

By applying the same argument as above for t = T -2 , . . . , 0 , we arrive at the optimal policy (60) and

where C V t and C Q t are independent of ( x, u ) . This completes the proof.

By the same argument as above, the optimal policy of the Rényi entropy regularized risk-sensitive control problem (17) in the linear quadratic Gaussian setting is also given by (60).

D Proof of Lemma 5

First, we give the precise statement of Lemma 5. To this end, for a, b ∈ R , define

Similarly, define B a,b ( U ) for upper bounded functions. For given g : U → R , a ∈ R , and α ∈ R \ { 0 , 1 } , define

If ρ ∈ L α ( U ) and α ∈ (0 , 1) , then it holds that H α ( ρ ) < ∞ . If α ∈ ( -∞ , 0) ∩ (1 , ∞ ) , we have H α ( ρ ) > -∞ .

Now, we are ready to state the duality lemma.

Lemma 10. For β, γ ∈ R \ { 0 } such that β < γ and for g ∈ B { β, -( γ -β ) } ( U ) , it holds that

and the unique optimal solution that minimizes the right-hand side of (68) is given by

In addition, for h ∈ B { γ,γ -β } ( U ) , it holds that

and the unique optimal solution that maximizes the right-hand side of (70) is given by

♦

Although the proof is similar to that of the duality between exponential integrals and Rényi divergence [40], it requires more careful analysis because we do not assume the upper boundedness of g and the lower boundedness of h , unlike in [40].

Proof. For notational simplicity, we often drop U as L α . First, we note that it is sufficient to prove that for α > 0 , α = 1 , g ∈ B { α -1 , -1 } , and h ∈ B { α, 1 } , it holds that glyph[negationslash]

and

glyph[negationslash]

are the unique optimal solutions to (72), (73), respectively. To see this, note that if (72), (73) hold for α > 0 , α = 1 , they hold for any α ∈ R \ { 0 , 1 } . Indeed, when α < 0 , let ¯ α := 1 -α > 1 and for h ∈ B { α, 1 } , let ¯ g := -h . Since ¯ g ∈ B { ¯ α -1 , -1 } , by (72), we have

Therefore, it holds that

which means that for any α < 0 and any h ∈ B α, 1 , (73) holds. Similarly, by considering ¯ h := -g ∈ B { ¯ α, 1 } for g ∈ B { α -1 , -1 } , we can see that for any α < 0 and any g ∈ B { α -1 , -1 } , (72) holds. Additionally, (72) and (73) with α = γ γ -β , g = ( γ -β ) ˜ g, h = ( γ -β ) ˜ h coincide with (68), (70) where g and h are replaced by ˜ g , ˜ h .

glyph[negationslash]

In what follows, for α > 0 , α = 1 , we prove (72). Note that when ρ ∈ L 1 -α , |H 1 -α ( ρ ) | < ∞ holds. Hence, for the minimization of (72), it is sufficient to consider ρ ∈ P α,g ∩ L 1 -α . The density ρ ∗ defined in (74) fulfills ρ ∗ ∈ P α,g ∩ L 1 -α because g ∈ B { α -1 , -1 } , and it can be easily seen that

First, we consider the case α > 1 . Define ˜ ρ ( u ) := exp(( α -1) g ( u )) , ϕ ( u ) := exp( -g ( u )) . Then, by Hölder's inequality, for any ρ ∈ P α,g ∩ L 1 -α , it holds that

Noting that α -1 > 0 and taking the logarithm of (76), we get for any ρ ∈ P α,g ∩ L 1 -α ,

glyph[negationslash]

Combining this with (75), the relation (72) holds, and by (75), ρ ∗ in (74) is an optimal solution. The equality of Hölder's inequality (76) holds if and only if there exist a 1 , a 2 ≥ 0 , a 1 a 2 = 0 such that a 1 ( ϕ ( u ) ρ ( u ) ) 1 -α = a 2 ρ ( u ) ϕ ( u ) holds ˜ µ -almost everywhere. Here, ˜ µ is the measure defined by ˜ ρ . This condition is satisfied only for ρ ∗ , that is, it is an unique optimal solution.

Next, we analyze the case α ∈ (0 , 1) . By Hölder's inequality, for any ρ ∈ P α,g ,

which yields

Then, similar to the case α > 1 , it can be seen that for α ∈ (0 , 1) , (72) holds and ρ ∗ is a unique optimal solution.

Next, we show (73) for α > 1 . Since α > 1 and h is upper bounded, it holds that ρ ∈ P α -1 ,h . The density ρ ∗∗ defined in (74) satisfies ρ ∗∗ ∈ P α -1 ,h ∩ L α because h ∈ B { α, 1 } , and one can easily see that

Define ̂ ρ ( u ) := exp(( α -1) h ( u )) ρ ( u ) , λ ( u ) := exp( -h ( u )) ρ ( u ) . Then, by Hölder's inequality, for any ρ ∈ L α , it holds that

It follows from the above that for any ρ ∈ L α ,

Hence, by the same argument as for (72), we can show that (73) holds for α > 1 , and ρ ∗∗ is a unique optimal solution.

Lastly, we show (73) for α ∈ (0 , 1) . For ρ ∈ L α , it holds that |H α ( ρ ) | < ∞ . Then, noting that α -1 < 0 , it is sufficient to perform the maximization in (73) for ρ ∈ P α -1 ,h ∩ L α . By Hölder's inequality, for any ρ ∈ P α -1 ,h , we have

Therefore,

and similar to the case α > 1 , we arrive at (73) for α ∈ (0 , 1) , and the unique optimal solution is ρ ∗∗ . This completes the proof.

E Proof of Theorem 6

In this appendix, we analyze the following problem:

where ε > 0 , η ∈ R \ { 0 , ε -1 } , the system is given by (48)-(50), and π t ( ·| x ) ∈ L 1 -εη ( U ) := { ρ ∈ P ( U ) | ∫ U ρ ( u ) 1 -εη d u < ∞} for any x ∈ X and t ∈ [ [0 , T -1] ] .

Define the value function and the Q-function associated with (78) as

For the analysis, we assume the following conditions.

Assumption 11. For any t ∈ [ [0 , T ] ] , c t is bounded below.

Assumption 12. The Q-function Q t in (80) satisfies

for any x X and t [ [0 , T 1] ] .

∈ ∈ -♦

For example, when c t is bounded for any t ∈ [ [0 , T ] ] , Q t is also bounded, and in addition, if µ L ( U ) < ∞ , (81) holds. In the linear quadratic setting, Assumption 12 also holds without the boundedness of c t and U .

Now, we prove Theorem 6 by induction. First, for t = T -1 , we have

The derivation is same as (85) and (86). By the definition of the Q-function in (80), it holds that

Since c T and c T -1 are bounded below, Q T -1 is also bounded below. Therefore, by Assumption 12, Q T -1 ( x, · ) ∈ B -( ε -1 -η ) , -ε -1 ( U ) (see (67) for the definition of B a,b ), and we can apply Lemma 10 with β = -( ε -1 -η ) , γ = η to (82). As a result,

and the unique optimal policy that minimizes the right-hand side of (82) is

Moreover, since Q T -1 is bounded below, V T -1 is also bounded below.

Next, we assume the induction hypothesis that for some t ∈ [ [0 , T -2] ] , { π glyph[star] s } T -1 s = t +1 is the unique optimal policy of the minimization in the definition of V t +1 , and V t +1 is bounded below. By definition,

♦

we have

Moreover, noting that

we get

By using Q t , the above equation can be written as

Since we assumed that V t +1 is bounded below, Q t is also bounded below. By combining this with Assumption 12, it holds that Q t ( x, · ) ∈ B -( ε -1 -η ) , -ε -1 ( U ) . Thus, by Lemma 10, the unique optimal policy that minimizes the right-hand side of the above equation is

and

Lastly, since Q t is bounded below, V t is also bounded below. This completes the induction step, and we obtain Theorem 6.

F Proof of Proposition 7

By using the relation ∇ θ log p θ ( τ ) = ∇ θ p θ ( τ ) /p θ ( τ ) , we obtain

In addition, by the expression

we get

Note that for any h : ( X ) t +1 × ( U ) t +1 → R , it holds that

It follows from the above that

By combining this with (90), we get

Lastly, for any function b : R n → R , it holds that

This completes the proof.

G Proof of Proposition 8

By definition,

The term between the brackets is

where

In addition, by the expression p π ( x 0: t , u 0: t ) = p ( x 0 ) π t ( u t | x t ) ∏ t -1 s =0 p ( x s +1 | x s , u s ) π s ( u s | x s ) ,

Now, define

Then, (94) can be rewritten as

By Jensen's inequality, for any η > -1 , η = 0 , it holds that glyph[negationslash]

where the equality holds if and only if π ( ·| x t ) glyph[lessmuch] ̂ π t ( ·| x t ) , and π t ( ·| x t ) / ̂ π t ( ·| x t ) is constant ̂ P x t -almost everywhere. Here, ̂ P x t is the probability distribution associated with ̂ π t ( ·| x t ) . Hence, the infimum (98) is attained only by π t = ̂ π t . This completes the proof.

H Details of the experiment

The implementation of the risk-sensitive SAC (RSAC) algorithm follows the stable-baselines3 [50] version of the SAC algorithm, which means that the RSAC algorithm also implements some tricks including reparameterization, minibatch sampling with a replay buffer, target networks, and double Q-network. Now, we introduce a series of hyperparameters listed in Table 1 shared for both SAC and RSAC algorithms.

As mentioned in Section 5, there were no significant differences in the control performance obtained or the behavior during training shown in Fig. 5 with those hyperparameters. However, when η is too small or too large, the training process becomes unstable due to the gradient vanishing problem and the gradient exponential growth problem, respectively, leading to training failure. To this end, we compare the robustness of the trained policies with RSAC ( η ∈ {-0 . 02 , -0 . 01 , 0 . 01 , 0 . 02 } ) and the standard SAC, which corresponds to η = 0 , in the experiment. For each learned policy, we do trail for 20 times. For each trail, we take 100 sampling paths to calculate the average episode cost. In Fig. 3, the error bars depict the max and min values, and the points depict the mean value among the 20 trails. We change the length of the pole l in the Pendulum-v1 environment to test the robustness of the learned policies ( l = 1 . 0 m in the original environment). For the training, we used an Ubuntu 20.04 server (GPU: NVIDIA GeForce RTX 2080Ti). The code is available at https://github.com/kaito-1111/risk-sensitive-sac.git .

Average episode cost

Learning steps

1750

n=-0.02

n=-0.01

SAC

1500

n=0.01

=0.02

1250

1000

750

500

250

10000

20000

30000

40000

50000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are made based on our theoretical results (Theorems 2, 3, 6, and Propositions 7, 8).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions and a complete proof of all our results (Theorems 2, 3, 6, and Propositions 7, 8) are provided in the main paper and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the information is disclosed in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the code via GitHub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are given in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars in Fig. 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on the computer resources is provided in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work does not involve human subjects or participants, and there are no data-related concerns such as privacy issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The contribution of this paper is theoretical and we do not anticipate any direct societal impact of the work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: In this work, we do not need data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For the experiment, we use OpenAI Gym, and it is properly mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit the documentation as a supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the core research question, \"What kind of control problem is solved by control-as-inference (CaI) with Renyi divergence?\"  The authors characterize the CaI objective that results from replacing Kullback-Leibler (KL) divergence with the more general Renyi divergence.  They show that the Renyi order parameter $\\alpha$ controls risk-sensitivity of the learned policy and refer to the result as risk-sensitive CaI (RCaI).  The paper also proposes a policy gradient method for optimizing the resulting objective via variational inference.  An experiment is provided in the Pendulum-v1 environment from OpenAI Gymnasium.\n\nSTRENGTHS:\nThe paper is relatively well-written, easy-to-read, and the ideas discussed are interesting and appear novel.  Risk-sensitive RL / control is an important problem area, and contributions made are likely to have impact.\n\nWEAKNESSES:\nWhile the work is interesting the paper lacks a strong motivation.  The research question of \"what kind of problem is solved by CaI with Renyi divergence?\" is interesting enough, but the authors propose no hypothesis or motivation for why one would want to consider Renyi divergence.  What is wrong with the existing formulation in terms of KL divergence?  I believe that there is a straightforward answer to this question, but the current manuscript does not address it.  The resulting RSAC algorithm is somewhat more complicated than the SAC counterpart, and so a strong justification for what is gained by considering RSAC in lieu of SAC, or indeed any of the other risk-sensitive RL approaches, would be necessary.\n\nOne way to justify the proposed method over existing ones is to empirically demonstrate some advantage.  However, the authors do not provide such empirical comparison.  There is no comparison of the proposed method(s) to baselines from the literature.  The authors only consider a single environment (Pendulum-v1).  Moreover, the notion of \"risk\" is not clear in the chosen environment.  Overall the experimental validation needs to be more convincing to recommend publication.  The authors should consider more environments, particularly where there is a precise notion of \"risk\", and compare to baseline methods such as SAC, MBPO, MPO, VMBPO, Mismatched no More, etc.\n\n**Detailed comments**\n* L284-286 : It is unclear what the authors intend to demonstrate to show that the proposed method \"works.\" It would be helpful to be more specific.  What are you showing?\n* Eq. (23) : The existence of the exponential function in the gradient suggests that learning may be numerically unstable.  The authors should address numerical stability of their approach beyond noting it in the conclusion.\n* L147-151 : These are known results and references should be provided for them\n\nQUESTIONS:\nSee \"Weaknesses\" section\n\nLIMITATIONS:\nThe authors explicitly note some limitations of the work in the Conclusions section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper generalizes the control as inference framework to the risk-sensitive setting using Renyi divergence variational inference. This yields a cost function with an exponential utility and log-probability regularization, weighted by the Renyi divergence parameter $\\eta$. From the Taylor expansion of the cost, we can see that $\\eta$ controls the level to which we are risk-averse or risk-seeking. And when $\\eta$ goes to zero, we recover the traditional MaxEnt control problem. Next, the authors show that the risk-sensitive optimal policy can be obtained by solving the soft Bellman equation, which relates it to many existing methods. They also show that for deterministic dynamics, their framework, RCaI, and MaxEnt give the same optimal policy. The authors then develop RCaI versions of the policy gradient and soft actor-critic algorithms. They also show that using Renyi entropy regularization in place of the standard KL divergence in maximum entropy control yields an optimal policy with the same structure. Finally, they provide a proof-of-concept experiment on the classical Pendulum benchmark using their risk-sensitive SAC algorithm. They show that the choice of risk-sensitivity parameter can improve robustness to dynamics mismatch.\n\nSTRENGTHS:\n- RCaI is a novel formulation of risk-sensitive control using the control-as-inference framework with a unique choice of divergence metric. They derive two RL algorithms from this framework, a policy gradient and soft actor-critic algorithm, which are easily swapped in for their risk-neutral equivalents.\n- The authors show that RCaI actually generalizes the MaxEnt formulation to incorporate risk sensitivity. They also show many interesting properties of the optimal policy and how it relates to existing approaches.\n- They show experimentally that RCaI may yield improvements in robustness by introducing the risk-sensitivity term.\n- The paper is well organized and overall written well. It provides a thorough related work section and does a good job explaining the novelty and results.\n\nWEAKNESSES:\n- The experimental evaluation is sparse, with only one simple task and no baselines other than RSAC with $\\eta=0$. They also do not evaluate the policy gradient method and contrast it with RSAC.\n\nQUESTIONS:\n- How does the risk-sensitive policy gradient method compare to RSAC and REINFORCE?\n\nLIMITATIONS:\nThe authors discuss limitations of their method in the conclusions. Specifically, they discuss the numerical instability of their method for large $\\eta$ cases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the control as inference framework of RL. Instead of minimizing the KL which is commonly done in control as inference (and has been shown to be equivalent to MaxEnt RL), they consider minimizing the Rényi divergence. They prove that this minimization is equivalent to minimizing a functional similar to the entropic risk measure, and show that the order of the Rényi divergence controls the risk behaviour of the policy obtained (averse/neutral/seeking). They then show that this minimization over policies is equivalent to solving a soft Bellman equation. They then take another route of generalizing maximum entropy control, by replacing the standard entropy with Rényi entropy, which they prove is also optimized by the same soft Bellman equation. They then show how this can be transformed into an implementable algorithm, by introducing a variant of policy gradient and soft actor-critic for their formulation. They round out their work with an experiment section to illustrate properties of their algorithm at varying levels of risk sensitivity.\n\nSTRENGTHS:\nI find this paper to be clearly written, and guides the reader through their contributions. The theoretical results and their proof also flow nicely and are clearly presented. \n\nI think that taking the two paths to generalize MaxEnt RL (replacing the minimization of the KL with Rényi and replacing the entropy regularization with Rényi entropy regularization) is a very nice approach and makes a strong case for why this is a natural generalization.\n\nOverall I think this is an interesting theoretical work and extension to the MaxEnt RL framework, and I think it can inspire future work.\n\nWEAKNESSES:\nI believe that the major weakness of this work is the experiment section. I understand the goal of this work is mainly theoretical, however, I believe the current experiments are unmotivated, and do not complement the theoretical results. In particular, there is currently only a single experiment, which I believe to be unfit for the following reasons:\n- the experiment measures the generalization across environments of the algorithm at different levels of $\\eta$. I am quite surprised by this,  as none of the theoretical results discuss generalization across environments. \n- the theoretical results prove that the algorithm optimizes a quantity that *does not* solely depend on the expectation of the cost function, but the plot shows the average cost obtained by the algorithm. \n- touching on the previous point, I think that what the experiment section can be quite helpful for, which it currently lacks, is provide the reader with a more intuitive understanding of the level of risk sensitivity as a function of $\\eta$. For example, perhaps one could plot the empirical distributions of returns obtained for different $\\eta$ (so that one can visualize the difference between risk-seeking/neutral/averse, and how continuous this behaviour is wrt $\\eta$).\n- there is only a single environment used (Pendulum-v1), with no justification as to why this was chosen.\n\nI believe that if the experiment section can be improved in the ways I highlighted above, the paper has the potential to be more interpretable and impactful, and I would be happy to update my score to reflect this.\n\nQUESTIONS:\n- It is written that for large $|\\eta|$ the algorithm is unstable. In the experiments, the largest $|\\eta|$ used is .02. Is the algorithm unstable for $\\eta$ larger than this? A better understanding of what range $\\eta$ can safely take would be helpful. \n- There have been a number of risk-senstive soft actor critic algorithms proposed in recent years, none of which are referenced in this paper. Is there a reason why you haven't mentioned them, and compared/contrasted your proposed algorithm to them?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations, and list them as items for future work (such as numerical instability for large $|\\eta|$).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}

comments: SUMMARY:
In this paper, the authors consider a risk-sensitive control problem with Renyi divergence. The contributions are primarily theoretical with some rudimentary experiments. The contributions include connection to risk-sensitive control under exponential cost formulation, with Renyi divergence leading to an additional regularization term. Extensions to a RL setting with a policy gradient theorem and an actor-critic version are proposed. An alternative risk-senstive control problem with Renyi
entropy is discussed.

STRENGTHS:
- Tackles an important risk-sensitive control problem and makes advances by employing a general divergence measure
- Connections to exponential utility formulation

WEAKNESSES:
- Some bits of RL extension are unclear to me (see questions below).

QUESTIONS:
1. The policy gradient theorem in Prop 7 has an expectation over trajectories. Does this mean for one update to the policy parameter, an entire trajectory needs to be simulated? In other words, is there a REINFORCE variant that updates after every sample transition?
2. The soft-actor critic extension is unclear to me. Does the proposed soft actor critic algorithm with gradient estimates in (34)-(36)? If yes, can you characterize the limiting policy wrt risk-sensitivity? What about compatibility issues that one usually sees with actor-critic algorithms that employ function approximation?
3. While this may be outside the scope of this work, can you comment on extensions to a long-run average cost formulation, with Renyi divergence?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16