Risk-Sensitive Variational Actor-Critic: A Model-Based Approach
Authors: Alonso Granados
Reza Ebrahimi
Jason Pacheco
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
variational inference
risk sensitive RL
Primary area: reinforcement learning

Abstract: Risk-sensitive reinforcement learning (RL) with an entropic risk measure typically requires knowledge of the transition kernel or performs unstable updates w.r.t. exponential Bellman equations. As a consequence, algorithms that optimize this objective have been restricted to tabular or low-dimensional continuous environments. In this work we leverage the connection between the entropic risk measure and the RL-as-inference framework to develop a risk-sensitive variational actor-critic algorithm (rsVAC). Our work extends the variational framework to incorporate stochastic rewards and proposes a variational model-based actor-critic approach that modulates policy risk via a risk parameter.  We consider, both, the risk-seeking and risk-averse regimes and present rsVAC learning variants for each setting.  Our experiments demonstrate that this approach produces risk-sensitive policies and yields improvements in both tabular and risk-aware variants of complex continuous control tasks in MuJoCo.

Published as a conference paper at ICLR 2025
RISK-SENSITIVE VARIATIONAL ACTOR-CRITIC: A MODEL-BASED APPROACH
Alonso Granados Department of Computer Science University of Arizona Tucson, AZ, USA
Reza Ebrahimi
School of Information Systems and Management University of South Florida Tampa, FL, USA
alonsogêcs arizona. edu
ebrahimim@ust edu
Jason Pacheco
Department of Computer Science University of Arizona Tucson, AZ, USA
pacheco arizona. edu
ABSTRACT
Risk-sensitive reinforcement learning (RL) with entropic risk measure typi- cally requires knowledge of the transition kernel or performs unstable updates w.r.t. exponential Bellman equations. As consequence, algorithms that optimize this objective have been restricted to tabular or low-dimensional continuous en- vironments. In this work leverage the connection between the entropic risk measure and the RL-as-inference framework to develop a risk-sensitive variational actor-critic algorithm (rsVAC). Our work extends the variational framework incorporate stochastic rewards and proposes a variational model-based actor-critic approach that modulates policy risk via risk parameter. We consider, both, the risk-seeking and risk-averse regimes and present VAC learning variants for each setting. Our experiments demonstrate that this approach produces risk-sensitive policies and yields improvements in both tabular and risk-aware variants of complex continuous control tasks in MuJoCo.
INTRODUCTION
Deep reinforcement learning (RL) algorithms have contributed to many breakthroughs in domains such as games (Mnih et al., 2015) and robotics (Levine et al., 2016). However, the standard objective of rewards, disregards the variability of the return due to
in RL, maximization of the expected the intrinsic uncertainty in the transition dynamics and the stochasticity of the rewards which can lead catastrophic behavior. Such catastrophic behavior is especially common in real-world applications, such as autonomous driving agents acting dangerously to achieve high reward (Chia et al., 2022) or financial losses in portfolio management (Lai al., 2). As consequence, risk-aware agents are important to adapt to inherent environmental risk.
Many risk measures have been studied introduce risk-sensitivity into RL algorithms. For instance, value at risk (VaR) (Chow et al., 2018), conditional value at risk (CVaR) (Chow & Ghavamzadeh, 2014; Greenberg al., 2022), mean-variane (Tamar et al., 2012: La & Ghavamzadeh, 2013), reward- volatility risk measure (Zhang et al., 2021) and Gini-deviation (Luo et al., 2024). In this work we focus on the entropic risk measure, an approach that incorporates risk into objective via the exponential utility function (Howard & Matheson, 1972; Borkar, 2002). Directly optimizing this objective is challenging: requires the knowledge the transition kernel or depends on unstable updates w.r.t. exponential Bellman equations (Noorani al., 2023).
We address these challenges by exploiting the connection between RL and probabilistic inference to obtain surrogate objective to the entropic risk measure. CL-as-inferena algorithms search for policies that maximize the probability of optimal trajectories rather than maximizing the expected return. However, it has been observed that this objective can produce unwanted risk risk-secking behaviour

---

Published as a conference paper at ICLR 2025
in the learned policy (Levine, 2018; O' O'Donoghue et al., 2019; Tarbouriech et al., 2023). Many such methods take a variational approach that constrain the posterior dynamics to equal those of the true environment (Haarnoja al., 2017: 2018), but can lead to overly stochastic policies (Fellows et al., 2019). Existing variational model-based methods allow posterior dynamics to vary (Chow et al., 201 8) but lead to risk-seeking policies that do not adapt aleatoric risk in the environment (Eysenbach et al., 2022). Furthermore, these methods assume a deterministic reward model that implicitly ignores its risk contribution the original objective.
In this work, we leverage the connection between RL and probabilistic inference to formulate a variational lower bound on the entropic risk measure that can be optimized using only experience from an agent. We optimize this surrogate objective using an EM-style algorithm that consists of learning variational dynamics and reward models that account for intrinsic uncertainty in the environment (E-step) and improve the objective w.r.t. a policy (M-step). Our comprehensive approach permits learning of risk-seeking and risk-averse policies for which the latter has been mostly ignored in the RL-as-inference literature. Our formulation also adapts to risk induced by stochastic rewards, further extension of the RL-as-inference literature which assumes deterministic rewards. Furthermore, we demonstrate the robustness of our method to other risk-aware algorithms in risk-sensitiv variants of Mujoco tasks. Code is available at https: : / github...... com/ Al http//Aoooooonaaaaaa VaC.
PRELIMINARIES: RISK-SENSITIVE REINFORCEMENT LEARNING
The RL framework consists of Markov decision process (MDP) defined by tuple (8, p. R). S, A and R are the state, action and reward spaces, respectively. The transition probability over the next state St+1 S given the current state St E S and action d₂ AA is denoted as p(5 St: at). the initial state distribution as p(8)). ). A policy π specifies a probability distribution over actions given a current state 8t- The reward E R is treated as random variable with distribution o(relst, az). The distribution over trajectory T = ($1, a1, T1, 82, a22 ..... ST, TT, ST+1) ) for a sampling policy π is given by pa (T) p(s)) ) II. p(s++1 st, (t)p(Tr St; (edaTTT(a s.). The standard objective in RL is
to find a policy that maximizes expected return: x arg max, E p EE rz].
2.1 ENTROPIC RISK MEASURE
In risk-sensitive RL with the entropic risk measure the goal is to find policy that maximizes:
Σ Tt
max 3 log (p-(()
exp
13
for risk parameter 3 E R. This objective is closely related to mean-variance RL (Mannor & Tsitsiklis, 2011) given that a Taylor expansion of Eq. (1) yields (r) [E, re] + 23 Var= rt)
+ O (g)) (Mihatsch & Neuneier, 2002; García & Fernández, 2015). The parameter 3 controls the risk- sensitivity of the objective producing risk-seeking policies for B 0 and risk-averse policies for 3 < O. Additionally, it reduces to the standard (risk-neutral) RI. objective when [3] - co. Based on define the soft value functions the cumulative rewards under the entropic risk:
this framework,
Va (s) = log E
exp
rt
$1 s).
Qr a) = log E exp
Σ
rt
$1 = 0..1 (2)
B
B
pr
These functions are recursively associated via Bellman-style backup equations:
E V* (st) log -[-(..)
(exp(Qr(ss a))l, (n az) = log E
exp
F't
Var (
(3)
p(.
3
These are known as soft value functions, due to the presence of operators log E[exp(-) that act as soft approximations to (.). Finally, we have the Bellman optimality equations,
(+()))
V V*(ss)= max Q"(st,
Q" a.) = log Ep(-(on.a))
exp
(4)
Although these value functions can be estimated using dynamic programming, they require knowledge of the transition dynamics and the reward model to compute the expectations, since unbiased sample- based estimates are not available due to the nonlinear log operation. Fig. 1 illustrates the impact of the risk- risk-sensitivity parameter 3 on the optimal policy in simple three arms MDP, and its effects in modulating risk-seeking and risk-averse policies. We emphasize that a risk neutral policy is recovered for large | values, while small | values produce risk-seekiingaavrse policies.
2

---

Published as a conference paper at ICLR 2025
1.50
L1
R
solt- values
Salt-4 values
3.25
0.79
L
R2
0.50
D
Right
Right
Laft
Down
0.50
0.0
5.0
10.0
12.5
15.0
17.5
20:
-20.
-17.
-15.
-125
-12.0
-15
-5.0
-25
00
Beta
Beta
Three arms environment Left: MDP with three actions (left, down and right) and initial state S. Action 'right' produces reward 0 and 10 with probability 0.9 and 0.1, respectively. Action 'left' produces reward O and 4 with probability 0.5 and 0.S, respectively. Finally, action 'down' has deterministic reward of A risk neutral agent would prefer action 'left' which has highest mean return while risk-seeking and risk-averse agents would prefer action 'right' and 'down', respectively. Middle: Soft-Q values as function of 3 for 3 >> 0. Observe that for small & the framework learns risky policy (red region) while for large 8 it recovers optimal risk neutral policy (green region). Right: Soft-Q values function of for 3 < O. Now the agent learns risk-averse policy when [8] small (red region) while recovers the neutral policy for larger [}] (green region).
Figure 1:
2.2 RISK-SENSITIVE VARIATIONAL. BOUND
In this work we leverage the well-established connection between RL and probabilistic infer- ence (Levine, 2018). Under this formulation we incorporate the rewards into probabilistic model by introducing set of binary auxiliary variables O₂ E {0. 1} that are independently distributed at each time as p(O,
r²)) & exp()). The event O = can loosely be interpreted as the agent having acted optimally at time t'. An important motivation for using this interpretation is that we can define surrogate objective for the entropic risk measure via the evidence lower bound (ELBO) on the log- marginal likelihood:
log P.. (Orr) log Ep..
exp Σ
> E.
Σ T
-K(G(T))l == -)(())) (5)
where the LHS is shorthand for the marginal likelihood of an optimal trajectory: log Pr (Opr 1). The log-marginal is equivalent to the entropic risk measure, up to a multiplicative constant 3 which controls risk-sensitivity and is bounded by the RHS. The bound in Eq. (5) arises from the application of Jensen's 's inequality where q(T) is variational distribution over trajectories. This bound is tight when the variational distribution equals the posterior over trajectories q(T) = p(r O1.r 1) almost everywhere. A variety of optimize Js by alternating improvements w.r.t. q and æ (Abdolmaleki al., 2018b; Peters al., 2010; Levine & Koltun, 2013; Chow et al., 2021).
(EM) style algorithms have been proposed to
3 VARIATIONAL MODEL/ / POLICY ITERATION
Using Js(q T) as a surrogate objective for the entropic risk measure we now propose two algorithms that approximately optimize it for, both, the risk-secking (3 0) and risk-averse (B 0) settings. We consider variational distributions over trajectories of the form,
Gx (7) - p(si) II
ग (aelss)q, (rels(rrree) (a)4a(s411sss...
(6)
Note that we incorporate a variational posterior distribution 9, over rewards. This stochastic reward model is extension of existing RL-as-inference methods that are constrained to deterministic rewards. Expanding the KL regularizer of Eq. (5) yields the variational objective,
Σ
Tt
log qa( ga($1+1ls,,a,) log p(relae, 04)
gr (rels,,,,
Ts(g,a) = E Egg (7)
3
p(
'The optimality interpretation is a loose one stemming from the reward time f, which increases the prohability of O, exponentially. This interpretation has become standard in the literature (Levine, 2018).
3

---

Published as a conference paper at ICLR 2025
To optimize Eq. (7) consider an EM-style algorithm where the E-step maximizes J3 w.r.t q and the M-step optimizes w.r.t. т. sessiivinitivity arises in Eq. (7) from the maximization w.r.t the variational distribution 9. Although the penalty discourages deviations from the true model, the agent willing to pay this penalty the increase in expected return is large enough to compensate this extra cost. When B > O, the variational model becomes optimistic (risk-seeking) expected return. When 3 < O, it becomes pessimistic (risk-averse), as instead, it aims to increase the expected cost. Finally, we recover the model when [3] + oc as the objective only suffers the deviation penalty.
aims to increase the
3.1 E-STEP
For a fixed policy т we denote the optimal variational distribution as o = arg max, Jp(q: T). Directly maximizing Jß can be computationally expensive as it requires optimizing 4 over the full trajectory. Instead, we consider a Bellman-like operator Ta as a partial optimization over for single transition where V is state-value function:
(V)) = E max E amt(: a) 9LE rougle
T
log
4r (r(ss,
+ mAx E qLEDs of maga
V(s')
log
ga(s'|s, 8, a)
(8)
B p(r(s,a)
p(s'|s, (s
In particular, we have the following theorem for the operator Ta (v)(s) (see Appendix for proof):
Theorem 1. Repeated application TF any value function such that V(8++1) = 0 converges to the optimal value function VE for all k; where:
2f (ra(st,, 188--182,,0) log p(ra(se, ac)
(sx) = (x) (r)
Σ
log p(8 182, at)
Hence, we can obtain the optimal value function by iteratively applying Ta to some initial value function Vo. We recover the optimal variational distributions using the optimal value function: Theorem 2. Let df and be the solution of arg max, Jolg. x). Then
(r(s,a) a) p()(s, exp 55) (a(s'(s,a) a eV
(10)
All proofs can be found in the Appendix.
3.2 M-STEP
We optimize T(q", TT) w.r.t. the policy using the variational distribution of from the E-step:
Σ re Blog (5(55+118t,. B log p(s141ls),a))
g (r (re ar) p(ri prilst,e at)
π = arg max Egi(r)
(11)
Observe that Eq. (11) is equivalent to learning the optimal policy for standard RI. problem with transition dynamics q and augmented rewards ft, so any RL algorithm can be used for the M-step. Although the expectation can now be estimated using casily-obtained samples from we still have the problem of needing to evaluate the dynamics and reward model the augmented reward, which might be unknown. In the following section address this by providing algorithm that can be optimized using off-policy data. Finally, we note that in the risk-averse setting Eq. (11) corresponds minimization w.r.t. the policy: arg min, , I (q",, 7). See Appendix B for an extended discussion of optimization in the risk-averse regime.
4 RSVAC: RISK SENSITIVE VARIATIONAL ACTOR-CRITIE
We now present a practical RL algorithm that approximately optimizes J J(q, т) using only collected experience by the agent. We make three design choices to approximate this objective: first, learn
4

---

Published as a conference paper at ICLR 2025
parameterized probabilistic networks, (o(su+1|$,, az) and (r(rsss,, dt), for the unknown transition dynamics and reward model; next, we represent the variational distributions using probabilistic networks, (o($++1|s, a+) and qs (r(rsss,, az), and approximate the maximization operation in the E-step with stochastic gradient descent; finally, we use an actor-critic architecture with function approximators to learn the optimal value function and policy from the M-step.
4.1 VARIATIONAL REWARD AND DYNAMICS MODEL OPTIMIZATION
We model the reward and dynamics as Gaussian distributions with mean and covariance given by neural networks and train them minimize cross-entropy using stochastic gradient descent:
(0) (s... [e St, ar]] J²( E-E (te. [log (s st, az)l (12)
where Den is experience replay buffer that stores previously seen interactions with the environment. We similarly parameterize the variational models using Gaussian distributions. To learn the variational reward model we approximate the optimization w.r.t. (o(ra St, at) in Eq. (8) by maximizing:
rt 3
9o (r((ess)
J, = Den
log pe(relse, oz)
(13)
w.r.t. its parameters Φ In particular, we use the reparameterization trick to obtain a lower variance estimator that can be optimized using stochastic gradient ascent,
følc: st, log go go(fale;s), po(fole; st: at)lss,
(x) -(s..a))Dem...MM
(14)
B
where fole; st, ae) is the reparameterized reward model and a noise vector sampled from a spherical Gaussian distribution. Similarly, leam variational dynamics by approximating the optimization w.r.t. go ( az) in Eq. (8) with:
0z)) + log 2o 82, 81, at)ss,, at)
Ja(Φ) De ew
V Volas(e: (1)((1)))) (z)
(15)
po (c: St, ar)]ss,
where again use the reparameterization trick to reparameterize the dynamics model, 8141 [e(e; St, az). and substitute the optimal value function with a critic Vo that can be dif- ferentiated Eq. (15) can be optimized using stochastic gradient ascent.
4.2 ACTOR-CRITIC OPTIMIZATION
We now present an actor-critic algorithm to optimize the M-step. As previously stated, the opti- mization in Eq. (11) is equivalent to the RL problem that has transition dynamics g6 (8(5++11 1 1 az),
[e(ra[se.ee] reward model (ra st, at), and augmented reward f; - 3 log po(st=1/a,.a.) - Bogg po(rs(s..as) We collect transitions from the variational model using branched rollout (Janner et al., 2019), i.e. we sample states under the true dynamics Deny and run the policy under Jo to generate new transitions which we store in the model replay buffer modeel We approximate the critic using neural network Qu a.) which we train by minimizing the squared TD-error:
(o(se=1/s..ae)
Jo(()= = (e ) Drase [(o) ax) - - VV
(16)
using stochastic gradient descent and samples from the model replay buffer. The optimal state- value function ($++1 is implicitly represented by Be "l, ac+1]] where O's is a target critic network that we update using an exponentially moving average of the Q weights (Lillicrap et al., 2015). We approximate this expectation using a single action sample from the policy иe. The policy ma is Gaussian distribution parameterized with neural networks and is trained to maximize Qu with an added entropy regularizer to improve exploration during learning (Haarnoj et al., 2018):
(+) = Ea, (0t fols; st)) log (f((( st)]s
(17)
fo(e; s.) and is noise vector sampled from spherical Gaussian distribution. Again we learn these parameters using stochastic gradient descent. One benefit of VAC is that it enjoys great flexibility any actor-critic method can be incorporated into framework as long as the rewards and samples come from the variational model. Pseudocode for the VAC algorithm can be found in Appendix F.
where we have reparameterized the policy
5

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Entropic risk. The risk-sensitive objective with entropic risk measure was first described by the seminal work of (Howard & Matheson, 1972). This work has inspired many methods in a variety of settings (Borkar, 2001; 2002; 2010; Borkar & Meyn, 2002; Coraluppi & Marcus, 1999; Di Masi & Stettner, 1999; Fleming & McEneaney, 1995; Hernándezz-Hernández & Marcus, 1996; Huang & Haskell, 2020). However, these algorithms are constrained to simple environments as they require knowledge of the transition dynamics assume access to simulator of the environment. In the setting with unknown transition dynamics, TD(0) and Q-learning-styy algorithms have been proposed by applying an exponential transformation to the risk sensitive objective, but estimating these value functions can lead to instabilities when introducing function approximators (Bäuerle & Rieder, 2014; Borkar, 2002; Fei et al., 2021b;a; Mihatsch & Neuneier, 2002; Noorani et al., 2023).
RL-as-inference.
Probabilistic inference methods for solving RL can be traced back to the Kalman- duality in linear-quadratic systems (Kalman, 1960) and later to linearly solvable MDPs (Todorov, 2006). The variational framework can be formulated searching for maximum likelihood policies on an augmented MDP with exponentiated rewards treated as probabilities and is equivalent to the risk-sensitive objective (Todorov, 2008; Levine & Koltun, 2013; Levine, 2018). These approaches learn risk-seeking policies as they only consider 3 > 0. Model-free variational approaches such as MaxEnt RL combat this behavior by removing the controller's ability to modify the variational dynamics, resulting in high-entropy policies. This penalization of determinism has been effective in some high-dimensional tasks (O' DoDonogh al., 2016; Nachum et al., 2017: Haarnoja al., 2018; Lee et al., 2020), but has been shown to lead to undesirable behavior (Fellows et al., 2019). Closely related KL-regularized RL methods include proximal operator on the policy (Peters al., 2010; Schulman et al., 2015; Chebotar et al., 2017: Noorani & Baras, 2021). More generally, EM-style algorithms jointly optimize their variational and prior policies (Peters Schaal, 2007: Neumann et al., 2011; Abdolmaleki et al., 2018b;a). Our variational formulation is most similar to the one used VMBPO (Chow et al., 1021), an EM-style algorithm that also learns variational dynamics. However, their approach only considers risk-seeking policies and deterministic rewards.
Connections to other methods. The role of the risk parameter 3 is to limit the disagreement between the variational and true environment dynamics and reward model through the KI. penalty. 3-VAE (Higgins et al., 2016) studies a similar objective to ours, albeit in the different context of representation learning, where 3 limits the capacity of the variational distribution learn disentangled representations. In Bayesian RL, a similar risk parameter balances the exploration-exploiatiee trade- off by modulating epistemic uncertainty (O D'Donoghue et al., 2019; O' Donoghue & Lattimore, 2021; O' D'Donoguue, 2023).
6 EXPERIMENTS
We evaluate the ability of rsVAC to learn risk-sensitive policies in a variety of risky environments. First, we consider risky variation of the tabular environment discussed in Eysenbach al. (2022), where exact inference is possible and equality to the entropic risk can be achieved. We next evaluate the inclusion of function approximators in continuous 2D) environment with stochastic transition dynamics where the goal is to land proximal to the environment boundary without crossing over. Finally, we compare rs VAC risk-sensitive baseline methods in variations of three challenging MuJoCo environments that incorporate risk in the manner introduced by Luo et al. (2024). In all cases we find that VAC capably learns iiskssenstiive policies in, both, the averree and risk-seeking regimes while simultaneously achieving high reward.
6.1 RISK IN TABULAR ENVIRONMENTS
Our motivation for using the tabular setting is that we can study the risk-sensitive behavior of our algorithm without introducing function approximation error. We consider risky variant of the gridworld presented in Eysenbach et al. (2022). In this environment the agent's 's goal, which starts from the top left corner, is to reach the star goal state (See Fig. 2a). We modify the original environment to incorporate alcatoric risk by including a cliff region (gray squares in Fig. 2a). Falling into the cliff incurs large negative reward and transition to the initial state. The agent can choose

---

Published as a conference paper at ICLR 2025
250
500
750
5000
1250
1040
220
200
100
1000
220
1040
506
1000
2590
2000
(a) Environment
(b) Risk-neutral
(c) Risk-seeking
(d) Risk-averse
Figure 2: Risky tabular setting, (a) The modified grid enviromment with cliff region given by gray states. We show the dynamics in the grid for the action 'right' each state, where represent transition probability between two states as a vectors with magnitude proportional to probability. (b, c, d) demonstrate risk preferences our algorithm, we sample 1000 episodes for three policies Q-learning (risk-neutral), / = (risk-sccking) and B = -0.5 (risk-averse) and compute histograms for the count of visited states.
WEPO
1og,0 Beta
kg200
8.3
120
055
Epsoddes
Lihandus
(a) Performance curves
Convergence for
(c) Convergence 3 <
Figure 3: Stochastic cliff performance. (a): We compare the expected return for independent runs for different algorithms. rsVAC for 3 >> 0 performs comparably to both Q-learning and VMBPO. (b,c): We show that dual optimization eventually converges the same optimal value for different initial settings of (.
from four actions (up, left, down and right) which can result in a transition to the chosen direction or moving randomly one of the four directions with equal probability.
We demonstrate that VAC can produce risk-sensitive policies by training the surrogate objective and 8 -0.5, along with the risk-neutral policy. We compare the risk
for the values of preferences of these policies by computing a histogram over states for 1000 trajectories. From observe that the risk-seeking policy (Fig. 2c) takes the shortest path to the these trajectories, goal, but in the process occasionally falls into the cliff. In contrast, the risk-averse policy (Fig. 2d) avoids entirely the cliff region resulting in longer trajectories. Finally, the nistralutral policy takes middle-of-the-rous approach between the two previous policies (Fig. 2b) where it rarely falls into the cliff but on average it takes to longer to reach the final state in comparison the tisk-seeking policy.
We now compare the average return performance of VsC when including dual optimization w.r.t. discussed in Appendix C. As comparison baselines consider Q-Jearning and VMBPO (Chow et al., 2021), model-based algorithm that also learns variational dynamics but is restricted to the risk-seeking setting. The performance curves in Fig. 3a show that rAVAC (3 > 0) is as efficient and performs as well or better than VMBPO and Q-learning. Figs. 3b and 3c demonstrate robustness of our dual optimization over 3, which converges to the same value regardless of initialization.
6.2 STOCHASTIC CONTINUOUS 2D ENVIRONMENT
We verify that our algorithm can learn risk-sensitive policies with function approximators a stochastic continuous environment. An agent begins in the middle of a 2D space and the goal is to navigate as near to the lower left-or-right corners as possible without crossing the side edges. The agent observes its 2D coordinates (ar, y), chooses direction (la/|2 1), and moves in that

---

Published as a conference paper at ICLR 2025
(a) Risk-Neutral (3 = 100)
(b) Risk-Seeking (B = 2)
(c) Risk-Averse (B
2)
Figure 4: Trajectories for stochastic 2D environment. We illustrate the learned policy by sampling 10 trajectories for different 9 values. (a) Policies trained with large magnitude tend risk-neutral. (b) Policies trained with small positive / are risk seeking and try to hit high reward ignoring potentially hitting the side wall. (c) Policies trained with negative | stay in the center part of the square.
Low-risk
Medium-risk
High-risk
p-21
3-2300
1E.
p-221
04
6.-
0.4
24
02-
62-
01
00
stops
stops
Figure 5: Exit regions for stochastic environment. We define regions depended on the agent's X-position when episode ends; low-risk ([z] < 2.8), medium-risk (2.8 S |x|] < 5.6), and high-risk (5.6 s |æ| < 7). We
calculate these percentage regions function of environment steps over different values.
direction with noise sampled from (a, 0.521). An episode terminates when the agent leaves the square given by {(x, y) |x| 7, ly| S 7]. The agent receives -0.1 reward every step with an additional positive reward proportional to its X-position ((100/7) |x/) when it exits the square through bottom, or - 100 reward if it leaves through either the left-or-right side the square.
In Fig. 4, we visualize trajectories for the different learned policies the true environment. Observe that small positive 3 values tend to produce risk-seeking policies where the agent aims to get as much reward (close to the walls) possible while ignoring the likelihood of hitting the sides of the square. Policies trained with negative produce risk- avoiding policies that stay in the center region. We also calculate the percentage of episodes that terminate in different risk-regions (Fig. 5) which demonstrate how 3 interpolates between different regions. We designate low-risk (left) as far from the walls, medium-risk (center), and high-risk (right) near the wall. We find that negative 3 primarily terminate in the low-risk region, whereas small positive 3 primarily terminates in the high-risk region and risk-neutral (3 100) terminates in the intermediate and lesser extent high-risk regions. We additionally include the visualizations for the learned variational dynamics in the Appendix (Fig. 7).
6.3 SIMULATED ROBOTIC BENCHMARKS
We use the MuloCo physics engine (Todorov et al., 2012) in Gymnasium (Towers et al., 2023) to evaluate our method on three continuous tasks (InvertedPendullm, HalfCheetah, and Swimmer). We follow the modifications made to the reward function in (Luo et al.. 2024) to produce risky regions in these environments based on the X-position of the agent. An additional stochastic reward sampled from (00 102) if X-position 0.01 in InvertedPendulum, - in HalfCheetah, > 0.5 Swimmer. Hence, we can test whether an agent is seekisee or risk-averse by calculating the percentage of time spent in the region of stochastic reward.
We compare our method to Mean Gini deviation (MG) (Luo et al., 2024), a policy gradient algorithm that optimizes the Gini deviation alternative risk measure and outperforms other mean-varianes
8

---

Published as a conference paper at ICLR 2025
InvertedPendullummv
Suimmeoyf
430
19:0
(20k)
Dnvitommntt Moata
Envirnment Mtpp CUO)
Invertadiannillamae
HairChectah vi
Sutmmeru4
1.0
Emw Inwrmmment stapa 1200
Enwiromenn stpa 1100
Figure 6: Risk-Averse MuJoCo. Top row: Average return on risky MuJoCo benchmarks. Bottom row: Percentage of steps episode in risky regions. The solid curves correspond to the mean and shaded regions one standard deviation over 10 random trials.
algorithms; mean- variance policy iteration (MVPI) (Zhang et al., 2021), a highly flexible algorithm that optimizes reward-volatility risk measure with the primary goal of reducing the performance gap between nestrll and risk-averse algorithms; and exponential TD (expTD) (Noorani et al., 2023), an actor-critic algorithm that optimizes the entropic risk- misk-meauure by using a critic that estimates the exponentiated return. To achieve fair comparison, we implement every actor-critic algorithm on top of TD3 (Fujimoto et al., 2018). For MG we follow the author implementation which uses PPO-style the
policy gradient to maximize the expected return (Schulman et al., 2017). For consistency same network architectures across all algorithms. We also update the policy at each environment step for all algorithms, with the exception of MG which requires the collection of 10 episodes before updating its model. See the Appendix for additional configuration details.
In Fig. 6, we report the total return (top-row) and percentage of timesteps visiting the noisy region (bottom-row) for each algorithm in a risk- averse configuration. We perform 10 runs of cach algorithm with different random seeds and report average and STDEV every 10k environment steps. Agents tested for 20 episodes per evaluation. For VAC, we include both a version where the actor-critic given by TD3 and another given by SAC (Haarnoja et al., 2018). The results show that rs VSC is effective at learning the stochasticity of the environment while producing better policies in terms of learning speed and final-perfer performaace.. MVPI also learns risk risk-averse policies in all three domains, but results in lower overall mean returns. We also perform an ablation analysis for the parameter B to demonstrate that our algorithm can learn both risk-seeking and risk-averse policies (Appendix Fig. 8).
7 CONCLUSION
In this work, we leveraged the connection between RL and probabilistic inference to formulate surrogate objective on the entropic risk measure. We proposed an EM-style algorithm that consists of learning variational dynamics and reward model that account for aleatoric uncertainty in the environment (E-step) and improves the objective w.r.t. a policy (M-step). Finally, we proposed practical algorithm (rsVAC) that permits learning of risk-seeking and risk-averse policies from experience replay alone. Our evaluations demonstrate that rs VAC is effective in learning sessitene policies in several challenging environments. In particular we show that, compared to baseline risk- sensitive methods, rs VAC performs capably in risky variants of challenging MuJoCo environments, and in all cases yields superior return.

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-22-1-0199.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint 1812.2222/000 2018a.
Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representutions, 2018b.
Nicole Bäuerle and Ulrich Rieder. More risk- iiskisensitvve markov decision processes. Mathematics of Operations Research, 19(1):105-120, 2014.
Dimitri Bertsekas. Dynamic programming and optimal control 4th edition, volume Athena scientific, 2020.
Dimitri P Bertsekas. Dynamic programming and optimal control 4th edition, volume 2. Athena scientific, 2012.
Vivek S Borkar. A sensitivity formula for risk- cost and the actor-critic algorithm. Systems & Control Letters, 44(5):339-344 2001.
Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2): 294-311, 2002.
Vivek S Borkar. Learning algorithms for risk-sensitive control. Proceedings of the 19th In- ternational Symposium Mathematical Theory of Networks and Systems-MINS, volume 2010.
Vivek S Borkar and Sean P Meyn. Risk-sensitive optimal control for markov decision processes with monotone cost. Mathematics of Operations Research, 27(1):192-209, 2002.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3381-3388. IEEE, 2017.
Wei Ming Dan Chia, Sye Loong Keoh, Cindy Goh, and Christopher Johnson. Risk assessment methodologies for autonomous driving: A survey. IEEE transactions on intelligent transportation systems, 29(10):16933-16999, 2022.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. Advances in neural information processing systems, 27, 2014.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Yinlam Chow, Brandon Cui, Moonkyung Ryu, and Mohammad Ghavamzadeh. Variational model- based policy optimization. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2292-2299. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 00.24963/jcai.22221333 URL https: //doi, wwt.///g/0/// 249634 /1996///jcai... 2021/312 Main Track.
Stefano P Coraluppi and Steven Marcus. Risk-sensitive, minimax, and mixed risk-neutralmminimax control of Markov decision processes. Springer, 1999.
10

---

Published as a conference paper at ICLR 2025
Giovanni B Di Masi and Lukasz Stettner. Risk-sensitive control of discrete- time markov processes with infinite horizon. SIAM Journal on Control and Optimization, 38(1): 1999.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Russ R Salakhutdinov. Mismatched more: Joint model-policy optimization for model-based rl. Advances in Neural Information Processing Systems, 15:23230-232443 2022.
Yingjie Fei, Zhuoran Yang. Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in neural information processing systems, 34:20436-20446 2021a.
Yingjie Fei, Zhuoran Yang. and Zhaoran Wang. Risk-sensitive reinforcement learning with function approximation: A debiasing approach. In International Conference on Machine Learning, pp. 3198-3207. PMLR, 2021b.
Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Wendell H Fleming and William M McEneaney. Risk-sensitive control on an infinite time horizon. SIAM Journal on Control and Optimization, 1386)1888-1195,, 1995.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639-32652, 2022.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning. pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:
2018.
Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp. 105-111. Elsevier, 1994.
Daniel Hernández- Hernandez and Steven I Mareus. Risk sensitive control of markov processes in countable state space. Systems & control letters, 29(3):147-155, 1996.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2016.
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7)356-369, 1972.
Wenjie Huang and William Haskell. Stochastic approximation for risk-aware markov decision processes. IEEE Transactions Automatic Control, 66(3):1314-1320 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. In ASME Transactions journal of basic engineering, pp. 82(1):35-45, 1960.
Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. Ad- vances in neural information processing systems, 26, 2013.

---

Published as a conference paper at ICLR 2025
Tze Leung Lai, Haipeng Xing, and Zehao Chen. Mean Mera-variancee portfolio optimization when means and covariances are unknown. The Annals of Applied Statistics, pp. 198-823, 2011.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 880500999, 2018.
Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. Advances in neural information processing systems, 26, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022
2015.
Yudong Luo, Guiliang Liu, Pascal Poupart, and Yangchen Pan. An alternative to variance: Gini deviation for risk-averse policy gradient. Advances in Neural Information Processing Systems, 36, 2024.
Shie Mannor and John N Tsitsiklis. Mean-variance optimization in markov decision processes. In Proceedings of the 28th International Conference on International Conference Machine Learning, pp. 177-184, 2011.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49: 267-290, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances neural information processing systems, 30, 2017.
Gerhard Neumann et al. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pp. 817-824, 2011.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-799)) 2005.
Erfaum Noorani and John S Baras. Risk-sensitive reinforcement learning and robust learning for control. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 2976-2981. IEEE, 2021.
Erfaun Noorani, Christos N Mavridis, and John S Baras. Exponential td learning: A risk-sensitive actor-critic reinforcement learning algorithm. In 2023 American Control Conference (ACC), pp. 4109-4109. IEEE, 2023.
Brendan O' Donoghue and Tor Lattimore. Variational bayesian optimistic sampling. Advances in Neural Information Processing Systems, 14:12577-121119 2021.
Brendan O' D'Donoghu, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy
gradient and q-learning. In International Conference on Learning Representations, 2016.
Brendan O' O'Donoguue, Ian Osband, and Catalin lonescu. Making sense of reinforcement learning and probabilistic inference. In International Conference on Learning Representations, 2019.
Takayuki Osogami. Robustness and sessiivnstt in markov decision processes. Advances in neural
information processing systems, 25, 2012.
12

---

Published as a conference paper at ICLR 2025
Brendan O' O'Donoghue. Efficient exploration via mristemii-risk-seekng policy optimization. Interna- tional Conference on Machine Learning, pp. 26382-26402, 2023.
Jan Peters and Stefan Schaal. Reinforcement learning by rewart-weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp.
745-750, 2007.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607-1612 2010.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference machine learning, pp- 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06:3 06347, 2017.
Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the twenty-ninth international conference machine learning, pp. 387-396, 2012.
Jean Tarbouriech, Tor Lattimore, and Brendan O'Donoguue. Probabilistic inference in reinforcement learning done right. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Emanuel Todorov. Linearly-solvable markov decision problems. Advances in neural information processing systems, 19, 2006.
Emanuel Todorov.
General duality between optimal control and estimation. 2008 47th IEEE Conference Decision and Control, pp. 4286-4292. IEEE, 2008
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG. Markus Krimmel, Rodrigo Perez- Vicent, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https: zenodo. org/ record/ 8127822.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mcan-variance policy iteration for risk-averse rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10905-10913, 2021.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A OPERATOR T PROOFS
For our formulation, we consider a finite-horizon problem for which we maximize the following objective over variational distributions ('( 1 8t, az) and (ra (8,,taa))) at):
T
log La(5 Se. (t, at) p (5 log of (relse. (ra(st,,,
(s1)
max Eg (r) (T)
Σ
r 3
(18)
2f oF .2
Let {4+" ,.., of " be an optimal set of variational distributions for this problem. By the prin- ciple of optimality, we have that the truncated set of variational distributions {g ,..",...q g is optimal for the subproblem where we start at Sk:
ax) log (8++ 18,, at)
log (re(s,,,
(SL)
max
E (T) o
Σ
(19)
B
(1 15,4115,,,00 (e)
pf (Telse, ut)
Lemma 1. Define the recursive value functions as:
TT 3
log 9 (rT(ST, 0T)
log q 1ST+1|5T,,0
VT((T)=
E or
max of
E 9.
+ max 9
pf (rTl 88, ar)
Pd (ST+1 ST 0T)
(*(8k)
E
max E
Tk 1
log
9. (k: ax)
+ +max
(88+1) log
1 ar) p Sk. ap.
oF
p= 1k: ax)
b: I...T
Then we have that V*(ss) = V (sk)
Proof. This analysis proceeds similar to the proof of Prop. 1.3.1 of Bertsekas (2020). We will show by induction that the functions V2 are equal to the optimal value functions V₁. For k = T, we have that
TT B
of (rrlst, aT) pf (rT ST, aT)
log o (sr+1ls),ar)
(ST) = max
E
log
+ E 4u
of
6
Pd (ST |ST, aT)
E or~*
max
E
FT B
log
d (rr ST aT) pf
+ max EE od o7
log
L (ST |S |sT aT)
Vi((T),
ST, (TT(8T,,7)
p -orr-lss- (T)
where the max and expectation operators commute by the principle of optimality. Now, let us assume that for some & and all 8 we have V++1(5*+1) = Vi+ ($x+1). Then
Vi2 (sx)
(Tn(T)
Σ
Ft
log (5($441/5,,0
log (ri pt (re ae)
of of of o
B
pa( 100541151,000 (s,,ae) df (ralssk, az)
t=
Pk
E
max 9.
log (TK|sKK ax)
1
F't 3
(86+11888, log 12 (8141 (8, ac)
log gl (r ax)
L (S6+1 ak) loo
max E
g ..... o7
max
E
pf pr(re(ee
p (sx+1 Sky as)
rk
(ra(sk,, ax)
ViL - log Ldt (sk+1 9k4 ax)
E
max
E
log
+ max IE 2x of
ag
of
B
pr Sk. at))
1/ ( |sk, ax)
E
max 95
E
Fk
log
or (rx 19k, ax)
+ max E. o5 of
(1++ -lo / (8k+ 182 aR) p/ 18k, ax)
V((()).
pf (rx (sk ax)
where we obtain the second equality by moving the principle of optimality, For the third equality, we use the definition of V 1+ and for the fourth equality we use the induction hypothesis. This completes our induction, and we have Vg (sx) = Vi((s) for
operator inside the expectation using the
all k.
14

---

Published as a conference paper at ICLR 2025
The ability to commute expectation and maximization operators, used in the previous proof, deserves additional discussion. Formal proofs of this result the general DP setting can be found in appendix A of Bertsekas (2012). We include discussion specific to our setting below, beginning with the assumption that each maximization step is finite:
Σ Pt - log (1++++1 az) L=
ge (r(r)stt, at) <00. log (rells ai)
Qx ak)
max g of 42 (+())
E
3
(1 s1,
Hence, for every O there exist set of variational distributions {at ...., o that satisfies that
T
log 2 18s41/11,00 as)
of (re (re)s,,ee)
E g (T)
Σ 1=
log
Qu(sk, ax)
(5( ax)
pt (relsee,
Then we have that,
T Σ
log (8(00+188 0z)
of (1)
max a 42 of .2
Bay
E (xx(r)
3
p p5((51115),
log (re)s)))
t=k
P1 3
2 (s1+1 log
o (ra]st,
Bog((T)
log
p' 18t; az)
p (Tee)tra))
T
g (3 (8r, ac)
log of (Te(ss)) 0c)
2 Eayws
max ti
Egg (r)
Σ
log
3
pl (8 18,
pf (re
l=
Since c > O is arbitrary it follows that
T
( 184,1 1/ (8 180,11181,00 az)
log (rels, of (r+ 88, a4)
max af 44 of..fff 49
Bar
Bxr(())
Σ
log
3
tmn
T
(8 18c, 18,,0)) (5
llog of (relst, at) pf (re(st,, at)
Bays
max oh "........ of
(x)
Σ
log
B
On the other hand, we have that
log (841
g (rels,,,
Easwx
max
E (T)
Σ
log (f
at of of
3
1=k
E Bayw
(r)
Σ
rt
log
y (8 (s)
of (r((ste,, log
3
p/ (3144411 st, (r a1)
L
for all sets of variational distributions {g ..... q J. So the inequality holds when taking the maximum,
log (1$641152,00) log (rels,,, of Se, ac)
Easrr
max at of of
E (x)(
Σ
3
(5 (8 at)
max
E Eaya
Eg((F)
T
Fz
log
(8++ (e)
log o (re(3te) (r (rr,st,00 a+)
B
p (5041 182, a+)
of 26 ..
t=
Combining these two results we obtain the sought equality:
T
log ( log of (rels, as)
Eays
max at ot....... of
By-(()
E
3
p (8 18141181,000 (s, az)
(re (rrel1r,0)
1=k
T
( (8 (t) ( az)
of (re(st,
max of
E Easro Pa
Ex,(T)
Σ
log
-log (re
of
B
t=k
15

---

Published as a conference paper at ICLR 2025
We now present main result for application of the Bellman- style operator.
Theorem 1. Repeated application of T to any value function such that (($++1) = 0 converges the optimal value function Vig for all k.
Proof. We demonstrate this by induction. After one application of Tx we have that
log or (rr(sr, aT) 1f (ST aT)
V(ST+1) log Le (8T+118T, Pd (ST+1|ST,
Ta (VT))
max
E
TT B
+max E o 43
of
E max
E 4
TT
log
of (rr ST ar)
+ mmax EE
log
2 (sT+1|sT or) Pd (3T aT)
= Vr((8),
B
p. (rT ST, ar)
2n 9
where the second equality uses the fact that V ((ST41) = 0. Now, let us assume that for some & and all Sk+1, have V(s++1) (8k+1)) Then
log (ru as) (x 88k ax)
((8++1) log 2 (Sk+1 1 [kk ax) ( 1 Sko ax. Vi+t(S llo (8 +1 18k: ax)) ( (8k, ank)
Ta (8)) max of
E
E 9
+max EE 42 o
B
=E E
max
Tk E
log
of (Ta|Sk; ak) p (rklskk ax)
+ max E z oa
Vi(s)),
B
where the second equality uses the induction hypothesis. This shows that after successive applica- tions of Tx recover the optimal value function Vy.
Lemma 2. For any state E S, action E A and reward distribution p(r|s, a). we have
gr(r|s,a) log p(r(s,a)
exp )
r
max (rsq.(r(s,a)
log (y-p(rlss.)
(20)
EAR
B
Analogously, for any state E S, action a E A. value function V and dynamics distribution p(s'(s,a), have
log [d(5'(s.0)
max Eg (g V(S)= 0(5'18,00
log E. (xwxptoonna))exx(((()) (21)
qdEAs
Proof. For Eq. 20, we have that
r -
-log (r(s),))
r
max (rlma)) g.EAR
max Brog. (rls,a) g.EAR
+ logp(r(ss,)) - g
B
p(r(s,a)
B
J. exp
r + logp((r)
r
log
= log Be-p(rloaa)
exp
3
where the second equality follows from Lemma in Nachum et al. (2017). Eq. follows from Lemma 5 in Chow et al. (2021).
Lemma 3. The operator Tx is monotonic.
Proof. If V, W. S + are functions such that V(s) < W(s), S. Then T (V)) T [W]ss, Vs E S. From Lemma 2, we have that
S
T (v)
E
log E rop
exp (
[1)]
+ log E [exp(V(ss))] =E, log E s'mp
r exp
3 + V(+')) )
s'rpup
where the second equality uses the fact that rls,r)s,a) = p(s'|s, a)p(sss a). Therefore,
( (=))()))
- + ( w(])]] (+)))
Ta (V)
Ex
log
E prrpp
exp
<
Ex
log E
exp
mrup
where we use the monotonicity of the exp. expectation and log operations.
Before proving Theorem 2, prove the following lemma:
16

---

Published as a conference paper at ICLR 2025
Lemma 4. Let qf be the solution of Eq. 20. Then
(rls, a) & p(r|s, a) exp (7).
exp
(22)
V Analogously, let 9a be the solution of Eq. 21. Then
ad (s'(s,a) a) x p(s')s, exp(((
(23)
Proof. For Eq. 22, we have that
exp (5 + logp(r(s, (r(s,
J, exp (5 log p(rls. (r)
(24)
where the first equality follows from Corollary 6 in Nachum et al. (2017). Eq. 23 follows from Lemma 3 in Chow al. (2021).
Theorem 2. Let of and dá be the solution of arg max, Jz(g, R). Then
(r x p(r(s, exp
Ca(8' (5(5'1s,a) x exxxx
(25)
3
Proof. We have that
log (r(r(s,a) a)
+ VV' log (a(5'18.a)
Jola" T) = (s) = max E8
33
-
p(r(s,a)
pls' (,
4r
(r)
(a(s"(s,a))
= =Eawx
max Br
log
max E Eg'ngs wEAs
Vi((') - log p 1/
[TEAR
p(r|s, s a)
where the second equality comes from the definition of V². Using Lemma 4, we conclude that
of (r(s,a) a(rls,a) exp (5) (a(5'18, x p(8' exp (5)).
B RISK-AVERSE M-STEP
In this section, we derive Eq. (11) for the two cases: 3 >> 0 and B < 0. For B > 0, we have that the M-step:
log (8r+ (e,, at) loog of (relsee at) p(relse, at)
arg max Is(a", TT) = arg max E (r)
Σ 3
p(set1
S7
(114211 at) -logg 2f (relst), p(re(se,a))
arg max B (T)
Erc-Blog Blr-Blog
p(s(41l5e, ai)
x
where the second equality comes from multiplying by B. For 0, we have that the M-step:
Σ Pt log (5 as)
log 9. (ralst,, at) p(ri (r
arg min Jo(q", TT
arg min Eg (r)
(8141|s;,a1
( St, st, at) Σ Erc - Blg
gf (relse) oz)
arg max (+)
3 log p(re|st, ax)
p(s (8,, ot)
where the arg min becomes arg max in the second equality due to multiplying through by negative 3. Hence, for both cases the M-step is equivalent Eq. (11). This shows that for 3 0, the overall optimization is equivalent to the andle--pintt problem: arg min, arg max, Ja тT. Thus, for the risk-averse setting do not have monotonic improvement the entropic objective. Nonetheless, it
17

---

approximation to the unconstrained objective when p(r)orr) 1), for which we have equality to the entropic objective (Osogami, 2012). This surrogate objective is also related to Robust MDPs (Nilim & El Ghaoui, 2005) by treating the maximization of the worst choice for an uncertain set, i.e. the set of variational distributions of the form g (T). Additionally, the objective equivalent the Minimax Criterion under inherent uncertainty (Gareía & Fernández, 2015; Heger, 1994) when B 7 0. Published as a conference paper at ICLR 2025
C DUAL OPTIMIZATION
Choosing suitable can be deciding factor between learning risk-sensitive policies divergence in practice. Hence, we propose Langrangian formulation that automatically tunes the risk-sensitive parameter for, both, the risk-secking (3 > 0) and risk-averse (3 0) settings. For O, observe that the maximization of Js(g, x) w.r.t. q reflects the Lagrangian of the following constrained optimization,
Σn s.t.
max B (T)
(26)
where hard-constraint the allowable divergence of the distribution 4, (T). We recognize as Lagrange multiplier and perform dual gradient descent (Boyd & Vandenberghe, 2004) via the loss function:
J(B) 3 (c KL (L( St, at) p($+11.77 / st,))))).
(27)
Although constraint in the primal problem of Eq. (26) suggests optimizing the dual parameter B w.r.t. the entire trajectory (L( (T)). This can lead to high variance for long trajectories. We instead impose the constraint only at single transition, which yields more stable learning. This approach most closely aligns with SAC (Haarnoja et al., 2018), which introduces a dual relaxation to modulate its policy entropy. For B < O, we now consider the following primal problem.
max (g)
Σ ce
s.t. (T) (T) S.
(28)
where the optimization is w.r.t costs C dynamics z that are within of the true dynamics KI. sense. Now consider the dual problem with Lagrange multiplier X:
rr. In other words, the agent aims to find the worst-case
min max (g)(7) ASU
Σe
+ A(E (T)
(29)
In particular, we observe that for any fixed À the maximization w.r.t. y is equivalent to maximizing J). with / -X. Hence, we propose a dual gradient descent optimization with loss function:
J(A) A( KT.(g(s++1;: 8t, ax) p(8++1, ra 8t, az))).
(30)
where we can recover by setting it to -X Again, we impose the constraint only single transitions, which yields more stable learning.
D ADDITIONAL EXPERIMENTS
D.1 VISUALIZATION VARIATIONAL DYNAMICS FOR RSVAC
In Fig. 8, we visualize the learned variational dynamics on the stochastic continuous 2D environment for a range 3 values. When 3 < O, we observe that the variational dynamics model is pessimistic and moves the agent towards the horizontal sides of the square. In contrast, when 3 >> 0 the variational dynamics guide the agent towards the regions of high reward and ignore the potential of hitting the walls.
D.2 ABLATION EXPERIMENTS FOR RSVAC
Ablation experiments for VAC using SAC as its actor-critic w.t.. risky MuJoCo benchmarks (InvertedPendullum, HalfCheetah and Swimmer) for range of 3 values. Fig. demonstrates that rs VAC is capable learning sensitiee policies in, both, the risk-averse and risk-secking regimes while achieving high reward.
18

---

Published as a conference paper at ICLR 2025
20
36
4.0
40-
15
-30
(a)
8
(c) 1100
50-
30
500
23
10
21
60
26
100 225
100 225
(d) 8 =2
(c) 8 10
(1 8
100
Figure 7: Visualizations of variational dynamics for linearly spaced coordinates stochastic continuous environment. From each state, we draw a vector to its expected next state colored by the agent's 's action: up (black). right (green), down (blue) and left (red).
Inverteediendtuaammww
Swimmeryy
30.0
12:0
25.00
19:00
20:00
Envinoement steps E3d
Drudk
Enchromment (14)
InvertedRendiilumee
HalCCheethh-vd
Swimmervut
1.C
0:2
Balanni 20.3
350
Figure 8: Ablation analysis w.r.t risk parameter 3. The solid curves correspond the and shaded regions one standard deviation over random trials. Top row: Average return. Bottom row: Percentage of steps on an episode risky regions.
D.3 ADDITIONAL MUJOCO EXPERIMENTS
Fig. 9 compares VSC (with TD3 its actor-critic) to other risk-averse baselines on the MuJoCo environment Ant-v4. Similarly to our previous experiments, we modify this environment by including an additional stochastic reward sampled from (/(.. 102) ) if X-position > 0.5. Again observe that rs VAC is effective at learning siskisennttivity while producing better policies in terms of average return.
19

---

Published as a conference paper at ICLR 2025
Ant- v4
Ant-v4
2000
NG
MVPi
2.6
ExsTD
1500
1000
2.3
5.0
-500
40
60
03
100
60
Environment stops (10k)
Environmant stops (1.0k]
Figure 9: Risky Ant-v4. Left: Average Ant-v4, Right: Percentage of steps episode in risky regions. solid curves correspond to the mean and shaded regions to + one standard deviation over random trials.
E IMPLEMENTATION DETAILS
E. STABILITY MODIFICATIONS
During the implementation of rsAAC, we noticed that the log-terms in the critic update have no effects on controlling the risk-sensitivity of the algorithm, while producing instabilities that can hurt the critic's 's convergence. For this reason, we remove them during optimization of rsVAC for continuous experiments. Another modification that we found that can improve learning for variational dynamics is the introduction of separate critic V optimized w.r.t. real environment data. This critic convenient as provides information about the return in future states while remaining independent of the variational dynamics it doesn't 't tend to become overly optimistic (or pessimistic) for 3 values with small magnitude.
E.2 CONTINUOUS 2D ENVIRONMENT
Table 1 lists the hyperparameters used by rs VAC for the stochastic continuous 2D environment.
Table Hyperparameters for stochastic continuous 2D environment
Schedule details
Environment steps before training
5000 steps
Environment steps per epoch
1000 steps
Model optimization
every steps
Number of model rollouts
128 rollouts
Rollout length
step
Network details
Discount factor
0.9
Soft target update
0.005
Experience buffer Denv
1,000,000
Model buffer Dmodel
128
Dynamics Network Architecture
MLP with hidden layers of size 256
Actor Network Architecture
MI.P with 2 hidden layers of size 256
Critie Network Architecture
ML.P with hidden layers of size 256
Network optimizer
Adam
Non-linear layers
ReLU
Learning rate
0.0003
20

---

Published as a conference paper at ICLR 2025
E.3 MUJOCO ENVIRONMENTS HYPERPARAMETERS
For MG and MVPI, we use the implementations in Luo et al. (2024) and follow the same hyperparam- suggested by the authors. We implement expTD (Noorani al., 2023) top of the TD3 imple- mentation in Luo et al. (2024) and select the -10 as its risky parameter from 1-1,-16 10, 220, -100). For VAAC we select its risk parameter from {-1.- -4, -8). For SAC we use remplemmmenanionn of that algorithm made available by other authors We use the same network architectures and learning rates for all algorithms. Table 2 lists the hyperparameters used for VAC and all actor-critic algorithms in risk-aware MuloCo benchmarks.
Table 2: Hyperparameters for risk-aware MuJoCo benchmark
Schedule details
Environment steps before training
5000 steps
Environment steps per epoch
1000 steps
Model optimization
every step
Number of model rollouts
256 rollouts
Rollout length
step
Network details
Discount factor
0.99
Soft target update
0.005
Experience buffer 'Dewv
1,000,000
Model buffer Dmustel
256
Reward Network Architecture
MLP with 2 hidden layers of size 256
Actor Network Architecture
MLP with hidden layers of size 256
Critic Network Architecture
MLP with 2 hidden layers of size 256
Network optimizer
Adam
Non-linear layers
ReLU
Learning rate
0.0003
initialization
-1 (-8 for invertedPendulum with TD3)
initialization
0.2 (for SAC implementation only)
F PSEUDOCODE OF RSVAC
This section contains the pscudocode for our algorithm, rs VAC.
hhans///gthhh.oom/ingggoiino pytorch
21

---

Published as a conference paper at ICLR 2025
Algorithm VAC
Initialize networks, parameters and replay buffers. for each epoch do
for each environment step do 0t ~ To(a,(a))
8t+1, Fy (8. ar)
Sample next state from environment.
Denv + Denv U {1t, at, 7t41,
Add tuple to experience buffer.
if model optimization then
{ss of, s 1+ r))) ~ Denv
Sample every tuple in experience buffer.
0 < (J((
Update prior dynamics pa- Update prior reward pa- Update variational dynamics da- Update variational reward ds-
0 + (J)
Φ 6 VJa(o)
Φ + 6 VJ,(())
for 1,2,.., M do
8t D
Sample state from experience buffer Deny Sample action using policy. Sample next state using variational dynamics. Sample reward using variational reward model. Add tuple to model buffer.
a; To (st)
81+1 ~ q6 S1+ (000(841188,6400
Vi ~ (o Sc; ae)
Dmodel f- D model U {(S,, ae 8141,
end for end if
for kt = 1,2 2, ..., K do
((}: oi 8:41 rf)
Dinddel
Sample mini-batch from model buffer DD model-
1 f- dh (J(t))
Update critic Q Update policy ma- Update target critic Q''
0 - (J
& + Tu + ()
end for
end for end for
22


Reviews:
[{"review_id": "MncqJRizMq", "reviewer": "Reviewer_AxU6", "summary": "This work presents an approach to optimize the entropic risk objective in RL by formulating it as a RL-as-probabilistic-inference objective, and optimizing a variational lower bound. This lower bound is solved using two iterative steps, where in one step the variational dynamics of the environment are learned, and in the other step, an actor-critic algorithm optimizes for a policy with a modified reward.\nThe risk-sensitivity of the approach arises from the sign of a parameter $\\beta$, and the notion of risk here is related to the variance of the returns. Experiments in tabular and continuous environments demonstrate that the algorithm can learn risk-seeking and risk-averse behaviors based on $\\beta$ and outperforms similar algorithms in the risk-averse setting.", "questions": "- You mention in L040 that optimizing the entropic risk requires knowledge of the transition kernel, citing Noorani et al, 2023. To my knowledge the algorithms proposed in that work (particularly R-AC -- Risk-sensitive online Actor-Critic) do not require knowledge of the transition kernel and propose an actor-critic algorithm to optimize the entropic risk much like this work. Could you expand on the comparison between the two works and what advantage this variational approach has to the R-AC of [Noorani et al., 2023]?\n- Regarding objective (7), you mention that the controller may still learn risk-seeking or risk-averse behavior in L160 “if the change in return sufficiently compensates”. What does this mean? \n- In Section 4, it is mentioned that the model learning objectives are optimized using SGD. Since this is an online algorithm and these steps are done iteratively, does this mean that one step of SGD was taken for the optimization at each environment collection step or multiple steps, or optimized up to some convergence criteria? (e.g. L214, L222, L233) How was this decided in practice?\n\n\n## Post-rebuttal\nIncreased my score as my questions have been answered", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The presentation of the paper was mostly clear, rigorous, and well-written.\nThe illustrative example of Figure 1 was clear and helpful in understanding. There were a good variety of experiments in different domains. The 2D stochastic environment experiments nicely show the effect of using $\\beta$ to modulate risk behavior of the policy. \n\nThe paper has originality and significance. From my understanding, these come from extending the RL-as-inference framework to stochastic rewards, and using this framework to also optimize for risk-averse policies.", "weaknesses": "- I think the presentation of the ELBO objective needs to be improved to clearly indicate that this is not an original objective of this paper. There are also many similarities to [Chow et al, 2021] in the overall formulation and the Bellman-like operators in the E-step optimization. The difference between these works needs to be highlighted more than what is given in L302-303. \n  - Since a major difference between the two works is given as VMBPO only optimizing for risk-seeking behavior, I am wondering: would you not be able to get a risk-averse policy by choosing the temperature $\\eta < 0$ in VMBPO as well?  \n\n\n- The algorithm presented, rsVAC, is quite complicated and potentially subject to a lot of instabilities in practice. My main concerns are:\n  - (E-Step, section 4.1) Learning the variational dynamics and reward requires first to learn a model of the true dynamics and reward, $p_\\theta$. How does the error in $p_\\theta$ affect the variational distributions?\n  - $V_\\psi$ is used in Eq. (15) to learn the variational dynamics, and on the other hand it is optimized in the actor-critic step using the learned variational dynamics. I expect this would be quite difficult to stabilize in practice. Could you comment on that? \n  - $\\beta$ parameter sensitivity: Is the convergence of the algorithm very dependent on the $\\beta$ chosen? How was this sensitivity observed in the experiments?\n- I don’t agree that Fig 3a shows rsVAC is more efficient at learning and obtains a higher return (L360). It looks like all the curves are within each other’s variability. I think more runs (~30) are needed before a conclusion can be made, which I feel is not unreasonable since this is a tabular environment.\n\nOverall I think the originality and contribution of the paper are significant enough to be accepted, however, I have doubts as to the usability of the main algorithm in practice. \n\nMinor comments:\n- I found the presentation of (12) confusing, it would be better if they are presented as equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjxIzL7PYy", "reviewer": "Reviewer_XK4d", "summary": "This paper proposes a risk sensitive model based actor critic algorithm based on variational inference principles; in a risk aware RL setting. The authors extend the setting to stochastic rewards, compared to past works on deterministic reward setting when studying risk sensitive RL, and proposes a EM style algorithm that alternates between learning the model and optimizing the policy. The algorithm is derived based on RL as probabilistic inference; studies the risk awareness in policy optimization setting; studies the challenges of model learning in stochastic reward setting - and experimentally demonstrates the efficacy of the proposed algorithm on a suite of benchmarks.", "questions": "1. Can the authors comment on the instability issues related to comment in weakness (1)? \n\n2. Have there been any ablation study related to instability when optimizing the different objectives, involved in model learning and then doing policy optimization? \n\n3. I would like to see more experiments comparing to past baselines studying risk aware RL; even if in deterministic settings. How well do those algorithms scale to stochastic rewards, and this can help demonstrate the effectiveness of rsVAC itself. \n\n4. Are there simple suite of tasks, not necessarily control based Mujoco ones - where rsVAC can be shown to be substantially useful? Or I do not see the novelty in the algorithm - and the way it is derived is quite standard too; so maybe I am missing something here? Section 6.2 seems useful in this context - but what happens to policy learning in this setting?\n\n5. Objective-wise, can the authors comment on what makes the proposed objectives interesting to look into, which past works have ignored?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper has some of the following strengths : \n\n1. Unlike past works, the authors study both the risk-seeking and risk-averse settings and proposes a model based policy optimization algorithm based on first principles of RL as inference. \n2. The algorithm is derived based on standard formulation of entropic risk measure and the proposed objectives are derived based on the risk criterion. \n3. The paper is well written with the derivation of the key objectives for learning the model (rewards and transitions) and policy optimization well explained. \n4. Overall, for this setting, there are few aspects to consider when learning both the model and the policy. The authors extend the setting to the stochastic rewards setting and shows how a suitable model can be learnt even under stochastic rewards - a common setting to consider when studying risk awareness of learnt policies in RL. \n5. The EM Style algorithm derivation and the alternative optimization of the objectives are natural to consider when studying in this setting - and the authors do a good job in deriving a suitable algorithm that can work empirically. \n6. Experiment results demonstrates the usefulness of the rsVAC algorithm under few tasks - and takes some standard baselines on top of which the risk awareness criterion can be easily integrated to study effectiveness of the proposed algorithm.", "weaknesses": "Although the work seems good, there are few weaknesses that concern me about this work : \n\n1. The alternative EM style optimization of the objectives seems to bring in a lot of instability in the overall learning process. This has been a common issue in several past works on risk aware RL; for example, past works that even studied mean-variance RL, or when minimizing a variance measure in presence of inherent stochasticity in the environment. Similar to past works - I am worried about the effectiveness of the algorithm, beyond tasks that are demonstrated here, with the minimal baselines that are used for comparison. \n\n2. Experimental results do not seem convincing enough, and it is not clear what are the different aims of each experiment. It would be useful to better structure the experiments section to make it clear what different aspects the rsVAC algorithm seems promising in? \n\n3. There are several baselines for comparison in this line of work. I understand the work extends it to stochastic rewards setting -but to be more comprehensive, it would be useful to even show how effective past algorithms are in the stochastic rewards setting? \n\n4. I am not sure if the chosen baselines for experiments are suitable for the tasks? Can the authors compare to some more baselines?\n\n5. The derivation of the algorithm, based on the EM style approach seems useful - but there has been past works which would do in a similar way? What’s the novelty here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ifVXFyB50N", "reviewer": "Reviewer_m2Pi", "summary": "This paper proposes to solve risk-sensitive reinforcement learning (RL) problems with entropic risk measures based on control as a inference perspective. After deriving the evidence lower bound (ELBO) of the log-marginal likelihood of optimal trajectory, the introduces a EM-based algorithm to maximize the ELBO. For E-step, the author introduces a Bellman-like operator and arguses that it is a contraction map. Based on this theoretical result, the E-step can be implemented as a successive application of this Bellman-like operator. M-step is introduced as a standard RL problem with augmented reward function. The proposed method can be interpreted as a model-based actor critic method. Experimental studies shows the effectiveness of the proposed method.", "questions": "### Suggestion\nPlease address my concerns, especially on the theoretical arguments. If the theoretical results are validated, I am happy to re-evaluate the manuscript.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Risk-sensitive RL is an important research direction and entropic risk measure is a popular risk measure. Practical algorithm for this objective has a large group of potential audiences (significance).\n- Experimental results are informative. In particular, the experimental studies in Section 6.1 and 6.2 are very good show cases of the concept of this paper (quality, significance).", "weaknesses": "Despite the strengths, the paper has clear weaknesses.\n\n### Major concerns\n\n- I am not convinced of the validity of the theoretical results. The proof is not concrete and self-complete, and the assumptions are not clearly stated. As a result, I am also not convinced of the validity of the proposed algorithm, despite its empirical performance. The followings are my concerns on the theoretical arguments.\n  - L.732, the distribution $p$ is different for $s^{\\prime}$ and $r$ as we can see from Eqs. (18) and (19). How the second equality holds?\n  - How the contraction statement in L.739 - L.743 follows? I believe the inequality in L.741 - L.742 is not trivial for the current definition of $\\mathcal{T}_{\\pi}$ and needs a formal proof.\n  - How the successive application of $\\mathcal{T}_{\\pi}$ results in the second equality in L.755? In general, maximum and expectation is not commute.\n  - Even though the main text does not assume that the horizon of MDP, $T$, is a random variable, \"the transient assumption of stopping MDPs\" is used in the beginning of the page 15.\n\n- In the abstract, the author states that \"algorithms that optimize this objective have been restricted to tabular or low-dimensional continuous environments\". However, the experiments in Section 6.3 is still limited to the smaller problems in Mujoco environments. To better demonstrate the scalability of the proposed method, experimental evaluation in larger domain, such as Humanoid, is necessary.\n\n\n### Minor issues\n\n- L.752, typo: T -> \\mathcal{T}_{\\pi}\n\n\n## After reading author responses and revised manuscripts\n- Thank you for revising the paper. The theoretical arguments are almost clear. I raise the score from 3 to 5 (as of 11/25).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oRICguHXAX", "reviewer": "Reviewer_azQM", "summary": "The paper presents a risk-sensitiv RL algorithm based on the idea of RL-as-inference. It presents a thorough derivation and a diverse set of experiments that highlights how the agent can be tuned to be risk seeking or avoiding with a simple parameter.", "questions": "For standard RL as inference, it is known that variational algorithms produce optimistic policies if the transition dynamics are learned and optimized over. This is acknowledged in Section 3, yet I would encourage the authors to provide a more in-depth discussion and maybe some experiments if and when this poses a problem, especially for risk-averse agents? Also, is this phenomenon reversed for risk-averse agents, do they become more pessimistic? I expect some of this depends on the ratio of r_max vs beta?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is easy to follow, if somewhat dense in places. The idea is straightforward an well executed, building on a well-established framework.\n\nThe experimental section is thorough and contains both interesting small scale and reasonable large scale experiments. It would have been interesting to see if more complex or large dynamics systems such as image-based observations pose a challenge to the approach (I assume it would due to the difficulty of parameterizing a dynamics model?), but this is clearly something that can be relegated to future work.\n\nOverall, my review is somewhat shorter than normal because I genuinely can't find too much to criticize. I think it's well done!", "weaknesses": "Overall, the paper does not have any major weaknesses that I would consider crucial for the decision to accept or reject.\n\nAs pointed out, the paper is somewhat dense in its writing. Some of the mathematical formalism could potentially moved to the appendix to declutter the main body, but given that the ideas presented are relatively standard and don't require exotic notation, this does not detract from the readability too much. The main thing contributing to this is that inline equations are used somewhat frequently.\n\nOne point that caused me a bit of confusion was that beta acts somewhat unintutively in that small absolute values of beta correspond to more extreme risk-averseness/seeking behavior, while large values correspond to the standard setting. This is briefly pointed out, but it could be elaborated on a bit more so that readers don't stumble.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)