Stochastic Approximation of Gaussian Free Energy for Risk-Sensitive Reinforcement Learning
Authors: Grégoire Delétang
Jordi Grau-Moya
Markus Kunesch
Tim Genewein
Rob Brekelmans
Shane Legg
Pedro A Ortega
Venue: NeurIPS 2021 Submitted (2021)
Keywords: risk-sensitivity
free energy
stochastic approximation
model-free reinforcement learning
safety

Abstract: We introduce a stochastic approximation rule for estimating the free energy from i.i.d. samples generated by a Gaussian distribution with unknown mean and variance. The rule is a simple modification of the Rescorla-Wagner rule, where the (sigmoidal) stimulus is taken to be either the event of over- or underestimating a target value. Since the Gaussian free energy is known to be a certainty-equivalent sensitive to the mean and the variance, the learning rule has applications in risk-sensitive decision-making. In particular, we show how use the rule in combination with the temporal-difference error in order to obtain risk-sensitive, model-free reinforcement learning algorithms.

Stochastic Approximation of Gaussian Free Energy for Risk-Sensitive Reinforcement Learning
Anonymous Author(s)
Affiliation
Address
email
Abstract
We introduce stochastic approximation rule for estimating the free energy from i.i.d. samples generated by Gaussian distribution with unknown mean and vari- ance. The rule is simple modification of the Rescorla- Waggerr rule, where the (sigmoidal) stimulus is taken to be either the event of over- or underestimating a target value. Since the Gaussian free energy known to be cerfainty-equivalen sensitive to the mean and the variance, the learning rule has applications in risk- sensitive decision fecision-making, tion with the reinforcement learning algorithms.
particular, show how to use the rule in combina- error in order to obtain risk sen-seens,,ve, model-free
Introduction
(( be the Gaussian pdf with mean j 11 Main contribution. Let p)
12 and precision p. Given a sequence #1, 22.... of i.i.. samples drawn from N(;; p. p) with unknown and consider the problem of estimating the free energy F, for given inverse temperature 8 R.
14 that is
3 log / N(: p. p) exp(3x) dæ μ +
8
F3
(1)
2p
This paper shows that (1) be estimated using surprisingly simple stochastic approximation rule. v E IR is the current estimate and new sample x arrives, update v according to
16
+ + 2xx a,(x v) (x)+), v),
where > 0 is learning rate and (2) is the scaled logistic sigmoid
(2(2) exp(-
18
The unique and stable fixed point of the learning rule (2) is equal to the desired free energy value
19
8"
2p
Motivation. Risk-sensitivity, the susceptibility to the higher-order moments of the return, is neces- 21 sary the real-world deployment Al agents. Wrong assumptions, lack of data, misspecification, limited computation, and adversarial attacks are just a handful of the countless sources of unforeseen
20
23 perturbations that could be present at deployment time. Such perturbations can easily destabilize risk-neutral policies, because they only focus maximizing expected return while entirely neglecting the variance. This poses serious safety concerns (Russell et al. 2015 Amodei al. 2016 Leike al. 2017).
25
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

27
sisk-sensitive control has a long history in control theory (Coraluppi 1997) and is an active area of research within reinforcement learning (RL). There are multiple different approaches risk- sensitivity in RI: for instance in Minimax RI, inspired by classical robust control theory, one derives A conservative worst-case policy over MDP parameter intervals (Nilim and El Ghaoui |005 [aamar et al. 2014); and the more recent CVaR approach relies on using the conditional-value- at-rikk as robust performance measure (Galichet et al. [013] Cassel et al., 2018). We refer the reader to (aarcía and Fernández (2015) for comprehensive overview, Here we focus one of the earliest and most popular approaches (see references), consisting of the use of exponentially-trane
28
29
30
31
32
33
34
values, or
35
equivalently, the free energy as the risk- cortainty-equivalne
(Bellman 1957 Howard and
36
Matheson 1972).
37
The certainty-equivallee of a stochastic value X E IR is defined as the representative deterministic value v E R that a decision- uses as a summary of X for valuation purposes. To illustrate, consider first-order Markov chain over discrete states S with transition kernel P(s state-emitted
38
39
40
rewards R(s) E TR, and discount factor ? E 0. 1). Typically RL methods use the expectation as the eertainty-qqiivalent of stochastic transitions (Bertsekas and Tsitsiklis 1995 Sutton and Barto 2018). Therefore they compute the value (s) of the current state 8 E S by (recursively) aggregating the future values through their expectation, e.g.
41
42
43
V(s)= / 100==P('110 P())))((())) +V)()))))+++)))))))))))))))))))) ds'
(4)
44 Instead, Howard and Matheson
1972) proposed using the free energy as the certainty-cuiivalent
45 that is,
3 log f P(s'ls) exp( +V()))))))++())))))))) ds'.
(ss) Fs(()
(5)
46 where B E R is the inverse temperature parameter which determines whether the aggregation is
47 risk-averse (3 < 0), risk-seeking (3 > 0), or even risk-neutral as a special case (3 = 0). Indeed,
48 the future values are bounded, then (s()) is sigmoidal in shape as a function of 8, with three special
49 values given by
(R(s') + ++)(+
3 + -00;
lim (x(() = 3
E|R(S")++VS*") + = s] maxxx-(Rss)) +s)('))
i + 0;
(6)
if 3 +00.
50 These limit values highlight the sensitivity to the higher-order moments of the return. Because of this
51 property, the free energy has been used as the vertainty-equivaled for assessing the value of both
52 actions and observations under limited control and model uncertainty respectively, each effect having
58 their own inverse temperature. The work by Grau-Moya et al. (2016) is a demonstration of how to incorporate multiple types of effects in MDPs.
54
The present work addresses longstanding problem pointed out by Mihatsch and Neuneier (2002). 56 An advantage of using expectations is that certaity--quivaleen such as (4) are easily estimated using stochastic approximation schemes. For instance, consider the classical Robbins-Monro update Robbins and Monro 1951)
57
58
t where x ~ P(z) is a stochastic target value, a is a learning rate, and v is the estimate of E Substituting x = R(s') + (s') and v = V(s) leads to the popular TD(0) update (Sutton and Barto
(x- v)
59
60
61
1990):
V(s) + Vs) + (R(s)) + +VV(5) - V(s)).
However, there is no model-free counterpart for estimating free energies (3) under general unknown distributions. The difficulty lies in that model-free updates rely on single (Monte-Carlo) unbiased samples, but these not available in the case of the free energy due to the log-term on the r.h.s.
63
65
of This shortcoming led Mihatsch and Neuneier (2002) to propose the alternative risk-sensitive
86 learning rule
+ auta a u. v), where u (1- k)
if if v) > 0 (11 R) (I v) < 0
67 and where e E 1-1;1] is risk-sensitivity parameter. While the heuristic p does produce risk- sensitive policies, these have no formal correspondence to free energies.
2

---

a) Anatomy of the weighted update
b) Weighted update for different e
Error function c(d,8)
1.0
e
0.5
0.05
0.0
0.00
p)
0.05
2-
-0.5
-0.10
-1.0-
f(a)
0.15
p
-0.20
Figure 1: Update rule and its error function. a) shows the update fax to the estimate e caused by the arrival of a sample æ, weighted by its probability density, The expected update determined by comparing the integrals of the positive and negative lobes. b) Illustration of weighted update functions f(x) for different values of the current estimate v. The positive lobes are either larger, equal, or smaller than the negative lobes for a that is either smaller, equal, or larger than the free energy respectively. Error function implied by the update rule. For a risk-neutral (3 = 0) estimator the еттоr function is equal to the quadratic error e(ő, 0) 382. For risk risk-averse estimator (3 < 0), the error function lopsided, penalizing under-estimates stronger than over-estimates. Furthermore, e(ō, (3) is even function in
As anticipated, our work contributes simple model-free rule for estimating the free energy in the 70 special case of Gaussian distributions. Starting from the Rescorla- Rescorla-Wagner Wagner rule
<- 2 + & u (x v),
(10)
where E {0, 1} is an indicator function marking the presence of stimulus (Rescorla 1972),
72 substitute by twice the soft-indicator function g - v) of (3). which activates whenever either
73 over- or underestimates the target value x, depending on the sign of the risk-sensitivity parameter Using the substitutions appropriate for RI, we obtain the risk-sensitive TD(0)-rule
74
V(s) V(s) + 2a (0()) 5.
(11)
where 5 = R(s') + >V''' Vs)) is the standard temporal-Ce -irooaaldifffrencce error. The learning rule is trivial to implement, works as stated for tabular RL, and is easily adapted to the objective functions of deep RL methods (Mnih al. [215). Finally, the learning rule is also consistent with findings in computational neuroscience (Niv et al. 2012), e.g. predicting asymmetric updates that are stronger for negative prediction errors in the risk-averse case (Gershman) 2015).
76
78
79
2
Analysis of the Learning Rule
80
Our central result the following lemma, which implies that the unique and stable fixed point of the 81
82 expected learning dynamics of (2) is given by the desired free energy.
83 Lemma 1. 2 Z2.... are i.i.d. samples from P(X) = N N p. p), then the expected update J (v)
84 of the learning rule ( is twice differentiable and such that
0,
if F
J(o) = - v) b)]
0,
if F F3:
0.
if v F
85 Proof. The expected update of is
J(v) ( N N(z.
v)(x dx,
(12)
where we have dropped the subscript 3 from as for simplicity. Using the Leibnitz integral rule it 87 easily seen that this function twice differentiable w.r.t. v, because the integrand is product of twice differentiable functions.

---

89 The resulting update direction will be positive if the integral over the positive contributions outweight
90 the negative contributions and vice versa. The integrand of (12) has symmetry property: splitting
91 the domain of integration R into (~o0; v] and (w; +o)), using the change of variable å = v, and
92
recombining the two integrals into one gives
(v) (())== : L. {N(u+ (:: N( d; p))
(13)
We will show that the integrand of (13 is either negative, zero, positive, depending on the value 94 of v. Define the weighted update f(x) as
f(x) f(v 8) == N(v + 6: μs. (5)6.
This function illustrated in Figure D.. We are interested in the ratio
95
f(v + 5) N(v 6: o()
(14)
ʃ(0 1 6) N(v - p. o(-))
which compares the positive against the negative contributions to the integrand in 13 The first fraction of the r.h.s. of (14) is equal to
97
(v + 1z 8)
N(v - 6: p. exp - + - 1)² - 6 expl -2p6(v 1)}_
-
98
Using the symmetry property (6) 1- - (- of the logistic sigmoid function, the second fraction can be shown to be equal to
o
o(6) = exp(
o 8) 1- (6)
100
Substituting the above back into
14 results in
for u < 4
3 1(u + = expp--2p0(v 14) + (8} for == + f(u- - 5)
3g
for v > μ 2p
also illustrated in Figure Dp. Therefore, the integrand in 13 is either positive (v p 2) zero
101
102
(v=
A ) negative (v p 2, allowing conclude the claim of the lemma.
3 Additional Properties
104
We discuss additional properties in order to strengthen the intuition and to elarify the significance of the learning rule; some practical implementation advice is given at the end.
105
108
Associated free energy functional. The Gaussian free energy F in (I is formally related the valuation of risk-sensitive portfolios used in finance (Markowitz) 1952). It well-known that the free energy is the extremum of the free energy functional, defined as the Lewbee--eeebrbeullllaee expectation of X:
107
108
109
Fa[p(z)] = Ep[ (L(p())||
(15)
110
This functional is convex in p for 3 and concave for 0. Taking either the minimum (for 3 < 0) or maximum (for 3 > 0) w.r.t. p(x) yields
111
8
8 2p
3 2
F3 extr extr Fy [p())] 1 p(a)
E[X]
Var[X],
(16)
B _p
112 that is, the Gaussian free energy is linear function of 3, where the intercept and the slope are equal
113 to the expectation and half of the variance of X respectively. The extremizer p" (x) is the Gaussian
p" (x) arg extr Fa [p(x)] N u a p.p)
(17)
p(m)
114 The above gives a precise meaning to the free energy as eettainty-equivalentt The choice of
115 non-zero inverse temperature 3 reflects a distrust in the reference probability density N(x; μ4, p) as a reliable model for X. Specifically, the magnitude of 3 quantifies the degree of distrust and the sign of
117 indicates whether it is an under- or overestimation. This distrust results in using the extremizer robust substitute for the original reference model for X.

---

119
Game-theoretic interpretation. In addition to the above, previous work (Ortega and Lee 2014
120
Eysenbach and Levine 2019 Husain et al.] (02T)) has shown that the free energy functional has an
interpretation as two-player game which characterizes its robustness properties. Following OOtega
122
and Lee (2014), computing the Sngendre-Fencher dual of the KL. regularizer yields an equivalent adversarial re-statement of the free energy functional
123
15
which for / > O is given by
{ p(x)(== c(x)] dr
/ (s dr.
max min
(18)
p(a) c(a))
124
where the perturbations o(x) E R are chosen by an adversary (Note: for the case B < 0 one obtains a Minimax problem over p(x)) and c(x) rather than a Maximin). From this dual interpretation, one sees that the distribution p(r)) is chosen as if it were maximizing the expected value of x' = x c(z),
125
126
127
the adversarially perturbed version of x. turn, the adversary attempts to minimize x', but at the
128
cost of an exponential penalty for c(ar). More precisely, given the distribution p(z), the adversarial
129
best-response (ignoring constants)
1 log pL 2B p(x 11) - -p(x 73)² + log
p(x)
P p
F3:
(19)
130
where the equality (a) is true for any choice p(z); (b) holds if p(x) = (a;; pl p) for some mean jú and precision pt and where (c) holds if p(x:) is the extremizer (17). Here we see that the adversarial
132
perturbations can be arbitrarily bad p(z) is not chosen cautiously: for instance, for the (Gaussian)
Dirac delta
pax) (.) 80000 5(x 14)
get ((=)=o((1))) ))
81000 +00.
(20)
Let x- be the instantaneous difference between the sample and the estimate. 135 If the update rule (2) corresponds to a stochastic gradient descent step, then what is the error function?
Error function.
136 That is, if
- v - a Voe(d, 3) = v 2a (6) 6,
137
then what is e(6, 8)? Integrating the gradient Voe(d, 3) with respect d gives
28
2
π
e(,,3) 2 (
log(1 + exp( + 3
liz exp( +
(21)
3
6.32
138 where log(1 + exp(z)) is the softplus function Dugas et al. 2001 and liz(2) is Spence's 's function (or dilogarithm) defined as
139
li2(z) = log 2)
dz,
z
140
and where the constant of integration error function is illustrated in Figure I for handful of values of B. In the limit 3 0, the error function becomes:
chosen that lims- e(ō, B) 0 for all 8 e R. This
142
1
lim e(,, 3) () - 8-0
143
thus establishing a connection between the quadratic error and the proposed learning rule.
Practical considerations. The free energy learning rule 2 be implemented stated, for instance either using constant learning rate a > 0 or using an adaptive learning rate a, > fulfilling the Robbins-Mone conditions Σ, aq 0 and Σ, a, 3c.
145
147
problem arises when most of the data falls within the near-zero saturated region of the sigmoid, which can occur due to an unfortunate initialization of the estimate v. Since then g - v) Rs 0 for most x, learning can be very slow. This problem can be mitigated using an affine transformation of the sigmoid that gaurantees minimal rate 7) > 0, such as
148
149
T²(2) = 17 + (1 2n))
(22)
151 which re-seales the sigmoid within the interval [n, - 17). We have found this adjustment to work well for [3] R 0, especially when it is only used during the first few iterations.
152
153 If one wishes to use the learning nule in combination with gradient-based optimization (as is typical
in deep learning architecture), we do not recommend using the error function (1 directly. Rather,
5

---

Estimation from Gaussian samples
Estimation from uniform samples
data
data
1000
2000
3000
4000
1000
2000
3000
4000
number of data points
number data points
Figure 2: Estimation of the free energy from Gaussian (left panel) and uniform samples (right panel). Each plot shows 10 estimation processes (9 in pink, 1 in red) per choice the inverse temperature, where B E {-4, 22 0, 2, 4). The true free energies shown in black. The estimation of the free energy is accurate for Gaussian data but biased for uniform data.
155 we suggest absorbing the factor (0) directly into the learning rate (where as before, x v)) simple way to achieve this consists in scaling the estimation error (() by said factor using 157 stop- graddiadnnt, that is,
(6) == (2 E'(),
(23)
since then the error gradient with respect to the model parameters @ will be
OE dv
(O - (5)
(24)
06 00
Finally, a large |3] chooses target free energy within tail of the distribution, leading slower 160
convergence. one wishes to approximate a free energy that sits at n standard deviations from the mean, then 3 should be chosen
((n) = 2n \pp
(25)
162
However, since ((n) is not scale invariant and the scale is unknown, good choice of 3 must be determined empirically.
183
4 Experiments
184
165
Our first experiment simple sanity check. We estimated the free energy in an online manner using the learning rule (2) from data generated by two i.i.d. sources: standard Gaussian, and uniform distribution over the interval [- 2]. Five different inverse temperatures were used (3 {-4,- -2, 0, 2, 4)). For each condition, we ran ten estimation processes from 4000 random samples using the same starting point (v = 1.5). The learning rate was constant and equal to
Estimation.
186
167
168
189
170
0.02.
171
The results are shown in figure In the Gaussian case, the estimation processes successfully stabilize around the true free energies, with processes having larger |B] converging slower, but fluctuating less. In the uniform case, the estimation processes do not settle around the correct free energy values for 3 A O; however, the found solutions increase monotonically with B. These results validate the estimation method only for Gaussian data, as expected.
173
174
175
Reinforcement learning. Next applied the risk-sensitive learning rule to RL in simple grid- world. The goal was to qualitatively investigate the types of policies that result from different risk-sensitivities. Shown in Figure 3a. the objective of the agent is navigate to terminal state containing reward pill within no more than 25 time steps while avoiding the water. The reward pill delivers one reward point upon collection, whereas standing in the water penalizes the agent with minus one reward point per time step. In addition, there a very strong wind: with 50% chance in each step, the wind pushes the agent block in randomly chosen cardinal direction.
177
179
181
182
trained R2D2 (Kapturowski al. 2018) agents with the risk-sensitive cost function (23 using five uniformly spaced inverse temperatures 3 ranging from -0.8 to 0.8. The architecture of our
185

---

b)
Return
Violations %
B
-0.8
-0.4
0.0
0.4
0.8
0.00
Figure 3: Comparison of risk-sensitive RL agents. a) The task consists in picking up reward located the terminal state while avoiding stepping into water. A strong wind pushes the agent into random direction 50% of the time. b) Bar plots showing the average return (blue) and the percentage of violations (red) for each policy. ordered from lowest highest B. e) State visitation frequencies for each policy, plus the optimal (deterministic) policy when there is no wind (black paths).
185
agents consisted of a first convolutional layer with 3-by-3- --reellleels and 128 channels, a dense layer with 128 units, and logit layer for the four possible actions (i.e. walking directions). The discount factor was set to y = 0.95. Each agent was trained for 500K iterations with batch size of 64, using Adam optimizer with learning rate 10-4 (Kingma and Ba 2014). The target network was updated every 400 steps. The inputs to the network were observation tensors of binary features representing 2D board. Note these agents didn't use any recurrent cells and therefore no backpropagation through time was used. To train all the agents in this experiment we used 154 CPU core hours at 2.4
186
187
186
190
192
GHz and 22.5 GPU hours.
193
To analyze the resulting policies, we computed the episodic returns and the percentage of time the agents spent in the water (i.e. the "violations") from 1000 roll-outs. The results, shown Figure Bp. reveal that the risk-neutral policy (3 0) has the highest average return. However, the percentage of violations increases monotonically with Fi shows the state-visitation probabilities estimated
194
195
196
from the same roll-outs. There are essentially three types of policies: risk-averse, taking the longest path away from the water; resk-neu taking middle path; and risk-seeking, taking the shortest route right next to the water. These are even more crisply revealed when the wind is de-activated. Interestingly, the risk-averse policy (3 -0.8) does not always reach the goal, which explains why its return is slightly lower in spite of committing fewer violations.
196
200
202
Bandits. In the last experiment we wanted to observe the premiums that risk-sensitive agents are willing to pay when confronted with choice between a certain and a risky option. To do so, we used a two-arm bandit setup, where one arm ("certain"") delivered a fixed reward and the other arm ("risky") stochastic one-more precisely, drawn from Gaussian distribution with mean l and precision p 2. Both the fixed payoff and the mean of the risky arm were drawn from a standard Gaussian distribution at the beginning an episode, which lasted twenty rounds. To build agents
203
205
207
that can trade off exploration versus exploitation, used memory- based meta-learmin (Wang et al. 2016 Santoro et al. 2016), which is known to produce near-optimal bandit players (Ortega et al. Mikulik et al. 2020).
209
2019
211
We meta-trained five R2D2 agents using risk-sensitives E -1.00, 1.0) on the two- armed bandit task distribution (also randomizing the certain/risky arm positions) with discount factor 0.95. The network architecture and training parameters were as in the previous RL experiment, with the difference that the initial convolutional layer was replaced with dense layer and an LSTM layer having 128 memory cells (Hochreiter and Schmidhuber) 1997). We used backpropagation through time for computing the episode gradients. The input to the network consisted of the action
212
213
215

---

A=-10
A -0.5
0.0
A 0.5
1-0
0.5
Reoeeemm Amm
0.0
0.0
0.0
0.0
0.0
25
0.5
0.5
0.5
3.0
0.5
0.5
0.0
0.5 0.5
5.5
0.5
0.5
0.0
0.5 0.5
0.0
0.5
Figure 4: Two-armed bandit policy profiles with different risk-sensitivities 3. The certain arm I pays deterministic reward, while the risky arm 2 pays stochastic reward drawn from (r; 24 p) with precision م 2. The agents were meta- veta-traiine on bandits where the payoffs (i.e. arm I's payoff and arm 2's mean) were drawn from a standard Gaussian distribution. The plots show the marginal probability of choosing the certain arm (blue) over the risky arm (red) after twenty interactions for every payoff combination. Each point in the uniform grid was estimated from 30 seeds. Note the deviations from the true risk-neutral indifference curve (black diagonal).
217
taken and reward obtained in the previous step. This setup allows agents adapt their choices to past interactions throughout an episode. To train all the agents in this experiment we used 88 CPU core hours 2.4 GHz and 10 GPU hours.
218
219
Figure 4 shows the agents' choice profile in the last (20th) time step. A true risk-neutral agent does not distinguish between certain and risky option that have the same expected payoff (black diagonal). The main finding is that the indifference region (i.e. close to a 50% choice in white color) evolves significantly with increasing 3, implying that the agents with different risk attitudes are indeed willing to pay different risk premia (measured as the vertical distance of the indifference region from the diagonal). We observe two effects. The most salient effect is that the indifference region mostly moves from being beneath (risk-averse) to above (risk-seeking) the true risk-neutral indifference curve as B increases. The second effect is that risk-averse policies (3 -1 and -0.5) contain
220
222
224
225
226
227
228
large region of a stochastic choice profile that appears to depend only on the risky arm's 's parameter.
229
We do not have clear explanation for this effect. Our hypothesis is that risk-averse policies assume
230
adversarial environments, which require playing mixed strategies with precise probabilities. Finally,
231
the risk-neutral agent (B- = 0) appears to be slightly risk-averse. We helieve that this effect arises due
232
to the noisy exploration policy employed during training.
5
Discussion
234
Summary of contributions. In this work we have introduced a learning rule for the online estima- tion of the Gaussian free energy with unknown mean and peecision/variance The learning rule (2) is obtained by reinterpreting the stimulus preseecce indicator component of the Rescorla- Bescorla-Wagner rule (soft) indicator function for the event of either over- or underestimating the target value. In Lemma]] [ we have shown that the free energy is the unique and stable fixed point of the expected learning dynamics. This is the main contribution.
236
(Rescorla 1972)
238
240
Furthermore, we have shown how to use the learning rule for risk- sensitive RL. Since the free energy implements certainty-equivalents that range from risk-averse to risk-seeking, we were able to formulate a risk-sensitive, model-free update in the spirit of TD(0) (Sutton and Barto 1990). thereby addressing a longstanding problem (Mihatsch and Neuneier 2002) for the special case of the Gaussian distribution. Due to its simplicity, the rule is casy to incorporate into existing deep RL algorithms, for instance by modifying the error using stop-gradion as shown in (23) In Section we also claborated on the role of the free energy within decision maaiin, pointing out its robustness properties and adversarial interpretation.
242
244
246
248
We also demonstrated the learning rule in experiments. Firstly, we empirically confirmed that the online estimates stabilize around the correct Gaussian free energies (Section 4 Estimation). Secondly, we showed how incorporating risk-attitudes into deep RL can lead to agents implementing
249
250
qualitatively different policies which intuitively make sense (Section (eection[-RI RL.. Lastly, we inspected the
252
premia risk-sensitive agents are willing to pay for choosing risky over a certain option, finding that agents have choice patterns that are more complex than we had anticipated (Section 4 Bandits).
8

---

254
Limitations. As shown empirically in Section 4 Estimation, an important limitation of the learning rule is that its fixed point only equal the free energy when the samples are Gaussian (or approximately Gaussian, as justified by the CI.T). Nevertheless, agents using the risk-sensitive TD(0) update (TT still display risk attitudes monotonic in 8, with / = 0 reducing to the familiar risk-neutral case.
255
257
259
While Lemma establishes the stable equilibrium of the expected update, it only guarantees conver- gence in continuous-time updates. To show convergence using discrete-time point samples, a stronger result required. In partieular, we conjecture that
280
261
[(w) = 2 [x μa, p)or(( - u)(x - u) dx - 2
(26)
If 26 is true, meaning that ((v) is 2-Lipschitz, then this could be combined with result in stochastic a approximation theory akin to Theorem in Jaakkola et al. (1944) to prove convergence.
263
264
A shortcoming of our experiments using R2D2 agents is that they deterministically pick actions that maximize the Q-value. However, risk-averse agents see their environments as being adversarial, and these i turn require stochastic policies in order to achieve optimal performance.
265
286
Conclusions.
Because it is impossible to anticipate the many ways in which canaaiically-canggii environment will violate prior assumptions, requiring the robustness of ML algorithms is of vital importance for their deployment in real-world applications. Unforeseen events can render their decisions unreliable-and in some cases even unsafe.
288
289
270
Our work makes a small but nonetheless significant contribution to risk-sensitivity in ML. In essence, it suggests a minor modification to existing algorithms, biasing valuation estimates in a sensiivee manner. In particular, we expect the risk-sensitive TD(OO-learniig rule to become an integral part of future deep RI. algorithms.
272
274
References
275
278
Amodei, D., Olah, C., Steinhardt. J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv preprint
277
278
Bellman, R. (1957). Dynamic Programming. Princeton University Press.
279
Bertsekas, D. P. and Tsitsiklis,
(1995). Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume I, pages 560-564. IEEE.
281
282
Cassel, A., Mannor, S., and Zeevi, A. (2018). A general approach to multi-armn bandits under risk criteria. In Conference On Learning Theory, pages 1295-1306. PMLR.
284
Coraluppi, S. P. (1997). Optimal control of Markov decision processss for performance and robustness. University of Maryland, College Park.
285
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. (2001). Incorporating second-order
287
functional knowledge for better option pricing. In Advances in Neural Information Processing Systems, volume 13. MIT Press.
Eysenbach, B. and Levine, S. (2019). If MaxEnt RL is the answer, what is the question?
290
Galichet, N., Sebag, M., and Teytaud, O. (2013). Exploration vs exploitation vs safety: Risk-aware aumti- bandits. In Asian Conference on Machine Learning, pages 245-260. PMLR.
291
282
García, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480.
293
294 Gershman, S. J. (2015). Do learning rates adapt to the distribution of rewards? Psychonomic Bulletin & Review, 1325
Mryu-Moya, J., Leibfried, F., Genewein, T., and Braun, D. A. (2016). Planning with information- processing constraints and model uncertainty in Markov decision processes. In Joint European Con- ference Machine Learning and Knowledge Discovery in Databases, pages 475-491. Springer.
297

---

299
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9()):1335
300
1780.
301
Howard, R. A. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Management
302
science, 18(7):35--36.
303
Husain, H., Ciosek, K., and Tomioka, R. (2021), Regularized policies are reward robust. In
304
International Conference Artificial Intelligence and Stafistics, pages 64-72. PMLR.
305
Jaakkola, T., Jordan, M. I., and Singh, S. P. (1994). On the convergence of stochastic iterative
dynamic programming algorithms. Neural computation, (00011885-10111
307
Kappen, H. J. (2005). Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(11):P11011
309
Kappen, H. J., Gómez, V., and Opper, M. (2012). Optimal control as graphical model inference
problem. Machine learning, 1592)1159-182.
Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. (2018). Recurrent experience
311
312
replay in distributed reinforcement learning. In International conference on learning representa- tions.
313
314
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv e-prints, pages arXiv-1412.
315
316
Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. (2017). AI safety gridworlds. arXiv preprint arXiv:1711.0988:
317
318
Markowitz, H. (1952). Portfolio selection. Journal of Finance, 1(7).
Mihatsch, O. and Neuneier, R. (2002). Risk-sensitive reinforcement learning. Machine learning. 2962)267.-
320
Mikulik, V., Delétang, G., McGrath, T., Genewein, T., Martic, M., Legg, S., and Ortega, P. A. (2020).
322
Meta-trained agents implement bayes-optimal agents. arXiv preprint arXiv:2010. 11223.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et (2015). Human-level control through deep reinforcement learning. nature, 518(7540):529-533
324
325
326
Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798.
328
Niv, Y., Edlund, A., Dayan, P., and O' O'Doherty, P. (2012). Neural prediction errors reveal sisk-sensitive reinforcemenn-llerrnng process in the human brain. Journal of Neuroscience, 3252)551
329
330
331
Ortega, D. A. and Braun, A. (2011). Information, utility and bounded rationality. In International
332
Conference on Artificial General Intelligence, pages 269-274. Springer.
333
Ortega, A. and Braun, D. A. (2013). Thermodynamics as a theory of decision-making with cnformatiinn-rocessiing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 269(2133):201 20683.
335
Ortega, P. A. and Lee, D. (2014). An adversarial interpretation of iitommmtion-thhorede bounded rationality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28.
337
Ortega, P. A., Wang. J. X., Rowland, M., Genewein, T., Kurth-Nelson Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., et al. (2019). Meta-learning sequential strategies. arXiv preprint arXhv:1905.03030
339
341 Rescorla, A. (1972). theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. Current research and theory, pages 64-99.
10

---

343
Robbins, H. and Monro, S. (1951). A stochastic approximation method. The annals of mathematical statistics, pages 400-407.
344
345
Russell, S., Dewey, D., and Tegmark, M. (2015). Research priorities for robust and beneficial artificial intelligence. Ai Magazine, 36(4):105-114.
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016). Meta-learning with
348
memory-augmented neural networks. In International conference on machine learning, pages 1842 1850. PMLR.
350
Sutton, and Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In
351
Learning and Computational Neuroscience: Foundations of Adaptive Networks, pages 497-537.
352
MIT Press.
353
Sutton, R. and Barto, G. (2018). Reinforcement learning: introduction. MIT Press.
354
Tamar, A., Mannor, S., and Xu, H. (2014). Scaling up robust MDPs using function approximation. In
International Conference Machine Learning, pages 181-189. PMLR.
Theodorou, E., Buchli, J., and Schaal, S. (2010). A generalized path integral control approach to
357
reinforcement learning. The Journal of Machine Learning Research, 11:3137-3181
358
Tishby, N. and Polani, D. (2.11). Information theory of decisions and actions. In Perception-action cycle, pages 601-636 Springer.
359
380
Todorov, E. (2007). Linearly-solvable Markov decision problems. In Advances neural information processing systems, pages 1369-1376.
Toussaint, M. (2009). Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pages 1049-1056.
363
364
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2016). Learning to reinforcement learn. arXiv preprint arXiv: 1610.666.
386
387
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. (2008). Maximum entropy inverse
388
reinforcement learning. In AAAI.

---

Checklist
389
370
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
372
contributions and scope? [Yes]
378
(b) Did you descrihe the limitations of your work? [Yes] see Section
374
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
Section[5]
375
376
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results... 378
379
(a) Did you state the full set of assumptions of all theoretical results? [Yes] in Section 2
380
(b) Did you include complete proofs of all theoretical results? [Yes] in Section
381
3. If you ran experiments..
382
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is proprietary.
384
385
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Section 4
386
387
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] see Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Section 4
390
381
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
392
(a) your work uses existing assets, did you eite the creators? [N/A]
393
(b) Did you mention the license of the assets? [N/A]
394
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
396
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
397
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
399
400
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
402
403
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
404
405
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "fnxLs0ZoXJy", "reviewer": "Reviewer_AeYb", "summary": "This paper proposed an algorithm to estimate free energy. It provided a lemma to justify its update rule and conducted experiments to show the validity of its algorithm.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. The algorithm it proposed is an improvement of former algorithm. The intuition behind this improvement require further explanation. The lemma it proved is a basic property. \n\nQuality. This work is complete and its result is supported. However, there might be some little mistake in it. For example, equation (5) should be double checked.\n\nClarity. This paper is written clearly.\n\nSignificance. I am not persuaded that it is significant by this paper. First, the author proposed using it for risk-sensitive RL. However, there are already several risk-sensitive RL algorithms. Why should we use free energy instead of other algorithm? Why is it superior? Secondly, the lemma here required Gaussian assumption and its only conclusion is that free energy is the fixed point. It did not include the rate of convergence. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The value of this algorithm (Why should we use it when we have other mature risk-sensitive RL algorithms? ) should be emphasized more.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "e2bLfvK56aI", "reviewer": "Reviewer_q5jG", "summary": "This paper is about computing the log of the expectation an exp-transformed random variable (free energy) by stochastic approximation, under the assumption that this random variable is drawn from a Gaussian distribution. While not stated explicitely in this way, it seems that the underlying motivation may come from computation neuroscience: that of obtaining an explainable learning rule, in line the Rescorla-Wagner rule. I'm saying this because under the Gaussian assumption, the free energy admits a closed-form solution where the unknows are the mean and standard deviation which we don't necessarily need to estimate by stochastic approximation unless with have specic reasons to do so. The asymptotics behavior of the learning rule is studied in lemma 2 via an analysis akin to the \"ODE method\" in stochastic approximation: by considering the continuous time limit of the expected deterministic counterpart. Rather than talking about stability in terms of the eigenvalues of the Jacobian, the proof considers the possible values taken by the deterministic learning rule and shows that it has the free energy as a root. An application in reinforcement learning is presented where the authors match the form of their stochastic approximation method to that of temporal difference learning for policy evaluation and later present results in the control setting with deep reinforcement learning. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is impeccably writen: the flow, clear notation and figures (latex-styled plots, pdf output), historical context and cited papers older than a year ago. This really sets the standard for what our community should be striving to achieve. It is really refreshing to read a paper structured like this one! It starts right from the begining with a clear and direct description of what has been found, in math. The high level motivation then follows. It's not wasting anyone's time trying to sell their work. I wouldn't be surprise to learn that the first author is a physicist who moved to computational neuroscience. \n\n# Issues\n\n- The experiments sections departs quite far from the original setup where the assumptions are clear and the analysis is tractable. It feels like those experiments (the deep rl ones) were designed to make the paper feel more \"neurips-y\" and appeal to a more ML-focused audience. I would have preferred to see simpler but more carefully designed experiments. More on that in the comments below\n\n- Regarding the application to TD, of course many questions are left unanswered. Note that you start with the policy evaluation setup and then go to the control setting in the experiments. Let's just start with policy evaluation: in the tabular setting, is the corresponding method stable? What could you say via the ODE method? Can you show that the corresponding deterministic operator is a contraction? If not, can you establish stability in a different way? Now one step further: in the policy evaluation setting, what happens if now throw in linear functiona approximation? Do you obtain a system of \"projected Bellman equations\"? And finally the control setting: take Q learning in the tabular setting, add your update rule, does it converge? What do you get when you apply the ODE method once more (see Kushner and Yin for Q learning)?\n\n(Reflecting on the above, perhaps nothing could be said due to the iid assumption.)\n\n# Notes\n\nLine 37: \"is defined\" it'd be nice to have a citation\n\nEquation 7: Time indices would be appropriate here and throughout the entire paper: $v_{t+1} = v_t + \\alpha_t \\cdot (x_t - v_t)$.\n\nLine 153: It doesn't look like the analytical error function (21) would lend itself easily to automatic differentiation due to the presence of the dilogarithm term. Tensorflow seems to offer tf.math.special.spence, but what is it really going on under the hood? What is the corresponding vector-Jacobian rule? \n\nLine 183: \"trained R2D2\" The experimental setup and the choice of figures is appropriate. However, using R2D2 seems to be overkill for the problem being solved here. I would have preferred to see a minimalistic vanilla Q-learning + SA Free energy demonstration than a deep RL setup for a grid world problem. \n\nLine 208: \"memory-based meta-learning\" again, the experimental setup seems overkill and clashes with the more methodical approach taken earlier in the paper. \n\nLine 239: \"This is the main contribution.\" I love this", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate how figure 2 also shows how the algorithm performs when the Gaussian assumption is not satisfied. \n\nSee above comments on issues regarding the TD case", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LIpITlxYAN", "reviewer": "Reviewer_sDnj", "summary": "The authors provide a novel approximation rule for the risk-sensitive criterion in RL. In particular they present an iterative update rule for the estimation of the free energy, which has direct application to risk-sensitive RL. They point out relevant properties of the approach and their relationship to game-theoretic and finance and evaluate these on a set of benchmark problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written very well.  While heavy on theory it is explained in an understandable way and the resulting methodology is simple. \n\nThe experiments make sense given the research question. I particular like that the authors start with simple toy problems (including verifying the estimator), before using to more complex domains such as 2-armed bandits\n\n\nCons:\nAs the authors point out the main limitation is the Gaussian assumption for the Returns.  In this case the paper would have been stronger had the authors constructed an (toy) experiment where the Gaussian assumption does not hold, ideally with long-tail reward distributions, as these are often seen in practice. \n\nDisclaimer: I did not check the math in Section 2, so for the review I will assume it to be correct.\n\nA question: The free energy as certainty-equivalent applied to RL always appeared a bit odd to me: If we have the expected return (units return) we are adding the variance (units squared return) to it, weighted by  \\beta.  This does not seem right. Wouldn't it make more sense to use \\mu + \\beta \\sigma here (as the standard deviation and the mean share the same unit)?  Could from your update rule also an analogous rule for this be derived?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KGYo9KZ7AGY", "reviewer": "Reviewer_L68y", "summary": "The authors introduced a stochastic approximation rule for estimating the free energy from\ni.i.d. samples generated by a Gaussian distribution of unknown mean and variance. They apply their result then to optimize the entropic risk measure in an RL task. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality:\nAt the best of my knowledge, the proposed update rule is indeed novel. The authors correctly give an overview of the state of the art related to the free energy, even if a related work section is missing.\n\n## Quality: \nThe main result of the authors is Lemma 1, which seems to be correct. However it is not possible to apply it to  MDPs with a transition kernel which does not follow a Gaussian distribution. This limitation is recognized by the authors in a specific section, nevertheless, they show its use as a heuristic for TD learning.\nThey set a RL experiment on a gridworld task, in which the desired behaviour is obtained.\nHowever, in order to prove that the heuristic is valid, it would have been nice to see it applied in a set of environments, with different difficulty levels. It would be interesting to have more general results about the update rule, for general distributions, or, at least, for some other families. \n\n## Clarity: \nThe paper is clearly written, however, the structure is unusual: there is not a clear separation between introduction, related works, problem formulation and proposed methods. Organizing better the contents would help the reader figuring out the real contribution of the paper.\n\n## Significance: \nThe result is interesting indeed, however, it is difficult to see if it will have an impact alone. There are indeed two problems that limits the application of the rule to RL:\n- the Gaussian distribution assumption;\n- the continuous time assumption.\nThe latter is necessary to prove the convergence to the fix point, however, RL is usually focused on discrete time settings.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative impact is present.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces some technical ideas that may interest a significant segment of the community.  The theoretical work is carried out under quite restrictive assumptions.  But the ideas may be of broader interest.
Decision: Reject