Risk-Aware Transfer in Reinforcement Learning using Successor Features
Authors: Michael Gimelfarb
Andre Barreto
Scott Sanner
Chi-Guhn Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
transfer learning
representation learning
successor features
risk-sensitivity
risk-awareness
safety

Abstract: Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning, while the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of transferring policies between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending the dynamic programming's policy improvement operation to sets of policies \emph{and} levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, while incorporating risk into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account. 

Risk-Aware Transfer in Reinforcement Learning using Successor Features
Michael Gimelfarb"
André Barreto DeepMind andrebarreto@google. com
University of Toronto
mike. walke.gimllffreee gww.geamrmmaaalll.mmmmmm utoronto.ca
Scott Sanner"
Chi-Guhn Lee University of Toronto cgleetmie. utoronto.
University of Toronto
ssannerßmie, utoronto. ca
Abstract
Sample efficiency and risk-awareness central the development of practical reinforcement learning (RL) for complex decision-making The former can be addressed by transfer learning and the latter by optimizing the return. However, the problem of transferring skills in risk-aware manner is not well-understood. In this paper, we address the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk measured by the variance reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending policy improvement via dynamic programming sets of policies and levels of risk-aversion. Next, we extend the idea of successor features (SF), value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, and incorporate risk-awareness into the decision-making. Experiments on a discrete navigation domain and control simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk the learned policies into account.
utility function of
1 Introduction
Reinforcement learning (RL. is a general framework for solving sequential decision-making problems, in which an agent interacts with an environment and receives continuous feedback in the form of rewards. However, many classical algorithms in RL do not explicitly address the need for safety, making them unreliable and difficult to deploy in some real-world applications [10]. One reason for this is the relative sample inefficiency of model-fres RL algorithms, which often require millions of costly or dangerous interactions with the environment or fail to converge altogether [4
46). Transfer
learning addresses these problems by incorporating prior knowledge or skills 23 Despite this, using the expected return as a measure of optimality could still lead to undesirable behavior such as excessive risk-taking, since pwoobabbbility catastrophic outcomes with negative reward and high variance could be underrepresented (29). For this reason, risk-awareness is becoming an important aspect in the design of practical RL [14]. Thus, an ultimate goal of developing reliable systems should be to ensure that they are both sample efficient and risk-aware.
Affiliate Vector Institute, Toronto, Canada.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

skills from Transfers Skills
Exploits Task Structure
Risk- Sensitive
RL
30
36
39
x
X
Transfer
15
19
25
26
38
/
X
/
Successor Features
2
g
/
V
X
RaSF (Ours)
V
Table I: Comparison of RaSF with relevant work in transfer learning and risk-aware RL.
We take a step in this direction by studying the problem of aisk-aware policy transfer between tasks with different goals. A powerful way to tackle this problem in the risk-neutral setting is the GPI/GPE framework, of which successor features (SF) are a notable example (2). Here, generalized policy improvement (GPI) provides a theoretical framework for transferring policies with monotone improvement guarantees, while generalized policy evaluation (GPE) facilitates the efficient evaluation of policies on novel tasks and is a key component in satisfying the assumptions of GPI in practice. Together, GPI/GPE provide strong transfer benefits in novel task instances even before any direct interaction with them has taken place, a phenomenon we call lask generalization. The key to the superb generalization of GPI/GPE lies in their ability to directly exploit the structure of the task space, taking advantage of subtle differences and commonalities between task goals to transfer skills seamlessly in composable manner. This property could be an effective way of tackling problems in offline RL [24), such as the transfer of skills learned in a simulator to a real-world environment. However in many cases, such as helicopter flight control [16], making one wrong decision could lead to catastrophic outcomes. Hence, being risk- could offer one way to avoid worst-case outcomes when transferring skills in real-world settings.
Contributions. We contribute novel successor feature framework for transferring policies with the goal of maximizing the entropic utility of return in MDPs (Section 2.2 Intuitively, the entropic utility encourages agents to follow policies with predictable and controllable returns characterized by low variance, thus providing a natural way to incorporate risk awareness. Furthermore, while our theoretical framework could be extended to other classes of utility functions, the entropic utility has many favorable mathematical properties optimal transfer (Lemma] on the mean-variance approximation, in which the sufficient statisties of the return distribution can be computed directly (Section (.3) or by leveraging recent developments in distributional RI. [6]. Our resulting approach, which we call Risk-Aware Successor Features (RaSF), able to exploit the task structure to achieve task generalization with respect to novel goal instances well as levels of risk aversion, where emphasis is placed on avoiding high volatility of returns. Our approach is also complementary to other advances in successor features, including feature learning [ universal approximation [8] exploration (21). and non- non-stationary reward preferences [4]. Empirical evaluations on discrete navigation and continuous robot control domains (Section 4 demonstrate the ability of RaSFs to better manage the trade-off between return and risk and avoid catastrophic outcomes, while providing excellent generalization on novel tasks in the same domain.
B 22] that we exploit directly this work achieve We also derive a form of risk-aware GPE based
D Theorem | and
Related Work. The entropic and mean-variance objectives are popular ways of incorporating 36
However, transferring learned skills between tasks while taking risk into account is a difficult problem. One way to implement risk-aware transfer learn critic [38] or teacher [43 that can guide an agent toward safer behaviors on future tasks. The risk-aware transfer of a policy from a simulator to a real-world setting has also been studied in the area of robotics [17]. Another approach for reusing policies the probabilistic policy reuse of Gareía
risk-awareness in RL 7 20 27
28
30
39 45
and Fernández [1]] but requires strong assumptions on the task space. Hierarchical RL (HRL) is another related approach that relies on hierarchical abstractions, enabling an agent to decompose tasks into hierarchy of sub-tasks, and facilitating the transfer of semporally-extended sub-tasks to the parent task. The CISR approach of Mankowitz et al.
26 is the first to investigate safety explicitly within HRL, followed up by work on safe options [8]
19 25 However, none of the
existing work takes advantage of the compositional structure of task rewards to transfer skills while optimizing the variance-adjusted return, which is the problem we tackle in this paper (see Table I

---

2 Preliminaries
2.1 Markov Decision Process
Sequential decision-making in this paper follows the Markov decision process (MDP), defined as four-tuple (S,A, P): 8 is set of states; A finite of actions; r S A x S + R is a bounded reward function, where r(s, a, s') is the immediate reward received upon transitioning A S - [0, so) is the transition function
to state s' after taking action a in state s; and for state dynamics, where P(s'|s, a) is the probability of transitioning to state s' immediately after taking action a in state
In the episodic MDP setting, decisions are made over a horizon T= (0,1,...77 where T E N. We define stochastic Markov policy mapping и S x T ((A), where (AA) denotes the set of all probability distributions over A. Similarly, deterministic Markov policy is a mapping S x T - A. In the risk-neutral setting, the goal is find a policy π that maximizes the expected sum of future rewards after initially taking action a in state
T
(7, a) = Esc IPP seae))
Σ L=b
r(s,, au St+1)
In this case, it is possible to show that a deterministic Markov policy x² is optimal 133 The theoretical framework in this paper also allows for time-dependent reward or transition functions.
2.2 Entropic Utility Maximization
We incorporate risk-awareness into the decision-making by maximizing the entropic utility U3 of the cumulative reward, defined for fixed / E IR
1 log E
Up [
OR
(1)
B
for real-valued random variables R on bounded support Ω R. An important property of the entropic utility is the Taylor expansion Ug [R] + _ Var[H]] + O(32). Interpreting the risk as return variance, 33 can now be interpreted as the risk aversion of the agent: choosing 3 < 0 (B > 0) is risk-neutral, e.g. Uo[R] [[
leads to risk-averse (risk-seeking) behavior, while
Specializing ( to the MDP setting, the goal is to maximize
T
(x a) = Ug
Σ 1=h
r(s, (1)8)) stt1)
(2)
over all policies starting from 8;, = and a₁ = a. As in the risk-neutral setting, it is possible to show
that a deterministic Markov policy is optimal [5]. Furthermore, 27, be computed iteratively
through time using the Bellman equation 9 32
2h a) = Us (r(s,a,')) + Qh., ms41()'))]
= log E EemP(Man) [exp {8 (r(s, s') + T...o(s' (
3
starting with 27,1, a) = In fact, is the only utility function that has this equivalence, and other key properties (Lemma D. while also satisfying time consistency that ensures the learned risk-aware behaviors remain consistent across time (22]. In this paper, we use (3) to establish a general GPI framework for risk-aware transfer learning with provable guarantees, and leverage approximations of to learn portable policy representations.
In reinforcement learning, the Bellman equation of dimensionality when S is high-dimensional or continuous, and since neither the dynamics nor the reward function often known. Instead, the agent interacts with the environment using stochastic exploration policy x², collects trajectories ((Se, Ot, St+1, (t+1))T= and updates ( a:) via sample approximations Ü, R Uş (36). Our goal is to ameliorate the relative sample-inefficinccy of RL through transfer learning that aim to generalize to the risk-aware setting.
applied directly since it suffers from the curse

---

2.3 Transfer Learning
We now formalize the general transfer learning problem. Let M be the set of all MDPs with shared transition function P but different (bounded) reward functions. A fixed set of source tasks M E M is instantiated, and their corresponding optimal policies T1.... Ta are estimated. Our main goal is transfer these resulting source policies to target task Mn+1 E M, obtain a policy 7x+1 whose utility is better than one learned from scratch using only fixed number of samples from Mn+1- We refer to this outcome positive transfer.
As discussed earlier, a standard way to implement transfer learning is the GPI/GPE framework of Barreto et al. (2). The core mechanism that enables positive transfer in the risk-neutral setting is called generalized policy improvement (GPI). Specifically, the set of source policies T Ta are evaluated on the target task M,+1 to obtain corresponding values Qn+1..... Q 1. Given a mechanism that can perform this policy evaluation step efficiently with some small error ε namely successor features discussed and extended in Section 3.3 agent then selects actions in greedy manner by following policy (s) E arg max, maxj=1 11... Q (s, a) in state s. The policy π corresponds to a strict policy improvement operator, and thus fulfills requirements for positive transfer.
3 Risk-Aware Transfer Learning
An obvious challenge of applying GPI in the risk-aware setting is that transferring optimal risk- neutral source policies does not guarantee risk-aware optimality in the target task. A much stronger observation is that, even if the source policies Ty are risk-aware, performing the policy evaluation step risk-neutral way can still break the risk-awareness of GPI. This makes the extension of GPI to the risk-aware setting a non-trivial problem.
3.1 A Motivating Example
To see this, consider the MDP shown in Figure 1 which involves navigating from an initial state 'S' to a goal state GG" in grid with stochastic transitions. Traps of two types (X, Y) are placed fixed cells, which upon entry terminate the episode with fixed costs, summarized compactly as pairs (c1, c2). We define two source tasks with costs (20, 20) and (0, 0) and target task with costs (20, 0). The optimal policies for 3 = -0.1 induce three different trajectories: safe (blue) and hazardous routes (red) for the source tasks,
risk-oware GPI nektreltal GPI
density
and a relatively safe route (green) for the target task. We also note that risk-awareness is prerequisite for this problem, since a risk-neutral agent prefers the hazardous path when optimizing any of the tasks.
0.0
-30
10
episode retur
We compute the GPI policies π with risk-averse = -0.1 and resk-neutral (3 0) policy evaluation, the middle and rightmost plots in the top row in Figure [ The bottom plot shows the distribution of returns collected over 5,000 test runs by acting according to the two GPI policies. Risk-averse policy evaluation results in optimal risk-averse GPI policy corresponding to the green trajectory, whereas risk-neutral policy evaluation does not, even though both source policies are optimal on their corresponding tasks. Interestingly, the risk-averse GPI policy a non-trivial stitching of the two source policies.
Figure/ Comparing risk-aware and nest-nl GPI. The risk-aware (for = 0.1) GPI policy is shown in the top-middle plot, while the risk- (for 3 = 0) GPI policy is shown in the top-right plot.
shown in
3.2 Risk-Aware Generalized Policy Improvement
Motivated by this example, we conjecture that the risk-awareness the GPI policy я is primarily dependent the way in which the source policies are evaluated in target task instances. In this section, we describe theoretical results that generalize the concept of generalized policy iteration to the problem of maximizing the entropic utility of returns.

---

We begin by summarizing key properties necessary for establishing convergence of awske GPI in the following lemma.
Lemma 1. Let B E R and X, Y arbitrary random variables on Q. Then:
(monotonicity) Pf Y) then [g[X] 2 U
(2) (cash invariance) Ug[XX + c] [Us[X] + e for every c E R
(3) (convexity) B < 0 0) then Ug is a concave convex)functtinn
(4) (non-expansion) for f.g - Ω, it follows that
[U([((X)] Usla(X)]1
sup Exl P
where Px (S)) is the set of all probability distributions on Q that are absolutely continuous w.r.l. the true distribution of X.
A proof is provided in Appendix B.2. Properties (1)-(3) characterize concave utilities [13 which intuitively can be seen as minimal requirements for rational recision-maaingg (1) a lottery that pays off more than another in every possible state of the world should always be preferred; (2) adding cash position should not increase risk; and (3) the overall utility can be improved by diversifying across different risks. Property (4) is a derivative of the first three, and thus the theoretical results in this work could be extended to the broader class of iterated concave utilities 34
We can now state the first main result of the paper.
Theorem (GPI for Entropic Utility). Let Ti, be arbitrary deterministic Markov policies with approximate entropic utilities hh 2% when evaluated in an arbitrary task M, with error 1 (8,
2hip (s, a)l S for all & E S, a E A. 1.... and E Define
(h(s) E arg max max 2h Vs E aEA =L...
(4)
Then,
(T, 2 max (8, - 2(T - 1t + 1)e, SS
Thus, evaluating the risk of source policies using U3 provides monotone improvement guarantees for GPI, and thus satisfies our definition of positive transfer. Another significant property of the bound in Theorem [ and the one in Theorem below, is the linear separation between the optimal utility and the approximation error E. Knowing how the optimality of π explicitly depends on and how errors are propagated throughout the transfer learning process, is critical for developing reliable RL with predictable behavior, and highlights key advantage of making GPI risk-aware. The additive relationship between the optimality and the approximation error in Theorem [ SFs are robust to approximation errors. This becomes particularly advantageous in our setting, since estimating utilities Ug accurately with GPE is more complicated than the risk neutrallrall setting.
further explains why
A stronger result for GPI can be derived when the source policies Tı, T2 ... Tn are s-optimal, and policy evaluation is once again performed using Ug. In this case, the optimality of GPI is determined by the similarity 6, between the source and target task instances.
Theorem 2. Let 2h 3 be the utilities of optimal Markov policies nद from task M, but evaluated in task
M with reward function r(8, a, s'). Furthermore, let 2x be such that 2 (s, a) 27, 2h.gk (s, a) S
for all E S, a E A. h E T and -1.. and let be the corresponding policy in 44. Finally, let 8, min sup. s...s' |r(s, 0, 8)) -r, a,,',,)) s')}. Then,
O (s, - (s a) < 2(T h + 1)(5, +8), S
These results are proved in Appendix B.2 for the episodic setting and in Appendix B.3 for the discounted setting. Also, please note that these bounds are tight (see, e.g. Nemecek and Parr (3] for the restaneutral setting). Finally, while not required in this work, the above results could be extended to more general settings in risk-averse control 34 though practical implementation of GPE in these settings remains open problem.
5

---

3.3 Risk-Aware Generalized Policy Evaluation
Following Barreto al. (2] let Ø S x S' + Rd be a bounded and task insk-independent feature map, and consider the following linear representation of rewards.
r(s, a, s') = ( a, a',")) w, Vs,
where Fd is task-dependent vector of reward parameters. The risk-neutral return becomes:
(%(s,a) = Ep
Σ Φst, Ce, S(11) 8h =s, on a, a m((s))
T
Ep
Σ
a(s a1 8(+1) 8/1 8, ag ar me(ss) w= (8,
(5)
L/mh
where ubg, (s, a) are the successor features (SFs) associated with the policy æ. The linear dependence of the return on w allows for instantaneous policy evaluation in novel tasks with arbitrary reward preferences w, making a particular and perhaps the canonical instantiation of GPE. More critically, T can be seen as desk-independent and highly portable linear feature representation of policies, and is the key the generalization ability of SF' on novel task instances.
The concept of GPE can be generalized to incorporate entire distributions of the return. Repeating the above derivation for the entropic utility (2), we have:
() = Us
Σ rss, Te(s)), s(-) U3 as)
corresponding to the random vector (s, = Ei' Φ of unrealized feature returns at time Thus, we have transformed the problem of estimating the utility of returns into the problem of estimating the distribution of V (s, a) w. The key question now how to estimate this distribution for fast GPE.
natural way do this is by applying second-order Taylor expansion for Ug. since allows us to precompute and cache the necessary moments of the return distribution:
Us (VT(s, = Ep(V Tw]
w] +
+0(3))
2
= (s, a) w + B w Varp (s, (s,))
2
covariance matrix for SFs. In the context of Theorems [ and 2 the term E encapsulates the errors in the approximation of u/; and E,, plus the terms contained in ( above. However, the main advantage of ? is that, like (5) is also analytic in w and allows for instantaneous policy evaluation with arbitrary reward preferences w. Interestingly, (7) also allows instantaneous policy evaluation with respect to different choices of B making it possible to revise or adapt the level of risk aversion on on-demand. In all these cases, 1bz and Σj provide task -ndeeeddeenndntt and portable representations of policies while also accounting for exogenous risk. This is the key to preserving the task generalization ability of SFs in the risk-aware setting, and (7) can now be seen as particular instantiation of GPE. We call this overall approach Risk-aware Successor Features (RaSF).
in which Varp ( a)] (s, a) is interpreted
The simplest approaches for estimating 27 in the exact (c.g. tabular) Q-learning setting are based on dynamic programming [37 [40], which would allow the overall approach to be easily integrated into existing SF implementations, In particular, the covariance satisfies the Bellman equation
Mg (s, a) Ep- [5, (8 Th+ (8')) 84. 8, an a)
where on are the Bellman residuals of + (s, a). The approximation के is known to converge the true value ub and a similar result also holds for updating the covariance based on (8)
Theorem (Convergence of Covariance). Let || be a matrix- matrix-compatible norm, and suppose there exists S x A T [0, oo) such that:

---

1.
11,8 - (s, & (n(s)a)
2. [B
a.))) (0))5n 1s)) - -b (6', (+')) s en(s,a).
Then,
- -E 5no% + ET 1s' Th+
Th (+7) - (sn(s,a).
A proof can be found in Appendix B.4. Appendix A.1 describes how 25g and Eg can be learned online from environment interactions, while Appendix A.4 discusses further generalizations of (7). There are, however, several limitations of estimating 27, in this way. First, obtaining accurate estimates of 2% requires accurate estimates of 4b% (thus estimating one quantity on top of another), making this approach difficult apply with deep function approximation. This claim further substantiated by Theorem 3 and preliminary experiments. A second issue that occurs is double sampling, when the same transitions are used to update the mean and covariance, resulting in accumulation of bias in the latter [ 47.. Our experiments on the reacher domain mitigate these issues by leveraging 7 while maintaining computational efficiency.
distributional RL approximate
4 Experiments
To evaluate the performance of RaSF, we revisit the benchmark domains in Barreto al. (2], which have been slightly modified for learning and evaluating risk-aware behaviors. We defer all experimental details to Appendix C.
4.1 Four-Room
Domain Description. The first domain consists of fam- ily of navigation tasks defined on discrete 2-D space di- vided into four rooms, as illustrated on the left in Figure 2 The environment has additional objects that can be picked up by the agent by occupying their cells. These objects belong to one of three possible classes, drawn as different shapes in Figure 2 which determine their reward. The po- sition of the objects remains fixed, but the rewards of their classes are reset every 20,000 transitions to random values sampled uniformly in [-1, +1]. To incorporate risk, traps are placed in fixed cells marked with X. For every time in- stant during which the agent occupies trap cell, the trap activates spontaneously with a small probability, resulting in an immediate penalty and termination of the episode (we refer to this event as a failure). The goal is to maximize the total reward accumulated over 128 random task instances while minimizing the number of failures.
Four-Room (Left): the shapes of the objects represent their classes, 'S' is the start state, 'G' is the final goal state, and 'X' is trap. Reacher (Right): colored and gray circles represent training and test tar- gets, respectively, while shaded re- gions represent areas of high risk.
Figure 2:
Baselines. order to demonstrate the power of our ap- proach in the absence of approximation errors, we define a simple instance of RaSFs in which 26th and " learned exactly using lookup tables and dynamic programming (equation (8)) We also apply modest discounting of rewards to ensure that the Q-function converges, as is standard in RL and discussed further in Appendix A.2. The vector w is also learned using immediate reward feedback and exact, sparse state features & provided to the agent. Due to its similarity to standard Q-learning, we call this approach RaSFQI. To provide a challenging baseline for comparison, we implemented another policy reuse algorithm (PRQL. [2] Further replacing the risk-neutral action selection mechanism of PRQL. with smart exploration
[6 allows PRQL to easily account for return
volatility, and we refer to this baseline as RaPRQL.
Main Results. The performance of these algorithms is shown in Figure 3 The cumulative reward obtained by RaSFQL is generally lower than SFQL,, as expected since a risk-averse agent should avoid the objects the bettom-left and top-right rooms and forgo their associated rewards. Interestingly, the performance of RaSFQL far exceeds that of RaPRQL and even PRQL, suggesting that the benefits
7

---

Average Retum
RaSPOL SFQL
RaPROL
500.
PROL
-20
230
24
25
26
28
30
122
123
124
125
126
127
128
500
500
24
25
26
30
122
123
124
125
126
127
128
Pailures per Toak
60
20
20
0-
26
28
30
122
123
124
125
126
127
Instance
Figure 3: Average returm, cumulative return and number of failures per task in the four-room domain,
for 3 = w -2. Shaded bands show one standard error over 30 independent runs.
1.00
120
RuSFCS1
0.75
Normaltee bed Retur
100
SFC51
C51
0.6
Failures
80
0.25
0.4-
0.00
40
Tak
RuSPCS1
RaCS1
20:
-0.50
-0.2
SPC51
Training
Training Instance
Training Instance
Figure 4: Performance training and tasks for the reacher domain, function of the number of experiences collected from the training tasks. Left: Normalized return on training tasks. Faded curves correspond C51 performance. Middle: Normalized return averaged across all tasks. Right: Total failures across all test tasks. Shaded bands show one standard error over 10 independent runs with different seeds.
of task generalization provided by GPI/GPE are quite strong. Furthermore, the number failures observed by RaSFQL gradually decreases over the task instances, while the number of failures of SFQL slightly increases. This is consistent with Theorem [ that guarantees monotone improvement in the risk-adjusted return of RaSFQL. On the other hand, while RaPRQL also learns avoid risk, fails slightly more often than RaSFQL. This suggests that the benefits of task generalization promised by GPI/GPE even allow risk-aware behaviors emerge sooner than by using generic policy reuse methods that are unable to exploit the task structure, namely PRQL. This aspect becomes critical for minimizing failures when deploying a trained policy library on novel task instances real-world setting. Further analysis and ablation studies are provided in Appendix D.1.
4.2 Reacher
Domain Description. The second domain consists that involve the maneuver of a robotic arm toward a fixed target location. As illustrated in the rightmost plot in Figure the agent only allowed train on 4 tasks, whose target locations are indicated by colored circles, and must be able to perform well on 8 test tasks whose target locations are indicated by the grey circles. Furthermore, we incorporate two sources reward volatility: (1) actions are perturbed by additive Gaussian noise; and (2) fixed regions around some of the target locations randomly incur negative rewards (failures), illustrated by faded circles Figure Please note that most of these high-volatilly regions centered on target locations of test tasks from which
of tasks based on the MuJoCo physics
engine 42

---

lod Seak
1.0-
120
0.8
100
SP632.01
(9:15)
S263161
0.5-
80
Fallures
0.4
0.2
.. [3164)
0.0
03177
-0.2
0
Trining Task Instance
Training Task Instance
Figure 6: Left: Normalized average test return for the reacher domain, for different values of
(legend values indicate the negative values of 3). Middle: Total failures across all test tasks for various values of (legend values indicate the negative values of (). Right: Evolutions of the arm tip position during three successful rollouts of the reacher domain according to the GPI policy obtained after training with 3 -3 (yellow pentagons indicate the initial states in each rollout). Only one training task and two test tasks are shown. The risk-averse agent learns to hover close to the goal while avoiding the high-volatility shaded regions.
the agent never learns directly. This stresses the agent's ability to avoid unforeseen dangers in the environment, in additional performing well previously unseen task instances.
Baselines. As discussed earlier, it is difficult to compute " directly using (8). A computationally tractable way avoid these issues to first approximate the density of ( a), and then extract the moments needed to com- pute (8). Specifically, we apply C51 6 by modeling TT a.... (s, (s, a) using histograms for each (s, a). However, T are high-dimensional, so we avoid the curse of dimensionality by modeling the marginals " rather than their full joint distribution. This still turns out to be an effective way of detecting high-variance scenarios in the environment. The final architecture is illustrated in Figure where the marginal distributions of TiTh are modeled as separate output "heads" with shared state encoder. The rest of the training protocol is identical to the SFDQN Barreto et al. (2), except that DQN is replaced by the C51 architecture above, further detailed in Appendix A.3. The risk-averse and risk-neutral instances this approach for modeling successor features are referred to as R and SFC51, respectively, while RaC51 and C51 replace successor features with universal value functions 335 for generalization across target locations.
41 02
(A
(T.
return
(s.))
Figure 5: Architecture for y ( a).
The performance of the algorithms on the reacher domain illustrated in Figure We first observe that the performance of all four training policies for RaSFC51 improves almost immediately, obtaining returns that exceed 75% of the performance a fully trained C51 agent, an observation that correlates highly with the original SFDQN. Interestingly, RaC51 is unable to achieve satisfactory performance on two out of the four training tasks, even after fully trained on all 4 tasks. Furthermore, the performance of RaSFC51 on the testing tasks far exceeds that of both RaC51 1 and C51, demonstrating the superior generalization ability of SFs in the risk-aware setting. Finally, the total number of failures across the test task instances also considerably lower for RaSFC51 than is for SFC5 1, and remains low during the entire training horizon. RaC51 and C51 fail frequently at the beginning of training, but less often later in training. While this may suggest that C51 is learning adequate risk-sensitive policies, it is mainly due to the fact that C51 is not able to generalize nearly as well as SFC51 for some of the test target locations, as elaborated in Appendix D.2.
Main Results.
Additional Analysis. We also conducted a sensitivity analysis for the risk- aversion parameter for each of the algorithms, summarized in Figure The performance of RaSFC51 and RaC51 decays gracefully as the level risk-aversion 3 increased magnitude, but the performance of RaSFC51 uniformly outperforms RaC51 for every value of 3 tested. The middle plot demonstrates that the number of failures also decreases drastically as / is increased in magnitude, which stabilizies around 3 -4. Furthermore, while C51 and RaC51 generally fail less often than SFC51 1 and RaSFC51, this

---

40
RaSFC51(4)
RaC51(4)
Signal to Noise
RaSFC51(3)
RaC51(3)
30
RaSFC51(2)
RaC51(2)
RaSFC51(1)
RaC51(1)
SFC51
C51
0
2
3
5
6
8 10 11
Task Instance
Figure 7: Left: Empirical signal-to-noise ratios of the GPI policies at the end of training. Right: Examples of worst-case behavior observed by RaSFC51 with 3 -3, which depict the most likely sources of high return variability. While the agent learns to successfully avoid the risky regions of the domain, high return variance between rollouts can sometimes still be accumulated in other ways, such as failure to find globally optimal policy (top left), inability to generalize the utility precisely on an unseen task for all states (bottom left), policy instability (top right) and divergence in the most extreme cases (bottom right) due to compounding of numerical errors.
is due to their inability to generalize their learned behavior to all novel target instances, as mentioned above. The final plot on the right shows that RaSFC51 indeed learns safer policies than SFC51 on both training and test instances. Specifically, RaSFC51 learns to hover as close to the goal as possible most cases, while still avoiding the high-volatility shaded regions.
Failure Modes and Limitations. Finally, we compared the empirical signal-to-noise ratio achieved by each baseline on each task using the returns of sampled trajectories at the end of training, and the results are summarized in Figure 7 RaSFC51 and SFC51 soundly outperform RaC51 and C51, and RaSFC51 generally obtains the highest ratios for medium values of B {-3. -2 -1.. However, RaSFC51 does not consistently outperform SFC51 on all domains, despite RaSFC51 avoiding the high-risk regions of the domain. We suspect that return variance is accumulated in other ways, some of which can be identified by examining the worst-case behaviors exhibited by the risk-aware GPI policy illustrated in the rightmost plots of Figure 7 However, we believe that these results are expected, Firstly, estimation only the expected return in an off- off-policy setting with bootstrapping and deep function approximation is already difficult problem [44], and addressing outside the scope of this work. Secondly, generalizing the empirical distribution learned on one set of tasks to completely new set of unseen tasks even within the same domain is an even more challenging problem. We believe that advances in distributional RL that can learn more accurate distributions will strongly benefit our approach, will methods suited for estimating the joint distributions correlated returns efficiently.
5 Conclusion
We presented Risk-aware Successor Features (RaSFs) for realizing policy transfer between tasks with shared dynamics, with the goal of maximizing the entropie utility of return. We extended GPI to the risk-aware setting, providing monotone convergence and optimality guarantees, assuming that the utility of source policies can be evaluated. To facilitate policy evaluation, we also extended the notion of GPE to the risk-aware setting. Together, risk-aware GPI and GPE inherit the superior task generalization abilities of successor features, while learning to avoid dangerous high-volatility regions. More generally, incorporating risk and safety in sequential decision -akinng is complex problem. The entropic utility objective does not capture tail risk nor other properties of the return distribution, which could be challenging but powerful extension of successor features.
"Tii computed as ă/ō, where á is the estimated return and à is the estimated standard deviation of the return accumulated across multiple rollouts. We use the signal-to- to-iie-t- ratio because it is independent of /3 and allows consistent comparison across baselines.
10

---

Acknowledgements
[5] Nicole Bäuerle and UIrich Rieder. More risk-sensitive markov decision processes. Mathematics of We would like to thank the anonymous reviewers for their thoughtful comments on the draft version of this paper. We would also like to thank Daniel Mankowitz for suggesting relevant research in the area of robust and risk-aware reinforcement learning and for providing insightful comments during the development of the paper.
Funding Transparency Statement
Michael Gimelfarb was funded by an Ontario Graduate Scholarship, DiDi Graduate Student Award and Vector Institute Postgraduate Affiliate Award, Scott Sanner was funded by an Ontario Early Researcher Award Grant, and Chi-Guhn Lee was funded by an NSERC Discovery Grant.
References
[1] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995. pages 30-37. Elsevier, 1995.
[2] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, David Silver, and Hado P van Hasselt. Successor features for transfer in reinforcement learning. In NeurlPS, 2017.
[3] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In ICML, pages 501-$10, 2018.
[4] André Barreto, Shaobo Hou, Diana Borsa, David Silver. and Doina Precup. Fast reinforcement learning with generalized policy updates. PNAS, (077444)3007-30000 2020.
Operations Research, 39(1):105 -001),00--1.0.11100001 2014.
[6] Mare G Bellemare. Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In ICML, pages 449-458, 2017.
[7] L Bisi, L Sabbioni, E Vittori, M Papini, and M Restelli. Risk-averss trust region optimization for reward-volatilig reduction. In IJCAI, pages 4583 45899, 2020.
[8] Diana Borsa, Andre Barreto, John Quan, Daniel Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In ICLR, 2018.
[9] Oscar Dowson, David P Morton, and Bernardo K Pagnoncelli. Multistage stochastic programs with the entropic risk measure. 2020.
[10] Gabriel Dulac-Aroldd Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. ICML Workshop RLARealLife, 2019.
[11] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Ncar-optimal risk-sample tradeoff in regret. In NeurlPS, volume 33, 2020.
[12] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in reinforcement learning agent. In AAMAS, pages 720 7279-7272000 2006.
[13] Hans Föllmer and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 6(4):429 447, 2002.
[14] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480. 2015.
[15] Javier García and Fernando Fernández. Probabilistic policy reuse for safe reinforcement learning. ACM Trans. Auton. Adapt. Syst. 13(3), March 2019. ISSN 1556-4665. doi: 10.1145/3300000.
[16] Clement Gehring and Doina Precup. Smart exploration in reinforcement Icarning using absolute temporal difference errors. In AAMAS, pages 1037-1044, 2013.
[17] David Held, Zoe McCarthy, Michael Zhang, Fred Shentu, and Pieter Abbeel. Probabilistically safe policy transfer. In ICRA, pages 5798-5805. IEEE, 2017.

---

[18] Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori. Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value risk optimization. In NewrIPS, pages 2619-2629, 2019.
[19] Arushi Jain, Khimya Khetarpal, and Doina Precup. Safe critic learning safety in the option critic architecture. The Knowledge Engineering Review, 36, 2021.
[20] Arushi Jain, Gandhary Patil, Ayush Jain, Khimya Khetarpal, and Doina Precup. Variance penalized on-policy and off-policy actor-critic. arXiv preprint arXiv:2/02.01985, 2021.
[21] David Janz, Jiri Hron, Przemystaw Mazur, Katja Hofmann, José Miguel Hemández- Lohato,, and Schastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning. In NeurlPS, 2019.
[22] Michael Kupper and Walter Schachermayer. Representation results for law invariant time consistent functions. Mathematics and Financial Economics, (33)119-210, 2009.
[23] Alessandro Lazaric. Transfer in reinforcement learning: framework and survey. In Reinforcement Learning, pages 143-173. Springer,, 2012.
[24] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offiin reinforcement learning: Tutorial, review. and Perspectives on Open Problems, 2020.
[25] Daniel Mankowitz, Timothy Mann, Pierre-Luc Bacon, Doina Precup, and Shie Mannor. Learning robust options. In AAAI, volume 32, 2018.
[26] Daniel J Mankowitz, Aviv Tamar, and Shic Mannor. Situational awareness by risk-conscious skills. arXiv preprint arXiv: 02847, 2016.
[27] Shic Mannor and John N Tsitsiklis. Algorithmic aspects of mean- variance optimization in markov decision processes. European Journal of Operational Research, 231(3):645 653, 2013.
[28] Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Alizadeh, Variance reduction for reinforcement learning in input-driven environments. In ICLR, 2019.
[29] Teador Mihai Moldovan and Pieter Abbeel. Risk aversion in markov decision processes via near optimal chernoff bounds. In NeurlPS, pages 1140-3148, 2012.
[30] David Nass, Boris Belousov, and Jan Peters. Entropic risk measure in policy search. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1101-1106. IEEE, 2019.
[31] Mark Nemecek and Ronald Parr. Policy caches with successor features. In ICML,, volume 139 of Proceedings of Machine Learning Research, pages 8025-8033. PMIR, 18-24 Jul 2021.
[32] Takayuki Osogami. Robustness and risk-sensitivity in markov decision processes. In NewrlPS, volume 25, pages 233-241, 2012.
[33] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[34] Andrzej Ruszezyáski. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125(2):235-261, 2010.
[35] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In ICML, pages 1312-1320. PMLR, 2015.
[36] Yun Shen, Michael J Tobia, Tobias Sommer. and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7): 1328, 2014.
[37] Craig Sherstan, Dylan R Ashley, Brendan Bennett, Kenny Young, Adam White, Martha White, and Richard S Sutton. Comparing direct and indirect defporal- methods for estimating the variance of the return. In UAI, 2018.
[38] Krishman Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep with safety critic. arXiv preprint arXiv:2010.1400 2020.
[39] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In ICML, pages 1651-1658. 2012.
[40] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Learning the variance of the reward-to to-go. The Journal of Machine Learning Research. 17(1):361-35) 2016.
12

---

[41] Matthew E Taylor and Peter Stone. Transfer learning for reinforvement learning domains: A survey. Journal Machine Learning Research, 10(7), 2009.
[42] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In
2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033 IEEE, 2012.
[43] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. NeurIPS, volume 33, 2020.
[44] Hado Van Hasselt, Yotam Doron. Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and deadly triad. arXiv preprint arXiv:1812.0000 2018.
[45] Whiteson, Zhang, and B Liu. Mean- variance policy iteration for risk- reinforcement learning. In AAAI. Association for the Advancement Artificial Intelligence, 2021.
[46] Yang Yu. Towards sample efficient reinforcement learning. In I/CAL, pages 5739-5743, 2018.
[47] Yuhua 7hu and Lexing Ying. Borrowing from the future: An attempt to address double sampling. In Mathematical and scientific machine learning, pages 246-268. PMLR, 2020.
13


Reviews:
[{"review_id": "cdJzJ997rv8", "reviewer": "Reviewer_cpAq", "summary": "This paper considers safety in transfer RL, and proposes a method, called Risk-aware successor features, that exploits the task structure to avoid risks and achieve task generalization between tasks with shared dynamics and different goals. The entropic utility is introduced to represent risk-awareness and specialized to entropic Q-value with bellman update. Risk-aware GPI and GPE are extended from GPI, GPE, and proved its convergence. Experiments on two domains show the method outperforms previous works.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents Risk-aware Successor Features (RaSFs) for realizing policy transfer between tasks with shared dynamics and different goals. The objective is to optimize the trade-off between expected return and risk, which is measured by the variance of return. \n\nStrengths:\n\n1. This paper extends risk-aware GPI, provides monotone guarantees and optimality of GPI, extends GPE to the mean-variance objective, inheriting the superior task generalization abilities of successor features. \n2. The analysis on both discrete and continuous domains has showed.\n3. The paper is clear to read, easy to follow.\n\nWeaknesses:\n\nWhy PRQL(2005) is a challenging baseline? Why not consider more recent policy transfer methods?\n\nTypos:\nLine269: The w are also learned --> is", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors only discuss some limitations in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "_yZC3tsX2N", "reviewer": "Reviewer_TjXA", "summary": "The article extends the Successor Feature framework to the risk-averse setting in which the Entropic Risk Measure is optimized in place of the standard risk-neutral objective. The proposed approach is tested on a gridworld domain and on a Mujoco one, this time integrated with a distributional RL algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality:\nThe approach is a direct extension of successor features, based on the properties of the Bellman equation of the entropic risk-measure and on its Taylor expansion. Related works are clearly cited in a dedicated section.\n\n\n## Quality: \nThe submission is technically sound from both the theoretical and the empirical point of view. The author determine the theoretical properties of the proposed approach an then tested them in valid environments.\nFrom the experimental side, I only have doubt about the choice of the performance measure. The author chose to show the expected return, together with the failures. However, it would have been fairer to show the optimized objective (the entropic risk measure). The latter may also suffer of instability issues, hence, it would have been interesting to see if this aspects affect the proposed technique. \n\n## Clarity: \nThe paper is well written and organized in a adequate way. Both the experimental analysis and the theory are sufficiently clear to understand the illustrated results. I would suggest anyway minor improvements on some specific parts of the article:\n- the example in 3.1, can be exposed in a clearer way, in particular, it is not clear what happens when \\beta is equal to 0: which are the optimal policies in the different tasks, in this case?\n- In Theorem 1, it is not clear, without looking at the proof, what is the difference between Q and \\Tilde{Q}.\n- in Figure 3, it is not immediate to understand the performance of the different techniques, maybe a table would do a better job.\n- In Figure 4, I would avoid to use faded curves, they are difficult to read. Also, in the second figure, the x-label should be \"Test task Instance\", from what I understand from the caption.\n- In the \"Reacher\" paragraph of the experimental section, it is not clear what \"without interactions effect\" means.\n- Showing the variance instead of the total failures is also an interesting alternative, justified by the fact that ERM approximate mean-variance, and the studied setting is not a CMDP.\n\n## Significance: \nThe proposed approach outperforms the baselines in the first environment, but not its risk-neutral version, even if it allows for a lower number of failures. A similar behavior is observed in the second environment. It is hence difficult to assess whether the proposed approach is better then its risk-neutral version, if the final perfomance w.r.t. the optimization goal is not shown. Nevertheless, the approach offers an interesting an theoretically justified way to transfer learning in risk-averse tasks, representing a clear advance in the state of the art.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative impact are present.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "X_1t7teZyc0", "reviewer": "Reviewer_TAaE", "summary": "In this paper, a (variance-based) risk-aware transfer method for multi-task RL is proposed. \nIn the theory part, GPI and GPE (and the notion of successor features) are extended to entropic utility (for the risk-aware transfer). \nBased on the results in the theory part (specifically, the GPE/successor features parts), the successor feature for entropic utility is proposed to use for risk-aware setting. \nIn the experiment part, the proposed method is evaluated in the Four-room and Reacher environments. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Key strength:  \nSolid theoretical results are provided.  \n\nTheories of (vanilla) GPI and GPE (Theories 1 and 2 in [2]) are extended to entropic utility (i.e., Theories 1 and 2 in the paper under review). Although the derivation (proof) flow for the theories is basically similar to those for the vanilla GPI and GPE in [2], the theoretical results are meaningful because the results extend the GPI and GPE into a risk-aware setting.  \nIn addition, Theorem 3 (convergence of covariance) is novel.  \n\nKey weakness:  \nEvaluation of the proposed method (RaSF) is conducted only in a simple environment. \nIn the evaluation, Four-Room (grid-world) and Reacher (simple control) are used for benchmark environments.  \nIt is unclear if it works well in more complex environments (e.g., Ant-maze with traps). \n\n\nI will improve my score if a convincing rebuttal to my comments on \"Key weakness\" is provided (e.g., the additional experiment result in more complex environments). \n\n\nMinor comments:  \nLine 76 Other previous work on safe options: Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, Yoshimasa Tsuruoka. Learning Robust Options by Conditional Value at Risk Optimization, In NeurIPS, 2019  \nLine 431 Deep rl -> Deep RL  \nLine 442 NeurIPS -> In NeurIPS  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "H4xl2h70qK1", "reviewer": "Reviewer_5osE", "summary": "This paper extends the successor feature framework including the GPE/GPI methods for generalisation across tasks with risk awareness. The paper includes experiments for the associated methods on toy domains.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper has good clarity, however the theoretical results do not seem super useful and the the paper simultaneously lacks on strong empirical evaluation.\n\nThe authors should try to address the following:\n- How is sample inefficiency a cause for not addressing safety, lines 21-24 are bit unclear.\n- A little derivation for taylor expansion in line 98 should be provided in main text\n- How useful are Theorem 1 and 2? Ideally one would want to have little uncertainty in the initial Q estimates for transfer as the policy objective is to maximise expected value under initial state distribution, however the theorems gives a very weak bound for h=0\n- The method not only assumes access to SFs, which while being problematic to compute themselves also requires estimates for additional quantities like their covariance matrix, I feel this pushes most of the burden of solving hard tasks under the carpet. It is not very convincing to me that tasks which are interesting to solve and pose real challenges with respect to risk uncertainties would simultaneously admit low dimensional SF descriptors (note this is really important for the method given computability of O(d^2) structures like covariance matrix, d being SF dimensionality). Further the lack of thorough experimentation on this front compounds my doubts.\n- Are there guarantees which extend for general \\beta for the GPI case? Currently it seems that the source and the target tasks assume same risk awareness coefficient.\n\nUpdates post discussion:\n- I think the authors have made convincing points to address some of the issues raised in my review, also since they will be adding more experiments as stated, I have therefore decided to increase my rating for acceptance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned before the main limitation of the work is lack of evidence to support existence and learnability of succinct SFs to describe the space of practical risk sensitive problem domains. Additionally the theoretical results presented do not appear very informative for generalization.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' feedback was instrumental in the reviewers' change of mind.
The paper is still borderline, but I trust the authors' promises to revise their paper according to the reviewers' suggestions. and I propose to accept this paper.
I will check the final version of this paper hoping to find all the additions that have been requested.
Decision: Accept (Poster)