Risk-Aware Transfer in Reinforcement Learning using Successor Features
Year: 2022

Abstract:
Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning and the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending policy improvement via dynamic programming to sets of policies and levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, and incorporate risk-awareness into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.

heading: 1 Introduction
text: Reinforcement learning (RL) is a general framework for solving sequential decision-making problems, in which an agent interacts with an environment and receives continuous feedback in the form of rewards. However, many classical algorithms in RL do not explicitly address the need for safety, making them unreliable and difficult to deploy in some real-world applications [10]. One reason for this is the relative sample inefficiency of model-free RL algorithms, which often require millions of costly or dangerous interactions with the environment or fail to converge altogether [44, 46]. Transfer learning addresses these problems by incorporating prior knowledge or skills [23, 41]. Despite this, using the expected return as a measure of optimality could still lead to undesirable behavior such as excessive risk-taking, since low-probability catastrophic outcomes with negative reward and high variance could be underrepresented [29]. For this reason, risk-awareness is becoming an important aspect in the design of practical RL [14]. Thus, an ultimate goal of developing reliable systems should be to ensure that they are both sample efficient and risk-aware.
∗Affiliate to Vector Institute, Toronto, Canada.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
We take a step in this direction by studying the problem of risk-aware policy transfer between tasks with different goals. A powerful way to tackle this problem in the risk-neutral setting is the GPI/GPE framework, of which successor features (SF) are a notable example [2]. Here, generalized policy improvement (GPI) provides a theoretical framework for transferring policies with monotone improvement guarantees, while generalized policy evaluation (GPE) facilitates the efficient evaluation of policies on novel tasks and is a key component in satisfying the assumptions of GPI in practice. Together, GPI/GPE provide strong transfer benefits in novel task instances even before any direct interaction with them has taken place, a phenomenon we call task generalization. The key to the superb generalization of GPI/GPE lies in their ability to directly exploit the structure of the task space, taking advantage of subtle differences and commonalities between task goals to transfer skills seamlessly in a composable manner. This property could be an effective way of tackling problems in offline RL [24], such as the transfer of skills learned in a simulator to a real-world environment. However in many cases, such as helicopter flight control [16], making one wrong decision could lead to catastrophic outcomes. Hence, being risk-aware could offer one way to avoid worst-case outcomes when transferring skills in real-world settings.
Contributions. We contribute a novel successor feature framework for transferring policies with the goal of maximizing the entropic utility of return in MDPs (Section 2.2). Intuitively, the entropic utility encourages agents to follow policies with predictable and controllable returns characterized by low variance, thus providing a natural way to incorporate risk-awareness. Furthermore, while our theoretical framework could be extended to other classes of utility functions, the entropic utility has many favorable mathematical properties [13, 22] that we exploit directly in this work to achieve optimal transfer (Lemma 1, Theorem 1 and 2). We also derive a form of risk-aware GPE based on the mean-variance approximation, in which the sufficient statistics of the return distribution can be computed directly (Section 3.3) or by leveraging recent developments in distributional RL [6]. Our resulting approach, which we call Risk-Aware Successor Features (RaSF), is able to exploit the task structure to achieve task generalization with respect to novel goal instances as well as levels of risk aversion, where emphasis is placed on avoiding high volatility of returns. Our approach is also complementary to other advances in successor features, including feature learning [3], universal approximation [8], exploration [21], and non-stationary reward preferences [4]. Empirical evaluations on discrete navigation and continuous robot control domains (Section 4) demonstrate the ability of RaSFs to better manage the trade-off between return and risk and avoid catastrophic outcomes, while providing excellent generalization on novel tasks in the same domain.
Related Work. The entropic and mean-variance objectives are popular ways of incorporating risk-awareness in RL [7, 11, 20, 27, 28, 30, 36, 39, 45]. However, transferring learned skills between tasks while taking risk into account is a difficult problem. One way to implement risk-aware transfer is to learn a critic [38] or teacher [43] that can guide an agent toward safer behaviors on future tasks. The risk-aware transfer of a policy from a simulator to a real-world setting has also been studied in the area of robotics [17]. Another approach for reusing policies is the probabilistic policy reuse of García and Fernández [15], but requires strong assumptions on the task space. Hierarchical RL (HRL) is another related approach that relies on hierarchical abstractions, enabling an agent to decompose tasks into a hierarchy of sub-tasks, and facilitating the transfer of temporally-extended skills from sub-tasks to the parent task. The CISR approach of Mankowitz et al. [26] is the first to investigate safety explicitly within HRL, followed up by work on safe options [18, 19, 25]. However, none of the existing work takes advantage of the compositional structure of task rewards to transfer skills while optimizing the variance-adjusted return, which is the problem we tackle in this paper (see Table 1).

heading: 2 Preliminaries

heading: 2.1 Markov Decision Process
text: Sequential decision-making in this paper follows the Markov decision process (MDP), defined as a four-tuple 〈S,A, r, P 〉: S is a set of states; A is a finite set of actions; r : S × A × S → R is a bounded reward function, where r(s, a, s′) is the immediate reward received upon transitioning to state s′ after taking action a in state s; and P : S × A × S → [0,∞) is the transition function for state dynamics, where P (s′|s, a) is the probability of transitioning to state s′ immediately after taking action a in state s.
In the episodic MDP setting, decisions are made over a horizon T = {0, 1, . . . T} where T ∈ N. We define a stochastic Markov policy as a mapping π : S × T → P(A), where P(A) denotes the set of all probability distributions over A. Similarly, a deterministic Markov policy is a mapping π : S × T → A. In the risk-neutral setting, the goal is to find a policy π that maximizes the expected sum of future rewards after initially taking action a in state s,
Qπh(s, a) = Est+1∼P (·|st,at)
[ T∑
t=h
r(st, at, st+1) ∣∣∣ sh = s, ah = a, at ∼ πt(st)] .
In this case, it is possible to show that a deterministic Markov policy π∗ is optimal [33]. The theoretical framework in this paper also allows for time-dependent reward or transition functions.

heading: 2.2 Entropic Utility Maximization
text: We incorporate risk-awareness into the decision-making by maximizing the entropic utility Uβ of the cumulative reward, defined for a fixed β ∈ R as
Uβ [R] = 1
β logE
[ eβR ] , (1)
for real-valued random variables R on a bounded support Ω ⊂ R. An important property of the entropic utility is the Taylor expansion Uβ [R] = E[R] + β2Var[R] +O(β
2). Interpreting the risk as return variance, β can now be interpreted as the risk aversion of the agent: choosing β < 0 (β > 0) leads to risk-averse (risk-seeking) behavior, while β = 0 is risk-neutral, e.g. U0[R] = E[R]. Specializing (1) to the MDP setting, the goal is to maximize
Qπh,β(s, a) = Uβ
[ T∑
t=h
r(st, πt(st), st+1)
] (2)
over all policies starting from sh = s and ah = a. As in the risk-neutral setting, it is possible to show that a deterministic Markov policy is optimal [5]. Furthermore, Qπh,β can be computed iteratively through time using the Bellman equation [9, 32]:
Qπh,β(s, a) = Uβ [ r(s, a, s′) +Qπh+1,β(s′, πh+1(s′)) ] = 1
β logEs′∼P (·|s,a)
[ exp { β ( r(s, a, s′) +Qπh+1,β(s′, πh+1(s′)) )}] ,
(3)
starting with QπT+1,β(s, a) = 0. In fact, (1) is the only utility function that has this equivalence, and other key properties (Lemma 1), while also satisfying time consistency that ensures the learned risk-aware behaviors remain consistent across time [22]. In this paper, we use (3) to establish a general GPI framework for risk-aware transfer learning with provable guarantees, and leverage approximations of (2) to learn portable policy representations.
In reinforcement learning, the Bellman equation is not applied directly since it suffers from the curse of dimensionality when S is high-dimensional or continuous, and since neither the dynamics nor the reward function are often known. Instead, the agent interacts with the environment using a stochastic exploration policy πe, collects trajectories {(st, at, st+1, rt+1)}T−1t=0 , and updates Qh,β(st, at) via sample approximations Ûβ ≈ Uβ [36]. Our goal is to ameliorate the relative sample-inefficiency of RL through transfer learning that we aim to generalize to the risk-aware setting.

heading: 2.3 Transfer Learning
text: We now formalize the general transfer learning problem. Let M be the set of all MDPs with shared transition function P but different (bounded) reward functions. A fixed set of source tasks M1, . . .Mn ∈ M is instantiated, and their corresponding optimal policies π1, . . . πn are estimated. Our main goal is to transfer these resulting source policies to a new target task Mn+1 ∈ M, to obtain a policy π∗n+1 whose utility is better than one learned from scratch using only a fixed number of samples from Mn+1. We refer to this outcome as positive transfer.
As discussed earlier, a standard way to implement transfer learning is the GPI/GPE framework of Barreto et al. [2]. The core mechanism that enables positive transfer in the risk-neutral setting is called generalized policy improvement (GPI). Specifically, the set of source policies π1, . . . πn are evaluated on the target task Mn+1 to obtain corresponding values Qπ1n+1, . . . Q πn n+1. Given a mechanism that can perform this policy evaluation step efficiently with some small error ε — namely successor features discussed and extended in Section 3.3 — an agent then selects actions in a greedy manner by following policy π(s) ∈ argmaxa maxj=1...n Q πj n+1(s, a) in state s. The policy π corresponds to a strict policy improvement operator, and thus fulfills our requirements for positive transfer.

heading: 3 Risk-Aware Transfer Learning
text: An obvious challenge of applying GPI in the risk-aware setting is that transferring optimal riskneutral source policies does not guarantee risk-aware optimality in the target task. A much stronger observation is that, even if the source policies πj are risk-aware, performing the policy evaluation step in a risk-neutral way can still break the risk-awareness of GPI. This makes the extension of GPI to the risk-aware setting a non-trivial problem.
3.1 A Motivating Example
To see this, consider the MDP shown in Figure 1, which involves navigating from an initial state ‘S’ to a goal state ‘G’ in a grid with stochastic transitions. Traps of two types (X, Y) are placed in fixed cells, which upon entry terminate the episode with fixed costs, summarized compactly as pairs (c1, c2). We define two source tasks with costs (20, 20) and (0, 0) and a target task with costs (20, 0). The optimal policies for β = −0.1 induce three different trajectories: safe (blue) and hazardous routes (red) for the source tasks, and a relatively safe route (green) for the target task. We also note that risk-awareness is prerequisite for this problem, since a risk-neutral agent prefers the hazardous path when optimizing any of the tasks.
We compute the GPI policies π with risk-averse (β = −0.1) and risk-neutral (β = 0) policy evaluation, as shown in the middle and rightmost plots in the top row in Figure 1. The bottom plot shows the distribution of returns collected over 5,000 test runs by acting according to the two GPI policies. Risk-averse policy evaluation results in an optimal risk-averse GPI policy corresponding to the green trajectory, whereas risk-neutral policy evaluation does not, even though
both source policies are optimal on their corresponding tasks. Interestingly, the risk-averse GPI policy is a non-trivial stitching of the two source policies.

heading: 3.2 Risk-Aware Generalized Policy Improvement
text: Motivated by this example, we conjecture that the risk-awareness of the GPI policy π is primarily dependent on the way in which the source policies are evaluated in target task instances. In this section, we describe theoretical results that generalize the concept of generalized policy iteration to the problem of maximizing the entropic utility of returns.
We begin by summarizing key properties necessary for establishing convergence of risk-aware GPI in the following lemma.
Lemma 1. Let β ∈ R and X,Y be arbitrary random variables on Ω. Then:
(1) (monotonicity) if P(X ≥ Y ) = 1 then Uβ [X] ≥ Uβ [Y ] (2) (cash invariance) Uβ [X + c] = Uβ [X] + c for every c ∈ R (3) (convexity) if β < 0 (β > 0) then Uβ is a concave (convex) function (4) (non-expansion) for f, g : Ω → Ω, it follows that
|Uβ [f(X)]− Uβ [g(X)]| ≤ sup P∈PX(Ω) EP |f(X)− g(X)|,
where PX(Ω) is the set of all probability distributions on Ω that are absolutely continuous w.r.t. the true distribution of X .
A proof is provided in Appendix B.2. Properties (1)-(3) characterize concave utilities [13], which intuitively can be seen as minimal requirements for rational decision-making: (1) a lottery that pays off more than another in every possible state of the world should always be preferred; (2) adding cash to a position should not increase its risk; and (3) the overall utility can be improved by diversifying across different risks. Property (4) is a derivative of the first three, and thus the theoretical results in this work could be extended to the broader class of iterated concave utilities [34].
We can now state the first main result of the paper.
Theorem 1 (GPI for Entropic Utility). Let π1, . . . πn be arbitrary deterministic Markov policies with approximate entropic utilities Q̃π1h,β , . . . Q̃ πn h,β when evaluated in an arbitrary task M , with error |Q̃πih,β(s, a)−Q πi h,β(s, a)| ≤ ε for all s ∈ S , a ∈ A, i = 1 . . . n and h ∈ T . Define
πh(s) ∈ argmax a∈A max i=1...n Q̃πih,β(s, a), ∀s ∈ S. (4)
Then, Qπh,β(s, a) ≥ max
i Qπih,β(s, a)− 2(T − h+ 1)ε, h ≤ T.
Thus, evaluating the risk of source policies using Uβ provides monotone improvement guarantees for GPI, and thus satisfies our definition of positive transfer. Another significant property of the bound in Theorem 1, and the one in Theorem 2 below, is the linear separation between the optimal utility and the approximation error ε. Knowing how the optimality of π explicitly depends on ε, and how errors are propagated throughout the transfer learning process, is critical for developing reliable RL with predictable behavior, and highlights a key advantage of making GPI risk-aware. The additive relationship between the optimality and the approximation error in Theorem 1 further explains why SFs are robust to approximation errors. This becomes particularly advantageous in our setting, since estimating utilities Uβ accurately with GPE is more complicated than the risk-neutral setting.
A stronger result for GPI can be derived when the source policies π1, π2 . . . πn are ε-optimal, and policy evaluation is once again performed using Uβ . In this case, the optimality of GPI is determined by the similarity δr between the source and target task instances. Theorem 2. Let Qπ ∗ i
h,β be the utilities of optimal Markov policies π ∗ i from task Mi but evaluated in task
M with reward function r(s, a, s′). Furthermore, let Q̃π ∗ i h,β be such that |Q̃ π∗i h,β(s, a)−Q π∗i h,β(s, a)| ≤ ε for all s ∈ S, a ∈ A, h ∈ T and i = 1 . . . n, and let π be the corresponding policy in (4). Finally, let δr = mini=1...n sups,a,s′ |r(s, a, s′)− ri(s, a, s′)|. Then,∣∣Qπh,β(s, a)−Q∗h,β(s, a)∣∣ ≤ 2(T − h+ 1)(δr + ε), h ≤ T. These results are proved in Appendix B.2 for the episodic setting and in Appendix B.3 for the discounted setting. Also, please note that these bounds are tight (see, e.g. Nemecek and Parr [31] for the risk-neutral setting). Finally, while not required in this work, the above results could be extended to more general settings in risk-averse control [34], though practical implementation of GPE in these settings remains an open problem.

heading: 3.3 Risk-Aware Generalized Policy Evaluation
text: Following Barreto et al. [2], let φ : S ×A× S ′ → Rd be a bounded and task-independent feature map, and consider the following linear representation of rewards,
r(s, a, s′) = φ(s, a, s′) ᵀ w, ∀s, a, s′,
where w ∈ Rd is a task-dependent vector of reward parameters. The risk-neutral return becomes:
Qπh(s, a) = EP
[ T∑
t=h
φ(st, at, st+1) ᵀ w ∣∣∣ sh = s, ah = a, at ∼ πt(st)]
= EP
[ T∑
t=h
φ(st, at, st+1) ∣∣∣ sh = s, ah = a, at ∼ πt(st)]ᵀw = ψπh(s, a)ᵀw, (5)
where ψπh(s, a) are the successor features (SFs) associated with the policy π. The linear dependence of the return on w allows for instantaneous policy evaluation in novel tasks with arbitrary reward preferences w, making it a particular — and perhaps the canonical — instantiation of GPE. More critically, ψπh can be seen as a task-independent and highly portable linear feature representation of policies, and it is the key to the generalization ability of SFs on novel task instances.
The concept of GPE can be generalized to incorporate entire distributions of the return. Repeating the above derivation for the entropic utility (2), we have:
Qπh,β(s, a) = Uβ
[ T∑
t=h
r(st, πt(st), st+1) ] = Uβ [Ψ π h(s, a) ᵀ w] , (6)
corresponding to the random vector Ψπh(s, a) = ∑T
t=h φt of unrealized feature returns at time h. Thus, we have transformed the problem of estimating the utility of returns into the problem of estimating the distribution of Ψπh(s, a) ᵀ w. The key question now is how to estimate this distribution for fast GPE.
A natural way to do this is by applying a second-order Taylor expansion for Uβ , since it allows us to precompute and cache the necessary moments of the return distribution:
Uβ [Ψ π h(s, a) ᵀ w] = EP [Ψπh(s, a) ᵀ w] +
β 2 VarP [Ψ π h(s, a) ᵀ w] +O(β2)
≈ ψπh(s, a) ᵀ w +
β 2 wᵀVarP [Ψ π h(s, a)]w = Q̃πh,β(s, a), (7)
in which VarP [Ψπh(s, a)] = Σ π h(s, a) is interpreted as a covariance matrix for SFs. In the context of Theorems 1 and 2, the term ε encapsulates the errors in the approximation of ψπh and Σ π h, plus the terms contained in O(β2) above. However, the main advantage of (7) is that, like (5), it is also analytic in w and allows for instantaneous policy evaluation with arbitrary reward preferences w. Interestingly, (7) also allows instantaneous policy evaluation with respect to different choices of β, making it possible to revise or adapt the level of risk aversion on-demand. In all these cases, ψπh and Σπh provide task-independent and portable representations of policies while also accounting for exogenous risk. This is the key to preserving the task generalization ability of SFs in the risk-aware setting, and (7) can now be seen as a particular instantiation of GPE. We call this overall approach Risk-aware Successor Features (RaSF).
The simplest approaches for estimating Σπh in the exact (e.g. tabular) Q-learning setting are based on dynamic programming [37, 40], which would allow the overall approach to be easily integrated into existing SF implementations. In particular, the covariance satisfies the Bellman equation
Σπh(s, a) = Es′∼P (·|s,a) [ δhδh ᵀ +Σπh+1(s ′, πh+1(s ′)) | sh = s, ah = a ] , (8)
where δh are the Bellman residuals of ψπh(s, a). The approximation ψ̃ π h is known to converge to the true value ψπh , and a similar result also holds for updating the covariance based on (8).
Theorem 3 (Convergence of Covariance). Let ‖ · ‖ be a matrix-compatible norm, and suppose there exists ε : S ×A× T → [0,∞) such that:
1. ‖ψ̃πh(s, a)−ψπh(s, a)‖2 ≤ εh(s, a)
2. ‖Es′∼P (·|s,a)[δ̃h(ψ̃πh(s′, πh+1(s′))−ψπh(s′, πh+1(s′))) ᵀ ]‖ ≤ εh(s, a).
Then, ∥∥∥Σπh(s, a)− Es′∼P (·|s,a) [δ̃hδ̃ᵀh + Σ̃πh+1(s′, πh+1(s′))]∥∥∥ ≤ 3εh(s, a). A proof can be found in Appendix B.4. Appendix A.1 describes how ψ̃πh and Σ̃ π h can be learned online from environment interactions, while Appendix A.4 discusses further generalizations of (7). There are, however, several limitations of estimating Σπh in this way. First, obtaining accurate estimates of Σπh requires accurate estimates of ψ π h (thus estimating one quantity on top of another), making this approach difficult to apply with deep function approximation. This claim is further substantiated by Theorem 3 and preliminary experiments. A second issue that occurs is double sampling, when the same transitions are used to update the mean and covariance, resulting in accumulation of bias in the latter [1, 47]. Our experiments on the reacher domain mitigate these issues by leveraging distributional RL to approximate (7), while maintaining computational efficiency.

heading: 4 Experiments
text: To evaluate the performance of RaSF, we revisit the benchmark domains in Barreto et al. [2], which have been slightly modified for learning and evaluating risk-aware behaviors. We defer all experimental details to Appendix C.
4.1 Four-Room
Domain Description. The first domain consists of a family of navigation tasks defined on a discrete 2-D space divided into four rooms, as illustrated on the left in Figure 2. The environment has additional objects that can be picked up by the agent by occupying their cells. These objects belong to one of three possible classes, drawn as different shapes in Figure 2, which determine their reward. The position of the objects remains fixed, but the rewards of their classes are reset every 20,000 transitions to random values sampled uniformly in [−1,+1]. To incorporate risk, traps are placed in fixed cells marked with X. For every time instant during which the agent occupies a trap cell, the trap activates spontaneously with a small probability, resulting in an immediate penalty and termination of the episode (we refer to this event as a failure). The goal is to maximize the total reward accumulated over 128 random task instances while minimizing the number of failures.
Baselines. In order to demonstrate the power of our approach in the absence of approximation errors, we define a simple instance of RaSFs in which ψπi and Σπi are learned exactly using lookup tables and dynamic programming (equation (8)). We also apply modest discounting of rewards to ensure that the Q-function converges, as is standard in RL and discussed further in Appendix A.2. The vector w is also learned using immediate reward feedback and exact, sparse state features φ provided to the agent. Due to its similarity to standard Q-learning, we call this approach RaSFQL. To provide a challenging baseline for comparison, we implemented another policy reuse algorithm (PRQL) [12]. Further replacing the risk-neutral action selection mechanism of PRQL with smart exploration [16] allows PRQL to easily account for return volatility, and we refer to this baseline as RaPRQL.
Main Results. The performance of these algorithms is shown in Figure 3. The cumulative reward obtained by RaSFQL is generally lower than SFQL, as expected since a risk-averse agent should avoid the objects in the bottom-left and top-right rooms and forgo their associated rewards. Interestingly, the performance of RaSFQL far exceeds that of RaPRQL and even PRQL, suggesting that the benefits
of task generalization provided by GPI/GPE are quite strong. Furthermore, the number of failures observed by RaSFQL gradually decreases over the task instances, while the number of failures of SFQL slightly increases. This is consistent with Theorem 1 that guarantees monotone improvement in the risk-adjusted return of RaSFQL. On the other hand, while RaPRQL also learns to avoid risk, it fails slightly more often than RaSFQL. This suggests that the benefits of task generalization promised by GPI/GPE even allow risk-aware behaviors to emerge sooner than by using generic policy reuse methods that are unable to exploit the task structure, namely PRQL. This aspect becomes critical for minimizing failures when deploying a trained policy library on novel task instances in a real-world setting. Further analysis and ablation studies are provided in Appendix D.1.

heading: 4.2 Reacher
text: Domain Description. The second domain consists of a set of tasks based on the MuJoCo physics engine [42] that involve the maneuver of a robotic arm toward a fixed target location. As illustrated in the rightmost plot in Figure 2, the agent is only allowed to train on 4 tasks, whose target locations are indicated by colored circles, and must be able to perform well on 8 test tasks whose target locations are indicated by the grey circles. Furthermore, we incorporate two sources of reward volatility: (1) actions are perturbed by additive Gaussian noise; and (2) fixed regions around some of the target locations randomly incur negative rewards (failures), illustrated by faded circles in Figure 2. Please note that most of these high-volatility regions are centered on target locations of test tasks from which
the agent never learns directly. This stresses the agent’s ability to avoid unforeseen dangers in the environment, in additional to performing well on previously unseen task instances.
Baselines. As discussed earlier, it is difficult to compute Σπi directly using (8). A computationally tractable way to avoid these issues is to first approximate the density of Ψπi(s, a), and then extract the moments needed to compute (8). Specifically, we apply C51 [6] by modeling Ψπi1 (s, a), . . .Ψ πi d (s, a) using histograms for each (s, a). However, Ψπi are high-dimensional, so we avoid the curse of dimensionality by modeling the marginals Ψπij rather than their full joint distribution. This still turns out to be an effective way of detecting high-variance scenarios in the environment. The final architecture is illustrated in Figure 5,
where the marginal distributions of Ψπi are modeled as separate output "heads" with a shared state encoder. The rest of the training protocol is identical to the SFDQN of Barreto et al. [2], except that DQN is replaced by the C51 architecture above, as further detailed in Appendix A.3. The risk-averse and risk-neutral instances of this approach for modeling successor features are referred to as RaSFC51 and SFC51, respectively, while RaC51 and C51 replace successor features with universal value functions [35] for generalization across target locations.
Main Results. The performance of the algorithms on the reacher domain is illustrated in Figure 4. We first observe that the performance of all four training policies for RaSFC51 improves almost immediately, obtaining returns that exceed 75% of the performance of a fully trained C51 agent, an observation that correlates highly with the original SFDQN. Interestingly, RaC51 is unable to achieve satisfactory performance on two out of the four training tasks, even after fully trained on all 4 tasks. Furthermore, the performance of RaSFC51 on the testing tasks far exceeds that of both RaC51 and C51, demonstrating the superior generalization ability of SFs in the risk-aware setting. Finally, the total number of failures across the test task instances is also considerably lower for RaSFC51 than it is for SFC51, and remains low during the entire training horizon. RaC51 and C51 fail frequently at the beginning of training, but less often later in training. While this may suggest that C51 is learning adequate risk-sensitive policies, it is mainly due to the fact that C51 is not able to generalize nearly as well as SFC51 for some of the test target locations, as elaborated in Appendix D.2.
Additional Analysis. We also conducted a sensitivity analysis for the risk-aversion parameter β for each of the algorithms, summarized in Figure 6. The performance of RaSFC51 and RaC51 decays gracefully as the level of risk-aversion β is increased in magnitude, but the performance of RaSFC51 uniformly outperforms RaC51 for every value of β tested. The middle plot demonstrates that the number of failures also decreases drastically as β is increased in magnitude, which stabilizies around β = −4. Furthermore, while C51 and RaC51 generally fail less often than SFC51 and RaSFC51, this
is due to their inability to generalize their learned behavior to all novel target instances, as mentioned above. The final plot on the right shows that RaSFC51 indeed learns safer policies than SFC51 on both training and test instances. Specifically, RaSFC51 learns to hover as close to the goal as possible in most cases, while still avoiding the high-volatility shaded regions.
Failure Modes and Limitations. Finally, we compared the empirical signal-to-noise ratios2 achieved by each baseline on each task using the returns of sampled trajectories at the end of training, and the results are summarized in Figure 7. RaSFC51 and SFC51 soundly outperform RaC51 and C51, and RaSFC51 generally obtains the highest ratios for medium values of β ∈ {−3,−2,−1}. However, RaSFC51 does not consistently outperform SFC51 on all domains, despite RaSFC51 avoiding the high-risk regions of the domain. We suspect that return variance is accumulated in other ways, some of which can be identified by examining the worst-case behaviors exhibited by the risk-aware GPI policy illustrated in the rightmost plots of Figure 7. However, we believe that these results are expected. Firstly, estimation of only the expected return in an off-policy setting with bootstrapping and deep function approximation is already a difficult problem [44], and addressing it is outside the scope of this work. Secondly, generalizing the empirical distribution learned on one set of tasks to a completely new set of unseen tasks – even within the same domain – is an even more challenging problem. We believe that advances in distributional RL that can learn more accurate distributions will strongly benefit our approach, as will methods suited for estimating the joint distributions of correlated returns efficiently.

heading: 5 Conclusion
text: We presented Risk-aware Successor Features (RaSFs) for realizing policy transfer between tasks with shared dynamics, with the goal of maximizing the entropic utility of return. We extended GPI to the risk-aware setting, providing monotone convergence and optimality guarantees, assuming that the utility of source policies can be evaluated. To facilitate policy evaluation, we also extended the notion of GPE to the risk-aware setting. Together, risk-aware GPI and GPE inherit the superior task generalization abilities of successor features, while learning to avoid dangerous high-volatility regions. More generally, incorporating risk and safety in sequential decision-making is a complex problem. The entropic utility objective does not capture tail risk nor other properties of the return distribution, which could be a challenging but powerful extension of successor features.
2This is computed as µ̂/σ̂, where µ̂ is the estimated return and σ̂ is the estimated standard deviation of the return accumulated across multiple rollouts. We use the signal-to-noise ratio because it is independent of β and allows consistent comparison across baselines.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their thoughtful comments on the draft version of this paper. We would also like to thank Daniel Mankowitz for suggesting relevant research in the area of robust and risk-aware reinforcement learning and for providing insightful comments during the development of the paper.
Funding Transparency Statement
Michael Gimelfarb was funded by an Ontario Graduate Scholarship, a DiDi Graduate Student Award and a Vector Institute Postgraduate Affiliate Award, Scott Sanner was funded by an Ontario Early Researcher Award Grant, and Chi-Guhn Lee was funded by an NSERC Discovery Grant.

Review summaries:
This paper considers safety in transfer RL, and proposes a method, called Risk-aware successor features, that exploits the task structure to avoid risks and achieve task generalization between tasks with shared dynamics and different goals. The entropic utility is introduced to represent risk-awareness and specialized to entropic Q-value with bellman update. Risk-aware GPI and GPE are extended from GPI, GPE, and proved its convergence. Experiments on two domains show the method outperforms previous works.

In this paper, a (variance-based) risk-aware transfer method for multi-task RL is proposed.  In the theory part, GPI and GPE (and the notion of successor features) are extended to entropic utility (for the risk-aware transfer).  Based on the results in the theory part (specifically, the GPE/successor features parts), the successor feature for entropic utility is proposed to use for risk-aware setting.  In the experiment part, the proposed method is evaluated in the Four-room and Reacher environments.