Risk-Aware Transfer in Reinforcement Learning using Successor Features

1 Introduction . Reinforcement learning ( RL ) is a general framework for solving sequential decision-making problems , in which an agent interacts with an environment and receives continuous feedback in the form of rewards . However , many classical algorithms in RL do not explicitly address the need for safety , making them unreliable and difficult to deploy in some real-world applications [ 10 ] . One reason for this is the relative sample inefficiency of model-free RL algorithms , which often require millions of costly or dangerous interactions with the environment or fail to converge altogether [ 44 , 46 ] . Transfer learning addresses these problems by incorporating prior knowledge or skills [ 23 , 41 ] . Despite this , using the expected return as a measure of optimality could still lead to undesirable behavior such as excessive risk-taking , since low-probability catastrophic outcomes with negative reward and high variance could be underrepresented [ 29 ] . For this reason , risk-awareness is becoming an important aspect in the design of practical RL [ 14 ] . Thus , an ultimate goal of developing reliable systems should be to ensure that they are both sample efficient and risk-aware . ∗Affiliate to Vector Institute , Toronto , Canada . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We take a step in this direction by studying the problem of risk-aware policy transfer between tasks with different goals . A powerful way to tackle this problem in the risk-neutral setting is the GPI/GPE framework , of which successor features ( SF ) are a notable example [ 2 ] . Here , generalized policy improvement ( GPI ) provides a theoretical framework for transferring policies with monotone improvement guarantees , while generalized policy evaluation ( GPE ) facilitates the efficient evaluation of policies on novel tasks and is a key component in satisfying the assumptions of GPI in practice . Together , GPI/GPE provide strong transfer benefits in novel task instances even before any direct interaction with them has taken place , a phenomenon we call task generalization . The key to the superb generalization of GPI/GPE lies in their ability to directly exploit the structure of the task space , taking advantage of subtle differences and commonalities between task goals to transfer skills seamlessly in a composable manner . This property could be an effective way of tackling problems in offline RL [ 24 ] , such as the transfer of skills learned in a simulator to a real-world environment . However in many cases , such as helicopter flight control [ 16 ] , making one wrong decision could lead to catastrophic outcomes . Hence , being risk-aware could offer one way to avoid worst-case outcomes when transferring skills in real-world settings . Contributions . We contribute a novel successor feature framework for transferring policies with the goal of maximizing the entropic utility of return in MDPs ( Section 2.2 ) . Intuitively , the entropic utility encourages agents to follow policies with predictable and controllable returns characterized by low variance , thus providing a natural way to incorporate risk-awareness . Furthermore , while our theoretical framework could be extended to other classes of utility functions , the entropic utility has many favorable mathematical properties [ 13 , 22 ] that we exploit directly in this work to achieve optimal transfer ( Lemma 1 , Theorem 1 and 2 ) . We also derive a form of risk-aware GPE based on the mean-variance approximation , in which the sufficient statistics of the return distribution can be computed directly ( Section 3.3 ) or by leveraging recent developments in distributional RL [ 6 ] . Our resulting approach , which we call Risk-Aware Successor Features ( RaSF ) , is able to exploit the task structure to achieve task generalization with respect to novel goal instances as well as levels of risk aversion , where emphasis is placed on avoiding high volatility of returns . Our approach is also complementary to other advances in successor features , including feature learning [ 3 ] , universal approximation [ 8 ] , exploration [ 21 ] , and non-stationary reward preferences [ 4 ] . Empirical evaluations on discrete navigation and continuous robot control domains ( Section 4 ) demonstrate the ability of RaSFs to better manage the trade-off between return and risk and avoid catastrophic outcomes , while providing excellent generalization on novel tasks in the same domain . Related Work . The entropic and mean-variance objectives are popular ways of incorporating risk-awareness in RL [ 7 , 11 , 20 , 27 , 28 , 30 , 36 , 39 , 45 ] . However , transferring learned skills between tasks while taking risk into account is a difficult problem . One way to implement risk-aware transfer is to learn a critic [ 38 ] or teacher [ 43 ] that can guide an agent toward safer behaviors on future tasks . The risk-aware transfer of a policy from a simulator to a real-world setting has also been studied in the area of robotics [ 17 ] . Another approach for reusing policies is the probabilistic policy reuse of García and Fernández [ 15 ] , but requires strong assumptions on the task space . Hierarchical RL ( HRL ) is another related approach that relies on hierarchical abstractions , enabling an agent to decompose tasks into a hierarchy of sub-tasks , and facilitating the transfer of temporally-extended skills from sub-tasks to the parent task . The CISR approach of Mankowitz et al . [ 26 ] is the first to investigate safety explicitly within HRL , followed up by work on safe options [ 18 , 19 , 25 ] . However , none of the existing work takes advantage of the compositional structure of task rewards to transfer skills while optimizing the variance-adjusted return , which is the problem we tackle in this paper ( see Table 1 ) . 2 Preliminaries . 2.1 Markov Decision Process . Sequential decision-making in this paper follows the Markov decision process ( MDP ) , defined as a four-tuple 〈S , A , r , P 〉 : S is a set of states ; A is a finite set of actions ; r : S × A × S → R is a bounded reward function , where r ( s , a , s′ ) is the immediate reward received upon transitioning to state s′ after taking action a in state s ; and P : S × A × S → [ 0 , ∞ ) is the transition function for state dynamics , where P ( s′|s , a ) is the probability of transitioning to state s′ immediately after taking action a in state s. In the episodic MDP setting , decisions are made over a horizon T = { 0 , 1 , . . . T } where T ∈ N. We define a stochastic Markov policy as a mapping π : S × T → P ( A ) , where P ( A ) denotes the set of all probability distributions over A . Similarly , a deterministic Markov policy is a mapping π : S × T → A . In the risk-neutral setting , the goal is to find a policy π that maximizes the expected sum of future rewards after initially taking action a in state s , Qπh ( s , a ) = Est+1∼P ( ·|st , at ) [ T∑ t=h r ( st , at , st+1 ) ∣∣∣ sh = s , ah = a , at ∼ πt ( st ) ] . In this case , it is possible to show that a deterministic Markov policy π∗ is optimal [ 33 ] . The theoretical framework in this paper also allows for time-dependent reward or transition functions . 2.2 Entropic Utility Maximization . We incorporate risk-awareness into the decision-making by maximizing the entropic utility Uβ of the cumulative reward , defined for a fixed β ∈ R as Uβ [ R ] = 1 β logE [ eβR ] , ( 1 ) for real-valued random variables R on a bounded support Ω ⊂ R. An important property of the entropic utility is the Taylor expansion Uβ [ R ] = E [ R ] + β2Var [ R ] +O ( β 2 ) . Interpreting the risk as return variance , β can now be interpreted as the risk aversion of the agent : choosing β < 0 ( β > 0 ) leads to risk-averse ( risk-seeking ) behavior , while β = 0 is risk-neutral , e.g . U0 [ R ] = E [ R ] . Specializing ( 1 ) to the MDP setting , the goal is to maximize Qπh , β ( s , a ) = Uβ [ T∑ t=h r ( st , πt ( st ) , st+1 ) ] ( 2 ) over all policies starting from sh = s and ah = a . As in the risk-neutral setting , it is possible to show that a deterministic Markov policy is optimal [ 5 ] . Furthermore , Qπh , β can be computed iteratively through time using the Bellman equation [ 9 , 32 ] : Qπh , β ( s , a ) = Uβ [ r ( s , a , s′ ) +Qπh+1 , β ( s′ , πh+1 ( s′ ) ) ] = 1 β logEs′∼P ( ·|s , a ) [ exp { β ( r ( s , a , s′ ) +Qπh+1 , β ( s′ , πh+1 ( s′ ) ) ) } ] , ( 3 ) starting with QπT+1 , β ( s , a ) = 0 . In fact , ( 1 ) is the only utility function that has this equivalence , and other key properties ( Lemma 1 ) , while also satisfying time consistency that ensures the learned risk-aware behaviors remain consistent across time [ 22 ] . In this paper , we use ( 3 ) to establish a general GPI framework for risk-aware transfer learning with provable guarantees , and leverage approximations of ( 2 ) to learn portable policy representations . In reinforcement learning , the Bellman equation is not applied directly since it suffers from the curse of dimensionality when S is high-dimensional or continuous , and since neither the dynamics nor the reward function are often known . Instead , the agent interacts with the environment using a stochastic exploration policy πe , collects trajectories { ( st , at , st+1 , rt+1 ) } T−1t=0 , and updates Qh , β ( st , at ) via sample approximations Ûβ ≈ Uβ [ 36 ] . Our goal is to ameliorate the relative sample-inefficiency of RL through transfer learning that we aim to generalize to the risk-aware setting . 2.3 Transfer Learning . We now formalize the general transfer learning problem . Let M be the set of all MDPs with shared transition function P but different ( bounded ) reward functions . A fixed set of source tasks M1 , . . .Mn ∈ M is instantiated , and their corresponding optimal policies π1 , . . . πn are estimated . Our main goal is to transfer these resulting source policies to a new target task Mn+1 ∈ M , to obtain a policy π∗n+1 whose utility is better than one learned from scratch using only a fixed number of samples from Mn+1 . We refer to this outcome as positive transfer . As discussed earlier , a standard way to implement transfer learning is the GPI/GPE framework of Barreto et al . [ 2 ] . The core mechanism that enables positive transfer in the risk-neutral setting is called generalized policy improvement ( GPI ) . Specifically , the set of source policies π1 , . . . πn are evaluated on the target task Mn+1 to obtain corresponding values Qπ1n+1 , . . . Q πn n+1 . Given a mechanism that can perform this policy evaluation step efficiently with some small error ε — namely successor features discussed and extended in Section 3.3 — an agent then selects actions in a greedy manner by following policy π ( s ) ∈ argmaxa maxj=1 ... n Q πj n+1 ( s , a ) in state s. The policy π corresponds to a strict policy improvement operator , and thus fulfills our requirements for positive transfer .

Summary: This paper considers safety in transfer RL, and proposes a method, called Risk-aware successor features, that exploits the task structure to avoid risks and achieve task generalization between tasks with shared dynamics and different goals. The entropic utility is introduced to represent risk-awareness and specialized to entropic Q-value with bellman update. Risk-aware GPI and GPE are extended from GPI, GPE, and proved its convergence. Experiments on two domains show the method outperforms previous works.