Risk-Aware Transfer in Reinforcement Learning using Successor Features

1 Introduction . Reinforcement learning ( RL ) is a general framework for solving sequential decision-making problems , in which an agent interacts with an environment and receives continuous feedback in the form of rewards . However , many classical algorithms in RL do not explicitly address the need for safety , making them unreliable and difficult to deploy in some real-world applications [ 10 ] . One reason for this is the relative sample inefficiency of model-free RL algorithms , which often require millions of costly or dangerous interactions with the environment or fail to converge altogether [ 44 , 46 ] . Transfer learning addresses these problems by incorporating prior knowledge or skills [ 23 , 41 ] . Despite this , using the expected return as a measure of optimality could still lead to undesirable behavior such as excessive risk-taking , since low-probability catastrophic outcomes with negative reward and high variance could be underrepresented [ 29 ] . For this reason , risk-awareness is becoming an important aspect in the design of practical RL [ 14 ] . Thus , an ultimate goal of developing reliable systems should be to ensure that they are both sample efficient and risk-aware . ∗Affiliate to Vector Institute , Toronto , Canada . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We take a step in this direction by studying the problem of risk-aware policy transfer between tasks with different goals . A powerful way to tackle this problem in the risk-neutral setting is the GPI/GPE framework , of which successor features ( SF ) are a notable example [ 2 ] . Here , generalized policy improvement ( GPI ) provides a theoretical framework for transferring policies with monotone improvement guarantees , while generalized policy evaluation ( GPE ) facilitates the efficient evaluation of policies on novel tasks and is a key component in satisfying the assumptions of GPI in practice . Together , GPI/GPE provide strong transfer benefits in novel task instances even before any direct interaction with them has taken place , a phenomenon we call task generalization . The key to the superb generalization of GPI/GPE lies in their ability to directly exploit the structure of the task space , taking advantage of subtle differences and commonalities between task goals to transfer skills seamlessly in a composable manner . This property could be an effective way of tackling problems in offline RL [ 24 ] , such as the transfer of skills learned in a simulator to a real-world environment . However in many cases , such as helicopter flight control [ 16 ] , making one wrong decision could lead to catastrophic outcomes . Hence , being risk-aware could offer one way to avoid worst-case outcomes when transferring skills in real-world settings . Contributions . We contribute a novel successor feature framework for transferring policies with the goal of maximizing the entropic utility of return in MDPs ( Section 2.2 ) . Intuitively , the entropic utility encourages agents to follow policies with predictable and controllable returns characterized by low variance , thus providing a natural way to incorporate risk-awareness . Furthermore , while our theoretical framework could be extended to other classes of utility functions , the entropic utility has many favorable mathematical properties [ 13 , 22 ] that we exploit directly in this work to achieve optimal transfer ( Lemma 1 , Theorem 1 and 2 ) . We also derive a form of risk-aware GPE based on the mean-variance approximation , in which the sufficient statistics of the return distribution can be computed directly ( Section 3.3 ) or by leveraging recent developments in distributional RL [ 6 ] . Our resulting approach , which we call Risk-Aware Successor Features ( RaSF ) , is able to exploit the task structure to achieve task generalization with respect to novel goal instances as well as levels of risk aversion , where emphasis is placed on avoiding high volatility of returns . Our approach is also complementary to other advances in successor features , including feature learning [ 3 ] , universal approximation [ 8 ] , exploration [ 21 ] , and non-stationary reward preferences [ 4 ] . Empirical evaluations on discrete navigation and continuous robot control domains ( Section 4 ) demonstrate the ability of RaSFs to better manage the trade-off between return and risk and avoid catastrophic outcomes , while providing excellent generalization on novel tasks in the same domain . Related Work . The entropic and mean-variance objectives are popular ways of incorporating risk-awareness in RL [ 7 , 11 , 20 , 27 , 28 , 30 , 36 , 39 , 45 ] . However , transferring learned skills between tasks while taking risk into account is a difficult problem . One way to implement risk-aware transfer is to learn a critic [ 38 ] or teacher [ 43 ] that can guide an agent toward safer behaviors on future tasks . The risk-aware transfer of a policy from a simulator to a real-world setting has also been studied in the area of robotics [ 17 ] . Another approach for reusing policies is the probabilistic policy reuse of García and Fernández [ 15 ] , but requires strong assumptions on the task space . Hierarchical RL ( HRL ) is another related approach that relies on hierarchical abstractions , enabling an agent to decompose tasks into a hierarchy of sub-tasks , and facilitating the transfer of temporally-extended skills from sub-tasks to the parent task . The CISR approach of Mankowitz et al . [ 26 ] is the first to investigate safety explicitly within HRL , followed up by work on safe options [ 18 , 19 , 25 ] . However , none of the existing work takes advantage of the compositional structure of task rewards to transfer skills while optimizing the variance-adjusted return , which is the problem we tackle in this paper ( see Table 1 ) . 2 Preliminaries . 2.1 Markov Decision Process . Sequential decision-making in this paper follows the Markov decision process ( MDP ) , defined as a four-tuple 〈S , A , r , P 〉 : S is a set of states ; A is a finite set of actions ; r : S × A × S → R is a bounded reward function , where r ( s , a , s′ ) is the immediate reward received upon transitioning to state s′ after taking action a in state s ; and P : S × A × S → [ 0 , ∞ ) is the transition function for state dynamics , where P ( s′|s , a ) is the probability of transitioning to state s′ immediately after taking action a in state s. In the episodic MDP setting , decisions are made over a horizon T = { 0 , 1 , . . . T } where T ∈ N. We define a stochastic Markov policy as a mapping π : S × T → P ( A ) , where P ( A ) denotes the set of all probability distributions over A . Similarly , a deterministic Markov policy is a mapping π : S × T → A . In the risk-neutral setting , the goal is to find a policy π that maximizes the expected sum of future rewards after initially taking action a in state s , Qπh ( s , a ) = Est+1∼P ( ·|st , at ) [ T∑ t=h r ( st , at , st+1 ) ∣∣∣ sh = s , ah = a , at ∼ πt ( st ) ] . In this case , it is possible to show that a deterministic Markov policy π∗ is optimal [ 33 ] . The theoretical framework in this paper also allows for time-dependent reward or transition functions . 2.2 Entropic Utility Maximization . We incorporate risk-awareness into the decision-making by maximizing the entropic utility Uβ of the cumulative reward , defined for a fixed β ∈ R as Uβ [ R ] = 1 β logE [ eβR ] , ( 1 ) for real-valued random variables R on a bounded support Ω ⊂ R. An important property of the entropic utility is the Taylor expansion Uβ [ R ] = E [ R ] + β2Var [ R ] +O ( β 2 ) . Interpreting the risk as return variance , β can now be interpreted as the risk aversion of the agent : choosing β < 0 ( β > 0 ) leads to risk-averse ( risk-seeking ) behavior , while β = 0 is risk-neutral , e.g . U0 [ R ] = E [ R ] . Specializing ( 1 ) to the MDP setting , the goal is to maximize Qπh , β ( s , a ) = Uβ [ T∑ t=h r ( st , πt ( st ) , st+1 ) ] ( 2 ) over all policies starting from sh = s and ah = a . As in the risk-neutral setting , it is possible to show that a deterministic Markov policy is optimal [ 5 ] . Furthermore , Qπh , β can be computed iteratively through time using the Bellman equation [ 9 , 32 ] : Qπh , β ( s , a ) = Uβ [ r ( s , a , s′ ) +Qπh+1 , β ( s′ , πh+1 ( s′ ) ) ] = 1 β logEs′∼P ( ·|s , a ) [ exp { β ( r ( s , a , s′ ) +Qπh+1 , β ( s′ , πh+1 ( s′ ) ) ) } ] , ( 3 ) starting with QπT+1 , β ( s , a ) = 0 . In fact , ( 1 ) is the only utility function that has this equivalence , and other key properties ( Lemma 1 ) , while also satisfying time consistency that ensures the learned risk-aware behaviors remain consistent across time [ 22 ] . In this paper , we use ( 3 ) to establish a general GPI framework for risk-aware transfer learning with provable guarantees , and leverage approximations of ( 2 ) to learn portable policy representations . In reinforcement learning , the Bellman equation is not applied directly since it suffers from the curse of dimensionality when S is high-dimensional or continuous , and since neither the dynamics nor the reward function are often known . Instead , the agent interacts with the environment using a stochastic exploration policy πe , collects trajectories { ( st , at , st+1 , rt+1 ) } T−1t=0 , and updates Qh , β ( st , at ) via sample approximations Ûβ ≈ Uβ [ 36 ] . Our goal is to ameliorate the relative sample-inefficiency of RL through transfer learning that we aim to generalize to the risk-aware setting . 2.3 Transfer Learning . We now formalize the general transfer learning problem . Let M be the set of all MDPs with shared transition function P but different ( bounded ) reward functions . A fixed set of source tasks M1 , . . .Mn ∈ M is instantiated , and their corresponding optimal policies π1 , . . . πn are estimated . Our main goal is to transfer these resulting source policies to a new target task Mn+1 ∈ M , to obtain a policy π∗n+1 whose utility is better than one learned from scratch using only a fixed number of samples from Mn+1 . We refer to this outcome as positive transfer . As discussed earlier , a standard way to implement transfer learning is the GPI/GPE framework of Barreto et al . [ 2 ] . The core mechanism that enables positive transfer in the risk-neutral setting is called generalized policy improvement ( GPI ) . Specifically , the set of source policies π1 , . . . πn are evaluated on the target task Mn+1 to obtain corresponding values Qπ1n+1 , . . . Q πn n+1 . Given a mechanism that can perform this policy evaluation step efficiently with some small error ε — namely successor features discussed and extended in Section 3.3 — an agent then selects actions in a greedy manner by following policy π ( s ) ∈ argmaxa maxj=1 ... n Q πj n+1 ( s , a ) in state s. The policy π corresponds to a strict policy improvement operator , and thus fulfills our requirements for positive transfer .

Summary: In this paper, a (variance-based) risk-aware transfer method for multi-task RL is proposed.  In the theory part, GPI and GPE (and the notion of successor features) are extended to entropic utility (for the risk-aware transfer).  Based on the results in the theory part (specifically, the GPE/successor features parts), the successor feature for entropic utility is proposed to use for risk-aware setting.  In the experiment part, the proposed method is evaluated in the Four-room and Reacher environments.