Safe Policy Learning from Observations

1 INTRODUCTION . Recent progress in Reinforcement Learning ( RL ) has shown a remarkable success in learning to play games such as Atari from raw sensory input ( Mnih et al. , 2015 ; Hessel et al. , 2017 ) . Still , tabula rasa , RL typically requires a significant amount of interaction with the environment in order to learn . In real-world environments , particularly when a risk factor is involved , an inept policy might be hazardous ( Shalev-Shwartz et al. , 2016 ) . Thus , an appealing approach is to record a dataset of other agents in order to learn a safe initial policy which may later be improved via RL techniques ( Taylor et al. , 2011 ) . Learning from a dataset of experts has been extensively researched in the literature . These learning methods and algorithms are commonly referred to as Learning from Demonstrations ( LfD ) ( Argall et al. , 2009 ) . In this paper , we consider a sibling , less explored problem of learning form a dataset of observations ( LfO ) . We define LfO as a relaxation of LfD , where : ( 1 ) we do no assume a single policy generating the data and ; ( 2 ) the policies are not assumed to be optimal , nor do they cover the entire state space . Practically , LfO is of interest since it is often easier to collect observations than expert demonstrations , for example , using crowdsourcing ( Kurin et al. , 2017 ) . Technically , LfO is fundamentally different from LfD , where the typical task is to clone a single expert policy . The data under the LfO setting is expected to be more diverse than LfD data , which in general can be beneficial for learning . However , it also brings in the challenge of learning from multiple , possibly contradicting , policy trajectories . In this work , we propose to solve the LfO problem with a three phases approach : ( 1 ) imitation ( 2 ) annotation and ; ( 3 ) safe improvement . The imitation phase seeks to learn the average behavior in the dataset . In the annotation part we approximate the value function of the average behavior and in the final safe improvement step ( section 5 ) , we craft a novel algorithm that takes the learned average behavior and its approximated value function and yields an improved policy without generating new trajectories . The improvement step is designed to increase the policy performance with a high confidence in the presence of the value estimation errors that exists in a LfO setup . Our three phases approach which we term Rerouted Behavior Improvement ( RBI ) , provides a robust policy ( without any interaction with the environment ) , that both eliminates the risks in random policy initialization and in addition , can boost the performance of a succeeding RL process . We demonstrate our algorithm both in a Taxi grid-world ( Dietterich , 2000 ) as well as in the Atari domain ( section 6 ) . In the latter , we tackle the challenge of learning from non-experts human players ( Kurin et al. , 2017 ) . We show that our algorithm provides a robust policy , on par with deep RL policies , using only the demonstrations and without any additional interaction with the environment . As a baseline , we compare our approach to two state-of-the-art algorithms : ( 1 ) learning from demonstrations , DQfD ( Hester et al. , 2018 ) and ; ( 2 ) robust policy improvement , PPO ( Schulman et al. , 2017 ) . 2 RELATED WORK . Learning from demonstrations in the context of deep RL in the Atari environment have been studied in Cruz Jr et al . ( 2017 ) , DQfD ( Hester et al. , 2018 ) and recently in Pohlen et al . ( 2018 ) . However , all these methods focus on expert demonstrations and they benchmark their scores after additional trajectory collection with an iterative RL process . Therefore , essentially these methods can be categorized as a RL augmented with expert ’ s supervised data . In contrast , we take a deeper look into the first part of best utilizing the observed data to provide the highest initial performance . Previously , LfO has often been solved solely with the imitation step , e.g . in AlphaGo ( Silver et al. , 2016 ) , where the system learned a policy which mimics the average behavior in a multiple policies dataset . While this provides sound empirical results , we found that one can do better by applying a safe improvement step to boost the policy performance . A greedy improvement method with respect to multiple policies has already been suggested in Barreto et al . ( 2017 ) , yet we found that practically , estimating the value function of each different policy in the dataset is both computationally prohibitive and may also produce large errors since the generated data by each different policy is typically small . In section 4 we suggest a feasible alternative . Instead of learning the value of each different policy , estimate the value of the average behavior . While such estimation is not exact , we show both theoretically and experimentally that it provides a surrogate value function that may be used for policy improvement purposes . There is also a significant research in the field of safe RL ( Garcıa & Fernández , 2015 ) , yet , here there may be multiple accepted definitions of this term ranging from worst-case criterion ( Tamar et al. , 2013 ) to baseline benchmark approach ( Ghavamzadeh et al. , 2016 ) . We continue the line of research of safe RL investigated by Kakade & Langford ( 2002 ) ; Pirotta et al . ( 2013 ) ; Thomas et al . ( 2015 ) but we focus on a dataset composed of unknown policies . Finally , there are two recent well-known works in the context of non-decreasing policy improvement ( also can be categorized as safe improvement ) TRPO and PPO ( Schulman et al. , 2015 ; 2017 ) . We compare our work to these algorithms and show two important advantages : First , our approach can be applied without an additional Neural Network ( NN ) policy optimization step and second we provide theoretical and experimental arguments why both approaches may be deemed unsafe when applied in the context of a LfO setup . 3 PROBLEM FORMULATION . We are dealing with a Markov Decision Process ( MDP ) ( Puterman , 2014 ) where an agent interacts with an environment and tries to maximize a reward . A MDP is defined by the tuple ( S , A , P , R ) , where S is a set of states and A is a set of actions . P : S × A → S is the set of probabilities of switching from a state s to s′ when executing action a , i.e . P ( s′|s , a ) and R is a reward function S × A → R which defines the reward r that the agent gets when applying action a in state s. An agent acts according to a policy π , and its goal is to find a policy that maximizes the expected cumulative discounted reward , also known as the objective function J ( π ) = E [ ∑∞ k=0 γ krk ∣∣s0 , π ] where γ < 1 is a discount factor , k is a time index and s0 is an initial state . We assume that all policies belong to the Markovian randomized set ΠMR s.t . π ∈ ΠMR is a probability distribution over A given a state s , i.e . π ( a|s ) .1 For convenient and when appropriate , we may simply write πi to denote π ( ai|s ) ( omitting the state ’ s dependency ) . In the paper we will discuss two important distance measures between policies . The first is the Total Variation ( TV ) δ ( π , π′ ) = 12 ∑ ai |πi − π′i| and the second is the KL divergence DKL ( π||π′ ) = − ∑ ai πi log π′i πi . 1Note that humans ’ policies can generally be considered as part of the history randomized set ΠHR where π ∈ ΠHR is a probability function overA given the states and actions history . In the appendix we explain how we circumvented this hurdle in the Atari dataset . These measures are often used to constrain the updates of a learned policy in an iterative policy improvement RL algorithm ( Schulman et al. , 2015 ; 2017 ) . For a given policy , the state ’ s value is the expected cumulative reward starting at this state , V π ( s ) = E [ ∑∞ k=0 γ krk ∣∣s , π ] . Similarly , theQ-value functionQπ ( s , a ) is the value of taking action a in state s and then immediately following with policy π . The advantage Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) is the gain of taking action a in state s over the average value ( note that ∑ a∈A π ( a|s ) Aπ ( s , a ) = 0 ) . We denote by P ( s0 k−→ s|π ) the probability of switching from state s0 to state s in k steps with a policy π . We define the LfO problem as learning a policy π solely by observing a finite set of trajectories of other behavior policies without interacting with the environment . Formally , we are given a dataset D of trajectories executed by N different players each with a presumably different policy . Players are denoted by pi , i = 1 , ... , N , and their corresponding policies are βi , with value and Q-value functions V i , Qi respectively . D is indexed as { xj } |D|j=0 , where the cardinality of the dataset is denoted by |D| and each record is the tuple xj = ( sj , aj , rj , tj , ij ) s.t . tj is a termination signal and ij is the player ’ s index . D may also be partitioned to D = { Di|i = 1 , .. , N } , representing the different players ’ records . The paper is accompanied with a running example , based on the Taxi grid-world domain ( Dietterich , 2000 ) , In the Taxi world , the driver ’ s task is to pickup and drop a passenger in predefined locations with minimal number of steps ( See figure 1 ) . For this example , we synthetically generated policies of the form βi = αi ( s ) πrand + ( 1− αi ( s ) ) π∗ , where π∗ is the optimal policy and αi ( s ) is a different mixing parameter for each different policy . Generally we divide the state space into two complementary and equal-size sets S∗i ∪S̄∗i = S , |S∗i |= |S̄∗i | . Where αi ( s ) = 0 for s ∈ S∗i and αi ( s ) = 0.75 for s ∈ S̄∗i . In the next sections , we will use different selections of S∗i to generate different types of datasets . For example , randomly pick half of the states to form S∗i is termed in the paper as a random selection ( see also appendix ) .

Summary: The paper looks at learning a policy from multiple demonstrators which should also be safely improved by an reinforcement learning signal. They define the policy as a mixture of policies from the single demonstrators. The paper gives a new way to estimate the value function of each policy where the overall policy is defined as mixture of the single policies. The paper subsequently looks at the standard error of the value function estimation and then define the policy improvement step in the presence of value estimation error. The resulting reroute constraint for the policy improvement step is evaluated on the taxi toy task as well as on 4 different atari domains.