Interpretable Meta-Reinforcement Learning with Actor-Critic Method

1 INTRODUCTION . Reinforcement learning problems have been studied for a long time and there are many impressive works that achieved human-level control in real world tasks ( Mnih et al. , 2013 ; Silver et al. , 2017 ; Vinyals et al. , 2019 ; Schrittwieser et al. , 2019 ) . These agents are trained separately on each task and may require huge sampled data and millions of trails . However , in a many real world tasks , the cost of sampling data is not negligible , thus we can not give agent a large number of trails in environment . In contrast , human can laverage past experiences and learn new tasks quickly in few trails , which is very efficient . Many tasks in fact share similar structures that can be extracted as prior knowledge , e.g. , shooting games aims to eliminate enemies with weapons in different environments , which can help agent generalize quickly through different tasks . Meta-learn ( Thrun & Pratt , 2012 ) reinforcement learning tasks can be a suitable chioce . Meta-reinforcement learning ( meta-RL ) aims to learn a policy that can adapt to the unknown environment within few interactions with environment . Meta-policy can be seen as a policy that can derive new a policy maximizes the performance in the new environment . Gradient-based algorithms in meta-RL ( Finn et al. , 2017 ; Stadie et al. , 2018 ; Rothfuss et al. , 2018 ; Liu et al. , 2019 ) showed that a meta-policy can be obtained by reinforcement learning a policy adapted by few reinforcement learning steps . The experiment results suggests that gradient-based methods can learn to sample and utilize sampled data in some extent . Nevertheless , the learning style and learned meta-policy are still far from human . Human learns a new task by interacting with the task sequentially and efficiently . With the obtaining of environment data , human gradually understanding where to sampling data and how to utilize the sampled data to adjust the policy , while gradient-based algorithms use parallel sampling neglecting the relations between data . Sampling independently is not data-efficient , usually needs a number of stochastic trajectories to do plicy adaptation . This causes the agent relying on the stochasticity to sample and only learns how to utilize data . Inspired by the human behavior , we propose a K-shot meta-RL problem that constrains on the data amount accessed by agent , e.g. , adapting policy within only two trails . Low resource environment simulates the real world tasks that have high costs on data obtaining , therefore , requires agent to learn a stable strategy to explore environment . To address the K-shot problem , we also propose a contextual gradient-based algorithm using actor-critic method . The adptation step uses a trail buffer D to store all the transitions in K-shot sampling and optimizes expected value for the states in D. The meta-learning step optimizes the expected return performed by adapted policy while learning the value functions and context encoder using soft actor-critic ( Haarnoja et al. , 2018 ) objectives . We learn the policy with reparameterized objective that derives an unbiased meta-gradient estimation and reduces the estimation variance for Q-value . Our contribution can be summarized as follows : • We reformulate and propose the K-shot meta-RL problem to simulate the real world environment . • We propose a new gradient-based objective to address the K-shot problem . • We introduce context based policy and value functions to perform efficient data sampling . • We use actor-critic method to reduce the variance and bias of estimation in Q-value and meta-gradien . 2 RELATED WORK . Meta-reinforce learning algorithms mainly have three different categories : gradient-based motheds ( Finn et al. , 2017 ; Stadie et al. , 2018 ; Rothfuss et al. , 2018 ; Liu et al. , 2019 ; Nichol et al. , 2018 ) , recurrent meta-learners ( Wang et al. , 2016 ; Duan et al. , 2016 ) , multi-task learners ( Fakoor et al. , 2019 ; Rakelly et al. , 2019 ) . Gradient-based algorithms like MAML ( Finn et al. , 2017 ) optimizing the policy updated by one step reinforcement learning , aiming at learning a good initialization of the policy weights . E-MAML ( Stadie et al. , 2018 ) considered the impact that the data obtained by meta-policy can influence the adapted policy ’ s performance and assigned credit for meta-policy . While ProMP ( Rothfuss et al. , 2018 ) modified the adaptation gradient estimator to be low variance on second-order gradient . Recurrent meta-learners ( Wang et al. , 2016 ; Duan et al. , 2016 ) use RNN as a meta-learner that can learn new task from environment data while exploring . The RNN learners are optimized with sequentially performed episodes end-to-end , which is more similar to the learning process of human and more interpretable in meta-policy . Multi-task learners ( Fakoor et al. , 2019 ; Rakelly et al. , 2019 ) aim at learning multi-task objective to solve meta-learning problems . They argue that meta-learning can be done by explicitly resuing the learned features through context variable . MQL ( Fakoor et al. , 2019 ) can even perform well without adaptation . PEARL ( Rakelly et al. , 2019 ) constructs context encoder to infer the latent task variable and also learns a multi-task objective . The trained policy can perform structured exploration by inferring the task while interacting with environment.Our approach is related closely to the gradient-based researches which also tries to reduce the estimation variance and bias of the second-order gradient , however , we estimate the second-order gardient with value functions , and we still want perform structured exploration in data expensive environments . 3 BACKGROUND . This section focuses on the problem definition and notation of reinforcement learning and metareinforcement learning problems . 3.1 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) problems aim to maximize the expectation of episode returns Eτ∼P ( τ |θ ) [ R ( τ ) ] = Eτ∼P ( τ |θ ) [ ∑ t γtr ( st , at ) ] ( 1 ) with single task and agent , where τ = { s0 , a0 , r0 , . . . } is the trajectory performed by the agent , s0 ∼ ρ0 is the initial state , at ∼ πθ ( at|st ) is the action sampled from the policy π that parameterized by θ , st+1 ∼ P ( st+1|at , st ) is the state at timestep t , and P ( st+1|at , st ) is the transition probability . The problem can be represented by a Markov Desicion Process ( MDP ) with tuple M = ( S , A , P , R , ρ0 , γ , H ) , where S ⊆ Rn is the set of states , A ⊆ Rm is the set of actions , P ( s′|s , a ) ∈ R+ is the system transition probability , R ( s , a ) ∈ R is the reward function of the task , and H is the horizon . Optimizing ( 1 ) usually uses gradient descent and the gradient is estimated using vanilla policy gradient ( VPG ) estimator ( Williams , 1992 ) ∇θEτ∼P ( τ |θ ) [ R ( τ ) ] = Eτ∼P ( τ |θ ) [ ∇θ log π ( τ ) R ( τ ) ] ≈ 1 N ∑ i ∑ t ∇θ log πθ ( ai , t|si , t ) ( H∑ t′=t R ( si , t′ , ai , t′ ) ) ( 2 ) 3.2 GRADIENT-BASED META-REINFORCEMENT LEARNING . Meta-reinforcement learning ( meta-RL ) aims to learn a fast adaptation procedure that can leverage the learned prior knowledge from training tasks and adapt to new tasks with few steps . A task T in meta-RL can also be defined by an MDPMT = ( S , A , PT , RT , ρ0 , γ , H ) . The task is drawn from a distribution T ∼ P ( T ) , for simplicity , we only consider tasks with different reward functions or system transitions but the same state and action space . Gradient-based meta-RL algorithms ( Finn et al. , 2017 ; Stadie et al. , 2018 ) are mainly based on the basic meta-objective ( Rothfuss et al. , 2018 ) J ( θ ) = ET∼P ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] , θ′ = U ( θ , T ) = θ + α∇θEτ∼PT ( τ |θ ) [ R ( τ ) ] , ( 3 ) where θ is the weights of meta-policy , and θ′ is the adapted weights after one step gradient descent . The meta-objective J ( θ ) optimizes the expectation of episode return sampled from the adapted policy πθ′ . The meta-gradient can be written as ∇θJ ( θ ) = ET∼P ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ ∇θ′ logPT ( τ ′|θ′ ) R ( τ ′ ) ∇θθ′ ] ] ∇θθ′ = I + α∇2θEτ∼PT ( τ |θ ) [ R ( τ ) ] ( 4 ) 4 METHOD . 4.1 REFORMULATE META-REINFORCEMENT LEARNING PROBLEM . Different tasks have different features in MDP , a task can be inferred from few important states and transitions in the environment , e.g. , different friction coefficients on floor , different rewards for the same state and action , or some states only exists in certain environments . We name these states and transitions the feature points of the environment . Humans usually learn a task sequentially and efficiently since they can easily recognize the feature points in an environment . The exploration policy of a human changes significantly after obtaining data from the envioronment , thus they can decide where to explore and learn a task quickly . However , as formula ( 3 ) , fast adaptation U ( θ , T ) usually refers to few gradient descent steps in initial weights θ , and unlike humans , the updating is performed in a batched style as normal reinforcement learning . Batched sampling usually contains a large number of trajectories in parallel , which can be inefficient for inferring the task . E-MAML ( Stadie et al. , 2018 ) also tried to improve the sampling efficiency of meta-policy by accounting for the fact that samples drawn from meta-policy will impact the adapted policy . Inspired by the learning procedure of human , we reformulate the meta-RL problem as K-shot meta-reinforcement learning . Definition . Given a task T ∼ P ( T ) , the agent samples data in trail phase and perform good policy in test phase . In trail phase , the agent can only sequentially sample K trajectories in total to adjust its policy , with each trajectory of H length . In test phase , the agent is required to perform only one trajectory and make the return as high as possible . K-shot meta-RL problem defined above constrains the amount of data that can be accessed by agent , and is more similar to the real world meta-RL problem , e.g. , super mario maker . In K-shot setting , meta-policy can still be updated using U ( θ , T ) with batched trajectories , since they can be seen as sampled independently in sequence . However , the variance of the gradient estimation grows as K descends , which means the performance becomes more unstable . To optimize the problem , we propose a new meta-objective JK−shot ( θ ) = ET∼P ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] , θ′ = U ( θ , D ) = θ + α∇θEs∼D [ V π ( s|c ) ] ( 5 ) for the K-shot setting . Here D is the state buffer sampled by meta-policy in trail phase , and V π ( s|c ) is the expected return of policy π at state s under context c ( see 4.2 for details ) . The state buffer D contains K ∗H states as described in definition , which means the agent can only use few states to update its policy . Due to the constraint on availble environment information , the agent is encouraged to learn to explore more important states that can help performing well in test phase .

Summary: Authors introduce a new meta-RL algorithm based on SAC. It uses a context variable $c$ that they condition the Q-function on and the adaptation mechanism which is based on the values of the value function (ie. $\mathbb{E_a} Q(\dot, a)$) instead of the true returns. Authors claim their method reduces variance and bias of the meta-gradient estimation, is closer to human learning, encourages the agent to learn to explore, is more data-efficient in test-time and has competitive performance among gradient-based algorithms.