Online Meta-Critic Learning for Off-Policy Actor-Critic Methods

1 INTRODUCTION . Off-policy Actor-Critic ( Off-PAC ) methods are currently central in deep reinforcement learning ( RL ) research due to their greater sample efficiency compared to on-policy alternatives . On-policy requires new trajectories to be collected for each update to the policy , and is expensive as the number of gradient steps and samples per step increases with task-complexity even for contemporary TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) and A3C ( Mnih et al. , 2016 ) algorithms . Off-policy methods , such as DDPG ( Lillicrap et al. , 2016 ) , TD3 ( Fujimoto et al. , 2018 ) and SAC ( Haarnoja et al. , 2018b ) achieve greater sample efficiency due to their ability to learn from randomly sampled historical transitions without a time sequence requirement , thus making better use of past experience . Their critic estimates the action-value ( Q-value ) function using a differentiable function approximator , and the actor updates its policy parameters in the direction of the approximate action-value gradient . Briefly , the critic provides a loss to guide the actor , and is trained in turn to estimate the environmental action-value under the current policy via temporal-difference learning ( Sutton et al. , 2009 ) . In all these cases the learning algorithm itself is hand-crafted and fixed . Recently meta-learning , or “ learning-to-learn ” has become topical as a paradigm to accelerate RL by learning aspects of the learning strategy , for example , through learning fast adaptation strategies ( Finn et al. , 2017 ; Rakelly et al. , 2019 ; Riemer et al. , 2019 ) , exploration strategies ( Gupta et al. , 2018 ) , optimization strategies ( Duan et al. , 2016b ) , losses ( Houthooft et al. , 2018 ) , hyperparameters ( Xu et al. , 2018 ; Veeriah et al. , 2019 ) , and intrinsic rewards ( Zheng et al. , 2018 ) . However , the majority of these works perform meta-learning on a family of tasks or environments and amortize this huge cost by deploying the trained strategy for fast learning on a new task . In this paper we introduce a novel meta-critic network to enhance existing Off-PAC learning frameworks . The meta-critic is used alongside the vanilla critic to provide a loss to guide the actor ’ s learning . However compared to the vanilla critic , the meta-critic is explicitly ( meta ) -trained to accelerate the learning process rather than merely estimate the action-value function . Overall , the actor is trained by gradients provided by both critic and meta-critic losses , the critic is trained by temporal-difference as usual , and the meta-critic is trained to generate maximum learning performance improvements in the actor . In our framework , both the critic and meta-critic use randomly sampled off-policy transitions for efficient and effective Off-PAC learning , providing superior sam- ple efficiency compared to existing on-policy meta-learners . Furthermore , we demonstrate that our meta-critic can be successfully learned online within a single task . This is in contrast to the currently widely used meta-learning research paradigm – where entire task families are required to provide enough data for meta-learning , and to provide new tasks to amortize the huge cost of meta-learning . Essentially our framework meta-learns an auxiliary loss function , which can be seen as an intrinsic motivation towards optimum learning progress ( Oudeyer & Kaplan , 2009 ) . As analogously observed in several recent meta-learning studies ( Franceschi et al. , 2018 ) , our loss-learning can be formalized as a bi-level optimization problem with the upper level being meta-critic learning , and lower level being conventional learning . We solve this joint optimization by iteratively updating the metacritic and base learner online while solving a single task . Our strategy is thus related to the metaloss learning in EPG ( Houthooft et al. , 2018 ) , but learned online rather than offline , and integrated with Off-PAC rather than their on-policy policy-gradient learning . The most related prior work is LIRPG ( Zheng et al. , 2018 ) , which meta-learns an intrinsic reward online . However , their intrinsic reward just provides a helpful scalar offset to the environmental reward for on-policy trajectory optimization via policy-gradient ( Sutton et al. , 2000 ) . In contrast our meta-critic provides a loss for direct actor optimization just based on sampled transitions , and thus achieves dramatically better sample efficiency than LIRPG reward learning in practice . We evaluate our framework on several contemporary continuous control benchmarks and demonstrate that online meta-critic learning can be integrated with and improve a selection of contemporary Off-PAC algorithms including DDPG , TD3 and SAC . 2 BACKGROUND AND RELATED WORK . Policy-Gradient ( PG ) Methods . On-policy methods usually update actor parameters in the direction of greater cumulative reward . However , on-policy methods need to interact with the environment in a sequential manner to accumulate rewards and the expected reward is generally not differentiable due to environment dynamics . Even exploiting tricks like importance sampling and improved application of A2C ( Zheng et al. , 2018 ) , the use of full trajectories is less effective than off-policy transitions , as the trajectory needs a series of continuous transitions in time . Off-policy actor-critic architectures aim to provide greater sample efficiency by reusing past experience ( previously collected transitions ) . DDPG ( Lillicrap et al. , 2016 ) borrows two main ideas from Deep Q Networks ( Mnih et al. , 2013 ; 2015 ) : a big replay buffer and a target Q network to give consistent targets during temporal-difference backups . TD3 ( Twin Delayed Deep Deterministic policy gradient algorithm ) ( Fujimoto et al. , 2018 ) develops a variant of Double Q-learning by taking the minimum value between a pair of critics to limit over-estimation . SAC ( Soft Actor-Critic ) ( Haarnoja et al. , 2018a ; b ) proposes a maximum entropy RL framework where its stochastic actor aims to simultaneously maximize expected action-value and entropy . The latest version of SAC ( Haarnoja et al. , 2018b ) also includes the “ the minimum value between both critics ” idea in its implementation . Meta Learning for RL . Meta-learning ( a.k.a . learning to learn ) ( Santoro et al. , 2016 ; Finn et al. , 2017 ) has received a resurgence in interest recently due to its potential to improve learning performance , and especially sample-efficiency in RL ( Gupta et al. , 2018 ) . Several studies learn optimizers that provide policy updates with respect to known loss or reward functions ( Andrychowicz et al. , 2016 ; Duan et al. , 2016b ; Meier et al. , 2018 ) . A few studies learn hyperparameters ( Xu et al. , 2018 ; Veeriah et al. , 2019 ) , loss functions ( Houthooft et al. , 2018 ; Sung et al. , 2017 ) or rewards ( Zheng et al. , 2018 ) that steer the learning of standard optimizers . Our meta-critic framework is in the category of loss-function meta-learning , but unlike most of these we are able to meta-learn the loss function online in parallel to learning a single extrinsic task rather . No costly offline learning on a task family is required as in Houthooft et al . ( 2018 ) ; Sung et al . ( 2017 ) . Most current Meta-RL methods are based on on-policy policy-gradient , limiting their sample efficiency . For example , while LIRPG ( Zheng et al. , 2018 ) is one of the rare prior works to attempt online meta-learning , it is ineffective in practice due to only providing a scalar reward increment rather than a loss for direct optimization . A few meta-RL studies have begun to address off-policy RL , for conventional offline multi-task meta-learning ( Rakelly et al. , 2019 ) and for optimising transfer vs forgetting in continual learning of multiple tasks ( Riemer et al. , 2019 ) . The contribution of our Meta-Critic is to enhance state-of-the-art Off-PAC RL with single-task online meta-learning . Loss Learning . Loss learning has been exploited in ‘ learning to teach ’ ( Wu et al. , 2018 ) and surrogate loss learning ( Huang et al. , 2019 ; Grabocka et al. , 2019 ) where a teacher network predicts the parameters of a manually designed loss in supervised learning . In contrast our meta-critic is itself a differentiable loss , and is designed for use in reinforcement learning . Other applications learn losses that improve model robustness to out of distribution samples ( Li et al. , 2019 ; Balaji et al. , 2018 ) . Our loss learning architecture is related to Li et al . ( 2019 ) , but designed for accelerating single-task Off-PAC RL rather than improving robustness in multi-domain supervised learning . 3 METHODOLOGY . We aim to learn a meta-critic that provides an auxiliary loss Lauxω to assist the actor ’ s learning of a task . The auxiliary loss parameters ω are optimized in a meta-learning process . The vanilla critic Lmain and meta-critic Lauxω losses train the actor πφ off-policy via stochastic gradient descent . 3.1 REVIEW OF OFF-POLICY ACTOR-CRITIC RL . Reinforcement learning involves an agent interacting with the environment E. At each time t , the agent receives an observation st , takes a ( possibly stochastic ) action at based on its policy π : S → A , and receives a scalar reward rt and new state of the environment st+1 . We call ( st , at , rt , st+1 ) as a single point transition . The objective of RL is to find the optimal policy πφ , which maximizes the expected cumulative return J . In on-policy RL , J is defined as the discounted episodic return based on a sequential trajectory over horizon H : ( s0 , a0 , r0 , · · · , sH , aH , rH ) . J = Ert , st∼E , at∼π [ ∑H t=0 γ trt ] . In the usual implemen- tation of A2C , r is represented by a surrogate state-value V ( st ) from its critic . Since J is only a scalar value , the gradient of J with respect to policy parameters φ has to be optimized under the policy gradient theorem ( Sutton et al. , 2000 ) : ∇φJ ( φ ) = E [ J ∇φ log πφ ( at|st ) ] . In off-policy RL ( e.g. , DDPG , TD3 , SAC ) which is our focus in this paper , parameterized policies πφ can be directly updated by defining the actor loss in terms of the expected return J ( φ ) and taking its gradient ∇φJ ( φ ) , where J ( φ ) depends on the action-value Qθ ( st , at ) . The main loss Lmain provided by the vanilla critic is thus Lmain = −J ( φ ) = −Es∼pπQθ ( s , a ) |a=πφ ( s ) , ( 1 ) where we follow the notation in TD3 and SAC that φ and θ denote actors and critics respectively . The main loss is calculated by a mini-batch of transitions randomly sampled from the replay buffer . The actor ’ s policy network is updated as ∆φ = α∇φLmain , following the critic ’ s gradient to increase the likelihood of actions that achieve a higher Q-value . Meanwhile , the critic uses Q-learning updates to estimate the action-value function : θ ← arg min θ ( Qθ ( st , at ) − rt − γQθ ( st+1 , π ( st+1 ) ) 2 . ( 2 )

Summary: In actor-critic algorithms, the policy tries to optimize cumulative discounted rewards by a loss formed with components from the critic. In this paper, the authors propose a novel way, namely, meta-critic, which utilizes meta-learning to learn an additional loss for the policy to accelerate the learning process of the agent. There are several advantages of the proposed method, it's learned online for a single task, it's trained with off-policy data, it provides sample efficiency and it's generally applicable to existing off-policy actor-critic methods.