Adversarially Guided Actor-Critic

1 INTRODUCTION . Research in deep reinforcement learning ( RL ) has proven to be successful across a wide range of problems ( Silver et al. , 2014 ; Schulman et al. , 2016 ; Lillicrap et al. , 2016 ; Mnih et al. , 2016 ) . Nevertheless , generalization and exploration in RL still represent key challenges that leave most current methods ineffective . First , a battery of recent studies ( Farebrother et al. , 2018 ; Zhang et al. , 2018a ; Song et al. , 2020 ; Cobbe et al. , 2020 ) indicates that current RL methods fail to generalize correctly even when agents have been trained in a diverse set of environments . Second , exploration has been extensively studied in RL ; however , most hard-exploration problems use the same environment for training and evaluation . Hence , since a well-designed exploration strategy should maximize the information received from a trajectory about an environment , the exploration capabilities may not be appropriately assessed if that information is memorized . In this line of research , we choose to study the exploration capabilities of our method and its ability to generalize to new scenarios . Our evaluation domains will , therefore , be tasks with sparse reward in procedurally-generated environments . In this work , we propose Adversarially Guided Actor-Critic ( AGAC ) , which reconsiders the actor-critic framework by introducing a third protagonist : the adversary . Its role is to predict the actor ’ s actions correctly . Meanwhile , the actor must not only find the optimal actions to maximize the sum of expected returns , but also counteract the predictions of the adversary . This formulation is lightly inspired by adversarial methods , specifically generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . Such a link between GANs and actor-critic methods has been formalized by Pfau & Vinyals ( 2016 ) ; however , in the context of a third protagonist , we draw a different analogy . The adversary can be interpreted as playing the role of a discriminator that must predict the actions of the actor , and the actor can be considered as playing the role of a generator that behaves to deceive the predictions of the adversary . This approach has the advantage , as with GANs , that the optimization procedure generates a diversity of meaningful data , corresponding to sequences of actions in AGAC . ∗Equal contribution . This paper analyses and explores how AGAC explicitly drives diversity in the behaviors of the agent while remaining reward-focused , and to which extent this approach allows to adapt to the evolving state space of procedurally-generated environments where the map is constructed differently with each new episode . Moreover , because stability is a legitimate concern since specific instances of adversarial networks were shown to be prone to hyperparameter sensitivity issues ( Arjovsky & Bottou , 2017 ) , we also examine this aspect in our experiments . The contributions of this work are as follow : ( i ) we propose a novel actor-critic formulation inspired from adversarial learning ( AGAC ) , ( ii ) we analyse empirically AGAC on key reinforcement learning aspects such as diversity , exploration and stability , ( iii ) we demonstrate significant gains in performance on several sparse-reward hard-exploration tasks including procedurally-generated tasks . 2 RELATED WORK . Actor-critic methods ( Barto et al. , 1983 ; Sutton , 1984 ) have been extended to the deep learning setting by Mnih et al . ( 2016 ) , who combined deep neural networks and multiple distributed actors with an actor-critic setting , with strong results on Atari . Since then , many additions have been proposed , be it architectural improvements ( Vinyals et al. , 2019 ) , better advantage or value estimation ( Schulman et al. , 2016 ; Flet-Berliac et al. , 2021 ) , or the incorporation of off-policy elements ( Wang et al. , 2017 ; Oh et al. , 2018 ; Flet-Berliac & Preux , 2020 ) . Regularization was shown to improve actor-critic methods , either by enforcing trust regions ( Schulman et al. , 2015 ; 2017 ; Wu et al. , 2017 ) , or by correcting for off-policiness ( Munos et al. , 2016 ; Gruslys et al. , 2018 ) ; and recent works analyzed its impact from a theoretical standpoint ( Geist et al. , 2019 ; Ahmed et al. , 2019 ; Vieillard et al. , 2020a ; b ) . Related to our work , Han & Sung ( 2020 ) use the entropy of the mixture between the policy induced from a replay buffer and the current policy as a regularizer . To the best of our knowledge , none of these methods explored the use of an adversarial objective to drive exploration . While introduced in supervised learning , adversarial learning ( Goodfellow et al. , 2015 ; Miyato et al. , 2016 ; Kurakin et al. , 2017 ) was leveraged in several RL works . Ho & Ermon ( 2016 ) propose an imitation learning method that uses a discriminator whose task is to distinguish between expert trajectories and those of the agent while the agent tries to match expert behavior to fool the discriminator . Bahdanau et al . ( 2019 ) use a discriminator to distinguish goal states from non-goal states based on a textual instruction , and use the resulting model as a reward function . Florensa et al . ( 2018 ) use a GAN to produce sub-goals at the right level of difficulty for the current agent , inducing a form of curriculum . Additionally , Pfau & Vinyals ( 2016 ) provide a parallel between GANs and the actor-critic framework . While exploration is driven in part by the core RL algorithms ( Fortunato et al. , 2018 ; Han & Sung , 2020 ; Ferret et al. , 2021 ) , it is often necessary to resort to exploration-specific techniques . For instance , intrinsic motivation encourages exploratory behavior from the agent . Some works use state-visitation counts or pseudo-counts to promote exhaustive exploration ( Bellemare et al. , 2016a ) , while others use curiosity rewards , expressed in the magnitude of prediction error from the agent , to push it towards unfamiliar areas of the state space ( Burda et al. , 2018 ) . Ecoffet et al . ( 2019 ) propose a technique akin to tree traversal to explore while learning to come back to promising areas . Eysenbach et al . ( 2018 ) show that encouraging diversity helps with exploration , even in the absence of reward . Last but not least , generalization is a key challenge in RL . Zhang et al . ( 2018b ) showed that , even when the environment is not deterministic , agents can overfit to their training distribution and that it is difficult to distinguish agents likely to generalize to new environments from those that will not . In the same vein , recent work has advocated using procedurally-generated environments , in which a new instance of the environment is sampled when a new episode starts , to assess generalization capabilities better ( Justesen et al. , 2018 ; Cobbe et al. , 2020 ) . Finally , methods based on network randomization ( Igl et al. , 2019 ) , noise injection ( Lee et al. , 2020 ) , and credit assignment ( Ferret et al. , 2020 ) have been proposed to reduce the generalization gap for RL agents . 3 BACKGROUND AND NOTATIONS . We place ourselves in the Markov Decision Processes ( Puterman , 1994 ) framework . A Markov Decision Process ( MDP ) is a tuple M = { S , A , P , R , γ } , where S is the state space , A is the action space , P is the transition kernel , R is the bounded reward function and γ ∈ [ 0 , 1 ) is the discount factor . Let π denote a stochastic policy mapping states to distributions over actions . We place ourselves in the infinite-horizon setting , i.e. , we seek a policy that optimizes J ( π ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) ] . The value of a state is the quantity V π ( s ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s ] and the value of a state-action pair Qπ ( s , a ) of performing action a in state s and then following policy π is defined as : Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] . The advantage function , which quantifies how an action a is better than the average action in state s , is Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . Finally , the entropyHπ of a policy is calculated as : Hπ ( s ) = Eπ ( ·|s ) [ − log π ( ·|s ) ] . Actor-Critic and Deep Policy Gradients . An actor-critic algorithm is composed of two main components : a policy and a value predictor . In deep RL , both the policy and the value function are obtained via parametric estimators ; we denote θ and φ their respective parameters . The policy is updated via policy gradient , while the value is usually updated via temporal difference or Monte Carlo rollouts . In practice , for a sequence of transitions { st , at , rt , st+1 } t∈ [ 0 , N ] , we use the following policy gradient loss ( including the commonly used entropic penalty ) : LPG = − 1 N t+N∑ t′=t ( At′ log π ( at′ |st′ , θ ) + αHπ ( st′ , θ ) ) , where α is the entropy coefficient and At is the generalized advantage estimator ( Schulman et al. , 2016 ) defined as : At = ∑t+N t′=t ( γλ ) t′−t ( rt′ + γVφold ( st′+1 ) − Vφold ( st′ ) ) , with λ a fixed hyperparameter and Vφold the value function estimator at the previous optimization iteration . To estimate the value function , we solve the non-linear regression problem minimizeφ ∑t+N t′=t ( Vφ ( st′ ) − V̂t′ ) 2 where V̂t = At + Vφold ( st′ ) . 4 ADVERSARIALLY GUIDED ACTOR-CRITIC . To foster diversified behavior in its trajectories , AGAC introduces a third protagonist to the actor-critic framework : the adversary . The role of the adversary is to accurately predict the actor ’ s actions , by minimizing the discrepancy between its action distribution πadv and the distribution induced by the policy π . Meanwhile , in addition to finding the optimal actions to maximize the sum of expected returns , the actor must also counteract the adversary ’ s predictions by maximizing the discrepancy between π and πadv ( see Appendix B for an illustration ) . This discrepancy , used as a form of exploration bonus , is defined as the difference of action log-probabilities ( see Eq . ( 1 ) ) , whose expectation is the Kullback–Leibler divergence : DKL ( π ( ·|s ) ‖πadv ( ·|s ) ) = Eπ ( ·|s ) [ log π ( ·|s ) − log πadv ( ·|s ) ] . Formally , for each state-action pair ( st , at ) in a trajectory , an action-dependent bonus log π ( at|st ) − log πadv ( at|st ) is added to the advantage . In addition , the value target of the critic is modified to include the action-independent equivalent , which is the KL-divergence DKL ( π ( ·|st ) ‖πadv ( ·|st ) ) . We discuss the role of these mirrored terms below , and the implications of AGAC ’ s modified objective from a more theoretical standpoint in the next section . In addition to the parameters θ ( resp . θold the parameter of the policy at the previous iteration ) and φ defined above ( resp . φold that of the critic ) , we denote ψ ( resp . ψold ) that of the adversary . AGAC minimizes the following loss : LAGAC = LPG + βV LV +βadvLadv . In the new objective LPG = − 1N ∑N t=0 ( A AGAC t log π ( at|st , θ ) + αHπ ( st , θ ) ) , AGAC modifies At as : AAGACt = At+ c ( log π ( at|st , θold ) − log πadv ( at|st , ψold ) ) , ( 1 ) with c a varying hyperparameter that controls the dependence on the action log-probability difference . To encourage exploration without preventing asymptotic stability , c is linearly annealed during the course of training . LV is the objective function of the critic defined as : LV = 1 N N∑ t=0 ( Vφ ( st ) − ( V̂t+ cDKL ( π ( ·|st , θold ) ‖πadv ( ·|st , ψold ) ) ) ) 2 . ( 2 ) Finally , Ladv is the objective function of the adversary : Ladv = 1 N N∑ t=0 DKL ( π ( ·|st , θold ) ‖πadv ( ·|st , ψ ) ) . ( 3 ) Eqs . ( 1 ) , ( 2 ) and ( 3 ) are the three equations that our method modifies ( we color in blue the specific parts ) in the traditional actor-critic framework . The terms βV and βadv are fixed hyperparameters . Under the proposed actor-critic formulation , the probability of sampling an action is increased if the modified advantage is positive , i.e . ( i ) the corresponding return is larger than the predicted value and/or ( ii ) the action log-probability difference is large . More precisely , our method favors transitions whose actions were less accurately predicted than the average action , i.e . log π ( a|s ) − log πadv ( a|s ) ≥ DKL ( π ( ·|s ) ‖πadv ( ·|s ) ) . This is particularly visible for λ → 1 , in which case the generalized advantage is At = Gt − Vφold ( st ) , resulting in the appearance of both aforementioned mirrored terms in the modified advantage : AAGACt = Gt − V̂ φold t + c ( log π ( at|st ) − log πadv ( at|st ) − D̂φoldKL ( π ( ·|st ) ‖πadv ( ·|st ) ) ) , with Gt the observed return , V̂ φold t the estimated return and D̂ φold KL ( π ( ·|st ) ‖πadv ( ·|st ) ) the estimated KL-divergence ( estimated components of Vφold ( st ) from Eq . 2 ) . To avoid instability , in practice the adversary is a separate estimator , updated with a smaller learning rate than the actor . This way , it represents a delayed and more steady version of the actor ’ s policy , which prevents the agent from having to constantly adapt or focus solely on fooling the adversary .

Summary: This paper proposed a new actor-critic framework with adversary guide for deep reinforcement learning (RL), and introduced new Kullback-Leiblier divergence bonus term based on the difference between actor network and adversary network to deal with the exploration in RL. The experimental results showed the merit of this method for exploration. Some comments are provided as follows.