Generative Adversarial Self-Imitation Learning

1 INTRODUCTION . A major component of Reinforcement learning ( RL ) is the temporal credit assignment problem that amounts to figuring out which action in a state leads to a better outcome in the future . Different RL algorithms have different forms of objectives to solve this problem . For example , policy gradient approaches learn to directly adapt the policy to optimize the RL objective ( i.e. , maximizing cumulative rewards ) , while value-based approaches ( e.g. , Q-Learning ( Watkins & Dayan , 1992 ) ) estimate longterm future rewards and induce a policy from it . Policies optimized for different objectives many have different learning dynamics , which end up with different sub-optimal policies in complex environments , though all of these objectives are designed to maximize cumulative rewards . In this paper , we explore a simple regularizer for RL , called Generative Adversarial Self-Imitation Learning ( GASIL ) . Instead of directly maximizing rewards , GASIL aims to imitate past good trajectories that the agent has generated using generative adversarial imitation learning framework ( Ho & Ermon , 2016 ) . GASIL solves the temporal credit assignment problem by learning a discriminator which discriminates between the agent ’ s current trajectories and good trajectories in the past , while the policy is trained to make it hard for the discriminator to distinguish between the two types of trajectories by imitating good trajectories . GASIL can potentially make long-term temporal credit assignment easier when reward signal is delayed , because reproducing certain trajectories is often much easier than maximizing long-term delayed rewards . GASIL can be interpreted as an optimal reward learning algorithm ( Singh et al. , 2009 ; Sorg et al. , 2010 ) , where the discriminator acts as a learned reward function which provides dense rewards for the agent to reproduce relatively better trajectories . Thus , it can be used as a shaped reward function and combined with any RL algorithms . Our empirical results on 2D Point Mass and OpenAI Gym MuJoCo tasks ( Brockman et al. , 2016 ; Todorov et al. , 2012 ) show that GASIL improves the performance of proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) , especially when rewards are delayed . We also show that GASIL is robust to stochastic dynamics to some extent in practice . 2 RELATED WORK . Generative adversarial learning Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been increasingly popular for generative modeling . In GANs , a discriminator is trained to discriminate whether a given sample is drawn from data distribution or model distribution . A generator ( i.e. , model ) is trained to “ fool ” the discriminator by generating samples that are close to the real data . This adversarial play allows the model distribution to match to the data distribution . This approach has been very successful for image generation and manipulation ( Radford et al. , 2015 ; Reed et al. , 2016 ; Zhu et al. , 2017 ) . Recently , Ho & Ermon ( 2016 ) proposed generative adversarial imitation learning ( GAIL ) which extends this idea to imitation learning . In GAIL , a discriminator is trained to discrimiate between optimal trajectories ( or expert trajectories ) and policy trajectories , while the policy learns to fool the discriminator by imitating optimal trajectories . Our work further extends this idea to RL . Unlike GAN or GAIL setting , however , optimal trajectories are not available to the agent in RL . Instead , our GASIL treats “ relatively better trajectories ” that the policy has generated as optimal trajectories that the agent should imitate . Reward learning Singh et al . ( 2009 ) discussed a problem of learning an internal reward function that is useful across a distribution of environments in an evolutionary context . Simiarly , Sorg et al . ( 2010 ) introduced optimal reward problem under the motivation that the true reward function defined in the environment may not be optimal for learning , and there exists an optimal reward function that allows learning the desired behavior much quickly . This claim is consistent with the idea of reward shaping ( Ng et al. , 1999 ) which helps learning without changing the optimal policy . There has been a few attempts to learn such an internal reward function without domain-specific knowledge in deep RL context ( Sorg et al. , 2010 ; Guo et al. , 2016 ; Zheng et al. , 2018 ) . Our work is closely related to this line of work in that GASIL learns a discriminator which acts as an interal reward function that allows the agent to learn to maximize external rewards more easily . Self-imitation There has been a line of work that introduces a notion of learning and inducing a good policy by focusing on good experiences that the agent has generated . For example , episodic control ( Lengyel & Dayan , 2008 ; Blundell et al. , 2016 ; Pritzel et al. , 2017 ) and the nearest neighbor policy ( Mansimov & Cho , 2017 ) construct a non-parametric policy directly from the past experience by retreiving similar states in the past and following the best decision made in the past . Instead , our work aims to learn a parametric policy from past good experiences . Self-imitation has been shown to be useful for program synthesis ( Liang et al. , 2016 ; Abolafia et al. , 2018 ) , where the agent is trained to generate K-best programs generated by itself . Our work proposes a different objective based on generative adversarial learning framework and evaluates it on RL benchmarks . More recently , Goyal et al . ( 2018 ) proposed to learn a generative model of preceding states of high-value states ( i.e. , top-K trajectories ) and update a policy to follow the generated trajectories . In contrast , our GASIL directly learns to imitate past good trajectories without learning a generative model . GASIL can be viewed as a generative adversarial extension of self-imitation learning ( Oh et al. , 2018 ) which updates the policy and the value function towards past better trajectories . Contemporaneously with our work , Gangwani et al . ( 2018 ) also proposed the same method as our GASIL , which was independently developed . Most of the previous works listed above including ours may not guarantee policy improvement under certain types of stochastic environments due to its bias towards positive outcome , though they have been shown to work well on existing benchmarks when used as a regularizer . Looking forward , dealing with stochasticity with this type of approach with stronger theoretical guarantees would be an interesting future direction . 3 BACKGROUND . Throughout the paper , we consider a finite state space S and a finite action space A . The goal of RL is to find a policy π ∈ Π : S × A → [ 0 , 1 ] which maximizes the discounted sum of rewards : η ( π ) = Eπ [ ∑∞ t=0 γ trt ] where γ is a discount factor and rt is a reward at time-step t. Alternatively , we can re-write the RL objective η ( π ) in terms of occupancy measure . Occupancy measure ρπ ∈ D : S ×A → R is defined as ρπ ( s , a ) = π ( a|s ) ∑∞ t=0 γ tP ( st = s|π ) . Intuitively , it is a joint distribution of states and actions visited by the agent following the policy π . It is shown that there is a one-to-one correspondence between the set of policies ( Π ) and the set of valid occupancy measures ( D = { ρπ : π ∈ Π } ) ( Syed et al. , 2008 ) . This allows us to write the RL objective in terms of occupancy measure as follows : η ( π ) = Eπ [ ∞∑ t=0 γtrt ] = ∑ s , a ρπ ( s , a ) r ( s , a ) . ( 1 ) where r ( s , a ) is the reward for choosing action a in state s. Thus , policy optimization amounts to finding an optimal occupancy measure which maximizes rewards due to the one-to-one correspondence between them . 3.1 POLICY GRADIENT . Policy gradient methods compute the gradient of the RL objective η ( πθ ) = Eπθ [ ∑∞ t=0 γ trt ] . Since η ( πθ ) is non-differentiable with respect to the parameter θ when the dynamics of the environment are unknown , policy gradient methods rely on the score function estimator to get an unbiased gradient estimator of η ( πθ ) . A typical form of policy gradient objective is given by : JPG ( θ ) = Eπθ [ log πθ ( at|st ) Ât ] ( 2 ) where πθ is a policy parameterized by θ , and Ât is an advantage estimation at time t. Intuitively , the policy gradient objective either increases the probability of the action when the return is higher than expected ( Ât > 0 ) or decreases the probability when the return is lower than expected ( Ât < 0 ) . 3.2 GENERATIVE ADVERSARIAL IMITATION LEARNING . Generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) is an imitation learning algorithm which aims to learn a policy that can imitate expert trajectories using the idea from generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) . More specifically , the objective of GAIL for maximum entropy IRL ( Ziebart et al. , 2008 ) is defined as : argmin θ argmax φ LGAIL ( θ , φ ) = Eπθ [ logDφ ( s , a ) ] + EπE [ log ( 1−Dφ ( s , a ) ) ] − λH ( πθ ) ( 3 ) where πθ , πE are a policy parameterized by θ and an expert policy respectively . Dφ ( s , a ) : S ×A → [ 0 , 1 ] is a discriminator parameterized by φ. H ( π ) = Eπ [ − log π ( a|s ) ] is the entropy of the policy . Similar to GANs , the discriminator and the policy play an adversarial game by either maximizing or minimizing the objective LGAIL , and the gradient of each component is given by : ∇φLGAIL = Eτπ [ ∇φ logDφ ( s , a ) ] + EτE [ ∇φ log ( 1−Dφ ( s , a ) ) ] ( 4 ) ∇θLGAIL = Eτπ [ ∇θ logDφ ( s , a ) ] − λH ( πθ ) ( 5 ) = Eτπ [ ∇θ log πθ ( a|s ) Q ( s , a ) ] − λH ( πθ ) , ( 6 ) where Q ( s̄ , ā ) = Eτπ [ logDφ ( s , a ) |s0 = s̄ , a0 = ā ] , and τπ , τE are trajectories sampled from πθ and πE respectively . Intuitively , the discriminator Dφ is trained to discriminate between the policy ’ s trajectories ( τπ ) and the expert ’ s trajectories ( τE ) through cross entropy loss . On the other hand , the policy πθ is trained to fool the discriminator by generating trajectories that are close to the expert trajectories according to the discriminator . Since logDφ ( s , a ) is non-differentiable with respect to θ in Equation 5 , the score function estimator is used to compute the gradient , which leads to a form of policy gradient ( Equation 6 ) using the discriminator as a reward function . It has been shown that GAIL amounts to minimizing the Jensen-Shannon divergence between the policy ’ s occupancy measure and the expert ’ s ( Ho & Ermon , 2016 ; Goodfellow et al. , 2014 ) as follows : argmin θ argmax φ LGAIL ( θ , φ ) = argmin θ DJS ( ρπθ ||ρπE ) − λH ( πθ ) ( 7 ) where DJS ( p||q ) = DKL ( p|| ( p+ q ) /2 ) +DKL ( q|| ( p+ q ) /2 ) denotes Jensen-Shannon divergence , a distance metric between two distributions , which is minimized when p = q .

Summary: This paper proposes a regularization technique for existing RL algorithms by encouraging them to learn to reproduce the best past trajectories which obtained higher reward than that of current policy. The proposed method in the paper has the same high-level idea as "Self-imitation learning" [Oh et.al. ICML 2018] with a different objective. Instead of performing imitation learning to distill the knowledge from past best trajectories, this proposes to use inverse reinforcement learning via GAIL objective [Ho and Ermano, 2016]. The best k trajectories from past experience are stored to train a discriminator which is then used to augment the external reward function with a discriminator reward.