Model Imitation for Model-Based Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has become of great interest because plenty of real-world problems can be modeled as a sequential decision-making problem . Model-free reinforcement learning ( MFRL ) is favored by its capability of learning complex tasks when interactions with environments are cheap . However , in the majority of real-world problems , such as autonomous driving , interactions are extremely costly , thus MFRL becomes infeasible . One critique about MFRL is that it does not fully exploit past queries over the environment , and this motivates us to consider the model-based reinforcement learning ( MBRL ) . In addition to learning an agent policy , MBRL also uses the queries to learn the dynamics of the environment that our agent is interacting with . If the learned dynamic is accurate enough , the agent can acquire the desired skill by simply interacting with the simulated environment , so that the number of samples to collect in the real world can be greatly reduced . As a result , MBRL has become one of the possible solutions to reduce the number of samples required to learn an optimal policy . Most previous works of MBRL adopt supervised learning with ` 2-based errors ( Luo et al. , 2019 ; Kurutach et al. , 2018 ; Clavera et al. , 2018 ) or maximum likelihood ( Janner et al. , 2019 ) , to obtain an environment model that synthesizes real transitions . These non-trivial developments imply that optimizing a policy on a synthesized environment is a challenging task . Because the estimation error of the model accumulates as the trajectory grows , it is hard to train a policy on a long synthesized trajectory . On the other hand , training on short trajectories makes the policy short-sighted . This issue is known as the planning horizon dilemma ( Wang et al. , 2019 ) . As a result , despite having a strong intuition at first sight , MBRL has to be designed meticulously . Intuitively , we would like to learn a transition model in a way that it can reproduce the trajectories that have been generated in the real world . Since the attained trajectories are sampled according to a certain policy , directly employing supervised learning may not necessarily lead to the mentioned result especially when the policy is stochastic . The resemblance in trajectories matters because we estimate policy gradient by generating rollouts ; however , the one-step model learning adopted by many MBRL methods do not guarantee this . Some previous works propose multi-step training ( Luo et al. , 2019 ; Asadi et al. , 2019 ; Talvitie , 2017 ) ; however , experiments show that model learning fails to benefit much from the multi-step loss . We attribute this outcome to the essence of super- vised learning , which elementally preserves only one-step transition and the similarity between real trajectories and the synthesized ones can not be guaranteed . In this work , we propose to learn the transition model via distribution matching . Specifically , we use WGAN ( Arjovsky et al. , 2017 ) to match the distributions of state-action-next-state triple ( s , a , s′ ) in real/learned models so that the agent policy can generate similar trajectories when interacting with either the true transition or the learned transition . Figure 1 illustrates the difference between methods based on supervised learning and distribution matching . Different from the ensemble methods proposed in previous works , our method is capable of generalizing to unseen transitions with only one dynamic model because merely incorporating multiple models does not alter the essence that one-step ( or few-step ) supervised learning fails to imitate the distribution of multi-step rollouts . Concretely , we gather some transitions in the real world according to a policy . To learn the real transition T , we then sample fake transitions from our synthesized model T ′ with the same policy . The synthesized model serves as the generator in the WGAN framework and there is a critic that discriminates the two transition data . We update the generator and the critic alternatively until the synthesized data can not be distinguished from the real one , which we will show later that it gives T → T ′ theoretically . Our contributions are summarized below : • We propose an MBRL method called model imitation ( MI ) , which enforces the learned transition model to generate similar rollouts to the real one so that policy gradient is accurate ; • We theoretically show that the transition can be learned by MI in the sense that T → T ′ by consistency and the difference in cumulative rewards can be bounded by the training error of WGAN ; • To stabilize model learning , we deduce guarantee for our sampling technique and investigate training across WGANs ; • We experimentally show that MI is more sample efficient than state-of-the-art MBRL and MFRL methods and outperforms them on four standard tasks . 2 RELATED WORK . In this section , we introduce our motivation inspired by learning from demonstration ( LfD ) ( Schaal , 1997 ) and give a brief survey of MBRL methods . 2.1 LEARNING FROM DEMONSTRATION . A straightforward approach to LfD is to leverage behavior cloning ( BC ) , which reduces LfD to a supervised learning problem . Even though learning a policy via BC is time-efficient , it can not imitate a policy without sufficient demonstration because the error may accumulate without the guidance of expert ( Ross et al. , 2011 ) . Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) is another state-of-the-art LfD method that learns an optimal policy by utilizing generative adversarial training to match occupancy measure ( Syed et al. , 2008 ) . GAIL learns an optimal policy by matching the distribution of the trajectories generated from an agent policy with the distribution of the given demonstration . Ho & Ermon ( 2016 ) shows that the two distributions match if and only if the agent has learned the optimal policy . One of the advantages of GAIL is that it only requires a small amount of demonstration data to obtain an optimal policy but it requires a considerable number of interactions with environments for the generative adversarial training to converge . Our intuition is that transition learning ( TL ) is similar to learning from demonstration ( LfD ) by exchanging the roles of transition and policy . In LfD , trajectories sampled from a fixed transition are given , and the goal is to learn a policy . On the other hand , in TL , trajectories sampled from a fixed policy are given , and we would like to imitate the underlying transition . That being said , from LfD to TL , we interchange the roles of the policy and the transition . It is therefore tempting to study the counterpart of GAIL in TL ; i.e. , learning the transition by distribution matching . Fortunately , by doing so , the pros of GAIL remain while the cons are insubstantial in MBRL because sampling with the learned model is considered to be much cheaper than sampling in the real one . That GAIL learns a better policy than what BC does suggests that distribution matching possesses the potential to learn a better transition than supervised learning . 2.2 MODEL-BASED REINFORCEMENT LEARNING . For deterministic transition , ` @ -based error is usually utilized to learn the transition model . Nagabandi et al . ( 2018 ) , an approach that uses supervised learning with mean-squared error as its objective , is shown to perform well under fine-tuning . To alleviate model bias , some previous works adopt ensembles ( Kurutach et al. , 2018 ; Buckman et al. , 2018 ) , where multiple transition models with different initialization are trained at the same time . In a slightly more complicated manner , Clavera et al . ( 2018 ) utilizes meta-learning to gather information from multiple models . Lastly , on the theoretical side , SLBO ( Luo et al. , 2019 ) is the first algorithm that develops from solid theoretical properties for model-based deep RL via a joint model-policy optimization framework . For the stochastic transition , maximum likelihood estimator or moment matching are natural ways to learn a synthesized transition , which is usually modeled by the Gaussian distribution . Following this idea , Gaussian process ( Kupcsik et al. , 2013 ; Deisenroth et al. , 2015 ) and Gaussian process with model predictive control ( Kamthe & Deisenroth , 2017 ) are introduced as an uncertainty-aware version of MBRL . Similar to the deterministic case , to mitigate model bias and foster stability , an ensemble method for probabilistic networks ( Chua et al. , 2018 ) is also studied . An important distinction between training a deterministic or stochastic transition is that although the stochastic transition can model the noise hidden within the real world , the stochastic model may also induce instability if the true transition is deterministic . This is a potential reason why an ensemble of models is adopted to reduce variance . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . We consider the standard Markov Decision Process ( MDP ) ( Sutton & Barto , 1998 ) . MDP is represented by a tuple 〈S , A , T , r , γ〉 , where S is the state space , A is the action space , T ( st+1|st , at ) is the transition density of state st+1 at time step t + 1 given action at made under state st , r ( s , a ) is the reward function , and γ ∈ ( 0 , 1 ) is the discount factor . A stochastic policy π ( a|s ) is a density of action a given state s. Let the initial state distribution be α . The performance of the triple ( α , π , T ) is evaluated in the expectation of the cumulative reward in the γ-discounted infinite horizon setting : R ( α , π , T ) = E [ ∞∑ t=0 γtr ( st , at ) ∣∣∣α , π , T ] = E [ H−1∑ t=0 r ( st , at ) ∣∣∣α , π , T ] . ( 1 ) Equivalently , R ( α , π , T ) is the expected cumulative rewards in a length-H trajectory { st , at } H−1t=0 generated by ( α , π , T ) with H ∼ Geometric ( 1 − γ ) . When α and T are fixed , R ( · ) becomes a function that only depends on π , and reinforcement learning algorithms ( Sutton & Barto , 1998 ) aim to find a policy π to maximize R ( π ) . 3.2 OCCUPANCY MEASURE . Given initial state distribution α ( s ) , policy π ( a|s ) and transition T ( s′|s , a ) , the normalized occupancy measure ρα , πT ( s , a ) generated by ( α , π , T ) is defined as ρα , πT ( s , a ) = ∞∑ t=0 ( 1−γ ) γtP ( st = s , at = a|α , π , T ) = ( 1−γ ) H−1∑ t=0 P ( st = s , at = a|α , π , T ) , ( 2 ) where P ( · ) is the probability measure and will be replaced by a density function if S or A is continuous . Intuitively , ρα , πT ( s , a ) is a distribution of ( s , a ) in a length-H trajectory { st , at } H−1 t=0 with H ∼ Geometric ( 1 − γ ) following the laws of ( α , π , T ) . From Syed et al . ( 2008 ) , the relation between ρα , πT and ( α , π , T ) is characterized by the Bellman flow constraint . Specifically , x = ρ α , π T as defined in Eq . 2 is the unique solution to : x ( s , a ) = π ( a|s ) [ ( 1− γ ) α ( s ) + γ ∫ x ( s′ , a′ ) T ( s|s′ , a′ ) ds′da′ ] , x ( s , a ) ≥ 0 . ( 3 ) In addition , Theorem 2 of Syed et al . ( 2008 ) gives that π ( a|s ) and ρα , πT ( s , a ) have an one-to-one correspondence with α ( s ) and T ( s′|s , a ) fixed ; i.e. , π ( a|s ) , ρ ( s , a ) ∫ ρ ( s , a ) da is the only policy whose occupancy measure is ρ . With the occupancy measure , the cumulative reward Eq . 1 can be represented as R ( α , π , T ) = E ( s , a ) ∼ρα , πT [ r ( s , a ) ] / ( 1− γ ) . ( 4 ) The goal of maximizing the cumulative reward can then be achieved by adjusting ρα , πT , and this motivates us to adopt distribution matching approaches like WGAN ( Arjovsky et al. , 2017 ) to learn a transition model .

Summary: In model-based reinforcement learning methods, in order to alleviate the compounding error induced in rollout-based planning, the paper proposes a distribution matching method, which should be better than a regular supervised learning approach (i.e. minimizing mean square error). Experiments on continuous control domains are presented to show the algorithm’s advantages. The author compares the proposed method with several well-known baselines. The proposed approach is interesting, however, there are some issues.