DEEP ADVERSARIAL FORWARD MODEL

Learning world dynamics has recently been investigated as a way to make reinforcement learning ( RL ) algorithms to be more sample efficient and interpretable . In this paper , we propose to capture an environment dynamics with a novel forward model that leverages recent works on adversarial learning and visual control . Such a model estimates future observations conditioned on the current ones and other input variables such as actions taken by an RL-agent . We focus on image generation which is a particularly challenging topic but our method can be adapted to other modalities . More precisely , our forward model is trained to produce realistic observations of the future while a discriminator model is trained to distinguish between real images and the model ’ s prediction of the future . This approach overcomes the need to define an explicit loss function for the forward model which is currently used for solving such a class of problem . As a consequence , our learning protocol does not have to rely on an explicit distance such as Euclidean distance which tends to produce unsatisfactory predictions . To illustrate our method , empirical qualitative and quantitative results are presented on a real driving scenario , along with qualitative results on Atari game Frostbite . 1 INTRODUCTION . The capability to predict the consequences of actions is an inherent part of biological decision making Clark ( 2013 ) . Though this capacity is not always required , it becomes useful for tasks that require planning and more generally forecasting ahead of the current situation . Such a capability has been shown to be particularly useful in the case of environments with complex dynamics ( Bertsekas , 1995 ) . Indeed , modeling the dynamics of the surrounding world has led to significant improvements in the decision process thus providing the ability to anticipate ( Weber et al. , 2017 ; Silver et al. , 2017 ) . In the literature of decision and control , an environment forecasting model is commonly called a Forward Model Henaff et al . ( 2017 ) ; Pathak et al . ( 2017 ) . Such a model estimates what the future observation will be with respect to the current one and possibly other conditioning variables . These models are trained in a supervised fashion , either online or offline . More recently , several papers have shown that forward modeling can be used to quantify the inherent curiosity of a learning decision policy . Curiosity is defined as the error of a forward model during the course of an episode in a considered environment . Such quantity has been recently used to maintain the explorative behave of a decision policy and has been shown to be more efficient than maintaining an entropy regulariser as an egocentric reward . In this paper , we introduce an algorithm that learns forward models without relying on any kind of task-specific loss . Our method leverages recent improvements in adversarial training Goodfellow et al . ( 2014 ) ; Chen et al . ( 2016a ) . As far as our knowledge goes , it is the first attempt to use adversarial learning for such a forward modeling task . More precisely , we train a forward model in the form of a neural network , in an unsupervised fashion , through a discriminator network which has to distinguish between true and generated samples . By using this adversarial learning paradigm , the need to find the appropriate loss for each specific task is removed . Furthermore , this learning approach does not put any constraint on the modality of the observations the model receives and generates . The rest of the paper is organized as follows , after presenting the main elements of the state of the art , we describe the proposed methods and model . Then , we detail the experiments and the encouraging associated results that demonstrate the benefits of the proposed method . 2 RELATED WORK . In decision theory , forward models are commonly defined as auto-regressive architectures which can be trained using collected agent trajectories . Given a past state st and current action at+1 , the model predicts the next state p ( st+1|st , at+1 ) = fθ ( st , at+1 ) of the environment , with ftheta a function , for example a neural network , parameterized by θ . As an alternative definition , other signals such as reward can also be taken into account . Several arguments support the learning of a forward model ( Bertsekas , 1995 ) such as easier control policy definition as an optimization problem . More recently , several papers have already proposed deep learning for such a task ( Leibfried et al. , 2016 ; Oh et al. , 2015 ) . However , these works are currently using either handcrafted or pixel-wise representations to derive a computable and differentiable loss . More generally , several fields of research can dramatically benefit from this forward modeling . Exploration efficiency : Curiosity-driven exploration is formulated as the error in a learning agent ’ s ability to predict the consequence of its own actions ( Pathak et al. , 2017 ) . In this context , this error is measured through a forward model of the considered environment . The main motivation of such a measurement used as an intrinsic reward is to cope with the sparsity of extrinsic reward . It has been recently studied in the domain of visual control in grid worlds and Atari 2600 video games Henaff et al . ( 2017 ) . Such a metric is used to encourage exploration during the policy learning process . Moreover , the prediction error has also been used in the feature space of an auto-encoder as a measure of interest of a state to explore ( Stadie et al. , 2015 ) . Imagination-based control : Predicting the sequence of future outcomes from a given action has been recently investigated as an additional source of information for complex decision making . Indeed , learning World Models ( Ha & Schmidhuber , 2018 ) has been proposed as a promising way to decrease the complexity of a policy model . The idea is to delegate a part of the overall control learning task to a dynamic model instead of learning the environment representation and the optimal policy in a monolithic model trained in an end-to-end fashion . Thus , it becomes possible to benefit from environment observations that are not necessarily correlated with the task and the controller . Moreover , such a technique has been investigated as a way to perform transfer learning through tasks given an environment dynamic ( Kansky et al. , 2017 ) . The predictron ( Silver et al. , 2017 ) is a recent example of such an approach . This policy network learns environment dynamics as part of its component to enrich its decision support . Imagination-based control has provided encouraging initial results ( Weber et al. , 2017 ) . However , the end-to-end nature of the learning setup proposed in these works makes the interpretability of the model difficult . Interpretability : By developing parts of a decision model that can directly be observed - for example by looking at generated images of the future - it becomes possible to improve the interpretability of the model ( Maes et al. , 2012 ) . Indeed , forward modeling makes it possible to observe the decision support of the policy compared to implicit representations that only the model can use . For these reasons , we posit that forward modeling is an important field of research in the domain of control . In this paper , we propose to leverage the recent advances in imagination-based control and conditional generative adversarial models along with inverse modeling to develop a novel and generic forward model that is able to cope with synthetic and realistic visual control scenarios . 3 DEEP ADVERSARIAL FORWARD MODEL . 3.1 PROBLEM FORMULATION . We consider a transition function T that maps the current observation st to the the next observation st+1 , i.e. , st+1 = T ( st , at ) where at the action taken by an agent . A forward model fθ tries to approximate the transition function T and thus give an estimate ŝt+1 , such that ŝt+1 = f ( st , at ) is close to st+1 . The task of learning the forward model consists of learning the parameter θ that will minimize the error between st+1 and ŝt+1 . 3.2 MODEL ARCHITECTURE . The forward model is made of two neural networks . Figure 1 shows the architecture of the forward model . First , the observation st is encoded through a pretrained ResNet34 He et al . ( 2016 ) that aims at compressing the information in the raw observation . This abstract representation Φ ( s ) of the raw observations follows the approach proposed by Pathak et al . Pathak et al . ( 2017 ) . Then , this representation is concatenated with the action at taken at this time step to obtain ct . This is necessary because , in many situations , the current observation is not a sufficient piece of information to predict the next observation . For example at can represent the position of the wheel and velocity in the case of a forward model for driving . Finally , this conditioning ct is fed to a generator network G to obtain an estimate of the observation at the next timestep : ŝt+1 = fθ ( st , at ) = G ( Φ ( st ) , at ) = G ( ct ) . ( 1 ) 3.3 TRAINING AND LOSS FUNCTIONS . Our forward model is trained using a linear combination of three losses applied on the general architectures shown on figure 1 . The main loss is an adversarial loss encouraging the forward model to produce realistic forward estimates ŝt+1 . This loss is complemented with a mutual information loss for encouraging the forward model to make use of the conditioner . Finally , a feature space loss on a pretrained convolutional network encourages visual content to be similar in ŝt+1 and st+1 . To the best of our knowledge , it is the first composition of such losses schema in this context of forward modeling . We detail these losses below . Adversarial Loss : First , we want to generate forward samples that are realistic enough with regard to the current observation . For this , we follow the standard formulation of Generative Adversarial Learning Goodfellow et al . ( 2014 ) . In the adversarial setting , the role of discriminator D is to provide the loss function for training the generator G which correspond to the actual forward model we want to learn . To do this , the discriminator network D learns to distinguish the real next observation st+1 from its estimate ŝt+1 produced by G. The output of D is used as a probability of its input being real versus generated by G. At the same time , G is trained to fool D , i.e. , it is trained to generate samples that are hard for D to distinguish from the real ones . This known paradigm leads to the following two-player min-max game with value function V ( G , D ) : min G max D V ( G , D ) = Exv pdata [ log ( D ( x ) ) ] + Ezv pnoise [ log ( 1−D ( G ( z ) ) ) ] . ( 2 ) Mutual-Information Loss : Secondly , we need to encourage the generator to use all the provided auxiliary information ct to generate samples that take into account the current conditioner . For this , we introduce an additional loss Lc inspired from the InfoGAN setup Chen et al . ( 2016b ) . An additional network Q - which in practice shares most of its layers with D - outputs an estimation ĉt of ct . The associated loss Lc is defined as the cross-entropy between ĉt and ct in the case of discrete distributions and the mean squared error between them in the case of continuous ones . The addition of the mutual information loss turns the GAN objective into minG , Q maxD VInfoGAN = V ( D , G ) −λ Lc . In reinforcement learning , such a loss is used to train an inverse model . The inverse model model tries to guess which actions cause the transition from st to st+1 . Here , we just use this loss as auxiliary information to constraint the generator to make usage of the conditional inputs ct . Content Loss : Finally , in order to ensure that the generator outputs visually similar content than the one in st+1 , this third loss computes a similarity information based on the aspect of the generated visual states . A naive approach would be to add a weighted pixel-wise loss function such as Mean Squared Error ( MSE ) . In practice , these kinds of losses are very prone to generate blurry samples as they encourage the textures to be overly smooth , thus leading to bad image quality . Recently , Ledig et al . ( Ledig et al. , 2016 ) showed that combining both adversarial and content losses greatly help the generator to output realistic samples by relying on an abstract representation of the image . Following this , we include a content-loss LΨ that computes the Euclidean distance between the output representation generated by a network Ψ for st+1 and G ( Φ ( st ) , at ) . In our specific setup , we adopt the method ( Ledig et al. , 2016 ) to use V GGij for Ψ , where V GGij means the feature map obtained after the j-th convolution ( after activation ) and before the i-th max-pooling layer in the VGG19 network ( Simonyan & Zisserman , 2014 ) . Thus we define LΨ as LΨ = 1 Ci , jWi , jHi , j Ci , j∑ c=1 Wi , j∑ x=1 Hi , j∑ y=1 ( Ψ ( ŝt+1 ) cxy −Ψ ( st+1 ) cxy ) 2 . ( 3 ) Our final loss L is defined as a linear combination of all the aforementioned individual losses : L = αLadv + βLc + γLΨ .

Summary: Model-based RL that work on pixel-based environments tend to use forward models trained with pixel-wise loss. Rather than using pixel-wise loss for an action-conditioned video prediction model ("Forward Model"), they use an adversarial loss combined with mutual-information loss (from InfoGAN) and content loss (based on difference in convnet features of VGG network, rather than pixels). They run experiments on video-action sequences collected from an Atari game (Frostbite), and on a Udacity driving dataset.