Exploration via Flow-Based Intrinsic Rewards

Exploration bonuses derived from the novelty of observations in an environment have become a popular approach to motivate exploration for reinforcement learning ( RL ) agents in the past few years . Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models . In this paper , we introduce the concept of optical flow estimation from the field of computer vision to the RL domain and utilize the errors from optical flow estimation to evaluate the novelty of new observations . We introduce a flow-based intrinsic curiosity module ( FICM ) capable of learning the motion features and understanding the observations in a more comprehensive and efficient fashion . We evaluate our method and compare it with a number of baselines on several benchmark environments , including Atari games , Super Mario Bros. , and ViZDoom . Our results show that the proposed method is superior to the baselines in certain environments , especially for those featuring sophisticated moving patterns or with high-dimensional observation spaces . 1 INTRODUCTION . Deep reinforcement learning ( DRL ) algorithms aim at developing the policy of an agent to maximize its cumulative rewards collected in an environment , and have gained considerable attention in a wide range of application domains , such as game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) and robot navigation ( Zhang et al. , 2016 ) . Despite their recent successes , however , one of the key constraints of them is the requirement of dense reward signals . In environments with sparse reward signals , it becomes extremely challenging for an agent to explore and learn a useful policy . Although simple heuristics such as -greedy ( Mnih et al. , 2013 ) , entropy regularization ( Mnih et al. , 2016a ) , and noisy network ( Fortunato , 2018 ) were proposed , they are still far from satisfactory in such environments . Researchers in recent years have attempted to deal with the challenge by providing an agent with exploration bonuses ( also known as “ intrinsic rewards ” ) to encourage an agent to explore even when the reward signals from environments are sparse . These bonus rewards are associated with state novelty to incentivize an agent to explore those novel states . A number of novelty measurement strategies have been proposed in the past few years , such as the use of information gain ( Houthooft et al. , 2016 ) , count-based methods utilizing counting tables ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ) , and prediction-based methods exploiting prediction errors of dynamics models ( Stadie et al. , 2015 ; Pathak et al. , 2017 ; Burda et al. , 2019a ; b ) . These prediction-based methods differ in the targets of prediction . Pathak et al . ( 2017 ) ; Burda et al . ( 2019a ) introduce a forward dynamics model for predicting the feature representation of the next state based on the current state and the action taken by the agent , which is known as next-frame prediction . Next-frame prediction for complex or rapid-changing observations , however , is rather difficult for forward dynamics models , especially when the prediction is made solely based on the current state and the taken action . It has been widely recognized that performing next-frame prediction typically requires complex feature representations ( Kingma & Welling , 2014 ; Goodfellow et al. , 2014 ; Mirza & Osindero , 2014 ; Lotter et al. , 2017 ; Xue et al. , 2016 ) . On the other hand , Burda et al . ( 2019b ) proposed a self-frame prediction strategy by employing a predictor network to predict the feature representation of the current state from a fixed and randomly initialized target network . Nevertheless , the attempt of self-frame prediction to predict the encoded current state inherently neglects motion features in the observations . Rapid changes or moving patterns in two consecutive observations essentially serve as important signals to motivate an agent for exploration . It is thus inherently reasonable to take these features into account for prediction-based methods . As a result , in this paper we introduce a flow-based intrinsic curiosity module , called FICM , for evaluating the novelty of observations . FICM generates intrinsic rewards based on the prediction errors of optical flow estimation ( i.e. , the flow loss ) . Observations with low prediction errors , or low intrinsic rewards , indicate that the agent has seen the observations plenty of times . On the contrary , observations are considered seldom visited when the errors from the predicted flow are non-negligible . The latter case then prompts FICM to generate high intrinsic rewards to encourage the agent for exploration . Fig . 1 provides an illustrative example of FICM for Super Mario Bros . It can be observed that the brighter parts of the flow loss in Fig . 1 ( c ) align with the attention areas ( Greydanus et al. , 2018 ) in Fig . 1 ( d ) , implying the significance of motion features in intrinsically motivated exploration . In addition , as exploration bonuses ideally decrease over time , the agent is motivated to continuously explore novel observations during the training phase . Fig . 2 validates that flow loss gradually declines to low values after millions of iterations , indicating the qualification of flow loss to serve as an intrinsic reward signal . In other words , FICM is capable of incentivizing the agent to focus on moving parts of the environment and explore novel observations . We validate the performance of FICM in a variety of benchmark environments , including Atari 2600 ( Bellemare et al. , 2013 ) , Super Mario Bros. , and ViZDoom ( Wydmuch et al. , 2018 ) . We demonstrate that FICM is preferable to the previous prediction-based exploration methods in terms of exploration efficiency of the agent in several tasks and environments , especially for those featuring sophisticated moving patterns . We further provide a comprehensive set of ablation analysis for the proposed FICM . The primary contributions of this paper are thus summarized as the following : • We propose a new flow-based intrinsic curiosity module , called FICM , which leverages on existing methods of optical flow estimation from the field of computer vision ( CV ) to evaluate the novelty of observations characterizing by sophisticated motion features . • The proposed FICM encourages an RL agent to learn the motion features and understand the observations from an environment in a more comprehensive manner . • The proposed FICM is able to encode high dimensional inputs ( e.g. , RGB frames ) and utilizes the information more effectively and efficiently . • Moreover , the proposed FICM requires only two consecutive frames to obtain sufficient information when estimating the novelty of observations . The remainder of this paper is organized as the following . Section 2 introduces the background material related to this work . Section 3 presents the proposed FICM framework . Section 4 demonstrates the experimental results and discusses their implications . Section 5 concludes the paper . For more details of the background knowledge , implementations , and setups , please refer to our appendices . 2 BACKGROUND . In this section , we provide an overview of the previous curiosity-driven exploration methodologies . We first introduce these methods , followed by a review of the two representative strategies employed by them : the next-frame prediction and the self-frame prediction strategies . To ease the understanding of FICM , we discuss the concepts of DRL as well as optical flow estimation in our appendices . 2.1 CURIOSITY-DRIVEN EXPLORATION METHODOLOGIES . Curiosity-driven exploration is an exploration strategy adopted by a number of DRL researchers in recent years ( Houthooft et al. , 2016 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2019a ; b ) in order to explore environments more efficiently . While conventional random exploration strategies are easily trapped in local minima of state spaces for complex or spare reward environments , curiosity-based methodologies tend to discover relatively un-visited regions , and therefore are likely to explore more effectively than conventional strategies within the same amount of time . In addition to the extrinsic rewards provided by the environments , most curiositydriven exploration strategies introduce intrinsic rewards generated by the agent to encourage itself to explore novel states . In this paper , we mainly focus on the prediction-based methods of curiositydriven exploration including two branches : the next-frame and the self-frame prediction strategies . 2.1.1 NEXT-FRAME PREDICTION STRATEGY . In Pathak et al . ( 2017 ) ; Burda et al . ( 2019a ) , the authors formulate the exploration bonuses as the prediction errors of a forward dynamics network , representing an agent ’ s ability to predict the consequence of its actions in a visual feature space . Given an observation ot , a feature representation φt is generated by an embedding network . A forward dynamics network f is then used to predict the next state representation by φt and the agent ’ s taken action at . The mean-squared error ( MSE ) ||f ( φt , at ) −φt+1 ) ||2 is minimized by forward dynamics network and serves as the intrinsic reward . After exploring the environment for millions of iterations , more states are visited and the errors decline , indicating the agent are familiar with most of the states . However , next-frame prediction solely based on the current state and the taken action is challenging for complex or rapid-changing observations . Therefore , we propose to employ optical flow estimation to deal with the challenge . 2.1.2 SELF-FRAME PREDICTION STRATEGY . In Burda et al . ( 2019b ) , the authors analyze the factors of prediction errors and introduce random network distillation ( RND ) to measure state novelty . RND is composed of two networks : a fixed and randomly initialized target network g , and a predictor network ĝ . Given an observation ot , the objective of RND is to minimize the MSE ||g ( ot ) − ĝ ( ot ) ||2 . Intuitively , the predictor network can predict correctly on those frequently seen states and produce less intrinsic rewards . By introducing a deterministic target network , RND avoids the problem of stochastic prediction errors in next-frame prediction . However , RND does not consider motion features , which are essential in motivating an agent for exploration . We propose to introduce optical flow estimation to capture those features and encourage our agent to explore an environment from a different perspective , as compared with RND . 3 METHODOLOGY . In this section , we present the motivation and design overview of FICM . We first provide an introduction to the concepts of FICM . Then , we formulate these concepts into mathematical equations . Finally , we explore two different implementations of FICM and discuss their features and benefits . 3.1 FLOW-BASED CURIOSITY-DRIVEN EXPLORATION . Our objective is to develop a new exploration bonus method that is able to ( 1 ) understand the motion features between two consecutive frames , ( 2 ) encode the observations efficiently , and ( 3 ) estimate the novelty of different observations . As a result , we propose to embrace optical flow estimation ( Ilg et al. , 2017 ; Meister et al. , 2018 ) , a popular technique commonly used in the field of computer vision ( CV ) for interpreting displacement of objects in consecutive frames , as our state novelty measurement strategy . FICM is capable of encoding its input observations efficiently even with two consecutive RGB frames ( instead of stacked , gray-scale frames ) , and its efficiency is demonstrated in Section 4.3 . Moreover , FICM yields higher intrinsic rewards when the agent encounters unfamiliar states than that in familiar states . It then motivates the agent to revisit those states , and gradually learns the features of them over time , as depicted in Fig . 2 . The workflow of the proposed FICM and the architectures of our optical flow predictors are illustrated in Fig . 3 and Fig . 4 , respectively .

Summary: The paper proposes a novel way to formulate intrinsic reward based on optical flow prediction error. The prediction is done with Flownet-v2 architecture and the training is formulated as self-supervision (instead of the ground-truth-based supervised learning in the original Flownet-v2 paper). The flow predictor takes two frames, predicts forward and backward flows, then warps the first/second frame respectively and compares the warped result with real frame. The comparison error serves as the intrinsic reward signal. The results are demonstrated on 7 environments: SuperMario + 5 Atari games + ViZDoom. On those environments, the proposed method performs better or on-par with ICM and RND baselines.