Exploration via Flow-Based Intrinsic Rewards
Authors: Hsuan-Kung Yang
Po-Han Chiang
Min-Fong Hong
Chun-Yi Lee
Venue:  (2020)
Keywords: reinforcement learning
exploration
curiosity
optical flow
intrinsic rewards

Abstract: Exploration bonuses derived from the novelty of observations in an environment have become a popular approach to motivate exploration for reinforcement learning (RL) agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. In this paper, we introduce the concept of optical flow estimation from the field of computer vision to the RL domain and utilize the errors from optical flow estimation to evaluate the novelty of new observations.  We introduce a flow-based intrinsic curiosity module (FICM) capable of learning the motion features and understanding the observations in a more comprehensive and efficient fashion. We evaluate our method and compare it with a number of baselines on several benchmark environments, including Atari games, Super Mario Bros., and ViZDoom. Our results show that the proposed method is superior to the baselines in certain environments, especially for those featuring sophisticated moving patterns or with high-dimensional observation spaces.

Under review as a conference paper at ICLR 2020
EXPLORATION VIA FLOW-BASER INTRINSIC REWARDS
Anonymous authors Paper under double-blind review
ABSTRACT
Exploration bonuses derived from the novelty of observations in environment have become a popular approach to motivate exploration for reinforcement learn- ing (RL) agents in the past few years. Recent methods such as auriosity-driven exploration usually estimate the novelty of new observations by the prediction er- of their system dynamics models. In this paper, we introduce the concept of optical flow estimation from the field of computer vision to the RI. domain and utilize the errors from optical flow estimation evaluate the novelty of new ob- servations. We introduce a flow-based intrinsic curiosity module (FICM) capable of learning the motion features and understanding the observations in more com- prehensive and efficient fashion. We evaluate our method and compare it with a number of baselines on several benchmark environments, including Atari games, Super Mario Bros., and ViZDoom. Our results show that the proposed method is superior to the baselines certain environments, especially for those featuring sophisticated moving patterns with high-dimensions observation spaces.
INTRODUCTION
Deep reinforcement learning (DRL) algorithms aim at developing the policy of an agent to maximize its cumulative rewards collected in an environment, and have gained considerable attention in a wide range of application domains, such as game playing (Mnih al., 2015 Silver al. (016) and robot navigation (Zhang et al., (016). Despite their recent successes, however, one of the key constraints of them the requirement of dense reward signals. In environments with sparse reward signals, becomes extremely challenging for an agent to explore and learn useful policy. Although simple heuristics such as c-greedy (Mnih et al.) (213), entropy regularization (Mnih et al.] (0166)) and noisy network [Fortunato] 2018) were proposed, they are still far from satisfactory in such environments.
Researchers in recent years have attempted to deal with the challenge by providing an agent with ex- ploration bonuses (also known as "intrinsic rewards") to encourage agent to explore even when the reward signals from environments are sparse. These bonus rewards are associated with state novelty to incentivize
agent to explore those novel states. A number of novelty measurement strategies have been proposed in the past few years, such as the use of information gain (Houthooft 2016), count-based methods utilizing counting tables (Bellemare al.) 2016 Ostrovski al. and prediction -based methods exploiting prediction errors of dynamics models (Stadie et al., Pathak al. 2017; Burda al. 2019a (b. These prediction-based methods differ in the tar- gets of prediction. Pathak et al. (2017 Burda et al. (2019a) introduce a forward dynamics model predicting the feature representation of the next state based on the current state and the action taken by the agent, which is known as next-frame prediction. Next-frame prediction for complex or rapid-changing observations, however, rather difficult for forward dynamies models, especially when the prediction is made solely based on the current state and the taken action. It has been widely recognized that performing next-frame prediction typically requires complex feature repre- sentations (Kingma & Welling 2014 Goodfellow et al. 2014 Mirza & Osindero 2014 Lotter et al.) 2017 Xue et al. 2016). On the other hand, Burda et al. (2019b) proposed self-frame prediction strategy by employing a predictor network to prediet the feature representation of the current state from a fixed and randomly initialized target network. Nevertheless, the attempt of self-frame pre- diction predict the encoded current state inherently neglects motion features in the observations.
et al.
2017
2015

---

Under review as a conference paper at ICLR 2020
(b) 0+1
(c) (d) Attention areas
Figure 1: Illustration of the relation between the prediction errors (i.e., flow loss) and the attention are two consecutive frames, (c) visualizes the flow loss, and (d) depicts the attention areas (Greydanus al. 2018) of agent.
Figure Prediction errors of FICM versus training iterations
of agent. (a) and (b)
for selected states.
Rapid changes or moving patterns in two consecutive observations essentially serve important signals to motivate an agent for exploration. It thus inherently reasonable take these features into account for prediction-bais -baeeddd methods. As a result, in this paper we introduce flow-based intrinsic curiosity module, called FICM, for evaluating the novelty observations. FICM generates intrinsic rewards based on the prediction errors of optical flow estimation (i.e., the flow loss). Observations with low prediction errors, or low intrinsic rewards, indicate that the agent has seen the observations plenty of times. On the contrary, observations the predicted flow are non-negligible. The latter case then prompts FICM to generate high intrinsic rewards encourage the agent for exploration. Fig. provides an illustrative example of FICM for Super Mario Bros. It can be observed that the brighter parts of the flow loss in Fig. (c) align with the attention areas (Greydanus et al. 2018) in Fig. (d), implying the significance motion features in intrinsically motivated exploration. In addition, as exploration bonuses ideally decrease over time, the agent is motivated to continuously explore novel observations during the training phase. Fig. 2 validates that flow loss gradually declines to low values after millions of iterations, indicating the qualification of flow loss to serve as an intrinsic reward signal. In other words, FICM is capable of incentivizing the agent focus moving parts of the environment and explore novel observations.
considered seldom visited when the errors from
We validate the performance of FICM in a variety of benchmark environments, including Atari 2600 (Bellemare et al.) 2013), Super Mario Bros., and ViZDoom (Wydmuch et al.) 2018). We demonstrate that FICM is preferable to the previous prediction based exploration methods in terms of exploration efficiency of the agent in several tasks and environments, especially for those featur- ing sophisticated moving patterns. We further provide a comprehensive set of ablation analysis for the proposed FICM. The primary contributions of this paper thus summarized the following:
We propose new flow-based intrinsic curiosity module, called FICM, which leverages on existing methods of optical flow estimation from the field of computer vision (CV) to evaluate the novelty of observations characterizing by sophisticated motion features.
The proposed FICM encourages an RL agent to learn the motion features and understand the observations from environment in
comprehensive manner.
The proposed FICM is able to encode high dimensional inputs (e.g., RGB frames) and utilizes the information more effectively and efficiently.
Moreover, the proposed FICM requires only two consecutive frames to obtain sufficient information when estimating the novelty of observations.
The remainder of this paper is organized the following. Section introduces the background ma- terial related to this work. Section (3rtsents the proposed FICM framework. (eeeioon/ddmmoostratts the experimental results and discusses their implications. Section 5 concludes the paper. For more details of the background knowledge, implementations, and setups, please refer to our appendices.
BACKGROUND
In this section, we provide an overview of the previous auriosity-driven exploration methodologies. We first introduce these methods, followed by review of the two representative strategies employed by them: the next-frame prediction and the self-frame prediction strategies. To ease the understand- ing of FICM, we discuss the concepts of DRL as well optical flow estimation in appendices.
2

---

Under review as a conference paper at ICLR 2020
2.1
CURIOSITY-DRIIE EXPLORATION METHODOLOGIES
Curiosity-driven exploration is an exploration strategy adopted by number of DRL researchers in recent years (Houthooft et al. 2016; Bellemare et al. 2016; Ostrovski al.] 2017 Pathak et al. 2017 Burda et a 2019a b) in order to explore environments more efficiently. While conventional random exploration strategies are easily trapped local minima of state spaces for complex or spare reward environments, curiosity-bai based methodologies tend to discover relatively un-visited regions, and therefore are likely to explore more effectively than conventional strategies within the same amount of time. In addition to the extrinsic rewards provided by the environments, most curiosity- driven exploration strategies introduce intrinsic rewards generated by the agent to encourage itself to explore novel states. In this paper, we mainly focus on the prediction -asedd methods of curiosity- driven exploration including two branches: the next-frame and the self-frame prediction strategies.
2.1.1 NEXT-FRAME PREDICTION STRATEGY
In Pathak et al. (2017); Burda et al. (2019)), the authors formulate the exploration bonuses as the prediction errors of a forward dynamics network, representing an agent's ability to predict the sequence of its actions in a visual feature space. Given an observation ot, a feature representation 6 generated by an embedding network. forward dynamics network is then used to predict the next state representation by Ø; and the agent's taken action az. The mean--quared eтror (MSE) ( a2) 1() 2 is minimized by forward dynamics network and serves as the intrinsic reward. After exploring the environment for millions of iterations, more states are visited and the errors decline, indicating the agent are familiar with most of the states. However, next-frame prediction solely based on the current state and the taken action is challenging for complex or rapid-changing observations. Therefore, we propose to employ optical flow estimation to deal with the challenge.
2.1.2 SELF-FRAME PREDICTION STRATEGY
In [urda et al. 2019b), the authors analyze the factors of prediction errors and introduce random network distillation (RND) to measure state novelty. RND is composed of two networks: fixed and randomly initialized target network g, and a predictor network g. Given an observation or, the o(o,) 1| Intuitively, the predictor network can predict correctly on those frequently seen states and produce less intrinsic rewards. By introducing a deterministic target network, RND avoids the problem of stochastic prediction errors in next-frame prediction. However, RND does not consider motion features, which are essential in motivating an agent for exploration. We propose introduce optical flow estimation to capture those features and encourage our agent to explore an environment from a different perspective, as compared with RND.
objective of RND is to minimize the MSE (19():)
3 METHODOLOGY
In this section, present the motivation and design overview of FICM. We first provide intro- duction to the concepts of FICM. Then, we formulate these concepts into mathematical equations. Finally, we explore two different implementations of FICM and discuss their features and benefits.
3.1 FLOW-BASED CURIOSITY- EXPLORATION
Our objective is to develop a new exploration bonus method that is able to (1) understand the motion features between two consecutive frames. (2) encode the observations efficiently, and (3) estimate the novelty of different observations. As result, we propose to embrace optical flow estimation (gg et al., 2017 Meister et al.] 2018), a popular technique commonly used in the field of computer vision (CV) for interpreting displacement of objects in consecutive frames, our state novelty measurement strategy. FICM is capable of encoding its input observations efficiently even with two consecutive RGB frames (instead of stacked, gray-scale frames), and its efficiency is demonstrated in Section 4.3 Moreover, FICM yields higher intrinsic rewards when the agent encounters unfamiliar states than that in familiar states. It then motivates the agent revisit those states, and gradually learns the features of them over time, as depicted in Fig. 2 The workflow of the proposed FICM and the architectures of our optical flow predictors are illustrated in Fig. 3 and Fig. 4 respectively.
3

---

Under review as a conference paper at ICLR 2020
Rackward Now (F becevadd
Warping
Flow predictor
111
Forward (F (Frrarr)
Figure 3: The workflow of the proposed flow-based intrinsic curiosity module (FICM).
3.2 BROO-BASED INTRINSIC CURIOSITY MODULE (FICM)
In this section, we formulate the procedure of FICM as formal mathematical equations. The main objective of FICM to leverage the optical flow between two consecutive observations as the en- coded representation of them. Given two raw input observations o; and 0r+-1 observed at consecutive timesteps and + 1, FICM takes the 2-tuple (ot, 0t+1) ) as its input, and predicts a forward flow I forward and backward flow Fbackword by its flow predictor G parameterized by a set of trainable parameters y. The two flows Fforward and Fbockward can therefore be expressed as the following:
F Foorward G(or, 1,4 O₁) and Phackword G( Ot, Oy).
F Forward and Fbackwerd are used to generate the observations 6, and o+41 1 via warping function (+) defined in Fischer et al. (2015): Olg al. (2017). The generated oR and o441 expressed
6, = W ((0+1 Fforward, B) and o1 = (or, Fbuckward. B),
(2)
where is the flow scaling factor. (**) (*) warps 0t+1 o, and or to 0,41 via F Forwward and Fbackward respectively using bilinear interpolation and wiement-wise multiplication with B. The interested reader is referred Fischer et al. (2015): Mg al. (2017) for more details of the warping algorithm. Please note that in this work, W(*) employs inverse mapping instead of forward mapping to avoid the common duplication problem flow warping (Beier & Neely 1992).
With the predicted observations ó, and o1411 e is iteratively updated to minimize the flow loss function LG of the flow predictor G, which consists of a forward flow loss Lt and a backward flow loss Lhd The goal of 9 is given by:
min Lo min (I LD7 min o + |1o, 6ell3), Øy
(3)
0, 6,
where (LS, Ld) are derived from the mean-squared error (MSE) between (0t+1, o141) and (or, 6t), respectively. this work, LG is interpreted by FICM as a measure of novelty, and serves as an intrinsic reward signal r² presented to the agent. The expression of r¹ is therefore formulated as:
o --r ' +L" S - - oiill +1oo 6,l1²). 2
(4)
where Ç is the reward scaling factor, and r-f and و are the forward and backward intrinsic rewards scaled from L' and Ld, respectively. Please note that yd is independent of the action taken by the agent, which distinguishes FICM from the intrinsic curiosity module (ICM) proposed in Pathak al. (2117). FICM only takes two consecutive input observations for estimating the prediction errors of optical flows, which serve
meaningful measure for evaluating and memorizing the novelty of observations in environments with figh-dimensional observation spaces. The experimental results presented in Section 4 validate the effectiveness of intrinsic reward y' and the proposed FICM.
4

---

Under review as a conference paper at ICLR 2020
D+1
Centrica
6065
Jacond
(a) The flw predictor of FICM-S
(b) The flow predictor of FICM-C
Figure 4: The flow predictor architectures of FICM-S and FICM-C.
3.3 IMPLEMENTATIONS OF FICM
In this work, propose different implementations of FICM: FICM-S and FICM-C. These two implementations adopt different flow predictor architectures based on FlowNerS and FlowNerC in- troduced by lg et al. (2017). respectively. We employ different implementations to validate that Eqs. (44 are generalizable to different architectures. rather than restricted to any specific pre- dictor designs. The flow predictor architectures of FICM-S and FICM-C are depicted in Fig. 4 (a) and Fig. E (b), respectively. The primary difference between the two predictors lies in their feature extraction strategy from the input observations og and 01+1. FICM-S encodes the stacked observa- tions (oc, 0++1) together to generate single feature embedding 6, while the other one encodes or and 0++1 in different paths to generate two separate feature embeddings Øt and Φ In order to preserve both coarse, high-level information and fine, low-level information for enhancing the flow prediction accuracy, the feature embeddings in the two predictor designs are later fused with the feature maps from their shallower parts of the networks by skips (Fischer al., 2015). The fused feature map is then processed by another convolutional layer at the end to predict the optical flow from O, 0t+11. Please note that the two input paths of Fig. 4 (b) are share-weighted in order to generate comparable embeddings. The flow predictor of FICM-C additionally employs a correlation layer from (Fischer al. 2015) perform multiplicative patch comparisons between Φe and Φ1+1-
4 EXPERIMENTAL RESULTS
In this section, we present the experimental results on a number of environments characterizing by complex motion features. We start by comparing the proposed methodology with the previous prediction-based exploration approaches on five Atari 2600 games (Bellemare et al. 2013) and Super Mario Bros., in the absence of any extrinsic reward signal. Next, we evaluate the performance of FICM on ViZDoom (Wydmuch et al.) 2018), combining intrinsic rewards with sparse extrinsic rewards. Finally, we present the ablation analysis of FICM from three different aspects. We further provide the training details and the setting of paramterrrmeters in the supplementary appendices.
4.1 EXPERIMENTS ON PURE EXPLORATION CAPABILITY
Environments. We first perform experiments on Super Mario Bros. and five different Atari games, including CrazyClimber, Enduro, KungFuMaster, Seaquest, and Skiing. During the training phase, the agents are not provided with any extrinsic reward or end-of-episode signal, in order to examine their abilities of pure exploration.
Baseline approaches. We compare the performance of our method against two baselines: (a) next-frame prediction with random CNN features (Burda et al. 2019a), and (b) self-frame predic- tion (Burda al.) 2019b), denoted as Forward Dynamics and RND, respectively. Both of the two approaches are prediction- based exploration approaches but differ in their prediction targets. In ad- dition, in order to present the best settings in terms of the agents' performance, our method takes RGB frames as inputs while the baselines remain the default settings described in their papers, that is, taking gray-scale frames as inputs. An ablation study the usage between RGB and gray-seale frames is further presented in Section 4.3 for fair comparisons.
Results. We plot the evaluation curves of our method (denoted FICM-C) and the baselines in different environments in Fig. BJdurig the training phase where the agents are trained without access to extrinsic rewards. It is observed that our method significantly outperforms the two baselines in
5

---

Under review as a conference paper at ICLR 2020
RND (seli-frem prediction)
Forward Dynemics (next-freme freme prediction)
Ours (FICM-C)
CrazyClimber
Enduro
KungFuMaster
Extrinsic Reward per Episode
302
80000
202
10000
5320
Mario
Seaquest
Skinng
10000
-10200
6500
0020
403
-20000
-25200
003
-32009
2008
201
Timesteps
Figure 5: Mean extrinsic rewards (with standard error) that pure exploration agents obtained in Super Mario Bros. and five selected Atari games. These environments feature complex moving patterns, and therefore our proposed FICM outperforms the baselines in Super Mario Bros., CrazyClimber, and Skiing, while delivering comparable results in Seaquest.
Enduro,
games Super Mario Bros., KungFuMaster, CrazyClimber, Enduro, and Skingg while delivering comparable performance to the baselines in games Seaquest. These games are characterized by moving objects that require the agents to concentrate on and interact with. As a result, they are favorable to the proposed exploration method, as the optical flow estimator is capable of capturing motion features and perceiving the changes in observations in more comprehensive fashion.
Since there is no extrinsic reward provided to the agent, the fluctuations and sudden drops in the evaluation curves in (ie.. KungFuMaster) only reflect the exploration processes of the agents. At different timesteps, the agent may focus its attention fluctuations in the evaluation curve. High scores in the curves indicate that the agents can explore hard-to- visit states (which usually associated with high extrinsic rewards) certain timesteps. The same fluctuations in the evaluation curves also occurred in the experiments conducted in Burda et al. (2019).. We further illustrate the best extrinsic return curves in our supplementary appendices.
exploring different states, leading to
4.2 EXPERIMENTS ON EXPLORATION WITH SPARSE EXTRINSIC REWARDS
Environments. We further conduct experiments on exploration with sparse extrinsic rewards. The environment evaluate on is the gaming environment, DoomMy WayHome-vO in ViZDoom [Wyd-] much et al. 2018), the same as those conducted in Pathak et al. (2017). In this environment, the agent required reach the fixed goal from certain spawning location in nine-rooms map, and only receives an extrinsic reward of '+1' if it accomplishes the task. We adopt two setups, sparse and very sparse reward settings, to evaluate the exploration ability of an agent. The two settings are different in the distance between the initial spawning location of the agent and the goal. The farther the goal is from the spawning location of the agent, the harder the map is for the agent explore.
Baseline approaches. We compare FICM with two baselines, 'ICM + A3C' (denoted as ICM) and 'IMM-pxels + A3C' (denoted as ICM-pixels) which are both proposed in Pathaa al. (2017). These baselines combine intrinsic curiosity module with training algorithms A3C (Mnih et al. 2016bb. ICM-pixels is close to ICM in architecture except for the absence of the inverse dynamics model, and it computes intrinsic rewards only dependent on forward model loss in next-frame prediction.
Results. We analyze and compare the evaluation curves of our proposed methods (denoted as FICM-C and FICM-S) with those of the baseline methods. The experimental results on the ViZDoom environment are depicted in Fig. 6 In the sparse reward setting, our methods and the baselines are able to guide the agent to reach the goal. However, in the very sparse reward setting, it is observed that ICM sometimes suffers from performance drop and is not always able to to obtain the maximum performance, while ICM-pixels even fails to reach the goal in this setting. On the contrary, proposed methods based on FICM are able to converge faster than the baselines and maintain stable different runs.
performance consistently in both the sparse and very sparse reward settings
The summation of extrinsic and intrinsic rewards are provided to the agents in this ViZDoom exper- iment, with the same setting adopted in Pathak et al. (2077) Although directly adding them up may

---

Under review as a conference paper at ICLR 2020
1.0
100
per Ephoods
0.8
8000
9000
6000
6000
0.4
4200
Extincic
0.2
2000
2000
07
54
1014
1554
IN
3N
1031
1500
01
2014
60M
20N
40M
Timesteps
Timestops
(8)
(b)
(A)
(b)
Figure 6: Comparison of the evaluation curves in ViZDoom with (a) sparse and (b) very sparse extrinsic rewards.
Figure 7: Analysis of feature encoding ability of (a) RGB versus gray-scale frames and (b) stacked versus non-stacked frames in Super Mario Bros.
be the most straightforward approach, it is definitely not the best way to do so since the objective of extrinsic and intrinsic rewards might be different and even confficted with each other. There- fore, combining them requires sophisticated engincering. Take Atari games for example, the agent may receive rewards from the environment by eliminating an enemy or collecting coins, however, the intrinsic rewards have no relationship with those extrinsic rewards. Careless combinations of them may confuse the agent during exploration, resulting in unsatisfactory performance. Investigat- ing novel methods to effectively combine extrinsic and intrinsic rewards to encourage an agent for exploration is thus promising future research direction yet it is beyond the scope of this paper.
4.3 ABLATION ANALYSIS
In this section, we present a set of ablation analysis to investigate the feature encoding abilities of our method and validate the proposed FICM in terms of three different aspects: (1) the dimensionality of the input frames, (2) the number of the stacked frames, and (3) the applicable domains of FICM.
RGB versus gcay-scale frames. We hypothesize that RGB frames contain more useful informa- tion than gray-scale ones to be utilized in prediction-based approaches. In order to validate this assumption, we eliminate the procedure of RGB to gray-scale conversion on the input frames when generating intrinsic rewards. For fair comparison, the RL agents still take gray-scale frames as their inputs. We examine whether FICM and the baselines, Forward Dynamics and RND mentioned in Section 4.1] are able to make use of the extra information to guide the agent for exploration. The experiments are evaluated on Super Mario Bros., and the results are plotted in Fig. (a). We present the results with and without RGB to gray-scale conversion as two different settings for both FICM and the baseline. According to the figure, the baselines perform nearly the same for both settings. In contrast, FICM using RGB frames outperforms the one using only gray-scale frames, indicating that FICM can encode the features more efficiently and utilize the information contained in RGB channels. The superior performance is mainly since the optical flow estimator is more capable of perceiving the motions of objects with color information, difference between foreground and background. As a result, FICM is able to predict optical flow more accurately, and the accurately predicted flow allows FICM to warp the observations more pre- cisely. The increase of input channels does not harm the ability of the flow predictor or warping function, as the same displacement vectors are used in each channel to get the warping results. On the contrary, it is difficult for Forward Dynamics and RND to obtain appropriate representations of high- -iggeiinensional inputs (i.e., RGB frames), since they are lack of eligible embedding networks to encode such RGB information. Fig. (a) reveals their inefficiency in encoding RGB information.
it is much easier distinguish the
Stacked versus non-stacked frames. We conduct another experiment to investigate the encod- ing efficiency of FICM and the baselines. We consider two different setups in terms of numbers of stacked frames: (a) two consecutive single frames (denoted as non-stacked) and (b) a pair of four stacked frames (denotes as stacked). In addition to the existing setups adopted in Section 4. we introduce their counterpart setups. The comparison results are plotted in Fig. (b). The results show that both FICM-C (stacked) and FICM-C (non-stacked) maintain high and consistent performance, indicating that FICM is able to encode the information efficiently even with only two consecutive frames. Furthermore, it is observed that the evaluation curve of Forward Dynamics (non-stacked)
7

---

Under review as a conference paper at ICLR 2020
2010
2
(b)
2%
0z
0++1
Flow
Attention areas
Evaluation curve
5M 10M
15M
Figure 8: Visualization of flow loss and attention areas in (a) Super Mario Bros. and (b) BeamRider. The first two columns are two consecutive frames and 0t+1- The third column shows the flow loss of the two frames, where the brighter parts reflect higher flow loss. The following three columns highlight the attention areas of the agents in red at 5M, 10M, and 15M timesteps. The last column shows the evaluation curves of pure exploration, representing the exploration process of the agents.
exhibits high variance highlighted the wide shaded area of the evaluation curve, showing that Forward Dynamics (non-stacked) suffers from unstable performance due to its poor encoding effi- ciency. Meanwhile, there is no much difference on performance between RND (stacked) and RND (non-stacked). Intuitively, self-frame prediction is less affected by stacked or non-stacked frames. Since there is significant difference in the evaluation curves of FICM-C (stacked) and FICM-C (non-stucked), we suggest that FICM only requires non-stacked frames, to generate intrinsic rewards.
Discussion of applicable domains. Despite the outstanding performance in the experiments ducted above, there might be some limitations of FICM which will result in the unsatisfactory re- sults: (1) insufficiency of movements of the major objects (c.g., enemies) and (2) excessive changes of irrelevant components (e.g., backgrounds) that distract the focus of FICM. For instance, if the ob- jects that the agent should pay attention to barely move, FICM generates negligible intrinsic rewards since there is only little discrepancy between two consecutive frames. On the other hand, if some irrelevant parts of the environments move relatively faster than the crucial objects, FICM may be distracted to concentrate on incorrect regions components, leading to unsatisfactory performance.
To prove and validate the hypothesis above, we further illustrate how FICM guides the agent to explore the environments by visualizing the flow loss and the attention areas (Greydanus et al. 2018) of the agents during the training phase in Fig. g It is observed that the flow loss is highly correlated the attention areas of the agents different timesteps for both games. The evidence reveals that FICM is indeed able to guide the agents to concentrate on the regions with high flow loss during the training phase, the attention areas grow wider and become brighter at those regions in later timesteps. However, flow loss may as well distract the agent to focus on irrelevant parts of the environments. Take BeamRider for example, the agent misled to concentrate on the background by the rich flow loss in it, resulting in its poor performance depicted by the evaluation curve in Fig. (b).. On the other hand, the agent is able master Super Mario Bros, as it concentrates on the enemies and is not distracted by irrelevant objects. The visualization further validates that FICM is able to guide an agent to concentrate on moving objects and motivate exploration an environment.
5 CONCLUSIONS
In this paper, we proposed FICM for evaluating the novelty of observations in RL exploration. FICM employs optical flow estimation errors as a measure for generating intrinsic rewards, which allow an RL agent to explore environments featuring moving objects or with high-dimensional observa- tion spaces (i.e., RGB inputs) in a more comprehensive and efficient manner. We demonstrated the proposed methodology and compared it against a number of baselines on Atari games, Super Mario Bros., and ViZDoom. According to our experiments, we observed that FICM is capable of focus- ing on essential objects, and guiding the agent to deliver superior performance to the baselines in certain environments. Moreover, we presented a comprehensive set of ablation analysis analyzing the encoding efficiency and applicable domains of FICM, and provided our insights from different aspects. Based on the experimental evidence discussed above, we therefore conclude that FICM is a promising strategy motivate exploration, especially for complex and rapid-changing observations.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
T. Beier and S. Neely. Feature-based image metamorphosis. In Special Interest Group on Computer Graphics (SIGGRAPH), pp. 35 42-42, Jul. 1992.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artificial Intelligence Research (JAIR), 47:253-279,
May 2013.
M. G. Bellemare, Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems (NeurIPS), pp. 1471-1479 Dec. 2016.
Y. Burda, H. Edwards, Pathak, A. Storkey, Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. In Proc. Int. Conf. Learning Representation (ICLR), May 2019a.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov, Exploration by random network distillation. In Proc. Int. Conf. Learning Reprexentations (ICLR), May 2019b.
Fischer, A. Dosovitskiy, and E. llgA. et al. FlowNet: Learning optical flow with convolutional networks. In Proc. IEEE Int. Conf. Computer Vision (ICCV), pp.
May 2015.
M. et al. Fortunato. Noisy networks for exploration. In Proc. Int. Conf. Learning Representations (ICLR), May 2018.
I. J. Goodfellow, J. Pouget-Abadie M. Mirza, B. Xu, D. W.-Farle,, S. Ozair, A. C. Courville, and
Y. Bengio. Generative adversarial networks. Proc. Advances Neural Information Processing Systems (NeurIPS), pp. 2680, Dec. 2014.
Greydanus, A. Koul, Dodge, and A. Fern. Visualizing and understanding atari agents. In Int. Conf. Machine Learning (ICML), pp. 1787-1796, Jun. 2018.
R. Houthooft, X. Chen, Duan, Schulman, D. Turck, and P. Abbeel. VIME: Variational in- formation maximizing exploration. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 1109-1117, Dec. 2016.
E. 1lg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet 2.0: optical flow estimation with deep networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1647-1655, Dec. 2017.
Evolution of
D. P. Kingma and M. Welling, Auto-encoding variational bayes. 1rXiv:1312.661 May 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 3431- 3440, Jun. 2015.
W. Lotter, G. Kreiman, and D. D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. wrxiv:66050080000 Mar. 2017.
Meister, J. Hur, and Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In Proc. of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pp. 7251-7259, 2018.
M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv: 1411.1884 Nov. 2014.
Mnih, K. Kavukeuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and Riedmiller. Playing atari with deep reinforcement learning. 1XXi::33/.56022 Dec. 2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, vol. 518, no. 7540, pp. 533-533, Feb. 2015.

---

Under review as a conference paper at ICLR 2020
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Int. Conf. Machine Learning (ICML) pp. 19377, Jun. 2016a.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proc. Int. Conf. Machine Learning (ICML). pp. 1928-1937, Jun. 2016b.
G. Ostrovski, M. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Proc. Int. Conf. Machine Learning (ICML), pp. 2721-2730, Jun. 2017.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. Int. Conf. Machine Learning (ICML), pp. 2778-27877 May 2017.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. TrXXiv:707.063320 Jul. 2017.
Samuel Schulter, Paul Vernaza, Wongun Choi, and Manmohan Krishna Chandraker. Deep net- work flow for multi-object tracking. Proc. IEEE Conf. Computer Vision and Puttern Recognition (CVPR), pp. 2730-2739, Jun. 2017.
D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre,, G. Driessche, Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillierap. M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, vol. 529, 7587, pp. 484-489, Jan. 2016.
B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv:1507.0088 1507000814 Nov. 2015.
M. Wydmuch, M. Kempkaand, and W. Jaskowski. ViZDoom competitions: Playing Doom from pixels. IEEE Trans. Games, Oct. 2018
Xue, J. Wu, K. L. Bouman, and B. Freeman. Visual dynamics: Probabilistic future frame synthesis via eross convolutional networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 91-99, Dec. 2016.
M. Zhang, Z. McCarthy, C. Finn, S. Levine, and P. Abbeel. Learning deep neural network policies with continuous memory states. In Proc. Int. Conf. Robotics and Automation (ICRA), pp. 520- 527, May 2016.
10

---

az), Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 BACKGROUND
A.1.1 DEEP REINFORCEMEN LEARNING (DRL)
DRL is method to train an agent to interact with an environment &. At each timesteps t, the agent receives state 8, form the state space of &, takes an action o, from the action space according to its policy [[]ss), receives reward r(s,, dt), and transits to next state 8t+1 p($ (s, at). The main objective of agent is maximize its discounted cumulative rewards R2
where E (0, 1] is the discount factor and T is the horizon. The action-value function (i.e., Q- function) of given policy и is defined as the expected return starting from state-action pair (s, as,,), expressed as (2ss, a) E [R s 0, = a, T.. With the advancement of Deep Neural Networks (DNNs), Deep Q-learning (DQN) (Mnih et al., 2013) was proposed take advantages of DNN to approximate the Q- Q-function. Asynchronous Advantage Actor-Critic (A3C) (Mnih et al.) (016b) further introduces asynchronous parallelization and optimization to accelerate the policy and value function updates of an agent. In addition, Proximal Policy Optimization (PPO) (Schulman et al.) 2017) takes the biggest possible improvement step on a policy without stepping so to prevent accidental performance collapse.
A.1.2 OPTICAL FLOW ESTIMATION
Optical flow estimation (Fischer al. 2015) is technique to evaluate the motion of objects between consecutive images. In usual cases, a reference image and target image are required. The optical flow is represented vector field, where displacement vectors reference image. These vectors represent where those pixels can be found in the target image. In recent years, number of deep learning approaches running on GPUs dealing with large displace- ment issues of optical flow estimation have been proposed (Fischer et al. 2015 Schulter et al., 2017] lig et af. 2017). Among these techniques, FlowNet 2.0 (Ig al. 2017) delivers the most accurate estimation. In this paper, we use simplified network of FlowNet 2.0 to generate optical flow.
assigned to certain pixels of the
A.2 IMPLEMENTATIONS OF FICM
FICM-S The flow predictor in FICM-S consists of several convolutional and deconvolutional lay- ers. The module first stacks two consecutive observations o, and 0++1 together, and feed the stacked observations (oe, 0t+1) into three convolution layers with 32, 64, and 96 filters, respectively, fol- lowed by an exponential linear unit (ELU) non-linear activation function. The encoded features are then fused with the feature maps from the shallower parts of the network by adding skips (Fischer et al. 2015 Long et al.) 2015), and fed into two deconvolutional layers with 64 and 32 filters. This skip layer fusion architecture allows the flow predictor to preserve both coarse, high layer informa- tion and fine, low layer information (Fischer et al. 2015 Long et al. (015). Finally, the feature map passed into convolutional layer with two filters to predict the optical flow from or to 0c41.
FICM-C
The flow predictor in FICM-C encodes two consecutive observations o, and o sep- arately instead of stacking them together. The input observations are passed through three convo- lutional layers generate feature maps dt and Φ The three convolutional layers of the two paths are whirh-weighted in order to generate better representations of ór and Φ as input obser- vations or and 0t+1 usually contain same or similar patterns. The feature maps á and d are then fed into correlation layer proposed by Fischer et al. (015), which performs multiplicative patch comparisons between two feature maps to estimate their correspondences defined as:
c(F1, = Σ ((dr( + 1), (22 + i)) xE(-RAA)((-AAA)
(5)
where In is the patch center in the first feature map and 22 is that in the second one. Here we fix the maximum displacement d to 2, which means only compute correlations c(#1, limited in a neighborhood of size D ;= 2d + 1 by constraining the range of ø2, Since the feature maps of our models have moderate resolutions, d 2 is sufficient to find their correspondences.
11

---

Under review as a conference paper at ICLR 2020
RND (self-freme prediction)
Forward Dynamics (next-freme prediction)
Ours (FICM-C)
CrezyClimber
400
Enduro
KungFuMann FMastrr
Extrinsic Reward per Episode
30000
300
100000
200
20000
50000
10000
Mario
Seaquest
Skiing
-3200
1200
10000
1000
-10000
800
-15200
20M
40M
Timesteps
Figure 9: Best extrinsic returns eight Atari games and Super Mario Bros.
Once the correspondences of o, and 0++1 is estimated, it is concatenated with the feature map from the fourth convolutional layer of the right path in Fig. h). The concatenated feature map later passed through one convolutional and deconvolutional layers, fused with the feature map came from the skip path, and then passed through another deconvolutional layer. Similar to Fig. 4a), the final feature map Fig. bb) is fed into convolutional layer with two filters to predict the optical flow from Of to 01+1.
A.3 BEST RETURN IN ATARI GAMES
In addition to mean rewards achieved by the agent, we further illustrate the highest rewards as measurement of how well the agents explore the environments, since certain states providing higher rewards are harder for the agent to visit. In Fig. 9 it observed that our method (denoted as FICM- C) outperforms the baselines (denoted as Forward Dynamics, RND) in most of the games, showing consistent results to those in Fig. 5
A.4 HYPERPARAMETER
The hyperparameters for the PPO and A3C algorithm are shown in Table I details of configurations of FICM and baselines in Table
We also provide the
Value Asynchronous Advantage Actor-Critic (A3C))
Hyperparameter
Learning rate of agent
le-4 4
Discount
0.99
GAE parameter (X)
1.0
Entropy cocff.
0.01
Number of threads
20
Optimization for RL agent
Adam
Proximal Policy Optimization (PPO)
Learning of agent
1e-4
Horizon
128
Discount (7)
0.99
GAE parameter (X)
0.95
Entropy coeff.
0.001
Number of optimization epochs
Number of minibatches
Number of parallel environments
128
Optimization for RL agent
Adam
Table The detail settings of hyper-parameters using in PPO and A3C algorithm.
12

---

Under review as a conference paper at ICLR 2020
Hyperparameter
Value
Experiments on Pure Exploration Capability
Training algorithm of RL agent Input size for RL agent Input size for FICM Learning rate of FICM Forward dynamic settings RND settings
84*84*4 (gray-scale)
84*84*3 (RGB)
1e-6
Same as settings provided (Burda
2019a
Same as settings provided (Burda el al. 2019b
Experiments on Exploration with Sparse Extrinsic Rewards
Training algorithm of RL agent
A3C
Input size for RL agent Learning of FICM
42*42*4 4 (gray-scale)
le-4
Other baseline settings
Same as settings provided in (Pathak 2017
Ablation Analysis
Training algorithm of RL agent Input size for RL agent Learning rate of FICM
PPO
84 (gray-scale)
1e-6
RGB versus gray-scale frames
Input size for FICM Input size for Forward dynamic Input size for RND
84*84* 3 (RGB) v.s. 84*84"1 (gray-scale)
84 12 (RGB) v.s. 84*84*4 (gray-scale)
84*84*3 (RGB) v.s. 84*84%1 (gray-scale)
Stacked versus non-stacked frames
Input size for FICM
84*84#1 (non-stacked) v.s. 84*84*4 (stacked)
Input size for Forward dynamic Input size for RND
84*84*1 (non-stacked) v.s. 84*84*4 (stacked)
84*84#1 (non-stacked) v.s. 84984*4 (stacked)
Other baseline settings
Same as settings provided in Burda 2019a and (Burda et al. 2019b
Table 2: The detail configurations of FICM and baselines.
13


Reviews:
[{"review_id": "SJxmXHNhtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Well motivated paper\n\nThe authors study the problem of exploration and exploitation in deep reinforcement learning. The authors propose a new intrinsic curiosity-based method that deploys the methods developed in optical flow. Following this algorithm, the agents utilize the reconstruction error in the optical flow network to come up with intrinsic rewards. The authors show that this approach boosts up the behavior of the RL agents and improves the performance on a set of test environments. \n\nA few comments that I hope might help the authors to improve the clarity of their paper. \n\n1) While the paper is nicely written, I would encourage the authors, of course, if they think necessary, to make the paper slightly more self-contained by explaining the optical flow problem, FlowNet, and warping approach. While a cruise reader might be required to either know literature in optical flow or go and study them along with this paper, it might be helpful for a bit more general readers to have these tools and approaches in access.\n\n2) Regarding the first line of introduction, I would recommend to rephrase it to one imply that the mentioned \"aim\" is one of the aims of the DRL study. \n\n3) In the fourth line of the intro, the authors mention that the current DRL methods are \"constraint\" to dense reward. I believe the authors' aim was to imply that these methods perform more desirably in dense reward settings rather than being constrained to such settings.\n\n4) I would also recommend to the authors to elaborate more on the term \"attention area\" Greydanue et al 2018.\n\n5) It would be helpful to have a better evaluation of this paper if the authors could clarify and motivate the choice of games in their empirical study. For example the empirical study in Fig 5.\n\n\n6) While I find this study interesting and valuable, the novelty of the approach might fall short to be published at a conference like ICLR with a low acceptance rate. This does not mean that there is anything unscientific about this paper, in fact, the scientific value of this work is appreciated and this work adds a lot to the community. \n\n7) It would also be useful to explicitly explain the advances of this approach over the next frame predictions approaches in stochastic environments. And also, if there is a shortcoming, what are those.\n\n8) Also, what the authors think would happen when the action directly does not change the scene, at least immediately. \n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xxeBsnFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel way to formulate intrinsic reward based on optical flow prediction error. The prediction is done with Flownet-v2 architecture and the training is formulated as self-supervision (instead of the ground-truth-based supervised learning in the original Flownet-v2 paper). The flow predictor takes two frames, predicts forward and backward flows, then warps the first/second frame respectively and compares the warped result with real frame. The comparison error serves as the intrinsic reward signal. The results are demonstrated on 7 environments: SuperMario + 5 Atari games + ViZDoom. On those environments, the proposed method performs better or on-par with ICM and RND baselines. \n\nI am leaning towards rejecting this paper. Two key factors motivate this decision. \nFirst, the motivation for this work is not fully clear: why would the error in flow prediction be a good driving force for curiosity? Optical flow has certain weaknesses, e.g. might not work well for textureless regions because it's hard to find a match. Why would those weaknesses drive the agent to new locations? \nSecond, the choice of tasks where the largest improvement is shown (i.e. 5 Atari games) seems not well-motivated and rather crafted for the proposed method. Those 5 Atari games are not established hard exploration games.\n\nDetailed arguments for the decision above:\n[major concerns]\n* Analysis is need on how the method deals with known optical flow problems: occlusion, large displacements, matching ambiguities. Those problems don't fully go away with learning and it is unclear how correlated corresponding errors would be with state novelty.\n* \"Please note that ri is independent of the action taken by the agent, which distinguishes FICM from the intrinsic curiosity module (ICM) proposed in Pathak et al. (2017)\" - but would it then be susceptible to spurious curiosity effects when the agent is drawn to motion of unrelated things? Like leaves trembling in the wind. ICM was proposed to eliminate those effects in the first place, but what is this paper's solution to that problem? Furthermore, the experiments on BeamRider show that this concern is not a theoretical one but quite practical.\n* \"CrazyClimber, Enduro, KungFuMaster, Seaquest, and Skiing\" - none of those Atari environments are known to be hard exploration games (which are normally Gravitar, Montezuma Revenge, Pitfall!, PrivateEye, Solaris, Venture according to Bellemare et al \"Unifying count-based exploration and intrinsic motivation\"). I understand that every game becomes hard-exploration if the rewards are omitted but then there is a question why those particular games. Moreover, if you omit the rewards the question remains how to select hyperparameters of your method. Was the game reward used for selecting hyperparameters? If not, what is the protocol for their selection? This is a very important question and I hope the authors will address this.\n* \"These games are characterized by moving objects that require the agents to concentrate on and interact with.\" - this looks like tailoring the task to suit the method.\n* Figure 6 - those results are not great compared to the results of Episodic Curiosity: https://arxiv.org/abs/1810.02274 . Maybe this is because of the basic RL solver (A3C vs PPO) but that brings up another question: why are different solvers used for different tasks in this paper? PPO is normally significantly better than A3C, why not use throughout the whole paper?\n[minor concerns]\n* Figures are very small and the font in them is not readable. Figure 2 is especially difficult to read because the axes titles are tiny.\n* \"complex or spare reward\" -> sparse\n* \"However, RND does not consider motion features, which are essential in motivating an agent for exploration.\" - this is unclear, why are those features essential?\n* \"We demonstrated the proposed methodology and compared it against a number of baselines on Atari games, Super Mario Bros., and ViZDoom.\" - please state more clearly that only 5 out of 57 Atari games are considered, here and in the abstract.\n* \"Best extrinsic returns on eight Atari games and Super Mario Bros.\" - but only 5 games are shown, where are the other 3?\n\nSuggestions on improving the paper:\n1) Better motivating the approach in the paper would help. Why using the flow prediction error as a curiosity signal?\n2) Better motivating the choice of the environments and conducting experiments on more environments would be important for evaluating the impact of the paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgBCOT4Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nPros\nSolid technical innovation/contribution: \n- The paper proposed a novel method FICM that bridged the intrinsic reward in DRL with optical flow loss in CV to encourage exploration in an environment with sparse rewards. To the best of my knowledge, this was the first paper proposed to use moving patterns in two consecutive observations to motivate agent exploration.\n\nBalanced view: \n- The authors discussed both the advantages of FICM and settings that FICM might fail to perform well, and conducted experiments to better help the readers understand such nuances. Such balanced view should be valuable to RL communities in both academia and industry.\n\nClarity:  \n- In general this was a very well-written paper, I had no difficulty in following the paper throughout. The proposed method (FICM) was clearly motivated, and the authors provided good coverage of related works. Notably, the authors reviewed two relevant methods upon which FICM was motivated, which made the paper self-contained.\n\n\nCons\nExperiments:\n- Experiments were conducted only using a few recent results as baselines (ICM, forward dynamics, RND). It would be interesting to compare FICM against simpler exploration baselines such as epsilon-greedy or entropy regularization.\n- I’d also like to see more extensive comparisons between FICM and ICM across different datasets, for example, Super Mario Bros. and the Atari games, instead of only comparing FICM against ICM on ViZDoom.\n\nSignificance of the innovation: \n- The proposed exploration method seemed to be applicable with a particular RL setting: the environment changes could be represented through consecutive frames (e.g., video games), and optical flow could be used to interpret any object displacements in such consecutive frames. And as the authors discussed, even under such constraints the applicability of proposed method depends on how much changes of the environment were relevant to the goal.\n\nReproducibility:\n- Although the authors discussed the experiment setting in detail in supplements, I believe open-sourcing the code / software used to conduct the experiments would be greatly help with the reproducibility of the proposed method for researchers or practitioners.\n\n\n\n\nSummary\nA good paper overall, but the experiments were relatively weak (common for most ICLR submissions) and the novelty was somewhat limited.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for improving exploration by implementing intrinsic rewards based on optical flow prediction error. The approach was evaluated on several Atari games, Super Mario, and VizDoom.

There are several strengths to this work, including the fact that it comes with open source code, and several reviewers agree it’s an interesting approach. R1 thought it was well-written and quite easy to follow. I also commend the authors for being so responsive with comments and for adding the new experiments that were asked for.

The main issue that reviewers pointed out, and which I am also concerned about, is how these particular games were chosen. R3 points out that these 5 Atari games are not known for being hard exploration games. Authors did conduct further experiments on 6 Atari games suggested by the reviewer, but the results didn’t show significant improvement over baselines.

I appreciate the authors’ argument that every method has “its niche”, but the environments chosen must still be properly motivated. I would have preferred to see results on all Atari games, along with detailed and quantitative analysis into why FICM fails on specific tasks. For instance, they state in the rebuttal that “The selection criteria of our environments is determined by the relevance of motions of the foreground and background components (including the controllable agent and the uncontrollable objects) to the performance (i.e., obtainable scores) of the agent.” But it doesn’t seem like this was assessed in any quantitative way.  Without this understanding, it’d be difficult for an outsider to know which tasks are appropriate to use with this approach. I urge the authors to focus on expanding and quantifying the work they depict in Figure 8, which, although it begins to illuminate why FICM works for some games and not others, is still only a qualitative snapshot of 2 games. I still think this is a very interesting approach and look forward to future versions of this paper.
Decision: Reject