Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a novel way to formulate intrinsic reward based on optical flow prediction error. The prediction is done with Flownet-v2 architecture and the training is formulated as self-supervision (instead of the ground-truth-based supervised learning in the original Flownet-v2 paper). The flow predictor takes two frames, predicts forward and backward flows, then warps the first/second frame respectively and compares the warped result with real frame. The comparison error serves as the intrinsic reward signal. The results are demonstrated on 7 environments: SuperMario + 5 Atari games + ViZDoom. On those environments, the proposed method performs better or on-par with ICM and RND baselines. 

I am leaning towards rejecting this paper. Two key factors motivate this decision. 
First, the motivation for this work is not fully clear: why would the error in flow prediction be a good driving force for curiosity? Optical flow has certain weaknesses, e.g. might not work well for textureless regions because it's hard to find a match. Why would those weaknesses drive the agent to new locations? 
Second, the choice of tasks where the largest improvement is shown (i.e. 5 Atari games) seems not well-motivated and rather crafted for the proposed method. Those 5 Atari games are not established hard exploration games.

Detailed arguments for the decision above:
[major concerns]
* Analysis is need on how the method deals with known optical flow problems: occlusion, large displacements, matching ambiguities. Those problems don't fully go away with learning and it is unclear how correlated corresponding errors would be with state novelty.
* "Please note that ri is independent of the action taken by the agent, which distinguishes FICM from the intrinsic curiosity module (ICM) proposed in Pathak et al. (2017)" - but would it then be susceptible to spurious curiosity effects when the agent is drawn to motion of unrelated things? Like leaves trembling in the wind. ICM was proposed to eliminate those effects in the first place, but what is this paper's solution to that problem? Furthermore, the experiments on BeamRider show that this concern is not a theoretical one but quite practical.
* "CrazyClimber, Enduro, KungFuMaster, Seaquest, and Skiing" - none of those Atari environments are known to be hard exploration games (which are normally Gravitar, Montezuma Revenge, Pitfall!, PrivateEye, Solaris, Venture according to Bellemare et al "Unifying count-based exploration and intrinsic motivation"). I understand that every game becomes hard-exploration if the rewards are omitted but then there is a question why those particular games. Moreover, if you omit the rewards the question remains how to select hyperparameters of your method. Was the game reward used for selecting hyperparameters? If not, what is the protocol for their selection? This is a very important question and I hope the authors will address this.
* "These games are characterized by moving objects that require the agents to concentrate on and interact with." - this looks like tailoring the task to suit the method.
* Figure 6 - those results are not great compared to the results of Episodic Curiosity: https://arxiv.org/abs/1810.02274 . Maybe this is because of the basic RL solver (A3C vs PPO) but that brings up another question: why are different solvers used for different tasks in this paper? PPO is normally significantly better than A3C, why not use throughout the whole paper?
[minor concerns]
* Figures are very small and the font in them is not readable. Figure 2 is especially difficult to read because the axes titles are tiny.
* "complex or spare reward" -> sparse
* "However, RND does not consider motion features, which are essential in motivating an agent for exploration." - this is unclear, why are those features essential?
* "We demonstrated the proposed methodology and compared it against a number of baselines on Atari games, Super Mario Bros., and ViZDoom." - please state more clearly that only 5 out of 57 Atari games are considered, here and in the abstract.
* "Best extrinsic returns on eight Atari games and Super Mario Bros." - but only 5 games are shown, where are the other 3?

Suggestions on improving the paper:
1) Better motivating the approach in the paper would help. Why using the flow prediction error as a curiosity signal?
2) Better motivating the choice of the environments and conducting experiments on more environments would be important for evaluating the impact of the paper.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Well motivated paper

The authors study the problem of exploration and exploitation in deep reinforcement learning. The authors propose a new intrinsic curiosity-based method that deploys the methods developed in optical flow. Following this algorithm, the agents utilize the reconstruction error in the optical flow network to come up with intrinsic rewards. The authors show that this approach boosts up the behavior of the RL agents and improves the performance on a set of test environments. 

A few comments that I hope might help the authors to improve the clarity of their paper. 

1) While the paper is nicely written, I would encourage the authors, of course, if they think necessary, to make the paper slightly more self-contained by explaining the optical flow problem, FlowNet, and warping approach. While a cruise reader might be required to either know literature in optical flow or go and study them along with this paper, it might be helpful for a bit more general readers to have these tools and approaches in access.

2) Regarding the first line of introduction, I would recommend to rephrase it to one imply that the mentioned "aim" is one of the aims of the DRL study. 

3) In the fourth line of the intro, the authors mention that the current DRL methods are "constraint" to dense reward. I believe the authors' aim was to imply that these methods perform more desirably in dense reward settings rather than being constrained to such settings.

4) I would also recommend to the authors to elaborate more on the term "attention area" Greydanue et al 2018.

5) It would be helpful to have a better evaluation of this paper if the authors could clarify and motivate the choice of games in their empirical study. For example the empirical study in Fig 5.


6) While I find this study interesting and valuable, the novelty of the approach might fall short to be published at a conference like ICLR with a low acceptance rate. This does not mean that there is anything unscientific about this paper, in fact, the scientific value of this work is appreciated and this work adds a lot to the community. 

7) It would also be useful to explicitly explain the advances of this approach over the next frame predictions approaches in stochastic environments. And also, if there is a shortcoming, what are those.

8) Also, what the authors think would happen when the action directly does not change the scene, at least immediately. 





review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 

Pros
Solid technical innovation/contribution: 
- The paper proposed a novel method FICM that bridged the intrinsic reward in DRL with optical flow loss in CV to encourage exploration in an environment with sparse rewards. To the best of my knowledge, this was the first paper proposed to use moving patterns in two consecutive observations to motivate agent exploration.

Balanced view: 
- The authors discussed both the advantages of FICM and settings that FICM might fail to perform well, and conducted experiments to better help the readers understand such nuances. Such balanced view should be valuable to RL communities in both academia and industry.

Clarity:  
- In general this was a very well-written paper, I had no difficulty in following the paper throughout. The proposed method (FICM) was clearly motivated, and the authors provided good coverage of related works. Notably, the authors reviewed two relevant methods upon which FICM was motivated, which made the paper self-contained.


Cons
Experiments:
- Experiments were conducted only using a few recent results as baselines (ICM, forward dynamics, RND). It would be interesting to compare FICM against simpler exploration baselines such as epsilon-greedy or entropy regularization.
- I’d also like to see more extensive comparisons between FICM and ICM across different datasets, for example, Super Mario Bros. and the Atari games, instead of only comparing FICM against ICM on ViZDoom.

Significance of the innovation: 
- The proposed exploration method seemed to be applicable with a particular RL setting: the environment changes could be represented through consecutive frames (e.g., video games), and optical flow could be used to interpret any object displacements in such consecutive frames. And as the authors discussed, even under such constraints the applicability of proposed method depends on how much changes of the environment were relevant to the goal.

Reproducibility:
- Although the authors discussed the experiment setting in detail in supplements, I believe open-sourcing the code / software used to conduct the experiments would be greatly help with the reproducibility of the proposed method for researchers or practitioners.




Summary
A good paper overall, but the experiments were relatively weak (common for most ICLR submissions) and the novelty was somewhat limited.


review_title: Official Blind Review #1

Meta-review: This paper proposes a method for improving exploration by implementing intrinsic rewards based on optical flow prediction error. The approach was evaluated on several Atari games, Super Mario, and VizDoom.

There are several strengths to this work, including the fact that it comes with open source code, and several reviewers agree it’s an interesting approach. R1 thought it was well-written and quite easy to follow. I also commend the authors for being so responsive with comments and for adding the new experiments that were asked for.

The main issue that reviewers pointed out, and which I am also concerned about, is how these particular games were chosen. R3 points out that these 5 Atari games are not known for being hard exploration games. Authors did conduct further experiments on 6 Atari games suggested by the reviewer, but the results didn’t show significant improvement over baselines.

I appreciate the authors’ argument that every method has “its niche”, but the environments chosen must still be properly motivated. I would have preferred to see results on all Atari games, along with detailed and quantitative analysis into why FICM fails on specific tasks. For instance, they state in the rebuttal that “The selection criteria of our environments is determined by the relevance of motions of the foreground and background components (including the controllable agent and the uncontrollable objects) to the performance (i.e., obtainable scores) of the agent.” But it doesn’t seem like this was assessed in any quantitative way.  Without this understanding, it’d be difficult for an outsider to know which tasks are appropriate to use with this approach. I urge the authors to focus on expanding and quantifying the work they depict in Figure 8, which, although it begins to illuminate why FICM works for some games and not others, is still only a qualitative snapshot of 2 games. I still think this is a very interesting approach and look forward to future versions of this paper.
Decision: Reject