Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary
This paper proposes a Rewarding Impact-Driven Exploration (RIDE), which is an intrinsic exploration bonus for procedurally-generated environments. RIDE is built upon the ICM architecture (Pathak et al. 2017), which learns a state feature representation by minimizing the L2 distance between the actual next state feature and the predicted next state feature while minimizing the cross-entropy loss between the true action and the estimated action from the consecutive state features. Finally, RIDE's intrinsic reward bonus is computed by L2 norm of the difference between the current state feature and the next state feature, divided by the square root of the visitation count of the next state within the episode. Experimental results show that RIDE outperforms the existing exploration methods in the procedurally-generated environments (MiniGrd), and is competitive in singleton environments.


Comments and questions:
- In reinforcement learning, the agent should explore the experiment due to uncertainty. If everything in the environment is certain to the agent, then it does not have to explore and just exploiting the past experience would be the best. My major concern about the paper is 'impact-driven' reward bonus may not account for the uncertainty. Constantly encouraging the states that have a high impact would not always good, and it may interfere to converge to an optimal policy.
- It seems that RIDE assumes that 'high-impact' states are always good, thus rewarded. It could be true on the conducted MiniGrid domains, but this assumption may not hold in general. Could 'impact-driven' exploration be realistic and be applied to more general problems?
- Similarly, in the problems where high-impact states have to be avoided, can RIDE still work effectively? For example, how about 'Dynamic-Obstacles' domains implemented in MiniGrid? In this task, RIDE may promote to chase obstacles that have to be avoided, interfering with learning optimal policy. It would be great to show the effectiveness of RIDE in such environments.
- In MiniGrid problems, if the colors of walls and the goal are changed at every episode, does RIDE work well?
- In Figure 4, why the intrinsic reward heatmaps are drawn only on the straight paths?
- Minor: In the last sentence of Section 3, "the current state and the next state predicted by the forward model" -> "the actual next state and the next state predicted by the forward model"


---
after rebuttal:

Thank the authors for clarifying my questions and concerns. Most of my concerns are addressed, and I raise my score accordingly.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new intrinsic reward method for model-free reinforcement learning agents in environments with sparse reward. The method, Impact-Driven Exploration, learns a state representation of the environment separate from the agent to be trained, based on a combied forward and inverse dynamics loss. The agent is then separately trained with a reward encouraging sequences of actions that maximally change the learned state.

Like other latent state transition models (Pathak et al. 2017), RIDE learns a state representation based on a combined forward and inverse dynamics loss. However, Pathak et al. rewards the agent for taking actions that lead to large difference between the actual next state and the predicted next state. RIDE instead rewards the agent for taking actions that lead to a large difference between the actual next state and the current state. However, because rewarding one-step state differences may cause an agent to loop between two maximally-different states, the RIDE loss term is augmented with a state visitation count term, which decreases intrinsic reward for a state based on the number of times that state has been visited in the current episode.

The experiments compare RIDE to a selection of other intrinsic reward methods in the MiniGrid, Mario, and VizDoom environments. RIDE provides improved performance on a number of tasks, and solves challenging versions of the MiniGrid tasks that are not solved by other algorithms.

Decision: Weak Accept.

The main weakness of the paper seems to be a limitation in novelty.
Previous papers such as (Pathak et al. 2017) have trained RL policies using an implicit reward based on learned latent states. Previous papers such as (Marino et al. 2019) have used difference between subsequent states as an implicit reward for training an RL policy. It is not a large leap to combine these two ideas by training with difference between subsequent learned states. However, this paper seems to be the first to do so.

Strengths:
The experiments section is very thorough, and the visualizations of state counts and intrinsic reward returns are insightful.
The results appear to be state of the art for RL agents on the larger MiniGridWorld tasks.
The paper is clearly-written and easy to follow.
The Mario environment result discussed in section 6.2 is interesting in its own right, and provides some insight into previous work.

Despite the limited novelty of the IDE reward term, the experiments and analysis provide insight into the behavior of trained agents and the results seem to improve on existing methods.
Overall, the paper seems like a worthwhile contribution.

Notes:
In section 2 paragraph 4, "sintrinsic" should be "intrinsic".
In section 3, at "minimizes its discounted expected return," seems like it should be "maximizes".
The explanation of IMPALA (Espeholt et al., 2018) should occur before the references to IMPALA on page 5.
Labels for the axes in figures 4 and 6 would be helpful for readability.

The motivation for augmenting the RIDE reward with an episodic count term is that the IDE loss alone would cause an agent to loop between two maximally different states.
It would be interesting to know whether this suspected behavior actually occurs in practice, and how much the episodic count term changes this behavior.
It is surprising that in the ablation in section A.5, removing the state count term does not lead to the expected behavior of looping between two states, but instead the agent converges to the same behavior as without the state count term.

Also, in Figure 9, was the OnlyEpisodicCounts ablation model subjected to the same grid search described in A.2, or was it trained with the same intrinsic reward coefficient as the other models?
Based on the values in Table 4, it seems like replacing the L2 term with 1 without changing the reward coefficient would multiply the intrinsic reward by a large value.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper addresses the problem of intrinsically motivating in DRL. In particular, it focuses on exploration of procedurally generated environments where many states are novel compared to training experiences. It offers an intrinsic reward based on large movement in a state embedding space where this state embedding representation is co-trained on the same data already collected for learning. The paper claims to overcome shortcomings of specific past approaches (e.g. count-based / curiosity).

The need for intrinsic motivation in exploration is well motivated, and the approach for training a state embedding is anchored in multiple past works. The use of movement in this state embedding as an intrinsic reward is importantly novel and valuable. The problematic propensity for RL researchers to train on the test environments or design agents that are confused by proverbial noisy TVs and/or sacrifice extrinsic rewards in favor of intrinsic rewards is satisfyingly discussed and addressed through detailed experiments.

This reviewer moves to accept the paper for its contributions to intrinsically motivated exploration with thorough discussion of how the technique addresses shortcomings of past methods. This reviewer is thankful that the authors do not overinterpret the MiniGrid results and that they provide intuition for why the state embedding functions capture what we want them to capture. The fact that this approach makes joint use of the whole (s,a,r,s') tuple feels significant, as does the fact that this approach does not require any changes to the policy network (e.g. presuming that features useful for computing intrinsic rewards are also going to be useful for directly acting to optimize extrinsic rewards).
 
Question:
- In partially observable environments that require agents to wait for something, should a RIDE-motivated agent consider changes in its own internal clocks (part of the recurrent state) impactful moves? If an environment might require a recurrent / history-aware action policy, should RIDE also be made history aware? Might a history-aware RIDE reward sufficiently motivate a stateless/reactive policy?
review_title: Official Blind Review #2

Meta-review: This paper tackles the problem of exploration in deep reinforcement learning in procedurally-generated environments, where the same state is rarely encountered twice. The authors show that existing methods do not perform well in these settings and propose an approach based on intrinsic reward bonus to address this problem. More specifically, they combine two existing ideas for training RL policies: 1) using implicit reward based on latent state representations (Pathak et al. 2017) and 2) using implicit rewards based on difference between subsequent states (Marino et al. 2019).

Most concerns of the reviewers have been addressed in the rebuttals. Given that it builds so closely on existing ideas, the main weakness of this work seems to be the novelty. The strength of this paper resides in the extensive experiments and analysis that highlight the shortcomings of current techniques and provide insight into the behaviour of trained agents, in addition to proposing a strategy which improves upon existing methods.

The reviewers all agree that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)