Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Sequence-level intrinsic exploration model for partial observable domains
This paper tackles the problem of RL in partially observable domains with sparse rewards. To address the sparse rewards issue, it proposes a sequence level intrinsic novelty model to guide policy learning. The sequence model is based on a dual-LSTM architecture. In general, this paper is well-written as easily accessible. Comprehensive experiments are provided to validate the effectiveness of the proposed methods. 
The main issue with the paper is lacking discussions regarding the effective of the biased incur by the intrinsic reward. Specifically, 
1)	How is the scaling factor beta determined? It would be nice if some discussions or experimental comparisons can be provided. 
2)	The paper mainly deals with problems with sparse rewards. I wonder how the proposed method perform will in non-sparse rewards cases.  My main concern is that in the non-sparse reward cases, the intrinsic reward will cause bias, which may not guarantee good final performance.  


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper extends the prediction-error based model by Pathak et. al., 2019 by learning a forward (and inverse) dynamics model for predicting a state feature multiple steps into the future (say, K-steps) given an open loop sequence of K actions as opposed to 1 step into the future, with the caveat that instead of using learnable state features, a random network is used for computing state features, similar to Random Network Distillation (RND) by Burda et. al., 2019. Also, their inverse dynamics models predicts the entire sequence of actions up to K steps. Experiments on VizDoom point-navigation tasks show that the proposed model does better than baselines as rewards get sparser. Ablations are provided to justify the choice of K in multi-step prediction, the choice of inverse dynamics and the choice of RND state features.

My decision is weak reject as:

1. The paper does a good job at clearly explaining their model, presenting results on relevant experiments and baseline comparisons for their model and justifying each modeling choice with ablations.

2. The novelty in their contribution is moderate - the idea of long-term future prediction is not new (e.g.: Ke et. al., 2019) (but using it for giving a curiosity bonus is new), the architecture choice is not significantly new.

3. I expected a more detailed ablation for the choice of K, given that “multi-step” has major emphasis in the paper, but Figure 7(a) only shows ablations for 3 vs 1 step predictions.


Elaborating on (3):
- My major concern is that the gap between 1-step prediction and 3-step prediction (in Figure 7 (a)) is not significant. Note that the version of their model with 1-step predictions does not completely reduce to Pathak et. al. 2019’s ICM model, as RND state features are used. I feel that it may be the case that the 1-step version of the proposed model is actually good enough to beat all the baselines and adding multiple steps gives marginal gains. This hypothesis needs to be verified by the authors with more experiments - I would like to see all the main experiments have an additional baseline of 1-step predictions. 

- Only two values of K are tested - 1 and 3, what happens with larger values of K?

Other comments:
The motivation for using long-term predictions to “infer more meaningful novelty” is fine on it’s own but seems to conflict with the choice of random network (RND) state features. Random features imply that a random “hash” of the observations is being computed which has no reason to have similar features for two nearby states. If there is any small amount of noise in the state transitions, this would mean that predicting far into the future is practically impossible given that the random feature of slightly incorrect states will be very different. Can the authors give reasons/motivations as to why such a model would work in the case of stochastic transitions and K is large or will it be brittle to stochasticity?

References:
All references are same as those cited in paper.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
The authors tackle the exploration problem by introducing SIM (Sequence-level Intrinsic exploration Module). In most existing literature, intrinsic motivation bonuses are scored based on individual states or transitions, and not over multi-step trajectories. SIM predicts novelty bonuses based on the prediction error of an open-loop forward dynamics model - the model consumes as input a sequence of observations (without paired actions) followed by a sequence of actions (without paired observations) to predict a feature vector associated with the next state. The error between this feature vector and the RND embedding of the true observation is used as a novelty bonus. 

Overall, the paper is easy to follow and well-motivated. The main experimental results show a reasonable improvement over baselines (RND, ICM). While the model contains many moving components that may seem ad-hoc, the ablation studies show the beneficial effect of each of the individual modeling choices (specifically, using multi-step predictions, the auxiliary inverse dynamics loss, and RND embedding). It would be nice if experiments could be performed on a more popular benchmark such as Atari, but overall I think this is an interesting paper with a reasonable contribution.

For additional motivation, "Why is posterior sampling better than optimism for reinforcement learning" (Osband & Van Roy 2016) offers some justification on the downsides of modeling novelty bonuses for each state/action independently.
review_title: Official Blind Review #2

Meta-review: This paper introduces a new architecture based on intrinsic rewards, to deal with partially observable and sparse reward domains.

The reviewers found the novelty of the work not particularly high, and had concerns about the general utility of the method based on the empirical evidence. This paper has numerous issues and could use significant revision in terms of writing, connections to literature, experiment design, and clarity of results. 

Much of the discussion focused on the scaling parameter. From an algorithmic point of view, the scaling parameter is very problematic. It is domain specific and when tuned per domain resulted in very different values. The ablation study showed that only two settings in one domain led to good performance, whereas the other resulted in no learning (for some reason the other two values were not plotted). 

There are concerns that the baselines were not completely fair. In many cases different domains were used to compare against RND and ICM, and there appears to be no tuning of these baselines for the new domains---this a problem due to the inherent bias in favor one's own method. In the solaris domain which was used in the RND paper, the results don't appear to match the RND paper, and in vizdoom the performance numbers are difficult to compare for ICM because a different metric is used---even if you don't like their performance numbers at least report them once so we can be confident the baselines are well calibrated. One reviewer pointed out the meta-parameters where different for RND than the published previous, but the paper does not describe what approach was used to tune those parameters and this is not acceptable. We cannot have much confidence that these results are reflective of those methods. Finally, there is no comment on how the performance numbers were computed and no description of how the errorbars where computed or what they represent. 

The paper focuses on partially observable domains, the evidence that this method is effective in closer to Markov settings is unclear. The Atari experiments do not yield significant results by large (solairs looks as if there is no learning occurring at all---a no comment about it in the text to explain). The paper claims evidence the approach can work well in both cases, but it was not even indicated if frame-stacking was used in the Atari experiments. In fact, the result was only alluded too in the conclusion---there was no reference in the main text to a specific result in the appendix. Text is very challenging to read. The language is informal and imprecise, and the paper frequently uses terms incorrectly or in different ways through (e.g., the use of the term novelty throughout)

This is clearly an interesting direction. The authors should keep working, but this paper is not ready for publication. I urge the authors to dig deeper in the literature to gain a more nuanced understanding of the topic. Barto et al's excellent paper on the topic is a great place to start: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3858647/ 
Decision: Reject