Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors present a method to guide exploration that prefers to go to areas of the state space for which it is more uncertain. This uncertainty is obtained by measuring the standard deviation of the next state prediction from an ensemble of models. The authors call this the disagreement measure At each step, a search is performed and the disagreement measure is obtained for each state visited. The disagreement measure for each action is compared to the distribution for all the states visited during the search. It it is above some threshold, then the action that maximizes the disagreement measure is taken. Otherwise, it takes the action determined by the search.

The algorithm presented was unclear. What does planner.choose_action do? Is the heuristic for best-first search (BFS) the disagreement measure? I don't understand how this should help the algorithm pick a good action to take.  The paper says, "The proposed action is the first edge on the shortest path to the best node in the subgraph searched so far." How is the best node determined?

Furthermore, is this search necessary? It seems like it is mainly used as a comparison for the disagreement measure. What if the agent behaved greedily with respect to the disagreement measure all the time. Pathak et al. (2017) used a similar method, but with an inverse model.

I am not quite sure about the comparisons the authors are making. In the case of BFS search, what does it mean to do BFS search with epsilon greedy? Also, this is an artificial curiosity method where curiosity is measured by the disagreement between the ensemble of models. However, there are no comparisons to other curiosity papers, such as Pathak et al. (2017).
review_title: A method for artificial curiosity using model uncertainty

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work tackled the problem of model-based RL in environments where the reward is sparse and many actions are needed to achieve some. Particularly, the authors tried to solve the issue of one-step false loop in the model, which avoids further exploration. Measuring the uncertainty about the built model through ensemble of models, they added a possibility of choosing an action different from what planner suggests, promoting exploration. The work is very-well written in general, especially sections of problem definition and related work. I also appreciate that the proposed method is compared with multiple planners and tested on two different tasks. Having said that, the main missing analysis for me is that the method was not tested on environments where false loop does not exist. Given the nature of the problem definition, i.e. learning the environment, it is counter-intuitive not to test the method on a few standard test-benchmarks without any assumption. The proposed method does not have to get the best result on environments without false loop, but it is important to see how it behaves when the built model is already good. Other than this, I have a few more questions/concerns: 

1) The RANDOM parameter seems a little strange, especially because it looks too high, i.e. .5. Some analysis on different values of this (or just with and without RANDOM) on performance would be great. Also, I suspect that change in RANDOM would also change the best QR. I think a plot similar to figure 7, but for RANDOM and combination of RANDOM and QR would improve the paper.

2) The method is about one-step false loop. I would appreciate if the authors talk about multistep false loop briefly. Could it be problematic in learning? If yes, could an extension of this method work?

3) Have the authors considered a QR that changes with number of steps? 
review_title: Interesting and well-written paper on model-based exploration

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a new method which can be combined with graph search algorithms to boost exploration when the uncertainty is high. This new mechanism, called TBV, can override actions given by the model to explore and verify model predictions. It is also shown in the experiments that TBV improves the model performance when combined with MCTS or BestFS. TBV utilizes graph structure of the problem and finds the solution much quicker for both MCTS and BestFS. 

While the presented method is interesting with high performance, I found many editorial errors in the writing. For example, in the second paragraph of section 3.3, ‘we concentrated of exploration….’, and ‘In our experiments, such a version proved to be effective in in discrete…’, just to name a few. There are so many errors like this and the paper needs serious rewriting. Also, having a conclusion or discussion can help the structure of the paper. 

Figure 3 is unclear if the blue line is without TBV with the legend ‘Right corridor visited’. It could be interesting to discuss extension of TBV into continuous environments. Reference format seems to have errors since there are underlines. 
review_title: Trust, but verify: model-based exploration in sparse reward environments

final_score: rating: 2: Strong rejection
review_content: This paper proposes an approach for encouraging exploration when planning over learned models of discrete reinforcement learning environment. The proposed method involves using an uncertainty-aware model (e.g., an ensemble of neural networks) to predict state-action transitions, together with a graph-based planner operating on this model. The key idea is to replace (with some probability) the planner's action with the action leading to the highest uncertainty in model prediction. The paper evaluates the proposed technique using two standard search planners (MCTS and BFS). 

Unfortunately, I think the significance and technical contribution of this work is minimal, an issue that mostly likely starts from a deficient literature review. What the authors refer to as Trust-But-Verify, it's just an ad-hoc instance of the well-known principle of *optimism in the face of uncertainty*, which underlies classic bandit and RL algorithms such as UCB1[1], UCT[2], Thompson Sampling [3, 4]. In the model-free setting, this idea has lead to numerous recent algorithms, many of which also use ensembles for uncertainty quantification [5-8]. In the model-based setting there are also some precedents of work using similar ideas [9-11]. There is also a large body of work treating the problem from the point of view of Bayesian RL (see [12] for a survey).  It is a bad sign that none of this body of previous work was discussed in the paper, which I would argue was the more relevant literature upon which the paper had to be positioned.

This could conceivably be excused if the paper technical and experimental contribution was impressive enough, but this is not the case.  In contrast to the literature outlined above (where proposed exploration strategies typically follow from rigorous statistical analysis), this work presents the proposed method as a heuristic rule, providing no insight as to why one should expect the approach to work well in general. Moreover, the experiments are done in relatively simple domain, and compared against simple baselines. Some of the baseline choices do not seem appropriate either. For example, why use $\epsilon$-greedy for exploration, instead of more robust strategies using upper confidence bounds? 

Finally, the writing on the paper can be improved in many places. For example, the paper refers to using the graph structure of the underlying problem, but what this graph structure refers to is not properly defined anywhere in the paper. I imagine it refers to the graph wherein edges represent non-zero probability transitions between states, but this is not clear from the text. Additionally, some paragraphs add little in terms of content. For example, the first paragraph of Section 3 is devoted to describe the basic problem that all model-based RL methods are trying to solve; this issue is ubiquitous so there is no need for a full example and so much text to describe this. Other sentences are unclear, such as "The optimistic and pessimistic errors are often of the same nature", which I am not sure what is referring to . Additionally, I didn't see a description of the learned model used in the experiments, which is not an obvious choice, since the environment is discrete. 

Overall, to end on a somewhat constructive note, I think the problem the authors are trying to solve is interesting and the proposed approach is based on the right intuitions. However, this work is still too immature for publication. I suggest to the authors to position their work properly with regards to the relevant literature, refine their technical contribution accordingly, and compare with more appropriate baselines.  

----------------------------------------------------------------------
[1] Auer, Peter, Nicolo Cesa-Bianchi, and Paul Fischer. "Finite-time analysis of the multiarmed bandit problem." Machine learning 47.2-3 (2002): 235-256.

[2] Kocsis, Levente, and Csaba Szepesvári. "Bandit based monte-carlo planning." European conference on machine learning. Springer, Berlin, Heidelberg, 2006.

[3] Thompson, William R. "On the likelihood that one unknown probability exceeds another in view of the evidence of two samples." Biometrika 25.3/4 (1933): 285-294.

[4] Russo, Daniel, et al. "A tutorial on thompson sampling." arXiv preprint arXiv:1707.02038 (2017).

[5] Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems (pp. 1471-1479).

[6] Ostrovski, G., Bellemare, M. G., Oord, A., & Munos, R. (2017, July). Count-Based Exploration with Neural Density Models. In International Conference on Machine Learning (pp. 2721-2730).

[7] Osband, I., Blundell, C., Pritzel, A., & Van Roy, B. (2016). Deep exploration via bootstrapped DQN. In Advances in neural information processing systems (pp. 4026-4034).

[8] Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., ... & Blundell, C. (2017). Noisy networks for exploration. arXiv preprint arXiv:1706.10295.

[9] Sanner, S., Goetschalckx, R., Driessens, K., & Shani, G. (2009). Bayesian real-time dynamic programming. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI-09) (pp. 1784-1789). IJCAI-INT JOINT CONF ARTIF INTELL.

[10] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. "Self-supervised exploration via disagreement." arXiv preprint arXiv:1906.04161 (2019).

[11] Shyam, Pranav, Wojciech Jaśkowski, and Faustino Gomez. "Model-based active exploration." International Conference on Machine Learning. 2019.

[12] Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2016). Bayesian reinforcement learning: A survey. arXiv preprint arXiv:1609.04436.
review_title: Improperly positioned, minor technical contribution, unclearly written

Meta-review: After reading the meta-reviews and the authors comment, the meta-reviewer thinks the paper is not ready for publication in a high-impact conference like ICLR. The paper is not well positioned with respect to the literature, and the proposed techniques are not well discussed in relation with predominant paradigms like optimism in the face of uncertainty.
Decision: Reject