Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proposed latent optimistic value exploration (LOVE) as a mechanism to leverage optimistic exploration for continuous visual control. The main idea is to use a small (~5) ensemble of latent models with shared encoders (and therefore shared learned latent space) but different transition, reward and value models. The variance of predictions from this ensemble can be used as uncertainty estimates of each action sequence while the mean is the typical policy learning objective. Then LOVE puts more weighs on the states with high variance during exploration to enforce the agent to visit (optimistically) uncertain states. This is inherently optimistic because there is a positive bias (beta > 0 and var > 0) addition to the expected reward. 

The idea of the paper is not novel and it has been explored before such as (T. Kurutach et al ICLR 2018) and unfortunately the authors did not provide enough context in the related works to distinguish their work with previous research. However, in its proposed form,  LOVE  is new to the best of my knowledge. Using the variance of predictions of an ensemble of agents is an interesting way of approximating uncertainty and the experiments demonstrate how this bias can improve the results. However, I find the experiments not to be convincing enough that the added complexity is necessary to achieve the improved performance:

First, the main claim of the paper is that the proposed method achieves better score by exploring better. However, there are many changes that can cause this improvement. For example, using an ensemble of the models with different transition, reward and value models is essentially using a bigger (i.e. with more parameters) model. There is an ablation study that demonstrates only using an ensemble doesn't work (LVE alternate), however, this is not convincing that the improvements are coming from better *exploration*. This can be done in multiple ways such as demonstrating that a base method (e.g. Dreamer) works better if trained on the data collected by LOVE. An ablation study with a negative beta can be also super helpful (although I believe the results of that is kinda trivial).

Second, there is no study that visualizes the importance of the ensemble. The authors used 5 particles throughout the experiments but why 5? I'm not suggesting optimizing this number as a hyper-parameter or anything like that but what I'm looking for is a study that clearly demonstrates that having a more accurate approximation of uncertainty is important. This can be achieved by studying the effect of number of particles on the performance of the agent. Visualizing the actual variance of the predictions and demonstrating that the predictions actually vary is also important. This tightly related to various values for beta which also requires another ablation study. There are also many other way of enforcing optimism that the proposed method can be compared to.

Overall, the authors are on the right track. The paper (except for related works) is very well written and easy to read. However, more experiments are required to clearly demonstrate why the method is working and how. 
review_title: On the right track but more experiments are necessary

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
--------------

This paper proposes LOVE (Latent Optimistic Value Exploration), a model-based exploration algorithm for POMDP or pixel-based control systems. The method builds upon Dreamer (Hafner et al. 2019) for learning latent models, and the variance of value estimates (one transition/reward/value model per particle) estimates the epistemic uncertainty of the world, which can be used as a UCB-style exploration reward. LOVE can achieve a comparable or better performance on a simple pointmass environment with a trap and standard DeepMind continuous control suites.


Strength:
- This paper studies a high-significance problem of learning representation for visual-based control and deep exploration. Self-supervised learning of latent representation and model-based exploration is an important and timely research problem to study.
- Overall, the paper is written well, with a clear motivation and a good organization of the method. Presentation in terms of pseudocode, plot, table, and hyperparameters look great.
- The method is straightforward and the idea behind the algorithm is very reasonable.

Weakness
- The paper has limited novelty; the core idea of model-based planning and disagreement-based exploration are taken from existing works, though combining them and making them work in a challenging pixel-based continuous control tasks would be a non-trivial work.
- Crucially, there are a few of *very* similar works in model-based exploration, which also builds upon Dreamer (please see the detailed comments below). There might be some technical difference, but a comprehensive comparison with existing works would be critical.
- It is not clearly discussed or mentioned in the paper why the proposed method can be called as a “deep” exploration, or how beneficial it would be compared to “shallow” explorations.

Detailed comments:
-----------------

There are some similar works in the model-based exploration literature, which this paper did not cite or compare with.

Plan2Explore (Sekar et al., ICML 2020): this paper learns a Dreamer-like model, where an ensemble of $q(h | s, a)$ is learned and the variance of prediction (of the latent variable $h$) is used as the uncertainty, or the “disagreement” intrinsic reward. The difference to LOVE would be, the uncertainty is measured by estimation of value function or latent representations. It will be interesting to see how these approaches can be compared.

Value estimation might be much harder than the transition model, especially when the task reward is sparse, and would not be directly applicable in a reward-free/unsupervised setup. In such scenarios, how can LOVE learn to explore the world?

More related works to consider:
- Ready Policy One (RP1; Ball et al., arXiv 2020)
- Model-based Active Exploration (MAX; Shyam et al., ICML 2019)
- Learning Awareness Models (Amos et al., ICLR 2018)

In terms of experiments/empirical evaluation, there is no baseline exploration algorithm presented that is based on Dreamer or pixel-based control, which makes the effectiveness of the proposed approach a bit difficult to be assessed. For example, what if we do a straightforward exploration (such as Curiosity [Pathak et al. 2017], RND [Bruda et al., 2018]) on Dreamer?


**Additional Comments**

- It would be great to provide more clarification/explanation on how LVE (LOVE with $\beta=0$) and Dreamer are different.
- Additional analysis and experiments would also have strengthened the paper, such as: how does the algorithm performance differ under different values of the planning horizon? How does the performance of the algorithm vary under different number $M$ of ensembles?
- As in Hafner et al., 2019a, it would be great to clarify the difference between $p(\cdot)$ and $q(\cdot)$ in their meaning, i.e. $p$ for distributions that samples from real environments, and $q$ for approximations.

review_title: Interesting problem and results, but lacking comparison with similar works

final_score: rating: 6: Marginally above acceptance threshold
review_content: The proposed method seems like a simple and effective combination of existing ideas and approaches. Overall while I like the approach but am leaning towards a reject since I think there are aspects in the experiment section that can be clarified and improved. However I am open to changing my decision if my concerns are addressed during the rebuttal period.

Strengths:
- The ideas presented are simple extensions of existing methods and are quite easily digested.
- The proposed approach does seem to learn faster on some of the tasks considered although some aspects of this need clarification.
- The experimental details presented in the Appendix are quite thorough and did help me understand some aspects of the work in more detail.

Issues and points worth clarifying:

- Apart from a number of points of clarification that are listed later, my main concern is with the experimental section of the paper. This work compares performance of the proposed method on 8 of the DeepMind Control Suite domains and one toy domain with very little ablation. In contrast, the original DrQ and DREAMER work which are used as baseline here show results on 15 and 20 control suite tasks on top of some results on Atari. 
- The number of domains becomes somewhat more important in light of the fact that on 2 of the 8 domains considered DrQ outperforms the proposed approach. The explanation regarding these 2 tasks having dense rewards does make sense but more data would help substantiate the claim.
- Another concern I have is with regard to the presentation of the results in Figure 3. Most of the curves are cut off before they have converged. This makes it hard to  sanity check against the existing results from the DREAMER and DrQ paper and it remains unclear to me if LOVE performs asymptotically as well as the baselines on some of the tasks. Perhaps this information could be included instead in Table 1 which as far as I can tell does not currently present any new information that is not present in Figure 3.
- Finally the paper could also include more ablations - especially regarding the effect of the ensemble size and how important the step increase schedule is for the beta parameter of the UCB objective.

Finally there are some points which I don’t fully understand based on my reading of the text and for which clarification would greatly help.
- The main text mentions that LOVE has exploration noise turned off since exploration happens through the UCB noise. Was this the same setting used for the LVE where beta is set to 0? My understanding is that LVE does have exploration noise but it would be good for the authors to confirm this.

- The parameter in the UCB objective Beta is progressively increased from 0 in delta steps of 0.001. Where does this number come from? Was this a hyperparameter?

- Learning multiple DREAMER models is an interesting idea but the obvious issue with this is that it could bloat wall clock learning time. How much slower is the proposed approach to vanilla DREAMER  in terms of wall clock time? While understandably the focus of this work is data efficiency, I think this is an important number to mention perhaps even in the Appendix to paint a more complete picture.

--Update (Nov 25)--

I am happy that the authors improved the paper with reviewer feedback. In particular I think the new ablations and comparison and mention of previous work makes the work more complete. The results on new sparse tasks are also interesting. I still think more can be done in terms of experimental validation (in particular my original note regarding early cutting of the curves has not been addressed). However overall I think the paper does meet the acceptance threshold as things stand.

review_title: This paper presents a method called Latent optimistic value exploration (LOVE) that combines a latent variable world model (DREAMER) with ensembling and a UCB objective for optimistic exploration under uncertainty.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---- Summary ----

The paper proposes LOVE, an adaptation of DOVE (Seyde’20) to latent variable predictive models (Seyde’20 only condsidered predictive models without latent variables). Seyde’20 proposes to use a generalization of Upper Confidence Bound to deep model-based RL, by training an ensemble of models and value functions, and training a policy to maximize mean + variance of this ensemble (similarly to Lowrey’19). The submission empirically demonstrates that tuning the learning rate and number of training steps per environment steps of Dreamer (Hafner’20) improves sample efficiency, using an ensemble of predictive models further improves data efficiency slightly (on cartpole and walker tasks), while on top of that the proposed exploration method slightly improves sample efficiency on the Hopper and sparse Cartpole tasks. 

---- Decision ----

The submission contains little technical novelty over prior work of Seyde (2020). The experimental results are weak, but somewhat justify the claims as there is a slight but consistent improvement on some tasks. However, the paper suffers from a major flaw in the empirical evaluation. Figure 3 and the relevant discussion describe LOVE as significantly outperforming the Dreamer baseline. This difference is largely due to the fact LOVE uses a different learning rate and number of epochs, which improves sample efficiency. The paper graciously provides the comparison to the fairly tuned baseline in the appendix as Figure 9, confirming this. The fairly tuned baseline needs to be moved from the appendix to the main paper, and the contribution section and the discussion of the experiments need to be rewritten accordingly. If this is provided, I will reevaluate the paper. In the current state of the paper, I am unable to consider its merits on the basis of this flaw.

---- Strengths ----

The paper is technically correct (except for the flaw explained above), and proposes a promising approach to a relevant problem of exploration in RL from images. The experimental results indicate that the proposed method could be effective.

---- Weaknesses ----

The major flaw of the paper is described earlier. In addition, there are two other major issues with experimental evaluation.

The experimental evaluation of the paper is rather weak. The proposed exploration method only improves performance in 2 out of 8 environments. This might be because the other environments do not require sophisticated exploration, in which case the method needs to be tested on more than 2 relevant environments. Sparse-reward versions of the evaluated environments can be easily designed and would be suitable for evaluating the method.

The second major issue is that the method is not evaluated against any competing exploration baselines, even though the paper cites multiple prior works on this. For instance, the paper claims that methods based on information gain or improving knowledge of the dynamics will not explore as efficiently as the proposed method. Both of these claims need to be empirically evaluated or toned down.

---- Additional comments ----

The related work section is missing the following papers: 
- Ball’20 is a model-based RL method that uses UCB for exploration
- Sekar’20 is a model-based RL method that uses latent ensemble variance for task-agnostic exploration

Ball’20, Ready Policy One: World Building Through Active Learning

Sekar’20, Planning to Explore via Self-Supervised World Models

## Update 

The new sparse tasks and comparison to Dreamer + Curious improve the paper and address some of my concerns. Specifically, a sizable improvement due to exploration is now seen on 3 tasks, Hopper, Cartpole Sparse, and Cheetah Sparse. The new maze task is also more challenging than the bug trap task.  

--- Final Decision ---

After the significant improvements in the experimental evaluation, I believe the paper provides a reasonable case for the proposed latent UCB method. It also provides an interesting discussion on the advantages of UCB-style methods, and an interesting observation that optimistic reward-based exploration can be effectively used even in absence of (positive) rewards. Even though the experimental evaluation of prior work on exploration is still rather lacking, I believe that these contributions are enough for the paper to be interesting to the ICLR audience. I raise my score to 6.

--- Remaining weaknesses ---

The experimental evaluation in the paper is still quite lacking in terms of baselines, making it impossible to judge whether the paper actually works better than prior work.

First, the proposed method contains two improvements, model ensembling, and optimistic exploration, but doesn't go much in-depth analyzing either of these improvements, instead trying to focus on both at the same time. This makes comparison to prior exploration methods hard because the proposed method receives an additional boost due to an ensemble of dynamics (the paper conveniently quantifies this boost in the LVE method, and it is shown to be rather large). For a more fair comparison, the ensemble of dynamics might be ablated (leaving only the ensemble of value functions), or the competing baselines could also be built on top of LVE.

Second, the paper only compares against one competing exploration method, Dreamer + Curious. There has been a large amount of proposed exploration methods, and it would be appropriate to evaluate the proposed UCB method against at least a few of them. For instance, the paper could compare against similar value function ensemble techniques (Osband'16, Lowrey'18, Seyde'20), or other cited work (Ostrovski'17, Pathak'17). Burda'18 is not cited, but perhaps should be compared against. All these methods can be relatively easily implemented on top of LVE for a fair comparison.

Burda'18, Exploration by random network distillation.

--- Additional comments ---

Would be great to clarify what is the observation space for the bug trap and maze tasks. For instance, you could add observations and predictions for these tasks to the appendix.
review_title: Review 1: Flawed evaluation, needs significant updates

Meta-review: The submission is acknowledged as having potential value in terms of proposing a new approach for exploration based on ensembles and value functions. However, there are lingering concerns about the discussion of what this paper brings to the table vis-a-vis prior work, together with a lack of clear demonstration of the explicit gains from the exploration mechanism and more experimental studies. The author(s) would do well to revise as per the feedback given and resubmit a version with a more compelling argument. 
Decision: Reject