Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This paper proposes an auxiliary reward for model-based reinforcement learning. The proposed method uses ensembles to build a better model of environment dynamics and suggests some rules to optimize the new ensemble-based dynamics and to estimate the intrinsic reward.

I am torn on this paper. I like the derivation of the method and the ideas behind it. I think it is an interesting direction of research. However, the experiments are limited to one domain and the paper needs proofreading. I will vote "weak accept" for this paper, as I think it is incremental and the experiments are too limited.

As I said above, the paper could use some proofreading. Some sections (like pages 1-2) are well written, but others are full of grammatical mistakes. There is also a lot of redundant information.

It is often stated that the ensemble model has better capacity than the single model. Some experimental proof of this better modelling capacity could help convince a reader that the ensemble is indeed beneficial and warranted (e.g., show that P_K is better than the single model P).

Some evaluation or discussion on the computational costs of the method would be beneficial. I assume the ensemble-based method is more computationally intensive. Would it perform better than single surprise if they were compared according to wall clock time?


Examples of minor issues:

- Page 3, after equation 7, sentence beginning with "In addition to the advantage that the mixture model (4) has the increased model order for modelling" is confusing, contains redundant elements, and is not bringing useful information. It should be revised
- page 4, in paragraph after eq. 11: the following sentence is grammatically incorrect, please revise: "Propositions 1 and 2 suggest a way to intrinsic reward design under the mixture model for tight gap between η(π ̃∗) and η(π∗)"
- in the same sentence, revise: "be close to the true η(π∗) of the true optimal policy π∗, as desired.",
- page 6 text above Figure 1: "single-modal" should be unimodal


POST REBUTTAL 

Thanks for writing your rebuttal. I have read it, as well as the other reviews. I think reviewer 1 touches on some important points, especially regarding the engineered sparse rewards. It seems the method is not properly justified given the environments used for its evaluation. Based on this, and the fact that the method is rather incremental, I would like to change my score to a weak reject. The method should be evaluated in a setting­ with truly sparse rewards.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an approach for using an ensemble of learning dynamics models to generate an intrinsic reward for reinforcement learning in sparse reward environments. The paper's results demonstrate that this approach out-performs other benchmark approaches across a set of continuous control tasks.

I'm curious about the motivation for taking the min surprise across the ensemble. Eqs 8-11 seem to motivate that this will make a tighter gap between the expected cumulative returns of the optimal policy and the policy with additional intrinsic rewards. This implies that this will converge to the same policy in the end, but I'm not sure that it implies that this is a good exploration strategy, an intrinsic reward of 0 would provide an even tighter bound. When you used the max or average return, was the problem that there was still too much intrinsic reward for the agent to converge? Or that it wasn't exploring enough or in the right places?

For the experiments, it would be very interesting to see the intrinsic rewards accumulated over time for each approach. That plot could be very enlightening as to what is going on. 

It would also be great to see the results on these tasks when they're not modified to be sparse reward. Is your method a big hindrance in that case? Or does it still help?

Your intro claims that typical model-free RL is about the circumstance where the agent receives non-zero reward for every time step. This is not true, many model-free RL papers look at tasks that have sparse reward on some or many steps.  I would agree that in many continuous control problems, shaping rewards are added to ease the exploration problem. 

For Figure 3, how does the model-ensemble TRPO (Kurutach et al) fit in? Is that algorithm represented by one of the curves?

Here's one more related work deriving intrinsic rewards from an ensemble of trained dynamics models:
http://www.sciencedirect.com/science/article/pii/S0004370215000764
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary

This paper proposes a model-based method for intrinsic rewards based on probabilistic neural network ensembles. For a particular ensemble element, an intrinsic reward is defined as a log-term that measures the deviation in prediction between a Gaussian mixture over all ensemble elements and the particular ensemble element (at the previous update period). The intrinsic reward signal applied in practice is the minimum of the aforementioned quantity w.r.t. all ensemble elements. The authors provide some theoretical motivation for their approach and validate their method in sparse-reward continuous robotics domains (Mujoco), using PPO as reference algorithm. Experiments are averaged over 10 seeds and compared against various other intrinsic reward baselines including PPO without intrinsic motivation. In these experiments, the method provided by the authors seems to dominate over competing approaches.

Quality

I find the quality low. First, while I appreciate that the authors try to provide some theoretical motivation for their method, there is quite a gap from theory to practice. The theory assumes that the world model is known and considers a stationary reward signal. In practice, the world model is approximated with the current network ensemble, a quantity that is changing over time yielding a non-stationary signal. Second, the experiments are conducted in a sparse-reward modification of Mujoco-type environments that are non-sparse by construction. Sparsity is introduced by accumulating instantaneous rewards over some time window before "releasing" them. This way of sparsifying rewards yields weird partial observability issues, e.g. the same state-action pair observed at the right moment in time yields significant reward whereas at the wrong moment in time yields no reward at all. Additionally, it is always guaranteed that there will be a reward signal at a certain frequency. There are probably better environments for studying the approach, like Atari for example. I do understand that the authors cite Oh et al. 2018 who apply the same technique of sparsification, but Oh et al. also conduct additional experiments in ALE.

Clarity

The paper is clearly written and easy to follow. On a side note, it could be stated more clearly that the presented approach is not model-based because no forward prediction is required for constructing intrinsic rewards (merely probability values are queried for observed transitions). There is one question I have though regarding the experiments where different intrinsic reward approaches are compared against each other. The paper states that all approaches normalize intrinsic rewards according to Equation (17)---why do all of them then need a different weighting factor \beta as stated in the second paragraph of Section 3.3? Furthermore, since \beta is fine-tuned for each intrinsic reward approach and each environment, can the authors please elaborate in detail how exactly this is accomplished (to ensure correct interpretability of the results)?

Originality

The originality is low. As stated by the authors, the proposed method is an incremental extension of Achiam and Sastry 2017 who proposed a similar method for non-ensemble methods.

Significance

The significance is low as well. The method's improvement over other intrinsic reward approaches is minor (the environments chosen by the authors are also not ideal). I feel the significance is reduced further by the fact that there are other model-based approaches that use models of the proposed kind for increasing sample-efficiency and performance considerably in non-sparse environments (e.g. Kurutach et al.2018).

Update

After reading the other reviews and the rebuttal, the concerns I have raised remain. I still feel this paper shouldn't be accepted to ICLR. However, given the extensive experimental analysis conducted by the authors, I feel a score of 1 from my side was too harsh. I therefore increase to 3.
review_title: Official Blind Review #1

Meta-review: This paper considers the challenge of sparse reward reinforcement learning through intrinsic reward generation based on the deviation in predictions of an ensemble of dynamics models. This is combined with PPO and evaluated in some Mujoco domains.

The main issue here was with the way the sparse rewards were provided in the experiments, which was artificial and could lead to a number of problems with the reward structure and partial observability. The work was also considered incremental in its novelty. These concerns were not adequately rebutted, and so as it stands this paper should be rejected.
Decision: Reject