Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel exploration strategy for RL that uses intrinsic rewards based on predicting long term values of random vector values rewards. The proposed approach contains the common RND exploration algorithm as a special case and is shown to work well on standard benchmarks, even when not given privileged access to episodic counts.
strengths_and_weaknesses: +Simple idea that seems to work well and generalizes prior work

+Good results on exploration benchmarks

+Nice use of ensembles to deal with aleatoric and epistemic uncertainty

-The environment details are sparse for someone who hasn't used these benchmarks before

-It is unclear how sensitive the algorithm is to the weights on true reward and intrinsic reward for RL?
questions: Figure 1: I appreciate the idea of having a simple motivating example, but the details are too sparse. How does the agent get to the end to see the spike? It seems that even predicting future rewards gives no signal until the very end so it would still appear to be a very hard exploration problem without a shaped true reward. What is the statespace here? is it just white and blue? x-pos? What is the task reward?

Line 226: Transitions from "dead" row are unclear. From the current description it seems that  an optimal policy be to take a bad action H-1 times (total reward 0) followed by a good action at the last timestep?

Line 232: I don't understand why you can't just always take the good action. It seems like this deterministic policy works regardless of what state you are in

How sensitive is the algorithm to the weights on true reward and intrinsic reward for RL?

limitations: yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose Random Curiosity with General Value Functions (RC-GVF) that essentially extends the intrinsic rewards obtained through Random Network Distillation (RND) to temporally extended outcomes of action sequences, in order to tackle long-horizon hard-exploration tasks in partially observable environments. To deal with aleatoric uncertainty, an ensemble of predictors is trained and the ensemble disagreement is combined with the prediction error as a multiplicative factor. The method is evaluated in two sets of environments: the diabolical lock and the MiniGrid environments.
strengths_and_weaknesses: I believe this is an interesting work touching on an important subject that is utilizing future novelty with long horizon novelty predictions to solve hard-exploration tasks.

However, I believe that the authors of the paper didn’t discuss the related work in this direction in enough detail. There are several recent papers that build upon the ensemble disagreement proposed in [1] (which is discussed in the current paper) and extend it to get multi-step novelty estimates into the future, such as [2] and [3]. (Especially in [3], multi-step novelty with RND as intrinsic reward is also proposed.). I believe these works should also be part of the discussion to give a more complete picture of long-horizon intrinsically-motivated exploration.

I still believe that the way the authors combined the RND intrinsic reward with general value functions is interesting and different in that they do not rely on model rollouts.

[1] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. “Self-supervised exploration via disagreement.” International conference on machine learning. PMLR, 2019.

[2] Sekar, Ramanan, et al. “Planning to explore via self-supervised world models.” International Conference on Machine Learning. PMLR, 2020.

([3] Lambert, Nathan, et al. “The Challenges of Exploration for Offline Reinforcement Learning.” arXiv preprint arXiv:2201.11861 (2022).)
questions: - Why is there no recurrency in the predictor network for the diabolical lock experiment?
- Why are there only two ensemble members? How does the number of ensemble members affect performance in regards to noise levels in the tested environments?
- I find the ablation study Figure 4c to be contradicting the statements made in section 3.3. Using $\gamma_z=0$ is the equivalent of intrinsic rewards by RND, in which case as the authors state no ensemble is needed (The deterministic target value of the pseudo-reward takes care of the stochasticity). So why is the ensemble ablation performed for this discount value? In fact why are there no ablations for $\gamma_z=0.6$?
- If I understand it correctly, the main difference between RC-GVF with $\gamma_z=0$ with and without recurrence is that the predictor network unlike in RND is not the same as the pseudo-reward generator and in the case of Minigrid doesn’t use an LSTM core. Did you do an ablation on the extra representational power in the different predictor network architecture of RC-GVF compared to the pseudo-reward generator also for the diabolical lock experiment?
limitations: -
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a generalisation of the random network distillation (RND) to deal with partially-observable environments. That is, the paper extends the notion of intrinsic rewards in RND to deal with long-horizon, sparse reward environments that are partially observable by (1) predicting the value of pseudo-rewards (as opposed to per-state rewards in RND), (2) using a recurrent value predictor network to create a belief state due to partial observability, and (3) using the disagreement among an ensemble of value predictors to explore where predictions are epistemically uncertain. The combination of this approach with any model-free on-policy RL agent as a basis leads to an approach to exploration through temporally-extended curiosity. To validate the approach, the authors have combined their approach with PPO and tested it on a long-horizon anti-reward-shaped task, as well as on a set of MiniGrid tasks with egocentric and panoramic observations. They conduct analysis and ablation studies to verify that all added components are necessary to achieve the best performance in such tasks over RND and other techniques.
strengths_and_weaknesses: ### Strengths:

- The paper moves in the direction of extending a simple yet powerful idea (namely, RND) to the more general problem setting of partially-observable environments with sparse rewards. 

- The approach is simple and fundamentally well justified; specifically, (1) GVF formulation is natural for going beyond predicting single-step rewards (as in RND), (2) using random targets as in RND has been previously shown effective (and simplifies the problem of selecting/discovering auxiliary tasks in Horde-style architectures), (3) using recurrence as agent's state-update function to deal with partial-observability is well-accepted and the authors have also justified its particular importance in the context of this paper, (4) using prediction disagreement among an ensemble is an effective proxy for epistemic uncertainty.    

- The approach is well-motivated through illustrative examples and challenging exploration problems (but with relatively simplistic dynamics; namely, Diabolical Locks and 6 MiniGrid environments).


### Weaknesses:

- I feel that the pseudo-reward generator should also be based on some representation of history (e.g. via a fixed and randomly initialised recurrent network). I discuss this further in the Questions section (first question).

- The trade-offs involved in choosing the discount factor are not discussed to a good extent. For example, why should we even care for low discounts? Why do they think $\gamma = 0.6$ works better, and not $\gamma = 0.9$ or $\gamma = 0.99$? Discussing intuitions and drawing on previous literature on the role of discounting (especially in combination with function approximation) could help improve this shortcoming.
  
- In relation to discounting, I also feel the range of discount factors used is somewhat arbitrary and not well motivated. The most commonly-used discount factors are missing in the tested values in Figure 4. Why?

- I feel that the panoramic-view experiment or at least the justifications around it are forced. The statement in footnote 1 of the paper is not a reasonable argument in my view. If the environment doesn't give you some information, assuming anything beyond it is privileged. Augmenting the agent with more sensors to get a panoramic view is privileged too. As such, I much rather see the result of your approach having access to the episodic count as opposed to a panoramic view.
 

questions: 1. In order to avoid state aliasing in the rewards, wouldn't it make more sense for the rewards to also be generated through a fixed, randomly initialised recurrent network that conditions on the sequence of observations, actions, and pseudo-rewards? For example, say there are only aliasing states between a trajectory that the agent has experienced repeatedly and a trajectory that the agent had not experienced before. Now, the history summaries would be the same and so would the value predictions, and the disagreement would be low. But if the trajectories were distinguishable by different pseudo-rewards, then the histories would be distinct and the epistemic uncertainty would be high for the trajectory that had not been explored before.

2. The authors have tried different discount factors for training the GVF predictors and argue that lower discounts are not good because the prediction horizon should be larger. Couldn't the worse performance of lower discount factors be due to optimisation issues under function approximation? (See van Seijen et al. (2019) for details on this issue.) 

3. I also feel the range of discount factors used is somewhat arbitrary and not well motivated. The most commonly-used discount factors (e.g. $\gamma = 0.9$ or $\gamma = 0.99$) are missing in the tested values in Figure 4. Why? I understand that these are not agent discounts and are only used for exploration, but I cannot process why still a higher value shouldn't in principle be preferred in episodic settings. I think too high wouldn't work well with function approximation due to optimisation issues, but a value of 0.99 should work in practice. Why is it not preferred in principle by the authors?

4. Does the basis PPO agent use the history summary or only use the latest observation? Is this setting the same among all baselines? 

5. Where can I find the result that you refer to in footnote 1?  

6. The approach seems also applicable in principle to continuous state-action environments. The authors do not claim this, but could you comment on whether you see any obstacles in such environments?

7. How do you justify using only 10 independent trials (or only 5 in one task)? PPO itself generally has very high variance across trials, and my experience is that using 20 trials is important in deriving any conclusions about performance characteristics. Now, in your context, the variation added due to pseudo-reward predictors could only make this worse. It is my view that, given that the domains are not too computationally expensive, it is critical to at least make sure the results are statistically reliable. Please share your view on this, and why the choice to run only 10/5 seeds.

Should these questions be adequately clarified/addressed, I'd be happy to increase my score. 

- H. van Seijen, M. Fatemi, A. Tavakoli. “Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning.” NeurIPS (2019).
limitations: The authors discuss some limitations of their work. I have asked for comments on other potential limitations in the Questions section.
They appropriately state that the nature of the work does not per se entail a potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a novel form of artificial curiosity that utilizes randomly initialized neural networks to represent generalized value functions. The authors utilize the insight that temporally extended predictions of environment quantities (such as state transitions, etc.) are often useful for generating artificial curiosity in partially observable environments. However, instead of using state sequences directly, they utilize randomly initialized NNs to annotate trajectories with pseudo-rewards and utilize the uncertainty in the prediction of these pseudo-rewards by a prediction network ensemble as a signal for artificial curiosity. They evaluate their method in lock-opening environments and in grid world tasks and show a performance improvement on curiosity baselines. 
strengths_and_weaknesses: Strengths:
1. The paper is well motivated. Long-term dependencies should be accounted for when generating artificial curiosity signals. 
2. The writing is easy to follow and the approach is intuitive.
3. The experiments are representative of hard exploration problems.
Weaknesses:
My only real qualm with the paper is the significance of the results. The method strongly outperforms the baseline in the lock task but really only outperforms the baselines in 2 of the grid-world tasks. Additionally, why are there no performance metrics for AGAC and NovelD on the lock task? I would be interested in knowing those numbers as a reader.
questions: 1. Are there any ablations on ensemble size?
2. As mentioned I would be interested in knowing the results of AGAC and NovelD on the lock task.

limitations: I believe the limitations of the method have been adequately discussed by the authors. I believe the paper is well written. While not state-of-the-art in terms of results, I believe there is some novelty in the method. I am on the fence about my decision about the paper.
review_title: null

Meta-review: metareview: This paper proposes an intrinsic motivation method which by and large extends RND, aiming to service the needs of agents in longer horizon exploration problems. There was a bit of a spread of scores amongst reviewers, but based upon reading the extensive discussion, I am recommending acceptance it seems that on the balance of opinions, the consensus leans towards the reviewers agreeing that the result is important, the method useful, and the paper is of a publishable nature.
confidence: Certain
Decision: Accept