Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents HyperDQN as a practical algorithm of Randomized least-square value iteration (RLSVI).  HyperDQN consists of two parametric models: first is a base model that is similar to that of a DQN agent and the second a meta model that parameterizes the last layer of the Q-network as a function of a latent variable. The purpose of this architecture is to generate posterior samples using the diverse Q-value functions represented through this architecture. The paper demonstrates this approach empirically across a variety of Atari games, SuperMarioBros and DeepSea. 

main_review: The paper begins with a strong motivation towards computationally implementing RLSVI in Deep RL using a base and meta models, and also points out reasons why this is non-trivial. Also, the main differences between BootDQN and HyperDQN are also discussed, which is important because BootDQN is a highly relevant prior work in this domain. Overall the paper is well-written and has strong motivations that are relevant to the RL community at large. 

The following are some of the questions based on my understanding of the paper and would be great if the authors could clarify some of them: 

1. The meta-model in HyperDQN is considered as a linear parameterized model and there is a theoretical justification for this. What would happen to the empirical/theoretical results if the meta-model is a non-linear function of the latent variable z? I am inclined to think that the representation power for the posterior distribution would be better and thus should produce improvements overall.

2. How are the sigma_w and sigma_p parameters chosen in the objective function? Have the authors considered a baseline where an agent uses an identical artificial noise term for modifying the Q_{target} (i.e., sigma_w z^{\top} \epsilon_i ) to the one used in HyperDQN? 

3. The Atari performance curves summarized in Fig 3 is not sufficient to demonstrate the utility of the presented approach. This curve is computed over all the Atari games (some of them require no exploration and some require a complex strategy for exploration). It would be more appropriate to show how the agent performs on “hard-exploration” and “easy-exploration” games (this classification is presented in Bellemare et al. 2016) rather than present a summary across all games. 

Also, why are the agents only trained for 20M frames? It is conventional in Atari to run the agents for 200M frames (Machado et al. 2018). This makes it easier to compare the performance of HyperDQN wrto the prior methods in Atari. 

A minor comment: why do some curves in Fig 3 start at 0 and another starts at a value lower than 0? 

Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. In NeurIPS, 2016.

Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018). Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61, 523-562.

4. How many random seeds were used to produce the learning curves in all the experiments (Atari, Mario and DeepSea)? It would be useful to add this information in the main text. Also, are the error bars reported in each of the presented learning curves?

summary_of_the_review: Overall, I think the contribution would be important to the RL community. My main concerns with the paper is that the presented approach claims to be helpful for exploration but the empirical results are not demonstrating this: One specific way to demonstrate this would be to show and emphasize that HyperDQN is able to achieve better returns on Atari games that are classified to be hard-exploration games. This would be a more powerful argument for HyperDQN.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new randomized exploration method termed HyperDQN. 

It aims to improve the popular Randomized Least Square Value Iteration (RLSVI) method by addressing RLSVI's major limitations of computational burden and its requirement of known good features in advance to adopt the linear setting. 

The principal of HyperDQN is to combine DQN with a probabilistic hypermodel which could generate posterior samples for the parameter values of the weights at the last linear layer of the Q-function. Both the hypermodel and the Q-function could be jointly optimized during training. By sampling the Q-function parameters, it helps the model to select exploratory action sequences.

 Empirical evaluation results are provided in Atari 2600 domain, SuperMarioBros and deep sea. Overall, the proposed method could outperform Bootstrapped DQN, Double DQN and another recent bootstrapping baseline termed OB2I in majority of the testified domains. 
main_review: $\textbf{Strength}$: 
- This work tackles an important problem of exploiting randomized exploration methods in deep reinforcement learning problems.
- This paper is very well written and easy to follow.
- Overall the method of combining hypermodel with Q-network is quite sound and very general to be applied on deep RL problems.
- The empirical evaluation results are very extensive where relevant baselines are considered for comparison.

$\textbf{Weakness}$:
- Very limited novelty where the idea is from the work [1]. 
- During comparison, only results at intermediate training stage  (i.e., 20M frames)  is presented while the detailed scores for the full training is missing. So it is unclear if the method could really be the next state-of-the-art method for Atari 2600 test suite.
-Only limited number of baselines are considered. There is not decent exploration methods, such as UCB-type exploration or ramdomized exploration with noisy nets, being considered for comparison.  


$\textbf{Detailed comments}$:

1. My primary concern is about the novelty of this paper. I found that both the formulation for hypermodel and the loss function for optimizing the hypermodel is inherited from the existing work [1]. In [1], a more general formulation for the hypermodel has been discussed which includes linear models, neural models and additive prior models. In HyperDQN, a linear hypermodel is adopted where there is nothing new for the method. Thus the contribution of this paper is not in terms of introducing a new method but is to show additional experimental results for an existing method on several deep RL domains. I feel such contribution might not be significant enough to be published as a full paper in ICLR.

2. This paper tackles the policy learning problem from a direction of considering randomized exploration method. It claims the randomized exploration method O2BI as a SOTA exploration method but I do not think so. Generally, the O2BI as well as HyperDQN is only compared with limited baselines when the training is incomplete. I feel that many other variants of UCB-type algorithms, such as noisy net [2] might be able to outperform O2BI or HyperDQN greatly with 200M training budget. For instance, prediction-error based exploration methods could effectively progress on the extremely hard exploration task Montezuma's Revenge while O2BI and HyperDQN make no progress on it. Therefore, a more inclusive exploration methods for discussion/comparison is desired. 

3. The training curves presented in Fig 12 come with no error bars and/or O2BI is missing from the curves.

4. I feel noisy net is also a related randomized exploration model which should be considered for discussion and/or experimental comparison.

[1] HYPERMODELS FOR EXPLORATION
[2] NOISY NETWORKS FOR EXPLORATION
summary_of_the_review: This paper is a well written one with extensive experimental results. 

The novelty of this method is rather limited as the method is exactly the same as the one proposed in a prior work. 

The empirical evaluation results appear to be less convincing to me because the results on Atari 2600 are only shown for intermediate model trained after 20M frames and the full training scores (inferred from the learning curves in Fig 12) appear to be much inferior than another variant of randomized exploration model which is the noisy net.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces HyperDQN, a novel exploration technique for deep reinforcement learning, which extends the ideas from an existing method, namely RLSVI, to the deep RL settings. HyperDQN optimizes a linear hypermodel, which is used to compute state-action values as a function of extracted features, to work as a module that converges to a sample from the posterior distribution of the Q-value function. The authors further present a specific cost function to jointly optimize the hypermodel and the feature extractor that can efficiently address the limitation of the RLSVI approach in deep RL scenarios. Empirical study demonstrates significant improvement compared to the competitors on several benchmarks.
main_review: Strengths: 
+ The outcome of the paper which adapts the ideas from RLSVI to deep reinforcement learning settings is significant and practical. 
+ The paper is very well written, clear, and easy to understand. The supplementary materials provide most of the information needed for acquiring a better understanding of the paper. 
+ The authors provide theoretical analysis and proofs to support their main ideas and design choices. 
+ The experimental results show superior performance of HyperDQN compared to state-of-the-art algorithms. 
+ The results are reproducible as the implementation and experimental details are provided in the paper.

Weaknesses: 
- The proposed approach provides incremental contributions w.r.t. the existing ideas from the previous work.
- The contributions are very specific to a class of problems and algorithms.

Comments: 
- The main ideas of the paper has been previously introduced in the literature and is further extended to the deep RL scenarios which make the presented contributions only somewhat novel. 
- The paper can be improved by providing some information and\or intuition about the generalizability of the proposed approach to the other domains. Additionally, discussions about the challenges in extending the proposed idea to the areas such as continuous control, offline RL, etc. would be highly appreciated. 
- Could you elaborate about the challenges and potential problems that could arise when jointly optimizing the hypermodel and the feature extractor? One of the main contributions of the paper is addressing the major problems with RLSVI which are restricting its applications in deep RL. While addressing these limitations regarding computational costs of RLSVI is discussed extensively, providing a specific cost function to jointly optimize the feature extractor and the hypermodel might lead to stability and convergence problems. Hence, it would be beneficial if the authors provide further insights into this matter.

summary_of_the_review: Overall, the paper introduces a novel approach for randomized exploration in deep RL, is well written, and shows strong promise in terms of performance compared to other state-of-the-art methods. Additionally, it includes valuable theoretical and experimental discussions that provide further support for their contributions. I thus vote for an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new posterior sampling-based exploration method, called *HyperDQN*, for deep reinforcement learning. Conceptually, *HyperDQN* is modified based on randomized least-square value iteration (RLSVI) and it resolves the limitations of RLSVI on feature engineering and high computational complexity. Technically, *HyperDQN* extends the Hypermodel technique, which is only used for bandit learning in its original paper. The performance of *HyperDQN* is evaluated on Atari 2600 games, SuperMarioBros and deep sea environment.
main_review: ### Strengths
Overall, I think it is a good trial to extend the idea of Hypermodel to the realm of deep reinforcement learning. Compared with BootDQN, *HyperDQN* will require much less computational power. Meanwhile, the paper provides a large amount of experiment results to show the superior performance of *HyperDQN*. Meanwhile, the paper is also written in a clear logical flow.

### Weaknesses
I apologize if I understand incorrectly. One of my major concern is on the novelty. Given the existence of Hypermodel, the *HyperDQN* seems to be a very straightforward application of it on deep reinforcement learning without more insights. Meanwhile, from my perspective, the Theorem 1 simply re-writes the motivation of Hypermodel in a mathematical way, using linear hypermodel to approximate the posterior distribution, which is already explained in [1].

Another concern I have is about the performance of *HyperDQN*. Specifically, although the experiments compare *HyperDQN* with BootDQN, it seems that this paper uses the version of BootDQN based on [2]. However, the version of BootDQN in [3] has much better performance than that in [2]. Further, it seems the BootDQN in [3] did not use $\epsilon$-greedy. Have you compred *HyperDQN* with the BootDQN in [3]?

**Honestly, I'm not very experienced in running experiments, so I apologize if the above comments on experiments contain some flaws.**

### Questions
- Does *HyperDQN* contain other insights that are not presented in [1]?
- Did the comparison experiments use BootDQN in [2] or [3]?
- Did you compare *HyperDQN* and BootDQN on deep sea?
- When you ran BootDQN, did you use number of ensembles significantly less than the one used in [3]?

### Suggestions on Writing
- In appendix C, it seems $b$ should be replaced by $\nu_b$ in equation C.6.
- At the bottom of page 4, it says $\theta_{\mathrm{predict}}\in\mathbb{R}^{d\times N_a}$, but in the previous paragraph, it also says $\theta_{\mathrm{predict}}\in\mathbb{R}^d$. This can potentially be confusing.
- In Theorem 1, it is may be better to replace "then $\theta:=\dots$ with" by "then $\theta:=\dots$ satisfies"

```
[1] Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Ian Osband, Zheng Wen, and Benjamin Van Roy. Hypermodels for exploration. In Proceedings of the 8th International Conference on Learning Representations, 2020.

[2] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems 29, pp. 4026–4034, 2016a.

[3] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems 31, pp. 8626–8638, 2018.
```
summary_of_the_review: The paper proposes a computationally lighter method *HperDQN* for exploration in deep reinforcement learning and shows its superior performance in experiments. However, the formulation of *HyperDQN* is conceptually not novel and its performance compared to other methods is also questionable.
review_title: null

Meta-review: This paper extends Randomized least-square value iteration (RLSVI), which is a method for exploration-exploitation tradeoff that is suitable for linear FA, to the deep RL setting. A key component is using Hypermodels of Dwaracherla et al. (ICLR, 2020) to generate the weights of last layer of the DNN. This generates a learnable randomness required in an RLSVI-like procedure.
The paper provides some theoretical results regarding Hypermodels, and provides extensive experiments to show that their method is a competitive one in solving exploration problems.

The majority of the reviewers are positive about this work. The concerns include the incremental nature of this work and the empirical results. In my opinion, the algorithmic contribution is reasonable, but somehow incremental. The theoretical results are minor, but acceptable. The empirical results are extensive, though they have some shortcomings.
I explain the algorithmic and empirical contributions below:


**Algorithmic Contribution:**
Similar formulation has been done by  Dwaracherla et al. But that work does not consider the RL setting, and instead focuses on the bandit setting. This work provides such an extension. A straightforward application of Hypermodels does not work for DRL, but some simple, yet crucial, tricks needs to be applied to make it work. The trick is to use Hypermodel to generate the weights of the last layer, instead of all layers. Although this is simple, the fact that it enables the method to work for DRL paper is significant.


**Empirical Results:**
The empirical results are quite extensive. There are two main issues with them though:

a) Many experiments on the Atari Suite are terminated after 20M samples. This is shorter than usual.
b) The experiments are only repeated for 3 runs (seeds) -- except one, which used 5 runs.

The authors' answer for (a) is that they have a limited compute budget, and running for 200M samples would cost them about $20K. Also they argue that 20M samples is enough to show the benefit in a better exploration method, as shown by some other papers.

After some inquiries, it seems that this $20K value has the correct order to run the experiments on a cloud (maybe within a factor of 2 or 3). Given this prohibitive cost, I am willing to accept that 20M samples might be sufficient for proving the main points of this paper.  I am not giving a large weight to this in my evaluation.

The main concern for me, however, is having only 3 independent runs of the algorithms, especially given that the issue under study is the efficiency of exploration-exploitation tradeoff, and that the proposed method has a lot of randomness built-in. This is the main weakness of the empirical results in my opinion, and not the 20M samples issue.

For instance, Figure 3 (Human-normalized score over 56 environments in Atari 2600 suite) does not have any confidence interval information. And Figure 4 has some shaded areas around the curves, but it is not clear whether it is standard deviation, standard error, or some other quantification of uncertainty.

Even though this is a borderline paper, I recommend acceptance of this work under the expectation that the authors should improve their empirical studies. In particular, I recommend much larger number of independent runs (seeds), maybe around 10 or so, with proper information about the uncertainty of the estimates. If running this for all games is prohibitive, showing the performance with more runs on a subset of the games is sufficient. Also I encourage the authors to consider NeurIPS 2021 paper "Deep Reinforcement Learning at the Edge of the Statistical Precipice".
Decision: Accept (Poster)