Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes Mixture Bootstrapped DQN (MB-DQN). The new algorithm modifies bootstrapped DQN by giving different backup lengths to each head. It shows better performance in several Atari environments, compared to bootstrapped DQN. This work also attempts to analyze the source of empirical benefits in terms of attention areas of the agents, the quality of data sampling, and the way of utilizing different backup lengths. There is also an ablation analysis showing that the MB-DQN is not limited to a single configuration.

Reasons for score: 

(+): The proposed method is tested considering several different aspects, including the agent’s attention area, data quality, and the sensitivity of parameters. These experiments provide a deeper view of the MB-DQN and try to explain why this new method helps with learning. 

(+): The paper is well organized and easy to read. The appendix provides enough details to reproduce the main result.

(-): This paper claims that while longer back-ups expedite learning, they do so at the expense of exploration: longer back-ups help in efficient assimilation of the reward information that the agent begins to exploit, as opposed to shorter TD(0)-like backups that propagate reward information relatively slowly, leading to seemingly exploratory behavior (reward information has not propagated everywhere so exploitation essentially amounts picking random actions). I find this assertion rather flawed as it suggests that inefficient learning is an avenue for better exploration. An agent tends to explore when it chooses a seemingly sub-optimal course of action to improve its knowledge, and it tends to exploit its current knowledge when it chooses the greedy action. Deliberately slowing the learning process such that even exploitation appears exploratory is a defeatist approach towards the exploration problem.

(-): While there is some empirical comparison with eligibility traces in section 4.3, the fundamental advantages of using multiple heads over TD-lambda remains unclear to me. Can we see the benefits clearly in the tabular/linear function approximation setting, or make a theoretical statement about the benefits? Or is this approach only useful in the Deep-RL setting where it helps with how neural networks interact with TD? Section 4.2 suggests that MB-DQN helps by changing the data distribution in the replay buffer, can we see the same effect in the tabular/linear function approximation setting, perhaps in the domain used in Figure 1? Studying these questions empirically in simpler settings can help untangle the interaction of NN-based non-linear function approximation and the general temporal credit assignment problem that traces are designed to solve. 

(-): The experimental results are averaged over only 3 different random seeds. It remains unclear how these seeds were chosen. More importantly, 3 runs are not sufficient to establish the statistical significance of the results. 
review_title: MB-DQN

final_score: rating: 7: Good paper, accept
review_content: Summary:
As the title suggests, the paper proposes using a mixture of multi-steps in estimating the action-space value function in DRL. Multi-step RL has been around for a while now, and this work attempts to extend it further to a mixture setting by exhibiting a performance boost when the proposed technique is used with Bootstrapped DQN.
Why multi-steps?
Multi-step return estimations constitute a high-variance low-bias method. Because of the “intuition” that different multi-steps would incorporate different knowledge, and the step parameter might be dependent on the environment dynamics, it would make sense to have several of them.
Why Bootstrapped DQN:
Provides easy incorporation of the multi-steps since it has multiple Q heads, and the authors could assign different multi-steps to them.

The paper is easy to read and follow. The different sections have a “natural” flow to them and the reasoning is clear.
To prove the advantage of using the algorithm, it has been tested on 8 of the Atari games and it exhibited better performance in several of them. A qualitative comparison was also performed based on the attention map visualization for 2 Atari games and it showed that MB-DQN infers more relevant sate-space regions.

“However” points:
The critical point is that the performance of the baseline Bootstrapped DQN is different from the one reported in the original paper. We would assume there is still some hyperparameter tuning missing for B-DQN. Yet, this does not disapprove of the power of the suggested MB-DQN since the same hyperparameters were used for both methods.
Since MB-DQN is expected to have a higher variance, it would be interesting to test in a noisy environment (non-deterministic action-reward?).
(Slight indentation problem in the first hyperparameters table that pushed the head return value to the first row)

Verdict: 
This is an interesting and powerful method. Well written paper. More extensive experiments would have made the proposed work more convincing. 
review_title: Interesting concept, well written paper. More experiements would have been more convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary of Contributions

The paper proposes the Mixture Bootstrapped DQN (MB-DQN) algorithm, an extension of bootstrapped DQN where each outputted value estimate uses a different multi-step TD target. They provide a motivating example suggesting that with shorter backups, the slower convergence results in greater exploration. They empirically show that in a variety of cases, simultaneously learning action value estimates with varying multi-step TD targets can lead to improvements over existing methods.

### Review

The proposed algorithm appears novel, and the results are interesting. However, I have the following concerns:

1) A high level description was provided on how actions were selected during evaluation, but none was provided as to how they were selected during training (i.e., the behavior policy). This information is important given the use of Peng's Q(λ) targets (*uncorrected* returns), as the update targets are now a mixture of on-policy sampled rewards, and off-policy bootstrapped values. Could the authors elaborate on the behavior policy used while training, and comment on its compatibility with the use of uncorrected multi-step returns? 

2) The motivating example compares learning with a 1-step TD target with that of a 5-step TD target. The paper suggests that due to the larger bias in a 1-step TD target (and consequently, slower convergence), the 1-step learner ends up exploring more. Especially in the 1-step case (where Peng's Q(λ) does not need corrections), the algorithm learns off-policy, making it seem like whether or not it explores more an artifact of a design choice in the behavior policy, and not something inherent about 1-step TD targets. Also of note, when looking at the heatmaps after 100,000 time steps, the 1-step learner has a very salient line taking a suboptimal path of going straight up for a bit, and then heading in a straight line toward the goal. Such a high frequency of this suboptimal path seems suggestive of the method actually *not* exploring enough. As such, I'm not sure the conclusions drawn from the example are convincing, and would appreciate if the authors could comment on the above or clarify any misunderstanding.

3) In addition to the motivating maze example, the paper repeatedly emphasizes that MB-DQN provides heterogeneity in the target values. Can the authors clarify what this precisely means, and elaborate on why this is a desirable property?

4) It appears a fixed learning rate was used for each algorithm. Due to the bias-variance tradeoff of multi-step TD methods, as well as the varying numbers of heads in the ablation studies, it's not clear whether a fair comparison was made between each algorithm instance. Could the authors comment on how hyper-parameters were chosen in the empirical evaluation?

5) Only 3 random seeds were used in their empirical evaluation, and the paper presents one standard deviation as a 68% confidence interval. The use of 3 seems insufficient for central limit theorem arguments, but the mention of 68% suggests an assumption that the results are normally distributed. Can the authors comment on the statistical significance of their results, and whether the number of runs are sufficient for the claims being made?

Based on the above, I'm recommending rejection at this time. I'm willing to raise my score should my concerns be addressed.

----- Post Discussion -----

I appreciate the clarifications made in the discussion regarding additional experimental details and whether the methods were fairly compared. However, given that much of the claims rely heavily on the empirical evaluation, I think further experiments with more rigorous statistical analysis is necessary. Even with the correctly plotted standard errors, the results still largely do not appear significant, suggesting that 5 seeds is just not enough. There are good recommendations by Henderson et al. (2017) and Colas et al. (2018) for the empirical evaluation, and I think it would additionally strengthen the paper to formalize the notion of heterogeneity (e.g., be able to approximately measure it, and convincingly argue that this is what's underlying any differences in performance). 
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper utilizes different step return targets for different heads in bootstrapped DQN, such that a more heterogeneous posterior estimation of policy value may be obtained. However, my main concern is the novelty. The step size can be viewed as a tunable hyperparameter, and the posterior computation is mainly credited to the usage of different randomized DQNs. Similarly, other tunable hyperparameters may also be diversified to introduce more heterogeneity such as learning rates, etc. 

Re- simulations, I appreciate the authors' efforts in conducting different experiments to understand their method, but I do have a few comments/thoughts as below:
- what is the performance of all-2-step bootstrapped DQN? It seems more natural to me to include the comparison with this baseline, since the performance of mixed-1-3 may be more close to all-2.
- I'm not sure if Figure 4 does do you a favor, since again we can always tune the hyperparameter of step size, and it does not need to be the same for different environments. This paper's method seems to not peak either of them.
- the reason behind the good performance of MB-DQN explained in section 4.2 is not obvious to me. I'm not sure about the point of comparing an agent trained with its own data versus one trained with other data, since we are just learning the policy value but not the policy itself, and there is no issue like off-policy correction, etc. Actually such techniques of data-sharing have been used a lot if we have multiple agents to collect data.
- in section 4.4, when comparing different setups of MB-DQN, it would be fairer to include more baselines used in part of the MB-DQN such as all-2, all-3.
 
review_title: The paper introduces different step return targets to bring more heterogeneity to the bootstrapped DQN, but the improvement is not significant enough.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ###################################

Summary: 

This paper combines the idea of multi-step returns for value-based reinforcement learning with Bootstrapped DQN. Updating value functions with different step returns has potential benefits, and was elegantly integrated with the Bootstrapped DQN structure, where different heads have different backup lengths. This work provides empirical results on Atari game domains, and further provides ablation analysis. 



###################################

Pros: 

1. This paper presents a simple, but meaningful approach to combine multi-step returns in value-function based RL and Bootstrapped DQN. The strength of having different backup length in value-based RL is that multi-step returns have lower bias (despite the high variance) and sensitivity to future rewards. The power of Bootstrapped DQN comes from its exploration capabilities. Bootstrapped DQN’s multiple heads are a natural way of integrating different lengths of returns. I like the originality of this idea presented by this paper. 

2. This paper provides good empirical results in Atari games, showing that mixed 1-3 step MB-DQN outperforms fixed-step baselines. 

3. This paper also provides ablation studies, adding more comprehensive analysis of this algorithm. Separating data generation agent and learning-only agent is valid; MB-DQN is compared with DQN-lambda and DQN-ensemble and shows better performance; and more experimental results with different numbers of heads and different configurations of return lengths are compared. 


###################################

Cons / Questions 

1. The choice of (step length) n_k is limited: this paper presented main empirical results using five heads of 1-step and five heads of 3-step, and also compared with mixed 1-2-3, 1-2, 1-3, 2-3 steps. In the supplementary materials, there is a result on 1-3-5 step configurations. What about results on larger n_k steps? Could you add some experiments on 10 or 20 steps or at least add insights on how it would perform? My guess is that including very large n_k to the mixture would lead to high variances and possibly degrade the performance. 

2. The original strength of Bootstrapped-DQN comes from the fact that it is better for exploration. (Multiple heads lead to explorations.) However, I don’t see comprehensive analysis in the perspective of explorations. To be specific, as authors mentioned in section 3.1., shorter-step return is better at exploring wider areas in the state spaces than longer-step returns, although longer-step returns might be faster at convergence. In terms of exploration perspective, is there an evidence that mixture of shorter & longer n_k’s is better than single fixed 1-step returns? 

###################################

Reasons for Score: 

I gave the score of 5 (marginally below the threshold). But if the concerns that were raised above are well addressed, I am happy to change my score to a higher score after the rebuttal period. Overall, I appreciate the novelty of this paper, and I think the idea of combining two powerful methods in value-based RL: multi-step returns and bootstrapped DQN is meaningful. 
review_title: Interesting combination of two ideas, but some concerns to be discussed..

Meta-review: This paper extends Bootstrap DQN with multi-step TD target. The initial submission had missing details, communication problems, and results lacking rigor. The authors made a clear effort to address the reviewers concerns.

This paper's contribution is supported primarily by the empirical results which need major work. The lack of statistical significance in the key results is a major problem. The new 5 run results (originally only 3 runs) shows no clear evidence of improving over the baseline. Additionally one must either justify the use of such few runs by investigating the distributions and using the proper statistical tools (Colas et al [2]) or simply do more runs. Regardless statistical significance in the precise sense is a requirement.

In addition other adjustments to the paper would strengthen it significantly: (1) The qualitative results like state visitations can be interpreted either in favour of the method or not, this could be improved with discussion or omitted---see [1]; (2) the discussion of heterogeneity was informal; (3) discussion of the impact and sensitivity of hyper-parameters should be included---this includes addressing the concern that the performance of the baseline was as strong as it could be; (4) the current results do clearly separate if the improvement in performance (if it can be shown to be significant) is due to improvements in the rep via auxiliary task effect or the multi-step return---the reviewer has made a nice suggestion for an experiment here.

In summary, the reviewers did not find the text and examples in the paper convincing as to why the proposed method should be better than bootstrap q, and the results are not significant and need more work. 

references that may be helpful:
[1] https://openreview.net/forum?id=rkl3m1BFDB&utm_campaign=RL%20Weekly&utm_medium=email&utm_source=Revue%20newsletter
[2] https://arxiv.org/abs/1806.08295
Decision: Reject