Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a new Bayesian Deep RL algorithm that learns deep Q networks (DQN) without using gradient descent. The updates of DQN's parameters follows from the tractable approximate Gaussian inference (TAGI) algorithm. Then it studies the empirical performance of proposed algorithm with DQN learned by gradient based optimization in several Atari benchmarks.
main_review: Strengths: 
1. This paper applies a Bayesian inference methods to do reinforcement learning and shows effectiveness on Atari level benchmarks.
2. It is potentially useful in the settings when one need to do deep RL without using gradient based optimization.

Weaknesses:
In general, the algorithm seems largely rely on an existing algorithm, TAGI, and the main contribution is extending this to the deep Q learning settings. In this case, I would expect a much better written paper in explaining the algorithm/contribution and stronger empirical results.
1. The property that it use Bayesian inference (gradient free) to update the NN's parameters seems to be an important contribution. First, it need to be better explained why no gradient itself is a pro. I would take it as just a property of an algorithm, without more advantage of disadvantage. Second it is very hard for me to understand the full update rule of each layer's parameter after reading this paper and its appendix. It might due to my limited knowledge in Bayesian inference, but I think this paper could be more self-contained by adding the full derivation of each layer's neural networks.
2. Most of the theoretical assumptions are not well explained and justified. For examples, what is the definition of A3 in math; why A5 is reasonable. In particular, one assumption about that different values has the same scale of variance seems very strong, especially in multi-step settings. Often the decisions not only effect the expectation, but also the variance (and distribution) of the return. Many distributional RL, safe RL literature make use of this and shows benefit.
3. It surprised me that this paper does not include any Bayesian deep RL baselines. Though some of them might not be gradient free like this paper. They shares many similarities such as the Gaussian assumptions.
summary_of_the_review: In summary, this paper studied how to learn DQN in a fully Bayesian way and is interesting. However too many parts of the paper related to its core contribution is unclear at this time. Unless it's significantly updated then this paper is not ready to published.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a method for combining deep Q-learning with tractable approximate Gaussian inference to allow for deep Bayesian reinforcement learning without the need for gradient based updates. They demonstrate this method can attain performance comparable to standard backprop trained agents in a number of environments.
main_review: The paper presents an interesting new solution to the problem of performing Bayesian deep RL, which is certainly a challenging problem for which a lot of use can be seen. As they note, the associated uncertainty is certainly very useful for driving appropriate strategies for the agent to balance exploration and exploitation. However I don't think the paper is quite ready to be published.

Firstly, I'm not sure this citation is correct:
J-A. Goulet, L.H. Nguyen, and S. Amiri. Tractable approximate Gaussian inference for Bayesian
neural networks. Journal of Machine Learning Research, 22:1–23, 2021.
I can't find the paper on the JMLR website (https://jmlr.org/papers/v22/), and as far as at least Google Scholar is showing it only seems to exist on arxiv. Of course, if I just haven't looked sufficiently hard enough (or in the wrong place) please correct me. This may seem like an odd complaint for the main review, but since the method presented in the paper is a direct extension of this TAGI method to deep RL it's important to be clear where the starting point is. This makes it a little harder to validate the claims made about the underlying TAGI method, since as far as I can see it does not appear to have been peer-reviewed yet. Having looked at it myself briefly, I will assume for the purposes of this review though that the claims hold.

In the experimental section there are no comparisons made to any other Bayesian deep RL methods despite them being discussed in the paper. This seems to be justified by saying that the presented method is sufficiently special and distinct because it is not a gradient based method. I'm not really sure why that's particularly important, but even so I think it very useful/necessary to have comparisons with those methods anyway so as to see the relative performance.

The comparison is always made to a deterministic neural network but importantly following a $\epsilon$-greedy exploration strategy, as opposed to the Thompson sampling done by this method. Other exploration strategies are available (upper confidence bound for example) and I think should all be considered (or at least explained why not considered) in the paper and experiments.

A large deal is made of the fact that no gradients are required - why this is such a major contribution should probably be discussed furthering the paper as I don't think that it's made particularly clear at the moment.

I really appreciate the use of the colours in the figures - I'm not sure though whether magenta and grey are the best choices as they don't particularly stand out.


summary_of_the_review: The paper presents a potentially promising method for Bayesian deep RL that can scale to large states spaces like Atari problems. I do not however believe that the paper is ready to be published as is since the experimental validation and comparison needs expanding, and further discussion on why exactly this method is so useful compared to the current literature would be useful.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines the TAGI algorithm with Q-learning algorithm and demonstrates that TAGI-DQN can achieve on-par performance with its backpropagation-based counterpart with fewer hyperparameters. 

main_review: Strength: 

This paper is well-written and easy to follow. The idea of applying Bayesian inference to obtain uncertainty estimation and sampling is interesting. 

Weakness:

Major concern:

Since the key idea of TAGI-DQN, the TAGI algorithm, is from an existing paper, I'm expecting more complete empirical evaluations and more systematic comparisons with the baseline model (gradient-based algorithm), and the current experiment part may not be enough. 
1. The comparisons should be implemented on more Atari games with more epochs since Atari games are very different for each task and may have different learning patterns in the early stage of training. (as shown in paper [1])
2. The performances of DQN here is much lower than the ones shown in other paper (e.g., [1,2]), which makes the conclusion less convincing. 

It would be more persuasive if adding TAGI-DQN on top of the codebase that has better performances, and run through more Atari games to conclude that TAGI-DQN can achieve similar performances with its gradient-based counterpart. 

Besides, the assumption homoscedastic variance of TAGI-DQN is not very realistic since the variance of $V$ at step $0$ should be very different from that at step $T$ (last step), which requires more strong empirical results to support.  

3. I think the focus of this paper is a little bit unclear to me. In the related work part, it seems the main contribution of TAGI-DQN is to propose an analytically tractable inference approach; whereas in the Discussion part, the main focus is to propose a gradient-free method compared with its backpropagation counterpart. If the first is the case, then I would like to see more comparisons between the existing BDL methods; if the latter is the case, I think the related work part should be modified to reflect more from this aspect. 

Minor concern:

1. The definition of A in eq(4) is not the same as the size of action space. 
2. It is not very clear to me what does "Atari experiment with on-policy RL" mean, since Q-learning is an off-policy algorithm, what does on-policy indicate here?
3. I do not see how eq(10) takes the maximization of the expected value as described one line above. 

1: Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." Thirty-second AAAI conference on artificial intelligence. 2018.

2:Dabney, Will, et al. "Distributional reinforcement learning with quantile regression." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.

summary_of_the_review: In summary, the idea of applying Bayesian inference as an alternative to backpropagation-based algorithm is interesting. However, it requires more complete experiments to demonstrate this, along with the other assumptions that simplify the computing complexity.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes how to adapt the Q-learning algorithm so that it is compatible with Bayesian deep learning, as opposed to the standard (semi-)gradient-based deep learning implementation. It explains how to propagate uncertainties about network parameters through the neural network computations and ultimately to the q-values, which enables action selection using Thompson sampling. Finally, it presents empirical results on CartPole, Lunar Lander, and five Atari games.
main_review: ### Strengths:
1. Potentially useful to be able to implement deep Q-learning with the tractable approximate Gaussian inference (TAGI) method, rather than with the standard semi-gradient approach.
2. Demonstrates an alternative implementation of deep Q-learning at the scale of Atari games.
3. Proposes a fully Bayesian way of accounting for neural network parameter uncertainty in the value function during action selection.
4. Offers a strategy for reducing the number of hyperparameters involved in deep Q-learning.

----

### Weaknesses:

This paper does not focus on a clear contribution. The abstract claims that using Bayesian methods to account for the uncertainty of the NN’s parameters is key to improving decision making, but the evaluation focuses only on reducing hyperparameters and obtaining comparable performance to existing methods. Each of the strengths listed in the section above could be a fine topic for a paper if executed well, but the submitted paper moves on before fully answering any of the questions it raises.

(Note: each of the numbered weaknesses below corresponds with the same numbered strength above.)

1. If the objective is to show that it is both possible and useful to implement deep Q-learning with TAGI, then I would expect to see a full derivation for _Q-learning_, as well as some motivation or evidence for  _why_ it is useful.
    - Currently the paper only appears to consider the derivation for _value estimation_. My (somewhat uninformed) impression of the original TAGI paper is that it demonstrated both regression and classification in deep neural networks. So if one were to take the perspective that estimating the value function amounts to solving a regression problem, then the original paper already demonstrated, at least in principle, that this was possible.
    - But the value estimation problem is only one aspect of Q-learning, and it’s not obvious from the derivation how actions would fit in. Eqn (10) defines $q(s_t,a_t)$ in terms of a fixed policy, which does not make the dependence on actions clear. Then the paper redefines $V(s_t, a_t)$ as being action-dependent, which seems like an error, without describing what this dependence looks like (eqn. (11)).
    - Finally, the paper does not provide much motivation for _why_ this alternative implementation would be desirable.
2. If the objective is to demonstrate that TAGI can support deep Q-learning at scale, then I would expect an answer to the question: “how well does it work?”. I feel like the paper provides a partial answer, but the experiments don’t give me enough confidence to answer the question decisively.
    - For instance, the results are only for 3 seeds, and the shaded regions are not labeled in the plots, so it’s unclear whether the results are significant. In general, my expectation with Atari would be to see at least 5 seeds, and ideally 10 or more. With CartPole and Lunar Lander, experiments are typically much cheaper to run, and I would expect to see more like 100+ seeds. See Henderson et al. (2018) for a discussion of why this is important.
    - It’s unclear what the baseline agent is for the Atari experiments. Is it standard DQN? If so, I’m surprised by the Atari baseline performance curves. Performance is much worse than the original for Breakout (\~150 here, vs. 401.2), Qbert (\~2900 vs. 10596), and Space Invaders (\~450 vs. 1976). After reading through the appendix, my impression is that the “Backpropagation” agent trains on the same amount of experience as the DQN from Mnih et al. (2015)’s Nature paper, but the hyperparameters and network architecture seem to be different. I’m left wondering why the paper uses these settings rather than comparing against the original published DQN.
3. Given that the method claims to offer a tractable way to account for NN parameter uncertainty, I would expect to see a clear indication that this enables an agent to make _better_ decisions than methods that either don’t take a fully Bayesian approach or that don’t account for such uncertainty at all. Instead, the paper only claims to achieve “comparable” performance, and only compares against one baseline. I feel that there are a few problems with this.
    - In my opinion, “comparable” doesn’t accurately characterize the performance. I would instead say the results are “mixed” versus the “backpropagation” agent. On the Atari games, TAGI appears to be better on three out of five, but much worse on one and somewhat worse on another.
    - Regardless, neither “comparable” nor “mixed” results clearly demonstrate _better_ decision making capabilities.
    - If the story is in fact more complicated---i.e. if properly accounting for parameter uncertainty doesn’t _always_ lead to better performance---then I would want to understand when to expect it to be better and when to expect it to be worse. The experiments suggest that this may be the case, but such a discussion is absent from the results presented in the paper.
    - The paper cites Azizzadenesheli et al. (2018) and Osband et al. (2016), both of which appear to account for uncertainty and allow for Thompson sampling during action selection. I’m left wondering why the paper doesn’t compare against those approaches when evaluating the performance of the proposed method.
4. I am unconvinced that the paper demonstrates that TAGI reduces the number of hyperparameters required for deep Q-learning. Examining the hyperparameters in the appendix, I don’t find the comparison conclusive because the decisions about what should count as a hyperparameter are subjective.
    - For example, half of the “backprop” agent’s hyperparameters are related to the epsilon schedule, whereas the same agent’s learning rate schedule is represented as a single hyperparameter. Alternate forms of epsilon schedule exist that could reduce this number significantly. The paper does not justify why this particular way of accounting for them is appropriate.
    - If the objective is simply to reduce the number of hyperparameters, are there other strategies worth comparing against? What makes this strategy for reducing hyperparameters so compelling? Is there a way to go fully Bayesian and incorporate the hyperparameters into the graph, so as to effectively get rid of _all_ of them?

----

### Questions:
1. Can you comment on why it is reasonable to assume that the value V is well approximated by a Gaussian distribution? Can we expect this assumption to hold in general, or only for specific environments?
2. Is the backprop baseline intended to exactly reproduce the Nature DQN result? If so, why are the results so different from the original paper? If not, what was the rationale for using the particular DQN architecture and hyperparameters for the Atari experiments?
3. Any idea why TAGI performs better than backprop on some Atari games but worse on others?
4. Was there a reason for not comparing against Azizzadenesheli et al. (2018) and Osband et al. (2016)? Is there a benefit to the proposed method over these other approaches, aside from mathematical rigor?
5. Why does TAGI use a horizon of 128 steps, but the backpropagation baseline uses a horizon of 5 steps? Is this a fair comparison?
6. I’m confused about the exploration rates for backprop on Atari. How are the different epsilon values used in the schedule?
7. What does it mean for the initial learning rate to be LogUniform(1e-4, 1e-2)? Does this mean the learning rate is chosen randomly for each seed?
8. The exploration fraction (0.1)---which I assume is the same thing as epsilon---for backprop on CartPole and Lunar Lander seems rather high. Are you decaying that to 0.02? If not, how was this value chosen?
9. It’s interesting that the variance for TAGI seems to be much lower than for backprop on CartPole and Lunar Lander, but much higher on Atari. Does this hold up with more seeds? What might explain this behavior?
10. Is there a potential connection between entropy regularized RL and this notion of using a decay function for the observation noise variance to more strongly incentivize the prior during early learning?

----

### Comments:
1. For a reader unfamiliar with TAGI, it’s unclear which parts of this paper are reviewing existing work and which are proposing original contributions.
2. In section 2.1, $J_Z$ and $J_\theta$ are undefined. I assume these refer to the Jacobian? It would be useful to say what they are explicitly.
3. In section 2.2, the “short form notation for the reward” makes it difficult to see how actions fit into equations (8)-(11).
4. Final paragraph of Sec. 5: “advanced actor critic” should be “advantage actor critic”. I also don’t think Mnih et al. (2016) is the right reference for A2C, since that paper only discusses the asynchronous version, A3C. OpenAI Baselines or “Learning to Reinforcement Learn” by Wang et al. (2016) would be more appropriate.
5. The formatting of the references is inconsistent. Some references use the full name of the publication, while some use abbreviations. Some include conference paper URLs; others don’t, etc. References should consistently use the same style, and use the full publication name (with proper capitalization) rather than the abbreviation.

----

### References:
1. **Henderson et al. (2018) –** ​​Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. "Deep Reinforcement Learning that Matters." In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 3207--3214, 2018.
2. **OpenAI Baselines –** either https://github.com/openai/baselines or https://arxiv.org/abs/1708.05144
3. **Wang et al. (2016) –** https://arxiv.org/abs/1611.05763


summary_of_the_review: The paper explores using Bayesian methods to guide action selection in deep Q-learning, and it makes some progress in that direction. I appreciate the time and effort spent in preparing this submission, and I do see some value in the proposed approach. Unfortunately, in its current form, the paper does not have a clear contribution. The paper attempts to answer too many questions and does not sufficiently answer any of them. Additionally, the main derivation is hard to follow and appears to be incomplete or incorrect. I recommend rejection.
review_title: null

Meta-review: The authors combine TAGI with Q-learning to create an approximate Bayesian Q-learning algorithm. They evaluate their approach and show it has comparable performance to DQN.

All of the reviewers were positive about the potential of the paper. Unfortunately, the paper suffers from lack of clarity, of motivation, and comparison to relevant approaches. All of the reviewers brought up nearly the same concerns and I agree with these concerns. The authors have not address them and the reviewers do not think this paper is ready for publication at this time. I agree and recommend rejection.

Evaluating TAGI-DQN is a valuable contribution, but alone it is not sufficient. The reviewers have made many suggestions on how to improve the paper, and I hope the authors follow up on these suggestions.
Decision: Reject