Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes to use TAGI, an approach for training BNNs, within DQN to enable Thompson sampling.
main_review: The paper has very limited novelty,  simply plugging TAGI (a previously proposed approach for training BNNs) into DQN. The results are also not particularly impressive, being run on a very limited set of environments and being outperformed by vanilla DQN on several. While having fewer hyperparameters to tune than training the network via gradient descent is a nice benefit, a more thorough evaluation would be needed to demonstrate the advantages of TAGI in reinforcement learning.

Why not compare to Thompson Sampling with Bayesian approaches to Q learning like in Azizzadenesheli or Osband et al? This seems like the more direct comparison than vanilla DQN.

The authors also describe their learning algorithm as being analytic, but still relies on repeated iterations, so it's unclear to me what the benefit of the "analytic" updates are. If the key benefit here is just the uncertainty is analytically computed when evaluating the network on new inputs, then the authors should compare the evaluation times of TAGI vs other approximate BNN methods that require multiple forward passes.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper adapts a recently proposed analytical closed-form Bayesian approach to optimizing Neural Networks to Q-Learning. The proposed TAGI-DQN architecture is optimized without gradient descent. RL policies based on TAGI-DQN are shown to perform well on a collection of OpenAI Gym environments, including a small number of Atari games. 
main_review: #### **High-level evaluation**
I am excited about the potential advances to RL in complex environments that may be ushered in by the possible contributions put forward in this paper. I however do not feel that the work, as currently presented, is wholly ready for publication and perhaps another round of revisions (including the addition of more relevant baselines -- see "Relevance" section below) would do this paper a great service. While reading, I got the sense that the authors didn't want to distract from their intended contributions by dwelling on how TAGI works. This is always a tricky balance in writing papers (how much background is too much?) but I did feel that there were too significant of conceptual+technical gaps introduced in the paper by keeping the discussion too high-level and informal through Section 2.  

#### **Originality**
The adaptation of the TAGI framework to RL domains is very intriguing. Analytical solutions to a Bayesian treatment of NNs could greatly improve challenges of sample efficiency within RL. This advantage is demonstrated in the experimental analysis across a variety of domains. 

#### **Quality and Clarity**
The major weaknesses of this paper concern the clarity of the formulation and utlization of the TAGI with DQNs. In particular, I found the discussion in Section 2.1 to be far too high-level for introducing a relatively complex new approach. A phrase that ran through my head while reading was “show don’t just tell”. The paper is quite light on technical details and it wasn’t due to space constraints as there was nearly 2 pages of unused space remaining in the submitted version of the paper. I feel that a more formal approach to how the two steps of TAGI interact with individual parameter distributions and their connections through hidden layers is necessary and would greatly improve the paper. This would make the discussion in Sections 2.2-2.3 much easier to follow and understand. This is especially true in the discussion starting on line 124 when inference through the TAGI-DQN is described.

Particularly egregious in this lack of clarity is the unmotivated and sudden jump at the end of Section 2.1 to RL. It’s not apparent how the discussion around optimizing parameters via TAGI with a single example or a mini-batch directly relates to learning sequentially with episodes of observations and so on. Additionally, without any formal framing, there’s an extensive burden placed on the reader to know exactly what parts of the following sections correspond to the steps within TAGI. With the connection seemingly drawn to the expected value of selecting an action $a_t$ when conditioned on some state $s_t$, it is also somewhat surprising that the family of Distributional RL (Bellemare, et al; 2017 ICML) approaches went uncited. I acknowledge that Dist. RL does not attempt to pass off as a Bayesian approach but the core component of these approaches is in the implicit distribution formed around the expected value of rewards conditioned on action selection. 

In line 100, was the word “discrete” intended in the place of “categorical”? 

It’s not clear why n-step returns are preferred? Typically this introduces significant variance in DQN based approaches. Is there something particular about TAGI that necessitates the use of multi-step returns?

The elimination of hyperparameters by moving to TAGI-DQN, described on line 130 seems to only consider those associated with SGD optimization (learning rate, optimization hyperparams, minibatch size, etc…). While this may require several components to tune, the architecture construction and size of replay buffer can have an outsized impact as well. While it’s nice to potentially not worry about hyperparameters associated with SGD, there are still significant choices made in the design of a model.

What’s the rationale for decaying the standard deviation $\sigma_{V}$? Is it to account for drifting biases as the parameters of the model are learned? Is this a “hack” that assumes improved model confidence over time? Is this perhaps too optimistic? What happens if an observed state in a later episode falls out of the assumed distribution? Is TAGI-DQN robust to outliers? These are important questions with regards to avoiding the Q-function approximation overfitting too early. 

Again, it’s a shame that so little technical content is provided in Section 2. The primary contribution of this paper is the extension of TAGI to RL domains and architectures. Without the details of how TAGI actually works, line 16 in Algorithm 1 becomes an intellectual dead-end within this paper as written.

In the OpenAI Gym experiments, the initialization of $\sigma_V$ is not mentioned. Additionally, it is not mentioned whether the comparison with the OpenAI default DQN is especially fair or not. For instance, what is the architecture of the DQN model? Is it the same as the TAGI network? Does TAGI use the same amount of computation or more in it’s optimization steps? Are the batch sizes consistent between the two implementations?

For the experiments on Atari, it’s unclear whether n is set to 128 or whether that horizon is for the decay of the standard deviation. While it’s impressive that TAGI outperforms DQN on several of these Atari games, the discussion or analysis about why it fails on some is missing. It would be nice to be able to develop some insight into what types of domains TAGI may not be appropriate for. Are there some complexities in the state spaces of Beam Rider or Space Invaders that make it harder for the solution approach? Do the representations learned by the CNN encounter (or otherwise get stuck in) local minima? I find it interesting that in both of these games, there is a point after which the baseline DQN solution diverges from the TAGI-DQN solution in performance, indicating that it’s learned something that wasn’t accessible to the TAGI network. 


#### **Significance**
If properly evaluated the contributions promised through a Bayesian method optimized via analytical means are potentially of great significance. However, I found the technical discussion and experimental demonstration to be mildly unconvincing (partially due to a lack of clarity as discussed above). Clearly, the performance of the TAGI-DQN is demonstrated to be better than a standard DQN model. However, the lack of comparison to comparable approximate Bayesian RL methods or appropriate ablations is disappointing. Without comparison to relevant baselines, it’s hard to feel confident about the proposed analytical solution from a practical standpoint. It’s possibly assumed that TAGI-DQN is automatically more efficient/effective due to the closed-form solution. I say this as a means to rigorously understand the specific advantages borne by the TAGI framework applied to a Bayesian form of the DQN.

In line 144, a “few” BNN approaches that have been applied to large RL domains are alluded to. It would be preferable if these papers are explicitly cited and discussed so as to provide greater context into how TAGI differs (aside from only being an analytical closed-form solution method, if there is anything else?). Is the way the network parameter distributions are factorized to make inference tractable in a similar way to any of these BNN approaches, what about variational inference types of solution methods (such as VariBad--Zintgraf, et al; 2020 ICLR)? How similar is that to TAGI? Answers to these types of questions that would greatly improve the framing of the results that are shown in the following sections of the paper. 


##### **Additional references**
Bellemare, Marc G., Will Dabney, and Rémi Munos. "A distributional perspective on reinforcement learning." International Conference on Machine Learning. PMLR, 2017.

Zintgraf, L., et al. "VariBAD: a very good method for Bayes-adaptive deep RL via meta-learning." Proceedings of ICLR 2020 (2020).
limitations_and_societal_impact: The authors did an adequate job discussing the computational limitations of their approach as well as the types of contemporary RL methods the TAGI optimization approach could be currently applied to. There are however remaining questions regarding some of the technical limitations of the TAGI framework in light of overfitting, or getting stuck in local optima as the solution method still relies on collecting data proactively while the policy is trained.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes applying TAGI (Goulet et al, 2019), a framework for approximate Bayesian inference in deep neural networks, to deep q-learning. Using assumptions of Gaussianity on various distributions, TAGI infers both the neural network parameters and unit activations, without needing gradient descent. Because all distributions are assumed to be Gaussian, the means and variances can be updated analytically with linear computational complexity. TAGI is essentially used to infer the correct Q-value function, and the approach is applied to lunar lander, cartpole, and several Atari games.
main_review: Overall, I liked the idea of introducing Bayesian approaches to deep RL is an interesting, particularly as it offers a more principled way of handling the exploration-exploitation tradeoff compared to epsilon-greedy exploration. However, I do have several reservations on both the theoretical and empirical aspects on this work, and for the reasons given below, I cannot recommend this paper for acceptance as this stage.

Strengths:
* introducing Bayesian inference into deep RL allows for the possibility for a more principled approach to exploration-exploitation via Thompson sampling; exploration is driven by greater posterior uncertainty, while a lower uncertainty will drive greater exploitation
* compared to alternative approaches to Bayesian deep learning, TAGI's inference can be done analytically with linear computational complexity with respect to the number of network parameters

Theoretical weaknesses:
* the authors have chosen not to not use a target network for TAGI-DQN. This is rather controversial as it changes the training target. DQN updates the Q-network towards a fixed TD target. For the learning target to be fixed, it needs a fixed bootstrap, which therefore requires a fixed target network. By not using a fixed target network, TAGI-DQN attempts to infer the network parameters that minimize a different learning objective, what Sutton & Barto (2ed, Ch 11.5) call the mean squared TD error, and this is known to converge to an incorrect result.

Empirical weaknesses:
* the Atari results were ran for 40M frames (only a fifth of the 200M that is more standard). My concern is that this is too early to tell what the asymptotic performance of these algorithms will be like. For instance, it'd be nice to know whether any of TAGI's early leads over the backprop algorithm will be stable over longer horizons. It'd also be nice to know whether TAGI training eventually converges to some asymptotic performance or whether instabilities will emerge over longer training horizons.
* I think this work could also benefit from some ablation studies. For instance, the authors chose not to include a target network for TAGI: as this is a striking design decision, it'd be helpful to see what the effects of it is via an ablation study. Presumably, TAGI explored by Thompson sampling (it was not made explicit in the paper), and it'd be interesting to see how this compares to a more traditional epsilon-greedy policy.
* Finally, it'd be helpful to also examine TAGI in a policy evaluation setting to see whether it can converge onto a correct, known policy. 

---Review update---

I thank the authors for their response to my questions during the discussion period. In light of their comments, my previous theoretical concerns have been addressed, and I have raised my score to 5 to reflect that. But I still feel that the paper's baselines were rather thin, and multiple suggestions for improvement have been offered by myself and the other reviewers. I'd like to mention the atari baselines in particular. The authors' current choice is unusual and never used beyond the original paper. The authors compare against n-step Q-learning of Mnih et al (2016), but the 1-thread version of it. This baseline is never used simply because it performs poorly -- it has no replay buffer or parallel actors to make up for it. I appreciate the authors' intention was to demonstrate an improvement in the online learning setting. However, I think it'd still be helpful to compare against a more widely used atari baseline as well, such as DQN and/or the multi-actor version of n-step Q-learning. I also believe it'd be helpful if the authors could provide results on more games, as the current sample size is small -- the authors show TAGI is better in only 3 out of the 5 games they ran. Based on this, it is difficult to conclude whether the RL practitioner should expect TAGI to do better than gradient descent if she were considering training an agent on a new task.
limitations_and_societal_impact: A discussion of societal impact was not relevant for this work, as it was more foundational in nature and would not introduce additional impacts beyond that inherent generically in machine learning.
review_title: null

Meta-review: The reviewers have raised valid concerns that were not adequately addressed by the rebuttal, placing the paper below the bar.
Decision: Reject