Analytically Tractable Bayesian Deep Q-Learning
Authors: Luong-Ha Nguyen
James-A. Goulet
Venue: ICLR 2022 Submitted (2022)
Keywords: Bayesian Learning
Probabilistic Methods
Uncertainty Quantification
Reinforcement Learning
Deep Q-learning

Abstract: Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to reach human performance on video game benchmarks using deep Q-learning (DQN). The current consensus of DQN for training neural networks (NNs) on such complex environments is to rely on gradient-descent optimization (GD). This consensus ignores the uncertainty of the NN's parameters which is a key aspect for the selection of an optimal action given a state. Although alternative Bayesian deep learning methods exist, most of them still rely on GD and numerical approximations, and they typically do not scale on complex benchmarks such as the Atari game environment. In this paper, we present how we can adapt the temporal difference Q-learning framework to make it compatible with the tractable approximate Gaussian inference (TAGI) which allows estimating the posterior distribution of NN's parameters using a closed-form analytical method. Throughout the experiments with on- and off-policy reinforcement learning approaches, we demonstrate that TAGI can reach a performance comparable to backpropagation-trained networks while using only half the number of hyperparameters, and without relying on GD or numerical approximations. 

Under review as a conference paper at ICLR 2022
ANALYTICALLY TRACTABLE BAYESIAN DEEP Q-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to reach human performance on video game benchmarks using deep Q-learning (DQN). The current consensus of DQN for training neural networks (NNs) on such complex environments is to rely on gradient-descent optimization (GD). This consensus ignores the uncertainty of the NN's parameters which is key aspect for the selection of an optimal action given a state. Although alternative Bayesian deep learning methods exist, most of them still rely on GD and numerical approximations, and they typically do not scale on complex benchmarks such the Atari game environment. In this paper, we present how we can adapt the temporal difference Q-eaaniin framework to make it compatible with the tractable approximate Gaussian inference (TAGI) which allows estimating the posterior distribution of NN's parameters using closed-form analytical method. Throughout the experiments with on- and off-policy reinforcement learning approaches, we demonstrate that TAGI can reach performance comparable to backpropagation- trained networks while using only half the number of hyperparameters, and without relying GD or numerical approximations.
INTRODUCTION
Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to reach human performance on video game benchmarks using deep Q-learning (DQN) (Mnih) iet al. (2015; Van Hasselt et al., (2016). Deep RL methods typically require an explicit definition of an
function in order to compromise between using the current policy and exploring the potential of new actions. Such an issue can be mitigated by opting for Bayesian approach where we estimate the uncertainty of the neural network's parameters using Bayes' theorem, and select the optimal action given a state according to the parameter uncertainty using Thompson sampling technique (Strens) 2000). Bayesian deep learning (BDL) methods based on variational inference (Kingma et al., 2015 Hernaneez-Lobatt & Adams 2015 Blundell et al.) 2015 Louizos
& Welling, 2006]) Osawa et al. 2019] Wu et al. 2019). Monte-Cane dropout (Gal & Ghahramani 2016), Hamiltonian Monte-Carlo sampling (Neal) 1995) have shown perform well on regression
and classification benchmarks, despite being generally computationally more demanding than their deterministic counterparts. Note that none of these approaches allow performing the analytical infer- ence for the weights and biases defining the neural network. Goulet et al. (2021) recently proposed the tractable approximate Gaussian inference (TAGI) method that allows estimating the posterior of the neural network's 's parameters using closed-for form analytical method. More specifically, TAGI leverages Gaussian conditional equations to analytically infer this posterior without the need of numerical approximations (i.e., sampling techniques or Monte Carlo integration) or optimization on which existing BDL. methods rely. addition, TAGI is able to maintain the same computational complexity deterministic neural networks based the gradient backpropagation. For convolu- tional architectures applied on classification benchmarks, this approach was shown to exceed the performance of other Bayesian and deterministic approaches based on gradient backpropagation, and to do so while requiring a smaller number of training epochs (Nguyen & Goulet) 2021).
In this paper, we propose a new perspective on probabilistic inference for Bayesian deep reinforcement learning which, up to now has been relying on gradient- based methods. More specifically, we present how can we adapt the temporal difference Q-learning framework (Sutton) 1988 Watkins & Dayan

---

Under review as a conference paper at ICLR 2022
1992) to make it compatible with TAGI. Section
first reviews the theory behind TAGI and the expected value formulation through the Bellman's $ Equation. Then, we present how the action- value function can be learned using TAGI. Section Bayesian reinforcement learning, and Section compares the performance the TAGI-DQN with the deterministic DQN and off-policy RT. approaches.
B
presents the related work associated with
2 TAGI-DQN FORMULATION
This section presents how to adapt the DQN frameworks in order to make them compatible with analytical inference. First, Section 2.1 reviewws the fundamental theory behind TAGI, and Section [2. reviews the concept of long-term expected value through the Bellman's 's equation (Sutton & Barto
2018). Then, Section 2.3 presents how to make the Q-learning formulation (Watkins & Dayan 1992) compatible with TAGI.
2.1 TRACTABLE APPROXIMATE GAUSSIAN INFERENCE
Bayesian deep learning aims to estimate the posterior of neural network's parameters 0 conditional on given training dataset DD {(x,, yı), vi E D: using Bayes' theorem that
[(0 f(
I(D)
where f0) is the prior Probability Density Function (PDF) of parameters, f(D is the likelihood
and f(D is the marginal likelihood or evidence. Until recently, the posterior f in Equation |
has been considered to be intractable (Goodfellow al. 2016 Goan & Fookes 2020). TAGI has
addressed this intractability issue by leveraging the Gaussian conditional equations; For a random vector parameter and observations Y for which the joint PDF satisfies
/ y - N (
H4
Eg
E
1).
BY
Σyp
Σy
the Gaussian conditional equation deseribing the PDF of 0 conditional on y is defined following
f(0ly) (( Holy: Egjy)
Pol = μe + Σ E (y UY)
Σøly = Eg - E Exe-
In its simple form, the intractable computational requirements for directly applying Equation limits trivial neural networks. order overcome this challenge, TAGI leverages the inherent conditional independence structure of hidden layers Z 1 | under the assumptions that parameters 0 are independent between layers. This conditional independence structure allows breaking down the computations for Equation into layer-wise two-step procedure; forward uncertainty propagation and backward update.
The first forward uncertainty propagation step is intended to build the joint prior between the hidden states of layer j. ZO), and the parameters g(3) directly connecting into it. This operation is made by propagating the uncertainty from the parameters and the input layer through the neural network. In order to maintain the analytical tractability of the forward step, must address two challenges: The first challenge is related to the product of the weights W and activation units A involved in the calculation of the hidden states so that
Σ 1=1
6 A + 1,00
(4)
where B is the bias parameter, (j) refers to the jth layer, i refers to a node in the current layer, refers to node from the previous layer, and A is the number of hidden units from the previous layer. The second challenge to propagate uncertainty through the non-linearity when activating hidden states with activation function ((.). ((+11 (2(o+t).
(5)
In order to tackle these issues, TAGI made the following assumptions
2

---

Under review as a conference paper at ICLR 2022
A1. The prior for the parameters and variables in the input layer Gaussians.
A2. The product of weight and an activation unit in Equation 4 is approximated by Gaussian random variable, WA = M(Jua Wa). whose moments are computed analytically using Gaussian multiplicative approximation (GMA).
A3. The activation function <(.) in Equation | is locally linearized using a first-order Taylor expansion evaluated at the expected value of the hidden unit being activated.
Note that the linearizaion procedure in the assumption A3 may seems to be a crude approximation, yet has been shown match exceeds the state-of-the-an performance on fully-connected neural networks (FNN) (Goulet et al. 2021), as well as convolutional neural networks (CNN) and generative adversarial networks (Nguyen & Goulet 202T). In order to maintain a linear computational complexity with respect to the number of weights for the forward steps, TAGI relies on the following assumptions
A4. The covariance for all parameters in the network and for all the hidden units within same layer are diagonal.
A5. The hidden states on a given layer are independent from the parameters connecting into the subsequent layer, i.e., z 1 g())
A6. The hidden states at layer j + 1 depend only the hidden states and parameters directly connecting into it.
The empirical validity of those assumptions as well the GMA formulations are provided by Goulet et al. (2021).
The second backward update-step consists in performing layer-wise recursive Bayesian inference going from hidden-layer to hidden-layer, and from hidden-layer to the parameters connecting into it (see Figure TT. The assumptions A A1-A6 allow performing analytical inference while still maintaining linear computational complexity with respect to the number of weight parameters in the network where only the posterior hidden states for the output layer are computed using Equation 3 For the remaining layers, those quantities are obtained using the Rauch-Tung -tun-Sttiebbel recursive procedure that was developed in the context of state-space models (Rauch et al. 1965). For simplicity, we
define the short-hand notation (0+, Z+} = (00+1), 20+1) and (0, Z = (0()) z Z0) so that the posterior for the parameters and hidden states are computed following
f(Zly)
N(z: HZw Σz(w)
Pzv
uz Jz (Mz*ly - (1z))
(6)
Ezw
Eg JJ (E2'/1 Σz-) J
Jz = Ezz- Ez
f00y) = N(O: po Egly)
14y = 140 + Jo (1 (1z))
Eow Eg + Jø E2rly Σz-) J
Jø Ee
Figure presents directed acyclic graph (DAG) describing the interconnectivity in such neural network.
TAGI allows inferring the diagonal posterior knowledge for weights and bias parameters, either using one observation at a time, or using mini-batches of data. As we will show in the next sections, this online learning capacity is best suited for RI. problems where we experience episodes sequentially and where we need to define tradeoff between exploration and exploitation, as a function of our knowledge of the expected value associated with being in a state and taking an action.
2.2 EXPECTED VALUE AND BELLMAN'S EQUATION
We define r(s, a, 8²) as the reward for being in a state s E RS, taking an action a E A {a), {a1,02, 02, (4)) and ending in state s' E R For simplicity, use the short-form notation
3

---

Under review as a conference paper at ICLR 2022
f(O6) y) f(z
(
2
((L.))
(
O
y
Figure 1: Compact representation of the variable nomenclature and the dependencies associated with feedforward neural network. The red node denotes the input vector, green nodes are vectors of hidden units 2, and purple node denote the observation vector. The gray arrows represent the weights and bias 0 connecting the different hidden layers and magenta arrows outline the flow of information that takes place during the inference step.
for the reward r(8, a, 8') = r(s) in order define the value as the infinite sum of discounted rewards
css)
*(((())))
(8)
k=
As do not know what will be the future states Sc+ for need to consider them random variables (St+k), so that the value (sr) becomes random variable as well,
V(s:) = r(s)) + Σ r ((S++t))
1=1
Rational decisions regarding which action to take among the set A is based the maximization of the expected value as defined by the action-value function
o
((,,, ar) = pv = (VV ae x)] = r(8:) +
Σ (r(
(10)
where it assumed that at each time f, the agent takes the action defined the policy æ. In the case of episode- based learning where the agent interacts with the environment, assume know the tuple of states and 8,41, so that we can redefine the value as
V(s a.) = r(s,) + ((((+1)+
x Ar(
(11)
k=1
= r(s:) + (V(8)t1,
Assuming that the value V NNx ,v, variables, we can reparameterize these equations as the sum of the expected value q(8, a) and zero-mean Gaussian random variable & (eee;; 0, 1), so that
ot) p in Equations
9 and 11 described by Gaussian random
V(8.a) q(s, a) -ovE
(12)
where the variance of and are assumed here to be independent of and a. Although in a more general framework this assumption could be relaxed, such an heteroscedastic variance term is outside from the scope this paper. Using this reparameterization, we can write Equation [I as the discounted difference between the expected values of two subsequent states
((81,a;) a.)
r(s:) + (1((1141:444)) ov, + 7ovin
(13)
r(si) + ( ((41) + ovE.
Note that in Equation 13 and yoV. can be combined in single standard deviation parameters σv with the assumption that E, L E,, +
In the case where at time t. we want to update the Q-values encoded in the neural net only after observing n-step returns (Mnih al. 2016), we can reformulate the observation equation so that
n-
(Ss, (4) = Σ
7r((S())) + - - g( an) + &ov = (1,2,.... ... -n-1 n-1}. 1}.
(14)
4

---

Under review as a conference paper at ICLR 2022
Note that in the application of Equation 14
employ the simplifying assumption that & 1 E1+s, Vi & O, as Equation 13 already makes simplifying ussumptions for the independence of of and E. Note that in a more general framework, this assumption could be relaxed. An example of n-step returns is presented the the algorithm displayed in $1 from Appendix A
The following subsections will present, for the case of categorical actions, how model the deter- ministic action-value function q(ss a) using a neural network.
2.3 TAGI DEEP Q-LEARNING FOR CATEGORICAL ACTIONS
Suppose we represent the environment's state at time & and + by s'). and the expected value for each of the A possible actions E A by the vector q E TRA In that context, the role of the neural network is to model the relationships between {s, and q. Figure presents a directed acyclic graph (DAG) describing the interconnectivity in such neural network, where red nodes denote state variables, green nodes are vectors hidden units 2, the blue box a compact representation for the structure of convolutional neural network, and where gray arrows represent the weights and bias O connecting the different hidden layers. Note that unlike other gray arrows, the magenta ones in (b) not directed ares representing dependencies, but they simply outline the flow of information that takes place during the inference step. For simplification purposes, the convolutional operations are omitted and all regrouped under the CNN box. In order learn the parameters of such network,
8
CNN
(
((L)
22
g(u)
q
(a) Neural network DAG for modelling the action-value function
O
((n))
CNN
g
1
0
(2)
TYE
(b) DAG for the temporal-difference
difference Q-learning configuration
Figure 2: Graphical representation of neural network structure for temporal-different Q-learning with categorical actions. The red nodes denote state variables, green nodes are vectors of hidden units 2, and the blue box a compact representation for the structure of convolutional neural network. The gray arrows represent the weights and bias O connecting the different hidden layers and magenta arrows outline the flow of information that takes place during the inference step.
need to expand the graph from Figure 2 to include the reward the error term øye, and q', the g-values of the time step + 1. This configuration is presented in Figure 2p where the nodes that have been doubled represent the states and s' which are both evaluated in network sharing the same parameters. Note that the value of q' is computed using the same network presented in Figure 2 in which this network acts predictor taking input s'. When applying Equation 13 q-values corresponding to specific action can be selected using a vector h; E {0, 1}AA having a single non-zero value for the i-th component identifying which action taken at time t so that
[g]. = h.q..
(15)
In the context TAGI-DON, we employ Thompson sampling Strens 2000) for the selection of an action given state. More specifically, the vector h E {0, 1) is defined such that the i-th non-zero value corresponds the index of the maximal value among q'_ vector of realizations from the neural network's 's posterior predictive output Q (g': AQID, EQrd). Because of the Gaussian assumptions in TAGI, this posterior predictive readily available from the forward uncertainty propagation step, outlined in $2.1
The red arrows in Figure 2p outline the flow of information during the inference procedure. The first step consists in inferring using the relationships defined in either Equation 1r [4] As this is linear equation involving Gaussian random variables, the inference is analytically tractable. From there, follow the same layer-wise recursive procedure presented in 2.. in order learn
5

---

Under review as a conference paper at ICLR 2022
the weights and biases in 0. With the exclusion of the standard hyperparameters related to network architecture, batch size, buffer size or the discount factor, this TAGI-DQN framework only involves a single hyperparameter, dv. the standard deviation for the value function. Note that when using CNNs with TAGI, Nguyen & Goulet (202T) recommended using a decay function for the standard deviation of the observation noise that after seing batches of n-steps,
ov maxxoy min (V))
(16)
because it allows putting more weight on the prior rather than on the likelihood during the inference step at the early stage. This effect then diminishes with time. The model in Equation ho three hyperparameters, the minimal noise parameter of the decay factor 7j and the initial noise parameter av. As it was shown by Nguyen & Goulet (202T) for CNNs and how we show in for RL problems, TAGI's performance is robust towards the selection of these hyperparameters.
A comparison of implementation between TAGI and backpropagation on deep Q-network with experience replay (Mnih et al. 2013) shown in Figure 3 A practical implementation of n-step TAGI deep Q-learning is presented in Algorithm from Appendix A
Algorithm TAGI-DQN with Experience Replay
Algorithm 2: DQN with Experience Replay
Initialize replay memory capacity N; Σy;
Initialize replay memory to capacity N
2 Initialize parameters 0;
Initialize parameters 6;
Discount factor
Discount factor 7:
for episode 1
Define (cpsilon- gredddy function);
Reset environment So:
for episode
for t= 1: do
Reset environment Soi
q(st, a) Q(s., ~ N NiN0 ( as) (8 (s),a))
for do a U ~ (4(0 1);
arg g(x
Sco1, =nvii==nnnnon/ Store {s1, (1411) R:
aEd
zandi(A)
arg max Qo(s., a) a
Sample random batch of {8,, a5. 85+11
8,41, cnviroment(a.)) (a:):
((5+41,a)) Q(s+e1.d') N Ning (15-1, o'), E8 (sp1r,a))
Store (st, Pt, (++1) in R;
13
d's+ arg max (()+1,,_)
Sample random batch (55.0,,1)
d'EAA
MY, ry + 7 (8)11
Vj =y max mmmxmmmxxaaa. (e))==1.a)))
15
Ey, = " (5y+1: 1/41) Ey:
Update 0 using gradient descent
16
Update using TAGI on f(Oly) (Equation
L 0.5 [u; Q0(s),a)))
Figure 3: Comparison of TAGI with backpropagation on deep Q-network with experience replay. L: loss function; Wi uniform distribution; randi: uniformly distributed pseudorandom integers.
3 RELATED WORKS
Over the last decades, several approximate methods have been proposed in order to allow for Bayesian neural networks (Neal) 1995 Kingma et al. 2013] [Hernandez-Lobato & Adams 2015 Blundell
et al., 2015 Louizos & Welling 2016 Osawa et al., 2019; Wu et al. 2019 Gal & Ghahramani 2016) with various degree of approximations. Although some these methods have shown to be capable of tackling classification tasks on datasets such ImageNet (Osawa et al.) 2019) none of them have been applied on large-scale RI. benchmark problems such as the Atari games environement. The key idea behind using Bayesian methods for reinforcement learning is to consider the uncertainty associated with Q-functions in order to identify tradcoff between exploring the performance of possible actions and exploiting the current optimal policy (Sutton & Barto 2018). This typically takes the form of performing Thompson sampling (Strens 2000) rather than relying on heuristics such as c-greedy.
For instance, MC dropout (Gal & Ghahramani 2016) introduced has method intrinsically suited for reinforcement learning. Nevertheless, five years after its inception, the approach has not yet been reliably scaled to more advanced benchmarks such the Atari game environment. The

---

Under review as a conference paper at ICLR 2022
2015 which was recently applied to simple RL problems (Lipton et al. 2018), and which has not yet been applied to more challenging environments requiring convolutional networks. On the other hand, Bayesian neural networks relying sampling methods such as Hamiltonian Monte-Carlo (Neal 1995) are typically computationally demanding to be scaled to RL problems involving such complex environment.
same applies to Bayes-by by-backprop (Blundell et al.
Although mainstream methods related to Bayesian neural networks have seldom been applied to complex RL problems, several research teams have worked on alternative approaches in order to allow performing Thompson sampling. For instance, Azizzadenesheli et al. (2018) have employed a deep Q-network where the output layer relies on Bayesian linear regression. This approach was shown to be outperforming its deterministic counterparts on Atari games. Another approach by [Osband et al. (2016) employs bootstrapped deep Q-networks with multiple network heads in order to represent the uncertainty in the Q-functions. This approach was also shown to scale to Atari games while presenting an improved performance in comparison with deterministic deep Q-networks. Finally, Wang & Zhou (2020) have tackled the same problem, but this time by modelling the variability in the Q-functions through a latent space learned using variational inference. Despite its good performance on the benchmarks tested, it did not allowed to be scaled to the Atari game environment.
The TAGI deep Q-network presented in this paper is the first demonstration that an analytically tractable inference approach for Bayesian neural networks can be scaled to problem challenging as the Atari game environment.
4 BENCHMARKS
This section compares the performance of TAGI with backpropagation-baed standard implementa- tions on off- and on-policy deep RL. For the off-policy RL, both TAGI- and backpropagation- based RL approaches are applied to deep Q-learning with experience replay (see Algorithm & 2) for the lunar lander and cart-pole environments, For the on-policy RL, TAGI is applied the n-step Q-learning algorithm (Mnih et al. 2016) and is compared with its backpropagation-baine -basdd counterpart. For this purpose, we perform the comparison on five Atari games including Beamrider, Breakout, Pong. Qbert, and Space Invaders. Note that these five games hyperparameters for the entire Atari games (Mnih et al. 2016; 2013) because they represent the features sharing with the remaining Atari games. All benchmark environments taken from the OpenAl Gym (Brockman et al. 2016
commonly selected for tuning
4.1 EXPERIMENTAL SETUP
In the first experiments with off-policy RL, we use a fully-connected multilayer perceptron (MLP) with two hidden layers of 256 units for the lunar lander environment, and with one hidden layer of 64 units for the cart pole environment. In these experiments, there is no need for input processing nor for reward normalization. Note that unlike for the deterministic deep Q-network with experience replay, TAGI does not separate the Q-network with the target network that is used for ensuring the stability during training and allows eliminating the hyperparameter related to the target update frequency. For the deep Q-network trained with backpropagation. we employ the pre-tuned implementation of OpenAl baselines (Dhariwal et al., 2017) with all hyperparameters set to the default values.
For the Atari experiments with on-policy RL, we use the same input processing and model architecture as Mnih et al. (2016). More specifically, the Q-network uses two convolutional layers (16-32 filters) and full-connected MLP of 256 units. The n-step Q-learning only uses a single network to represent the value function for each action, and relies on a single learning agent in this experiment. The reason behind this choice is that TAGI current main library is only available on Matlab which does not support running a Python multiprocessing module such as the OpenAl gym. In the context of TAGI, we an horizon of 128 steps. As recommended by AAdrychhwicc et al. (202I) and following practical implementation details (Pytorch 20119] 2020), each return in n-step Q-learning algorithm is normalized by subtracting the average return from the current n-steps and then dividing by the empirical standard deviation from the set of n returns.
The standard deviation for the value function, (av), is initialized at 2. σv is decayed each 128 steps with a factor 7) = 0.9999. The minimal standard deviation for the value function min = 0.3. These hyperparameters values were not grid-searched but simply adapted to the scale of the problems
7

---

Under review as a conference paper at ICLR 2022
and are kept constant for both experiments. The complete details of the network architecture and hyperparameters are provided in Appendix and
4.2 RESULTS
For the first set of experiments using off-policy RL, Figure 4 presents the average reward over 100 episodes for three runs for the lunar lander and cart pole environment. The TAGI-based deep Q-learning with experience replay shows faster and more stable learning than the one relying on backpropagation.
300
200
Average rewand
Average reward
150
150
TAGE
100
Backpropegation
-150
-300
0.5 Number of steps (M)
0.5 Number of steps (M)
(a) Lnder-an
(b) CartPole-vO
Figure 4: Illustration of average rewards over 100 episodes of five runs for one million time steps for the TAGI-based and backpropagation-baal -based deep Q-learning.
Table I shows that the average reward over the last 100 episodes obtained using TAGI are greater than the one obtained using backpropagation. Figure compares the average reward over 100 episodes
Table I:
Average reward over the last 100 episodes for the lunar lander and cart pole experiments. TAGI: Tractable Approximate Gaussian Inference.
Method
Lunar lander
Cart pole
TAGI
279.9 + 6.1
199.3 = 1.0
Backpropagation
197.2 + 84.3
137.7 + 16.5
for three runs obtained for TAGI, with the results from Mnih al. (2016) for the second set of experiments on Atari games. Note that all results presented were obtained for a single agent, and that the results for the backpropagation-ttriinn trainee networks are only reported the end of each epoch.
Results show that TAGI outperforms the results from the original n-step Q-learming algorithm trained with backpropagation (Mnih al. (2016) on Breakout, Pong, and Qbert, while underperforming on Beam Rider and Space Invaders. The average training time of TAGI for an Atari game is approximately 13 hours on GPU calculations benchmarked on a 4-core-intel desktop of 32 GB of RAM with NVIDIA GTX 1080 Ti GPU. The training speed of TAGI for the experiment of the off- policy deep RL is approximately three times slower on CPU caleulations than the backpropagation- trained counterpart. The reason behind this slower training time is because of its intrinsically different inference engine making TAGI's implementation incompatible with existing librarics such TensorFlow Pytorch. TAGI's library development is still ongoing and is not yet fully optimized for computational efficiency. Overall, these results for on- and off policy RL approaches confirm that TAGI can be applied to large scale problems such deep Q-learning.
DISCUSSION
Although the performance of TAGI does not systematically outperform its backpropagation based counterpart, it requires fewer hyperparameters (see Appendix C). This advantage is one of the key
8

---

Under review as a conference paper at ICLR 2022
8,000
TAGI
180
21
Averige reward
6,000
Backpropa gation
Average reward.
135
Average reward
10
4,000
90
2,000
45
-10-
10
30
20
30
50
-1
20
50
Number spochs
Number of epocls
Number of epochs
(a) Beam Rider
(b) Breakout
(c) Pong
12,000
800
Average reward
Average reward
640
9,000-
480
6,000
320
3,000-
160
20
30
40
50
10
20
30
40
50
Number of epocts
Number of epochs
(d) Qbert
(e) Space Invaders
Figure 5: Illustration of average reward over 100 episodes of three runs for five Atari games. The number of epochs is used here for the comparison of TAGI and backpropagation-trained counterpart obtained by Mnih et al. ()016). Each epoch corresponds to four million frames. The environment identity are (Atari Game NoF ameski ip---
aspects for improving the generalization and reducing the computational cost of the hyperparameter tuning process which are the key challenges in current state of deep RI. (Irpan] 2018 Henderson et al. 2018). For instance, this paper, the TAGI's hyperparameters relating to the standard deviation of value function (ov) are kept constant across all experiments. Moreover, since these hyperparameters were not subject to grid-search
order to optimize the performance, the results obtained here representative of what user should obtain by simply adapting the hyperparameters fit the specificities and scale the environment at hand.
More advanced RL approaches such advanced actor critic (A2C) (Mnih et al. 2016) and proximal policy optimization (PPO) (Schulman et al., 2017) employ two-networks architectures in which one network is used approximate value function and other is employed to encode the policy. The current TAGI-RL framework is not yet able to handle such architectures because training a policy network involves an optimization problem for the selection of the optimal action. Backpropagation- based approach currently rely on gradient optimization to perform this task, while TAGI will require developing alternative approaches order maintain the analytical tractability without relying on mudient--aseed optimization.
CONCLUSION
This paper proposes Bayesian reinforcement learning framework that combines TAGI with deep Q-learming. The proposed framework allows estimating the uncertainty of neural network's 's param- eters and taking an action given a state according to the parameter uncertainty using Thompson sampling. Throughout the experiments with on- and off off-policy reinforcement learning approaches, we demonstrate that TAGI can reach a performance comparable to backpropagation-tral -raindd networks while using only half the number of hyperparameters. These results challenge the common belief that for large scale problems such as the Atari environment, neural networks can only be trained using the gradient backpropagation. We have shown here that this current paradigm is longer the only alternative as TAGI has linear computational complexity and can be used to learn the parameters of complex networks in analytically tractable manner, without relying gradient-based optimization.

---

Under review as a conference paper at ICLR 2022
REFERENCES
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pictquin, M. Michalski. S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study, In International Conference on Learning Representations, 2021. URL htattps://openeee
net/ forum? daarummid=nIAxxssn....
K. Azizzadenesheli, E. Brunskill, and A. Anandkumar. Efficient exploration through Bayesian deep q-networks. In IEEE Information Theory and Applications Workshop, pp. 1-9, 2018.
C. Blundell, J. Comebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 150.05444, 2015.
G. Brockman, Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint 16X.::66000014000 2016.
P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov. Openai baselines. https://gil
com
ines 2017.
Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICMI. proceeding,, pp. 1050-1059, 2016.
E. Goan and C. Fookes. Bayesian neural networks: An introduction and survey. In Case Studies in Applied Bayesian Data Science, pp. 45-87. Springer, 2020.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT Press, 2016.
J-A. Goulet, L.H. Nguyen, and S. Amiri. Tractable approximate Gaussian inference for Bayesian neural networks. Journal of Machine Learning Research (accepted for publication), 22:1-23. September 2021.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
J. M. Hernández-Lobato and R. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Leurning, pp. 1861-1869, 2015.
A. Irpan. Deep reinforcement learning doesn't 'ttt work yet. Ttts:://w..aex.rppan 2018/02/14/r1-hatt/ html 2018
D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterivation trick. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, 2015.
Z. Lipton, X. Li, J. Gao, I.. Li, F. Ahmed, and I.. Deng. Bbq-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. ICMI. proceeding, pp. 1708-1716, 2016.
V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13121 December 2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, and G Ostrovski. Human-level control through deep reinforcement learning, nature, 51897540):529-533, 2015.
V. Mnih, Adria P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In proceddos,,,, pp. 1928-1937. PMLR, 2016.
R. M. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
10

---

Under review as a conference paper at ICLR 2022
Luong-Ha Ha Nguyen and James-A Goulet. Analytically tractable inference in deep neural networks. 4th Workshop on Tractable Probabilistic Modeling (UAI), 2021.
K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, and M. E. Khan. Practical deep learning with Bayesian principles. In Advances Neural Information Processing Systems proceedings, 2019.
I. Osband, C. Blundell, A. Pritzel, and Benjamin V. Roy. Deep exploration via bootstrapped dqn. In NEURIPS proceedings, pp. 4033 4041, 2016.
Pytorch. Pytorch examples for reinforce algorithm.
http:///glthub..com//yyrorotttl
2019.
Pytorch. Pytorch examples for actor ertic algorithm. httpss//giichab.ccmmppttot httpp////sssteeeeeeeeerttttttaaa
pttptttrrrrriiii.p.......... 2020.
H. E. Rauch, C. T. Striebel, and F. Tung. Maximum likelihood estimates of linear dynamic systems.
AIAA journal, 345):1445-1450, 1965.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
M. Strens. A Bayesian framework for reinforcement learning. In peoceddess,,,,, pp. 943-950, 2000.
R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1): 9-44, 1988.
R. Sutton and G. Barto. Reinforcement learning: An introduction. MIT Press, 2nd edition, 2018.
H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Z. Wang and M. Zhou. Thompson sampling via local uncertainty. In ICML proceedings, volume 119, pp. 10115-10125, 13-18 Jul 2020. URL http:://prrrceeeingg....... wwt.sm....rreevl11......
wang20ab. html
C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):27--292, 1992.
A. Wu, S. Nowozin, E. Meeds, R. E. Turner, J. M. Hernández-Lobato, and A. L. Gaunt. Deterministic variational inference for robust Bayesian neural networks. In ICIR proceedings, 2019.

---

Under review as a conference paper at ICLR 2022
A ALGORITHM
This section presents the n-steps Q-learning algorithm with Tractable Approximate Gaussian Infer- ence (TAGI).
Algorithm 3: n-step Q-learning with TAGI
Initialize 0 Σv; number of steps (N)
Initialize memory R. capacity N;
steps 00;
for episode 1 E do Reset environment So: for t 1: T do
Steps = steps 1;
g(s;, a) Q(s,, a) ~ (N (5e, a), 2g ());
a, arg max ((,,, a):
aEA
8/ ( Store (sc: dcs ry) in R; if steps mod N ((s++1,a') Q(5(+1(( N (W (st+1, d'), Eg (8)+1,
then
13
d'+1 = arg max a()+1, aEA
Take N samples of {sj, aj. rý} from R;
, = 11 ($t+1, 4(41); Dー = Eg (441,
for j = N - 1: do
" - + yp 1 my = 7 د + Ev:
Update 0 using TAGI; Initialize memory R to capacity N;
B MODEL ARCHITECTURE
This appendíx contains the specifications for each model architecture in the experiment section. D refers to a layer depth; W refers to a layer width; refers to the layer height case of convolutional or pooling layers; K refers the kernel size; P refers to the convolutional kernel padding: S refers to the convolution stride; a refers to the activation function type; ReLU refers to rectified linear unit; N₂ refers to the number of actions.
Table 2: Model Architecture for Cart pole
Layer
D x H
K x
K
P
Input
4 1 X 1
Full connected
64 x 1 1
ReLU
Output
2x1x1
C HYPERPARAMETER
This appendix details the hyperparameters for each model architecture in the experiment section
12

---

Under review as a conference paper at ICLR 2022
Table 3: Model Architecture for Lunar lander
Layer
D x W x H K x
P S
Input
8 1 X 1
Full connected
256 1
ReLU
Full connected
256
ReLU
Output
4x1x X
Table 4: Model Architecture for Atari domain
Layer
D W x H
K x
K
P
S
Input
x 84 84
Convolutional
16 x 20 x 20
8 8
ReLU
Convolutional
32 x 9 x 9
4x4
2
ReLU
Full connected
256 x 1x1
ReLU
Output
N₂ X x1x11
Table 5: Hyperparameters for Cart pole and Lunar lander
Method
#
Hyperparameter
Value
1
Initial standard deviation for the value function (av)
2 0.9999
2
Decay factor (n)
Minimal standard deviation for the value function (omin)
0.3
TAGI
4
Buffer size
50 000
5
Batch size
10
6
Discount (7)
0.99
1
Learning rate
5 x 10-4 10-5
Adam epsilon
3
Adam 81
0.9
4
Adam B2 Buffer size
0.999
5
50 000
Backprop
6
Exploration fraction
0.1
Final value of random action probability
0.02
Batch Size
32
9 Discount (7)
0.99
10 Target update frequency
500
11 Gradient norm clipping coefficient
10
13

---

Under review as a conference paper at ICLR 2022
Table Hyperparameters for Atari domain
Method
#
Hyperparameter
Value
Horizon
128
2 Initial standard deviation for the value function (av)
3
Decay factor (1))
0.9999
4
Minimal standard deviation for the value function (apin)
0.3
TAGI
5
Batch size
32
6 Discount (7)
0.99
7
Number of actor-learners
Horizon
2 Initial learning rate Learning rate schedule
LogUniform(10 10
3
LinearAnneal 0)
4 RMSProp decay parameter
0.99
5 Exploration rate (G))
0.1
6 Exploration rate 2 (62)
0.01
Backprop
7
Exploration rate 3 (Eg)
0.5
8
Probability of exploration rate
0.4
9 Probability of exploration rate 2
0.3
10 Probability of exploration rate
0.3
11 Exploration rate schedule (first four million frames)
Anneal from E1,
12 Batch size
13 Discount (7)
0.99
14 Number of actor-learnrs
14


Reviews:
[{"review_id": "z-dvrWLG-ou", "reviewer": "Reviewer_6wux", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents an interesting new solution to the problem of performing Bayesian deep RL, which is certainly a challenging problem for which a lot of use can be seen. As they note, the associated uncertainty is certainly very useful for driving appropriate strategies for the agent to balance exploration and exploitation. However I don't think the paper is quite ready to be published.\n\nFirstly, I'm not sure this citation is correct:\nJ-A. Goulet, L.H. Nguyen, and S. Amiri. Tractable approximate Gaussian inference for Bayesian\nneural networks. Journal of Machine Learning Research, 22:1–23, 2021.\nI can't find the paper on the JMLR website (https://jmlr.org/papers/v22/), and as far as at least Google Scholar is showing it only seems to exist on arxiv. Of course, if I just haven't looked sufficiently hard enough (or in the wrong place) please correct me. This may seem like an odd complaint for the main review, but since the method presented in the paper is a direct extension of this TAGI method to deep RL it's important to be clear where the starting point is. This makes it a little harder to validate the claims made about the underlying TAGI method, since as far as I can see it does not appear to have been peer-reviewed yet. Having looked at it myself briefly, I will assume for the purposes of this review though that the claims hold.\n\nIn the experimental section there are no comparisons made to any other Bayesian deep RL methods despite them being discussed in the paper. This seems to be justified by saying that the presented method is sufficiently special and distinct because it is not a gradient based method. I'm not really sure why that's particularly important, but even so I think it very useful/necessary to have comparisons with those methods anyway so as to see the relative performance.\n\nThe comparison is always made to a deterministic neural network but importantly following a $\\epsilon$-greedy exploration strategy, as opposed to the Thompson sampling done by this method. Other exploration strategies are available (upper confidence bound for example) and I think should all be considered (or at least explained why not considered) in the paper and experiments.\n\nA large deal is made of the fact that no gradients are required - why this is such a major contribution should probably be discussed furthering the paper as I don't think that it's made particularly clear at the moment.\n\nI really appreciate the use of the colours in the figures - I'm not sure though whether magenta and grey are the best choices as they don't particularly stand out.\n\n", "summary_of_the_paper": "The authors present a method for combining deep Q-learning with tractable approximate Gaussian inference to allow for deep Bayesian reinforcement learning without the need for gradient based updates. They demonstrate this method can attain performance comparable to standard backprop trained agents in a number of environments.", "summary_of_the_review": "The paper presents a potentially promising method for Bayesian deep RL that can scale to large states spaces like Atari problems. I do not however believe that the paper is ready to be published as is since the experimental validation and comparison needs expanding, and further discussion on why exactly this method is so useful compared to the current literature would be useful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "yUUa8IDdiIV", "reviewer": "Reviewer_Aorz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n1. Potentially useful to be able to implement deep Q-learning with the tractable approximate Gaussian inference (TAGI) method, rather than with the standard semi-gradient approach.\n2. Demonstrates an alternative implementation of deep Q-learning at the scale of Atari games.\n3. Proposes a fully Bayesian way of accounting for neural network parameter uncertainty in the value function during action selection.\n4. Offers a strategy for reducing the number of hyperparameters involved in deep Q-learning.\n\n----\n\n### Weaknesses:\n\nThis paper does not focus on a clear contribution. The abstract claims that using Bayesian methods to account for the uncertainty of the NN’s parameters is key to improving decision making, but the evaluation focuses only on reducing hyperparameters and obtaining comparable performance to existing methods. Each of the strengths listed in the section above could be a fine topic for a paper if executed well, but the submitted paper moves on before fully answering any of the questions it raises.\n\n(Note: each of the numbered weaknesses below corresponds with the same numbered strength above.)\n\n1. If the objective is to show that it is both possible and useful to implement deep Q-learning with TAGI, then I would expect to see a full derivation for _Q-learning_, as well as some motivation or evidence for  _why_ it is useful.\n    - Currently the paper only appears to consider the derivation for _value estimation_. My (somewhat uninformed) impression of the original TAGI paper is that it demonstrated both regression and classification in deep neural networks. So if one were to take the perspective that estimating the value function amounts to solving a regression problem, then the original paper already demonstrated, at least in principle, that this was possible.\n    - But the value estimation problem is only one aspect of Q-learning, and it’s not obvious from the derivation how actions would fit in. Eqn (10) defines $q(s_t,a_t)$ in terms of a fixed policy, which does not make the dependence on actions clear. Then the paper redefines $V(s_t, a_t)$ as being action-dependent, which seems like an error, without describing what this dependence looks like (eqn. (11)).\n    - Finally, the paper does not provide much motivation for _why_ this alternative implementation would be desirable.\n2. If the objective is to demonstrate that TAGI can support deep Q-learning at scale, then I would expect an answer to the question: “how well does it work?”. I feel like the paper provides a partial answer, but the experiments don’t give me enough confidence to answer the question decisively.\n    - For instance, the results are only for 3 seeds, and the shaded regions are not labeled in the plots, so it’s unclear whether the results are significant. In general, my expectation with Atari would be to see at least 5 seeds, and ideally 10 or more. With CartPole and Lunar Lander, experiments are typically much cheaper to run, and I would expect to see more like 100+ seeds. See Henderson et al. (2018) for a discussion of why this is important.\n    - It’s unclear what the baseline agent is for the Atari experiments. Is it standard DQN? If so, I’m surprised by the Atari baseline performance curves. Performance is much worse than the original for Breakout (\\~150 here, vs. 401.2), Qbert (\\~2900 vs. 10596), and Space Invaders (\\~450 vs. 1976). After reading through the appendix, my impression is that the “Backpropagation” agent trains on the same amount of experience as the DQN from Mnih et al. (2015)’s Nature paper, but the hyperparameters and network architecture seem to be different. I’m left wondering why the paper uses these settings rather than comparing against the original published DQN.\n3. Given that the method claims to offer a tractable way to account for NN parameter uncertainty, I would expect to see a clear indication that this enables an agent to make _better_ decisions than methods that either don’t take a fully Bayesian approach or that don’t account for such uncertainty at all. Instead, the paper only claims to achieve “comparable” performance, and only compares against one baseline. I feel that there are a few problems with this.\n    - In my opinion, “comparable” doesn’t accurately characterize the performance. I would instead say the results are “mixed” versus the “backpropagation” agent. On the Atari games, TAGI appears to be better on three out of five, but much worse on one and somewhat worse on another.\n    - Regardless, neither “comparable” nor “mixed” results clearly demonstrate _better_ decision making capabilities.\n    - If the story is in fact more complicated---i.e. if properly accounting for parameter uncertainty doesn’t _always_ lead to better performance---then I would want to understand when to expect it to be better and when to expect it to be worse. The experiments suggest that this may be the case, but such a discussion is absent from the results presented in the paper.\n    - The paper cites Azizzadenesheli et al. (2018) and Osband et al. (2016), both of which appear to account for uncertainty and allow for Thompson sampling during action selection. I’m left wondering why the paper doesn’t compare against those approaches when evaluating the performance of the proposed method.\n4. I am unconvinced that the paper demonstrates that TAGI reduces the number of hyperparameters required for deep Q-learning. Examining the hyperparameters in the appendix, I don’t find the comparison conclusive because the decisions about what should count as a hyperparameter are subjective.\n    - For example, half of the “backprop” agent’s hyperparameters are related to the epsilon schedule, whereas the same agent’s learning rate schedule is represented as a single hyperparameter. Alternate forms of epsilon schedule exist that could reduce this number significantly. The paper does not justify why this particular way of accounting for them is appropriate.\n    - If the objective is simply to reduce the number of hyperparameters, are there other strategies worth comparing against? What makes this strategy for reducing hyperparameters so compelling? Is there a way to go fully Bayesian and incorporate the hyperparameters into the graph, so as to effectively get rid of _all_ of them?\n\n----\n\n### Questions:\n1. Can you comment on why it is reasonable to assume that the value V is well approximated by a Gaussian distribution? Can we expect this assumption to hold in general, or only for specific environments?\n2. Is the backprop baseline intended to exactly reproduce the Nature DQN result? If so, why are the results so different from the original paper? If not, what was the rationale for using the particular DQN architecture and hyperparameters for the Atari experiments?\n3. Any idea why TAGI performs better than backprop on some Atari games but worse on others?\n4. Was there a reason for not comparing against Azizzadenesheli et al. (2018) and Osband et al. (2016)? Is there a benefit to the proposed method over these other approaches, aside from mathematical rigor?\n5. Why does TAGI use a horizon of 128 steps, but the backpropagation baseline uses a horizon of 5 steps? Is this a fair comparison?\n6. I’m confused about the exploration rates for backprop on Atari. How are the different epsilon values used in the schedule?\n7. What does it mean for the initial learning rate to be LogUniform(1e-4, 1e-2)? Does this mean the learning rate is chosen randomly for each seed?\n8. The exploration fraction (0.1)---which I assume is the same thing as epsilon---for backprop on CartPole and Lunar Lander seems rather high. Are you decaying that to 0.02? If not, how was this value chosen?\n9. It’s interesting that the variance for TAGI seems to be much lower than for backprop on CartPole and Lunar Lander, but much higher on Atari. Does this hold up with more seeds? What might explain this behavior?\n10. Is there a potential connection between entropy regularized RL and this notion of using a decay function for the observation noise variance to more strongly incentivize the prior during early learning?\n\n----\n\n### Comments:\n1. For a reader unfamiliar with TAGI, it’s unclear which parts of this paper are reviewing existing work and which are proposing original contributions.\n2. In section 2.1, $J_Z$ and $J_\\theta$ are undefined. I assume these refer to the Jacobian? It would be useful to say what they are explicitly.\n3. In section 2.2, the “short form notation for the reward” makes it difficult to see how actions fit into equations (8)-(11).\n4. Final paragraph of Sec. 5: “advanced actor critic” should be “advantage actor critic”. I also don’t think Mnih et al. (2016) is the right reference for A2C, since that paper only discusses the asynchronous version, A3C. OpenAI Baselines or “Learning to Reinforcement Learn” by Wang et al. (2016) would be more appropriate.\n5. The formatting of the references is inconsistent. Some references use the full name of the publication, while some use abbreviations. Some include conference paper URLs; others don’t, etc. References should consistently use the same style, and use the full publication name (with proper capitalization) rather than the abbreviation.\n\n----\n\n### References:\n1. **Henderson et al. (2018) –** ​​Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. \"Deep Reinforcement Learning that Matters.\" In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 3207--3214, 2018.\n2. **OpenAI Baselines –** either https://github.com/openai/baselines or https://arxiv.org/abs/1708.05144\n3. **Wang et al. (2016) –** https://arxiv.org/abs/1611.05763\n\n", "summary_of_the_paper": "The paper describes how to adapt the Q-learning algorithm so that it is compatible with Bayesian deep learning, as opposed to the standard (semi-)gradient-based deep learning implementation. It explains how to propagate uncertainties about network parameters through the neural network computations and ultimately to the q-values, which enables action selection using Thompson sampling. Finally, it presents empirical results on CartPole, Lunar Lander, and five Atari games.", "summary_of_the_review": "The paper explores using Bayesian methods to guide action selection in deep Q-learning, and it makes some progress in that direction. I appreciate the time and effort spent in preparing this submission, and I do see some value in the proposed approach. Unfortunately, in its current form, the paper does not have a clear contribution. The paper attempts to answer too many questions and does not sufficiently answer any of them. Additionally, the main derivation is hard to follow and appears to be incomplete or incorrect. I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uTJr0LWMSLq", "reviewer": "Reviewer_vNfB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. This paper applies a Bayesian inference methods to do reinforcement learning and shows effectiveness on Atari level benchmarks.\n2. It is potentially useful in the settings when one need to do deep RL without using gradient based optimization.\n\nWeaknesses:\nIn general, the algorithm seems largely rely on an existing algorithm, TAGI, and the main contribution is extending this to the deep Q learning settings. In this case, I would expect a much better written paper in explaining the algorithm/contribution and stronger empirical results.\n1. The property that it use Bayesian inference (gradient free) to update the NN's parameters seems to be an important contribution. First, it need to be better explained why no gradient itself is a pro. I would take it as just a property of an algorithm, without more advantage of disadvantage. Second it is very hard for me to understand the full update rule of each layer's parameter after reading this paper and its appendix. It might due to my limited knowledge in Bayesian inference, but I think this paper could be more self-contained by adding the full derivation of each layer's neural networks.\n2. Most of the theoretical assumptions are not well explained and justified. For examples, what is the definition of A3 in math; why A5 is reasonable. In particular, one assumption about that different values has the same scale of variance seems very strong, especially in multi-step settings. Often the decisions not only effect the expectation, but also the variance (and distribution) of the return. Many distributional RL, safe RL literature make use of this and shows benefit.\n3. It surprised me that this paper does not include any Bayesian deep RL baselines. Though some of them might not be gradient free like this paper. They shares many similarities such as the Gaussian assumptions.", "summary_of_the_paper": "This paper proposed a new Bayesian Deep RL algorithm that learns deep Q networks (DQN) without using gradient descent. The updates of DQN's parameters follows from the tractable approximate Gaussian inference (TAGI) algorithm. Then it studies the empirical performance of proposed algorithm with DQN learned by gradient based optimization in several Atari benchmarks.", "summary_of_the_review": "In summary, this paper studied how to learn DQN in a fully Bayesian way and is interesting. However too many parts of the paper related to its core contribution is unclear at this time. Unless it's significantly updated then this paper is not ready to published.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fZosh-eEbPe", "reviewer": "Reviewer_H4dH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\nThis paper is well-written and easy to follow. The idea of applying Bayesian inference to obtain uncertainty estimation and sampling is interesting. \n\nWeakness:\n\nMajor concern:\n\nSince the key idea of TAGI-DQN, the TAGI algorithm, is from an existing paper, I'm expecting more complete empirical evaluations and more systematic comparisons with the baseline model (gradient-based algorithm), and the current experiment part may not be enough. \n1. The comparisons should be implemented on more Atari games with more epochs since Atari games are very different for each task and may have different learning patterns in the early stage of training. (as shown in paper [1])\n2. The performances of DQN here is much lower than the ones shown in other paper (e.g., [1,2]), which makes the conclusion less convincing. \n\nIt would be more persuasive if adding TAGI-DQN on top of the codebase that has better performances, and run through more Atari games to conclude that TAGI-DQN can achieve similar performances with its gradient-based counterpart. \n\nBesides, the assumption homoscedastic variance of TAGI-DQN is not very realistic since the variance of $V$ at step $0$ should be very different from that at step $T$ (last step), which requires more strong empirical results to support.  \n\n3. I think the focus of this paper is a little bit unclear to me. In the related work part, it seems the main contribution of TAGI-DQN is to propose an analytically tractable inference approach; whereas in the Discussion part, the main focus is to propose a gradient-free method compared with its backpropagation counterpart. If the first is the case, then I would like to see more comparisons between the existing BDL methods; if the latter is the case, I think the related work part should be modified to reflect more from this aspect. \n\nMinor concern:\n\n1. The definition of A in eq(4) is not the same as the size of action space. \n2. It is not very clear to me what does \"Atari experiment with on-policy RL\" mean, since Q-learning is an off-policy algorithm, what does on-policy indicate here?\n3. I do not see how eq(10) takes the maximization of the expected value as described one line above. \n\n1: Hessel, Matteo, et al. \"Rainbow: Combining improvements in deep reinforcement learning.\" Thirty-second AAAI conference on artificial intelligence. 2018.\n\n2:Dabney, Will, et al. \"Distributional reinforcement learning with quantile regression.\" Thirty-Second AAAI Conference on Artificial Intelligence. 2018.\n", "summary_of_the_paper": "This paper combines the TAGI algorithm with Q-learning algorithm and demonstrates that TAGI-DQN can achieve on-par performance with its backpropagation-based counterpart with fewer hyperparameters. \n", "summary_of_the_review": "In summary, the idea of applying Bayesian inference as an alternative to backpropagation-based algorithm is interesting. However, it requires more complete experiments to demonstrate this, along with the other assumptions that simplify the computing complexity. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors combine TAGI with Q-learning to create an approximate Bayesian Q-learning algorithm. They evaluate their approach and show it has comparable performance to DQN.

All of the reviewers were positive about the potential of the paper. Unfortunately, the paper suffers from lack of clarity, of motivation, and comparison to relevant approaches. All of the reviewers brought up nearly the same concerns and I agree with these concerns. The authors have not address them and the reviewers do not think this paper is ready for publication at this time. I agree and recommend rejection.

Evaluating TAGI-DQN is a valuable contribution, but alone it is not sufficient. The reviewers have made many suggestions on how to improve the paper, and I hope the authors follow up on these suggestions.
Decision: Reject