Analytically Tractable Bayesian Deep Q-Learning
Authors: Luong-Ha Nguyen
James-A. Goulet
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Reinforcement Learning
Bayesian Deep Learning
Tractable Inference
Analytical Inference
Deep Q-learning

Abstract: Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to reach human performance on video game benchmarks using deep Q-learning (DQN). The current consensus for training neural networks on such complex environments is to rely on gradient-based optimization. Although alternative Bayesian deep learning methods exist, most of them still rely on gradient-based optimization, and they typically do not scale on benchmarks such as the Atari game environment. Moreover none of these approaches allow performing the analytical inference for the weights and biases defining the neural network. In this paper, we present how we can adapt the temporal difference Q-learning framework to make it compatible with the tractable approximate Gaussian inference (TAGI), which allows learning the parameters of a neural network using a closed-form analytical method. Throughout the experiments with on- and off-policy reinforcement learning approaches, we demonstrate that TAGI can reach a performance comparable to backpropagation-trained networks while using fewer hyperparameters, and without relying on gradient-based optimization.

34 formulation through the Bellman's Equation. Then, we present how the action-value function can Analytically Tractable Bayesian Deep Q-Learning
Anonymous Author(s) Affiliation
Address
email
Abstract
Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to reach human performance on video game benchmarks using deep Q-learning (DQN). The current consensus for training neural networks such complex environments is to rely on gradient- basedd optimization. Although alterna- live Bayesian deep learning methods exist, most of them still rely on gradient-based optimization, and they typically do not scale on benchmarks such as the Atari game environment. Moreover none of these approaches allow performing the analytical inference for the weights and biases defining the neural network. In this paper, we present how we can adapt the temporal difference Q-learning framework to make it compatible with the tractable approximate Gaussian inference (TAGI), which neural network using closed-form: analytical
2
10
11
allows learning the parameters method. Throughout the experiments with on- and off off-policy reinforcement learn- ing approaches, we demonstrate that TAGI can reach performance comparable nackpropagation-trainee networks while using fewer hyperparameters, and without relying on gradient-ban baseed optimization.
13
14
15
Introduction
Reinforcement learning (RL) has gained increasing interest since the demonstration it was able to
18 reach human performance on video game benchmarks using deep Q-learning (DQN) [17 [26]. Deep RL methods typically require an explicit definition of exploration-exploitatoo function in order to compromise between using the current policy and exploring the potential new actions. Such an issue can be mitigated by opting for Bayesian approach where the selection of the optimal action to follow is based on Thompson sampling (23). Bayesian deep learning methods based on variational
20
22
23
inference [12 10 5 20 29] Monte-Carlo dropout [8], or Hamiltonian Monte-Carbh sampling
24
18] have shown to perform well on regression and classification benchmarks, despite being generally computationally more demanding than their deterministic counterparts. Note that none of these approaches allow performing the analytical inference for the weights and biases defining the neural network. Goulet et al. g recently proposed the tractable approximate Gaussian inference (TAGI) method which allows learning the parameters of a neural network using a closed-form analytical method. For convolutional architectures applied on classification benchmarks, this approach was shown to exceed the performance of other Bayesian and deterministic approaches based on gradient backpropagation, and to do so while requiring smaller number of training epochs 199.
25
26
27
28
29
30
31
32 In this paper, we present how can we adapt the temporal difference Q-learning framework 24 28 to
33 make it compatible with TAGI. Section first reviews the theory behind TAGI and the expected value
35 be learned using TAGI. Section presents the related work associated with Bayesian reinforcement
36 learning, and Section 4 compares the performance of simple TAGI-DON architecture with the one obtained for its backpropagation-triin trained counterpart.
37
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

38
The second wackward update-step consists in performing layer-wise recursive Bayesian inference which goes from hidden-layer hidden-layer and from hidden-layer to the parameters connecting into it. Given the Gaussian approximation for the joint prior throughout the network, the inference can be done analytically while still maintaining a linear computational complexity with respect the number of weight parameters in the network. TAGI allows inferring the diagonal posterior knowledge TAGI-DON Formulation
2
39
This section presents how to adapt the DQN frameworks in order make them compatible with analytical inference. First, Section reviews the fundamental theory behind TAGI, and Section [2.] reviews the concept of long-term expected value through the Bellman's 's equation [25]. Then, Section
40
41
42
2.3
presents how to make the Q-learning formulation
28 compatible with TAGI.
43
2.1
Tractable Approximate Gaussian Inference
TAGI [9] relies
main steps; forward uncertainty propagation and backward update. The first forward uncertainty propagation step is intended to build the joint prior between the neural network parameters and the hidden states. This operation is made by propagating the uncertainty from the model parameters and the input layer through the neural network. TAGI relies on the Gaussian assumption for the prior of parameters as well for the variables in the input layer. In order to maintain the analytical tractability of the forward step, we rely on the Gaussian multiplicative approximation (GMA) which consists in approximating the product of two Gaussians by Gaussian random variable whose moments match those calculated exactly using moment generating functions. In order to propagate uncertainty through mon-linear activation functions, a second approximation made by locally linearizing these function at the expected value of the hidden unit being activated. Although this linearization procedure may seems to be crude approximation, it has been shown to match or exceeds the state-of-the-ar performance on fully-connected neural networks (FNN) (9]
45
47
48
49
50
51
52
53
54
55
56 as well as convolutional neural networks (CNN) and generative adversarial networks 19. TAGI succeeds in maintaining linear computational complexity for the forward steps, (1) by assuming 58 a diagonal covariance for all parameters in the network and for all the hidden units within a same layer, and (2) by adopting layer-wise approach where the joint prior is only computed and stored for the hidden units on pairs successive hidden layers, as well as the hidden units within a layer and the parameters connecting into it. This layer-wiss approach is allowed by the inherent conditional independence that is built-in feed-forwan neural network architectures.
60
82
63
64
65
66
68 for weights and bias parameters, either using one observation at a time, or using mini-batches of data. As we will show in the next sections, this online learning capacity is best suited for RL problems where we experience episodes sequentially and where we need to define a tradeoff between exploration and exploitation, as a function of our knowledge of the expected value associated with being in state and taking an action.
70
72
73
2.2
Expected Value and Bellman's Equation
74
We define r(s, a')) as the reward for being in state s E RS, taking an action A {a), {a,.a2... a.......... .... and ending in state s' E RR. For simplicity, we use the short-form notation
75
76
for the reward r(s, a, s') = r(s) in order to define the value as the infinite sum of discounted rewards
77
(ss)
Σ *((((+))
k=0
78 As we do not know what will be the future states 8 for k: > 0, we need to consider them as random
79 variables (S++&), so that the value V(s;) becomes random variable as well,
v(s)) = r(tt) +
x ((((+)))
(2)
1=1
80 Rational decisions regarding which action take among the set is based the maximization of the
81 expected value as defined by the action-value function
oc
(1t 01) = pv EVV(s at, T) = r(st) +E
Σ
2 *r
(3)

---

82 where it is assumed that at each time t, the agent takes the action defined in the policy я. In the case 83 of spisode-based learning where the agent interacts with the environment, we assume we know the
84 tuple of states and 8,41, so that we can redefine the value as
V(s,, 0z) = r(s)) +t (((++1) +
r Ar(S41144)
(4)
= r(s)) + 7V (8441, 0+11)
85 Assuming that the value V ~ (N o in Equations 2 and 4 described by Gaussian random
86 variables, we can reparameterize these equations as the sum of the expected value q(s, a) and a zero-mean Gaussian random variable & ~ (Ne 0, 1), so that
87
V(s,a) g(s a) avv,
(5)
where the variance of and E are assumed here to be independent of and a. Although in more 89 general framework this assumption could be relaxed, such an heteroscedastic variance term is outside from the scope of this paper. Using this reparameterization, difference between the expected values of two subsequent states
write Equation ( as the discounted
81
q(s,, az)
r(s:) + (7(( ((+1) ov,&, + +0v0 E++1 r(s:) + (1(1 a(+1) + ovE.
(6)
92 Note that in Equation 6 dv, and vo can be combined in a single standard deviation parameters 93 Vv with the assumption that & 1 E,, Vi j..
94 In the case where a time I, we want to update the Q-values encoded in the neural net only after observing n-step returns (15),
reformulate the observation equation so that
1-1-1
((s 04)
Σ
(r((s)))) + " (g(sn, an) + & Vt = (1,2, ,n , 1}.
F=0
96
Note that in the application of Equation 7 we employ the simplifying assumption that & IL E44s. Vi + as Equation 6 already makes simplifying assumptions for the independence of ot and E. Note that in a more general framework, this assumption could be relaxed. An example of n-step returns is presented in the the algorithm displayed in $1 from the supplementary material.
97
98
99
100
The following subsections will present, for the case of categorical actions, how to model the deter- ministic action-value function g(s, a) using a neural network.
102
2.3
TAGI Deep Q-learning for Categorical Actions
Suppose we represent the environment's state at time & and t by {s. s'), and the expected value for each of the A possible actions E AA by the vector Q E IR.,. In that context, the role of the neural network is to model the relationships between (s, a} and q. Figure [a presents a directed acyclic graph (DAG) describing the interconnectivity in such neural network, where red nodes denote state variables, green nodes are vectors of hidden units z, the blue box compact representation for the structure of a convolutional neural network, and where gray arrows represent the weights and bias connecting the different hidden layers. Note that unlike other gray arrows, the red ones in (b) are not directed arcs representing dependencies, but they simply outline the flow of information that takes place during the inference step. For simplification purposes, the convolutional operations omitted and all regrouped under the CNN box [19. In order to learn the parameters 0 of such network, we need to expand the graph from Figure Th to include the reward r. the error term y 114 and q', the v-values of the time step L 1. This configuration is presented in Figure Ip where the
103
105
106
107
109
111
113
115 nodes that have been doubled represent the states & and s' which are both evaluated in a network
116 sharing the same parameters. When applying Equation g-values corresponding to specific action
can be selected using a vector h E (0. l having a single non-zero value for the i-th component identifying which action was taken a time that
117
118
4: = lg]. h.q.
(8)
119 During the network's training, analogously to Thompson sampling 233, the vector h E (0, 1)^A is defined such that the i-th non-zero value corresponds to the index of the largest value among q',
3

---

8 g
CNN
1
2
q
(a) Neural network DAG for modelling the action-value function
O
((ru)
CNN
0
1
0
!2
OYE
g(a)
q'
(b) DAG for the temporal-differer Q-learning configuration
Figure I: Graphical representation of neural network structure for temporal temporal-difference Q-learning with categorical actions. The red nodes denote state variables, green nodes are vectors of hidden units 2, and the blue box compact representation for the structure of convolutional neural network. The gray arrows represent the weights and bias f connecting the different hidden layers and the red arrows outline the flow of information that takes place during the inference step.
vector of realizations from the neural network's posterior predictive output Q ~ (q': pQ EQrd). Because of the Gaussian assumptions in TAGI, this posterior predictive is readily available from the forward uncertainty propagation step, as outlined in $2.1
121
123
124
The red arrows in Figure D outline the flow of information during the inference procedure. The first step consists in inferring q using the relationships defined in either Equation 6 7 As this is a linear equation involving Gaussian random variables, the inference is analytically tractable. From there, one order to learn the weights and biases in O. With the exclusion of the standard hyperparameters related to network architecture, batch size, buffer size or the discount factor, this TAGI-DQN framework only involves a single hyperparameter, dv, the standard deviation for the value function. Note that when using CNNs
125
127
can follow the same layer-wise recursive procedure proposed by Goulet al.
128
129
131
with TAGI, Nguyen and Goulet [19] recommended using a decay function for the standard deviation of the observation noise that after seing batches of n-steps,
of max(oV") min , ov)
The model in Equation 9 has three hyperparameters, the minimal noise parameter opin the decay factor 77 and the initial noise parameter σy. As was shown by Nguyen and Goulet [19] for CNNs and how show in for RL problems, TAGI's performance is robust towards the selection of these hyperparameters.
134
136
137
A comparison of implementation between TAGI and backpropagation on deep Q-network with experience replay [17 is shown in Figure P A practical implementation of n-step TAGI deep learning is presented in Algorithm from the supplementary material.
138
3
Related Works
141
Over the last decades, several approximate methods have been proposed in order to allow for Bayesian neural networks [18] [2 10
29, g with various degree of approximations. Although some these methods have shown to be capable of tackling classification tasks on datasets such ImageNet few of them have been applied on large-scale RI. benchmark problems. The key idea behind using Bayesian methods for reinforeement learning is to consider the uncertainty associated with fuuctiis in order to identify tradeoff between exploring the performance of possible actions and exploiting the current optimal policy (25]. This typically takes the form of performing Thompson sampling
5 [4
20
143
20
145
147
23 rather than relying heuristics such as e-greedy.
149
For instance, MC dropout was introduced has method intrinsically suited for reinforcement learning. Nevertheless, five years after its inception, the approach has not yet been reliably scaled to more advanced benchmarks such as the Atari game environment. The same applies Bayes-
150
151
by-backprop
[ which was recently applied to simple RL problems [R], and which has not yet been applied to more challenging environments requiring convolutional networks. On the other hand, Bayesian neural networks relying sampling methods such as Hamiltonian Monte-Carlo
153

---

Algorithm TAGI-DQN Experience Replay
Algorithm 2: DQN with Experience Replay
Initialize replay memory R. capacity N; Evs
Initialize replay memory R to capacity N
2 Initialize parameters
Initialize parameters 0:
3 Discount factor 7:
Discount factor
for episode 1:E.e do
Define (epsillon-gree) greelly function);
Reset environment 80i
for cpisode 1: do
for 1:
Reset environment for = 1 do a U 24(0, 1):
((5t. Q(ss. (11 (g
(s.
arg
8r+1. cnviroment(e)):
randi(A)
1 <
10
Store {$t: at "t, 9(41) R:
0g
arg max Qo( n) a
aEAA
11
Sample random batch of 93:
8141, enviroment(as))
12
((5+1.0a)) (())+1.a))= N( (3)=1.0"), E (8)41.
Store (9, in R:
o3+ aug (8)+)
Sample random batch of (5
deA
14
m3 ry + t (#541, (534),0)41)
Vj Ty max Qo(#)-1.0);;
a'Eal
15
m 2 (s)+1: Ev;
14
Update @ using gradient descent on
Update using TAGE on PDF(0ly)
(,5 Qo(ss,a))]:
Figure 2: Comparison of TAGI with backpropagation deep Q-network with experience replay. PDF: probability density function; L: loss function; U: uniform distribution; randi: uniformly distributed pseudorandom integers.
155
18
are typically computationally demanding to be scaled to RL problems involving such a complex
156
environment.
157
Although mainstream methods related to Bayesian neural networks have seldom been applied to complex RL problems, several research teams have worked alternative approaches in order to allow performing Thompson sampling. For instance, Azizzadenesheli et al. [4] have employed a deep Qenetwork where the output layer relies Bayesian linear regression. This approach
158
159
shown to be outperforming its deterministic counterparts on Atari games. Another approach by Osband et al. (2I) employs bootstrapped deep Q-networks with multiple network heads in order to represent the uncertainty in the Q-functions. This approach was also shown to scale to Atari games while presenting an improved performance in comparison with deterministic deep Q-networks. Finally, Wang and Zhou 27 have tackled the same problem, but this time by modelling the variability in the Q- functions through latent space learned using variational inference. Despite its good performance on the benchmarks tested, it did not allowed to be scaled to the Atari game environment.
161
162
163
164
185
186
167
The TAGI deep Q-network presented in th is paper is the first demonstration that analytically tractable inference approach for Bayesian neural networks can be scaled to a problem as challenging the Atari game environment.
189
170
Benchmarks
171
This section compares the performance of TAGI with backpropagation-bast standard implementa- tions on off- and on- on-policy deep RL. For the off- policy RL, both TAGI-based and backpropagation- based RL approaches are applied to deep Q-learning with experience replay (see Algorithm & 2) for the lunar lander and cart pole environments. For the on-policy RL, TAGI is applied to the n-step counterpart [15]. We perform the comparison for five Atari games including Beamrider, Breakout, Pong, Qbert, and Space Invaders. Note that these five games are commonly selected for tuning hyperparameters for the entire Atari
172
173
174
175
176
Q-learning algorithm and is compared with its backpropagation-basee
178
games [1] [6]. All benchmark environments are taken from the OpenAl Gym (6).
5

---

180
4.1
Experimental Setup
181
In the first experiments with off-po RL, we use a fully- cally-ooneected multilayer perceptron (MLP) with two hidden layers of 256 units for the lunar lander environment, and with one hidden layer of 64 units for the cart pole environment. In these experiments, there is no need for input processing nor for reward normalization. Note that unlike for the deterministic Q-network, TAGI does not use target Q-network for ensuring the stability during training and allows climinating the hyperparameter related the target update frequency. For the deep Q-network trained with backpropagation, employ the pre-tuned implementation of OpenAl baselines 7 with all hyperparameters set to the default values.
185
185
187
189
For the Atari experiments with on on-policy RL, we use the same input processing and model architecture as Mnih et al. [15]. The Q-network uses two convolutional layers (16-32) and a full-connected MLP of 256 units. TAGI re-teep Q-learning only uses single network to represent the value function for each action, and relies on single learning agent. The reason behind this choice is that TAGI current main library only available on Matlab which does not support running a Python multiprocessing module such the OpenAI gym. In the context of TAGI, we use horizon of 128 steps and as recommended by Andrychowicz et al. [3] and following practical implementation details [ 2], each return in n-step Q-learning algorithm is normalized by subtracting the average return from the current n-steps and then dividing by the empirical standard deviation from the set of n returns.
190
191
193
195
197
The standard deviation for the value function, (σv), is initialized at 2. σv is decayed each 128 steps with a factor 1) = 0.9999. The minimal standard deviation for the value function min = 0.3. These hyperparameters values were not gric-hcarehed but simply adapted to the scale of the problems and are kept constant for all experiments. The complete details of the network architecture and hyperparameters provided in the supplementary material.
199
201
203
4.2
Results
204
For the first set of experiments using off-policy RL, Figure presents the average reward over 100 episodes for three runs for the lunar lander and cart pole environment. The TAGI-based deep Q-learning with experience replay shows faster and more stable learning than the one relying on backpropagation, while not requiring a target network.
205
300
200
Average reward
Average reward
150
150
TAGI
100
Backproangation
-150
-300
0.5
Numher of steps (M)
Number of steps (M)
(a) LundrLander-v2
(b) CartPole-vO
Figure 3: Illustaation of average rewards over 100 episodes of three runs for one million time steps for the TAGI-based and backpropagation-basse deep Q-learning.
207
Table shows that the average reward over the last 100 episodes obtained using TAGI greater than the one obtained using backpropagation.
Table I:
Average reward the last 100 episodes for the lunar lander and cart pole experiments. TAGI: Tractable Approximate Gaussian Inference.
Method
Lunar lander
Cart pole
TAGI
277.6 6.3
199.2 + 1.3
Backpropagation
166.7 + 103.6
130.3 = 16.9

---

209
Beam Rider Figure 4 compares the average reward over 100 episodes for three runs obtained for TAGI, with 210
the results from Mnih et al. [15] for the second set of experiments on Atari games. Note that all results presented were obtained for a single agent, and that the results for the backpropagation-train networks are only reported at the end of each epoch.
212
3,400
21
TAGI
Averago roward
Average reward
Average reward
2,550
Backpropegation
30
10
1,700
20-
850
10
-10
0
-21
10
Number epochs
Number of epochs
Number of epochs
(a) Beam Rider
(b) Breakout
(c) Pong
4,200
550
Average rewardd
Average reward
3,150
440-
2,100
330
1,050
220
To
110
10
Number epochs
Number of epochs
(d) Qbert
(e) Space Invaders
Figure 4: Illustration of average reward over 100 episodes of three runs for five Atari games. The number of epochs is used here for the comparison of TAGI and backpropagatton-trainee counterpart obtained by Mnih et al. 15] Each epoch corresponds to four million frames. The environment identity are (Atari Game NoFrameSkip--u
214 Results show that TAGI outperforms the results from the original n-step Q-learning algorithm trained with backpropagation [15] Breakout, Pong. and Qbert. while underperforming
216
and Space Invaders. The average training time of TAGI for an Atari game is approximately 13 hours GPU calculations benchmarked
4-core-intel desktop of 32 GB of RAM with NVIDIA GTX 1080 Ti GPU. The training speed of TAGI for the experiment of the off-policy deep RL is approximately three times slower on CPU calculations than the backpropagation-tran -rriindd counterpart.
218
219
220
The reason behind this slower training time is because of its intrinsically different inference engine, so that TAGI's implementation is not compatible with existing libraries such TensorFlow or Pytorch.
222
TAGI's library development is still ongoing and it is not yet fully optimized for computational efficiency. Overall, these results for on- and off policy RL approaches confirm that TAGI can be
223
224
applied to large scale problems such as deep Q-learning.
Discussion
225
Although the performance of TAGI does not systematically outperform its backpropagation- based counterpart, it requires fewer hyperparameters (see $3 in supplementary material). This advantage 228 is one of the key aspects for improving the generalization and reducing the computational cost of the hyperparameter tuning process which are the key challenges current state of deep RL [II]. For instance, in this paper, the TAGI's hyperparameters relating to the standard deviation of value function (av) are kept constant across all experiments. Moreover, since these hyperparameters were not subject to grid-search in order to optimize the performance, the results obtained here are representative of what user should obtain by simply adapting the hyperparameters fit the specificities and scale of the environment hand.
228
227
229
231
233

---

235
More advanced RL approaches such as advanced actor critic (A2C) [5] and proximal policy opti- mization (PPO) [22] employ two-networks architectures in which one network is used to approximate value function and other is employed to encode the policy. The current TAGI-RI. framework is not yet able handle such architectures because training policy network involves an optimization problem for the selection of the optimal action. Backpropagatiinnbbaal bbaee approach currently rely on
236
238
239
240
gradient optimization to perform this task, while TAGI will require developing alternative approaches
in order to maintain the analytical tractability without relying on gradient-based optimization.
6
Conclusion
242
243
This paper presents how to adapt TAGI to deep Q-learning: Throughout the experiments, we demon- strated that TAGI could reach a performance comparable to backpropagation-tee trained networks while using fewer hyperparameters. These results challenge the common belief that for large scale problems such as the Atari environment, neural networks can only be trained by relying on gradient backpropa- gation. We have shown here that this current paradigm is no longer the only alternative as TAGI has linear computational complexity and can be used to learn the parameters complex networks in an analytically tractable manner, without relying gradient-basee optimization.
244
245
246
248
References
Pytorch examples for reinforce algorithm. http:////ghhhh...ooooooooooaoaaa www.rrreeenneeeeeeennereereeeeee
252
py 2019.
253
[2] Pytorch examples for actor crtic algorithm. http:/hhmmmtttttttttteheeeeeeete
httppp//ghtt.bb.c.ooooorrrrrree/ reinforcement, reinnforeeantt_eariingnn actor cwwiiiiiipppp.. attorrcrrtiiccppppppppcac. py 2020.
254
255
[3]
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussemot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference Learning Representations, 2021.
256
K. Azizzadenesheli, E. Brunskill, and A. Anandkumar. Efficient exploration through Bayesian deep g-networks. In IEEE Information Theory and Applications Workshop, pages 1-9,2018 2018.
259
260
[5] C. Blundell, J. Cornebise, K. Kavukcuoglu, and Wierstra. Weight uncertainty in neural networks. arXiv preprint w 2015.
261
282
[6] G. Brockman, V. Cheung. L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.0140 2016.
283
284
[7] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov. Openai baselines. attps://githhe com/ http///eaaaaaaeeeeeeeeeeeee 2017.
[8] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In ICML proceeddcs,,,gg,, pages 1050-1059, 2016.
268
J-A. Goulet, L.H. Nguyen, and S. Amiri. Tractable approximate Gaussian inference for Bayesian neural networks. arXiv preprint, 2020.
269
270
[10] M. Hernández-Looenn and Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning, pages 1861-1869, 2015. 2015.
271
272
[1]] A. Irpan. Deep reinforcement learning doesn't 't work yet. http:://www.aaerirpaa.co/o//1/1
273
rl hard. html 2018
274
[12] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 28, 2015.
278
277
[13] Z. Lipton, X. Li, J. Gao, L. Li. F. Ahmed, and L. Deng. Bbq-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
278
279
280
[14] C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In ICML proceedings, pages 1708- 17181 2016.
8

---

282
[15] V. Maih, Adria P. Badia, M. Mirza, A. Graves. T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML proceedings, pages 1928-1937. PMLR. 2016.
284
[16] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602, December 2013.
286
287
[17] V. Mnih, Kavukcuoglu, Silver, A.A. Rusu, J. Veness, M.G. Bellemare, Graves, M. Riedmiller, A.K. Fidjeland, and G Ostrovski. Human-level control through deep reinforcement learning. nature, 51 5187540):529-535 523-333,,,,3, 2015.
288
289
290
[18] R. M. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
291
[19]
L. H. Nguyen and J-A. Goulet. Analytically tractable inference in deep neural networks. arXiv preprint, 2021.
292
293
[20] K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen. R. E. Turner, R. Yokota, and M. E. Khan. Practical deep
294
learning with Bayesian principles. In Advances in Neural Information Processing Systems proceedings. 2019.
296
[21] I. Osband, C. Blundell, A. Pritzel, and Benjamin V. Roy. Deep exploration via bootstrapped dqn. In NEURIPS proceedings, pages 4033-4041, 2016.
[22] J. Schulman, F. Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.
289
arXiv preprint 1707,07777.0647 2017.
[23]
M. Strens. A Bayesian framework for reinforcement learning. In ICML ICML, proceedings, pages 943-950, 2000.
[24] R. S. Sutton. Learning predict by the methods of temporal differences. Machine learning, 3(1):9-44,
302
1988.
[25] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 2nd edition, 2018.
[26] H. Van Hasselt. A. Guez, and D. Silver. Deep reinforcement learning with double q-leaminng In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
305
[27] Z. Wang and M. Zhou. Thompson sampling via local uncertainty. In ICML proceedings, volume 119, pages 10115-10125, 13-18 Jul 2020.
307
308
[28] C. J. Watkins and P. Dayan. Q-learinng Machine learning, 8(3-4):279 292, 1992.
[29] A. Wu, S. Nowozin, F. Meeds, R. F. Turner, M. Hernández-Looato, and L.. Gaunt. Deterministic variational inference for robust Bayesian neural networks. In ICLR proceedings, 2019.
310
Checklist
312
1. For all authors...
313
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
314
315
(b) Did you describe the limitations of your work? [Yes]
316
(c) Did you discuss any potential negative societal impacts your work? [N/A]
317
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
318
2. If you are including theoretical results...
319
(a) Did you state full set of assumptions of all theoretical results? [N/A]
320
(b) Did you include complete proofs all theoretical results? [N/A]
321
3. If you ran experiments..
322
(a) Did you include code, data, and instructions needed to reproduce main experimental results (either in the supplemental material or as a URL)? [No] The code will be made available upon the publication of the paper
323
324
325
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)?
328
[Yes]
327
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]

---

(d) Did include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes]
331
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
334
(c) Did you include any new assets either in the supplemental material or as URL? [No]
335
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
ing/curating? [N/A]
337
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
339
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
343
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
10


Reviews:
[{"review_id": "f2k2TuhpBSh", "reviewer": "Reviewer_JJGA", "summary": "This paper proposes applying TAGI (Goulet et al, 2019), a framework for approximate Bayesian inference in deep neural networks, to deep q-learning. Using assumptions of Gaussianity on various distributions, TAGI infers both the neural network parameters and unit activations, without needing gradient descent. Because all distributions are assumed to be Gaussian, the means and variances can be updated analytically with linear computational complexity. TAGI is essentially used to infer the correct Q-value function, and the approach is applied to lunar lander, cartpole, and several Atari games.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I liked the idea of introducing Bayesian approaches to deep RL is an interesting, particularly as it offers a more principled way of handling the exploration-exploitation tradeoff compared to epsilon-greedy exploration. However, I do have several reservations on both the theoretical and empirical aspects on this work, and for the reasons given below, I cannot recommend this paper for acceptance as this stage.\n\nStrengths:\n* introducing Bayesian inference into deep RL allows for the possibility for a more principled approach to exploration-exploitation via Thompson sampling; exploration is driven by greater posterior uncertainty, while a lower uncertainty will drive greater exploitation\n* compared to alternative approaches to Bayesian deep learning, TAGI's inference can be done analytically with linear computational complexity with respect to the number of network parameters\n\nTheoretical weaknesses:\n* the authors have chosen not to not use a target network for TAGI-DQN. This is rather controversial as it changes the training target. DQN updates the Q-network towards a fixed TD target. For the learning target to be fixed, it needs a fixed bootstrap, which therefore requires a fixed target network. By not using a fixed target network, TAGI-DQN attempts to infer the network parameters that minimize a different learning objective, what Sutton & Barto (2ed, Ch 11.5) call the mean squared TD error, and this is known to converge to an incorrect result.\n\nEmpirical weaknesses:\n* the Atari results were ran for 40M frames (only a fifth of the 200M that is more standard). My concern is that this is too early to tell what the asymptotic performance of these algorithms will be like. For instance, it'd be nice to know whether any of TAGI's early leads over the backprop algorithm will be stable over longer horizons. It'd also be nice to know whether TAGI training eventually converges to some asymptotic performance or whether instabilities will emerge over longer training horizons.\n* I think this work could also benefit from some ablation studies. For instance, the authors chose not to include a target network for TAGI: as this is a striking design decision, it'd be helpful to see what the effects of it is via an ablation study. Presumably, TAGI explored by Thompson sampling (it was not made explicit in the paper), and it'd be interesting to see how this compares to a more traditional epsilon-greedy policy.\n* Finally, it'd be helpful to also examine TAGI in a policy evaluation setting to see whether it can converge onto a correct, known policy. \n\n---Review update---\n\nI thank the authors for their response to my questions during the discussion period. In light of their comments, my previous theoretical concerns have been addressed, and I have raised my score to 5 to reflect that. But I still feel that the paper's baselines were rather thin, and multiple suggestions for improvement have been offered by myself and the other reviewers. I'd like to mention the atari baselines in particular. The authors' current choice is unusual and never used beyond the original paper. The authors compare against n-step Q-learning of Mnih et al (2016), but the 1-thread version of it. This baseline is never used simply because it performs poorly -- it has no replay buffer or parallel actors to make up for it. I appreciate the authors' intention was to demonstrate an improvement in the online learning setting. However, I think it'd still be helpful to compare against a more widely used atari baseline as well, such as DQN and/or the multi-actor version of n-step Q-learning. I also believe it'd be helpful if the authors could provide results on more games, as the current sample size is small -- the authors show TAGI is better in only 3 out of the 5 games they ran. Based on this, it is difficult to conclude whether the RL practitioner should expect TAGI to do better than gradient descent if she were considering training an agent on a new task.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A discussion of societal impact was not relevant for this work, as it was more foundational in nature and would not introduce additional impacts beyond that inherent generically in machine learning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "5374_d6VtoN", "reviewer": "Reviewer_5v2P", "summary": "This paper adapts a recently proposed analytical closed-form Bayesian approach to optimizing Neural Networks to Q-Learning. The proposed TAGI-DQN architecture is optimized without gradient descent. RL policies based on TAGI-DQN are shown to perform well on a collection of OpenAI Gym environments, including a small number of Atari games. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **High-level evaluation**\nI am excited about the potential advances to RL in complex environments that may be ushered in by the possible contributions put forward in this paper. I however do not feel that the work, as currently presented, is wholly ready for publication and perhaps another round of revisions (including the addition of more relevant baselines -- see \"Relevance\" section below) would do this paper a great service. While reading, I got the sense that the authors didn't want to distract from their intended contributions by dwelling on how TAGI works. This is always a tricky balance in writing papers (how much background is too much?) but I did feel that there were too significant of conceptual+technical gaps introduced in the paper by keeping the discussion too high-level and informal through Section 2.  \n\n#### **Originality**\nThe adaptation of the TAGI framework to RL domains is very intriguing. Analytical solutions to a Bayesian treatment of NNs could greatly improve challenges of sample efficiency within RL. This advantage is demonstrated in the experimental analysis across a variety of domains. \n\n#### **Quality and Clarity**\nThe major weaknesses of this paper concern the clarity of the formulation and utlization of the TAGI with DQNs. In particular, I found the discussion in Section 2.1 to be far too high-level for introducing a relatively complex new approach. A phrase that ran through my head while reading was “show don’t just tell”. The paper is quite light on technical details and it wasn’t due to space constraints as there was nearly 2 pages of unused space remaining in the submitted version of the paper. I feel that a more formal approach to how the two steps of TAGI interact with individual parameter distributions and their connections through hidden layers is necessary and would greatly improve the paper. This would make the discussion in Sections 2.2-2.3 much easier to follow and understand. This is especially true in the discussion starting on line 124 when inference through the TAGI-DQN is described.\n\nParticularly egregious in this lack of clarity is the unmotivated and sudden jump at the end of Section 2.1 to RL. It’s not apparent how the discussion around optimizing parameters via TAGI with a single example or a mini-batch directly relates to learning sequentially with episodes of observations and so on. Additionally, without any formal framing, there’s an extensive burden placed on the reader to know exactly what parts of the following sections correspond to the steps within TAGI. With the connection seemingly drawn to the expected value of selecting an action $a_t$ when conditioned on some state $s_t$, it is also somewhat surprising that the family of Distributional RL (Bellemare, et al; 2017 ICML) approaches went uncited. I acknowledge that Dist. RL does not attempt to pass off as a Bayesian approach but the core component of these approaches is in the implicit distribution formed around the expected value of rewards conditioned on action selection. \n\nIn line 100, was the word “discrete” intended in the place of “categorical”? \n\nIt’s not clear why n-step returns are preferred? Typically this introduces significant variance in DQN based approaches. Is there something particular about TAGI that necessitates the use of multi-step returns?\n\nThe elimination of hyperparameters by moving to TAGI-DQN, described on line 130 seems to only consider those associated with SGD optimization (learning rate, optimization hyperparams, minibatch size, etc…). While this may require several components to tune, the architecture construction and size of replay buffer can have an outsized impact as well. While it’s nice to potentially not worry about hyperparameters associated with SGD, there are still significant choices made in the design of a model.\n\nWhat’s the rationale for decaying the standard deviation $\\sigma_{V}$? Is it to account for drifting biases as the parameters of the model are learned? Is this a “hack” that assumes improved model confidence over time? Is this perhaps too optimistic? What happens if an observed state in a later episode falls out of the assumed distribution? Is TAGI-DQN robust to outliers? These are important questions with regards to avoiding the Q-function approximation overfitting too early. \n\nAgain, it’s a shame that so little technical content is provided in Section 2. The primary contribution of this paper is the extension of TAGI to RL domains and architectures. Without the details of how TAGI actually works, line 16 in Algorithm 1 becomes an intellectual dead-end within this paper as written.\n\nIn the OpenAI Gym experiments, the initialization of $\\sigma_V$ is not mentioned. Additionally, it is not mentioned whether the comparison with the OpenAI default DQN is especially fair or not. For instance, what is the architecture of the DQN model? Is it the same as the TAGI network? Does TAGI use the same amount of computation or more in it’s optimization steps? Are the batch sizes consistent between the two implementations?\n\nFor the experiments on Atari, it’s unclear whether n is set to 128 or whether that horizon is for the decay of the standard deviation. While it’s impressive that TAGI outperforms DQN on several of these Atari games, the discussion or analysis about why it fails on some is missing. It would be nice to be able to develop some insight into what types of domains TAGI may not be appropriate for. Are there some complexities in the state spaces of Beam Rider or Space Invaders that make it harder for the solution approach? Do the representations learned by the CNN encounter (or otherwise get stuck in) local minima? I find it interesting that in both of these games, there is a point after which the baseline DQN solution diverges from the TAGI-DQN solution in performance, indicating that it’s learned something that wasn’t accessible to the TAGI network. \n\n\n#### **Significance**\nIf properly evaluated the contributions promised through a Bayesian method optimized via analytical means are potentially of great significance. However, I found the technical discussion and experimental demonstration to be mildly unconvincing (partially due to a lack of clarity as discussed above). Clearly, the performance of the TAGI-DQN is demonstrated to be better than a standard DQN model. However, the lack of comparison to comparable approximate Bayesian RL methods or appropriate ablations is disappointing. Without comparison to relevant baselines, it’s hard to feel confident about the proposed analytical solution from a practical standpoint. It’s possibly assumed that TAGI-DQN is automatically more efficient/effective due to the closed-form solution. I say this as a means to rigorously understand the specific advantages borne by the TAGI framework applied to a Bayesian form of the DQN.\n\nIn line 144, a “few” BNN approaches that have been applied to large RL domains are alluded to. It would be preferable if these papers are explicitly cited and discussed so as to provide greater context into how TAGI differs (aside from only being an analytical closed-form solution method, if there is anything else?). Is the way the network parameter distributions are factorized to make inference tractable in a similar way to any of these BNN approaches, what about variational inference types of solution methods (such as VariBad--Zintgraf, et al; 2020 ICLR)? How similar is that to TAGI? Answers to these types of questions that would greatly improve the framing of the results that are shown in the following sections of the paper. \n\n\n##### **Additional references**\nBellemare, Marc G., Will Dabney, and Rémi Munos. \"A distributional perspective on reinforcement learning.\" International Conference on Machine Learning. PMLR, 2017.\n\nZintgraf, L., et al. \"VariBAD: a very good method for Bayes-adaptive deep RL via meta-learning.\" Proceedings of ICLR 2020 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did an adequate job discussing the computational limitations of their approach as well as the types of contemporary RL methods the TAGI optimization approach could be currently applied to. There are however remaining questions regarding some of the technical limitations of the TAGI framework in light of overfitting, or getting stuck in local optima as the solution method still relies on collecting data proactively while the policy is trained.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "14"}, "extra_text": {}}, {"review_id": "1MMToB8f1Ah", "reviewer": "Reviewer_tBCT", "summary": "The paper proposes to use TAGI, an approach for training BNNs, within DQN to enable Thompson sampling.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has very limited novelty,  simply plugging TAGI (a previously proposed approach for training BNNs) into DQN. The results are also not particularly impressive, being run on a very limited set of environments and being outperformed by vanilla DQN on several. While having fewer hyperparameters to tune than training the network via gradient descent is a nice benefit, a more thorough evaluation would be needed to demonstrate the advantages of TAGI in reinforcement learning.\n\nWhy not compare to Thompson Sampling with Bayesian approaches to Q learning like in Azizzadenesheli or Osband et al? This seems like the more direct comparison than vanilla DQN.\n\nThe authors also describe their learning algorithm as being analytic, but still relies on repeated iterations, so it's unclear to me what the benefit of the \"analytic\" updates are. If the key benefit here is just the uncertainty is analytically computed when evaluating the network on new inputs, then the authors should compare the evaluation times of TAGI vs other approximate BNN methods that require multiple forward passes.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have raised valid concerns that were not adequately addressed by the rebuttal, placing the paper below the bar.
Decision: Reject