Title: Iterated Deep Q-Network: Efficient Learning of Bellman Iterations for Deep Reinforcement Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: deep reinforcement learning

bellman operator

approximate value iteration

atari games
Abstract: Value-based reinforcement learning~(RL) methods strive to obtain accurate approximations of optimal action-value functions. Notoriously, these methods heavily rely on the application of the optimal Bellman operator, which needs to be approximated from samples.Most approaches consider only a single Bellman iteration, which limits their power. In this paper, we introduce iterated Deep Q-Network (iDQN), a new DQN-based algorithm that incorporates several consecutive Bellman iterations into the training loss. iDQN leverages the online network of DQN to build a target for a second online network, which in turn serves as a target for a third online network, etc., thereby taking into account future Bellman iterations. While using the same number of gradient steps, iDQN allows for better learning of the Bellman iterations compared to DQN. We evaluate iDQN against relevant baselines on 54 Atari 2600 games to showcase its benefit in terms of approximation error and performance. iDQN greatly outperforms its closest baselines, DQN and Random Ensemble Mixture, while being orthogonal to more advanced DQN-based approaches.

1

Abstract

Value-based reinforcement learning (RL) methods strive to obtain accurate approximations of optimal action-value functions. Notoriously, these methods heavily rely on the application of the optimal Bellman operator, which needs to be approximated from samples. Most approaches consider only a single Bellman iteration, which limits their power. In this paper, we introduce Iterated Deep Q-Network (iDQN), a new DQN-based algorithm that incorporates several consecutive Bellman iterations into the training loss. iDQN leverages the online network of DQN to build a target for a second online network, which in turn serves as a target for a third online network, and so forth, thereby taking into account future Bellman iterations. While using the same number of gradient steps, iDQN allows for better learning of the Bellman iterations compared to DQN. We evaluate iDQN against relevant baselines on 54 Atari 2600 games to showcase its benefit in terms of approximation error and performance. iDQN outperforms its closest baselines, DQN and Random Ensemble Mixture, while being orthogonal to more advanced DQN-based approaches.

1 Introduction

Deep value-based Reinforcement Learning algorithms have achieved remarkable success in various 16

fields, from nuclear physics (Degrave et al., 2022) to construction assembly tasks (Funk et al., 2022). 17

These algorithms aim at learning a function as close as possible to the optimal action-value function, 18

19

20

on which they can build a policy to solve the task at hand.

To obtain an accurate estimate of the optimal action-value function, the optimal Bellman operator is used to guide the learning procedure in

the space of Q -functions (Bertsekas, 2019) through successive iterations, starting from any Q -function 21

to the optimal action-value function. In Reinforcement Learning, as opposed to Dynamic Programing, 22

the reward function and system dynamics are not assumed to be known (Bertsekas, 2015). This forces 23

us to approximate the optimal Bellman operator with an empirical Bellman operator. This problem 24

has received a lot of attention from the community (Fellows et al. (2021), Van Hasselt et al. (2016)). 25

On top of that, the use of function approximation results in the necessity of learning the projection of 26

the empirical Bellman operator's iteration on the space of approximators. In this work, we focus on 27

the projection step. 28

We propose a way to improve the accuracy of the learned projection by increasing the number of 29

gradient steps and samples that each Q -function estimate has been trained on. This idea, implemented 30

in the training loss function, uses the same total number of gradient steps and samples than the 31

classical approaches. At a given timestep of the learning process, this new loss is composed of the 32

consecutive temporal differences corresponding to the following Bellman iterations needed to be 33

learned, as opposed to DQN (Mnih et al., 2015), where only one temporal difference related to the 34

first projection step is considered. Each temporal difference is learned by a different neural network, 35

making this method part of the DQN variants using multiple Q estimates during learning. Those 36

consecutive temporal differences are computed in a telescopic manner, where the online network of 37 the first temporal difference is used to build a target for the second temporal difference and so on. 38 This loss implicitly incurs a hierarchical order between the Q estimates by forcing each Q estimate to 39 be the projection of the Bellman iteration corresponding to the previous Q estimate, hence the name 40 iterated Deep Q-Network (iDQN). In the following, we start by reviewing algorithms built on top of 41 DQN, highlighting their behavior in the space of Q -functions. We then introduce a new approach to 42 Q-learning that emerges naturally from a graphical representation of DQN. In Section 5, we show 43 the benefit of our method on the Arcade Learning Environment benchmark (Bellemare et al., 2013). 44 Our approach outperforms DQN and Random Ensemble Mixture (REM, Agarwal et al. (2020)), its 45 closest baselines, establishing iDQN as a relevant method to consider when aggregating significant 46 advances to design a powerful value-based agent such as Rainbow (Hessel et al., 2018). We also 47 perform further experimental studies to bring evidence of the intuition on which iDQN is built. We 48 conclude the paper by discussing the limits of iDQN and pointing at some promising follow-up ideas. 49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

2 Preliminaries

We consider discounted Markov decision processes (MDPs) defined as M = ⟨S , A , P , R , γ ⟩ , where S is the state space, A is the action space, P : S × A × S → R is the transition kernel of the dynamics of the system, R : S × A → R is a reward function, and γ ∈ [0 , 1) is a discount factor (Puterman, 1990). A deterministic policy π : S → A is a function mapping a state to an action, inducing a value function V π ( s ) ≜ E [ ∑ + ∞ t =0 γ t R ( S t , π ( S t )) | S 0 = s ] representing the expected cumulative discounted reward starting in state s and following policy π thereafter. Similarly, the action-value function Q π ( s, a ) ≜ E [ ∑ + ∞ t =0 γ t R ( S t , A t ) | S 0 = s, A 0 = a, A t = π ( S t ) ] is the expected discounted cumulative reward executing action a in state s , following policy π thereafter. Q-learning aims to find a function Q from which the greedy policy π Q ( s ) = arg max a Q ( · , a ) yields the optimal value function V ∗ ( · ) ≜ max π : S→A V π ( · ) (Puterman, 1990). The (optimal) Bellman operator Γ ∗ is a fundamental tool in RL for obtaining optimal policies, and it is defined as:

for all ( s, a ) ∈ S × A . It is well-known that Bellman operators are contraction mappings in L ∞ -norm, such that their iterative application leads to the fixed point Γ ∗ Q ∗ = Q ∗ in the limit (Bertsekas, 2015). We consider using function approximation to represent value functions and denote Θ the space of their parameters. Thus, we define Q Θ = { Q ( ·| θ ) : S × A → R | θ ∈ Θ } as the set of value functions representable by parameters of Θ .

3 Related Work

To provide an overview of the related work, we propose to view the related algorithms from the perspective of their behavior in the space of Q -functions, which we denote by Q . Due to the curse of dimensionality, covering the whole space of Q -functions with function approximators is practically infeasible, as it requires a large number of parameters. Therefore, the space of representable Q -functions Q Θ only covers a small part of the whole space Q . We illustrate this in Figure 1a by depicting the space of representable Q -functions as a subspace of Q . One can deduce two properties from this gap in dimensionality. First, the optimal Q -function Q ∗ is a priori not representable by any chosen function approximator. Second, the same is true for the optimal Bellman operator Γ ∗ applied to a representable Q -function. That is why in Figure 1a, both functions Q ∗ and Γ ∗ Q are drawn outside of Q Θ . Additionally, thanks to the contracting property of the optimal Bellman operator || Γ ∗ Q -Q ∗ || ∞ ≤ γ || Q -Q ∗ || ∞ , we know that the iterated Q given by Γ ∗ Q is at least γ closer to the optimal Q ∗ than the initial Q (Bertsekas, 2015). The goal of most value-based methods is to learn a Q -function that is as close as possible to the projection of the optimal Q -function on the space of representable Q -functions, shown with a dotted line in Figure 1a.

This perspective allows us to represent a variety of Q-learning algorithms proposed so far in an 82 intuitive way in a single picture. For example, Figure 1a depicts how Deep Q-Network (DQN) by 83 Mnih et al. (2015) works. With a target network ¯ Q 0 , DQN aims at learning the iterated target network 84 Γ ∗ ¯ Q 0 , also called 'target', using an online network Q 1 . The loss used during training is shown in red. 85

86

87

88

89

90

91

92

loss

loss

(a) DQN learns the optimal Bellman iteration of the Q -function represented by the target network ¯ Q 0 . It uses an online network Q 1 for that purpose.

(b) DQN updates the target network to make the target change. This means moving one Bellman iteration forward in the space of Q -functions.

(a) Two common ways of improving DQN is to develop a more efficient empirical Bellman operator, noted ˜ Γ ¯ Q , or to modify the space of representable Q -functions, noted ˜ Q Θ .

loss

(b) REM uses multiple Q -functions to explore areas in the space of Q -functions. A random convex com- bination of the stored Q -functions is sampled at each gradient step.

In the optimal case, after a pre-defined number of training steps, the online network should represent the projection of the iterated target network onto the space of representable

Q

-functions (shown with a dotted line). This perspective also gives a way to understand the hyper-parameter related to the

frequency at which the target network is updated. It is the number of training steps before moving to the next Bellman iteration. When the target network is updated, it will be equal to the online network,

and the next Bellman iteration will be computed from there, as shown in Figure 1b. It is important to note that in DQN, the empirical Bellman operator is used instead of the optimal Bellman operator.

93

The term included in the loss at every gradient step is a stochastic estimation of the optimal Bellman

95

3.1 DQN Variants

The DQN paper has inspired the community to develop further methods which improve its efficiency. 96 Alarge number of those algorithms focuses on using a better empirical Bellman operator (Van Hasselt 97 et al. (2016), Fellows et al. (2021), Sutton (1988)). For instance, double DQN (Van Hasselt et al., 98 2016) uses an empirical Bellman operator designed to avoid overestimating the return. This results 99 in a different location of the Bellman iteration, as shown in Figure 2a. Other approaches consider 100 changing the space of representable Q -functions (Wang et al. (2016), Osband et al. (2016)). The hope 101 is that the projection of Q ∗ on the space of representable Q -function is closer than for the classical 102 neural network architecture chosen in DQN. It is important to note that adding a single neuron to 103 one architecture layer can significantly change the space of representable Q -function. Wang et al. 104 (2016) showed that performance can be increased by including inductive bias in the neural network 105 architecture. This idea can be understood as a modification of the space of Q -functions, as shown 106 in Figure 2a where the new space of representable Q -function is colored in yellow. Furthermore, 107 algorithms such as Rainbow (Hessel et al., 2018) leverage both ideas. Other approaches, however, 108 such as prioritized replay buffer (Schaul et al., 2015), cannot be represented in the picture. 109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

loss

loss

loss

loss

(a) iDQN learns two Bellman iterations at once. It uses a second target network ¯ Q 1 to learn the second Bellman iteration. This second target network stays close to the first online network Q 1 .

(b) iDQN updates the online and target networks to move one Bellman iteration ahead. Each network takes the value of the network corresponding to the following iteration.

3.2 Random Ensemble Mixture

Among the variants of DQN, ideas involving learning several Q -functions (Osband et al. (2016), Agarwal et al. (2020)) are particularly close to our method. Even if they are close, they remain orthogonal in the sense that they can be combined with our idea to create a more powerful agent. Random Ensemble Mixture (REM, Agarwal et al. (2020)) has been shown to be state-of-the-art for DQN variants with several Q -functions. Instead of exploring the space of Q -functions point by point as DQN does, REM moves in this space by exploring area by area, where the areas are the convex hull of the Q -functions stored in memory. As represented by the red line in Figure 2b, the loss used by REM is

where θ denotes the parameters of the online network and ¯ θ the target parameters, D is the replay buffer, ∆ is the standard simplex and l is the Huber loss (Huber, 1992), and Q α i = ∑ k α k Q k i , i ∈ { 0 , 1 } . For a Bellman iteration i , the k th learnt Q -function is noted Q k i . Figure 4a shows how this loss is computed with the neural network's architecture used in REM.

4 Iterated Deep Q-Networks

Wepropose an approach built on top of DQN. The main idea emerges naturally from the representation developed in Section 3. In DQN, Figure 1 illustrates that to learn two Bellman iterations, we first need to wait until the first iteration is learned, and then we need to update the target before starting to learn the second iteration. Conversely, we propose to use a second online network that learns the second Bellman iteration while the first Bellman iteration is being learned. The target for the second online network is created from a second target network that is frequently updated to be equal to the first online network. Figure 3a shows how iDQN behaves in the space of Q -function. It is important to understand that in iDQN, both online networks are learned at the same time. This idea can be further applied, as more Q -functions can be considered for learning the following Bellman iterations. Repeating the same schema, we can learn a following Bellman iteration by adding a new online network that would use a new target network set to be equal to the last online network. In Figure 3a, it would mean adding an online network Q 3 , learnt to be the projection of the target Γ ∗ ¯ Q 2 , where ¯ Q 2 is a target network periodically updated to the value of Q 2 . This iterative process can be continued until memory usage becomes an issue. Once again, the loss can be observed on the representation in Figure 3a (see red lines), it writes

where θ is the online parameters and ¯ θ the target parameters, K is the number of Bellman iterations 139 considered at once. The k th learned Q -function corresponds to the k th Bellman iteration and is noted 140 Q k . The way the loss is computed from the neural network's architecture is presented in Figure 4a. 141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

(a) Losses and neural networks architectures.

(b) After the same number of gradi-

ent steps, iDQN has already started

to learn the second Bellman itera-

tion, noted

Q

2

.

In iDQN, updating the target networks does not bring the target parameters to the next Bellman iteration like in DQN. It simply refines their positions to be closer to the online networks to allow better estimates of the iterated Q -functions. To be able to go further in the Bellman iterations, we periodically consider a new online Q -function to learn and discard the first online and target network, as shown in Figure 3b. For this example, Q 3 is now considered and Q 1 and ¯ Q 0 are left aside. We call this procedure a rolling step . In practice, the rolling step is simple to implement. A new head to the neural network of Figure 4a is added, with the index K +1 and the first head is removed. It leads us to introduce a new hyper-parameter that indicates at which frequency the rolling step is performed. It is worth noticing that if K is set to 1 and if the rolling step frequency is synchronized with the target update frequency in DQN, then we recover DQN, i.e., iDQN with K = 1 is equal to DQN.

In DQN, the actions are drawn from the online network. For iDQN, one needs to choose from which of the multiple online networks to sample. One could stick to DQN and choose the first online network. One could also use the last online network since it is supposed to be the one that is closer to the optimal Q -function, or one could pick an online neural network at random as it is done in Bootstrapped DQN (Osband et al., 2016). We do not consider taking the mean as REM proposes because the online Q -functions are expected to follow a specific arrangement in space. Taking their mean could lead to unwanted behavior. We investigate these sampling strategies in Section 5.1. One can see how iDQN uses the same algorithm as DQN through its pseudo-code shown in Algorithm 1 in the supplementary material. The only two changing parts are the behavioral policy and the loss.

4.1 Understanding the Loss of iDQN

The crucial advantage of iDQN is coming from the loss. In addition to the loss used in DQN, it 162 contains K -1 more terms. Those terms concern the future Bellman iterations, hence the fact that 163 iDQN allows for a better learning of the Bellman iterations . In practice, each Bellman iteration 164 is learned with K times more gradient steps than in DQN while having the same overall number 165 of gradient steps. This means that each selected sample is used K times more or, in other words, 166 that each network sees K times more samples. As mentioned earlier, updating the target in DQN 167 moves the learning procedure one Bellman iteration further. The same goes for the rolling step for 168 iDQN. From the fact that each Bellman iteration is learned with K times more gradient steps than in 169 DQN, we can allow iDQN to perform the rolling step more frequently than DQN updates the target, 170 which means that iDQN will do more Bellman iterations at the end of the training, while still learning 171 each iteration better than DQN. Figure 4b pinpoints the advantage of iDQN with K = 2 over DQN. 172

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

iDQN. When DQN updates its target network Q 0 to ¯ Q 1 , the new online network Q 2 is located at ¯ Q 1 . When the rolling step is performed for iDQN, ¯ Q 1 is located at the same position as the ¯ Q 1 of DQN 1 but Q 2 has already been learnt and is closer to the optimal Q -function than the Q 2 of DQN. This phenomenon is even stronger when K increases. Another way to understand iDQN is to see iDQN as a way to pre-train the next online Q -functions instead of taking them equal to the online network as it is done in DQN.

5 Experiments

We evaluate our proposed algorithm on 54 Atari 2600 Games (Bellemare et al., 2013) 2 . Many implementations of Atari environments along with classical baselines are available online (Castro et al. (2018), D'Eramo et al. (2021), Raffin et al. (2021), Huang et al. (2022)). We choose to mimic the implementation choices made in Dopamine (Castro et al., 2018) since it is the only one to release the evaluation metric for all relevant baselines to our work and the only one to use the evaluation metric recommended by Machado et al. (2018). Namely, we use the game over signal to terminate an episode instead of the life signal. The input given to the neural network is a concatenation of 4 frames in gray scale of dimension 84 by 84 . To get a new frame, we sample 4 frames from the Gym environment (Brockman et al., 2016) configured with no frame skip, and we apply a max pooling operation on the 2 last gray scale frames. We use sticky actions to make the environment stochastic (with p = 0 . 25 ). The training performance is the one obtained during learning. By choosing an identical setting as Castro et al. (2018) does, we can take the baselines' training performance of Dopamine without the need to train them again ourselves. To certify that the comparison is fair, we compared our version of DQN to their version and concluded positively (see Figure A of the supplementary material).

Hyperparameter tuning. The hyperparameters shared with DQN are kept untouched. The two additional hyperparameters (rolling step frequency and target parameters update frequency) were set to follow our intuition on their impact on the performance. As a reminder, the frequency at which the rolling step is performed is comparable to the target update frequency in DQN. Since iDQN allows more gradient steps per iteration, we set this hyperparameter to be 25 % lower than the target update frequency in DQN ( 6000 compared to 8000 ). To further ensure that our code is trustworthy, Figure A in the supplementary material shows that DQN achieves similar training performances than iDQN with K = 1 and the rolling step frequency is set to be equal to the target update frequency of DQN. It is important to note that decreasing the target parameters update results in a more stable training but also a higher delay with the online networks which can harm the overall performance. We set it to 30 , allowing 200 target updates per rolling step. We choose K = 5 . This choice is further discussed in Section 5.1. To make the experiments run faster, we designed the Q -functions to share the convolutional layers. Additionally, we consider the first layers of the neural network useful for extracting a feature representation of the state space. This is why this choice can potentially be beneficial to our algorithm. Further details about the hyperparameters can be found in the supplementary material.

Performance metric. As recommended by Agarwal et al. (2021), we choose the interquartile mean (IQM) of the human normalized score to report the results of our experiments with shaded regions showing pointwise 95 %percentile stratified bootstrap confidence intervals. IQM is a trade-off between the mean and the median where the tail of the score distribution is removed on both sides to consider only 50 %of the runs. 5 seeds are used for each game.

Main result. iDQN greatly outperforms DQN (Adam) on the aggregation metric, proposed in Agarwal et al. (2021). Figure 5a shows the IQM human normalized score over 54 Atari games according to the number of frames sampled during the training. In the last millions of frames, iDQN reaches a higher IQM human normalized score than DQN (Adam). iDQN performs better than REM as well, showing that our approach should be preferred when using multi-head Q networks. We do not consider other variants of DQN to be relevant baselines to compare with. The ideas used in Rainbow, IQN or Munchausen DQN (Vieillard et al., 2020) can be included in iDQN algorithm to build an

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

iDQN K=5

DQN (Nature)

DQN (Adam)

C51

REM

0

50

100

150

200

Number of Frames (in millions)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

IQM Human Normalized Score

(a) Training performance.

0

2

4

6

8

Human Normalized Score (

)

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of runs with score  >

(b) Performance profile.

iDQN K=5

DQN (Nature)

DQN (Adam)

C51

REM

0

50

100

150

200

Number of Frames (in millions)

0.0

0.5

1.0

1.5

IQM Human Normalized Score

BankHeist

0

50

100

150

200

Number of Frames (in millions)

0

1

2

3

4

Enduro

0

50

100

150

200

Number of Frames (in millions)

0.0

0.2

0.4

0.6

0.8

ChopperCommand

0

50

100

150

200

Number of Frames (in millions)

0.0

0.5

1.0

1.5

Frostbite

0

50

100

150

200

Number of Frames (in millions)

1.0

0.5

0.0

Skiing

even more powerful agent. We further discuss it in Section 5.1. To visualize the distribution of final scores, we plot the performance profile in Figure 5b. It shows the fraction of runs with a higher final score than a certain threshold given by the X axis. In some ranges of human normalized score, iDQN statistically dominates DQN. For example, there are more games in which iDQN achieves 1 . 5 times a human performance ( τ = 1 . 5 ) than DQN. It is harder to distinguish iDQN and REM because the gap in final performance between iDQN and REM is smaller than between iDQN and DQN. In Figure 6, we selected 5 games where different behaviors can be observed. On some games like BankHeist and Enduro , iDQN overtakes all its baselines. Interestingly, in ChopperCommand , DQN (Adam) and REM fail at outperforming DQN (Nature), while iDQN is comparable with C51 in performance. This shows that efficiently learning the Bellman iterations plays an important role in some environments. In Frostbite , REM is more efficient than DQN (Adam). iDQN outperforms both algorithms in this game, being the only one achieving superhuman performances. Finally, in Skiing , REM and C51 failed to be better than a random agent, while iDQN sticks to the performance of DQN (Adam). We believe this behavior comes from the fact that iDQN is close to DQN in principle, which minimizes the risk of failing when DQN succeeds. We present the training performance of all the remaining games in Figure B of the supplementary material.

5.1 Ablation Studies

We perform several ablation studies to showcase the different behaviors of iDQN. We first investigate 241 the importance of the number of Bellman iterations K taken into account in the loss. As shown in 242 Figure 7 for the games Asteroids and Asterix , increasing K to 10 iterations could be beneficial. In 243 Qbert , the gain seems not certain. We believe further tuning of hyperparameters should bring iDQN 244 with K = 10 to yield better performances than iDQN with K = 5 . We insist that no hyperparameter 245 tuning has been performed to generate this plot. In order to have the same number of gradient steps 246 per Bellman iteration for K = 5 and K = 10 , we simply halved the frequency at which the rolling 247 step is performed for K = 10 , bringing it to 3000 since we doubled K . Interestingly, the performance 248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

iDQN K=5

iDQN K=10

DQN (Adam)

0

50

100

150

200

Number of Frames (in millions)

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

0.0150

IQM Human Normalized Score

Asteroids

0

50

100

150

200

Number of Frames (in millions)

0.0

0.5

1.0

1.5

2.0

2.5

Asterix

0

50

100

150

200

Number of Frames (in millions)

0.0

0.2

0.4

0.6

0.8

1.0

iDQN uniform sampling

iDQN first online Q sampling

iDQN last online Q sampling

0

50

100

150

200

Number of Frames (in millions)

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

IQM Human Normalized Score

Asteroids

0

50

100

150

200

Number of Frames (in millions)

0.0

0.5

1.0

1.5

2.0

2.5

Asterix

0

50

100

150

200

Number of Frames (in millions)

0.0

0.2

0.4

0.6

0.8

1.0

never drops in the 3 considered games. Therefore, we recommend increasing K as much as the computational resources allow it.

In Section 4, we mentioned several sampling strategies. Figure 8 illustrates the different possibilities mentioned earlier: sampling from a uniform online Q -function, sampling from the first online Q -function like DQN does, or sampling from the last online Q -function, it is supposed to be the closest to the optimal Q -function. No significant difference exists except for the game Asteroids , where sampling from a uniform online Q -function seems to yield better performance throughout the training. We believe that the increase in performance comes from the fact that the online Q-functions generate a wider variety of samples compared to only sampling from the first or last online Q-function. We recommend sampling actions from an online Q -function chosen at random.

iDQN heavily relies on the fact that the learned Q functions are located at different areas in the space of Q -functions. We computed the standard deviation of the output of the learned Q -functions during the training in Figure 9 to verify this assumption. The figure shows that the standard deviation among the Q -function is indeed greater than zero across the 3 studied games. Furthermore, we can observe that the standard deviation decreases during training, hinting that they become increasingly closer. This matches the intuition that at the end of the training, the Q -functions should be close to the projection of the optimal Q -function, hence being close to each other.

6 Discussion

In Figure 10, we compare iDQN with other powerful baselines to show the gap between those baselines and our approach, which does not use the benefit of a prioritized replay buffer and a n -step return. The curves for other algorithms shown in Figure 10 depict the publicly available metrics for those algorithms. 3 The training performance of IQN and Munchausen DQN without the n -step return would be interesting to analyze, but our limited resources do not allow us to run those baselines. The major improvement of Rainbow over C51 is made by using a prioritized replay buffer and

0

50

100

150

200

Number of Frames (in millions)

0.02

0.04

0.06

0.08

IQM inter-head standard deviation

Asteroids

0

50

100

150

200

Number of Frames (in millions)

0.04

0.06

0.08

0.10

0.12

Asterix

0

50

100

150

200

Number of Frames (in millions)

0.01

0.02

0.03

0.04

0.05

0.06

BankHeist

iDQN K=5

REM

DQN (Nature)

Rainbow (C51 + 3-step return + PER)

QR-DQN + 3-step return

IQN + 3-step return

DQN (Adam)

Munchausen + IQN + 3-step return

C51

0

50

100

150

200

Number of Frames (in millions)

0.0

0.5

1.0

1.5

2.0

IQM Human Normalized Score

0

2

4

6

8

Human Normalized Score (

)

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of runs with score  >

adding a 3-step return which gives hope for following works to study the potential strength of any 273 Rainbow-like iDQN approach. The training performances of iDQN on 54 Atari games along with 274 those more advanced methods are available in Figure C of the supplementary materials. 275

Our approach introduces two new hyperparameters, rolling step frequency and target parameters 276 update frequency, that need to be tuned. However, we provide a thorough understanding of their 277 effects to mitigate this drawback. While extreme values of some hyperparameters were found to be 278 harmful to the performance of DQN, e.g., changing the target update frequency, little variation of 279 the presented values was found to have only a small impact on the overall performance. Regarding 280 the resources needed to train an iDQN agent, more computations are required to get the gradient of 281 the loss compared to DQN. Thanks to the ability of JAX (Bradbury et al., 2018) to parallelize the 282 computation, iDQN with K = 5 only requires 1 to 2 times more time to run. With the released code 283 base, each run presented in this paper can be run under 3 days on an NVIDIA RTX 3090. 284

285

7 Conclusion

In this paper, we have presented a way to learn the Bellman iterations more efficiently than DQN. 286 The underlying idea of iDQN comes from an intuitive understanding of DQN's behavior in the space 287 of Q -functions. It allows each Q estimate to be learned with more gradient steps without increasing 288 the overall number of gradient steps. iDQN outperforms its closest baselines, DQN and REM, on the 289 Atari 2600 benchmark. While we proposed an approach to Q -learning that focuses on the projection 290 step of the Bellman iterations, an interesting direction for future work would be to investigate which 291 other past improvements of DQN, in combination with iDQN, would lead to a new state-of-the-art 292 for value-based methods. 293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

References

Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pp. 104-114. PMLR, 2020.

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 34, 2021.

Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Bertsekas, D. Reinforcement learning and optimal control . Athena Scientific, 2019.

Bertsekas, D. P. Dynamic Programming and Optimal Control 4 th Edition , Volume II . Athena Scientific, 2015.

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.

Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Bellemare, M. G. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 , 2018.

Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419, 2022.

D'Eramo, C., Tateo, D., Bonarini, A., Restelli, M., and Peters, J. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research , 22(131):1-5, 2021.

Fellows, M., Hartikainen, K., and Whiteson, S. Bayesian bellman operators. Advances in Neural Information Processing Systems , 34:13641-13656, 2021.

Funk, N., Chalvatzaki, G., Belousov, B., and Peters, J. Learn2assemble with structured representations and search for robotic architectural construction. In Conference on Robot Learning , pp. 1401-1411. PMLR, 2022.

Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18, 2022. URL http://jmlr.org/papers/v23/ 21-1342.html .

Huber, P. J. Robust estimation of a location parameter. Breakthroughs in statistics: Methodology and distribution , pp. 492-518, 1992.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562, 2018.

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Puterman, M. L. Markov decision processes. Handbooks in operations research and management science , 2:331-434, 1990.

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22 (268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015.

Sutton, R. S. Learning to predict by the methods of temporal differences. Machine Learning , 3(1): 9-44, August 1988. URL http://www.cs.ualberta.ca/~sutton/papers/sutton-88.pdf .

Tieleman, T., Hinton, G., et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012.

Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Vieillard, N., Pietquin, O., and Geist, M. Munchausen reinforcement learning. Advances in Neural Information Processing Systems , 33:4235-4246, 2020.

Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In International conference on machine learning , pp. 1995-2003. PMLR, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on learning the projection of the empirical Bellman operator's iteration on the space of function approximator (neural model). This being done through increasing the number of gradient steps using multiple heads with a certain form of update. While retaining the same total number of gradient steps and samples compared to common approaches, the proposed method seems to provide better results (at the cost of retaining multiple heads and more computation).\n\nSTRENGTHS:\nThe idea is interesting, novel and practical. The paper also **experimentally** shows noticeable improvement over various baselines.\n\nWEAKNESSES:\n- The presented method is quite simple and could have been presented much more efficiently with simple math and direct explanation rather than lengthy discussions over multiple figures. \n\n- The choice of $K$ seems to have a significant impact on the behaviour, which also varies depending on the domain. Suggestion: some formal analysis (e.g., the algorithm's variance) could be useful to provide better insight about what to expect from larger $K$ in terms of other characteristics such as the transition kernel.\n\nQUESTIONS:\n- The main idea of the paper is simple: using multiple heads for a Q network and compute the Bellman loss for each of them using its preceding head’s target, which provides multiple grad steps at once for a given transition (also performs the rolling step more frequently). This has been presented through a long (and figure-based) discussion. While I did like the figure-based discussion of the “related work” section, I believe a set of equations can be far more efficient to explain the method and compare it against other techniques. Honestly, looking at Eq 2 with a short paragraph of explanation is sufficient (and way better) to understand what is going on. The current form is ponderous and hard to follow.\n\n- In section \"Preliminaries\": the policy is assumed to be **deterministic**. Why? (this is a significant limitation and calls for more discussion.)\n\n\n- It’s been said both in the abstract and section 4.1 that iDQN concern the future Bellman iterations. Looking at the loss function (Eq 2), all the $Q_k$’s corresponding loss terms are computed using the same transition tuple. I do not see why it has anything to do with “future Bellman iterations”. \n\n\n- I liked figures 1 & 2 with the corresponding discussion of section 3. One missing point here is the value-mapping idea: to functionally map $Q$ into a difference space and perform the learning, then map back to the space of $Q$. This basically alters the actual $Q$ space and is inherently different from choosing a different approximator. Hence, the approximator’s space boundary may retain better properties such as remaining homogeneously close to the $Q^*$ for a larger subset of states. See this paper [and possibly others]: https://arxiv.org/abs/2203.07171  note that this paper also discusses orchestration of multiple mappings...\n\n- For the ensemble techniques, this paper is of particular interest, which also provides a neat way of diversification (hence, requiring smaller ensemble size): https://arxiv.org/abs/2110.01548. \n\n\nOther comments:\n\n- L78--79 --> “... it is at least $\\gamma$ closer ...” --> if being picky, this is not accurate. Rather, the distance is shrunk by $\\gamma$ (as a multiplier, not a subtractor). \n\n- Beginning of L86 -->. “In the optimal case...” --> what do you mean? \n\n- Eq 2 --> what is $r’$ ?\n\nLIMITATIONS:\nWhile the authors mentioned at the end of Introduction that \"We conclude the paper by discussing the limits of iDQN ...\", they apparently forgot to do so! No discussion of limitations is provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Iterated Deep Q-Network (iDQN), a new DQN-based algorithm that incorporates multiple Bellman iterations into the training loss. The paper highlights the limitations of traditional RL methods that only consider a single Bellman iteration and proposes iDQN as a solution to improve learning. The algorithm leverages the online network of DQN to build targets for successive online networks, taking into account future Bellman iterations. The paper evaluates iDQN against relevant baselines on 54 Atari 2600 games and demonstrates its benefits in terms of approximation error and performance.\n\nSTRENGTHS:\n1. The proposed iDQN algorithm introduces a novel approach to incorporate multiple Bellman iterations into the training loss, addressing the limitations of traditional RL methods.\n2. The paper provides a well-structured review of relevant literature, discussing the behavior of various Q-learning algorithms in the space of Q-functions. This analysis helps in understanding the motivation behind iDQN and its relationship with other methods.\n3) The empirical evaluation on selected Atari games demonstrates the superiority of iDQN over its closest baselines, DQN and Random Ensemble Mixture. This provides empirical evidence of the effectiveness of the proposed approach.\n\nWEAKNESSES:\n1. It would be helpful if the paper included more comparisons with widely-known baselines in the field. While the paper compares iDQN to DQN and Random Ensemble Mixture, it would be valuable to see how iDQN performs against other popular RL algorithms like R2D2.\n2. Some parts of the paper could be further clarified to improve the reader's understanding. For example, the explanation of the loss function and the graphical representations of DQN variants could be made more concise and intuitive.\n\nQUESTIONS:\n1. Can you demonstrate the performance table of Atari 26/54 games?\n2. Can you continue to improve the demonstration of Figure 1,2,3? It's difficult to understand what they are demonstrating.\n3. Will you provide code or implementation details?\n\nLIMITATIONS:\nIt would be helpful if the paper included more comparisons with widely-known baselines in the field. This paper has no negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of how to get accurate approximations of optimal Q-functions. The paper introduces a new algorithm called iterated DQN (iDQN). iDQN incorporates multiple consecutive Bellman iterations into the training process, which aims to allow for better approximation of optimal action-value functions. It uses the online network of DQN to build a target for a second online network, and so on, for considering future Bellman iterations. The authors conducted several experiments based on Atari games by comaping iDQN with baseline methods, including DQN and Random Ensemble Mixture.\n\nSTRENGTHS:\n- Significance and Originality: The topic that the paper studies - how to learn the Bellman iterations efficiently, is an important topic in reinforcement learning. The authors propose a simple yet effective method, called iterated DQN, to improve the learning efficiency. Specifically, iterated DQN uses a second online Q-network for learning the second Bellman iteration simultaneously, where the target for the second online Q-network is created from a second target network according to the first online network. In this way, the loss can include K-1 more terms compared to the original loss used in DQN. The way for using such kind of ensemble seems novel, which allows for improved efficiency.\n\n- Clarity: The paper is well-written and easy to follow, with very clear illustrations for the update for DQN, REM, and iterated DQN as in Figures 1-4.\n\nWEAKNESSES:\n- Quality: The paper presents a simple yet effective idea, but it could be further strengthened particularly in theoretical and empirical analysis. First, the authors could provide a theoretical guarantee for iterated DQN by examining its convergence speed, in addition to the intuitive explanation given in Section 4.1. This would lend credibility to their claims. Second, the empirical validation raises concerns, as iterated DQN's performance is only marginally better than that of previous baseline methods such as DQN (Adam), C51, and REM. This modest improvement does not strongly support the paper's assertions. Lastly, it would be beneficial for the authors to include a memory comparison, as employing more Q-networks may lead to increased memory costs, which is an important consideration for practical applications.\n\nQUESTIONS:\n1. Can the authors also develop a theoretical guarantee for iterated DQN by analyzing its convergence speed besides the intuitive explanation in Section 4.1? It would be better if  the claim is also demonstrated theoretically.\n\n2. My main concern for the paper is about the empirical validation part. Iterated DQN actually performs very closely to previous baseline methods including DQN (Adam), C51, and REM, with quite small improvement margin, which does not support the claim in the paper well. \n\n3. Can the authors also show the memory comparison, as using more Q-networks can induce a larger memory cost?\n\nLIMITATIONS:\nThe authors have discussed some of the limitations by considering other value-based methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an extension to DQN aimed at improving projection steps in Q value updates. There are two main contributions of the paper:\n\n- The paper intuitively explains the Q-value learning characteristics of DQN variants caused by a mismatch between the optimal Bellman operator and the set of representable Q functions.\n- The authors propose the iterated DQN (iDQN) method, which keeps track of a collection of K online Q-functions. When updating these Q networks, the previous Q function is used as the target network.\n\nExperiment results show that iDQN outperforms a collection of DQN variants on the standard Atari benchmark. Further ablation studies explore the effect of K and sampling strategies for iDQN and conclude that bigger K and uniform sampling of Q networks are in general preferable.\n\nSTRENGTHS:\n- The figures explaining the projection characteristics of existing DQN variants are very intuitive and provide excellent motivation for the proposed method.\n- The iDQN method, the newly introduced hyperparameters, and the experiment settings are communicated clearly and transparently.\n- The results on Atari are convincing.\n- The ablation studies on the choice of K and sampling method are insightful.\n\nWEAKNESSES:\n- The figures used to explain the projection behaviors of DQN variants are created for illustration, but not from actual experiments.\n- As discussed by the authors, iDQN doesn't outperform more recent DQN variants which employ other tricks to improve performance.\n\nQUESTIONS:\n- The online Q networks are known to be unstable during training. Could later Q networks suffer from compounding stability issues as they use previous online Q networks as targets?\n- Connecting back to weakness #1, is it possible to create a visualization of the projection behaviors in some toy environments?\n\nLIMITATIONS:\nThe authors discussed how iDQN is not able to outperform more recent DQN variants using other tricks. It would also be nice if the authors can discuss the training stability of iDQN.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a new variant of DQN algorithm, iDQN, is proposed by replacing the classical Bellman iteration with several consecutive Bellman iterations and using multiple Q networks.\nIntuitively, this new Bellman operator propogates reward sigals more efficiently thus speeds up learning, with the cost of more computation and memory.\nAs the number of consecutive Bellman iterations increases, it is shown that the learning performance of iDQN in Atari games is also increased.\n\nSTRENGTHS:\nAs far as I know, the proposed method is novel. The new algorithm, together with several baselines, are tested in 54 Atari games.\nMany illustrative pictures are included to help understand the new Bellman operator.\nThe paper is generally well-written.\n\nWEAKNESSES:\n1. It will make this work much better if a theoretical analysis of the proposed Bellman operator is provided, such as convergence speed, the affect of the number of Q networks, etc.\n2. The performance of iDQN is only slightly better than baselines, such as DQN(Adam). A summarized result (e.g. the first figure in [DQN Zoo](https://github.com/deepmind/dqn_zoo)) can make the comparison in Atari games much clearer.\n3. Missing related works about ensemble methods + DQN, e.g. Averaged DQN and Maxmin DQN.\n4. Misssing baselines: DQN + n-step return. Both iDQN and this baseline try to accelerate the propogations of reward signals. Furthermore, although it is mentioned that \"We do not consider other variants of DQN to be relevant baselines to compare with.\", more explanations are necessary.\n5. It is claimed that \"Our approach introduces two new hyperparameters, rolling step frequency and target parameters update frequency, that need to be tuned. However, we provide a thorough understanding of their effects to mitigate this drawback. \". However, I don't find the understanding thorough enough.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe limitations of iDQN are that it takes more memory and computation than DQN. Jax is used to parallelize the computation, making the training time increase acceptable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}