Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
Authors: Théo Vincent
Daniel Palenicek
Boris Belousov
Jan Peters
Carlo D'Eramo
Venue: Accepted by TMLR ()

Abstract: The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, until now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated $Q$-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari $2600$ games and MuJoCo continuous control problems.

Published in Transactions on Machine Learning Research (02/2025)
Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
Théo Vincent
theo, vincent&&ffi..ccc
DPKT, SATROL Team & TU Darmstaát
Daniel Palenicek
TU Darmstadt & Hessian.ot
Boris Belousov DFKI, SATROL Team
b www.sbblooosov@rrhoooloomnnggo de
Jan Peters
peters@ias. tu-aaasstaadddaaa...... de
DPKI, SAIROL Team TU Darmstadt Hessian.dt
Carlo D'Eramo
carlo. deromme wwipmmoohhni--uueruuuuu....
University of Würzburg TU Dormstadt & Hessian.ai
Reviewed on OpenReview: https: openreview. net/ pet/formm? Ld-L12BBB
Abstract
The vast majority of Reinforcement Learning methods is largely impacted by the com- putation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample- efficiency the lcarning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application an empirical approximation Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator once, benefiting the underlying learning algorithm. However, until now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated Q-Network (i-QN), novel principled approach that enables multiple consecutive Bellman updales by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN theorctically grounded and that it can be scamlessly used valuo- based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari 2600 games and MuJoCo continuous control problems. Our code is publicly available at https: githbb. com/ theovincent/ i-DQN and the trained models are uploaded at https: //hgggiggaaae. co/ TheoVincent/ Atari_ i i-QN.
the
1 Introduction
Deep Reinforcement Learning (RL) algorithms have achieved remarkable success in various fields, from nuclear physics (Degrave et al., 2022) to construction assembly tasks (Funk et al., 2022). These algorithms aim at obtaining good approximation of an action-value function through consecutive applications of the Bellman operator P to guide the learning procedure in the space of Q-functions, i.e., Qo + IQ, Q1 + IQ = Q2 .... ... (Bertsekas, 2019). This process is known as value iteration. Approximate value iteration (AVT) (Farahmand, 2011) extends the value iteration scheme to function approximation by adding projection step, Qo rQu x Q1 + IQ1 z Q2 ..... Thus, the 8th Bellman update IQ gets projected back onto the chosen Q-function space via new function Qx approximating lQ This projection step

---

Published in Transactions on Machine Learning Research (02/2025)
also appears in opprozimate policy evaluation (APE), where empirical version of the Bellman operator
for behavioral policy is repeatedly applied to obtain its value function (Sutton & Barto, 1998).
In this paper, we discuss and tackle two effi- ciency issues resulting from the AVI learning scheme. (i) Projection steps are made sequen- tially in the sense that the following Bellman update is only considered by the learning pro- once the current projection step is frozen. For example, IQ1 starts to be learned only once Q1 is frozen. This harms the efficiency of the training. (ii) Samples are only used to learn one-step application of the Bellman op- crator at each gradient step, roducing sample- efficiency. We propose a novel approach to overcome these limitations by learning con- secutive Bellman updates simultancously, We leverage neural network function approxima- tion to learn consecutive Bellman updates in a telescopic manner, forming a chain where each neural network learns the application of the Bellman operator over the previous one, as shown in Figure 1. This leads to a hierarchical ordering between the Q-estimates, where each one is the projection of the Bell- man update corresponding to the previous one, hence the name iterated Q-Network (i-QN). Importantly, iQQ distributes the samples across all considered projection steps, thereby increasing the number of samples that each Q-function is learned from. Our approach can be seamlessly used in place of the regular one-step Bellman update by any value-based or actor-critic method, e.g., Deep Q-Network (DQN) (Mnih et al., 2015), Soft Actor-Critic (SAC) (Haarnoja et al., 2018). In the following, we motivate our approach theoretically and provide an algorithmic implementation that we validate empirically on Atari 2600 (Bellemare et al., 2013) and MuJoCo control problems (Todorov et al., 2012).
Q Q-Network
iterated Q-Network
rQ,
rQ
rQ,
rQ
K-1
Q₂
targer
C ON [QQ Q111
C - LQN
+ C ON
TQQ
Q
Figure 1: Iterated Q-Network (ours) uses the online network of regular Q-Network approaches to build a target for a second online network, and on, through the application of the Bellman operator T. The resulting loss Li- comprises K temporal difference errors instead of just one as in LQN.
Contributions. (1) We introduce iterated Q-Network (i-QN), novel approach that enables learning multiple Bellman updates at once. (2) We provide intuitive and theoretical justifications for the advantages and soundness of i-QN. (3) We show that i-QN can be seamlessly combined with value-based and actor-critic methods to enhance their performance, conducting experiments on Atari games and MuJoCo tasks
2 Preliminaries
We consider discounted Markov decision processes (MDPs) defined M = (8, A, P, R, 7), where S and A are measurable state and action spaces, P S x A A(S)* is transition kernel, R S x A - A(R) is reward function, and y E [0, 1) is a discount factor (Puterman, 1990). A policy is a function inducing an action-value function (*((s,a) a) A E, (E 7'R(s,, a,) = 8, 0g = a] that gives the expected discounted cumulative return executing action a in state 8, following policy π thereafter. The objective is to find an optimal policy म* = arg max, V*(.,, where (=()) = Ea-n(() '0²(( .))]. Approximate value iteration (AVI) and approximate policy iteration (API) are two common paradigms to tackle this problem (Sutton & Barto, 1998). While AVT aims to find the optimal action-value function (' A max, (-(, (- +);
S + A(A),
API alternates between approximate policy evaluation (APE) to estimate the action-value function of the current policy and policy improvement that improves the current policy from the action-value function obtained from APE.
Both paradigms aim to find the fixed point of a Bellman equation by repeatedly applying a Bellman operator starting from a random Quuntioon. AVI uses the optimal Bellman operalor r", whose fixed point is Q", while APE relies on the Bellman operator 1 associated with a policy x, whose fixed point is Q" (Bertsekas,
1 (X') denotes the of probability measures over set. X.
2

---

Published in Transactions on Machine Learning Research (02/2025)
TQ,
rQ,
TQ1
rQ1
2
TQ₂
rQ2
Qi
Q
Q
Q2
2
2
0
Q3
(a) Q-Network
(b) iterated Q-Network (ours)
Figure 2: Graphical representation of the regular Q-Network approach (left) compared to our proposed iterated Q-Network approach (right) in the space of Q-functions Q The regular Q-Network approach proceeds sequentially, i.e., Q2 is learned only when the learning process of Q1 is finished. With iterated Q-Network, all parameters are learned simultaneously. The projection of Q* and projections of the Bellman update are depicted with a dashed line?. The losses are shown in red.
2015). For any state E S and action a E A, D* and I are defined as
(s) En + max (s)))
(1)
a'EA
(2)
It is well-known that these operators are contraction mappings in -oom,, such that their iterative ap- plication leads to their fixed point in the limit (Bertsekas, 2015). However, in model-free RL, only sample estimates of those operators are used since the expectations cannot be computed in closed form. This ap- proximation, coupled with the use of function approximation to cope with large state-action spaces, forces learning the values of a Bellman update before being able to compute the next Bellman update. This pro- cedure, informally known as projection step, results in a sequence of projected functions (Q.) that does not shown correspond to the one obtained from the consecutive applications of the true Bellman operator, in Figure 2a, where I equals I* for AVI and n for APE, and Q* represents Q' for AVI and Q for APE. We denote Qe as the space of function approximators, where θ is the space of parameters. To learn Bell- man update TQ,, given a fixed parameter vector & E O for a sample s,a,r, r, Temporal Difference (TD) learning (Hasselt, 2010; Haarnoja et al., 2018) aims to minimize
(s. (
(ex,0,10.,,,,,r,55
(3)
over parameters 0 E O where f is an empirical estimate of T and QN stands for Q-Network (Dayan & Watkins, 1992; Mnih et al., 2015). As an example, in Q-learning setting, ())(sa)= =r + max, Q(s', d'), In current approaches, single Bellman update is learned at a time. The training starts by initializing the target parameters 8, and the online parameters 0,. The distance between Qo, representing the second Q-function Q1, and IQ representing the first Bellman update IQu. is minimized via the loss in Equation 3, as shown in Figure 2a. After a predefined number of gradient steps, the procedure repeats, as in Figure 2a, where the second and third projection steps are blurred to stress the fact that they are learned sequentially. However, this sequential execution is computationally inefficient because it requires many non-parallelizble gradient steps to train single projection. Moreover, this procedure is sample-inefficient since, at each gradient step, samples are used to learn only one Bellman update. In this work, we present a method that learns multiple Bellman updates at each gradient from a single sample batch. Importantly, our method scales deep RL as shown in Section 6.
for Q Q and a sample (s,arr,))
2The existence of projection Qa depends only on the choice of the fuuction approximator. Note that if the projection does not exist if the function appreximators are powerful enough to cover the entire space of Q-functious, i.c., Qe Q the presented abstraction is still valid. Indeed, any case, applying the Bellman operator moves away the target from the Q-funciion. Therefore, learning Bellman updates still required.
3

---

Published in Transactions on Machine Learning Research (02/2025)
3
Related work
Several methods have been proposed on top of Q- learning to improve various aspects. A large num- ber of those algorithms focus on variants of the em- pirical Bellman operator (Van Hassclt et al., 2016; Fellows et al., 2021; Sutton, 1988). For instance, double DQN (Van Hasselt et al., 2016) uses an em- pirical Bellman operator designed to avoid overes- timating the return. As shown in Figure 3, this results in different location of the Bellman up- date I'Q compared to the classical Bellman update QQ. Other approaches consider changing the space of representable Q- (-functions Qe (Wang et al., 2016; Osband et al., 2016; Fatemi & Tavakoli, 2022; Ota et al., 2021), attempting to improve the projection of Q on Qe compared to the one for the chosen baseline's neural network architecture. It is important to note that adding a single neuron to one archi- tecture layer can significantly change Qg Wang et al. (2016) show that performance can be increased by including inductive bias into the neural network architecture. This idea can be understood a modilica- tion of Qe, as shown in Figure 3 where the new space of representable Q-function Qe is colored in yellow. Furthermore, algorithms such as Rainbow (Hessel et al., 2018) leverage both ideas.
îQ îQ 2
Figure 3: Other empirical Bellman operators can be represented using another notation I than the classical empirical Bellman operator f. Changing the class of
function approximators Qe results in a new space Qe.
To the best of our knowledge, only a few works consider learning multiple Bellman updates concurrently. Using a HyperNetwork (Ha et al., 2016), Vincent al. (2024) propose to approximate the Q-function pa- rameters resulting from consecutive Bellman updates. While the idea of learning a Hyper seems promising, one limitation is that it is challenging to scale to action-value functions with millions of param- eters (Mnih et al., 2015). Similarly, Schmitt et al. (2022) provide a theoretical study about the learning of multiple Bellman updates concurrently. They consider an off-policy learning setting for linear function approximation and focus on low-dimensional problems. Crucially, their study shows that, in the limit, the concurrent learning of a sequence of consecutive Q- Q-functions converges to the same sequence of Q-functions when learning is carried out sequentially. This finding leads to the following statement: we do not need to wail unlil one Q-function has converged before learning the following one. In this work, we pursue this idea by proposing a novel approach that scales with the number of parameters of the Q-function.
4 Learning multiple Bellman updates
The motivation behind learning multiple Bellman updates arises from well-known result in AVI, which establishes upper bound on the performance loss IIQ* - -QTM||, where πy is the greedy policy of QN, i.e., the last Q-function learned during training (Theorem 3.4 from Farahmand (2011), stated in Appendix A for completeness). The only term of this upper bound that is controlled by the optimization procedure is the sum of approzimation e Emm [ Q where is the distribution of the statc-action pairs in the replay buller. Therefore, minimizing this sum of terms is crucial to oblain low performance losses. Importantly, for 0,8 E O, minimizing 0 + E(s...e) (0)(0)0, s, r')) )) is equivalent to minimizing 0 + T"Qa-Qollă.. under the condition that the dataset of samples D is rich enough to represent the true Bellman operator as proven in Proposition A. of Appendix A. This means that by learning one Bellman update time, classical Q-Network approaches minimize only one approximation error at a time. Therefore, by optimizing for the immediate approximation error, classical Q-Ntwork approaches can be seen as greedy hlgs aiming at minimizing the sum of approximation errors. Unfortunately, this does not guarantee a low sum of approximation errors at the end of the training as greedy algorithms can get trapped in local minima or even diverge with the number of iterations. This is why, in this work, we propose a novel approach that considers several consecutive Bellman updates simultaneously to aim at directly minimizing the sum of
We ignore the factors og weighting the sum approximation errors as they do not play a significant role in the analysis.
4

---

Published in Transactions on Machine Learning Research (02/2025)
rQ,
TQ
rQ
rQ,
rQ,
rQ
TQ
QN window
rQ
QN loss
Q
Windows shift by one Q-function every steps
i-Q window
TQ TQ
IQQ
rQ
rQ
IQ
ON loss
Q2
Figure 4: Left: Graphical representation of i-QN where each online networks Qa learns from its respective target network QK-1- Every G steps, each target network Q& (A: >> 0) is updated to its respective Qs. Right: iQQ considers a window of K Bellman updates as opposed to QN methods that consider only 1 Bellman update. Every T steps, the windows are shifted forward to consider the following Bellman updates.
approximation errors, as illustrated in Figure 2b. We complement this abstract visualization with Figure 10 in Appendix C.1, where we verify the described behavior in practice on simple two-dimensional problem by applying our approach to Fitted-Q iteration (Ernst el al., 2005).
To learn K consecutive Bellman updates, we consider ensemble of K online parameters (0s) Each Q-function Qo is responsible for learning one Bellman update. Each Bellman update is learned from a target network so that the minimization of the TD-error leads to the minimization of the corresponding approximation error as previously discussed. This means that we also consider K target parameters (0+) where each online parameter Og is learned to fit its corresponding Bellman update computed from the target parameters 0x-1- The target parameters are updated in two different ways to ensure that the considered networks are learning consecutive Bellman updates. First, to enforce the structure of a chain, we update each target network 0g (k > 0) to its corresponding online network 0₂ every D steps as Figure (left) illustrates, in which note Q+ Qo, and Qx Qs. for clarity. Then, learning all the Bellman updates is not feasible in practice as it would require storing many Q-functions parameters. Instead, we learn a window composed of K consocutive Bellman updates that is shifted every steps as shown in Figure 4 (right). The window is shifted by updating each larget parameter Og to the following online parameter 0x+1 (see Figure 4 (left)). Classical QN approaches also fit in this representation as they consider a window of size
Unfortunatcly, while i-QN minimizes cach approximation error by considering consecutive Bellman updates, the sum of approximation errors is not necessarily minimized. This is due to the semi-gradient updale rule which creates gaps between each frozen target network Qū, and the online network Qo, they represent shown in Figure 4 (left). This means that even when each approximation error [[Q - Qv 112, decreases by minimizing the corresponding QN loss between each 02 and - i.e. ΣK Q1 Q2, 11 decreases, the sum of approximation errors || Q0, +K [' 20, 12,,v (without the target parameters) does not necessarily decrease.
Nevertheless, in the following, we derive a sufficient condition under which the considered sum of approxi- mation errors is guaranteed to decrease, thus proving i-QN's soundness. For that, we focus the timesteps where the gap between the target networks and the online networks they represent is null. This happens every D timesteps as each target network Qù is updated to its respective online network Qo,- Therefore, we note (= the value of the parameters the tth time the target parameters (0x) are updated their respective online parameters (0 We stress that on those timesteps, the target parameters do not need to be distinguished from the online parameters anymore as they are equal. Proposition 4.1 states that between two timesteps and t 1, the sum of approximation errors is guaranteed to decrease (Equation 5) if the decrease of each approximation error after optimization compared to their previous version before optimization is greater than certain quantity: the displacement of the targets, i.e,, || -I" Ila, (Equation 4). The proof, which rclies mainly on the triangular incquality, is available in Appendix A.
5

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm Iterated Deep Q-Network (i-DQN). Modifications to DQN are marked in purple.
1: Initialize the first target network O, and the K online parameters (0z) and an empty replay buffer D. For k = -,,., K - set 02 + 0g the rest of the target parameters.
2: repeat
Sample kb U(T,.,,K}
Take action a ~ (-groody(Qo, (st, Observe reward ra, next state 8t+1- Update D < DU(ss, ae, de, 8t-1)]
every G steps Sample mini-batch B {(ss a,,,,,r,)) from D.
7:
8:
for k = 1,.., K do Iin paralled]
Compute the loss w.r.t. 0R, LDQN E(GanddeE (r maxe' Qø (s', Qo (s))))
10:
Updale Os from Vn, LDON.
11:
every T' steps
12:
K Update 0g t 6 for k E [1,.,,-T 1}.
Shift the parameters 0g + 0k-1, for E (O,..,K-1
13:
every D steps
14:
Let E N, (0t) be a sequence of parometers of O and be probability distribution over actien pairs. If. for every K], | Qottll2
Proposition 4.1.
Qo - Q0el22.
12,,
(4)
h opprox error before optimisation
oppoppo envor after optémisation
displacement of the target
then, we have
Σ
| ||
Qoc+: 112
[QQ QQ; 12,
(5)
approx after optimization
of approx errors before optimisation
We recall that, when the dataset of samples rich enough to represent the true Bellman operator, minimizing + [T l ( is equivalent to minimizing 0 Σ E(s,a, LQN (/%--,11 8,,,,, Crucially, for E N (1,., K}, each 0++1 is learned to minimize 0 + E ((,,a, LQN (0000-00110000 s,,,,, starting from 6. This means that each gf+1 is learned to minimize 0 + |[ Qo1l... Therefore, the optimization procedure is designed to make Equation 4 valid, i.c., to maximize the left-hand side of the incquality. This is why, argue that Equation 4 is useful to demonstrale that i-QN is theoretically sound. We stress that the presented condition is not meant to be used in practice. Nonetheless, we empirically evaluate the aforementioned quantities in Section 5 to show that the presented condition explains i-QN increased performances.
4.1 Practical implementation
The presented approach can be seen as a general framework to design iterated versions of standard value- based algorithms. Indeed, i-QN is an approach orthogonal the choice of Lov; thus, it can enable multiple simultancous Bellman updates in any algorithm basod on value function estimation. As an example, Algo- rithun showcases an iterated version of DQN (Mnih et al., 2015) that we call ilerated Deep Q-Nelwork (i- DQN). We recall that when K 1, the original algorithm is recovered. Similarly, Algorithm 2 is iterated version of SAC (Haarnoja et al., 2018) that we call iterated Soft Actor-Crütt (i-SAC). In the actor-critic setting, Polyak averaging (Lillicrap et al., 2015) is usually performed instead of hard updates. Therefore, we adapt the way the targets are updated i-SAC, i.e. we update the value of On to T0, + (1 - -)0,, where T E [0, and the enforced chain structure will shift the rest the parameters forward.
The availability of sequence of approximations of consocutive Bellman updates raises the question of how to use them to draw actions in Q-learning setting and how to train the policy in actor-critic setting. We

---

Published in Transactions on Machine Learning Research (02/2023)
K-1
K-4
K-7
K-10
K-20
Car-On-Hill
Sum o0 Mooodoe xeooxmmmm errors
K
4
7
10
20
12
Performance loss
1.2
% of the time the CSAE does not decrease Average decrease of the
17
23
21
2.4
0.3
1.5
2.6
3.9
7.8
1.0
CSAE (x10-*)
% of the time Eq. 5 holds when Eq. holds
100
100
100
100
100
0.8
Proportion of the CSAE's decrease when Eq. 4 holds
100
68
69
66
67
8
15
22
29
36 40
Bellman iteration k
Table 1: Empirical evaluation of the considered sum of approci- mation errors (CSAE) and Equations 4 and on car-on-hil with i-QN applicd FQI for different window sizes K.
Figure 5: Distance between the optimal value function V and T at each Bell- man iteration k for different K.
consider those choices to be similar as they both relate to the behavioral policy. We recall that, in sequential approaches, the single online network is the only possible choice. For iterated Q-Network approaches, we choose to sample a Q-function uniformly from the set of online networks to avoid the caveat passive learning identified by Ostrovski et al. (2021). We empirically justify this choice in Section 6.1.1, where we investigate different sampling strategies.
5 Motivating example
In this section, provide empirical analysis of i-QN on fitted Q-iteration (FQI) in the car-on-hill environment (Ernst et al., 2005) using non-linear function approximation (Riedmiller, 2005). The experiment setup is detailed in Appendix C.2. We study i-FQT's behavior with N = 40 Bellman iterations for varying window sizes, E {1,4,7, 7, 10, 20). We recall that K = corresponds to the regular FQI approach. As the experiment done offline, we impose a budget on the total number of non-parallelizabee gradient steps. This means that we have to shift the windows faster for lower values of K so that each algorithm has learned = Bellman updates the end of the training. On the left y-axis of Figure 5, show that for higher values of K, obtain lower performance losses (in solid lines) after only few Bellman iterations. In dashed lines, we show that continuing the training further improves the performance as the iterated approach keeps learning the last K - projection steps since the considered windows still contain those terms. This motivates the idea learning consecutive Bellman updates simultancously.
We now push this analysis further by computing the key quantities introduced in Section 4. On the right increases. As expected, this is in
y-axis of Figure 5, the sum of the 40 approximation errors decreases accordance with the aforementioned thcoretical results of Theorem 3.4 from Farahmand (2011). The effect of performing multiple Bellman updates is also evident from a visual interpretation of the plots, where can see that increasing the value of K results in shrinking the plots obtained for smaller values of K to the left. This is explained by the ability of i-QN to look ahead of multiple Bellman iterations, thus anticipating the behavior of less far-sighted variants and making better use of the sample. It is important to note that car-on-hill needs approximately 20 FQT iterations to be solved, for which reason i-FQT with K 20 has an evident advantage over the other values of K.
As stated in Section 4, i-QN does not minimize the sum of approximation errors directly due to semi- gradient updates. This can be verified empirically from the first line of Table 1 as the percentage of time the considered sum of approximation errors does not decrease (Equation 5 docs not hold) is positive when K > 1. Nenetheless, it appears that, on average, the considered sum of approximation errors decreases more when increases, as the second line Table 1 indicates. This is coherent with the results shown in Figure 5, as the sum of approximation errors is lower for higher values of K. The third line of Table demonstrates that when the proposed condition of Proposition 4.1 is verified (Equation 4 holds), the considered sum of approximation errors always decreases (Equation 5 holds). This confirms that this condition is sufficient..
7

---

Published in Transactions on Machine Learning Research (02/2025)
Finally, the last line of the table reports the proportion of the decrease in the considered sum of approximation errors that happens when the condition of Proposition 4.1 is verified (Equation 4 holds) compared to when it is not verified. The proposed condition is responsible for R 70% of the decrease of the considered sum of approximation errors. This shows that the proposed condition is relevant to explain i-QN's ability to minimize the sum of approximation errors for the considered example. For K = 1, this metric at 100%, which is expected since the proposed condition equivalent to a decrease the considered sum of approximation errors. Indeed, when K 1, the displacement to the target (the left term of Equation 4) is null since 0g. the only target parameter, is constant in t.
6 Experiments
We evaluate our proposed i-QN approach on deep value-based and actor-critic settings. As recommended by Agarwal et al. (2021), we choose the interquartile mcan (IQM) of the human normalized score to report the results of our experiments with shaded regions showing pointwise 95% percentile stratified bootstrap confidence intervals. IQM is a trade-off between the mean and the median where the tail of the score distribution is removed on both sides to consider only 50% of the runs. seeds are used for each Atari game, and 10 seeds are used for each MuJoCo environment.
6.1 Atari 2600
We evaluate the iterated version of DQN and implicit quantile network (IQN) (Dabney et al., 2018) on 20 Atari 2600 games. Many implementations of Atari environments along with classical baselines are avail- able (Castro et al., 2018; D'Eramo et al., 2021; Raffin et al., 2021; Huang et al., 2022). We choose to mimic the implementation choices made in Dopamine RI. (Castro et al., 2018) since it the only one to release the evaluation metric for the baselines that we consider and the only one use the evaluation metric rec- ommended by Machado et al. (2018). Namely, we use the game over signal to terminate an episode instcad of the life signal. The input given to the neural nelwork is a concatenation of 4 frames in grayscale of dimension 84 by 84. To get a new frame, we sample 4 frames from the Gym environment (Brockman et al., 2016) configured with no frame-skip, and we apply a max pooling operation on the 2 last grayscale frames. We use sticky actions to make the environment stochastic (with = 0.25). The performance is the one obtained during training. By choosing an identical setting as Castro et al. (2018), we leverage the baselines' training performance reported in Dopamine RL. To ensure that the comparison is fair, we compared our version of DQN and IQN to their version and verified their equivalence (see Figures 12 in Appendix C.3).
Hyperparameter settings. We use the same hyperparameters of the baselines, except for the target update frequency, which we set 25% lower than the target update frequency of the basclines (6000 compared to 8000). This choice is made to benefit from i-QN's ability to learn each projection step with more gradient steps and samples while shifting the window faster than sequential approaches. We stress that i-QN has access to the same number of mon-paralle gradient steps and the same number of samples. We choose D= and we let the Q-functions share the convolutional layers. We present the architecture of i-QN in Figure 11 in Appendix C.3. Further details can be found in Table 3 in Appendix C.3. To ensure that our implementation is trustworthy, Figure 12 in Appendix C.3 shows that the training performances of DQN and IQN are comparable to the ones of i-DQN and i-IQN with K = 1, as expected. Finally, we note T the target update frequency and G the number of environment interactions per gradient steps.
Atari results. i-DQN with K 5 outperforms DQN on the aggregation metric, as shown in Figure 6 (left), both in terms of sample-cfficiency and overall performance. In Figure 15 (left) in Appendix E, the performance profile, showing the distribution of final scores, illustrates that i-DQN statistically dominates DQN on most of the scores. Moreover, i-DQN improves over DQN's optimality gap during training as presented in Figure 15 (right). Notably, the iterated version of IQN with K = 3 greatly outperforms the sequential approach. All the individual training curves for the 20 Atari games are available in Figure 14 in Appendix F..
8

---

Published in Transactions on Machine Learning Research (02/2025)
i-DQN DQN
I-IQN
I-DQN K-4, G-4
DQN, G-1
i-DON K-4, T-2000
IQN
DQN, G=4
DQN. T-8000
Atari
Asterix
Asterix
1QM Human Norm Score
1QM Human Norm Score
1QM Human Norm Score
2
2
2
0
0
100
200
100
200
100
200
Num Frames (in millions)
Num Frames (in millions)
Num Frames (in millions)
i-DQN greatly outperforms DQN with G = 1, which is a variant of vanilla DQN (G = 4) that has access to the same amount of gradient steps as i-DQN. This shows that simply increasing the number of gradient steps of DQN is not effective. Right: i-DQN with K = 4, T = 2000 parallelizes the training of DQN T = 8000 successfully since it yields similar performances after 3125 projection steps (in orange). This parallelization saves samples and gradient steps which can be used later to outperform DQN.
Figure 6: Left: i-DQN and i-1QN outperform their respective sequential approach. Middle:
6.1.1 Ablation studies
We propose various ablation studics on different Atari games to help understand i-QN's behavior.
Gradient steps executed in parallel. The iterated approach performs K times more gradient steps than the sequential one as it considers K online networks instead of 1. Crucially, the batch of samples is shared between the networks, which means that the iterated approach observes the same amount of samples during training. Additionally, those additional gradient steps are executed in parallel so that the training time remains controllable when enough parallel processing power is available. We refer to Appendix D for further discussions. Nonetheless, we compare i-DQN with K = (i-DQN, K = 4, = 4) to a version of DQN which has access to the same number of gradient steps (DQN, G = 1). We also set the target update frequency of i-DQN T to be the same DQN, i.e., 8000 that the windows shift at equal speed. We report the performances in Figure 6 (middle) and add vanilla version of DQN (DQN, G = 4) for completeness. DQN with G = leads to overfitting while having training time 89% longer than i-DQN. This result shows that simply allowing DQN to perform more gradient steps is not beneficial.
Learning consccutive projection steps simultancously. In Section 4, we argue that we can effectively learn multiple Bellman updales simultaneously. We also provide a condition under which this idea is guaran- teed to be beneficial. We now evaluate version of i-DQN with K 4, and with target update frequency set at fourth of the one of DQN (T 2000 instead of 8000). With this setting, cach projection step is learned the same number of samples and gradient steps with DQN and i-DQN. We report the results in Figure 6 (right) and mark in orange the performance after 3125 (100 x 250000/8000 projection steps and in red the performance after 6250 projection steps. i-DQN requires times fewer sam- ples and gradient steps to reach those marked points compared to DQN since i-DQN's window shifts 4 times faster. Interestingly, the performances of i-DQN and DQN after 3125 projection steps are similar, showing that waiting for one projection step finish to learn the next one is not necessary. After the 6250 projection steps, the performance of i-DQN is slightly below the one of DQN. We believe that this comes from the fact that, at this point of the training, DQN has access to 4 times more environment interactions than i-DQN. Nevertheless, by learning the 3 following projection steps while the first one is being learned, i-DQN is more efficient than DQN.
25 250000 22000002000 = 3125)
Comparing different window sizes. In Figure 7 (left), we report the performances for K 1,5 and 10. Notably, higher performance achieved with higher window sizes. In the main experiment presented in Figure& 6 (left), we choose K 5 because provides a good trade-off between sample efficiency and additional training time. See Section D for more details about training time.
9

---

Published in Transactions on Machine Learning Research (02/2025)
i.QNN K-5
iDDN K-5 3-step
DQN (i-DQN K-1)
i-DQN Asteroids
i-DQN K-10
DQN S.step
DQN 3-slep
Asterix
Qbert
Atari
10M Human Nora Score
0.015
1.0
IQM Human Norm Score
1-
0.010
0.5
0.005
2
0.000
0.0
100
200
100
200
0
100
200
Num Frames (in millions)
Num millions)
millions)
100
200
Num Frames (in miltions)
Ablation study on the number of Bellman updates K taken into account in the loss. Greater performances are reached for greater values of K in Asleroids and Aslerix. In Qbert, iDQN with K = 10 might allow the agent to overfit the targets since each Bellman update is learned with 10 times more gradient steps. We recall that DQN is equivalent to iDQN with K = 1. Right: i-DQN with K = 5 performs differently than DQN + 5-step return on 4 randomly selected games (Breakout, DemonAttack,
Figure 7: Left:
ChopperCommante and Krull). DQN + 3-step return can be improved by simultaneously learning 5 Bellman updates.
uniform sampling
first online Q sampling
last online Q sampling
i.DQN w/ target nets
DDQ wia turget nets
10M Etunann Norm Soon
Asterix
Asteroids
1.0
Obert
Hisman Norm Soor
Asterix
7.5
Breakout
2-
0.000
5.0-
n.ons-
2.5
100
200
0.000
0.0
0
100
200
100
200
0.0
100
200
Nom Frames (in millions)
Num Prames millions)
Num Frames (in millions)
Num Frames millions)
Nurn Frmmas (in millions)
Figure 8: Left: Ablation study on how actions are sampled when i-DQN interacts with the environment. Right: Ablations Asterix and Breakout indicate improved performance with target networks.
Iterated Q-network vs. n-step return. We point out that i-QN is not equivalent to n-step return (Watkins, 1989). While our approach aims at learning multiple consecutive projection steps simultaneously, n-step return applies Bellman updates considering sequence of consecutive rewards and the bootstrapped target at the th step. In other words, n-step return derives another empirical Bellman operator as it computes the target as interpolation between one-step bootstrapping and Monto-Carlo estimate. Nonetheless, we compare i-DQN with K = 5 to DQN with 5-step return in Figure 7 (right). DQN with 5-step return behaves substantially worse than i-DQN with K = 5 on 4 randomly selected Atari games. The individual training curves are shown in Figure 16 of Appendix E.
One parallel can be done to link the two approaches in the APE setting. Indeed, using n-step return can be seen as applying the empirical Bellman operator n times. With n 2, (f'))²((s,a) = To + f a(1))) To + (r) + ( (s2))) = ro + 7r, + Q (s2)). Thus, combining the idea of learning K Bellman updates with n-step return artificially brings the length of the window to x K where K consecutive projections of the Bellman operator applied n times are learned. This fact is also discussed in Section 5.3 of Schmitt et al. (2022). It is worth noticing that this is not the case for AVI since the max operator is not linear. With n = 2, (f*)Q(ss,) a) = ry + maxa, f"*((1, a1) = To + y maxa, (r1 + 7 max,, Q(82, az)) )) * To + 7r1 + 72 max,, Q(x2, mz). Interestingly, n-step return can be seamlessly used with i-QN. In Figure 7, we show that the iterated version of DQN with 3-step return outperforms its sequential counterpart.
Action sampling. As explained in Section 4, the availability of several online Q-functions corresponding to K consecutive Bellman updates raises the question of which network to use for the sampling actions. Informally, we can argue that along the chain of Q-functions in i-DQN, on the one hand, the first network is the one with the best estimate of its respective Bellman update as it has seen the most samples, but it is the most distant from the optimal Q-function. On the other hand, the last network is the less accurate but the most far-sighted one. This creates a trado-off between accuracy and distance from the optimal Q-function, that we face by sampling network uniformly each step. This also helps mitigate passive learning rOstrovski et al., 2021) each network can interact with the environment. In Figure 8 (left), we
10

---

Published in Transactions on Machine Learning Research (02/2025)
compare the performance of our uniform sampling strategy to variant of i-DQN where only the first and last. network sample actions. The uniform sampling strategy shows a slight superiority over the others.
Relevance of the target networks. We evaluate a version of i-DQN without target networks. In this version, each online parameter 0g learned from the previous online parameter of the chain 8x-1- In Figure (right), we compare this version of i-DQN to the vanilla version of i-DQN where each target parameter 0z updated to its respective online parameter 0g every D gradient steps. Despite having fewer memory requirements, the version of i-DQN without target networks obtains worse results and even drops performance on Asterir. This supports the idea that target networks are useful for stabilizing the training.
In Appendix E, we present additional ablation studies about the choice of sharing the convolutional layers, the distance between the networks during training, and a comparison between i-DQN, a version of DQN with the same total number of parameters as i-DQ,, a comparison between i-QN and Bootstrapped DQN (Osband et al., 2016), and we discuss the link between i-QN and Value Iteration Networks (Tamar et al., 2016).
6.2 MuJoCo continuous control
We evaluate our proposed iterated approach over two actor-critic algorithms on 6 different MuJoCo environments. We build i-SAC on top of SAC and i-DroQ on top of DroQ (Hiraoka el al., 2022). We K 4, meaning that the iterated versions consider Bellman updates in the loss in- stead of 1. Therefore, set the soft target update rate to 4 times higher than the soquential
baseline algorithms (r 0.02 compared to 0.005) while keeping all other hyperparameters identical. Figure 9 shows the training performances of i-SAC and i-DroQ against their respective baselines. The IQM return normalized by the final performance of SAC. Both iterated versions outperform their respective sequential counterparts. While i-DroQ dominates DroQ until the end of the training at million environment interactions, i-SAC finally con-
SAC
i-SAC DroQ --- i-DroQ
MuJoCo
IQM SQC Norm retur
1.00
0.75
verges to the same performances as SAC. Per en-
SAC
vironment plots are available in Figure 21 of Ap-
0.50
pendix F. In Figures 20 and 22 of Appendix F, we
verify that the gain in performances of i-SAC and
0.25
i-DroQ over SAC and DroQ is not due to differ- ence in the soft larget update rate т. For that, make the soft target update rate of SAC and DroQ match the one of i-SAC and i-DroQ. The iterated ap- proaches still outperform the sequential approaches, validating that learning multiple Bellman updates in parallel is beneficial. Table 4 gathers all the hy- perparameters used in this section.
0.0
0.5
1.0 le6
Env Steps
Figure 9: Training curves of i-SAC and i-DroQ (K 4) along with SAC and DroQ. The iterated versions outperform their sequential counterparts. The curves are normalized by the final performance of SAC.
7 Discussion and conclusion
We have presented iterated Q-Network (i-QN), a new approach that considers multiple consecutive projection steps the loss overcome the limitations of the one-step Bellman updates. We have theoretically and empirically analyzed its benefit across several problems when applied to value-based and actor-critic mothods. Remarkably, HIQN provides 13% improvement over IQN final performances aggregated over 20 Atari games. Future works could investigate the idea of learning several iterations in parallel for other sequential RL algorithms. For example, trust-region methods are sequential algorithms where data collection and policy optimization alternate sequentially.
Limitations. While i-QN is sample efficient, it requires additional training time and a larger memory requirement than the soquential approach. We quantify this limitation in Appendix D. Importantly, if enough parallel processing power is available, i-QN's training time becomes similar the one of regular Q-Network as the additional gradient steps used by iQQN parallelizable.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgments
This work was funded by the German Federal Ministry of Education and Rescarch (BMBF) (Project: 011S22078). This work was also funded by Hessian.ai through the project 'The Third Wave of Artificial Intelligence 3AI by the Ministry for Science and Arts of the state of Hessen and by the grant "Einrich- fung eines Labors des Deutschen Forschungszentrum für Künstliche Intelligenz (DFKI) an der Technischen Universität Darmstadt".
Carbon Impact
As recommended by Lannelongue & Inouye (2023), we used GroenAlgorithms (Lannelongue et al., 2021) and ML CO2 Impact (Lacoste et al., 2019) to compute the carbon emission related to the production of the electricity used for the computations of our experiments. We only consider the energy used to generate the figures presented in this work and ignore the energy used for preliminary studies. The estimations vary between 1.39 and 1.56 tonnes of CO2 equivalent. As a reminder, the Intergovernmental Panel on Climate Change advocates a carbon budget of 2 tonnes of CO2 equivalent per year per person.
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline rein- forcement learning. In International Conference on Machine Learning, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Dimitri Bertsekas. 2015.
Dynamic programming and optimal control 4th edition, volume II. Athena Scientifie,
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: compos- able transformations of Python-NumPPy programs, 2018.
Steven Bradtke. Reinforcement learning applied to linear quadratic regulation. In Advances in Newral Information Processing Systems, 1992.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech 'Zarebba. Openai gym, 2016.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv, 2018.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, 2018.
Peter Dayan and CJCH Watkins. Q-learning. Machine Learning, 1992.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo E'walds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 2021.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 2005.
Amir-massoud Farahmand. Regularization in reinforeement learning. PhD thesis, University of Alberta, 2011.
Mehdi Fatemi and Arash Tavakoli. Orchestrated value mapping for reinforcement learning. arXiv, 2022.
Mattie Fellows, Kristian Hartikainen, and Shimon Whiteson. Bayesian bellman operators. In Advances in Newral Information Processing Systems, 2021.
Niklas Funk, Georgia Chalvatzaki, Boris Belousov, and Jan Peters. Learn2assemble with structured repre- sentations and search for robotic architectural construction. In Conference on Robot Learning, 2022.
David Ha, Andrew Dai, and Quoe Lc. Hypernetworks. In International Conference on Learning Represen- lations, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
Hado Hasselt. Double q-learning. In Adoances in Neurol Information Processing Systems, 2010.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference Arlificial Intelligence, 2018.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Repre- sentations, 2022.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file file implementations deep reinforcement learning algorithms. Journal of Machine Learning Research, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint NTi:19100007000 2019.
Loie Lannelongue and Michael Inouye. Carbon footprint estimation for computational research. Nature Reviews Methods Primers, 2023.
Loic Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: quantifying the carbon footprint of computation. Advanced Science, 2021.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv, 2015.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowl- ing. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep rein- forcement learning. Advances in Neural Information Processing Systems, 2021.
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiu, 2021.
Martin Puterman. Markov decision processes. Handbooks in operations research and management science, 1990.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 2021.
Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, 2005.
Simon Schmitt, John Shawe- Taylor, and Hado Van Hasselt. Chaining value functions for off-policy learning. In AAAI Conference on Artificial Intelligence, 2022.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 1988.
Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT Press, 1998.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. Advances Neural Information Processing Systems, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In International Conference on Intelligent Robots and Systems, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. Tn AAAI Conference on Artificial Intelligence, 2016.
Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters, Marcello Restelli, and Carlo D'Eramo. Paramcterized projected bellman operator. In AAAI Conference on Artificial Intelligence, 2024.
Théo Vincent, Fabian Wahren, Jan Peters, Boris Belousov, and Carlo D'Eramo. Adaptive g-network: On- the-fly target selection for deep reinforcement learning. International Conference on Learning Repre- sentations, 2025.
Ziyu Wang, Tom Schaul, Matleo Hessel, Hado Hasselt, Marc Lanclot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, 2016.
Christopher Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
14

---

Published in Transactions on Machine Learning Research (02/2025)
A Proofs
Theorem 3.4 from Farahmand (2011). Let N No, and p, two probability measures on S X A. Por any sequence (Qx) 2 where Rg depends on the reward function and the discount factor, we have
- - 11..p S CNA.R + inf F(r F(r:N,p,)) Ex= a Q - Qu1lē, rE(o.]]
where CN..R,, F(r:N,p,n) and (aw)k do not depend on (Qx) uy a greedy policy computed from QN.
Proposition 4.1. Let E N, (0%) be a sequence of parometers of θ and v be probability distribution over stale-action pairs. If, for every & E {1,.,,, K},
IT 2
| [
*Qot: r -o 12,
(4)
k approx error before optimization
khapppoo enror after optimization
displacement of the target
then, we have
Q 0-1
Q
T Q0L Qo: 112.
(5)
swm of opprex errors ofter optimisation
sum of approx ervors before optimisation
Proof. For N, let ( be sequence of parameters of θ and be probability distribution over state-action pairs. We assume that for every k E {1,.., KJ condition Equation 4 holds.
Now, we show that for every k E K), 1| k Q 1a, S || 1 Qog 11a... From there, Equation can be obtained by applying the square function to both sides of the inequality and summing over k.
Let h E (1,.,,K) KJ. To show that || Qott Q8-1122 S S1r -Qoz 11a,s start with the left side of
the incquality
| Q0-1112,0
|| r'Q0;_, r= -Qoet1 Qoc+1112. 12,2
[1 1 1000; 10; [1"' -Qost
Q2 Qot 11
The second last inequation comes from the triangular inequality, and the last inequation comes from the assumption that Equation holds.
Proposition A.1. Let (0,0) be a pair of parameters of θ and D = {(s, r',')} be set of samples. Let v the distribution represented by the state-action pairs present in D. We note Dz.a {(r, (s,,,,, E DJ, ((s,a) E D. If for every state-action pair (s, E D, (r,yy.. [fQg(s;)) =rQp(,aa) then,
Σ Lox(010, -aarrss)==M[[C0CC (DDD.
M - Qo1la, constant w.r.t. 0
(6)
where M. a constant w.r.t. Thus, minimizing 0 + Elaansrr Lox(0 a, 8²) equivalent to minimizing + [[ Qo11
15

---

Published in Transactions on Machine Learning Research (02/2025)
Proof. The proof is inspired from Vincent et al. (2025). Let (8,0) be pair of parameters of @ and D ((s,a, a,,,,))) be set of samples. Let be the distribution of the state-action pairs present in D. In this proof, we note I' as r'r to stress its dependency on the reward and the next state s'. For every state-action pair (s, in D, we define the set s. {(r, as,, E D) and assume that (r,,"))D,,.. [ r Qp(s, a)] TQp(s, a). Additionally, we note the cardinality of D, Ms,a the cardinality Da,a and D the set unique stale-action pairs D. We write
Σ (,0,000
Σ
(fre (o(s,)() Qo(s,a))*
(..))
Σ
Qono - rQp + (Qa(s,a) +)g((s)))-((sss.. Qo(s,a))
(sar))))
Σ
(ol,))(s,a) - -(Qp(sa))
(s.0.))))
+
Σ
(TQg(s,a)-Qu - a)(s,a))
(s,),),)))
+2
Σ
(QQlta) (TQa(s,)) - QQ(ssa))
(a.0)n)))DD
The second last equation is obtained by introducing the term [Q a) and removing it. The last equation
is obtained by developing the previous squared term. Now, we study each of the three terms:
((sDnneED (QQ(taa))) is independent of EGeande (TQa(s, a)( equal to M |[[ Qo112, by definition of
Σ ().)))))))
(T)g(s,)--Qoce - Q)(s,)))
Σ
Σ (fr= -(0((,,)) (TQp(s, Qo(s,a)) - =0
(B,a)ED
since, for every (s, E D.
Σ (ao[o(s,a--(((( Mai (Firo)) Qo((s)) -(Qg(s,))))
the last equality holds from the assumption.
Thus, we have
Σ (, = M x |[ 12, + constant w.r.t 0
(EDD
This is why minimizing Elsanshe CQN 8, is equivalent. to minimizing 0 |T 0
Qo 2ol2...
16

---

Published in Transactions on Machine Learning Research (02/2025)
B
Pseudocodes
Algorithm Iterated Soft Actor-Critic (i-SAC). Modifications to SAC are marked in purple.
1:
Initialize the policy parameters á, 2 x (K 1) parameters (10) x)) and an emply replay buffer D.
2: repeat
3: Take action ~ T=(-)s;;; Observe reward Tt, next state st41i D t DU[(sE de, Te, St+1))
4: for UTD updates do Sample a mini-batch B {(s,a, r, 6,)) from D.
for k L,.,, K; 2,2 do (in parallel] Compute the loss w.r.t. 0g
min Ooz oo (6',') - llo loggo(d((5)) -Qoz) )² where a' ~ =T(-1s)). (E(112)
LSAC= = Σ (r+ (s,a.),)))B
8:
Update of from Vog LSAC-
Update of =0f + (1 7)0 for i E {1,2}.
10:
Sample K UfI,.., K. Compute the policy loss w.r.t o, Aator = (1,2) Qo 20 (s,a) a Ta(als), where TT(-s))
11:
12:
Update & from VaCoer
C Experiments details
We used the optimizer Adam (Kingma & Ba, 2015) for all experiments. We used JAX (Bradbury et al., 2018) as a deep learning framework. The neural nerworks used by the i-QN approaches in the experiments presented in Section 6 are initialized following the same procedure as the neural networks used for their respective sequential counterparts.
C.1 Behavior of Q-network and iterated Q-network on a Linear Quadratic Regulator
2
Qo
Qo
-0.2
-0.2
Q1
Q2
-0.4
-0.4
loss
105s
6
loss
r"Qo
r" Q0Q
Q
Q
Q"
Q1
-1.0
-1.0
-1.0
-0.9
0.8
-.7
-0.6
-0.5
-0.
-0.3
-1.0
-0.9
-0.8
00.
-06
-0.5
-0.4
-0.3
Figure 10: Graphical representation of QN (left) and i-QN (right) in the space of Q-functions Q for the LQR experiment.
Figures 2a and 2b are schematically representing the behavior of QN and i-QN approaches in the space of Q- functions. Figure 10 shows that those representations are accurate for an offline problem: Linear Quadratic Regulator (Bradtke, 1992). In this problem, the state and action spaces are continuous and one-dimensional. The dynamics are linear: for a state and an action a, the next state is given by s' 0.8s 0.9a, and the reward quadratic r(s,a) a) 0.5s² + 0.4sa 0.5a². We choose to parametrize the space of Q-functions with
17

---

Published in Transactions on Machine Learning Research (02/2025)
2 parameters (M,G) such that, for state and an action a, Q(s,a) = Ma² Gs?. To reduce the space of representable Q-finctions, constrain the parameter M to be negative and the parameter G to be between -0.4 and 0.4. Starting from some initial parameters, we perform 30 gradient steps with a learning rate of 0.05 using the loss of QN and i-QN. Both figures show the space representable Q-functions Qe in green, the optimal Q-function Q", the initial Q-function Qo and its optimal Bellman update N* The projection of the optimal Bellman update is also shown with a dotted line. As we claim the main paper, i-QN manages to find Q-function Q2 closer to the optimal Q-function Q' than Qi found by QN. Figure 10 (left) closely resembles Figure 2a. Likewise, Figure 10 (right) looks like Figure 2b, showing that the high-level ideas presented in the paper are actually happening in practice.
C.2 Experiments the car-on-hill environment
Experimental setting. The dataset contains 50.000 samples collected with uniform policy from the initial state [-0.5, 0J. We use neural network with one hidden layer of 50 neurons. The batch size is 100 and we set D = 1. Each i-FQI run has access to 20.000 gradient steps. The results are average over 20 seeds. In Figure 5, we report the performance loss I|Q - p0xxlla.. where QUN is the action-value function associated with the greedy policy of QN, which can be associated with the last network Iearned during training. P is usually associated to the slate-action distributtio on which we would like the chosen policy lo perform well. Therefore, we choose p as a uniform distribution over the state space and action space on a 17 x 17 grid suggested by Ernst et al. (2005). In Figure 5, compute the sum of approximation errors Eee Q - Q& 1 where is the state-action distribution encountered during training, i.e., the data stored in the replay buffer.
C.3 Experiments on the Atari games
Atari game selection. The Atari benchmark is a highly compute-intensive benchmark. Depending on the hardware and the codebase, one seed for one Atari game can take between 1 day to days to run DQN GPU. This is why we could not afford to the experiments on the 57 games with 5 seeds. The 20 games were chosen before doing the experiments and never changed afterward. They were chosen such that the basclines and Rainbow have almost the same aggregated final scores for the 20 chosen games as the aggregated final scores shared by Dopamine RL as shown in Table 2. The Atari games used for the ablation studies were randomly selected. We stress that the purpose of those ablations is to highlight i-QN's behavior in the presented games, is not meant to draw some conclusions on the general performance of i-QN over the entire benchmark.
Table 2:
The TQM Human normalized final scores of the baselines and Rainbow aggregated the 55 available Atari games are similar to the ones aggregated over the 20 chosen games.
DQN
Rainbow
IQN
Final score over the 55 Atari games available in Dopamine RL
1.29
1.71
1.76
Final score the 20 chosen Atari games
1.29
1.72
1.85
18

---

Published in Transactions on Machine Learning Research (02/2025)
îQ.
TQ
TQ T
4
4
K
:
Head
Head
Head k
Head
K
convs
convs
DQN
i-DQN
Losses and neural networks architectures of DQN and i-DQN. The dotted lines link the outputs of the neural networks to the objects they represent. The flash signs stress how each projection step is being learned, where f is the empirical Bellman operator. The target networks are represented in blue while the online networks are in purple. For i-DQN, the convolution layers of Qo are stored separately since it is the only target Q-uuctioon that is not updated every D steps.
Figure 11:
DQN (our implementation)
i-DON K-1
DQN- + 3-step (our implementation)
DQN (dopamine)
DQN + 3-step (dopamine)
Breakout
DemonAttack
10M Human Norm Score
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
100
200
0
100
200
Num Frames (in millions)
Num Frames (in millions)
IQN wio 3-step (our implementation)
i-ION w/o 3-step
IQN four implementation)
IQN wlo 3-stop (dopamins)
IQN (dopamine)
KungFuuMsttrr
KungFuMaster uMaster
1QM Human Norm Score
1.5
1.0
1.0
0.5
0.5
0.0
0.0
0
100
200
0
100
200
Num Frames (in millions)
Num Frames (in millions)
Figure 12: Left: Our implementations of DQN (top) and IQN (bottom) yield similar performance as the implementation of Dopamine RL. This certifies that we can compare the results relcased in Dopamine RL with our method. AL the top, DQN and i-DQN with K = 1 have similar behavior. This certifies the trustworthiness of our code base. The same applies to IQN and i-IQN with K = 1 at the bottom. Right: We draw similar conclusions when adding a 3-step return.
19

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Summary of all hyperparameters used for the Atari experiments. We note Conva 2D con- volutional layer with C' filters of size axb and of stride d, and FC E fully connected layer with E ncurons.
Environment
Discount factor
0.99
Table 4: Summary of all hyperparameters used for the MuJoCo experiments. We note FC E a fully con- nected layer with E neurons.
Horizon H
27 000
Full action space
No
Reward clipping
clip( 1)
Environment
All algorithms
Discount factor
0.99
Number of epochs
200
Horizon H
1000
Number of training
250
All algorithms
steps per epochs Type of the replay buffer D Initial number of samples in D Maximum number of samples in D
Number of training steps
1 1.000 000
FIFO
Type of the replay buffer Initial number of samples in D Maximum number of samples in D
FIFO
20.000
000
1 1000 000
Gradient step
1 10000000
4
frequency G
1 for SAC/i-SAC
Starting e
UTD
20 for DroQ/i-DroQ
Ending
10
Batch size
256
linear decay
250 000
Learning rate
10
duration
Adam A1
0.9
Batch size
32
Policy delay
Learning rate
6.25 x 10 3
Actor and critic architecture
FC 256
Adam e
1.5 x 10 4
-FC 256
Couvg, 3,2
SAC & DroQ
Torso architecture
-Covv
Soft target update frequency 7
Conv
5 x 10-3
FFC 512
Head architecture
-FC nA
i-SAC & i-DroQ
Activations
ReLU
Soft target update
2 x 10
frequency
DQN IQN
D
Target update frequency T
8000
iDQQ i-IQN
Target update
600
frequency
30
20

---

Published in Transactions on Machine Learning Research (02/2025)
D Training time and memory requirements
In Figure 13, we report the performances presented in the main experiments of the paper (Figure 6 (left) and Figure 9) with the training time as the 2-axis instead the number of environment interactions. Computations are made on an NVIDIA GeForce RTX 4090 Ti. For iDQN and iIQN, the game Breakout is used compute the training time. For iSAC and iDroQ, the training time is averaged over the considered enviromments. The iterated approach requires longer training time compared the sequential approach. Notably, 1-QN reaches the final performances of the sequential approach faster the first line of Table 5 reports. Indeed, i-IQN reaches TQN's final performance 18h23m before TQN finished its training. Nenetheless, westress that the main focus of this work is on sample efficiency, as it remains the main bottleneck in practical scenarios. We recall that DroQ and i-DroQ use UTD of 20, which slows down the training significantly. We also report the additional GPU vRAM usage of i-QN compared to QN on the second line of Table The additional memory requirements remain reasonable. Finally, while the number of additional FLOPs increases linearly with K for gradients update, sampling an action from an i-QN agent does not take more FLOPs than sampling action from QN agent each Q-network has the same architecture (see Table 5).
i-DQN DQN
i-IQN
i-SAC
i-DroQ
IQN
SAC
DroQ
Atari
MuJoCo
IQM Human Norm Score
IQM SAC Norm return
2
1.00
0.75
0.50
0.25
0
10
21
31
41
2
4
Clock time (in hours)
Clock time (in hours)
Figure 13: i-QN and QN performances according to the training time on Atari games (left) and MuJoCo environments (right). i-QN requires more training time than QN approaches. Nonetheless, i-QN reaches QN's final performances faster.
Difference between the time i-QN took to reach QN's final performance and QN's training time measure of time efficiency. Despite taking longer to train, i-QN is always more time- efficient than QN's approaches for the four considered instances. Second line: The additional GPU vRAM usage of i-QN remains reasonable. Third line: The additional FLOPs for gradient update scale with K. Fourth line: There are no additional FLOPPs for sampling an action as each Q-network has the same architecture.
Table 5: First line:
i-DQN vs DQN for =5
i-IQN vs IQN for K =
i-SAC vs SAC i-DroQ vs DroQ for =4
for = 4
Time saved by i-QN to reach QN's final performance
7h31m
18h23m
6m
1h55m
Additional GPU vRAM usage
+0.3 Gb
+0.9 Gb
+0.1 Gb
+0.4 Gb
Additional FLOPs for gradient
x5.6
x2.2
x4.4
x4.6
update on a batch of samples
Additional FLOPs for sampling an action
x1
x1
x1
x1
21

---

Published in Transactions on Machine Learning Research (02/2025)
E
Training curves for Atari games
i-DQN DQN
i-TQN
IQN
Assault
BankHeist
Berzerk
1QM Human Norm Score
Alien
10
1.5
0.75
0.2
0.50
5
1.0
0.25
0.5
0.1
0.00
01
0.0
10M Human Norm Score
Breakout
Centipede
2
ChopperCommant
15
DemonAttack
7.5
0.2
10
5.0
2.5
0.0-
0.0
Enduro
Frostbite
Gopher
Gravitar
1QM Human Norm Score
5
0.5
IceHockey
Jamesbond
Krull
KungFuMann
1QM Humen Nom Score
15
7.5
1.0
10
5.0
0.5
5-
0.5
2.5
0.0
01
0.0
10M 1QMHummn Human Nom Score
1.0
Riverraid
Seaquest
Skiing
10-
StarGunner
0.5
0.5
0.0
0.0
100
200
100
200
0
100
200
0
100 200
Num Frames (in millions)
Num Frames (in millions)
Num (in millions)
Num Frumes millions)
Figure 14: Training curves of i-DQN with K 5 and i-IQN with K = 3 along with DQN, Rainbow and i-QN on 20 Atari games. As reminder, IQN is also using 3-step return. In most games, the iterated approach outperforms its respective sequential approach.
22

---

Published in Transactions on Machine Learning Research (02/2025)
DQN
IDQN
IQN
1QQ
Atari
Atari
1.0
Fraction of with score
Optimality Gap
0.75
0.5
0.50
0.25
5.0
7.5
100
200
Human Norm Score (C)
Num Frames (in millions)
Figure 15: Left: Performance profile. The figure shows the fraction of runs with a higher final score than a certain threshold given by the x-axis. The iterated versions, iDDQN and i-IQN, statistically dominate their respective sequential approach on most of the domain. Right: Optimality gap. The figure shows the average distance the human level of the runs that do not reach super-human performances. i-DQN and i-IQN achieve lower optimality gap, meaning that those algorithms also improve on games where human performance is not reached by their sequential counterparts.
i-DQN K=5
i-DQN K-5 3-step
DQN + 5-step
DQN 3-step
Breakout
DemonAttack
ChopperCommand
Krull
[OM Humon Norn Score
7.5
1.5
10
5.0
1.0
2.5
0.5
0.0
100
200
100
0 Num Frames (in millions)
100
200
Num Frames (in millions)
Num Frumes (in millions)
Num Frumes (in millions)
Figure 16: i-DQN with K 5 behaves differently than DQN + 5-step return. DQN + 3-step return is
boosted when learning K Bellman updates simultancously (i-DQN K = + 3-step).
Shared architecture vs. Independent networks. In Figure 17, show the performance of i-DQN when the convolution layers are shared and when the neural networks are independent from each other on 2 Atari games. In ChopperCommand, having fully independent networks seems more beneficial than sharing the convolutional layers. We conjecture that this due to the fact that the Bellman updates are far away from each other, hence the difficulty of representing consecutive Bellman updates with shared ionaal
İ-QNK-5 DQN
DQN with as many parameters as i-DQN
I-DQN (shared networks) ChopperCommand
+DDQ (indspendent netwrrs)) CrazyClimber
10M Human Norm Score
1.0
IQM Human Norm Score
Asterix
Breakout
0.5
5.0
0.0
Num Frames (in millions)
Num Frames (in millions)
0
100
200
0
100
200
Num Frames (in millions)
Num Frames (in millions)
Figure 17: Left: Independent networks can perform better than networks with shared convolutions. Right: Inflating the number of parameters of DQN until it reaches the same number of parameters i-DQN with K 5 uses does not. lead to performances high i-DQN on the considered games. Notably, while iDQQ uses the same architecture as DQN, the inflated version of DQN uses larger architecture.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Asterix
Asteroids
BankHeist
0.05
1QM inter-head std
0.100
0.06
0.04
0.075
0.04
0.03
0.050
0.02
0.02
0
100
200
0
100
200
100
200
Num Frames (in millions)
Num Frames (in millions)
Num Frames (in millions)
Figure 18: Standard deviation of the output of the 5 online networks of i-DQN averaged over 3200 samples. The standard deviation is greater than zero, indicating that the online networks are different from each other. The signal has a tendency to decrease, which matches our intuition that the Q-functions become increasingly close to each other as they get closer to the optimal Q-function.
i-DQN, head change for each sample Asterix
i-DQN, constant head along episode Asteroids
Bootstrapped DQN
Qbert
10M Human Norm Score
1.0
0.01
0.00
0.0
100
200
100
200
100
200
Num Frames (in millions)
Num Frames (in millions)
Num Frames (in millions)
Figure 19: i-DQN (i-DQN, head change for each sample) outperforms Bootstrapped DQN on the 3 considered Atari games. Keeping fixed head along each episode (i-DQN, constant head along episode) does not seem beneficial compared to sampling a new head at each environment interaction.
layers. Agarwal et al. (2020) share the same conclusion for Random Ensemble Mixture (REM) (Agarwal et al., 2020). They explain that independent. networks are more likely to cover a wider space in the space of Q-functions. In CrazyClimber both algorithms converge to the same score. Nonetheless, we argue that a shared architecture provides a reasonable trade-off between increased performances and additional memory computation and training time.
Distance between the online networks. i-DQN hcavily relics on the fact that the learned Q-functions are located within different areas in the space of Q-functioss. To verify this assumption,, we computed the standard deviation of the output of the learned Q-functions during the training Figure 18. This figure shows that the standard deviation among the Q-functions indeed greater than zero across the 3 studied games. Furthermore, the standard deviation decreases during training, suggesting they become increasingly closer. This matches the intuition that at the end of the training, the iteration of the Q-functions should lie at the boundary of the space of the space of representable Q-functions, close to each other.
i-DQN vs. DQN with an inflated number of parameters. i-DQN requires more parameters than DQN as it learns K Bellman updates in parallel. We point out that each network of i-DQN uses the same number of parameters as DQN, which means that both algorithms use the same amount of resources at inference lime. Nevertheless, we consider a version of DQN having access to the same nunber of parameters as i-DQN by increasing the number of neurons in the last hidden layer. In Figure 17, the inflated version of DQN does not provide an increase performance as high as i-DQN. This supports the idea that i-QN's benefit lies in how the networks are organized minimize the sum of approximation errors, as explained in Section 4.
24

---

Published in Transactions on Machine Learning Research (02/2025)
I-Q Bootstrapped DQN (Osband et al., 2016). Bootstrapped DQN is a method that also uses ensemble of N Q-functions. However, Booistrapped DQN and i-QN have orthogonal objectives: while Bootstrapped DQN focuses on deep exploration, I-QN aims at learning multiple Bellman updates concurrently. Nevertheless, in Figure 19, we propose to compare Bootstrapped DQN with N = to i-DQN with K 5 (i-DQN, head change for each sample) such that both algorithums use the same number of Q-functions. Indeed, i-DQN requires K networks. i-DQN outperforms Bootstrapped DQN on the 3 considered Atari games. Additionally, we evaluate version of i-DQN where the same Q-function is kept for interacting with the environment (i-DQN, constant head along episode). This version of i-DQN performs slightly worse that the version of i-DQN where head is sampled at each environment interaction. This suggests that keeping the same head along an episode to hope for deeper exploration, as suggested by Osband et al. (2016), is less elficient than sampling a head at each environment interaction to avoid learning the other heads passively as identified by Ostrovski et al. (2021).
I-Q vs. Value Iteration Networks (VIN) (Tamar et al., 2016). While i-QN's goal to learn multiple Bellman updates concurrently, the goal of VINs is train policy end-to-end and to include planning steps in finite MDP M that is different from the original MDP M inside the policy architecture. This means that while i-QN's objective is to learn multiple Q-functions at once, VIN's objective is to learn the transition dynamics and the reward of the finite MDP M. This is why, the VIN framework entails some drawbacks that do not belong to the i-QN framework making the comparison impractical. Indeed, VINs do not scale well with the number of Bellman iterations (planning steps) needed for solving the task because the number of parameters used by VTNs scales linearly with the number of planning steps as opposed iQQ where the considered window of Bellman updates can be effectively shifted as explained in Section 4, Additionally, opposed to i-QN, VINs assume that, for cach MDP M, there exists *finite* MDP M such that the optimal plan for M contains useful information about the optimal policy of the original MDP M. This assumption is not guaranteed to hold for all MDPs and more importantly, some design choices need to be made to model M in practice.
F Training curves for MuJoCo control problems
i-SAC, 0.02
i-rrQQ, 0.02
SAC, 0.02
DroQ. T= 0.02
MuJoCo
IQM SAC Norm return
1.00
0.75
0.50
0.25
0.0
0.5 Env Steps
1.0
1e6
Figure 20: The itcrated versions of SAC and DroQ with K 4 also outperform their sequential versions when all the hyperparameters are the same.
25

---

Published in Transactions on Machine Learning Research (02/2025)
i-SAC, == 0.02
i-DroQ. == 0.02
SAC, 0.005
DroQ. 0.005
le3
Hopper
le3
Ant
le4
HalfCheetah
1.25
6
1QM return
3
1.00
4
2
0.75
2
0.50
1e3
Walker2d
1e3
Humanoid
1e5
HumanoidStanduu
1.75
1QM retur
4
1.50
2
2
1.25
1.00
0.0
0.5 Num Steps
1.0
0.0
0.5 Num Steps
1.0
0.0
0.5
1.0
le6
le6
Num Steps
le6
Training curves of i-SAC with K = 4, i-DroQ with = 4 along with SAC and DroQ on 6 MuJoCo environments. In most problems, the iterated approach outperforms the sequential approach.
Figure 21:
i-SAC, 0.0.2
i-DroQ, T= 0.02
SAC, 0.02
DroQ. 0.02
le3
Hopper
le3
Ant
le4
HalfCheetah
1.25
6
IQM return
3
1.00
4
2
0.75
0.50
1e3
Walker2d
1e3
Humanoid
1e5
HumanoidStandup
1QM retum
4
4
1.75
1.50
2
2
1.25
0.0
0.5 Num Steps
1.0 le6
0.0
0.5 Num Steps
1.0
0.0
0.5 Num Steps
1.0
le6
le6
Training curves of i-SAC with K = 4, i-DroQ with = 4 along with SAC and DroQ on 6 MuJoCo environments. Compared to Figure 21, all algorithms use the same soft target update frequency 7 = 0.02. In most problems, the iterated approach outperforms the sequential approach.
Figure 22:
26


Reviews:
[{"review_id": "B1LUs1CNCd", "reviewer": "Reviewer_T2xJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The contribution is novel.\n* The algorithm is simple and easy to implement.\n* The algorithm is theoretically grounded. \n* Ablation study takes into consideration the number of gradient steps, which network is used for action sampling, etc.\n* Appropriate experimental settings i.e., both mujoco and atari.\n* Use the baselines from the well established dopamine library.\n\nWeaknesses:\n* Ablation studies do not take into consideration computational complexity nor parameter counts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduced iterated Q-Network (i-QN) to learn multiple consecutive Bellman updates. The authors demonstrate the effectiveness of i-QN on the mujoco and atari benchmarks. Furthermore, they show that their algorithm is theoretically grounded.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please include a discussion of computational complexity, FLOPS, etc.\n* Ideally, you should keep the number of parameters constant. Specifically, is it better to scale the number of networks or having a bigger network?\n\nMinor:\n* Figure 6: The colors of DQN, G=1 and DQN, G=4 are difficult to distinguish.", "broader_impact_concerns": "Not applicable since this work is theoretical in nature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MrQb9FRGhq", "reviewer": "Reviewer_7Lyx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe i-QN architecture is a very interesting form of ensemble; rather\nthan training each ensemble member identically, the ensemble members act\nas targets for a (ordered) *sequence* of Bellman targets. Notably, the\nsequence of Bellman backups required for computing parameter updates can be\nevaluated in parallel, so as to not dramatically increase training time.\n\nThe motivation for the i-QN architecture is nice, leveraging results\nproviding upper bounds on policy suboptimality. Empirically, the\nperformance of the various i-QN implementations appears to be\nconsistently good; I appreciated that the authors implemented i-QN\nwithin a variety of base deep RL methods (e.g., including multiple\nexamples of value-based discrete-action methods and continuous control\nactor-critic methods).\n\n## Weaknesses\n\nThe main weakness of the paper, in my opinion, is a lack of comparison\nwith the most similar methods in the literature involving either an\nensemble of Q-functions or multiple applications of the Bellman\noperator; most notably, Bootstrapped DQN and Value Iteration Networks\n(see Requested Changes). More specifically, I believe the following\ncrucial experiments / discussions are missing:\n\n1.  Comparison of i-QN to these existing architectures / methods;\n2.  Study of which components are responsible for performance\n    improvements (e.g. is it due to increased capacity? due to ensemble\n    diversity? is it actually due to reduced error of accumulated\n    Bellman errors?).\n\nMoreover, as discussed in Requested Changes, some of the writing (particularly surrounding technical details) is imprecise and/or unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel architecture, called i-QN, for critics in\ndeep RL, wherein an ensemble of $K$ critics is trained so as to minimize\nthe error accumulated over $K$ applications of the Bellman operator. The\nauthors show that minimizing such a loss can lead to tighter upper\nbounds on policy suboptimality in theory. Empirically, across a variety\nof benchmarks, it is shown that i-QN consistently results in greater\nperformance or sample efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  In the abstract, \"till now\" is too informal, it should be \"until\n    now\".\n2.  The name \"i-QN\" collides with the already popular IQN\n    algorithm/architecture (Implicit Quantile Network) – it may be best to consider a different\nname.\n1.  In the introduction, you describe the following as an efficiency\n    issue in deep RL: \"projection steps are made sequentially, thus forcing to learn\na Bellman update only at the end of the previous projection step\". I\ndon't understand what this means—I think this could be spelled out and\nmotivated in a little more detail.\n1.  I think it would be worth explicitly spelling out what you mean by\n    the empirical estimate $\\hat\\Gamma$ of the Bellman operator. Readers that\nhaven't studied TD-learning in detail might not be comfortable with this\nnotion / might not realize that what you're describing here is the usual\napproach to value-based deep RL. Likewise, there is more than one\nempirical Bellman operator (e.g. the standard one vs that used in double\nQ-learning). Similarly, I believe you should also cite some more\nfundamental algorithms here, such as Q-Learning and DQN, which almost\nall readers have likely implemented.\n1.  The notation $Q^{\\ast/\\pi}$ is pretty odd. Since you often want to\nsimultaneously describe both Q-functions, it might be preferable to\nwrite $Q^\\bullet$ for $\\bullet\\in\\{\\ast,\\pi\\}$.\n1.  There should be discussion comparing the proposed approach and\n    methods such as Value Iteration Networks, which are closely related.\n1.  At the beginning of section 4, it says \"By learning one Bellman\n    update at a time, classical $Q$-Network approaches minimize each term at a time\". I\ndon't know that this is exactly correct. Q-learning approaches do not\nminimize $\\|\\Gamma^\\ast Q_{k-1} - Q_k\\|$, they make indivual updates in\nthe direction minimizing those losses. It sounds like you're referring\nmore to something like FQI here. Maybe this is fine, since approaches\ninvolving target critics are perhaps more akin to FQI than Q-learning,\nbut this should be explained.\n1.  Why do you frequently write the sum of subsequent Bellman errors as\n$\\|\\Gamma^\\star Q\\_{\\overline{\\theta}\\_0} - Q\\_{\\theta\\_1}\\|^2 + \\sum\\_{k=2}^K\\|\\Gamma^\\star Q\\_{\\overline{\\theta}\\_{k-1}} -\nQ\\_{\\theta\\_k}\\|^2$ as opposed to a single term\n$\\sum\\_{k=1}^K\\|Q\\_{\\overline{\\theta}\\_{k-1}} - Q_{\\theta\\_k}\\|^2$?\n1.  In section 4, i-QN is discussed several times before it is actually\n    defined. For example, at the bottom of page 4 it says \"As i-QN is a semi-gradient\nmethod…\", but this is before i-QN has been described explicitly.\n1.  The discussion just above section 4.1 is difficult to understand. It\n    says \"the optimization procedure is designed to make Equation 4 valid\". Which\noptimization procedure is this referring to? It would be helpful to\nunderstand how this claim is true.\n1.  There should be a discussion (and experiments?) comparing i-QN with\n    something like Value Iteration Networks \\[1; VIN\\] and its descendants. Both attempt to\npredict several iterations of Bellman updates. When should one prefer\ni-QN to VIN and vice-versa? It appears that i-QN has the benefit of\nenabling parallelized applications of the Bellman operator over its\nheads, unlike VIN. If we could implement a \"Transformer analogue of\nVIN\", would i-QN still retain any benefits?\n1.  Likewise, there should be more discussion as well as experiments\n    comparing i-QN with Bootstrapped DQN \\[2\\]. In the introduction, Bootstrapped DQN\nis described as \"\\[improving\\] the projection of $Q^{\\ast/\\pi}$\non $\\mathcal{Q}_\\Theta$\". However, there is really more to it than that;\nBootstrapped DQN was designed to represent a posterior over Q-functions\nfor exploration with an ensemble. i-QN also introduces a form of\nensemble. Would Bootstrapped DQN perform worse than i-QN with the same\n$K$? There are other subtle differences between Bootstrapped DQN and\ni-QN. For example, in Bootstrapped DQN, you uniformly sample a\nQ-function head at the beginning of each episode, and act greedily with\nrespect to it until the episode ends; in i-QN, you uniformly sample a\nQ-function head at each *step* and act $\\epsilon$-greedily. I would\nthink the Bootstrap DQN version could potentially be more useful for\n(deep) exploration, especially if the $K$ heads in i-QN encode any form\nof epistemic uncertainty. Do you think the improved performance of i-QN\ncan (at least in part) be due to improved exploration induced by the\ndisagreement between heads? This should be discussed in the manuscript.\n1.  With regard to the IQM plots, it would be nice to see the suite of\nstatistics (e.g., optimality gap, etc) recommended by Agarwal et al. as\nopposed to only IQM, which hides some useful information.\n1.  Figure 6 middle uses colors that are very difficult to distinguish.\n    It would also be helpful to include a reminder around this figure about what\n$G, T$ are.\n1.  In the \"Gradient steps executed in parallel\" part of section 6.1.1,\n    you say that Figure 6 middle tests i-DQN with $K=4$ and $G=1$, but in the\nFigure, it says i-DQN with $K=4$ and $G=4$. Which one is it? The latter\n(setting described in the Figure) makes more sense to me in the context\nof the experiment.\n\n## References\n\n1.  Tamar, Aviv, et al. \"Value iteration networks.\" Advances in neural\ninformation processing systems 29 (2016).\n1.  Osband, Ian, et al. \"Deep exploration via bootstrapped DQN.\"\n    Advances in neural information processing systems 29 (2016).", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EBuH8Keh5t", "reviewer": "Reviewer_L6Jg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is simple and effective - a recipe that is important in scaling Deep RL. The method is generally applicable to any  Deep RL method, which uses critic updates, and adds an overhead of memory and time which scales linearly with the length of iterated updates. \n2. The experiments are well designed to ablate where the benefit comes from when compared to basic DQN and IQN and variations on top of it. \n3. The method gives a significant improvement in sample efficiency across both Atari and Mujoco domains.\n\nWeaknesses:\n1. Why are Steps 13 and 14 needed when there are Steps 11 and 12 in algorithm 1? Why is there a need to update the targets to current parameters rather than always updating to the online networks which uses the target parameters? If I understand correctly step 13 and 14, target Q_0 remains unupdated. Why   I felt the explanation of window size update with D and N was lacking and hard to understand. I think the second paragraph of Section 4 should be more clearly written to explain the core method of the paper clearly. \n2. It seems in the algorithm, it is not made clear how $\\hat{\\theta}_0$ is initialized.\n3. It is not clear to me how the theoretical result in Proposition 4.1 argues for decreasing the approximation error of full-gradient updates. Shouldn't the equation in line 2 of Page 5 should have $Q_{\\theta_0}$ rather than the target notation? Can the assumptions of the proof  be explained more clearly?\n4. I found the projection argument throughout the paper to be weak. Where is the projection argument used for? In all figures, it seems the projection is brining it back the targets back to the function space,but in Deep RL it is not clear we are limited by realizability in the function space. If the point is just to say that it takes a number of gradient updates for the function value to converge to the target value, then maybe that it might benefit to state that clearly. \n5. One aspect of the experiment that remained unexplored was: How does the iterative Bellman update method compare to the methods which manage to successfully increase the UTD (Update to data) ratio? It seems they should be equivalent in terms of the empirical Bellman operator both methods are trying to perform. [1,2]\n\n\n[1]: Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).\n[2]: Hussing, Marcel, et al. \"Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence.\" arXiv preprint arXiv:2403.05996 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new way to successfully perform multiple iterations of Bellman updates at once. The authors proposed to replace the traditional 1-step Bellman updates of the form $(T \\bar{Q} - Q)^2$ by a sequence of parallel updates where k Q-networks and targets are maintained. Each Q-network is updated based on the target of the previous Q-network and these updates can be performed in parallel and together. Authors demonstrate a theoretical condition when their method can help in reducing approximation errors, but in generally show through empirical results that their method can successfully leverage more gradient steps by increasing sample efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Section 4 (the part before section 4.1) should be written with greater clarity and correctness. This would be crucial for recommending acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimous in agreeing this paper should be accepted as is, which I agree with. It is well-written, well-motivated, and provides adequate evidence for the claims.
Decision: Accept (poster)