Bridging the performance-gap between target-free and target-based reinforcement learning
Authors: Théo Vincent
Yogesh Tripathi
Tim Faust
Abdullah Akgül
Yaniv Oren
Melih Kandemir
Jan Peters
Carlo D'Eramo
Venue: ICLR 2026 Poster (2026)
Keywords: deep reinforcement learning
Q-learning
function approximation
Primary area: reinforcement learning
TL;DR: We introduce iterated Shared Q-Network, a new algorithm improving the sample-efficiency of target-free algorithms to bridge the gap with target-based algorithms.

Abstract: The use of target networks in deep reinforcement learning is a widely popular solution to mitigate the brittleness of semi-gradient approaches and stabilize learning. However, target networks notoriously require additional memory and delay the propagation of Bellman updates compared to an ideal target-free approach. In this work, we step out of the binary choice between target-free and target-based algorithms. We introduce a new method that uses a copy of the last linear layer of the online network as a target network, while sharing the remaining parameters with the up-to-date online network. This simple modification enables us to keep the target-free's low-memory footprint while leveraging the target-based literature. We find that combining our approach with the concept of iterated $Q$-learning, which consists of learning consecutive Bellman updates in parallel, helps improve the sample-efficiency of target-free approaches. Our proposed method, iterated Shared $Q$-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems while using a single $Q$-network, thus stepping towards resource-efficient reinforcement learning algorithms.

Published as a conference paper at ICLR 2026
BRIDGING THE PERFORMANCE-GAN BETWEEN TARGET- FREE AND TARGET- BASED REINFORCEMENT LEARNING
Théo Vincent .2,- Yogesh Tripathi' 1.2 Tim Faust 1.2
Abdullah Akgül
Yaniv Oren Melih K Jan Peters 1.2erss!.2244, 1,2,4.7
Carlo D'Eramo
DDFII, SAIROL 2 TU Darmstadt University of Southern Denmark Hessian.ai
5 Delft University of Technology
UUniversity of Würzzurg
"Robotics Institute Germany
ABSTRACT
The use of target networks in deep reinforcement learning is a widely popular solu- tion to mitigate the brittleness of amiigrraieet approaches and stabilize learning. However, target networks notoriously require additional memory and delay the propagation of Bellman updates compared to an ideal fagget-free approach. In this work, we step out of the binary choice between target-free and target-based algorithms. We introduce a new method that uses a copy of the last linear layer the online network as a target network, while sharing the remaining parameters with the up-to-date online network. This simple modification enables us to keep the target-free' low-memory footprint while leveraging the target- sarget-based literature. We find that combining our approach with the concept of iterated Q-learning, which consists of learning consecutive Bellman updates parallel, helps improve the sample-effiiieccy of target-free approaches. Our proposed method, iterated Shared Q-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems while using a single Q-network, thus stepping towards resource-efficient reinforcement learning algorithms.
INTRODUCTION
Originally, Q-learning (Watkins & Dayan, 1992) was introduced as a reinforcement learning (RL) method that performs asynchronous dynamic programming using a single look-up table. By storing only one Q-estimate, Q-learning benefits from an up-to-date estimate and a low memory footprint. However, replacing look-up tables with non-linear function approximators and allowing off-policy samples make the method more scalable introduces training instabilities (Sutton & Barto, 2018). To address this, Mnih et al. (2015) introduce Deep Q-Network (DQN), an algorithm that constructs the regression target from an older version of the online network, known the larget network, which is periodically updated to match the online network (see "Target Based" in Figure 1). This modification to the lemporal-differenee difference objective helps mitigate the negative effects of function approximation and bootstrapping (Zhang et al., 2021), two elements of the deadly triad (van Hasselt et al., 201 8). Recently, new methods have demonstrated that increasing the size of the Q-network the learning speed and final performance of temporal difference methods (Espeholt et al., 2018; Schwarzer et al., 2023; Nauman et al., 2024; Lee al., 2025). Numerous ablation studies highlight the crucial role of the target network in maintaining performance improvements over smaller networks (Figure in Schwarzer et al. (2023), and Figure 9b in Nauman et al. (2024)). Interestingly, even methods initially introduced without target network (Bhatt et al. (2024) and Kim et al. (2019)) benefit from its reintegration (Figure 5 in Palenicek et al. (2025) and Gan al. (2021)).
enhance
While temporal difference methods clearly benefit from target networks, their utilization doubles the memory footprint dedicated to Q-networks. This ultimately limits the size of the online network due to the constrained memory capacity of the processing units (c.g. the vRAM of GPU). This limitation not only problematic for learning on edge devices where memory constrained, but also for applications that inherently require large network sizes, such as handling nigh-dimensional state spaces (Boukas et al., 2021; férez-Dattar et al., 2019), processing multi-modal inputs (Schneider al., 2026), constructing mixtures of experts (Obando Ceron et al., 2024; Hendawy et al., 2024). This motivates the development of target-free methods (see "Target Free" in Figure 1).
Coorespondance to: theo. vi nccettdfki.c. Code available at: https: / /i thub. com/ theovi httpr//stetttttnte//e////

---

Published as a conference paper at ICLR 2026
Target network
Online network
Feature extractor
Linear layer
T
Bellman operator
Target Based
Target Free
Shared Features
itevated Shared Features
approaches, where linear layer represents the target network, sharing the rest of the parameters with the online network (Shared Features). We apply the concept of iterated Q-learning (Vincent al., 2025b), which consists of learning multiple Bellman updates in parallel, to reduce the performance gap between target-free and larget-based approaches (iterated Shared Features).
Figure 1: We propose simple alternative aarget-based/targee
In this work, introduce alternative to the binary choice between target-free and target- based approaches. We propose storing only the parameters of the last linear layer, while using the parameters of the online network substitute the other layers of the target network (see "Shared Features" in Figure 1). Although this simple modification alone helps reduce the performance gap between target-free and target-based DQN (see "iS-DQN K 1 in Figure right), explain in this work how it opens up the possibility of leveraging the target-based literature to reduce this gap further, while maintaining low memory footprint, Notably, this approach is also orthogonal to regularization techniques that have been shown to be effective for target-free free algorithms (Kim et al., 2019; Bhatt al., 2024; Gallici al., 2025). Therefore, we will build upon these approaches benefit from their performance gains.
In the following, we leverage the concept iterated Q-learning (Vincent al., 2025b) to enhance the learning speed (in terms of the number of environment interactions) of target-free algorithms, which a major bottleneck in many real-world applications. The concept iterated Q-learning, initially introduced
target-based approach, aims at learning multiple Bellman iterations in parallel. This leads to a new algorithm, termed iterated Shared Q-Network (iS- AS-QNN, pronounced "ice- "ice-QN" to emphasize that it contains a frozen head. iS-QN utilizes single network with multiple linear heads, where each head is trained to represent the Bellman target of the previous one (see "iterated Shared Features" in Figure 1). Our evaluation of iS-QN across various RL settings demonstrates that it improves the learning speed of target-free methods while maintaining a comparable memory footprint and training time.
BACKGROUND
Deep Q-Network (Mnih et al., 2015) The optimal policy of a Markov Decision Process (MDP) be obtained by selecting for each state, the action that maximizes the optimal action-value function Q". This function represents the largest achievable expected sum of discounted rewards given a state-action pair. In the context of discrete action spaces, Mnih al. (2015) approximate the optimal action-value function with a neural network Qo. represented by a vector of parameters 0. This neural network is learned approximate its Bellman iteration I'Qa, leveraging the contraction property of the Bellman operator I' in the value space to guide the optimization process toward the operator's s fixed point, i.e., the optimal action- value function Q*. In practice, a sample estimate of the Bellman iteration is used, where for sample (s, r, s'), TQo(s, a) r + max Qo(s', a') where 7 the discount factor. However, this learning procedure network Qo learns from its own values, which change at each optimization step due to function approximation, and because of the compound effect of the overestimation bias. mitigate these issues, the authors introduce target network with parameters Ø to stabilize the regression target IQ and periodically update these parameters to the online parameters 0 every T steps. On the negative side, this doubles the memory footprint dedicated to Qentwwokks.
unstable because the neural
Iterated Q-Network (Vincent et al., 2025b) By using a target network, DQN slows down the training process multiple gradient steps dedicated to each Bellman iteration, IQ is delayed by some gradient steps compared to I JºQ.. To increase the learning speed, Vincent et al. (2025b) propose learn consecutive Bellman iterations in parallel. This approach uses sequence of online parameters (0.){, and sequence of target parameters K-1 Each online network Q is
2

---

Published as a conference paper at ICLR 2026
trained to regress IQ Similarly to DQN, each target parameter 0, is updated to the online parameter lc+1 every T steps. Importantly, the structure of chain is enforced by setting each 0g to 6g every D & T steps so that each Q0 which is learned to regress TQ; are forced to approximate TQu.- This results in Qøx = TQ ~ = I Qe,, thus learning K consecutive Bellman iterations in parallel. Importantly, DQN can be recovered by setting K 1. While the feature representation can be shared across the online Q-networks, iterated Q-Network or (i-QN) has the drawback of requiring an old copy of the online networks to stabilize training. significantly increasing the memory footprint. In the following, we will explain how the concept of i-QN can help reduce the performance gap between target-free free and baaed approaches while maintaining a low memory footprint.
3 RELATED WORK
Other works have considered removing the target network in different RL scenarios. Vasan et al. (2024) introduce Action Value Gradient, algorithm designed to work well in streaming scenario where no replay buffer, no batch updates, and no target networks are available. Gallici et al. (2025) also develop a method for streaming scenario, in which they rely on parallel environments cope with the non-stationarity of the sample distribution. Gradient Temporal Difference learning is another line of work that does not use target networks (Sutton et al., 2009; Maei et al., 2009: Yang et al., 2021; Patterson et al., 2022; Elelimy et al., 2025). Instead. they compute the gradient w.r.t. the regression target as well as the gradient w.r.t. the predictions, which doubles the compute requirement. Additionally, to address the double sampling problem, another network is trained to approximate the temporal difference value, which also increases the memory footprint.
Alternatively, some works construct the regression target from the online network instead of the target network, but still use a target network in some other way. For example, Ohnishi et al. (2019) compute the TD(0) loss from the online network and add term in the loss to constrain the predictions of the online network for the next actton pair (s', d') to remain close to the one predicted by the target network. Piché et al. (2021; 2023) develop similar approach, enforcing similar values for the state-action pair (s, a). Lindström et al. (2025) show that the target network can be removed after a pretraining phase in which they rely on expert demonstrations. Hendawy al. (2026) construct the target by using the lowest estimate between the online and target networks' predictions to learn from up-to-date estimates without suffering from the overestimation bias.
Many regularization techniques have been developed, attempting to combat the performance drop that occurs when removing the target network. We stress that our approach is orthogonal these regularization techniques and we show in Section 5 that our method improves the performance of target-free methods equipped with these advancements. Li & Pathak (2021) encode the input of the Q-network with learned Fourier features. While this approach seems promising, the authors acknowledge that the performance degrades for nigh-dimensional problems. Shao et al. (2022) remove the nnrget-netoork and search for an action that maximizes the Q-network predictions more than the action proposed by the policy. Searching for a better action requires additional resources and is only relevant for actor-criti algorithms. Kim et al. (2019) leverage the MellowMax operator to get rid of the target network. However, the temperature parameter needs to be tuned (Kim, 2020), which increases the compute budget, and follow-up work demonstrates that the reintegration of the target network is beneficial (Gan et al., 2021). We combine the MellowMax operator with the presented approach in Section D.I and demonstrate that it bridges the performance gap between the target-free and target-based approaches. Finally, Bhatt et al. (2024) point out the importance of using batch normalization (loffe & Szegedy, 2015) address the distribution shift of the input given to the critic. Our investigation reveals that it degrades the performance in a discrete action setting (see Figure 21, right).
The idea of learning multiple Bellman iterations has been introduced by Schmitt et al. (2022). They demonstrate convergence guarantees the case of linear function approximation. Then, Vincent et al. (2024) used this approach to learn recurrent hypernetork generating a sequence of Q-functoons where each Q-function approximates the Bellman iteration of the previous Q-function. Finally, Vincent et al. (2025b) introduced iterated Q-Network as a far-sighted version of DQN that learns the K following Bellman iterations in parallel instead of only learning the following one. While promising, those approaches rely on separate copy of the learnall parameters to stabilise the training process, which increases the memory footprint. In this work, we propose to leverage the potential of iterated Q-leaming to boost the learning speed of target-free algorithms.
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 iterated Shared Deep Q-Network (iS-DQN). Modifications to DQN are in purple.
Initialize network Qe with K heads, where each head defined by the parameters wk. We note 02 = (w,w)), and w the shared parameters such that 0 = (w, wo..., WK). D is an empty replay buffer.
2: Repeat
3: Set u Uniform([l,
4: Take action a ~ c-greedy- (Qo, 0 )): Observe reward r, next state s'.
5: Update D f- D DU((s,,,
6:
every G steps
7:
Sample mini-batch B = (s, Q8 (8, and
0, 8') from D.
8:
Store [Qn(s', (), Qx(s'
[Qa(s, a)... .., Qx(s, (s, a)] Qs(
9:
Compute the loss cis-QN
b [.] indicates a stop gradient operation.
-ssaan)) Σ ([r maxe 1 a')1 (s,
10:
Update 0 from VOLIS-ON
every T' steps
12:
Update wi f- Wkt1, for E {O.....K K-11
4
METHOD
Our goal is to design new algorithm that improves the learning speed of target-free value-based RL methods without significantly increasing the number of parameters used by the Q-networks. To achieve this, we consider single Q- Q-network parameterized with K + heads. We note we the "o,,wo..,.xk"
parameters of the h head, w the shared parameters, and define 0 = (w, (w,wb, ... and 02 = ww) Following Vincent al. (2025b), for sample d (s,a,r, 8'), the training loss is
K
CIS-ON (0) = Σ
CON (Ok, 0k-1),
where CON can be chosen from any temporal-different learning algorithm. For instance, DQN uses fa (0x, 1 = ([r + max,' Qo, "hh_,(d',, (s', a')l - Qa (s, a))a, where [-] indicates a stop gradient operation. We stress that wo not learned. However, every T steps. each wy is updated to wh+1+ similarly to the target update step in DQN. This way, iS-QN allows to Iearn K Bellman iterations in parallel while only requiring a small amount of additional parameters on top of a target-free approach. Indeed, in the general case, the size of each head wi is negligible compared to the size of shared parameters w. Algorithm 1 summarizes the changes brought the pseudo-code of DQN to implement this approach.
In Figure 2, we compare the training paths defined by the Q-functions obtained after each target update of the proposed approach (top) and the target-based approach (bottom). For each given sample, the target-based approach learns only Bellman iteration at a time and proceeds to the following one after 7' training steps. In contrast, the iterated Shared Features approach learns several consecutive Bellman iterations in parallel for each given sample. The considered window also moves forward every T training steps. As the window shifts, the network represents Q-functions that closer to the optimal Q-function since every Q-function is learned to represent the Bellman iteration of the previous Q-function. Similarly the target-based and target-free approaches, the online parameters are updated with the gradient computed through the forward pass of the state-action pair (s, a). as indicated with blue arrows. In Figure we depict our approach with of heads can be increased at minimal cost. We note that the first Q-function is considered fixed in this representation, even if the head is the only frozen element and the previous layers are shared with the other learned @esttmmates. We remark that iS-QN with K = implements the "Shared Features" approach presented in Figure 1. Interestingly, the target-free approach can also be depicted in Figure 2. Indeed, not using a target network is equivalent to updating the target network to the online network after each gradient step. Consequently, the target-free approach can be understood as the target-base representation with window shifting every step. Therefore, the target-free frgee-frreedie approach passes through the Bellman iterations faster, creating instabilities as the optimization landscape may direct the training path toward undesirable Q-funtiions.
However, the number
4

---

Published as a conference paper at ICLR 2026
(s) )) (s,
(s² .) (s,a)
iterated Shared Features
Bellman operator
Gradient path
Windows shift by one Q-function every steps
[ Q1
r 122-22-0 -
Q Q2 I 0
r r
Linear luyer
Q
Q3
Feature extractor
Online parametens
Target Based
Target parameters
(s, 1) (s, a)
(s', .) (s,
Figure 2: Comparison of the training path defined by the target networks obtained after each target update during training between the target-base based- approach (bottom) and the iterated Shared Features approach (top). While both approaches wait for 7 training steps before shifting their respective window by one Q-function, our approach already considers the following Bellman iterations using multiple heads, where each head represents the Bellman iteration of the previous head.
Area Under the Curve
Performance Curve
Number of Parameters
100%
K=49
iS-DQN
K=9
1QM Return
2
4 M parameters (16 Mb RAM)
K=4
50%
K=1
1
iS-DQN K=9
TF-DQN
TE-DON
TB-DQN
TB-DQN
0
0
95 100 105
20
40 60
80 100
TB-DON
TF-DON
Kml
Km4
CNN + LN
in % of TB-DQN
Number of Frames (in millions)
is DS-DQN
Figure 3: Reducing the performance gap in online RL on 15 Atari games with the CNN architecture and LayerNorm (LN). While removing the target network leads to a 10% drop in AUC (left), is- iS-DQN K = (using 10 linear heads), not only closes the gap but improves over the target- based approach by 6%. Importantly, iS-DQN uses comparable number of parameters TF-DQN (right).
In the following, we apply iterated Shared Features to several target- based approaches on multiple RL settings, demonstrating that it reduces the gap between target-free and target- based methods. For each algorithm A, we note TB-A as its target-based version, TF-A as its target-free version, and iS-A as the iterated shared approach, where "iS" stands for iterated Shared. For example, for experiments with the DQN algorithm, iterated Shared version. Importantly, we incorporate the insights provided by Gallici et al. (2025) to use LayerNorm (Ba et al., 2016) for the experiments with discrete action spaces, beneficial, even for the target-based approach. Similarly, we use BatchNorm (loffe Szegedy, 2015), suggested Bhatt al. (2024), improve sample-efficiency in continuous action settings, except for the target-based approach, as it degrades performances (see Figure 25, right).
note TF-DQN, the target-free version, and iS iS-DQN, the
found it
EXPERIMENTS
We evaluate iS-QN in online, offline, continuous control, and language-based RL scenarios to demonstrate that it can enhance the learning speed of target-free methods. We focus on the learning speed because, in this work, we are interested in the sample efficiency target-fre free methods. We use the Area Under the performance Curve (AUC) to measure the learning speed. The AUC has the benefit of depending less on the training length compared to the end performance, as accounts for the performance during the entire training. It also favors algorithms that constantly improve during training over those that only emerge at the end training. thus penalizing algorithms that require many samples perform well. In each experiment, normalized by the AUC of the target-base based approach, to facilitate comparison. By normalizing the AUCs, the resulting metric can also be interpreted as the average performance gap observed during
report the AUC of each algorithm,
5

---

Published as a conference paper at ICLR 2026
Area Under the Curve
Arca Under the Curve
Performance Curve
K=49
K=9
IS-DON
K=9
K=3
IQM JQM Return
K=4
K=1-
K=1
TF-DQN
TF-DQN
TB-DQN
TB-DQN
TR-DON
90
95
100
40
60
80
100
20 40
60
80 100
IMPALA + LN in of IB-DQN
CNN
in of TB- TB-
Number Frames (in millions)
Figure 4: Left: Reducing the performance gap in online RL on 10 Atari games with the IMPALA architecture and LayerNorm (LN). Similar to the results with the CNN architecture, iS-DQN bridges the gap between the target-free and target-base based approaches. Middle and Right: Reducing the performance gap in online RL on 15 Atari games with the CNN architecture. Removing the target network of the vanilla DQN algorithm results in 60% performance drop (100% 40%). By using iS-DQN with K = 3, the performance drop is divided by 4 (100% 85% = 15% 60% 4), thereby confirming the benefit of this approach.
training between the considered approach and the target-based approach. We the Inter-Quantile Mean (IQM) and 95% stratified bootstrapped confidence intervals to allow for more robust statistics as advocated by Agarwal et al. (2021). The IQMs are computed over 5 seeds per Atari game, 10 seeds per DMC Hard tasks, and 5 seeds for Wordle. 15 Atari games are used for the experiments on the CNN architecture, and 10 games for the experiments on the IMPALA architecture to reduce the computational budget. Importantly, all hyperparameterr are kept untouched with respect to the standard values (Castro al., 2018), only the architecture modified, as described Section Extensive details about the selection process of the Atari games, the metrics computation, the hyperparameters, and the individual learning curves are reported the appendix.
5.1 ONLINE DISCRETE CONTROL
First, we evaluate iS-DQN on 15 Atari games (Bellemare et al., 2013) with the vanilla CNN architecture (Mnih et al., 2015) equipped with LayerNorm. As expected, the target-free approach yields an AUC 10% smaller than the target-based approach, as shown in Figure 3 (left). This performance drop constant across the training, see Figure 3 (middle). Interestingly, iS-DQN K = improves over TF-DQN by simply storing an old copy of the last linear head. As more Bellman iterations are learned in parallel, the performance gap between iS-DQN and TB-DQN shrinks. Remarkably, iS-DQN K = 9 even outperforms the target-based approach by 6% in AUC. We note a slight decline in performance for iS-DQN K = 49. We conjecture that this is due to the shared feature representation not being rich enough to enable the network to learn 49 Bellman iterations in parallel with linear approximations. Importantly, Figure 3 (right) testifies that this performance boost is achieved with approximately half of the parameters used by the target-based approach, truly reducing the memory footprint required by the Q-functions.
Our evaluation with the IMPALA architecture (Espeholt et al., 2018) with LayerNorm confirms the ability of iS-DQN to reduce the performance gap between target-free and target-based approaches. Indeed, Figure 4 (left) indicates that removing the target network leads an 8% performance drop while iS- iS-DQN annuls the performance gap as more Bellman iterations are learned in parallel, i.e., as K increases. Interestingly, as opposed the CNN architecture, increasing the number of heads to learn 49 Bellman iterations in parallel is beneficial in this scenario. We believe this is due to IMPALA architecture's 's ability to produce a richer representation than the CNN architecture, thereby allowing more Bellman iterations to be approximated with linear mapping. The plots of the performance curve and the number of parameters are similar to the ones for the CNN architecture, see Figure 19.
Finally, we confirm the benefit of the iterated Shared Features approach by removing the normalization layers for all algorithms with the CNN architecture in Figure 4 (right). We observe a major drop in performance for TF-DQN, leading to 60% performance gap (100% 40%). Notably, iS-DQN K = 1 reduces this performance gap to 18% (100% 82%). This highlights the potential of simply storing the last linear layer and using the features of the online network to build lightweight regression

---

Published as a conference paper at ICLR 2026
Area Under the Curve
Performance Curve
1.0
r
r
iS-CQL
1QM Return
ES-CQL
0.5
iS-CQL ES-CQL TF-COL TB-CQL
TF-CQL
TB-CQL
0.0
70
85 100
10
20
30 40 50
Ensemble Shared Features
IMPALA LN in % of TBB-QLL
Gradient Steps (x 62.5K)
Figure 5: Reducing the performance gap in offline RI on 10 Atari games with the IMPALA archi- tecture and LayerNorm (LN). is- iS-CQL shrinks the performance gap from 26% 6%. Interestingly, applying the idea of sharing parameters to Ensemble DQN (Ensemble Shared Features, ES- ES-CQL) also reduces the performance gap, demonstrating that this idea not limited iterated Q-learning and can be applied other target-based approaches.
Area Under the Curve
Performance Curve Number of Parameters
iS-SAC
K=9
1QM Norm Return
1.0
iS-SAC K=9
100%
TF-SAC
parameters
K=1
TB-SAC
(20 Mh RAM)
0.5
51% 1
TF-SAC
TB-SAC
0.0
0.0
50
75
100
0.25
0.5
0.75 1.0
TB-SAC
TF-AC
SimbaV2 BN in % of TB-SAC
Environment Steps (in millions)
iS-SAC
Figure 6: Reducing the performance gap in online RL on the 7 DMC Hard tasks with the SimbaV2 architecture and BatchNorm (BN), iS-SAC recovers the performance drop incurred by removing the target network (left). This performance boost is made while reducing the total number of parameters by 49% (right).
target. While increasing the number learned Bellman iterations brings a small benefit, the performances are slightly decreasing for higher values of K, indicating that LayerNorm is beneficial to provide useful representations when considering higher number of linear heads.
5.2 OFFLINE DISCRETE CONTROL
We consider an offline RL setting in which the agent has access to 10% of the dataset collected by vanilla DQN agent trained with a budget of 200 million frames (Agarwal et al., 2020), sampled uniformly. We adapt the loss for learning each Bellman iteration to the one proposed by Kumar et al. (2020b). This leads to iterated version of Conservative Q-Learning (CQL). In Figure 5, iS-CQL K = 9 reduces the performance gap by 20 percentage points, ending up with performance gap of 6% compared to 26% for TF-CQL. Additionally, we evaluate another way of sharing features to show that this idea not limited to iterated Q-learning. Instead of building a chain of Q-functions represented by linear heads, we define ensemble of pairs linear heads. Each pair contains frozen head representing target network Q that is used to train the learned head representing the associated online network Q. as depicted in Figure 5 (right). We evaluate this variant that we call Ensemble Shared Features (ES-CQL), with 5 pairs of heads, i.e. 10 heads, to match the number of heads used by iS-CQL
as the number heads of iS-QN is always equal to + Importantly, ES-CQL also outperforms TF-CQL, reinforcing the idea that sharing parameters and using linear heads a fruitful direction.
5.3 ONLINE CONTINUOUS CONTROL
We investigate the behavior of is IS-QN on the DeepMind Control suite (Tassa et al., 2018). focusing on the hard tasks. We select Soft Actor-Critic (SAC, Haarnoja et al. (2018)) the base algorithm and
7

---

Published as a conference paper at ICLR 2026
Area Under the Curve
Performance Curve
Number of Parameters
iS-ILQL
K=9
-2
100%
88 M parameters
IQM Return
(352 Mb of RAM)
K=1
67%
-3
TF-ILQL
IS-ILOLAE K=9
TE-ILQL ILQI
TB-ILQL
95
100 105 in % of TB-ILQL
110
0
2
TB-ILOL
TF-ILQU
GPT-2
Gradient Steps (x 100K)
iS-ILQL
Figure 7: Reducing the performance gap in offline RL on Wordle with the GPT-2 small architecture. iS-ILQL K 9, not only closes the gap but improves over the target-based approach by more than 5%. Importantly, iS-ILQQ saves 33% of RAM compared to the target- based approach (right).
adapt the architecture to the one proposed by Lee et al. (2025) (SimbaV2) so that the target-based approach corresponds to the state-of-the-an This experiment allows us to test iS-QN on different learning dynamics, as the target updates are done with an exponentially moving average instead of a hard update, and the loss for the critic uses categorical distribution to learn the distribution of the return. Interestingly, Figure 6 (left) shows that only using an old copy of the last layer of the critic to construct the regression target (iS- SAC K 1) recovers the performance drop incurred by the target-free approach compared to the target-based approach. Importantly, Lee et al. (2025) design the critic with significantly more parameters than the actor, as commonly done in the actor-critic literature (Mysore et al., 2021; Mastikhina et al., 2025). This means that iS iSCSAC K=1 reduces the total number of parameters by 49%. see Figure (right). When considering more heads learn the following Bellman updates, we find it beneficial to give more importance to the first Bellman updates by scaling the future terms in the loss by discounting factor of 0.25. We leave the investigation of finding the best way to weight each term in the loss to future work. In Section D.4, we provide a first direction leveraging the concept of meta-gradient reinforcement learning (Xu et al., 2018) to tune learnable coefficients assigned to each term in the loss during training. We note that in this setting, iS- JS-SAC K = 9 only performs on par with K = 1. Nonetheless, iS-SAC K = is still performing better than the best target-frre approach, having overlapping confidence intervals with the barget-based approach, which serves as the gold standard, as requires additional parameters.
5.4 SCALING UP TO LANGUAGE MODELS
In this experiment, we evaluate iS-QN on an offline RL language processing task. Specifically, we focus on Implicit Language Q-Learning (ILQL, Snell et al. (2023)), method introduced with a target network. It adapts implicit Q-learning (Kostrikov et al., 2023) to the language domain by sampling action tokens from policy, learned with supervised learning, and weighted by the advantage computed from the Q-function. We evaluate II.QI on the Wordle game (Lokshtanov Subercaseaux, 2022), multi-turn game where the agent guesses a hidden word and receives feedback after each attempt. As in Snell et al. (2023), we choose the GPT-2 small architecture, which results in TB-ILQL using 264 million parameters. In Figure 7 (left), note that while performance drop noticeable, the target-free approach does not perform significantly worse than the target- based variant. Importantly, sharing parameters and learning K = 9 Bellman iterations in parallel improves the learning speed of the target-free approach by 10% without significantly increasing the memory footprint. This leads iS-QN to save 88 million parameters compared to the original approach.
5.5 STREAMING REINFORCEMENT LEARNING
We consider the streaming setting in which the RL agent only learns from stream of data. In this setting, the agent does not have access to a replay buffer, batch updates, or parallel environments. These constraints position the RL agent in a drastically different setting from the previously considered ones, as replay buffers and batch updates greatly stabilize the learning process (Vasan et al., 2024). Elsayed et al. (2024) introduced Stream Q(A), algorithm adapted for the streaming setting. It is an adaptation of the original Q(A) algorithm that uses observation and reward normalization, sparse initialization, layer norm, and an adaptive step-size optimizer. In this experiment, we combine
8

---

Published as a conference paper at ICLR 2026
Area Under the Curve
Performance Curves
Stream Q64
(in 56 of TF. Stream QU3)
Breakout
CrazyClimber
Assault
4
4
2
K-1
2
TF-Stream QC2
0
0
90
100
110
120
TF-Stream Q(2)
is Stream Q(2) K=4
SpaceInvaders
MsPacman
Qbert
Asterix
1QM Return
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.0
20
40
60
80
100
1
20
40
60
80
100
20
40 60
80
100
20
40
60
80
100
CNN + LN
Number of Frames (in millions)
Figure 8: Increasing the learning speed in a streaming scenario on 7 Atari games with the CNN architecture and LayerNorm (LN). iS-Stream Q(A) K = 3 improves over the target-frre approach by more than 10%, outperforming or performing on par with the baseline on out of the 7 games.
Stream Q()) with the presented approach to obtain iS-Stream Q().. We remark that Stream Q(A) was introduced without a target network; therefore, we focus only on this version (TF-Stream Q()).. We fix target update period of T 10 for iS-Stream Q(A). In Figure 8, report the performance of the target-free approach and the presented approach for three values of K across 7 Atari games that were selected for their diversity in human- aoman-normalized score (see Figure 10). iS-Stream Q(A)
K = 1 performs similarly to the target-free variant. Remarkably, increasing the number of heads to 4 (K' 3) improves the learning speed compared to the target-free approach. In this setting, increasing the number of heads until 9 Bellman iterations are learned in parallel is not helpful. The absence of batch updates increases the variance of the gradient steps, potentially leading to unstable updates for the shared parameters for large number of heads.
5.6 WHY IS IS-QN IMPROVING OVER TARGET- APPROACHES?
We now provide some insights to understand why iS-QN reduces the performance gap between target-free and target- based approaches. First, we investigate the change in the learning dynamics that happens when the features are shared between the online and the target heads ("Shared Features" or equivalently, "iterated Shared Features" with K" = 1, see Figure 1). To evaluate the impact on the learning dynamics, we compute, for each gradient step of an iS-DQN K = 1 agent, the gradient with respect to the loss of iS-DQN, as well as the gradients that the target- bagget-based loss and the target-free loss would produce. These quantities determine how the parameters evolve during training. We then report the cosine similarity between the gradients w.r.t. the iS-QN loss and the TB-DQN loss, and the cosine similarity between the gradients w.r.t. the TF-DQN loss and the TB-DQN loss in Figure (left) for 15 Atari games. Interestingly, the gradients obtained by the target- based approach are closer to the gradients of is SSNN K = 1 than the gradients of the target-free approach, especially at the beginning of the training. This means that by simply using copy of the last linear layer and sharing features, IS-D learning dynamics become closer to those of the target-based approach.
At first sight, the fact that iS-QN uses frozen heads on top of features changing each gradient step might seem like an uncommon practice in machine learning. However, this design choice is already part of the reinforcement learning literature. Indeed, in Deep Q-Network, the Q-network is designed with multiple heads, each one representing the prediction for a specific action. For each sample, only the selected head corresponding to the sampled action is updated, while the other heads, built on top of the features that are getting updated, remain frozen. This is likely to contribute to the policy churn phenomenon identified by Schaul et al. (2022), highlighting that the greedy-policy changes for significant proportion of the states in the replay buffer after single batch update. To measure the impact of sharing features, we introduce the notion of target churn, which we define as the absolute value of the difference between the regression target before and after each batch update. We report the cumulative target churn of iS DQN, reinitialized to zero after each target update, normalized by the target churn of TF-DQN in Figure 9 (middle). Conveniently, the target-based approach has a constant target churn of zero since the batch update does not influence the fully separated target network, and the normalization brings the target churn of the target-free approach to constant value

---

Published as a conference paper at ICLR 2026
Cosine Similarity
Target churn
Srank
90
0.5
1.0
10M
10M
1QM (in
0.4
0.5
DN K-9
70
IS-DON K=1
iS-DON K-9
VaC and V VaLTH and VoLTH
TE-DON
IS-DOO K=1
TB-DQN
TF-DON TB DBN
0.3
0.0
50
5
10
15
20
5
10 15
20
1
5
10
15
20
CNN + LN
Number of Frames (in millions)
Figure 9: Left: The cosine similarity between the gradients w.r.t the loss of DS- and TB-DQN is larger than the cosine similarity between the gradients w.r.t. the loss of TF-DQN and TB-DQN. Therefore, iS-DQN brings the learning dynamics of the target-free approach closer to those of the target-based approach. Middle: The target churn is the difference between the regression targets computed before and after each batch update. The target predictions of iS-DQN are less influenced by batch updates than the ones computed from the target- approach. Right: The effecctive rank (srank) of the features in the penultimate layer is higher for iS-QN, resulting in a higher expressivity.
of 1. Remarkably, the target churn of iS-DQN K = 1 and 9 lies in between 0 and 1, indicating that is IS-QN's targets are more stable than the ones of the target-free approach. We note that the target churn for K- 9 is larger than K 1, due the influence of the additional terms in the loss.
Beyond improving the learning dynamics of TF-DQN, iS-DQN also provides a richer state repre- sentation. We measure the representation expressivity by reporting the effective rank (srank) of the features in the penultimate layer (Kumar et al., 2020a) in Figure 9 (right). Interestingly. the srank obtained by iS-DQN K 1 is closer the srank of TB-DQN than the srank of TF-DQN, which further demonstrates the benefit of using the last linear layer to construct the target. Notably, learning K Bellman iterations in parallel increases the representation capacity of the network by large margin. This behavior is also visible in the offline setting, where iS-CQL reaches similar srank as the target-based approach at the end of the training (see Figure 23, middle). This confirms the benefit of iS-QN to foster richer representational capacity.
6 LIMITATION AND CONCLUSION
The proposed approach introduces the number of Bellman updates K to learn in parallel as new hyperparameter and there seems to be different optimal value for each setting. However, we usually observe a stable increase in performance as K grows until the benefit of the approach starts to diminish. Therefore, we recommend increasing until performance begins to decrease, as is commonly done with the learning rate. In Sections D.2, D.3, and D.4, we provide extensive analysis of the dependency of iS-QN on its hyperparameters.
this work, we focus on reducing the memory footprint of the function approximators. Depending on the setting, other objects such the replay buffer and the optimizer can occupy a large portion of the RAM. We remark that the proposed approach can be combined with other works addressing these issues (Vasan al., 2024). Additionally, the proposed approach reduces the memory footprint during training but uses the same amount during inference, which is complementary less during inference (Graesser et al., 2022), As reported in Figure 11, iS-QN does not reduce the training time or the number of Hoating-poin operations, except for the language processing task for which the temporal-differeen
pruning methods that use more memory during training and
eтror can be computed with a single pass through the network.
We introduced a simple yet efficient method for mitigating the performance drop that occurs when removing the target network in deep value-based reinforcement learning, while maintaining a low memory footprint. This made possible by storing a copy of the last linear layer of the online network and using the features of the online network as input to this frozen linear head to construct the regression target. From there, more heads can be added to learn multiple Bellman iterations in parallel. We demonstrated that this new algorithm, iterated Shared Q-Networks, improves over the target-free free approach and yields higher returns when the number of heads increases. We believe that combining iS-QN with mixed-preeee precision training methods is a promising direction for future resource-constraing constrained settings without sacrificing performance. In Section D.5, we provide pilot study demonstrating positive results.
work to facilitate online learning in resource-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by "Third Wave AI", funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and Art, and by the grant "Einrichtung eines Labors des Deutschen Forschungszentrum für Künstliche Intelligenz (DFKI) an der Technischen Universität Darmstadt". The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR @ @FAU) of the Friedrich- Alexander- Universsttt provided by federal and Bavarian state authorities. NHR @ FAU hardware is partially funded by the German Research Foundation (DFG) 4407 19683. This work was also partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG).
Telangen-Nürbeeee (FAU) under the NHR project b187cb. NHR funding is
REPRODUCIBILITY STATEMENT
Special care was taken to ensure this work is reproducible. The code is available at https: // github.
and is shared in the supplementary material. It contains the list of dependencies and their exact version that was used to generate the results. To ease reproducibility, all hyperparameters are listed in Appendix E, and the individual training curves shown in Appendix F.
LARGE LANGUAGE MODEL USAGE
large language model was helpful in polishing writing, improving reading flow, and identifying remaining typos.
CARBON IMPACT
As recommended by Lannelongue & Inouye (2023), we used Glgorihg (Lannelongue et al., 2021) and ML CO2 Impact (Lacoste et al., 2019) to compute the carbon emission related to the production of the electricity used for the computations of our experiments. We only consider the energy used to generate the figures presented this work and ignore the energy used for preliminary studies and for building the computing infrastructure. The estimations vary between 2.63 and 2.88 tonnes of CO2 equivalent. As a reminder, the Intergovernmental Panel on Climate Change advocates carbon budget of 2 tonnes of CO2 equivalent per year per person.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Stat, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization deep reinforcement learning for greater sample efficiency and simplicity. In International Conference Learning Representations, 2024.
Ioannis Boukas, Damien Ernst, Thibaut Théate, Adrien Bolland, Alexandre Huynen, Martin Buch- wald, Christelle Wynants, and Bertrand Cornélusse. deep reinforcement learning framework for continuous intraday market bidding. Machine Learning, 2021.
11

---

Published as a conference paper at ICLR 2026
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rim, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- -Mina,, and Qiao Zhang. JAX: composable transformations Python+NummPy programs, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprini arXiv:
2016.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
arXiv:1812.00110. 2018.
Esraa Elelimy, Brett Daley, Andrew Patterson, Marlos Machado, Adam White, and Martha White. Deep reinforcement learning with gradient eligibility traces. Reinforcement Learning Journal, 2025.
Mohamed Elsayed, Gautham Vasan, and A Rupam Mahmood. Streaming deep reinforcement learning finally works. arXiv preprint vrXiv:2410.14600 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-leamnr architectures. In International Conference on Machine Learning, 2018.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foer- ster, and Mario Martin. Simplifying deep temporal difference learning. In International Conference on Learning Representations. 2025.
Yaozhong Gan, Zhe Zhang, and Xiaoyang Tan. Stabilizing q learning via soft mellowmax operator. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
Laura Graesser, Utku Evei, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference Machine Learning, 2022.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances Neural Information Processing Systems, 2020.
Tuomas Haamojja Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018
Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In International Conference on Learning Representations, 2024.
Ahmed Hendawy, Henrik Metternich, Théo Vincent, Mahdi Kallel, Jan Peters, and Carlo D' D'Eramo. Use the online network if you can: Towards fast and stable reinforcement learning. In International Conference on Learning Representations, 2026.
Sergey loffe and Christian Szegedy. Batch normalization: reducing internal covariate shift. In International Conference on Machine Learning, 2015,
Accelerating deep network training by
Seungehan Kim. Adaptive tuning of temperature in mellowmax using meta-gradients. Master Thesis, Brown University, 2020.
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In International Joint Conference on Artificial Intelligence, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under paramtterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020b.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1rX1v:1910.09700, 2019.
Loic Lannelongue and Michael Inouye. Carbon footprint estimation for computational research. Nature Reviews Methods Primers, 2023.
Loic Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: quantifying the carbon footprint of computation. Advanced Science, 2021.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. In International Conference on Machine Learning, 2025.
Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. In Advances in Neural Information Processing Systems, 2021.
Alexander Lindström, Arunselvan Ramaswamy, and Karl-Johan Grinnemo. Pre-training deep q- networks eliminates the need for target networks: An empirical study. In The 14th International Conference on Pattern Recognition Applications and Methods (ICPRAM), 2025.
Daniel Lokshtanov and Bernardo Subercaseaux. Wordle is np-hard. In International Conference on Fun with Algorithms, 2022.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
Hamid Maei, Csaba Szepesvári, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard Sutton. Convergent cemporal--ifferente learning with arbitrary smooth function approximation. In Advances in Neural Information Processing Systems, 2009.
Olya Mastikhina, Dhruv Sreenivas, and Pablo Samuel Castro. Optimistic critics can empower small actors. Reinforcement Learning Journal, 2025.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
Siddharth Mysore, Bassel El Mabsout, Renato Mancuso, and Kate Saenko. Honey. shrunk the actor: A case study preserving performance with smaller actors in actor-critic rl. In IEEE Conference on Games, 2021.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoš, and Marek Cygan, Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024.
Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. In International Conference Machine Learning, 2024.
Shota Ohnishi, Eiji Uchibe, Yotaro Yamaguchi, Kosuke Nakanishi. Yuji Yasui, and Shin Ishii. Constrained deep q gllarnin gradually approaching ordinary q-learning. Frontiers in Neurorobotics, 2019.
13

---

Published as a conference paper at ICLR 2026
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Daniel Palenicek, Florian Vogt, and Jan Peters. Scaling off-policy reinforcement learning with batch and weight normalization. In Advances in Neural Information Processing Systems, 2025.
Andrew Patterson, Adam White, and Martha White. A generalized projected bellman error for off-policy value estimation in reinforcement learning. Journal of Machine Learning Research, 2022.
Rodrigo Pérez-Dattari, Carlos Celemin, Javier Ruiz-del Solar, and Jens Kober. Continuous control for high-dimensional state spaces: An interactive learning approach. In International Conference on Robotics and Automation, 2019.
Alexandre Piché, Joseph Marino, Gian Maria Marconi. Valentin Thomas, Christopher Pal, and Mohammad Emtiyaz Khan. Beyond target networks: Improving deep q-learning with functional regularization. In NeurlPS Workshop on Deep Reinforcement Learning. 2021.
Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan. Bridging the gap between target networks and functional regularization. Transactions on Machine Learning Research, 2023.
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. In Advances in Neural Information Processing Systems, 2022.
Simon Schmitt, John Shawe-Taylon Taylor, and Hado Van Hasselt. Chaining value functions for off-polis learning. AAAI Conference on Artificial Intelligence, 2022.
Tim Schneider, Cristiana de Farias, Roberto Calandra, Liming Chen, and Jan Peters. Active perception for tactile sensing: A task-agnostic attention-based approach. In International Conference on Learning Representations, 2026.
Julian Schritwieser,, Ioannis Antonoglou. Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2020.
Max Schwarzer, Johan Samir Ohando Ceron, Aaron Courville, Mare Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, 2023.
Lin Shao, Yifan You, Mengyuan Yan, Shenli Yuan, Qingyum Sun, and Jeannette Bohg. Grac: Self-guided and self-regularized actor-critic. In Conference on Robot Learning, 2022.
Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine Offline rl for natural language generation with implicit language q learning. In International Conference on Learning Representations, 2023.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast -eedeett-descent methods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning, 2009.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1807. 00690, 2018
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv/1812.02666 2018
14

---

Published as a conference paper at ICLR 2026
Gautham Vasan, Mohamed Elsayed, Alireza Azimi, Jiamin He, Fahim Shariar, Colin Bellinger, Martha White, and A. Rupam Mahmood. Deep policy gradient methods without batch updates, target networks, replay buffers. In Advances in Neural Information Processing Systems, 2024.
Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters, Marcello Restelli, and Carlo D D'Eramo. Parameterized projected bellman operator. In AAAI Conference on Artificial Intelligence, 2024.
Théo Vincent, Tim Faust, Yogesh Tripathi, Jan Peters, and Carlo D D'Eramo. Eau de q-network: Adaptive distillation of neural networks in deep reinforcement learning. Reinforcement Learning Journal, 2025a.
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, and Carlo D'Eramo. Iterated g-network: Beyond one-step bellman updates in deep reinforcement learning. Transactions Machine Learning Research, 2025b.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 1992.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems, 2018
Guang Yang, Yang Li, Di' an Fei, Tian Huang, Qingyun Li, and Xingguo Chen. Dhqn: a stable approach to remove target network from deep q-learning network. In International Conference Tools with Artificial Intelligence, 2021.
Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with target network. International Conference Machine Learning, 2021.
15

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
A Experiment Setup
16
Training Time and Floating-pou poitt Operations
17
C Algorithmic Details
18
Additional Experiments
19
D.I Comparison with MellowMax DQN
19
D.2 Ablation Study the Target Update Period
19
D.3 Ablation Study on the Number of Bellman Updates
20
D.4 Automatic tuning the Importance of Each Bellman Update
20
D.5 Pilot Study: Mixed Precision Training of iS-QN
22
E List of Hyperparameters
23
F
Individual Learning Curves
25
F.1
Deep Q-Network with CNN and LayerNorm
25
F.2
Deep Q-Network with IMPALA and LayerNorm
26
F33
Deep Q-Network with CNN
27
F.4
Conservative Q-Learning with IMPALA and LayerNorm
F.5
Soft Actor-Critic with SimbaV2 V2 and BatchNorm
A EXPERIMENT SETUP
Atari setup We build our codebase fol- lowing Machado et al. (2018) standards and taking inspiration from Castro et al. (2018) codebase. Namely, signal to terminate an episode instead of the life signal. The input given the neu- ral network is concatenation of 4 frames in grayscale of dimension 84 by 84. To get new frame, we sample 4 frames from the Gym environment (Brockman 2016) configured with no frame-skip, and apply a max pooling operation on the 2 last grayscale frames. We use sticky actions to make the environment stochastic (with p 0.25).
Atari games selection
VideoPinball Y
Boxing
Breakoul
the game over
Demon Attack
CrazyClimber
Assault
Ipacelnvaders
Y
Fishing
Asterix
Enduro Y
Selected games the CNN experiments
Pong
Tutankham
al.,
Qbert
Selected games for the IMPALA experiments
MsPacmaa
BeamRider
10
15
20
Human-normalized Score of DQN at 200M Frames
Figure 10: The Atari games selected for the experiments of this paper were chosen to cover a variety of normal- ized returns obtained by DQN after 200M frames. To lower the computational budget of the experiments with the IMPALA architecture, we reduced the set of games to 10 by removing 5 games, while maintaining diversity,
Atari games selection Our evaluations on the CNN architecture were performed on the 15 games recommended by Graesser al. (2022). They were chosen for their diversity of Human-normalized score that DQN reaches after being trained on 200 million frames, as shown in Figure 10. As the IMPALA architecture increases the training length, we removed 5 games, while maintaining diversity the final scores to reduce the computational budget. For the offline experiment, we used the datasets provided by Gulcehre et al. (2020). As the game Tutankham is not available in the released dataset, we replaced with Qbert, indicated with an asterisk in Figure 10. The codebase is adapted from the code released by Vincent et al. (2025a).
Deep Control suite setup Our codebase follows the implementation details of Lee et al. (2025). Before running the experiment presented in Section 5.3, we took special care that our codebase reproduces the evaluation performance shared by the authors. As takeaway from this exercise, note that the precision with which the state and reward are normalized matters, using float32 leads to lower performance than using float64. We invite interested readers to examine our code for more details. We emphasize that the performances reported in this work correspond those
16

---

Published as a conference paper at ICLR 2026
collected during training, not the ones obtained during separate evaluation phase, as they are closer to the initial motivation behind online learning (Machado et al., 2018).
Wordle setup Our codebase is fork of the repository shared by the authors (Snell al., 2023), from which we implemented the target-free and the iterated Shared Features approaches. We refer to the original paper for extensive details about the setup. Every algorithm was given budget of 600 000 gradient steps. This value was chosen by running TB-ILQL until the performance reported in Table of Snell et al. (2023) was reached.
Streaming RL setup We follow the implementation choices made by Elsayed et al. (2024). We detail here the differences with the classical Atari setup described earlier. Namely, the source of stochasticity is replaced from sticky actions to random number of NO OP actions chosen uniformly between and 30. The life loss signal is used for episode termination instead of the game over signal. The last hidden linear layer is composed of 256 neurons instead of 512. The neural network weights are initialized with a sparsity level of 90%. The optimizer's hyperparameters are k and the default learning rate is 1. The trace coefficient is set to À 0.8.
Computing the Area Under the Curve For each experiment, we report the normalized IQM AUC. For that, we first compute the undiscounted return obtained for each epoch, averaged over the episodes, as advocated by Machado et al. (2018). Then, we sum the normallied returns over the epochs and compute the IQM and 95% stratified bootstrap confidence intervals over the seeds and games. Finally, we divide the obtained values by the IQM the target-based approach to facilitate the comparison. The Human--ormalized scores computed from human and random scores that were reported in Schrittwieser et al. (2020). As discussed in Section 5, the normalized AUCs can also be interpreted as the average performance gap between the considered algorithm and the target-based approach. Indeed, dividing the two sums of performances across the training is equivalent to dividing the two averages of performances across the training because the normalizing factors cancel out.
B TRAINING TIME AND FLOATING-POINT OPERATIONS
DQN
SAC
ILQL
1e8
le10
1e12
FLOPs
4.3
1.4
3.3
2.2
0.7
1.6
0.0
0.0
0.0
CNN
IMPALA
SimBaV2
GPT-2
Run time
(in hours)
15
1.1
54
7
0.6
27
0.0
TB-DON
TE-DQN
K=1
K=49
TB-SAC
TF-SAC
TB-ILOU
TF-IQQU
iS-DQN
iS-SAC
iS-ILQL
Figure 11: While TE-DON and iS-DQN require fewer parameters, their training time is similar to TB-DQN since each algorithm uses similar amount of computation, as indicated by the number of floating-point operations (FLOPs) per gradient steps. Left: All algorithms on the Atari benchmark require similar amount of FLOPs and training time. Middle: As reported in Figure 25, the target-based approach does not benefit from BatchNorm for the DMC benchmark. This is why TB-SAC does not use BatchNorm and therefore has lower amount of FLOPs compared to the other approaches. Importantly, the difference training time between the algorithms less visible across the algorithms. Right: Thanks to the way the embeddings are computed, the target-free approach and is-ILQL can compute the TD error from a single pass through the neural network, which lowers the training time.
The presented approach is designed to reduce the memory footprint of basee methods, while performing better than the target-free approach. In Figure 11 (bottom), we report the training time hours required by all algorithms. On the top row, we report the number of floating point operations (FLOPs) required by all algorithms to perform on an NVIDIA GeForce RTX 4090 Ti with the game Asterix for the DQN experiments, and with the task Dog-walk for the SAC experiments. As expected, all algorithms require the same training time
gradient step. Computations were made
17

---

Published as a conference paper at ICLR 2026
and FLOPs because the same amount of computation is needed. Indeed, forward pass through the network for estimating the value of the next state is necessary to compute the temporal-ditee -ifferecee error. We note two exceptions. First, for experiments with SAC, the amount of FLOPs is reduced for the sarget-based approach, as it does not use BatchNorm. However, the difference in training time remains small. Second, for the experiments with II.QI., the training time for the target-free approach and iS-ILQL is smaller than that of the target-based approach since only one forward pass is required for computing the temporal difference instead of two forward passes. This results in the target-free approach and is iS-ILQL having a small training time. While this reduction is not visible in the amount of FLOPs per gradient steps, we verified that the amount of FLOPs per loss computation (forward pass only) is indeed lower: TB-ILQL: 3.3 x 10"H FLOPs, TF-ILQL: 2.0 10 FLOPs, iS-ILQL K 1: 2.1 x 1010 FLOPs, and iS-ILQL K = 2.5 x 10S0 FLOPs.
We point out that when computing the algorithm's runtime, the environment steps are simulated, which does not reflect the real-world duration. For example, an episode of an Atari game can be executed significantly faster in simulation than in real life. This means that if real-world durations were taken into account, sample-efficient algorithms would achieve better performance earlier than less sample-efficien methods.
C ALGORITTMIC DETAILS
Aggregating individual losses In Equation I, we define the loss of iS-QN the sum of losses over each Bellman iteration. Other ways of aggregating the losses are possible. Nonetheless, we decided to stick the version proposed by Vincent al. (2025b) and leave this investigation for future work. We provide a first alternative in Section 5.3 that provides a performance boost by discounting the following terms by a factor of 0. 2.25. While is true that taking the sum of temporal differences increases the magnitude of the loss, it has a different impact on the updates than simply multiplying the learning rate by the number of terms in the loss. Indeed, the Adam optimizer (Kingma & Ba, 2015) first normalizes the gradient with a running statistic before applying the learning rate. Therefore, changing the aggregation mechanism has greater impact on the direction of the update than on its magnitude. This is why we do not compare iS-QN against baselines instantiated with different learning rates.
Following Vincent et al. (2025b), at each environment interaction, an action is sampled from a single head chosen uniformly as shown in Line 3 in Algorithm 1. The authors motivate this choice by arguing that it allows each Q-function to interact with the environment, thereby avoiding passive learning, identified by Ostrovski et al. (2021). This choice is further justified by an ablation study (see Figure 19 in Vincent al. (2025b)) demonstrating stronger performance against another sampling strategy consisting of sampling one head for each episode, as proposed in Osband al. (2016).
Sampling actions
In the experiment on continuous control (Section 5.3), the policy network is used to sample actions. To align with the choice of computing the discounted of temporal differences, the critic estimate in the policy loss is calculated as the average discounted prediction over the sequence of Q-predictions given by the heads. The experiment on the language task (Section 6) also uses a policy network to sample actions, but weighs each prediction with the predicted advantage from the critic. To align with the choice made for the experiment on continuous control, the average over the weights corresponding to each head is computed to obtain single scalar value to weight each action probability.
18

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
In this section, we train each agent for 40M frames rather than 100M to reduce the computational budget.
D.1
COMPARISON WITH MELLOWMAX DQN
Breakout
Seaquest
Area Under the Curve
IQM retur
Area Under the Curve
IQM return
IS-Mellow
K=9
K=1
2
0.05
TF-Mellow
Madiow K T1-M-lbo
Mallow K-5 TE-Miloo -TB-Milio
TB-Mellow
0
0.00
80
100
10
20
30
40
60
80
100
20 30 40 Number of Frames (in millions)
in % of TB-Mellow
Number of Frames (in millions)
in % of TB-Mellow
CNN LN
Figure 12: When using the MellowMax operator, the presented approach (S-Meelow) effectively reduces the performance gap between the target-free approach (TF-Mellow) and the target-based approach (TB-Mellow).
We combine the MellowMar Max operator (Kim et al., 2019) with the presented idea. We select the games that were presented in the original paper (Kim al., 2019), and the same temperature coefficients (w = 000 for Breakout, and w 30 for Seaquest). Remarkably, in Figure 12, the performance gap between the target-free free and target-based approaches is reduced by the presented approach.
D.2 ABLATION STUDY ON THE TARGET UPDATE PERIOD
TB-DQN
iS-DQN K
iS-DQN K=9
Area Under the Curve
40
35
30
25
6
10
12
4
8
10
12
8
10
12
Target update period T (x1K)
CNN LN
Game: Tutanklam
TF-DQN
Figure 13: Ablation study on the target update period T for the game Tutankham. iS-DQN's performance remains stable when varying the target update period. This shows that it not sensitive to the target update period for this game.
We perform ablation study the frequency which the heads of the iS-DQN agent updated. This hyperparameter plays a role similar to the target update period T in DQN, allowing both algorithms to propagate rewards, explained in Section 4. In Figure 13, report the performance of the sarget-based approach and the presented approach for 5 different values of T on the game Tutankham. As reminder, T 8000 the default value used in Section 5. We also report the performance of the target-free variant as horizontal line since this algorithm does not use this hyperparameter. Interestingly, the performance iS-DQN remains similar across different values of T. demonstrating its robustness to this hyperparameter in this game. Remarkably, iS-DQN K = 9 outperforms both farget-free and target-based variants across all target update periods.
19

---

Published as a conference paper at ICLR 2026
D.3
ABLATION STUDY ON THE NUMBER OF BELLMAN UPDATES
Width x 0.5
Width x 1.0
Width 2.0
Km49
K=39
iS-DON
K=29
K=19
K=9
K-1
TF-DQN
TB-DQN
90
100
110
90
100
110
90
100
110
CNN + LN
Area Under the Curve (in of TB-DQN)
CazyClintar T'ranhbur
Figure 14: Ablation study the number of Bellman updates learned in parallel K and the size of the shared features. iS- iS-DQN achieves similar performance for large values of K and consistently outperforms the target-free approach.
We analyze the sensitivity of iS-DQN with respect to the number of Bellman updates K learned in parallel. As discussed in Section 5.1, the richness of the shared features can affect the behavior of the method. To account for this aspect, we vary the size of the last linear layer from half the default value (256 neurons, "Width 0.5") to twice the default (1024 neurons, "Width 2.0"). To perform this study, we choose the games CrazyClimber and Tutankham as the performance of iS-DQN on those games for the default setting (512 neurons, "Width 1.0") reflects the overall performance presented in Figure on the 15 games. In Figure 14, we report the performance of DS- for the three different settings with the number of heads varying from (K 1) to 50 (K 49). As expected, the performance of iS-DQN is similar for large values of K (K 2 9). Importantly, every value of K' results in improved performance compared to the target-free variant, even if slight decrease in performance occurs at the largest values of K. This is expected, as the potential of learning multiple Bellman updates in parallel can be better realized with the higher representational capacity.
D.4 AUTOMATIC TUNING OF THE IMPORTANCE OF EACH BELLMAN UPDATE
In this section, we explore a way of tuning the importance of each Bellman update during training. For that, propose weighting each term in the loss with different coefficient. We note the coefficient (a Intuitively, more weight should be given to a Bellman update that produces a gradient aligned with the gradients of the other Bellman updates. We will show that meta-learning those coefficients using the concept of meta- grediegr reinforcement learning (Xu et al., 2018) provides a natural way achieve this goal. Indeed, the update rule coming from the meta-gradient algorithm of a coefficient og linked to the tt Bellman update, depends on the dot product between the gradient of the loss of the th Bellman update and the gradient of the sum of all Bellman updates:
How well aligned will the gradient of the loss of the 8t head be with its current value?
Og O Aado Vulaje FNN (0(a)s, ((0)a-1) Van Fa T
(Bc, 0x-1)
K
+ dado Vu(a)
Σ
TV TV CoN (Os, 0x-1)
How well aligned will gradient overall loss shared paramoters be with the value of the gradient of the 1tth head?
where d = r.) is sample, is the vector of coefficients (ax (aw) ((a) is the parameters after one gradient step starting from 0, w is the shared parameters, we is the parameters of the 4h head such that 0x = (w, wx), Ag is the learning rate the parameters, and A is the meta learning rate.
In the following, we define the meta-ootimization problem, analyse the empirical results, and finally derive Equation 2. To define the meta-optimization problem, first stress the de- pendence of the learned parameters on the meta-parameters by noting them as a function of oz (*(a), Therefore, the inner loop of the meta-gradient optimization problem is defined as
20

---

Published as a conference paper at ICLR 2026
(a)) ming Σ 1= 0x axLa (1k,
0x-1)) This leads to the following optimization problem:
K
K
min
QN
(a)x-11 s.t. (a)a) min
axLa (Ok: 0k-1)
LO
MsPacman
Asterix
Breakout
ag
Bellman update coefficients
(48
(77
A6
20
40 1 Number of frames (in millions)
40
20
20
40
(Z5
(44
Meta iS-DON K=9-
iS-DQN K=9
iS-DQN K=1
90 100 110 120 90 100 110 120 90 100 110 Area Under the Curve (in % of TB-DQN)
120
[1)
CNN + LN
Figure 15: Top: Evolution of the meta-learned coefficients for each Bellman update during training
for 3 Atari games. Remarkably, the coefficients converge to similar values, indicating that setting
equal weights for each Bellman update is good static choice for this setting. Bottom: Meta iS-DQN K = 9 performs on with iS-DQN K = 9, which is coherent with the values of the learned coefficients.
We evaluate this approach for K on Atari games, selected for their diversity in human- normalized scores (see Figure 10). We parameterize the meta meta-parameters (aw) as logits (zx) and apply the softmax function such that the loss a convex combination the individual terms. In Figure 15 (top), we report the learned meta-parane pattmpparamtees, which we use as proxy for the importance of each Bellman update during training. Remarkably, after 10M frames, the coefficients converge to identical values, giving equal importance to each Bellman update. This indicates that setting equal weights for each Bellman update is a good static choice for this setting, as proposed in Equation 1. This explains why in Figure 15 (bottom), Meta iS SSDDQN K 9 performs par with iS-DQN K 9. as iS-DQN performs better than iS-DQN K 1.
9 uses equal static weights during training. Importantly, Meta is iS-DQN
To derive Equation 2, we assume that the meta-parameters only influence one update step, as in Xu al. (2018). We also simplify the computations by choosing SGD the optimizer instead of Adam, noting ((a) the parameters after one gradient step, starting from 0. The meta-parameter update, for sample d (s,a,r,") is
K
K
0k + as - Ao V Ork Σ
ON (())) = a - Aa
Σ
ON
(((aa),,((a)))))
(3)
oz
1=1
1=1
where Van (0(a), 0()); 0())
0( (
ON (f(a):
Vv.
+ Vo(o), CN (B(a), 0()) )V (())
(4)
Equation 4 comes from the chain rule. We remark that for any k. Vo. C. the target network is frozen. We split the parameters of the heads from the shared parameters in Section 4: ((a); = (w(a), w())), where represents the shared parameters and wi the parameters of the 7th head. We now focus on the gradient of the 7th head parameters with respect to ax:
CON (0x, 02-1) = 0, since
Va, (a:); (8) Vay
wi
V
Σ
aj
CON (0,, (0,,0)-1)
(
-o
Ve CON
(2
(0)
j=1
1=1
(0,, CO 0,-1)) (d) - C0N 0,-))
Ay Voz oz
a. V
(O,
(5)
(a) comes from the definition of w(a);, (b) uses the fact that w; and À do not depend oye, (c) uses the fact that the parameters of the 7th head do not influence the parameters of the other heads, and (d) uses the fact that Va, 0g 1 kmi. Similarly, derive the gradient of the shared parameters w()
21

---

Published as a conference paper at ICLR 2026
with respect to arg
k
k
V Va (a)
Vou
w - Ao v Σ
La (0,, (0):
S -2
Σ
ag V ON
(0,,
(3-))
j=1
( -ho A0 Σ V aa V. (0,.
V
Ik=
(0,,
V
CON (Ox: Ok-1).
(6)
(e) comes from the definition of w(a). (f) uses the fact that w and Xg do not depend on og and the linearity of the sum, (g) uses the linearity of the gradient operator, (h) uses the fact that
Va, aj = ==jz and in (i). we evaluate the indicator function. Plugging Equations and 6 in Equation 4, we obtain:
IN (O(a),
Vo(o ON (f(a);, (())-)))
u(a): ON ((())), w(a)) + Vi (Ola), ((a): V
1 Val)) 27 (0(a); 0 ((0))))0000)))11111)))
(0,, 0-11)
- ON (0(a); -1) (Ox. 6
()) comes from Equation 4, (k) uses the decomposition between the head parameters and the shared parameters, and (1) comes from Equations 5 and 6. Finally, plugging Equation Equation 3 and using the linearity of the dot product, we obtain the result presented in Equation 2:
ON (0(())) (())) V 05 5 + Aa Va(a) f3
wh (2 0k-1)
ON
dado
Vala) ( (((a),, CON (0+.0x-1)
= + Vo 4
ON (O(()) (())-))
(O 0-11)
ON
K ((()))
T
+ +aado
VaLON 02-1)-
D.5
PILOT STUDY: MIXED PRECISION TRAINING OF IS-QN
float16 operations
float32 operations
Area Under the Curve
IQM retum
Area Under the Curve
IQM return
JSDDNN
K=1
2
2
TF-DQN
1
1
TB-DQN
0
0
75
100
125
10
20
30
40
75
100
125
10
20
30 40
in % of TB-DQN
Number of Frames (in millions)
in % of TB-DQN
Number of Frames (in millions)
CNN + LN
Gamos: Breakont, DetoonAtack, Tutsnkhars, Assanli
Figure 16: When reducing the precision the operations performed during the forward and backward passes of the neural network from float32 float16, is -S-QN still bridges the performance gap between the target-free and target- based approaches.
To further reduce the resources required during training, the presented approach be combined with other techniques that also aim at reducing resource requirements. We provide an initial study that combines the idea of performing operations with lower precision to further reduce the memory
22

---

Published as a conference paper at ICLR 2026
footprint (Micikevicius et al., 2018) with the presented approach. Specifically, the computations during the forward and backward passes of the neural network are performed in float16b rather than float32. In Figure 16, show that the performance gap between the target-free and target- -ased approaches remains when the operations are performed with lower precision. Remarkably, iS-DQN == closes the performance gap, and iS-DQN K further boosts the learning speed of the larget-free approach, outperforming the target-based approach when the precision of the operations reduced. Overall, we observe slight decrease in performance across all algorithms, which is expected, as the precision of the operations during the forward and backward passes decreases.
E
LIST OF HYPERPARAMETERS
Our codebase is written in Jax (Bradbury et al., 2018). The details of hyperparameters used for the experiments are provided in Table 1 (Atari), Table (DMC Hard), and Table (Wordle). In each experiment, the same hyperparameters. as those provided in the original target-based approaches are used without further tuning. We note Conva, 2D convolutional layer with C' filters of size x b and stride d, and FC E fully connected layer with E neurons. When added, LayerNorm is placed before each activation function, and BatchNorm is placed after the activation function. Additionally, when BatchNorm is used, the state-action and next state-next action pairs are first concatenated and then passed as a single batch to the network suggested by the authors of CrossQ and CrossQ + WN (Bhatt al., 2024; Palenicek et al., 2025).
Table I: Summary of the shared hyperparameters used for the Atari experiments. The CNN architecture described here. We used three stacked layers of size 32, 64, and 64 with last linear layer of size 512 for the IMPALA architecture (Espeholt et al., 2018).
Shared hyperparameters
Discount factor 7
0.99
DQN hyperparameters
Horizon H
27 000
Number of training
Full action space
No
250 000
steps per epoch
Reward clipping
clip( 1,
Target update
Batch size
32
period T
8.00
Conv 32
Type of the
Torso architecture
Conv 1,
replay buffer D
FIFO
Conv 3.54
Initial number
20 000
FC 512
of samples in
Head architecture
FFC n1 [TB-QN, TF-QN]
Maximum number
1000 000
FC (K + +1) n. [iS-QN]
of samples in D
Activations
ReL.U
Gradient step
4
period G
CQL hyperparameters
Starting
1
Number of gradient
62 500
Ending
0.01
steps per epoch
linear decay
Target update
250 000
2000
duration
period
Batch size
32
Dataset size
5 000
Learning rate
6.25 10
Learning rate
5 x 10 5
Adam c
1.5 x 10
Adam e
3.125 x 10 4
CQL weight
0.1
23

---

Published as a conference paper at ICLR 2026
Table 2: Summary of the shared hyperparameters used for the DMC Hard experiments.
Table 3: Summary of the shared hyperparameters used for the Wordle experiments.
Environment
Environment
Discount factor yy
0.99
Dataset
Wordle Twitter dataset
Horizon H
1000
Discount factor y
0.99
Action repeat
2
Number of tokens
35 (alphabet colors)
Experiments
Rewards
I for incorrect guess, 0 for correct guess
Batch size
256
Policy architecture
Si Actor
Experiments
Critie Torso architecture
V2 Critic
Batch size
1024
GPT-2 small
Policy architecture
FC
(Dropout p = 0.1)
FC aatmss
GPT-2 small
Critic Head architecture
Torso architecture Q, V
[TB-SAC, TF-SAC]
(Dropout p = 0.1)
FC (K + 1) attom
FC 1536
[iS-SAC]
FC n. ITB-ILQL, TF-ILQL]
Activations
ReLU
Head architecture Q
BatchNorm
TF-SAC, iS-SAC
-FC (K + 1) V1,4
Number of training steps
[iS-II.QL.]
500
FC 1536
Soft target update
5 x 10-3
-FC 1 [TB-IL.LL, TF-ILQL]
Initial number of samples in D
Head architecture V
000
-FC (K + +)
Maximum number
[iS-ILQL]
1 000
samples in D
Activations
ReLU
Initial learning rate
1 10-1
Number of gradient steps
600 000
Final learning rate
1 x 10-5
Soft target update
10
Optimizer
Adam
Learning rate
10-5
Optimizer
Adam
Simba V2 hyperparameters
Double
No
IL hyperparameters
Distributional critic
Inverse temperature 3 CQL weight
4.0
101
bins Ratoms
10
24

---

Published as a conference paper at ICLR 2026
F
INDIVIDUAL LEARNING CURVES
F.1 DEEP Q-NETWORK WITH CNN AND LAYERNORM
TB-DQN
TF-DQN
iS-DQN K=9
Performance Curve
Distance to Human Score
Dormant Neurons
1.0
Optimality Gap
IQM Return
% params
0.5
10
0.0
7
20
40
60
80
100
80 100 Number of Frames (in millions)
40
5 10
15
CNN LN
Figure 17: Reducing the performance gap in online RL on 15 Atari games with the CNN architecture and LayerNorm. Left: iS-DQN K
only reduces the performance gap but outperforms the larget-based approach. Middle: iS-DQN annuls the performance gap for the games where the score is below the human level. Right: is- iS-DQN exhibits a lower amount of dormant neurons the beginning of the training compared to the target-free approach.
TB-DQN
TF-DQN
iS-DQN with K=9
VideoPinball
Boxing
Breakout
IQM Return
20
5
5.0
2.5
0
0-
DemonAttack
CrazyClimber
Assault
IQM Return
3
5
2
0
SpaceInvaders
FishingDerby
Asterix
1QM Return
1.5
2
1.0
0.5
0
Enduro
Pong
Tutankham
IQM Return
2
0
0
0-
Qbert
MsPacman
BeamRider
IQM Return
0.50
0.2
0.25
0
0.0
20
40
60
100
40
60
80
100
60
80
100
CNN + LN
Number of Frames (in millions)
Figure 18: Per game training curves of iS-DQN, TF-DQN, and TB-DQN with the CNN architecture and LayerNorm. Except Asterix, iS-DQN outperforms or is par with the target-free approach (TF-DQN).
25

---

Published as a conference paper at ICLR 2026
F.2
DEEP Q-NETWORK WITH IMPALA AND LAYERNORM
TB-DQN
-
TF-DQN - iS-DQN K=9
Performance Curve
Distance to Human Score
Number of Parameters
0.6
100%
Optimality Gap
1QM Return
M parameters
4
0.4
(16 Mh of RAM)
50%
2
0.2
0.0
0
20
40
60
80
100
1
20
40
60
80
100
TF-DON
K=l
Km
IB-DON
IMPALA + LN
Number of Frames (in millions)
iS- IS-DQN
Figure 19: Reducing the performance gap in online RL on 10 Atari games with the IMPALA architecture and LayerNorm. Left: iS-DQN K = 9 is outperforms the target-free approach. Middle: iS-DQN annuls the performance gap for the games where the score is below the human level. Right: iS-DQN requires significantly fewer parameters than the target-based approach while reaching similar performance.
TB-DQN
TE-DQN
iS-DQN K=9
VidcoPinball
Breakout
IQM Return
20
10
0-
CrazyClimber
Assault
IQM Return
5.0
2.5
SpaceInvaders
Asterix
1QM 1QM Retum
10
50
0
Enduro
Tutankham
1QM Return 1QM Return
0
0-
MsPacman
BeamRider
0.75
0.50
0.50
0.25
0.25
20
40
60
80
100
20
40
60
80
100
IMPALA
Number of Frames (in millions)
Figure 20: Per game training curves of iS-DQN, TF-DQN, and TB-DQN with the IMPALA architec- ture and LayerNorm. Our approach outperforms or is on par with the target-free approach (TF-DQN) on all games.
26

---

Published as a conference paper at ICLR 2026
F.3
DEEP Q-NETWORK WITH CNN
TF-DQN
TB-DQN
TF-DQN iS-DQN K=3
TE-DQN with BatchNorm Performance Curve
Performance Curve
Distance to Human Score
2
Optimality Gap
Crasce Antra, Crunyliesher Syudltendee Trackthen VideoPoball
IQM Return
1.0
10M R
0.5
0.0
0
20 40 60 80
20 Number of Frames (in millions)
100
40
60
80
100
20 40 60 80 100
CNN
Num Frames (in millions)
Figure 21: Reducing the performance gap in online RL on 15 Atari games with the CNN architecture. Left: iS-DQN K = 3 significantly reduces the performance gap between the target-free and target- based approaches. Middle: is iS-DQN annuls the performance gap for the games where the score is below the human level. Right: Including BatchNorm the architecture damages the performance on the 5 considered games of the target-based approach. This is why BatchNorm is not included for the experiments with TB-DQN.
TB-DQN
TF-DQN
iS-DQN K=3
VideoPinball
Boxing
Breakout
1QM Return
5
20
5
0
DemonAttack
CrazyClimber
Assault
IQM Return
2.5
2
0.0
0
SpaceInvaders
FishingDerby
Asterix
1QM Return
2
0
0
Enduro
Pong
Tutankham
IQM IQM Return
2
1
0
Qbert
MsPacman
BeamRider
1QM Return
0.2
0.25
0-
0.00
0.0
20
40
60
80
100
20
40
60
80
100
1
20
40
60
80
100
CNN
Number of Frames (in millions)
Figure 22: Per game training curves of iS-DQN, TF-DQN, and TB-DQN with the CNN architecture. Remarkably, iS-DQN outperforms the target-free approach (TF-DQN) on games.
27

---

Published as a conference paper at ICLR 2026
F.4
CONSERVATIVE Q- Q-LEARNING WITH IMPALA AND LAYERNORM
Distance to Human Score
Srank
Dormant Neurons
1.00
Optimality Gap
iS-CQL K=9 TF-CQL TB-CQL
is-CQL
0.75
ES-CQL
TE-QLL
0.50
TB-CQL
10 20
30
40
50
75
80
85
10
15
IMPALA LN Gradient Steps (x 62.5K)
At the End of Training (in %)
Figure 23: Reducing the performance gap in offline RL on 10 Atari games with the IMPALA architecture and LayerNorm. Left: iS-CQL significantly reduces the performance gap for the gameS where the score below the human level. Middle: the end the training, iS-CQL. and ES-CQL. lead to higher srank than the target-free approach, which indicates a higher representation capability, Right: All methods converge to a low amount of dormant neurons at the end of the training.
TB-CQL
TF-CQL
ES-CQL
iS-CQL CrazyClimbor
VideoPinball
Breakout
IQM Return
10
2
0.2
5
0-
0.0
0
Assault
Spacelnvaders
Asterix
IQM Return
0.2
1.5
1.0
0.1
0.5
0.0
Enduro
Qbert
MsPacman
2
0.4
IQM Return
0.5
0.2
1
0.0
0.0
BeamRider
10
20
30
40
50
10
20
40
50
1QM Return
0.4
Gradient steps (x 62.5K)
0.2
0.0
0
10
30
40
50
IMPALA LN
Figure 24: Per game training curves of iS-CL, TF-CQL, and TB-CQL with the IMPALA architecture and LayerNorm. Except on VideoPinball, iS-CQL outperforms or is on par with the target-free approach (TF-CQL).
28

---

Published as a conference paper at ICLR 2026
F.5
SOFT CRROICC WITH SIMBAV2 AND BATCHNORM
TB-SAC
TB-SAC -
TF-SAC
iS-SAC K=9
TB-SAC with BatchNorm
Srank
Dormant Neurons
Performance Curve
75%
10%
1QM Norm Return
50%
5%
25% 25%
%%.0 0.25 0.5 0.75
8.0
0.0
0.25
0.5
0.75 1.0
1.0
0.25 0.5 0.75 1.0
SimbaV2
Environment Steps (in millions)
Environment Steps millions)
Figure 25: Reducing the performance gap in online RL the DMC Hard tasks with the SimbaV V2 architecture and BatchNorm. Left: As opposed to iS-SAC, the target-free approach suffers from low srank, which indicates lower representation capability, Middle: The percentage of dormant neurons remains low during training for all methods, not exceeding 7%. Right: The target- based approach does not benefit from BatchNorm. This why it is not included in the experiments with TB-SAC. Importantly, all algorithms use -orm as described in Lee et al. (2025).
TB-SAC
TF-SAC - iS-SAC with K=9
Humanoid-stand
Humanoid-walk
Humanoid-run
IQM Return
500
500
100
0
0
0
Dog-stand
Dog-walk
Dog-run
1000
IQM Return
500
500
500
0)
Dog-trot
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
IQM Return
500
Environment Steps (in millions)
0
0.0
0.25
0.5
0.75
1.0
SimbaV2
Figure 26: Per task training curves of iS-SAC, TF-SAC, and TB-SAC with the SimbaV2 architecture and BatchNorm. iS-SAC consistently performs better than or on par with the target-free approach. Interestingly, SS-SAC even outperforms the target-based approach on the humanoid tasks.
29


Reviews:
[{"review_id": "kRB3bJf3QY", "reviewer": "Reviewer_siCu", "summary": "Proposes iterated shared Q-learning (iS-QL), which combines iterated Q-learning (prior work) with the novel approach of \"Shared Features\" for the target network. \"Shared Features\" replaces the standard target network with a single linear layer that shares its input features with the final linear layer of the online Q-network. Similar to a standard target network, the new target network head is a delayed copy of the online network head's parameters. This greatly reduces the memory required for training, since there is no longer a full target network, yet mostly retains the improved stability of a standard target network.", "questions": "> mitigate the brittleness of semi-gradient approaches and stabilize learning\n\nIs this referring to two issues, or one issue?\n\n&nbsp;\n\n> Our proposed method, iterated Shared Q-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems, while using a single Q-network, thus being a step forward towards resource-efficient reinforcement learning algorithms\n\nYou might split this sentence into two, and also say \"stepping towards\" instead of \"thus being a step forward towards\"\n\n&nbsp;\n\n> However, replacing look-up tables with non-linear function approximators and allowing off-policy samples to make the method more tractable\n\nYou might specify \"tractable\" more precisely here\n\n&nbsp;\n\n> This ultimately limits the size of the online network due to the constrained Video Random Access Memory (VRAM) of GPUs.\n\nMemory limits also apply to, for example, TPUs\n\n&nbsp;\n\n> We propose storing only the smallest possible part of the target network, i.e., the parameters of the last linear layer,\n\nThis is not the \"smallest possible part of the target network\", because you could for example lag only half of the last linear layer. Even though I would guess that would not work well\n\n&nbsp;\n\n> Iterated Q-Network (Vincent et al., 2025)\n\n\"$Q$-network\" and \"Q-network\" are used inconsistently. It would be better to stick with one\n\n&nbsp;\n\n> The optimal policy of a Markov Decision Process (MDP) with a discrete action space can be obtained by selecting for each state, the action that maximizes the optimal action-value function $Q^*$\n\nThis applies to continuous-action MDPs too\n\n&nbsp;\n\n> This is why Mnih et al. (2015) approximate the optimal action-value function with a neural network $Q_\\theta$, represented by a vector of parameters $\\theta$.\n\nThis confused me for a second, since I misread it as trying to explain why they use a neural network in particular\n\n&nbsp;\n\n> leveraging the contraction property of the Bellman operator $\\Gamma$ to guide the optimization process toward the operator’s fixed point, i.e., the optimal action-value function $Q^*$.\n\nThe Bellman operator is often not a contraction (even when it can provably converge to $Q^*$). See for example Section 11.4.3 \"Comparison between the two convergence proof techniques\" in https://sites.google.com/view/rlfoundations/home\n\n&nbsp;\n\n> Where $\\gamma$ is the discount factor linked to the MDP of interest.\n\n\"the discount factor linked to the MDP of interest\" is a bit unclear, particularly since the discount factor used for training should often ideally be smaller than the discount factor used for evaluation. https://www.ijcai.org/Proceedings/16/Papers/626.pdf\n\n&nbsp;\n\n> Gallici et al. (2025) also develop a method for a streaming scenario, in which they rely on parallel environments to cope with the non-stationarity of the sample distribution\n\nWould it be fair to say they also rely on LayerNorm for this?\n\n&nbsp;\n\n> gradient w.r.t. the\n\nIs there a missing backslash in the LaTeX there? The space after the last \".\" looks a bit large\n\n&nbsp;\n\n> Importantly, we incorporate the insights provided by Gallici et al. (2025) to use LayerNorm (Ba et al., 2016) for the experiments with discrete action spaces, as we found it beneficial, even for the target-based approach. Similarly, we use BatchNorm (Ioffe & Szegedy, 2015), as suggested by Bhatt et al. (2024), to improve sample-efficiency in continuous action settings\n\nThis seems to imply BatchNorm helped more than LayerNorm for continuous-action settings? If so, it might help to make that even clearer.\n\n&nbsp;", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Experimental results suggest that iS-QL indeed increases return compared to target-free algorithms, but with significantly less memory usage than target-based algorithms.\n\nThe Fig 7 results on an LLM show iS-QL getting even higher return than the target-based algorithm, which is extra promising because that experiment is, among the experiments in the paper, debatably the closest to a real-world problem.\n\nLarge variety of experiments (online, offline, continuous control, and language-based RL)\n\nThe hyperparameter tuning protocol is clear (except for some things mentioned in \"Weaknesses\" of this review)\n\nCode is included in the supplementary materials, and at a glance it looks fairly comprehensive", "weaknesses": "Introduces a hyperparameter, $K$, the number of Bellman iterations to learn in parallel. It appears to be mildly sensitive. Especially since, if I understand right, Fig 6 introduces an additional discounting hyperparameter for the iS-SAC $K = 9$ case.\n\nAs made clear in the paper (which is a strength), sometimes iS-QL does not match the returns of standard target-based algorithms.\n\nIt should be made clearer whether the target-based approach is given LayerNorm in continuous-action settings, or no normalization at all. If it is given no normalization at all, that would be unfair since iS-QL is given BatchNorm (which the paper notes does not help the target-based approach).\n\nFig 7 shows 600k gradient steps, but the appendix says 800k. Further, neither value is explained in the paper, and I don't think the original ILQL paper used either of those values.\n\nResults use 5 or 10 seeds per task (but the large variety of experiments greatly mitigates this overall).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8s5B3w4N6", "reviewer": "Reviewer_Hr47", "summary": "The paper proposes an alternative to target networks, namely keeping only a frozen copy of the last linear layer as the “target,” while sharing all upstream features with the online network. Building on this, the authors use iterated Shared Q‑Networks (iS‑QN), a method that uses multiple linear heads that learn consecutive Bellman updates in parallel. Across Atari (online/offline), DMC Hard (SAC), and Wordle (ILQL), this closes or even beats target‑based baselines while using far less memory. The paper also offer some empirical explainations why the method stabilizes learning.", "questions": "1. How does your work compare with the work of Elsayed 2024 [1] and Vasan 2025, which as you mention in your work, remove the need for the target network? When/why should one be preferred over the other? Why did you not compare your method with theirs? \n\nGiven this is my main issue with this work, I would consider raising my score if this is adequately addressed.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The main strength of this work lies in the problem they are solving, which is removing the target network from deep value based RL methods. Their idea can be integrated into DQN code with minimal changes while preserving the memory usage of a target free method and enjoying the stability of a target based method. The experiments appear extensive and well executed.", "weaknesses": "The main weakness lie in this paper's similarity to that of Elsayed 2024 [1] and the follow up by Vasan 2025, which also claims to solve a similar problem, this work would benefit greatly from a more direct comparison with these works. By properly placing your contributions in the context of these works, it would greatly enhance and appropriately detail the contributions of your work. \n\nWhile somewhat orthogonal to this work, it would be nice if the authors could give some theoretical explains (even on simple mdps/toy problems) as to why their method enjoys a best of both worlds (target free memory, target based stability) thought this might be hard with deep networks. \n\n[1] Elsayed, Mohamed, Gautham Vasan, and A. Rupam Mahmood. \"Streaming deep reinforcement learning finally works.\" arXiv preprint arXiv:2410.14606 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SwG6q95gY2", "reviewer": "Reviewer_959t", "summary": "This paper proposes Iterated Shared Q-Network (iS-QN), a simple method that stores only the final linear layer of the target network while sharing all other layers with the online network. Combined with iterated Q-learning, iS-QN learns multiple Bellman updates in parallel, reducing memory use and improving sample efficiency. Experiments across Atari, DMC, and Wordle tasks show that iS-QN bridges or surpasses the performance gap between target-free and target-based RL with minimal additional cost.", "questions": "1. What is the empirical or theoretical trade-off between K and variance/stability?\n2. How does feature sharing interact with normalization layers (LayerNorm vs BatchNorm) beyond empirical observation?\n3. Can the authors provide ablation on the frequency T of head updates, similar to target-network refresh rate in DQN?\n4. Future work: combining iS-QN with network pruning or quantization could yield compelling results for edge RL; a brief pilot experiment would add value.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The notion of retaining only the last linear layer of the target network (shared features) is conceptually simple but impactful. It effectively merges the strengths of both paradigms without introducing significant architectural or computational overhead.\nMoreover, the evaluation spans multiple benchmarks (Atari, DMC, Wordle), architectures (CNN, IMPALA, SimbaV2, GPT-2), and settings (online/offline/continuous).\nThe breadth of validation strongly supports the claim of generality and practicality.\nThe paper achieves significant empirical improvements over many benchmark examples.\nDetailed hyperparameters, FLOP analyses (Fig. 10), and appendices ensure reproducibility.\nCode release and dependency listings are also promised upon acceptance", "weaknesses": "Despite solid empirical results, the paper provides no formal convergence or stability analysis. For instance, how the partial sharing of target parameters affects the contraction properties of the Bellman operator remains unaddressed.\n\nThe optimal number of parallel Bellman updates (K) varies across domains (e.g., K = 9 works for Atari, K = 1 for SAC). The paper admits this (Sec. 6) but does not analyze the scaling law or provide tuning guidance.\nWhile works like Kim et al. (2019), Bhatt et al. (2024), and Gallici et al. (2025) are cited, the paper lacks direct quantitative comparison under identical settings.\nThe FLOP and runtime discussion (Fig. 10) shows that iS-QN doesn’t actually reduce compute per iteration. Clarifying whether sample efficiency translates into wall-clock speed-up would strengthen the practical argument.\nThe exposition occasionally interchanges terms like iS-QN, iS-DQN, iS-CQL, etc., without immediate redefinition. A concise schematic or unified notation table would improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNM40sy5oU", "reviewer": "Reviewer_Bpoz", "summary": "This paper presents iterated Shared Q-Learning, a deep reinforcement learning method designed to bridge the gap between target-free and target-based approaches. The proposed technique stores only the last linear layer of an old Q-network as a frozen target, while sharing the remaining parameters with the current online network. Coupling this shared features innovation with iterated Q-learning reduces the memory footprint compared to classic target-network approaches without sacrificing stability. Extensive experiments (Atari, DMC continuous control, language model RL) show that iS-QN recovers much of the performance lost by omitting target networks, sometimes even outperforming classic methods in both online and offline settings.", "questions": "1. Is there a way to adaptively select or anneal K during training?\n\n2. Can you answer W2? Adding studies or discussions on continual learning/resource-constrained real-world deployments might be a good idea.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated, as reusing all but the final linear layer for target calculation cuts memory usage, making deep RL feasible even on resource-constrained hardware or for large architectures.\n\n2. The framework is tested on diverse discrete and continuous control tasks, as well as language-based RL, complemented with analyses on training dynamics, expressivity of representations, and the stability of regression targets.", "weaknesses": "1. The number of parallel Bellman updates (heads, K) needs tuning, and optimal values may vary with domain and architecture.\n\n2. While memory use is reduced, floating-point operations and wall-clock training time are not improved compared to baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)