Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
Authors: Ahmed Hendawy
Henrik Metternich
Théo Vincent
Mahdi Kallel
Jan Peters
Carlo D'Eramo
Venue: ICLR 2026 Poster (2026)
Keywords: deep reinforcement learning
q-learning
actor-critic
function approximation
Primary area: reinforcement learning
TL;DR: MINTO is a simple, yet effective target bootstrapping method for temporal-difference RL that enables faster, more stable learning and consistently improves performance across algorithms and benchmarks.

Abstract: The use of target networks is a popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as a bootstrapped target is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing a novel update rule that computes the target using the **MIN**imum estimate between the **T**arget and **O**nline network, giving rise to our method, **MINTO**. Through this simple, yet effective modification, we show that MINTO enables faster and stable value function learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and offline RL, as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.

Published as a conference paper at ICLR 2026
USE THE ONLINE NETWORK IF YOU CAN: TOWARDS FAST AND STABLE REINFORCEMENT LEARNING
Ahmed Hendawy 1.2,+ Henrik Metternich'
Théo Vincent Mahdi Kallel
Jan Peters1:2.3,4 Carlo D'Eramo
Technical University of Darmstadt Roobttic Institute Germany (RIG)
2Hessian.Al German Research Center for AI (DFKI)
University of Würzburg
ABSTRACT
The use of target networks is popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as bootstrapped tar- get is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing novel update rule that computes the target using the MINimum estimate between the Target and Online network, giving rise our method, MINTO. Through this simple, yet effective modification, we show that MINTO enables faster and stable value func- tion learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and of- fline RI., as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.
INTRODUCTION
Reinforcement Learning (RL) has demonstrated exceptional performance and achieved major break- throughs across diverse spectrum of decision-making challenges. This success spans domains from mastering complex environments like video games (Mnih et al., 2013; 2015; Hessel et al., 2018) and strategic board games (Silver al., 2016; 2017), solving high-dimensional problems in continuous control (Haarnoja et al., 201 8a; Schulman et al., 2017). Noteworthy applications in- clude learning complex locomotion skills (Haarnoja et al., 2018b; Rudin al., 2022) and enabling sophisticated, real-world capabilities such as robotic manipulation (Andrychowicz et al., 2020; Lu et al., 2025). The foundation of this success lies primarily Deep RL, initiated by the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013), which marked the first successful application of deep neural networks in RL. To make that happen, Mnih et al. (2013) introduce various techniques to mitigate mainly the deadly triad issue (Sutton et al., 1998; Van Hasselt al., 2018) due to the usage of function approximators, off-policy data, and target bootstrapping. Particularly, Mnih et al. (2013) introduce the concept of target networks to mitigate the negative impact of the deadly triad issue (Zhang et al., 2021), where the regression target is computed using lagged copy of the online network to promote stability during training. This prevents the online network from directly chasing its own rapidly changing estimates, thereby mitigating the problem of moving targets. A problem that presents an obstacle towards using fresh estimates from the online network. While the target network has been highly successful in improving stability and convergence, it inherently slows down learning since updates are based on delayed targets. This naturally raises an important question: how can we accelerate learning by leveraging the most recent online estimates while still preserving the stability of the training process?
de) is the corresponding author. Code is available at https: / / thub. com/ AhmedMagdytenndary /MI NTO.
Ahhmed Hendawy fahmea..endawyyee

---

Published as a conference paper at ICLR 2026
Recent studies have suggested that relying solely the online network for target bootstrapping
can improve performance in certain methods (Bhatt et al., 2024: Kim et al., 2019; Kapturowski et al., 2023). Surprisingly, later findings indicate that reinstating target network in these same approaches can further enhance results (Palenicek et al., 2026; 2025; Gan et al., 2021). Building on these insights, we propose a complementary perspective: leveraging both the online and target networks jointly to compute the regression target, thereby aiming to combine the benefits of stability and fast learning. In deep RL, the problem of moving targets especially evident due to the use of neural networks and the resulting uncontrolled fluctuations in the values of unseen states. This issue becomes even more critical in value-basse based methods, where overestimation bias drives the online estimates to steadily increase over time. These issues motivate the search for an appropriate selection criterion that can mitigate the impact of overestimation bias when employing the online network for target computation, hence faster, yet stable learning.
Building on this motivation, we propose MINTO, a simple yet effective technique that computes re- gression targets by taking the MINimum estimated value between the Target and Online networks. By relying on the target network when the online estimate is relatively higher, MINTO reduces over- estimation bias, alleviates the moving- moving-target problem, and ensures stable learning. At the same time, by incorporating the online network when its estimate is lower, MINTO leverages fresher informa- tion, enabling faster learning. Thanks its simplicity, MINTO wide range of off-policy methods, including both value-based and actor-critic algorithms, across on- line and offline RL settings. In this work, we present an extensive empirical evaluation showcasing the benefits of our approach. In addition, we benchmark MINTO against related baselines that ex- ploit online estimates, whether for similar objectives or for different purposes. Our contributions can be summarized as follows: we advocate for a principled combination of online and target networks when computing bootstrapped targets, enabling faster learning while preserving stability. We intro- duce MINTO, a technique that is simple in design yet effective in practice, that computes regression targets as the minimum between online and target estimates, thereby mitigating the moving-targe problem and reducing overestimation bias. We further demonstrate MINTO's broad applicability and effectiveness by integrating it into both value- based and actor-critic methods, across online and offline RL settings. Extensive empirical results show that MINTO consistently outperforms conven- tional target-network designs and related baselines.
be seamlessly integrated into
RELATED WORK
Many works focus on developing simpler RL algorithms that closer the original Q-learning algorithm to benefit from up-to-date Bellman updates. A reasonable approach is to use additional resources or privileged information to remove the target network. For example, Gallici et al. (2025) demonstrated that cleverly using parallel environments mitigates the need for target network. How- ever, real-world applications are often limited to a single process. Interestingly, Lindström et al. (2025) show that constructing the regression target from the online network alone is stable after a pre-training phase using expert data. While this study gives hope that target-free algorithm is feasible, it still relies on additional resources that are not available in the general case. Shao et al. (2022) introduce a cross-entropy method to refine the actions suggested by the policy to construct the bootstrapped estimate. While promising, this technique is limited to the actor-critic setting, and requires additional resources to optimize the landscape defined by the Q-function. In this work, we do not consider having access to additional resources or privileged information in building a general-purpose algorithm for off-policy learning.
are interested
Another approach is to rely on single estimator, thereby reducing the number of parameters used during training. For example, Kim et al. (2019) replace the maximum operator with the MellowMax operator to construct target-free algorithm. However, a following work (Gan al., 2021) suggests that reintroducing the target network is beneficial, indicating that the improvements made by the first approach come from the different nature of the update instead of up-to-date bootstrapped estimates. This makes MellowMax mostly orthogonal to our approach. Kapturowski et al. (2023) explore incorporating recent online estimates through a trust region around the target-network values, yet their method is comprehensive algorithm with multiple interacting components contributing to performance. Some works intervene in the architecture of the function approximator to stabilize the training dynamics when using only one estimator. For example, Li & Pathak (2021) design neural networks processing the state given as input in the Fourier space. Nonetheless, their anal- ysis shows that the approach struggles to handle high-dimensional input spaces. More recently,
2

---

Published as a conference paper at ICLR 2026
Bhatt al. (2024) introduce CrossQ, an actor-critic algorithm that uses batch normalization (loffe & Szegedy, 2015) to account for the distribution match between the state-action pair and the next state-next action pair. A follow-up work demonstrates that this idea can work better with a target net- work (Palenicek et al., 2025), moving away from the objective of constructing the regression target from up-to-date estimates, and making this method orthogonal to our approach. In the following, we also choose to rely on a target network since Vincent et al. (2026) show that target-free methods still underperform compared to target-based methods. Closer our approach are the hybrid methods where the regression target is built from the online network, but an old copy of the online network used to regularize the update. Zhu & Rigotti (2021) introduce Self-correcting Q-learning (ScQL), which evaluates the Q-estimate of the next state at the action that maximizes combination of the target and online network. Piché et al. (2023) regularize the online network predictions with the prediction given by the target network. In Section 5.1, we compare MINTO to those methods. Building the regression target from the online network accelerates the overestimation bias, which degrades performance. The overestimation bias is a long-standing problem in off-policcy RL (Has- selt, 2010), which arises from the interaction between the maximum operator and the stochastic nature of the bootstrap estimate. A first attempt to combat this issue is to learn two independent estimates of the Q-function and evaluate one Q-estimate on the best action suggested by the other estimator to build the regression target (Hasselt, 2010). While this idea is scalable to deep set- tings (Van Hasselt et al., 2016), it leads to underestimation. which hinders exploration. The Clipped Double Q-Learning (CDQ) trick, proposed by Fujimoto al. (2018), is an effective technique for alleviating overestimation bias by taking the minimum between two critic networks. CDQ remains widely used in modern off-policy actor-critic algorithms (Haarnoja et al., 2018b; Lee et al., 2024; 2025; Palenicek et al., 2025) and has inspired several related approaches. For instance, Maxmin Q-learning (Lan et al., 2020) reduces overestimation bias by taking the minimum across several estimators before applying the Bellman maximization operator, but at the cost of training multiple networks. In contrast, our method applies the minimum operator between the target and online network, incorporating the latest online estimates in stable manner to achieve faster learning.
PRELIMINARIES
We define the problem as Markov Decision Process (MDP) (Puterman, 1990),
S, A.A.P. P,r, M, >, where S is the state space, is the action space, P: S A A(S) is the transi- tion distribution where P(s'|s, a) is the probability of reaching state from state 5 after performing action a, r SxA - (R) is the reward distribution, y is the initial state distribution, and 10 is a discount factor. A policy æ S + ((A) maps each state to a distribution over the action space. The policy induces action- value function (" a) E, EE y'r(s,, a,) |s0 s. an = al that defines the expected discounted cumulative return of executing action a in state 8 while following the policy thereafter. The goal of the agent is to find the policy ι that maximizes the expected return starting from some initial state. TD learning methods are suitable set of solutions to this goal, more precisely Q-Learning (Watkins & Dayan, 1992). Q-Learning is an off-policy algorithm that aims to learn the state-action value function Q, known as the Q-function, of the optimal policy Tr" Q= =Q", by utilizing the Bellman optimality equation Bellman (1957):
Q* a) = B|r(s, a) max (ss'sa'))) a'EAA
(I)
Therefore, the optimal policy is defined using this value function by acting greedily at a given state s. π argmax,e (s, a). Q-learning offers a recursive update to approximate the state-action value function Q', given a transition sample (s, a, r, 8') generated by any behavioral policy:
Q(s. + (s) + a(s, a)ly Q(s,a),,
where y = r + max,'eA Q(s'ss) a') is the target value which is computed via bootstrapping with the current estimate, and ass,, a) is the step-size. In the target value, the maximum expected value of the next state is approximated by applying maximum operator on a single estimate. This introduces maximization bias and causes Q-learning to overestimate the state-action values (Hasselt, 2010). When dealing with high dimensional state and action spaces, the tabular form of the value functions is not suitable and function approximators are needed. Particularly, in Deep RL, the state-action value function is modeled by a neural network Qo(s, a) with some learnable parameters A. In Mnih et al. (2013), DQN was introduced as the first successful application of neural networks in RL..
3

---

Published as a conference paper at ICLR 2026
Mnih al. (2013) introduce series of algorithmic components, most notably the introduction of the target network. Target networks Qğ allow a stable learning of value functions by computing the target value using older copy of the online network Qo. y + Y m Qz a). This is done by periodically updating the target parameters 0 to the online parameters 0 every 7 steps. Hence preventing the chase of moving target due to learning Qo from its value that eventually results in unstable learning. Despite the success, this results in a slow learning of the value function as well as the policy due relying on out-dated estimates. This raises question: can we find practical Bellman update rule that results in a stable and fast learning?
4
METHODOL.OGY
4.1
MINTO
The aim of this paper is to identify suitable criterion for incorporating recent online estimates without sacrificing the stability traditionally provided by the target network. This naturally suggests that the online and target networks should work side by side to achieve both fast and stable learning.
To this end, we propose a lightweight and effective technique that leverages online estimates only rapid fluctuations in the target. In
when they are unlikely to introduce harmful overestimation cases where the online estimates are higher than the target one, we instead rely on the target network to ensure stability. Concretely, achieve this by applying the MINimum operator the estimated values of the Target and Online networks, giving rise to our method, MINTO.
Following our method, the bootstrapped target can be computed as follows:
+ max min (Qa(s', a²), Qo(s',a')).
(3)
a'EAA
Given the new regression target, we can compute the regression loss, similar in DQN, as follows:
[(0) (Tyl (o(s,a))²
(4)
where [.] refers to the stop gradient operator, prevent the backpropagation of gradients to the online network presented in the regression target equation (Eq. 3).
In practice, MINTO can be integrated into DQN and, more generally, into any algorithm that relies on temporal-differenc learning (see Appendix B), by modifying only few lines of code. The method is lightweight, requiring only a single additional feedforward pass of the online network on the next state. When implemented in an efficient deep learning framework such as JAX (Bradbury al., 2018), this overhead negligible.
4.2 ANALYZING MINTO
Area Under the Curve
To evaluate the impact of MINTO, we design empirical study that seeks to answer two cen- tral questions: (Q1) Is the minimum operator an appropriate criterion for combining online and target estimates? (Q2) Can the empirical evidence substantiate the rationale for adopt- ing the minimum operator? We empirically ex- amine how the minimum operator employed in MINTO stands relative alternative operators for combining online and target network esti- mates. Our evaluation conducted on 15 Atari games and considers the following baselines: Online Only, which relies exclusively on the online estimates during training: Target Only, equivalent to DQN using solely the target net- work estimates; Max, which selects the larger of the two estimates; Mean, which averages them; Random, which chooses between the on- line and target networks with equal probability; and finally Min, the operator at the core of our proposed method, MINTO.
Min (ours)
Random
Mean
Max
Target Only
Online Only
25
50
75
100
125
in % of DQN
Results of benchmarking the Minimum operator utilized by MINTO against other poten- tial operators on 15 Atari games with the CNN ar- chitecture. We report the AUC metric using IQM and the confidence interval computed across seeds. Methods are trained for 50 million frames.
Figure 1:
4

---

Published as a conference paper at ICLR 2026
Arca Under the Curve
Performance Curve (CNN)
Performance Curve (IMPALA)
IMPALA
MINTO
IQM Return
DQN
MINTO
CNN
DQQ
MINTO (ours)
100
110
120
130
20
40
60
80
100
10
20
30
40
50
in % of DQN
Number of Frames (in millions)
Figure 2: Results of benchmarking MINTO and DQN on 15 Atari games with CNN, and IMPALA with LayerNorm (LN) architectures, All results are reported using IQM and confidence interval on seeds and over all games. Left: We report the AUC metric for MINTO and DQN while utilizing both architectures. Center and Right: We illustrate the performance learning curves of MINTO and DQN under both architecture options.
In Fig. I, we clearly observe the advantage of the minimum operator over the alternative candidates, thereby addressing Q1. As expected, the Online Only function performs poorly, as it relies on rapidly changing hootstrapped targets that lead to unstable training dynamics (see Fig. 33). On the other hand, the Random function fails to match the performance of MINTO, instead converging toward the behavior of Target Only (DQN). The Mean function also aligns closely with Target Only, without offering any additional benefit. Interestingly, the worst results obtained with the Max operator, which represents the opposite selection criterion to ours. In this case, instability arises from severe overestimation bias, which in turn drives large and uncontrolled increases in target values. Taken together, these findings support our motivation for adopting the minimum operator: enables the inclusion of recent online estimates in stable manner by mitigating the overestimation bias they may introduce, thus providing an empirical answer to Q2.
4.3 CONVERGENCE OF MINTO
While empirical study highlights the effectiveness of the minimum operator, it equally impor- tant to establish whether its use is theoretically justified in terms of convergence. Since analyzing convergence under function approximation is notoriously difficult, Our analysis leverages the Generalized Q-learning framework of Lan et al. (2020), which guarantees convergence for update operators satisfying specific non-expansion properties. Owing its close relation to Maxmin Q-learning, we show that MINTO can be cast special case of this general framework, the same spirit as how has been used to analyze other Q-learning variants in their work. This observation leads directly to the following corollary.
focus on the tabular setting.
Corollary 1 (Convergence of MINTO). Let the MINTO operator be defined as MINTO (Q.) maxae minjer Qsalj), where T is set of historical time indices. Under the slandard stochastic approximation assumptions for the learning rate (Assumption in Lan al. (2020)), the Q-values generated by the MINTO update rule converge the optimal action-valuee Q"
Proof. The proof relies on demonstrating that the MINTO operator satisfies the two conditions on the target operator G (Assumption in Lan et al. (2020)). We provide the detailed verification of these conditions in Appendix A.
5 EXPERIMENTAL RESULTS
We now turn to the empirical evaluation, where we demonstrate MINTO's broad applicability and effectiveness by integrating it into both value-based and actor-critic methods across online and offline RL settings, and show consistent advantages over conventional terget-network designs and related baselines.
5

---

Published as a conference paper at ICLR 2026
5.1 ONLINE RL AND DISCRETE CONTROL
We evaluate MINTO on benchmark of 15 Atari games (Bellemare et al., 2013) recommended by Graesser et al. (2022). These games are chosen for their diversity in human nomaa-normmlized scores achieved by DQN after training. All methods employ the standard CNN architecture introduced by Mnih et al. (2015), and our experiments follow the evaluation protocol of Machado et al. (2018) us- ing the official Dopamine hyperparameters (Castro et al., 2018). Additional implementation details and hyperparameters are provided in Appendix C.1.
We begin by evaluating the performance gains of MINTO over DQN (Mnih et al., 2013) terms of both sample efficiency and asymptotic performance. Our objective is to demonstrate that MINTO not only accelerates learning by incorporating fresher estimates from the online network but also achieves higher final performance. To capture both aspects in single measure. we report the Area Under the Curve (AUC) metric, alongside learning curves for MINTO and DQN. To further assess the method's s effectiveness with more advanced architectures,
extend the comparison to the IM-
PALA architecture (Espeholt et al., 2018; Castro et al., 2018), evaluating the same metrics. In this setting, we additionally apply LayerNorm (LN), motivated by prior work showing its potential for improving learning stability and performance (Lee et al., 2024; Nauman et al., 2024; Gallici et al., 2025: Vincent al., 2026).
In Fig. 2, we observe a clear advantage of MINTO over DQN. Specifically, MINTO achieves an improvement of approximately 18% in AUC when using the vanilla CNN architecture, and about 24% when employing IMPALA with LN. The performance curves in Fig. 2 (center and right) further illustrate MINTO's superior sample efficiency and higher asymptotic performance. These results suggest that incorporating more recent estimates from the online network accelerates learning while also improving final performance.
The idea of leveraging the online network alongside the target network for computing re- gression targets is not confined to MINTO. Prior work has explored alternative ways of uti- lizing the online network for different purposes. is therefore important to evaluate the effec- tiveness of our selection criterion, the minimum operator, against established methods from the literature. In this study, we compare against three representative baselines: Double DQN (Van Hasselt al., 2016), Functional Regular- ization DQN (FR-DQN) (Piché et al., 2023), and Self-correcting DQN (SeDQN) (Zhu & Rigotti, 2021). All methods are implemented with the CNN architecture, and we report ag- gregate results over all 15 games using the AUC metric. In Fig. 3, we observe that MINTO consistently outperforms all baselines, includ- ing methods such as Double DQN and ScDQN that are specifically designed mitigate overestimation. This indicates that combining the online network with the minimum operator provides an especially effective mechanism for addressing this issue. Furthermore, MINTO achieves clear gains over FR-DQN, which relies entirely on the online network for target bootstrapping while regularizing its updates using a fixed network (target net- work). This highlights the role of our selection criterion, the minimum operator, in regulating the contribution of online estimates. It is also worth noting that methods such as ScDQN and FR-DQN require additional hyperparameters, whereas MINTO introduces none.
Area Under the Curve
MINTO
Maxmin DQN
ScDQN
FR-DQN
Double DQN
DQN
100
110
120
in % of DQN
Figure 3: Results of benchmarking MINTO against related baselines on 15 Atari games with the CNN architecture. We report the AUC metric using IQM and the confidence interval computed across 5 seeds.
Given the similarity of our method to the ensemble-based approach Maxmin DQN (Lan et al., 2020), which also employs a minimum operator, it is essential to benchmark MINTO against this prior work. Maxmin DQN applies the minimum operator across multiple estimates produced by an ensemble of target networks, with the goal of reducing the overestimation bias introduced by the maximum operator in the Bellman optimality equation. For this comparison, we also report results in Fig. 3 using the AUC metric. Although Maxmin DQN relies on an ensemble of Q- Q-functions (N = in our study), MINTO achieves better performance. This demonstrates the advantage of leveraging

---

Published as a conference paper at ICLR 2026
Arca Under the Curve
Performance Curve
Breakout
Online Selection Ratio
0.6
IMPALA
1QN+ MINTO
JOM Return
IQN
0.4
CNN
1ON+ MINTO
CNN
0.2
IMPALA
IQN
IQN
INN+MINTO (ouns
0.0
100
105
110
20
40
60
80
100
40
60
80
100
in % of IQN
Number of Frames (in millions)
Figure 4: Results of benchmarking MINTO+IQN and IQN on 15 Atari games in the online RL setting with the CNN, and IMPALA with LayerNorm (LN) architectures. All metrics are computed over 5 seeds. Left: We report the AUC metric computed by the IQM and its confidence interval. Center: We demonstrate the performance learning curves of both methods in terms of IQM return and confidence interval. Right: We report the frequency of selecting the online estimate during the course of training the Breakout game, considering the mean and standard deviation.
up-to-date estimates from the online network, in combination with the minimum operator, to miti- gate overestimation bias that can arise from blindly incorporating online estimates. Simultaneously, MINTO avoids the additional memory and compute overhead required by Maxmin DQN.
5.2
DISTRIBUTIONAL RL
We also evaluate MINTO within the context of distributional RL (Bellemare al., 2017), ad- vanced value-based paradigm that has achieved state-of-the-art performance on the Atari bench- mark the online RL setting (Castro al., 2018). Specifically, we consider Implicit Quantile Networks (IQN) (Dabney et al., 2018), a distributional RL method that employs quantile regression to approximate the entire return distribution rather than only its expected value. IQN accomplishes this by learning an implicit quantile function for the state-actis follow the implementation guidelines provided by Castro al. (2018) when benchmarking IQN on the Atari suite. Comprehensive implementation details and hyperparameter settings are reported in Appendix C.2.
values. For our experiments, we
We evaluate the effect of integrating MINTO into IQN by modifying its target computation ac- cording to our proposed technique. Appendix B provides further details, including pseudo-code illustrating the changes introduced (see Alg. 2). We benchmark MINTO+IQN against the original IQN on 15 Atari games using the CNN, and the IMPALA with LN architectures. In Fig. 4 (left), we report the AUC metric for both methods, which captures improvements in both learning speed and asymptotic performance. Under both architectures, MINTÓ enhances IQN by approximately 7% according this metric. Fig. 4 (center) further presents the learning curves, illustrating consistent gains in sample efficiency and final performance. While the improvement smaller than in the DQN case (and in later cases we consider), these results clearly demonstrate that MINTO enhances the performance of IQN, establishing it as a general and effective improvement even for state-of-the-art distributional methods.
To verify that the online network is selected frequently during training, we track the online selection ratio and report for a representative game, Breakout, Fig. 4 (right). The results show that the target network dominates in the very early stages of training, while the use of the online network gradually increases as training progresses. Later in training, the online network is selected approx- imately 45% of the time. Each point the curve corresponds to the average online selection ratio between two consecutive target network updates, during which the parameters the online network increasingly diverge from those of the target network.
5.3
OFFLINE RL
The applicability of our method extends beyond online RL. Given its simplicity, MINTO can be readily incorporated into offline RL methods by modifying the Bellman update rule, allowing to
7

---

Published as a conference paper at ICLR 2026
Area Under the Curve
Performance Curve (CNN)
Performance Curve (IMPALA)
1.0
IMPALA
COL
1.5
IQM Return
CQL
1.0
CNN
CQU-+MINTO
0.5
COL
CQL+MINTO (ours)
0.0
100
150
200
250
0
10
20
30
40
50
10
20
30
40
50
in of CQL
Gradient Steps (x 62.5K)
Figure 5: Results of benchmarking CQL and CQL+MINTO 14 Atari games with CNN, and IMPALA with LayerNorm (LN) architectures. All results are reported using IQM and confidence interval on seeds and over all games. Left: We report the AUC metric for both methods while utilizing the two architectures. Center and Right: We illustrate the performance learning curves of CQL and CQL+MINTO after employing both architecture options.
investigate its potential in this setting. Offiin RI. aims to learn an optimal policy from static dataset of previously collected experience, without any further interaction with the environment. central challenge in this paradigm is the distributional shift problem: the learned policy may query the Q- function on state-action pairs absent from the dataset. Overestimation of these outtof-distributtin (OOD) actions can then misguide the policy toward suboptimal behaviors.
We consider popular offline RL algorithm, Conservative Q-Learning (CQL), which regularizes Q-function learning by penalizing OOD values while encouraging in-distribution values. To inte- grate MINTO into this framework, we simply modify the computation of the regression target using Eq. (see Alg. 3). As in the online setting, we the hyperparameters from Castro et al. (2018). Additional implementation details and hyperparameters are provided in Appendix C.3.
In Fig. 5, we present the results of evaluating original CQL on 14 Atari games using both the vanilla CNN architecture and the IMPALA archi- tecture with LN. For the offline setting, we use the datasets provided by Gulcehre al. (2020). We note that one game, Tutankham, is excluded from our evaluation since it is not available in the released dataset.
method, denoted CQL+MINTO, against the
The results of our experiments demonstrate clear benefit of applying MINTO to offline RL. MINTO consistently improves the performance of CQL terms of both sample efficiency and final performance, as reflected in the AUC met- ric (see Fig. 5 (Ieft)) and the learning curves (see Fig. 5 (center and right)). In particular, MINTO tremendously boosts the performance of CQL with a CNN architecture by roughly 125% in AUC. Although the performance gain smaller with the IMPALA with LN archi- tecture, MINTO still improves CQL by around 20%, with clearer sample-efficiency advan- tage visible in Fig. 5 (right). The improvements are substantial, highlighting the central role of recent online estimates, aspect overlooked by the original formulannt,,,onn, as well as the effectiveness of the minimum operator in mit- igating the overestimation introduced by these estimates.
Area Under the Curve
CQL+MMNTT-
Maxmin COL -
CQL+MINTO
ScCQL
FR-CQL
Double COL-
COL-
100
150
200
250
300
of CQL
Figure 6: Results of benchmarking MINTO
against related baselines on 14 Atari games in the offline RL setting using the CNN architecture. We report the AUC metric using IQM and the confi- dence interval computed across seeds.
Building on our comparison in Fig. 3, we adapt the online RL baselines to the offline RL setting by using CQL as the underlying RL algorithm. This integration is straightforward due to the close relation between CQL and DQN. As shown in Fig. 6, MINTO consistently outperforms all basclines that rely single estimator (N 1), mirroring observations in the online RI. setting. How-
8

---

Published as a conference paper at ICLR 2026
SimbaV1
SimbaV2
SimbaV1+MINN (ours)
SimbaV2-MINTO (ours)
MuJoCo (5 Tasks)
HBench (14 Tasks)
DMC-Hard (7 Tasks)
1QM Norm Return
1.5
750
0.4
1.0
500
0.2
0.5
250
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0,6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
IQM Norn Return
750
0.75
0.2
500
0.50
0.1
250
0.25
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Enviromment Steps (in millions)
Figure 7: Results of evaluating the impact of MINTO on SimbaV2 (top) and SimbaVI (bottom) on three continuous control benchmarks: MuJoCo (left), HBench (center), and DMC-Hard (right). We show the performance curves of all methods each benchmark while reporting normalized IQM return and confidence interval computed on 10 seeds.
ever, unlike the online setting, Maxmin CQL, which employs two estimators (N = 2), achieves notably higher performance than MINTO. We attribute this to the conservative nature of Maxmin CQL, whose cautious updates are less detrimental in the offline scenario compared to the online case where exploration plays major role. This observation motivates to investigate whether introducing fresh estimates through MINTO into such strong baseline can yield further gains. By incorporating recent online estimates into the target computation, find that MINTO indeed enhances the performance of Maxmin CQL, as illustrated in Fig. 6.
5.4 ONLINE RL AND CONTINUOUS CONTROL
Similarly, MINTO is not restricted to value-based methods. In off-policy actor-critic algorithms, where dedicated network models the policy, and the Q-function (critic) is trained using boot- strapped targets from a target critic network, our proposed update rule can also be applied. There multiple ways incorporate MINTO into actor-critic methods, depending in particular on the number of critic networks employed. We defer a discussion of these design choices to Appendix B.
To examine the effect of incorporating up-to-date estimates in the actor-critic setting, we adopt two recent architectures that are used in combination with Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), namely SimbaV1 (Lee al., 2024) and SimbaV2 (Lee al., 2025). SAC is widely used actor-critic algorithm that seeks to learn an optimal stochastic policy by encouraging explo- ration through entropy maximization. In contrast, SimbaV1 and SimbaV2 are recently proposed architectures designed to scale performance with model size by leveraging the concept of simplicity bias, implemented in practice through normalization layers and residual connections. Alg. 4, we highlight the modifications introduced to SAC in order to integrate MINTO.
We benchmark SimbaV1 and SimbaV2 with and without our proposed method, MINTO. The eval- uation is conducted 26 continuous control tasks, including both manipulation and locomotion, drawn from three different benchmarks: MuJoCo (Todorov al., 2012), Humanoid Bench (HBench) (Sferrazza et al., 2024), and the DeepMind Control Suite Hard (DMC-Hard) (Tassa et al., 2018). For each actor- method, we report the aggregated performance curves on three benchmarks. Additional details the experimental setup and hyperparameters are provided in Appendix C.4.
In Fig. 7, we observe a clear improvement in sample efficiency when applying our method on the MuloCo and HBench benchmarks. In contrast, performance DMC-Hard comparable to the

---

Published as a conference paper at ICLR 2026
base algorithm (SimbaV1) or slightly lower (as with SimbaV2), Overall, however, MINTO yields consistent positive impact across all benchmarks and both actor-critic methods, suggesting the value of incorporating recent online estimates when computing regression targets in continuous control and actor-critic settings. Detailed results are in the Appendix D.6 including an additional study on CrossQ+ WN (Palenicek al., 2025).
CONCLUSION
We introduce MINTO, a simple bootstrapping rule that combines online and target networks by tak- ing their minimum estimate. This design mitigates the overestimation bias that can arise from blindly incorporating online estimates into target computation, thereby alleviating the moving- prob- lem, while still exploiting recent updates for faster learning. Across online. offline, value-basse based, and actor-critic methods, MINTO consistently improves sample efficiency and final performance with- out introducing additional hyperparameters and with only negligible overhead. These results estab- lish MINTO as a practical and effective alternative to conventional target-network designs, pointing toward a promising direction for advancing stable and efficient deep RL. Although MINTO consis- tently improves performance across diverse settings, there are natural trade-offs to consider. Rely- ing exclusively the minimum operator may, in some cases, be overly conservative in low-noise environments, leading to slight underestimation. In addition, by dampening optimistic estimates, MINTO may interact with exploration strategies in ways that are not yet fully understood. These observations open promising avenues for future research, such as adaptive operator selection that dynamically balance online and target estimates based on uncertainty or learning dynamics. Beyond these considerations, a separate line of future work lies in testing MINTO in additional challenging scenarios, such as multi-task and multi-agent RL, to assess its scalability and broader applicability.
ACKNOWLEDGMENTS
The authors would like to thank Daniel Palenicek and Florian Vogt for their valuable support in conducting the study on CrossQ+WN (Palenicek et al., 2025). This work was funded by Hessian.Al through the project 'The Third Wave of Artificial Intelligence 3AT by the Ministry for Science and Arts of the state of Hessen. This work was also funded by the Deutsche Forschungsgemein- schaft (DFG, German Research Foundation) under Germany's Excellence Strategy (EXC-3057/1 "Reasonable Artificial Intelligence', Project No. 533677015). Caleulations for this research were conducted on the Lichtenberg high-performance computer and the Intelligent Autonomous Sys- tems (IAS) cluster at TU Darmstadt, Furthermore, the authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Cen- ter (NHR @ FAU) of the Friedrich-Alexance AlexanderUniieersttäi Erlangen-Nürnberg (FAU) under the NHR project b18ccb. NHR funding provided by federal and Bavarian state authorities. NHR FAU hardware is partially funded by the German Research Foundation (DFG) 4407 190711666 19683. The au- thors also gratefully acknowledge the "Julia 2" HPC provided by the Saluss-Maximllaannnnnnnea Würzburg. "Julia 2" was funded as DFG project "Forschungsgrolgerrit nach Art 9IbGG" under INST 93/1145-1 FUGG.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our results. To further pro- mote transparency and replication, we publicly release our code at https: /tgg//ttttt/sss hhub.com/ AhmedMaadyHeendawy /MINTO. To illustrate the simplicity of our approach. we provide in the appendix both a JAX code snapshot showing how MINTO can be integrated into DQN with minimal changes to the target computation and pseudocode for all algorithms where we integrate our method. Detailed descriptions of the experimental setup, architectures, and hyperparameters are provided in Appendix C. For theoretical results, we clearly state assumptions and include the complete proof of convergence in Appendix A. For empirical evaluation, we report results across multiple random seeds, following standard evaluation protocols (Machado et al., 2018), and provide per-task break- downs in Appendix D. Offline RL experiments use publicly available datasets (RL Unplugged), and we specify all data processing and training procedures in the appendix. Together, these details should facilitate exact replication and further validation of our findings.
10

---

Published as a conference paper at ICLR 2026
LARGE LANGUAGE MODEL USAGE
large language model was helpful in polishing writing, improving reading flow, and identifying remaining typos.
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of artificial intelligence research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement
learning. In International conference on machine learning, pp. 458. PMLR, 2017.
Richard Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, edi- tion, 1957.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman Milne, and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http:
//github.ccm com/ jaammmjaaxmm///m/m
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
arXiv:1812.06110, 2018.
Will Dabney, Georg Ostrovski. David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pp. 1096- 1105. PMLR, 2018
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan. Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Fo- erster, and Mario Martin. Simplifying deep temporal difference learning. In International Con- ference on Learning Representations, 2025.
Yaozhong Gan, Zhe Zhang, and Xiaoyang Tan. Stabilizing learning via soft mellowmax operator. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforeement learning, Advances in neural information process- ing systems, 33:7248-7259, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. Pmlr, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: orXiv:/812.055 011..8......555 2018b.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. 2015.
Steven Kapturowski, Víctor Campos, Ray Jiang, Nemanja Rakicevic, Hado van Hasselt, Charles Blundell, and Adria Puigdomenech Badia. Human-level atari 200x faster. In The Eleventh Inter- national Conference Learning Representations, 2023.
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for target network in deep q-learning In Proceedings of the twenty eighth international joint conference on artificial intelligence, 2019.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2024.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspher- ical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502. 15280, 2025.
Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. In Advances in Neural Information Processing Systems, 2021.
Alexander Lindström, Arunselvan Ramaswamy, and Karl-Johan Grinnemo. Pre-training deep q- networks eliminates the need for target networks: An empirical study. In The 14th International Conference on Pattern Recognition Applications and Methods (ICPRAM), 2025.
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv: 250..18551.1 2025.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.5660, 2013.
12

---

Published as a conference paper at ICLR 2026
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 518(7540):529-5333 2012.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample efficient continuous control. Ad- vances in neural information processing systems, 37:1 1038-113071, 2024.
Daniel Palenicek, Florian Vogt, and Jan Peters. Scaling off off-policy reinforcement learning with batch and weight normalization. In Advances in Neural Information Processing Systems, 2025.
Daniel Palenicek, Florian Vogt, Joe Watson, Ingmar Posner, and Jan Peters. Xqe: Well-conditioned optimization accelerates deep reinforcement learning. International Conference on Learning Rep- resentations, 2026.
Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christo- pher Pal, and Mohammad Emtiyaz. Khan. Bridging the gap between target networks and func- tional regularization. Transactions on Machine Learning Research, 2023.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Nikita Rudin, David Hoeller. Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on robot learning, pp. 91-100. PMLR, 2022.
Patrick Schnell, Luca Guastoni, and Nils Thuerey. Temporal difference learning: Why it can be fast and how it will be faster. The Thirteenth International Conference on Learning Representa- tions, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
Lin Shao, Yifan You, Mengyuan Yan, Shenli Yuan, Qingyun Sun, and Jeannette Bohg. Grac: Self- guided and self-regularized actor-critic. In Conference on Robot Learning. 2022.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844889 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 17110:1712.01815, 2017.
Richard Sutton, Andrew G Barto, et al. Reinforcement learning; An introduction, volume 1. MIT press Cambridge, 1998.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefiraney, al. Deepmind control suite. arXiv preprint arXiv:1801.00699 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
13

---

Published as a conference paper at ICLR 2026
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Mo- dayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02643 2018.
Théo Vincent, Tim Faust, Yogesh Tripathi, Jan Peters, and Carlo D D'Eramo. Eau de g-network:
Adaptive distillation of neural networks in deep reinforcement learning. Reinforcement Learning Journal, 2025a.
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, and Carlo D'Eramo. Iterated q- network: Beyond one-step bellman updates in deep reinforcement learning. Transactions on Machine Learning Research, 2025b.
Théo Vincent, Yogesh Tripathi, Tim Faust, Abdullah Akgül, Yaniv Oren, Mehil Kandemir, Jan Peters, and Carlo D' Eramo. Bridging the performance gap between target-free and target-based reinforcement learning. International Conference Learning Representations, 2026.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning. 8(3):799-22,, 1992.
Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with target network. In International Conference Machine Learning, pp. 12621-12631. PMLR, 2021.
Rong Zhu and Mattia Rigotti. Self-correcting q-learning. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11192, 2021.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Proof of Corollary 1
16
A.I Proof of Condition A1.1
16
A.2 Proof of Condition A1.2
16
B Algorithmic Details
17
Implementation Details
20
C.I Online RL and Discrete Control
20
C.2 Distributional RL
C.3 Offline RL
22
C,4 Online RL and Continuous Control
22
Individual Results
24
D.I Online RL and Discrete Control
24
D.2 Maxmin DQN MINTO
27
D.3
Distributional RL.
28
D.4 Offline RL
30
................................
D.5 Maxmin CQL vs. MINTO
33
D.6 Online RL and Continuous Control
34
D.7 Ablation on Target Operators
44
Analysis of Value Estimation Error and Bias
46
E.1
Experimental Setup
46
E.2 Algorithms and Hyperparameters.
46
E,3 Discussion
47
15

---

Published as a conference paper at ICLR 2026
A PROOF OF COROLLARY
To prove the convergence of MINTO, we show that its target operator, GAINTO, satisfies the two conditions of Assumption from the Generalized Q-learning framework Lan al. (2020).
Assumption 1 (Conditions on G) Ler G : RaxNxK H R be the target operator where Q= = (Q%) R=xN*K,
A, and | n, i .... arbitrary state. G must satisfy:
fo...., K - 1}, and S is Ni [.,....N.....
A1.1: If all input action-values identical, Q% Qai Vi, k. Vj, k; and Va, then
G(Q+) = maxa Q%
A1.2: G is non-expansion w.r.t. the max norm, (G(Q)) G(Q S maxa,iJ 10 Q-il-
The action-value function Q, is represented by tensor with an ensemble of N action-value func- tions and for each, historical time action- values.
The MINTO operator is defined (MINTO(Q.)) = maxag.4 (minjet Qsa(j)) for given state where T a set of historical time indices, and j the time index with an abuse of notations. This special case of the Generalized Q-learning where N 1 and K 2, hence dropping the i index. In the analysis, we consider a general variant of MINTO, where K > 1, by considering a set of historical time values using the indices set T such that Qsa (Q K), K)..... Qua 1)) for given state and action. In practice, T includes only (L- 1) and (L - K), representing the online and target network estimates, respectively.
A.1 PROOF OF CONDITION A1.1
Assume the input Q-values Q (j) are identical for all j E T and all A.
MINTO (Q=)
max aeAA
min
Qaa(i)
JET
max Qea
aeA
This is the maximum action-value among the inputs. Thus, Al.1 is satisfied.
A.2 PROOF OF CONDITION A1.2
Let Q and Q be two distinct sets of historical Q-values for a given state s, assuming that N
1.
MINTO (Q+)
MINTO (Q.)
max .EAA
min Qoa (i)
max aeA
min Qua Qaa(i)
(i)
(5)
JET
JET
max
min Qua (j) JET
min D'e (i) JET
(6)
< max
max QQa( (j) - Qua Vsa(i)l jET
max
(j) - Qaa
(8)
eEA.JT
The first inequality holds because the max operator is non-expansion. The second inequality holds because the min operator is also non-expansion.
The final term is the maximum absolute difference over the subset of Q-values used by the MINTO operator. This maximum necessarily less than or equal to the maximum over the entire set of all possible Q-values, i.e.:
aEAJET Quaa()) Q'aa S max Qsa(i) Q (i)
o.j
Therefore, (GMINT((Q.). MINTO (M.)) S maxa.) (Qaa()) - Qua(()) Thus, A1.2 is satisfied.
Conclusion:
Since both conditions of Assumption are satisfied, the convergence of MINTO is guaranteed by Theorem of Lan et al. (2020) under the standard stochastic approximation assump- tions for learning rate stated in Assumption 2 of Lan et al. (2020).
16

---

Published as a conference paper at ICLR 2026
B ALGORITHMIC DETAILS
The pseudocode blocks below present MINTO and its extension to IQN, CQL, and SAC in Alg. I, Alg. 2, Alg. 3, and Alg. 4 respectively. Changes the pseudocode of the underlying algorithms colored red. Note that [-] denotes the stop-gradee soopggrddeent operator. For MINTO we want to explicitly stop the gradient, as the target computation relies on both the online well the target parameters. In contrast to other methods, like DoubleDQN, for example, the gradient is not stopped by the arg max operator, as the online values are used directly.
Algorithm DON+MINTO.
1: Initialize online and target paramters 0, 8, an empty replay buffer B, and frotal
2: repeat
3: Sample an initial state from
4: for forl= to Plarriznn do
5:
Sample an action ar eegrredy(QQ(ss:: Execute action a, in environment, observe reward rz and next state 8t-1
6:
7:
Store transition (st, a2 Tx, 8t+1) in B
8:
Sample batch of B transitions (sb, ab. Pb, s)) from B Compute the TD-target yb. + max,' min(Qa(s( a'), Qo(s), if s/, is terminal then y/b f- rb
9:
11:
Compute the loss C(0) Obtain the gradient V and perform an update step Every T steps update the target network 0 <-
Σ
vol Qo ab))²
12:
13:
14:
88+1 is terminal break
15:
end for
16:
Exotal f tsotal + t
until Eooal Tiocal
Algorithm IQN+MINTO
1: Initialize online and target network parameters 0, 0, and replay buffer B, and beota = 0.
2:
repeat
3: Sample initial state
4: for 0 to Phorizon do
Select action ~ e-greedy Σ Zo(se, a, T.) where ~ U10,
5:
6:
Execute az observe reward rt and next state 8141 Store transition (st, az, Tx, 8t+1 in B
7:
8:
Sample minibatch (sb, ab. Tb; 56) from
9:
Sample {T {T
[0
Compute target quantiles:
= arg max min a'
Σ Zg Lg(s
a' Ti),
EN Lo(sh,
y, rb min
a" +y), ( a=, a²)
11:
if s/ terminal then yj <- Th Compute quantile regression loss:
12:
N
(()) NNB ΣΣΣ
PR ([
Zo(sn, ab. T:))
13:
Perform gradient step on V
14:
Every T steps update target network 0 St+1 is terminal break end for
15:
16:
17:
hotal + Lsotal +
18:
until hoaa Thooal
17

---

Published as a conference paper at ICLR 2026
Algorithm 3 CQL+MINTO
Initialize online and target critic parameters 0, 0, an empty replay buffer B, and touu 0.
2: Load offline dataset 'D into B
3: repeat
4: Sample a batch of B transitions (sp, dp. rb, s))= from B
5: Compute target Q-values for next states:
yb rb + max min +m
6: if <gg is terminal then ybb +- rb
7: Compute standard TD loss:
(T)
Σ b=1
(1yb) Q0(s,,a)))
2B
8:
Compute conservative regularizer:
B
Loqu( = ax. -Ba- log Σ exp((O((s,, a)) - Σ Qu(os, an)
9:
Total loss:
(()) CTD + (0)
10: Update critic parameters: 0 + 0 - (V
Every T steps update target network:
12:
fiotl f- Eaxal +
until Lotal Rsteps
In Alg. 4, SAC adapted to use single Q-function critic, following the approach taken in Simba (Lee et al. (2024; 2025)) on many benchmarks, which also relies single critic. This choice motivated our design, as it simplifies computation while remaining effective when combined with the MINTO operator in the target computation. In practice, also experimented with double- critic setup (see Fig. 18-21, 24, and 26), where we evaluate the Clipped Double Q-Learning (CDQ) trick with and without MINTO integration of online estimates.
For MINTO integration with CDQ, we considered two variants. the first variant (as in SimbaV1 and SimbaV2), the online estimates from both critics are aggregated to compute single shared target value, which is then used to update both networks. In the second variant (as in CrossQ+WN), each critic is updated using only its corresponding online estimate when forming the target. This is a design choice, and a more detailed investigation is left for future work.
In general, the largest performance gains were observed with the single-critic variant, since applying the minimum operator on all online and target networks, in the CDQ case, can lead to overly con- servative updates. Nevertheless, the performance gains remain clear after integrating MINTO into the CDQ trick, especially as shown in Fig. 24 and 26 when using the CrossQ+Wy QN (Palenicek et al., 2025) algorithm. The stochastic policy and temperature updates are retained as in standard SAC.
Overall, the MINTO modifications presented in Alg. 1, 3, and are straightforward to implement and can be integrated with minimal changes to the underlying algorithms. This makes MINTO practical approach for extending existing value-based and actor-critic methods without introducing significant complexity.
18

---

Published as a conference paper at ICLR 2026
Algorithm SAC+MINTO.
1: Initialize policy parameters 6, critic's online and target parameters 0, 0, empty replay buffer
B, and fuw 0.
2: repeat
3: Sample an initial state 8g from j
4: for 0 to Thorizon do
5:
Sample action ag ~ (-(s))
6:
Execute a in environment, observe reward Tr and next state 8+41 Store transition (8,, a,, 8t+1) in B
7:
8:
Sample batch of B transitions (sb, ab: rb, s), b=1 from B
9:
Sample af ~ (-(())) and compute log Ta (0)
Compute target:
y6 + y [min (Qa(s), Qo(s), ab)) log (a((()))
11:
if s/ is terminal then y/ rb Update critic by minimizing
12:
B
(())
[[v6l Qo(si, ot))²
27
b=1
13:
Update policy ø using gradient of
(Ø) 1 Σ
(a lagog (asssss) Qo(ss, ab))
d=1
14:
Optionally update temperature by minimizing
Jaa)
B 1=1
Σ
a log Ta (b) + Husre))
15:
Every T steps update target critic: Ø - 78 + (1 - +) if st+1 is terminal break
16:
17:
end for
18:
hotal laoal 6
19:
until Enotal 2 Reccal
19

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
C.I ONLINE RL AND DISCRETE CONTROL
For our online reinforcement learning experiments in the discrete-action setting, we use Atari envi- ronments as the benchmark domain. The implementations of DoubleDQN, FR-DQN, ScDQN, and MINTO share the same lightweight and reproducible framework for DQN variants written in JAX. The codebase' has been released. It is inspired by the code² released by Vincent et al. (2025b). All algorithms share the same network architectures and training setups, differing only in the algorithm- specific modifications detailed in Tables 1 and 2. MINTO is implemented by modifying the target computation.
Listing 1: JAX implementation of the MINTO TD target.
def compute_mirto_taree
self, target_parame,
online_params, sample,
):
gonl conlins_next f.lf htlff.etwrrk..plly.. (onli_aaonie_pprraaa ine_params,
cample.next_state
__onnine_nexxt jax. www.llxxxtttp_rrdieeee (e_onl ine_next) q t_.aarget_neae next = self. self.network.apply (target__parass
> sample. _ext_state]
q next japppax.ppii.pppppm.p....p..mppp (jnp. (jnp.minimuml (q
online next, t_target_nexte
11
return (
12
reward rewad
13
(1 cample.isttermnnal (self. wtl.
9 next
Hyperparameter
DQN
Maxmin DQN
MINTO
Replay Buffer Capacity
1,000,000
Batch Size
32
Update Horizon
Discount Factor (1)
0.99
Learning Rate
6.25 x 10
Horizon
27,000
Architecture Type
CNN
Features
322 64, 64, 512]
Epochs
100
Training Steps per Epoch
250,000
Data to Update
Initial Samples
20,000
Epsilon End
0.01
Epsilon Duration
250,000
Target Update Frequency (7)
8,000
Ensemble Size (N)
Table 1: Hyperparameter settings for DQN, Maxmin DQN, and MINTO on Atari. Most parameters shared, with algorithm-specife hyperparameters explicitly listed. Identical values are merged for clarity.
<https:////
20

---

Published as a conference paper at ICLR 2026
Hyperparameter
DoubleDQN
FR-DQN
ScDQN
MINTO
Replay Buffer Capacity
1,000,000
Batch Size
32
Update Horizon
Discount Factor (7)
0.99
Learning Rate
6.25 x 10
Horizon
27,000
Architecture Type
CNN
Features
[32, 64, 64, 512]
Epochs
100
Training Steps per Epoch
250,000
Data to Update
4
Initial Samples
20,000
Epsilon End
0.01
Epsilon Duration
250,000
Target Update Frequency (2)
8,000
Regularization Parameter (x)
1.0
Self Correcting Parameter (1)
3.0
Table 2: Hyperparameter settings for the four DQN variants evaluated on Atari, including Dou- bleDQN, FR-DQN ScDQN, and MINTO. Most parameters are shared across all algorithms, while the table explicitly lists the algorithm-specinf hyperparameters, the regularization parameter k for FR-DQN and the self-correcting parameter /3 for ScDQN. Identical values are merged for clarity.
C.2 DISTRIBUTIONAL RL
For the distributional reinforcement learning experiments, we use Implicit Quantile Networks (IQN). The implementation builds directly on the same JAX codebase the DQN experiments, ensuring consistency in architecture and training setup. The key algorithm-specific hyperparameters are listed
in Table 3.
Hyperparameter
IQN
IQN+MINTO
Replay Butfer Capacity
1,000,000
Batch Size
32
Update Horizon (n)
Discount Factor (?)
0.99
Learning Rate
5.0 x10
Adam
3.125 x x10
Horizon
27,000
Architecture Type
CNN
Features
[32, 64, 64, 512]
Epochs
100
Training Steps per Epoch
250,000
Data Update
4
Initial Samples
20,000
Epsilon End
0.01
Epsilon Duration
250,000
Target Update Frequency (T)
8000
Table 3: Comparison of IQN hyperparameters for n 1 using IQN and IQN+MINTO. U Update
horizon is set to 1.
21

---

Published as a conference paper at ICLR 2026
C.3 OFFLINE RL
For the offline reinforcement learning experiments on Atari, we use datasets from RL Unplugged (Gulcehre et al., 2020), which provide standardized and diverse benchmarks. The im- plementation is built on a stable and well-tested codebase to ensure reproducibility and fair com- parison, released by Vincent al. (2025a). The codebase has been released. All methods are run with their default hyperparameters, and the most important settings are reported in Table 4.
Hyperparameter
CNN
IMPALA
CQL
CQL+MINTO
CQL
CQL+MINTO
Dataset Size
5,000,000
Batch Size
32
Update Horizon
Discount Factor (7)
0.99
Epochs
100
Learning Rate
5 x 10
Adam (c)
5 x 3.125
Training Steps per Epoch
62,500
Tradeoff Factor (a)
0.1
Target Update Frequency (T)
2000
Layer Norm
no
yes
Table 4: Comparison of CQL and CQL+MINTO hyperparameters for the two different network architectures CNN and IMPALA.
C.4 ONLINE RL AND CONTINUOUS CONTROL
For our continuous-control experiments with online reinforcement learning, we adopt SimbaVI I and V2. The implementation is based on the official Simba V2 codebase" (Lee et al., 2025), ensur- ing consistency with the original work. All experiments use the default hyperparameters provided by the authors, with the most relevant ones summarized in Tables 5 and 6.
Hyperparameter
V1mbaV1, Simba MINNTO
DMC-Hard HumanoidBench
MuJoCo
Discount Factor (r)
0.99
0.995
Learning Rate
1.0 x 10
Weight Decay
0.01
Target (7)
0.005
Update Horizon (7)
Temperature Initial Value
0.01
Temperature Target Entropy
0.5 x [A]
Batch Size
256
Buffer Max Length
1,000,000
Buffer Min Length
5,000
Num Train Envs
Action Repeat
2
Max Episode Steps
1000
Table 5: Comparison of SimbaV1 hyperparameters across DMC-Hard, HumanoidBench. and Mu-
joco locomotion environments. Identical values merged.
2 httper//ggttuubb.cmmmuuuaa////// http:///iiiiumuuuegeeeeee unplugged https/////tthubb.bm/m///joo/// /bmab
22

---

Published as a conference paper at ICLR 2026
Hyperparameter
SimbaV2.
V
DMC--ard
HumanoidBench
MuJoCo
Actor Shift
Critic Shift
Critic Umax
5.0
Critic Unin
-5. 5.0
Critic Num Bins
101
Discount Factor (r)
0.99
0.995
Learning Rate Init
1.0 10
Learning Rate End
5.0 10-5
Update Horizon (n)
Target (7)
0.005
Temperature Initial Value
0.01
Temperature Target Entropy
-0.5 [A]
Buffer Max Length
1,000,000
Buffer Min Length
5,000
Num Train Envs
Action Repeat
2
Max Episode Steps
1000
Table 6: Comparison of V2 hyperparameters across DMC-Hard, Humanoid Bench, and Mu-
JoCo. Identical values are merged for clarity.
23

---

Published as a conference paper at ICLR 2026
D INDIVIDUAL RESULTS
D.1 ONLINE RL AND DISCRETE CONTROL
DQN
MINTO (ours)
VidcoPinball
Boxing
Breakout
1QM Return
20
10
5
5
0
DttocAttack
CrazyClimber
Assault
IQM Return
4
50
5.0
2
2.5
SpaceInvaders
FishingDerby
Asterix
1QM Return
50
10
2
0
Enduro
Pong
Tutankham
IQM Return
1.0
1.0
0.5
0.5
Qbert
MsPacman
BeamRider
IQM Return
0.4
0.50
0.2
0.25
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Number of Frames (in millions)
Figure 8: Individual Results of benchmarking MINTO and DQN on 15 Atari games using the IM- PALA architecture with LayerNorm. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
24

---

Published as a conference paper at ICLR 2026
DQN
MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
7.5
20
5.0
5.0
2.5
2.5
DemonAttack
CrazyClimber
Assault
IQM Return
5.0
4
2.5
Fishing Derby
Asterix
IQM Return
2
2
0
0
Enduro
Pong
Tutankham
IQM Return
1.0
1.0
0.5
0.5
Qbert
MsPacman
BeamRider
IQM Retum
0.50
0.50
0.25
0.25
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Number Frames (in millions)
Figure 9: Individual Results of benchmarking MINTO and DQN on 15 Atari games using the CNN network architecture. Reported metrics interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
25

---

Published as a conference paper at ICLR 2026
Double DQN
FR-DQN
ScDQN
MINTO (ours)
VideoPinball
Boxing
Breakout
1QM Return
7.5
20
5.0
5.0
2.5
2.5
DemonAttack
CrazyClimber
Assault
IQM Return
4
4
2
2
Spacelnvaders
Fishing Derby
Asterix
IQM Retur
2
2
0
Enduro
Pong
Tutankham
IQM Return
1.0
1.0
0.5
0.5
Qbert
MsPacman
BeamRider
IQM Return
0.5
0.50
0.25
0.0
20 40
60
80
100
20
40
60
80
100
20
40
60
80
100
Number of Frames (in millions)
Figure 10: Individual Results of benchmarking Double DQN, FR-DQN, ScDQN, and MINTO on 15 Atari games using the CNN network architecture. Reported metries are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
26

---

Published as a conference paper at ICLR 2026
D.2
MAXMIN DQN vs. MINTO
Maxmin DQN
MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
10
20
DemonAttack
CrazyClimber
Assault
IQM Return
2
2
0
SpaceInvaders
FishingDerby
Asterix
1QM Retum
1.5
2
1.0
0.5
0
Enduro
Pong
Tutankham
IQM Return
1.0
0.5
Qbert
MsPacman
BeamRider
1QM Return
0.5
0.50
0.5
0.25
0.0
0.0
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Number of Frames (in millions)
Figure 11: Individual Results of benchmarking Maxmin DQN and MINTO on 15 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
27

---

Published as a conference paper at ICLR 2026
D.3
DISTRIBUTIONAL RL
IQN
IQN+MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
7.5
20
10
5.0
5
2.5
DemonAttack
CrazyClimber
Assault
IQM Retum
10
4
50
5
0
SpaceInvaders
FishingDerby
Asterix
IQM Return
10
25
0-
0
Enduro
Pong
Tutankham
IQM Return
2
1.0
1.25
1.00
0.75
0.5
Qbert
MsPacman
BeamRider
1QM Return
1.0
0.50
0.50
0.5
0.25
0.25
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Number of Frames (in millions)
Figure 12: Individual Results of henchmarking IQN and IQN+MINTO on 15 Atari games using the IMPALA architecture with LayerNorm. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across seeds per game.
28

---

Published as a conference paper at ICLR 2026
IQN
IQN+MINTO (ours)
VideoPinball
Boxing
Breakout
1QM Return
10
20
5
0
DemonAttack
CrazyClimber
Assault
IQM Return
4
7.5
10
5.0
2
2.5
Spacelnvaders
Fishing Derby
Asterix
IQM Retur
2
2
0
0
Enduro
Pong
Tutankham
IQM Return
1
1.0
0.5
0
Qbert
MsPacman
BeamRider
IQM Return
0.75
0.50
0.25
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Number of Frames (in millions)
Figure 13: Individual Results of benchmarking IQN and IQN+MINTO on 15 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
29

---

Published as a conference paper at ICLR 2026
D.4
OFFLINE RL
CQL
CQL+MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
7.5
10
0.25
5.0
0.00
DemonAttack
CrazyClimber
Assault
IQM Return
10
2
5
0
SpaceInvaders
FishingDerby
Asterix
IQM Return
2
0.5
1
0
0.0
Enduro
Pong
Qbert
IQM Return
2
1.0
1
0.5
MsPacman
BeamRiderr
Performance Curve
IQM Return
0.50
0.5
0.25
0.0
0
10
20
30
40
50
0
10
20
30
40
50
10
20
30
40
50
Gradient steps (X 62.5K)
Figure 14: Individual Results of benchmarking CQL and CQL+MINTO on 14 Atari games using the IMPALA architecture with LayerNorm. Reported metries are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
30

---

Published as a conference paper at ICLR 2026
CQL
CQL+MINTO (ours)
VideoPinball
Boxing
Breakout
1QM Return
0.5
5
0.0
0
DemonAttack
CrazyClimber
Assault
1QM Return
2
0.5
2
0.0
0.
SpaceInvaders
FishingDerby
Asterix
1QM Retam
2
0.5
0.0
0.0
Enduro
Pong
Qbert
IQM Return
1.0
1.0
0.5
0.5
0.5
0.0
MsPacman
BeamRider
Performance Curve
IQM Retum
0.4
1
0.2
0.2
0.0
0
10
20
30
40
50
0
10
20
30
40
50
0
10
20
30
40
50
Gradient steps (x 62.5K)
Figure 15: Individual Results of benchmarking CQL and CQL+MINTO 14 Atari games using the CNN architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
31

---

Published as a conference paper at ICLR 2026
Double CQL
FR-CQL
ScCQL
CQL+MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
0.5
0
0.0
AttacA
CrazyClimber
Assault
IQM Return
2
0.5
2
0.0
0
0
SpaceInvaders
Derby
Asterix
IQM Return
2
0.5
0.1
0.0
0.0
Enduro
Pong
Qbert
1QM 1QM Return
1.0
0.5
0.5
0.0
MsPacman
BeamRider
Performance Curve
1QM Return
0.25
0.2
0.00
0.0
0
10
20
30
40
50
0
10
20
30
40
50
10
20
30
40
50
Gradient steps (X 62.5K)
Figure 16: Individual Results of benchmarking Double CQL, FR-CQL, ScCQL, and CQL+MINTO on 14 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
32

---

Published as a conference paper at ICLR 2026
D.5
MAXMIN CQL vs. MINTO
Maxmin CQL
CQL+MINTO (ours)
Maxmin CQL+MINTO (ours)
VideoPinball
Boxing
Breakout
IQM Return
7.5
10
0.5
5.0
2.5
0.0
0
DemonAttack
CrazyClimber
Assault
IQM Return
0.5
2
0.0
SpaceInvaders
FishingDerby
Asterix
1QM Return
2
0.25
0
0.00
Enduro
Pong
Qbert
IQM Return
1.5
1.0
1.0
0.5
0.5
0.5
0.0
MsPacman
BeamRider
Performance Curve
0.5
IQM Return
1
0.2
0.0
0.0
10
20
30
40
50
10
20
30
40
50
0
10
20
30
40
50
Gradient steps (x 62.5K)
Figure 17: Individual Results of benchmarking Maxmin CQL, CQL+MINTO, and Maxmin CQL+MINTO on 14 Atari games using the CNN network architecture. Reported metrics are in- terquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
33

---

Published as a conference paper at ICLR 2026
D.6
ONLINE RL AND CONTINUOUS CONTROL
D.6.1 SIMBA
Clipped Double Q
SimbaV2
SimbaV2+MINTO (ours)
Humanoid-v4
HalfChoctah-v4
Ant-v4
10000
IQM Retum
10000
6000
5000
4000
5000
2000
Walker2d-v4
Hopper-v4
Performance Curve
4000
IQM IQM Norm Return
1.5
6000
IOM Return
1.0
4000
2000
2000
0.5
0..0
0.0
0.2
0.4
0.8
0.0
0.4
0.8
0.2
0.4
0.6
0.8
1.0
Environment Steps millions)
Figure 18: Individual Results of benchmarking SimbaV2 and SimbaV2+MINT on the MuJoCo environments. Reported metries are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
Clipped Double Q
SimbaV1
(ours)
Humanoid-v4
HalfCheetah-y4
Ant-v4
10000
6000
1QM Retim
6000
4000
4000
5000
2000
2000
Walker2d-v4
Hopper-v4
Performance Curve
1QM Norm Return
3000
1QMRe Retur
2000
2000
1000
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 19: Individual Results of benchmarking SimbaV1 and SimbaV1+MINTE on the MuJoCo environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 5 environments.
34

---

Published as a conference paper at ICLR 2026
Clipped Double Q
SimbaV2
SimbaV++MINTO (ours)
hl hl-walk-vO
h1-stand-vO
hl al-run-vO
750
750
400
10M Return
500
500
200
250
250
h1 -0
h1 -1-hurdle-v0
hl-crawl-v0
1000
200
1QM Return
4000
800
100
600
2000
400-
hl -1-maze-v0
h1-sit simple-v0
h1 hard-v0
1QM Return
300
750
750
500
500
200
250
250
100
h1 --balance_a simple-v0
Ill-blancessimple--yeeeee
hl -balancce hard-v()
h1-stair-vO
150
750
10M 1QM Retum
400
500
100
200
250
50
h1 --ldiide-vO
h1 hl-pole-v0
Performance Curve
TQM Norm Return
750
1OM Retum
400
500
0.5
200
250
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 20: Individual Results of benchmarking SimbaV2 and Simba LimbaV2+MINTO on the Hu- manoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% con- fidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
35

---

Published as a conference paper at ICLR 2026
Clipped Double Q
SimbaV1
+iINTO (ouns)
hl bl-walk-vO
h1-stand-vO
hl -luu---
750
750
10M Return
200
500
500
250
250
100
hl-rrach-vo
h1 -1-huudle-v0
hl-crawl-v0
200
IQM Return
750
2000
100
500
250
hl 1--aaze-vO
h1-sit simple-v0
h1-sit hard-v0
750
750
1QM Return
300
500
500
200
250
250
100
h1 -bbllance 1l-balance_simples simple-vO
hl -bbllancee hard-v0)
hl-stair-vO
750
400
10M Retum
500
50
200
250
h1 -ssiide-vO
h h1-pole-vO
Performance Curve
750
IQM Norn Retum
300
1QM Retum
0.50
500
200
100
250
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 21: Individual Results of benchmarking SimbaV1 and SimbaV1+MINTI on the Hu- manoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% con- fidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
36

---

Published as a conference paper at ICLR 2026
SimbaV2
+ (ours)
humanoid-stand
humanoid-walk
humanoid-run
dog-stand
1000
IQM Return
500
400
100
500
200
0
dog-walk
dog-run
dog-trot
Performance Curve
1000
TQM Norm Retum
0.75
IQM IQM Return
400
0.50
500
200
0.25
0.0 0.4
0.0 Environment Steps (in millions)
0.4 0.8
0
0.0 0.2 0.8
0.00
0.0
0.2
0.6 0.8 1.0
Figure 22: Individual Results of benchmarking Simba and SimbaV2+MINTO on the 7 DMC- Hard environments. Reported metries interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
SimbaV1
SinnaVI+MINTO (uurs)
humanoid-stand
humanoid-walk
humanoid-run
dog-sland
750
200
1000
IQM Return
500
500
100
500
0-
0
dog-walk
dog-run
dog-trot
Performance Curve
600
TQM Norm Return
0.75
750
1QM Rehurn
400
500
500
0.50
200
250
0.25
0-
0.00
0.2
0.4
0.6
1.0
0.0
0.2 0.4 1.0
0.2 0.4 Envirommnn Steps (in millions)
1.0
0.0
0.2
0.4 0.6 0.8 1.0
Figure 23: Individual Results of benchmarking SimbaVI and SimbaV1+MINTH on the 7 DMC- Hard environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
37

---

Published as a conference paper at ICLR 2026
D.6.2 CROSSQ+WN
Clipped Double
CrossQ+Wr WN
CrossQ+WN MINTO (ours)
Humanoid-v4
HallCheetah-v4
Ant-v4
6000
6000
10000
1QM Return
4000
4000
5000
2000
2000
Walker2d-v4
Hopper-v4
Performance Curve
IQM Norm Return
3000
1.0
1QM Return
4000
2000
0.5
2000
1000
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 24: Individual Results of benchmarking CrossQ+WN and CrossQ+WN MINTO on the MuloCo environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of normalized return over all 5 environments.
CrNssO+WN
CrossQ+WN + MINTO (ours)
humanoid-stand
humanoid-walk
humanoid-run
dog-sland
1000
750
150
1QM Retum
400
500
100
250
0
0
0
dog-walk
dog-run
dog-trot
Performance Curve
750
IQM Norm Return
1QM Retum
500
200
400
0.4
250
200
0
0
0.0
0.0
0.2
1.0
0.0
0.2 0.4 0.6 1.0
0.0
0.8
1.0
0.8 1.0
Environment Steps (in millions)
Figure 25: Individual Results of benchmarking CrossQ+WV WN and CrossQ+WN + MINTO on the DMC-Hard environments. Reported metrics are interquartile (IQM) scores with 95% confi- dence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
38

---

Published as a conference paper at ICLR 2026
Clipped Double Q
Cross++WN WN
CrosQ++WN + MINTO (ours)
hl hl-walk-vO
h1-stand-v0
hl-run-vO
750
750
750
10M Return
500
500
500
250
250
250
hl -eachchvv
h1 A1-hurdle-v0
hl-crawl-v0
200
IQM Return
2000
100
500
250
hl al-maze-v0
h1-sit simple-v0
h1-sit hard-v0
750
750
1QM Return
300
500
200
250
250
100
h1 --balance_aa simple-v0
Ill-alancessimple-00
hl -balance hard-v0
h1-stair-v0
10M 1QM Retum
600
75
400
400
50
200
200
25
h1 -1-slide-vO
hl-pole-vO
Performance Curve
IQM Norm Return
400
600
IQM Retum
400
0.5
200
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 26: Individual Results of benchmarking CrossQ+WN and CrossQ+WN MINTO on the 14 Humanoid Bench environments. Reported metries interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
39

---

Published as a conference paper at ICLR 2026
D.6.3
FUNCTIONAL REGULARIZATION VS. MINTO
Clipped Double Q
FR-SimbaV2
VR-SimbaV2+MMNTO (ours)
Humanoid-v4
HalfCheetah-v4
Ant-v4
10000
IQM Return
10000
6000
5000
4000
5000
2000
Walker2d-v4
Hopper-v4
Performance Curve
1QM Norm Return
1.5
10M Return
6000
3000
1.0
4000
2000
2000
1000
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 27: Individual Results of benchmarking FR-SimbaV2 and Simba on the 5 Mu- JoCo environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
Clipped Double Q
FR-SimbaV1
SimbaV1++MNTO (ours)
Humanoid-v4
HalfCheetah-v4
Ant-v4
6000
10000
6000
IQM Return
4000
4000
5000
2000
2000
Walker2d-vl
Hopper-v4
Performance Curve
Norm Return
1QM Return
4000
3000
1.0
2000
2000
0.5
1000
0.0
0.2
0.4
0.8
0.0
0.2
0.6
0.0
0.0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 28: Individual Results of benchmarking FR-SimbaV1 and SimbaV1+MINTO on the 5 Mu- JoCo environments. Reported metries are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
40

---

Published as a conference paper at ICLR 2026
Clipped Double Q
FR-SimbaV2
SimbaV2+MINTO (ours)
h1- 11-walk-vO
h1-stand-vO
hl al-run-vO
400
750
750
10M Return
500
500
200
250
250
0
hl -each-vv-
h1 -1-hurdle-v0
hl-crawl-v0
200
IQM Return
2000
100
500
250
hl -1-maze-v0
h1-sit simple-v0
h1-sit hard-v0
1QM Return
300
750
750
500
500
200
250
250
100
h1 --balance_aa simple-v0
Illbalancessimple-yee
hl -balance hard-v()
hl-stair-vO
750
10M 1QM Retum
400
500
100
200
250
50
h1 -1-slide-vO
hl-pole-v0
Performance Curve
600
IQM Norm Return
750
1OM Retum
400
500
0.5
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 29: Individual Results of benchmarking FR-SimbaV2 and manoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% con- fidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
on the 14 Hu-
41

---

Published as a conference paper at ICLR 2026
Clipped Double Q
FR-SimbaV1
SimbaVI+MINTO (ouns)
hl b1-wwlk-vO
h1-stand-vO
hl Al-run-vO
750
750
10M Return
200
500
500
250
250
100
h1 -e
h1 A1-hurdle-v0
hl-crawl-v0
4000
200
IQM Return
750
2000
100
500
250
hl -1-maze-vO
h1-sit simple-v0
h1 hard-v0
750
750
1QM Return
300
500
500
200
250
250
100
h1 -bbllancee o1-balance_simplee simple-vO
hl -bbllancee hard-v0)
hl-stair-vO
750
400
10M Retum
500
50
200
250
hl -ssiide-vO
h hl-pole-vO
Performance Curve
750
IQM Norn Retur
300
1QM Retumm
0.50
200
500
100
250
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (in millions)
Figure 30: Individual Results benchmarking FR-SimbaVI and SinbaVI+MINII on the 14 Hu- manoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% con- fidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
42

---

Published as a conference paper at ICLR 2026
FR-SimbaV2
Ambaa22+MINT (ours)
humanoid-stand
humanoid-walk
humanoid-run
dog-stand
200
1000
600
IQM Return
500
100
500
200
0
dog-walk
dog-run
dog-trot
Performance Curve
1000
600
TQM Norm Retum
0.75
IQM IQM Return
400
0.50
500
200
0.25
0.0
0.4
0.0 Environment Steps (in millions)
0.4 0.8
0
0.0 0.2
0.8
0.00
0.0
0.2
0.6
0.8 1.0
Figure 31: Individual Results of benchmarking FR--Simba2 and SinbaV2+MINTO on the DMC- Hard environments. Reported metries interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
Fimbaa1
MinNaV (ours)
humanoid-stand
humanoid-wal
humanoid-run
dog-sland
1000
750
IQM Return
200
500
500
500
100
0-
0
dog-walk
dog-run
dog-trot
Performance Curve
600
TQM Norm Return
0.75
750
1QM Rehurn
400
500
500
0.50
200
250
0.25
0-
0.00
0.2
0.4
0.6
1.0
0.0
0.2 0.4 1.0
0.2 0.4 Environment Steps (in millions)
1.0
0.0
0.2
0.4
0.6
0.8 1.0
Figure 32: Individual Results of benchmarking FR-SimbaVI and Hard environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all environments.
on the DMC-
43

---

Published as a conference paper at ICLR 2026
D.7 ABLATION ON TARGET OPERATORS
Online Only
Target Only
Max
Mean
Random
Min (ours)
VidcoPinball
Boxing
Breakout
IQM Return
5
0-
0
0.
DemonAttack
CrazyClimber
Assault
1QM Return
2.5
2
2
0
0.0
0
Space Invaderss
Fishing Derby
Asterix
IQM Return
1
2
0.5
0
0
0.0
Enduro
Pong
Tutankham
IQM Return
2
0
0
Qbert
MsPacman
BeamRider
1QM Return
0.5
0.5
0.25
0.0
0.0
0.00
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Number of Frames (in millions)
Figure 33: Individual Results of benchmarking the Minimum operator of MINTO against other potential operators Atari games using the CNN architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
44

---

Published as a conference paper at ICLR 2026
Final Performance
Performance Curve
2
Min (ours)
Random
IQM Return
Mean
Max
Target Only
Online Only
0
2
10
20
30
40
50
IQM Return
Number of Frames (in millions)
Figure 34: Cumulative results of benchmarking the Minimum operator of MINTO against other potential operators on 15 Atari games using the CNN architecture. Both figures show the interquar- tile mean (IQM) scores with 95% confidence intervals of the final performance of each operator. Left: The final performance of each operator in a bar chart. Right: The performance curve for each operator over 50 million frames.
45

---

Published as a conference paper at ICLR 2026
E ANALYSIS OF VALUE ESTIMATION ERROR AND BIAS
DQN
(ouns)
Averaged Return
Brror Estimato MSB
Averaged Empirical Bias
Training Loss
1.00
0.8
0.015
13M Raaura
0.75
10M MSE
0.50
0.6
-0.2
0.010
0.25
0.005
0.00
0.4
0.000
1000
2000
3000
1000
2000
3000
1000
2000
3000
1000
2000
5000
Episades
Episades
Episodes
Equsodes
Figure 35: Results of benchmarking DQN, Double DQN, Maxmin DQN, and MINTO on the Car on Hill environment, showing the Average Return, Value Estimation Error Mean Squared Error (MSE), Averaged Empirical Bias, and Training Loss. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 100 seeds.
In this section, examine how several value-based RL algorithms perform the Car-on-the- Hill control task. Our analysis centers on their estimation biases, learning stability, and overall performance, shown in Fig. 35. We further aim evaluate whether MINTO's of recent online estimates enables faster and more stable learning. We also examine the degree to which MINTO exhibits estimation bias when compared with the corresponding baseline algorithms.
E. EXPERIMENTAL SETUP
We consider the Car-on-the-Be environment from Ernst al. (2005), modeled Markov deci- sion process with continuous state space s = (zz, v) (position and velocity) and discrete action space A {0, 1). Episodes terminate upon either reaching the goal region (reward +1) entering failure region (reward -), with intermediate rewards equal to zero. The episode is limited to 100 steps with a discount factor 7 0.95.
To obtain the true value function Q', we follow the procedure in Ernst et al. (2005) by computing an exact-lookup approximation on fixed evaluation grid. For each mate-action pair (s,a), we simulate the deterministic transition and, the successor state is non-terminal, we perform a breadth- search (BFS) over future trajectories up maximum depth K 50. If success terminal state is discovered at depth k; we assign the discounted return > ; if only failure is reachable within depth K, we assign -- This procedure yields a signed, discounted value that we treat as the ground-trute target (*
For evaluation, we construct a 10 10 grid over the state bounds provided by the environment. For each grid point, both actions are considered, yielding 200 fixed test states. The corresponding ground-truth values Q* (s,, are precomputed once and loaded during training.
E.2 ALGORITHMS AND HYPERPARAMETERS
We compare four value-hased deep RI. methods: DQN (Mnih al., 2013), Double DQN (Van Has- selt et al., 2016), Maxmin DQN (Lan et al., 2020), and our approach, MINTO. All agents use a fully connected neural network with two hidden layers of 64 units each and ReLU activations. We train the networks using the Adam optimizer with a learning rate of 10-3 and mini-batcher of size 64 sampled from replay buffer of capacity 100 000 transitions. A warm-up of 000 environment steps is used before performing gradient updates. Target networks are updated every 1 100 environment steps using a hard update.
Exploration follows an E-greedy strategy, where is annealed linearly from 1.0 to 0.05 over the first 50 000 steps, and kept fixed afterwards. All agents are trained for 3 000 episodes. After each episode, evaluate the current Q-network the fixed test grid and compute: (i) Value Estimation Error in Mean Squared Error (MSE) B[(((s, a) (s, a)) (ii) Averaged Empirical Bias EJQ(s, a) Q* a)]. Policy performance is monitored by logging episodic Average Return during training and evaluating the greedy policy (with 0) every 20 episodes. In Fig. 35, these metrics, in addition to
46

---

Published as a conference paper at ICLR 2026
the training loss, are reported in the form of interquartile mean (IQM) scores with 95% confidence intervals across 100 seeds.
E.3 DISCUSSION
As anticipated, DQN exhibits strong overestimation tendency, while Double DQN mitigates this effect only moderately (see Fig. 35). Maxmin DQN, which relies on the minimum over two largely independent value estimates, shows marked underestimation. In contrast, MINTO produces sub- stantially lower overall bias without the severe downward bias characteristic of Maxmin DQN. Im- portantly, the bias overshoot commonly observed in DQN and Double DQN is absent MINTO.
While MINTO's estimation error is broadly comparable to that of the baseline methods, it displays slight increase variance, an expected outcome of incorporating an online component into the target calculation. Nevertheless, MINTO solves the task faster than all other methods and does without exhibiting learning instability, despite incorporating online estimates into the target (see Fig. 35). Although
is deterministic environment with minimal stochasticity, MINTO still performs robustly and avoids the excessive conservatism normally associated with strong under- estimation. Finally, in line with Schnell al. (2025), we observe similar pattern of training loss increase followed by decrease for the discussed methods, and notably the pattern persists for MINTO but lower amplitude.
47


Reviews:
[{"review_id": "dOK4YjiJjY", "reviewer": "Reviewer_HyDL", "summary": "for the bootstrapping target in deep Q-learning, uses the minimum of the online net and target net value estimates, instead of using just the target net value estimate\n\nthey call their algorithm \"MINTO\"", "questions": "\"$Q$-learning offers a recursive update to approximate the state-action value function $Q$\"\n\nShould that last $Q$ be $Q^*$ instead of $Q$?\n\n&nbsp;\n\ndoes the \"preliminaries\" section allow for stochastic rewards? if not, is that intentional?\n\n&nbsp;\n\n\"The foundation of this success lies primarily in Deep RL, initiated by the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013), which marked the first successful application of deep neural networks in RL.\"\n\nNeural Fitted Q Iteration was also a success, even though DQN could maybe be considered an even bigger success\n\n&nbsp;\n\n\"A problem that presents a challenge and obstacle\"\n\nwhy both?\n\n&nbsp;\n\n\"In deep RL, the problem of moving targets is especially evident due to the use of neural networks and the resulting uncontrolled fluctuations in the values of unseen states.\"\n\nwhat is this contrasting against? linear RL?\n\n&nbsp;\n\n\"For example, Gallici et al. (2025) demonstrated that cleverly using parallel environments eliminates the need for a target network\"\n\nsome people might consider \"eliminates the need for a target network\" as too strong of a claim\n\n&nbsp;\n\n\"This makes MellowMax orthogonal to our approach.\"\n\nyou might say \"somewhat orthogonal\" to help avoid confusion\n\n&nbsp;\n\n\"making this method orthogonal to our approach\"\n\nlikewise\n\n&nbsp;\n\n\"known as $Q$-function\"\n\ntypo, should be \"known as _a_ $Q$-function\" (or \"_the_ $Q$-function\")\n\n&nbsp;\n\n\"(Mnih et al., 2013) introduce a series of algorithmic components, more importantly, is the introduction of the target networks.\"\n\ntypos\n\n&nbsp;\n\n\"Despite the success, this results in a slow learning of value function as well as the policy due to relying on out-dated estimates\"\n\ntypo, should be \"learning of _the_ value function\"\n\n&nbsp;\n\n\"bellman\"\n\ntypo, should be \"Bellman\"\n\n&nbsp;\n\n\"can we find a practical bellman update rule that results in a stable and fast learning?\"\n\nwhat does \"practical\" specify here?\n\n&nbsp;\n\n\"When implemented in an efficient deep learning framework such as JAX (Bradbury et al., 2018), this overhead is negligible.\"\n\nI don't understand. does JAX automatically parallelize the additional forward pass with the target network's forward pass?\n\n&nbsp;\n\n\"addressing Q1.As\"\n\ntypo\n\n&nbsp;\n\nwhy is Fig 1 with 10 Atari games, but Fig 2 with 15 games?\n\n&nbsp;\n\nFig 2 shows CNN and IMPALA results on 100 vs 50 million frames. why not both 100 or both 50?\n\n&nbsp;\n\n\"into both value-based and actor–critic methods\"\n\nthis is maybe slightly confusing wording because actor-critic methods are partially value-based. maybe \"both actor-critic and purely value-based methods\" would be slightly clearer.\n\n&nbsp;\n\n\"Offline RL aims to learn an optimal policy from a large and static dataset\"\n\noffline RL does not need a \"large\" dataset to be offline RL\n\n&nbsp;\n\n\"A central challenge in this paradigm is the distributional shift problem: the learned policy may query the Q-function on state–action pairs absent from the dataset\"\n\noffline policy evaluation does not necessarily involve a learned policy. offline policy evaluation is also offline RL, and also may query the value function on state or state-action inputs not present in the dataset", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "applies MINTO on many base algorithms (DQN, IQN, CQL, SAC) and architectures (CNN, IMPALA, SimbaV1, SimbaV2), and fairly consistently shows some improvements in returns\n\nno additional hyperparameters\n\nsimple\n\na good number of fair baseline algorithms -- for ex, includes mean, max, random, and min of {online, target} nets in Fig 1, and later also includes Double DQN, FR-DQN, ScDQN, and Maxmin DQN\n\nnovel as far as I am aware\n\nwritten clearly", "weaknesses": "I think it would be great if the paper were even more clear about how the hyperparameters were set. while Appendix C.3 says \"All methods are run with their default hyperparameters\", it's not immediately clear to me how the values in Table 1 were set. are they the default hyperparameters from any existing codebase? if not, why not, and how were they set?\n\n5 seeds is not many seeds. (but the experiments cover so many base algorithms, architectures, codebases, and environment suites that I suspect this is not a big issue)\n\nno tests on toy tasks, along the lines of Baird's counterexample. I think there is a non-negligible chance that MINTO will get \"stuck\" on some such problems, where both TD and DQN sometimes empirically require their loss to increase before it will decrease. see for ex Fig 1e here: https://openreview.net/pdf?id=j3bKnEidtT. MINTO empirically appears to work on a wide variety of other benchmarks though, so this whole bullet might be irrelevant (even including the possible case that MINTO fails on some toy tasks).\n\na bit verbose, for ex the paper calls MINTO \"simple yet effective\" 4 times (once with a comma after simple)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bEnUZS1heP", "reviewer": "Reviewer_vd9n", "summary": "The paper proposes a technique (MINTO) to make better use of the online network when also using a target network for learning the value function. The technique simply changes the update target value to be the minimum between the target network and online network's value estimates.\nExperiments on a variety of benchmarks show that this technique is beneficial and can outperform common alternatives such as double Q-learning or using the minimum betweeen two target networks.", "questions": "Suggestions and comments(not directly impacting the score):\n- In the ablations, it is reported that taking the mean between the online and target network estimates was not helpful. Do you have any hypotheses why this woudl be the case? Using the mean between two target critics has been found to sometimes an effective alternative to the min between two critics [1] and sometimes underestimation bias has been found to be harmful.\n\n\n- The proposed method reminds me of the online network trust region method proposed in [2] since they both attempt to make use of the online network safely. It could be beneficial to include it as a baseline or provide some comparisons.\n\n- The algorithm labelled  \"Maxmin DQN\" in Fig.3. seems to be very similar to the clipped Double-Q learning update proposed in the TD3 paper [3] and I had not seen any mention of this.  \n\n\n[1] \"Scaling for compute and sample-efficient continous control\" Nauman et al.\n\n[2] \"Human-level Atari 200x faster\" Kapturowski et al.\n\n[3] \"Addressing function approximation error in actor-critic methods\" Fujimoto et al.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper was easy to follow and the proposed technique is very simple to understand and implement.\nThe experiments are comprehensive and show that method works in a variety of settings (Q-learning algorithms, actor-critic, distributional RL, continuous-control, atari).\nI appreciated testing the variants that would be most relevant such as mean or max of the online and target network estimates. \n\nIn summary, I think the proposed technique would make a nice addition to the deep RL toolbox.", "weaknesses": "I did not identify any major weaknesses.\n\nIt could have been nice to include a more detailed analysis of why MINTO may be effective. In particular, thinking through the MINTO updates and comparing to using the target net as usual, the online network value replaces the target net value when the online network's value is lower. This means we can only use the most recent information (online network) if it is lowering the bootstrap target. \nI wonder if you could study the accuracy of the value estimates in a policy evaluation setting where we could accurately estimate true value functions. Perhaps MINTO produces better estimates overall.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVtGXQg7N4", "reviewer": "Reviewer_Fczs", "summary": "This paper proposes a new method for using the target network in RL. \nSpecifically, the authors modify the target computation by taking the minimum of the target Q-network and the online Q-network. \nAlbeit simple, it shows substantial improvements across several RL settings, including online RL (DQN, SAC (Simba-V1, Simba-V2)), distributional RL (IQN), and offline RL (CQL).", "questions": "**Q1. Missing comparison with Clipped Q-learning and related baselines**\n* To fully support the claim, the proposed method should be experimentally compared with other bias reduction methods (e.g., methods introduced in DQN experiment). \n* Among those, I think at least **comparison with Clipped Q-learning [1] is necessary**, which is equivalent to MaxMin DQN with N=2 --- the strongest baseline reported for DQN in the paper, and also widely used.\n\nI would like to emphasize that I believe this paper is strong and has substantial potential. However, this point is a critical concern for me. I currently lean toward a weak reject, but if the authors can provide the requested experimental comparisons (e.g., with Clipped Q-learning), I would be inclined to raise my recommendation, even significantly if the results meet the expectations.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "**S1. Simplicity of the method**\n* The proposed method is conceptually simple and easy to implement; it only modifies the target calculation by taking the minimum between the target and online Q-values. \n* This adjustment can be integrated easily into existing RL frameworks, as demonstrated in Algorithms 1–4. \nSuch simplicity makes the method practical and broadly applicable.\n\n**S2. Substantial improvement across diverse RL domains**\n* Despite its simplicity, the method demonstrates strong empirical performance. \n* It improves results across various RL domains --- online, offline, and distributional settings --- and across both continuous and discrete action spaces. \n* The reported gains over baseline methods in the DQN setting, suggest that the proposed approach effectively mitigates overestimation bias than other methods.\n\n**S3. Clear organization and presentation**\n* The paper is well structured and easy to read.\n* Experimental results are clearly presented.", "weaknesses": "**W1. Comparison with other regularization methods limited in DQN setting**\n* Currently, the comparison with other bias-reduction or regularization methods is limited to the DQN (discrete online RL) setting. \nThis restricts the strength of the empirical claims, especially since the method is positioned as a general improvement applicable across multiple RL domains.\n* Including comparisons with well-established bias-reduction methods in other settings (e.g., SAC, IQN, and CQL) will clarify whether the observed improvements generalize beyond DQN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEMc0uvXQ2", "reviewer": "Reviewer_t6g5", "summary": "This paper proposes MINTO, a novel update rule for deep reinforcement learning that computes regression targets as the minimum estimate between the online and target networks. MINTO is designed to address the stability-efficiency trade-off inherent in standard RL, where target networks help stabilize learning but slow down convergence, while online networks foster efficiency but risk instability. Experiments on online, offline, discrete, and continuous RL benchmarks such as Atari games and robotic control tasks demonstrate consistent and significant improvements in sample efficiency and final performance over various baselines.", "questions": "1. Have you studied the impact of potential underestimation? While you claim that it is slight, is there any theoretical bound to it?\n\n2. The paper briefly mentions possible unexplored interactions between MINTO and exploration strategies. Can the authors clarify how MINTO affects exploratory behavior?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated and the minimum-operator is easy to integrate into existing RL algorithms.\n\n2. The framework is validated on a wide set of benchmarks (Atari, MuJoCo, offline and online RL) and compared to several baselines, demonstrating consistent gains in both stability and sample efficiency.\n\n3. The modifications require minimal changes to underlying algorithms and introduce only a tiny computational cost, making practical adoption straightforward.", "weaknesses": "1. The main evaluation is conducted against SimbaV1 and SimbaV2, which appear to be less strong baselines. This certainly raises the question whether this method can generalize to strong baselines such as Rainbow.\n\n2. The exclusive reliance on the minimum operator can be too conservative, possibly causing underestimation and inhibiting exploration in low-noise or optimistic environments.\n\n3. The code is not attached and will be released upon acceptance. This raises certain concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)