Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning

## Abstract

Abstract The use of target networks is a popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as a bootstrapped target is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing a novel update rule that computes the target using the MIN imum estimate between the T arget and O nline network, giving rise to our method, MINTO . Through this simple, yet effective modification, we show that MINTO enables faster and stable value function learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and offline RL, as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.

## 1 Introduction

Reinforcement Learning (RL) has demonstrated exceptional performance and achieved major breakthroughs across a diverse spectrum of decision-making challenges. This success spans domains from mastering complex environments like video games (Mnih et al., 2013; 2015; Hessel et al., 2018) and strategic board games (Silver et al., 2016; 2017), to solving high-dimensional problems in continuous control (Haarnoja et al., 2018a; Schulman et al., 2017). Noteworthy applications include learning complex locomotion skills (Haarnoja et al., 2018b; Rudin et al., 2022) and enabling sophisticated, real-world capabilities such as robotic manipulation (Andrychowicz et al., 2020; Lu et al., 2025). The foundation of this success lies primarily in Deep RL, initiated by the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013), which marked the first successful application of deep neural networks in RL. To make that happen, Mnih et al. (2013) introduce various techniques to mitigate mainly the deadly triad issue (Van Hasselt et al., 2018) due to the usage of function approximators, off-policy data, and target bootstrapping.

Particularly, Mnih et al. (2013) introduce the concept of target networks to mitigate the negative impact of the deadly triad issue (Zhang et al., 2021), where the regression target is computed using a lagged copy of the online network to promote stability during training. This prevents the online network from directly chasing its own rapidly changing estimates, thereby mitigating the problem of moving targets. A problem that presents a challenge and obstacle towards using fresh estimates from the online network. While the target network has been highly successful in improving stability and convergence, it inherently slows down learning since updates are based on delayed targets. This naturally raises an important question: how can we accelerate learning by leveraging the most recent online estimates while still preserving the stability of the training process?

Recent studies have suggested that relying solely on the online network for target bootstrapping can improve performance in certain methods (Bhatt et al., 2024; Kim et al., 2019). Surprisingly, later findings indicate that reinstating a target network in these same approaches can further enhance results (Palenicek et al., 2025; Gan et al., 2021). Building on these insights, we propose a complementary perspective: leveraging both the online and target networks jointly to compute the regression target, thereby aiming to combine the benefits of stability and fast learning.

In deep RL, the problem of moving targets is especially evident due to the use of neural networks and the resulting uncontrolled fluctuations in the values of unseen states. This issue becomes even more critical in value-based methods, where maximization bias drives the online estimates to steadily increase over time. These issues motivate the search for an appropriate selection criterion that can mitigate the impact of maximization bias when employing the online network for target computation, hence faster, yet stable learning.

Building on this motivation, we propose MINTO, a simple yet effective technique that computes regression targets by taking the MINimum estimated value between the Target and Online networks. By relying on the target network when the online estimate is relatively higher, MINTO reduces maximization bias, alleviates the moving-target problem, and ensures stable learning. At the same time, by incorporating the online network when its estimate is lower, MINTO leverages fresher information, enabling faster learning.

Thanks to its simplicity, MINTO can be seamlessly integrated into a wide range of off-policy methods, including both value-based and actor–critic algorithms, across online and offline RL settings. In this work, we present an extensive empirical evaluation showcasing the benefits of our approach. In addition, we benchmark MINTO against related baselines that exploit online estimates, whether for similar objectives or for different purposes.

Our contributions can be summarized as follows: we advocate for a principled combination of online and target networks when computing bootstrapped targets, enabling faster learning while preserving stability. We introduce MINTO, a simple yet effective technique that computes regression targets as the minimum between online and target estimates, thereby mitigating the moving-target problem and reducing maximization bias. We further demonstrate MINTO’s broad applicability and effectiveness by integrating it into both value-based and actor–critic methods, across online and offline RL settings. Extensive empirical results show that MINTO consistently outperforms conventional target-network designs and related baselines.

## 2 Related Work

Many works focus on developing simpler RL algorithms that are closer to the original Q-learning algorithm to benefit from up-to-date Bellman updates. A reasonable approach is to use additional resources or privileged information to remove the target network. For example, Gallici et al. (2025) demonstrated that cleverly using parallel environments eliminates the need for a target network. However, real-world applications are often limited to a single process. Interestingly, Lindström et al. (2025) show that constructing the regression target from the online network alone is stable after a pre-training phase using expert data. While this study gives hope that a target-free algorithm is feasible, it still relies on additional resources that are not available in the general case. Shao et al. (2022) introduce a cross-entropy method to refine the actions suggested by the policy to construct the bootstrapped estimate. While promising, this technique is limited to the actor-critic setting, and requires additional resources to optimize the landscape defined by the $Q$-function. In this work, we do not consider having access to additional resources or privileged information as we are interested in building a general-purpose algorithm for off-policy learning.

Another approach is to rely on a single estimator, thereby reducing the number of parameters used during training. For example, Kim et al. (2019) replace the maximum operator with the MellowMax operator to construct a target-free algorithm. However, a following work (Gan et al., 2021) suggests that reintroducing the target network is beneficial, indicating that the improvements made by the first approach come from the different nature of the update instead of up-to-date bootstrapped estimates. This makes MellowMax orthogonal to our approach. Some works intervene in the architecture of the function approximator to stabilize the training dynamics when using only one estimator. For example, Li & Pathak (2021) design neural networks processing the state given as input in the Fourier space. Nonetheless, their analysis shows that the approach struggles to handle high-dimensional input spaces. More recently, Bhatt et al. (2024) introduce CrossQ, an actor-critic algorithm that uses batch normalization (Ioffe & Szegedy, 2015) to account for the distribution match between the state-action pair and the next state-next action pair. A follow-up work demonstrates that this idea can work better with a target network (Palenicek et al., 2025), moving away from the objective of constructing the regression target from up-to-date estimates, and making this method orthogonal to our approach. In the following, we also choose to rely on a target network since Vincent et al. (2025) show that target-free methods still underperform compared to target-based methods.

Closer to our approach, hybrid methods have been developed, where the regression target is built from the online network, but an old copy of the online network is used to regularize the update. Zhu & Rigotti (2021) introduce Self-correcting $Q$-learning (ScQL), which evaluates the $Q$-estimate of the next state at the action that maximizes a combination of the target and online network. Piché et al. (2023) regularizes the online network predictions with the prediction given by the target network. In Section [5.1](https://arxiv.org/html/2510.02590v1#S5.SS1), we compare MINTO to those methods.

Building the regression target from the online network accelerates the overestimation bias, which degrades performance. The overestimation bias is a long-standing problem in off-policy RL Hasselt (2010), which arises from the interaction between the maximum operator and the stochastic nature of the bootstrap estimate. A first attempt to combat this issue is to learn two independent estimates of the $Q$-function and evaluate one $Q$-estimate on the best action suggested by the other estimator to build the regression target (Hasselt, 2010). While this idea is scalable to deep settings (Van Hasselt et al., 2016), it leads to underestimation, which hinders exploration. Maxmin Q-learning (Lan et al., 2020) reduces overestimation bias by taking the minimum across several estimators before applying the maximum operator, but at the cost of training multiple networks in parallel. By contrast, our method applies the minimum operator between the target and online network, incorporating the latest online estimates in a stable manner to achieve faster learning.

## 3 Preliminaries

We define the problem as a Markov Decision Process (MDP) Puterman (1990), $<\mathcal{S},\mathcal{A},P,r,\mu,\gamma>$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the transition distribution where $P(s^{\prime}|s,a)$ is the probability of reaching state $s^{\prime}$ from state $s$ after performing action $a$, $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function, $\mu$ is the initial state distribution, and $\gamma\in[0,1)$ is a discount factor. A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ maps each state to a distribution over the action space. The policy induces an action-value function $Q^{\pi}(s,a)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a]$ that defines the expected discounted cumulative return of executing action $a$ in state $s$ while following the policy $\pi$ thereafter. The goal of the agent is to find the policy $\pi$ that maximizes the expected return starting from some initial state.
TD learning methods are a suitable set of solutions to this goal, more precisely Q-Learning Watkins & Dayan (1992). Q-Learning is an off-policy algorithm that aims to learn the state-action value function $Q$, known as $Q$-function, of the optimal policy $\pi^{*}$, $Q^{\pi^{*}}=Q^{*}$, by utilizing the Bellman optimality equation Bellman (1957):

$$ $Q^{*}(s,a)=\mathbb{E}[r(s,a)+\gamma\max_{a^{\prime}\in\mathcal{A}}Q^{*}(s^{\prime},a^{\prime})].$ (1) $$

Therefore, the optimal policy is defined using this value function by acting greedily at a given state $s$, $\pi^{*}=\text{arg}\text{max}_{a\in\mathcal{A}}Q^{*}(s,a)$. Q-learning offers a recursive update to approximate the state-action value function $Q$, given a transition sample $(s,a,r,s^{\prime})$ generated by any behavioral policy:

$$ $Q(s,a)\leftarrow Q(s,a)+\alpha(s,a)[y-Q(s,a)],$ (2) $$

where $y=r+\gamma\max_{a^{\prime}\in\mathcal{A}}Q(s^{\prime},a^{\prime})$ is the target value which is computed via bootstrapping with the current estimate, and $\alpha(s,a)$ is the step-size. In the target value, the maximum expected value of the next state is approximated by applying maximum operator on a single estimate. This introduces a maximization bias and causes Q-learning to overestimate the state-action values Hasselt (2010).

When dealing with high dimensional state and action spaces, the tabular form of the value functions is not suitable and function approximators are needed. Particularly, in Deep RL, the state-action value function is modeled by a neural network $Q_{\theta}(s,a)$ with some learnable parameters $\theta$. In Mnih et al. (2013), DQN was introduced as the first successful application of neural networks in RL. (Mnih et al., 2013) introduce a series of algorithmic components, more importantly, is the introduction of the target networks. Target networks $Q_{\bar{\theta}}$ allow a stable learning of value functions by computing the target value using an older copy of the online network $Q_{\theta}$, $y=r+\gamma\max_{a^{\prime}\in\mathcal{A}}Q_{\bar{\theta}}(s,a)$. This is done by periodically updating the target parameters $\bar{\theta}$ to the online parameters $\theta$ every $T$ steps. Hence preventing the chase of a moving target due to learning $Q_{\theta}$ from its own value that eventually results in unstable learning. Despite the success, this results in a slow learning of value function as well as the policy due to relying on out-dated estimates. This raises a question: can we find a practical bellman update rule that results in a stable and fast learning?

## 4 Methodology

### 4.1 MINTO

The aim of this paper is to identify a suitable criterion for incorporating recent online estimates without sacrificing the stability traditionally provided by the target network. This naturally suggests that the online and target networks should work side by side to achieve both fast and stable learning.

To this end, we propose a simple yet effective technique that leverages online estimates only when they are unlikely to introduce harmful overestimation or rapid fluctuations in the target. In cases where the online estimates are higher than the target one, we instead rely on the target network to ensure stability. Concretely, we achieve this by applying the MINimum operator to the estimated values of the Target and Online networks, giving rise to our method, MINTO.

Following our method, the bootstrapped target can be computed as follows:

$$ $y=r+\gamma\max_{a^{\prime}\in\mathcal{A}}\min\big(Q_{\bar{\theta}}(s^{\prime},a^{\prime}),Q_{\theta}(s^{\prime},a^{\prime})\big).$ (3) $$

Given the new regression target, we can compute the regression loss function, similar in DQN, as follows:

$$ ${\mathcal{L}}(\theta)=\tfrac{1}{2}\big(\lceil y\rceil-Q_{\theta}(s,a)\big)^{2},$ (4) $$

where $\lceil.\rceil$ refers to the stop gradient operator, to prevent the backpropagation of gradients to the online network presented in the regression target equation (Eq. [3](https://arxiv.org/html/2510.02590v1#S4.E3)).

In practice, MINTO can be integrated into DQN, and more generally into any off-policy algorithm (see Appendix [B](https://arxiv.org/html/2510.02590v1#A2)), by modifying only a few lines of code. The method is lightweight, requiring only a single additional feedforward pass of the online network on the next state. When implemented in an efficient deep learning framework such as JAX (Bradbury et al., 2018), this overhead is negligible.

### 4.2 Analyzing MINTO

Figure: Figure 1: Results of benchmarking the Minimum operator utilized by MINTO against other potential operators on $15$ Atari games with the CNN architecture. We report the AUC metric using IQM and the confidence interval computed across $5$ seeds. Methods are trained for $50$ million frames.
Refer to caption: x1.png

To evaluate the impact of MINTO, we design an empirical study that seeks to answer two central questions: (Q1) Is the minimum operator an appropriate criterion for combining online and target estimates? (Q2) Can the empirical evidence substantiate the rationale for adopting the minimum operator?

We empirically examine how the minimum operator employed in MINTO stands relative to alternative operators for combining online and target network estimates. Our evaluation is conducted on $15$ Atari games and considers the following baselines: Online Only, which relies exclusively on the online estimates during training; Target Only, equivalent to DQN using solely the target network estimates; Max, which selects the larger of the two estimates; Mean, which averages them; Random, which chooses between the online and target networks with equal probability; and finally Min, the operator at the core of our proposed method, MINTO.

In Fig. [1](https://arxiv.org/html/2510.02590v1#S4.F1), we clearly observe the advantage of the minimum operator over the alternative candidates, thereby addressing Q1.As expected, the Online Only function performs poorly, as it relies on rapidly changing bootstrapped targets that lead to unstable training dynamics (see Fig. [19](https://arxiv.org/html/2510.02590v1#A4.F19)). On the other hand, the Random function fails to match the performance of MINTO, instead converging toward the behavior of Target Only (DQN). The Mean function also aligns closely with Target Only, without offering any additional benefit. Interestingly, the worst results are obtained with the Max operator, which represents the opposite selection criterion to ours. In this case, instability arises from severe overestimation bias, which in turn drives large and uncontrolled increases in target values. Taken together, these findings support our motivation for adopting the minimum operator: it enables the inclusion of recent online estimates in a stable manner by mitigating the overestimation bias they may introduce, thus providing an empirical answer to Q2.

### 4.3 Convergence of MINTO

While our empirical study highlights the effectiveness of the minimum operator, it is equally important to establish whether its use is theoretically justified in terms of convergence. Since analyzing convergence under function approximation is notoriously difficult, we focus on the tabular setting. Our analysis leverages the Generalized Q-learning framework of Lan et al. (2020), which guarantees convergence for update operators satisfying specific non-expansion properties. Owing to its close relation to Maxmin Q-learning, we show that MINTO can be cast as a special case of this general framework, in the same spirit as how it has been used to analyze other Q-learning variants in their work. This observation leads directly to the following corollary.

###### Corollary 1 (Convergence of MINTO) .

Let the MINTO operator be defined as $G^{\text{MINTO}}(Q_{s})=\max_{a\in\mathcal{A}}\min_{j\in\mathcal{T}}Q_{sa}(j)$, where $\mathcal{T}$ is a set of historical time indices. Under the standard stochastic approximation assumptions for the learning rate (Assumption 2 in Lan et al. (2020)), the Q-values generated by the MINTO update rule converge to the optimal action-values, $Q^{*}$.

###### Proof.

The proof relies on demonstrating that the $G^{\text{MINTO}}$ operator satisfies the two conditions on the target operator $G$ (Assumption 1 in Lan et al. (2020)). We provide the detailed verification of these conditions in Appendix [A](https://arxiv.org/html/2510.02590v1#A1).
∎

###### Corollary 1 (Convergence of MINTO) .

###### Proof.

## 5 Experimental Results

We now turn to the empirical evaluation, where we demonstrate MINTO’s broad applicability and effectiveness by integrating it into both value-based and actor–critic methods across online and offline RL settings, and show consistent advantages over conventional target-network designs and related baselines.

Figure: Figure 2: Results of benchmarking MINTO and DQN on $15$ Atari games with CNN, and IMPALA with LayerNorm (LN) architectures. All results are reported using IQM and confidence interval on $5$ seeds and over all games. Left: We report the AUC metric for MINTO and DQN while utilizing both architectures. Center and Right: We illustrate the performance learning curves of MINTO and DQN after employing both architecture options.
Refer to caption: x2.png

### 5.1 Online RL and Discrete Control

We evaluate MINTO on a benchmark of $15$ Atari games (Bellemare et al., 2013) recommended by Graesser et al. (2022). These games are chosen for their diversity in human-normalized scores achieved by DQN after training. All methods employ the standard CNN architecture introduced by Mnih et al. (2015), and our experiments follow the evaluation protocol of Machado et al. (2018). Additional implementation details and hyperparameters are provided in Appendix [C.1](https://arxiv.org/html/2510.02590v1#A3.SS1).

We begin by evaluating the performance gains of MINTO over DQN (Mnih et al., 2013) in terms of both sample efficiency and asymptotic performance. Our objective is to demonstrate that MINTO not only accelerates learning by incorporating fresher estimates from the online network but also achieves higher final performance. To capture both aspects in a single measure, we report the Area Under the Curve (AUC) metric, alongside learning curves for MINTO and DQN. To further assess the method’s effectiveness with more advanced architectures, we extend the comparison to the IMPALA architecture (Espeholt et al., 2018; Castro et al., 2018), evaluating the same metrics. In this setting, we additionally apply LayerNorm (LN), motivated by prior work showing its potential for improving learning stability and performance (Lee et al., 2024; Nauman et al., 2024).

In Fig. [2](https://arxiv.org/html/2510.02590v1#S5.F2), we observe a clear advantage of MINTO over DQN. Specifically, MINTO achieves an improvement of approximately $18\%$ in AUC when using the vanilla CNN architecture, and about $24\%$ when employing IMPALA with LN. The performance curves in Fig. [2](https://arxiv.org/html/2510.02590v1#S5.F2) (center and right) further illustrate MINTO’s superior sample efficiency and higher asymptotic performance. These results suggest that incorporating more recent estimates from the online network accelerates learning while also improving final performance.

Figure: Figure 3: Results of benchmarking MINTO against related baselines on $15$ Atari games with the CNN architecture. We report the AUC metric using IQM and the confidence interval computed across $5$ seeds.
Refer to caption: x3.png

The idea of leveraging the online network alongside the target network for computing regression targets is not confined to MINTO. Prior work has explored alternative ways of utilizing the online network for different purposes. It is therefore important to evaluate the effectiveness of our selection criterion, the minimum operator, against established methods from the literature. In this study, we compare against three representative baselines: Double DQN (Van Hasselt et al., 2016), Functional Regularization DQN (FR-DQN) (Piché et al., 2023), and Self-correcting DQN (ScDQN) (Zhu & Rigotti, 2021). All methods are implemented with the CNN architecture, and we report aggregate results over all $15$ games using the AUC metric.
In Fig. [3](https://arxiv.org/html/2510.02590v1#S5.F3), we observe that MINTO consistently outperforms all baselines, including methods such as Double DQN and ScDQN that are specifically designed to mitigate overestimation. This indicates that combining the online network with the minimum operator provides an especially effective mechanism for addressing this issue. Furthermore, MINTO achieves clear gains over FR-DQN, which relies entirely on the online network for target bootstrapping while regularizing its updates using a fixed network (target network). This highlights the role of our selection criterion, the minimum operator, in regulating the contribution of online estimates. It is also worth noting that methods such as ScDQN and FR-DQN require additional hyperparameters, whereas MINTO introduces none.

Given the similarity of our method to the ensemble-based approach Maxmin DQN (Lan et al., 2020), which also employs a minimum operator, it is essential to benchmark MINTO against this prior work. Maxmin DQN applies the minimum operator across multiple estimates produced by an ensemble of target networks, with the goal of reducing the overestimation bias introduced by the maximum operator in the Bellman optimality equation. For this comparison, we also report results in Fig. [3](https://arxiv.org/html/2510.02590v1#S5.F3) using the AUC metric. Although Maxmin DQN relies on an ensemble of $Q$-functions ($N=2$ in our study), MINTO achieves better performance. This demonstrates the advantage of leveraging up-to-date estimates from the online network, in combination with the minimum operator, to mitigate overestimation bias that can arise from blindly incorporating online estimates. Simultaneously, MINTO avoids the additional memory overhead required by Maxmin DQN.

### 5.2 Distributional RL

Figure: Figure 4: Results of benchmarking MINTO+IQN and IQN on $15$ Atari games in the online RL setting using the CNN architecture. All metrics are computed over $5$ seeds. Left: We report the AUC metric computed by the IQM and its confidence interval. Center: We demonstrate the performance learning curves of both methods in terms of IQM return and confidence interval. Right: We report the frequency of selecting the online estimate during the course of training on the Breakout game, considering the mean and standard deviation.
Refer to caption: x4.png

We also evaluate MINTO within the context of distributional RL (Bellemare et al., 2017), an advanced value-based paradigm that has achieved state-of-the-art performance on the Atari benchmark in the online RL setting (Castro et al., 2018). Specifically, we consider Implicit Quantile Networks (IQN) (Dabney et al., 2018), a distributional RL method that employs quantile regression to approximate the entire return distribution rather than only its expected value. IQN accomplishes this by learning an implicit quantile function for the state–action values. For our experiments, we follow the implementation guidelines provided by Castro et al. (2018) when benchmarking IQN on the Atari suite. Comprehensive implementation details and hyperparameter settings are reported in Appendix [C.2](https://arxiv.org/html/2510.02590v1#A3.SS2).

We evaluate the effect of integrating MINTO into IQN by modifying its target computation according to our proposed technique. Appendix [B](https://arxiv.org/html/2510.02590v1#A2) provides further details, including pseudo-code illustrating the changes introduced (see Alg. [2](https://arxiv.org/html/2510.02590v1#alg2)). We benchmark MINTO+IQN against the original IQN on $15$ Atari games using the standard CNN architecture. In Fig. [4](https://arxiv.org/html/2510.02590v1#S5.F4)(left), we report the AUC metric for both methods, which captures improvements in both learning speed and asymptotic performance. MINTO enhances IQN by approximately $7\%$ according to this metric. Fig. [4](https://arxiv.org/html/2510.02590v1#S5.F4)(center) further presents the learning curves, illustrating consistent gains in sample efficiency and final performance. While the improvement is smaller than in the DQN case (and in later cases we consider), these results clearly demonstrate that MINTO enhances the performance of IQN, establishing it as a general and effective improvement even for state-of-the-art distributional methods.

To verify that the online network is selected frequently during training, we track the online selection ratio and report it for a representative game, Breakout, in Fig. [4](https://arxiv.org/html/2510.02590v1#S5.F4)(right). The results show that the target network dominates in the very early stages of training, while the use of the online network gradually increases as training progresses. Later in training, the online network is selected approximately $45\%$ of the time. Each point on the curve corresponds to the average online selection ratio between two consecutive target network updates, during which the parameters of the online network increasingly diverge from those of the target network.

### 5.3 Offline RL

Figure: Figure 5: Results of benchmarking CQL and CQL+MINTO on $14$ Atari games with CNN, and IMPALA with LayerNorm (LN) architectures. All results are reported using IQM and confidence interval on $5$ seeds and over all games. Left: We report the AUC metric for both methods while utilizing the two architectures. Center and Right: We illustrate the performance learning curves of CQL and CQL+MINTO after employing both architecture options.
Refer to caption: x5.png

The applicability of our method extends beyond online RL. Given its simplicity, MINTO can be readily incorporated into offline RL methods by modifying the Bellman update rule, allowing us to investigate its potential in this setting. Offline RL aims to learn an optimal policy from a large and static dataset of previously collected experience, without any further interaction with the environment. A central challenge in this paradigm is the distributional shift problem: the learned policy may query the Q-function on state–action pairs absent from the dataset. Overestimation of these out-of-distribution actions can then misguide the policy toward suboptimal behaviors.

We consider a popular offline RL algorithm, Conservative Q-Learning (CQL), which regularizes Q-function learning by penalizing out-of-distribution values while encouraging in-distribution values. To integrate MINTO into this framework, we simply modify the computation of the regression target using Eq. [3](https://arxiv.org/html/2510.02590v1#S4.E3) (see Alg. [3](https://arxiv.org/html/2510.02590v1#alg3)). Additional implementation details and hyperparameters are provided in Appendix [C.3](https://arxiv.org/html/2510.02590v1#A3.SS3).

In Fig. [5](https://arxiv.org/html/2510.02590v1#S5.F5), we present the results of evaluating our method, denoted as CQL+MINTO, against the original CQL on $14$ Atari games using both the vanilla CNN architecture and the IMPALA architecture with LN. For the offline setting, we use the datasets provided by Gulcehre et al. (2020). We note that one game, Tutankham, is excluded from our evaluation since it is not available in the released dataset.

The results of our experiments demonstrate a clear benefit of applying MINTO to offline RL. MINTO consistently improves the performance of CQL in terms of both sample efficiency and final performance, as reflected in the AUC metric (see Fig. [5](https://arxiv.org/html/2510.02590v1#S5.F5) (left)) and the learning curves (see Fig. [5](https://arxiv.org/html/2510.02590v1#S5.F5) (center and right)). In particular, MINTO tremendously boosts the performance of CQL with a CNN architecture by roughly $125\%$ in AUC. Although the performance gain is smaller with the IMPALA with LN architecture, MINTO still improves CQL by around $20\%$, with a clearer sample-efficiency advantage visible in Fig. [5](https://arxiv.org/html/2510.02590v1#S5.F5) (right). These findings are noteworthy, as one might expect our modified update rule to yield more conservative estimates in CQL, potentially slowing down learning. Instead, the improvements are substantial, highlighting the central role of recent online estimates, an aspect overlooked by the original CQL formulation..

### 5.4 Online RL and Continuous Control

Figure: Figure 6: Results of evaluating the impact of MINTO on SimbaV2 (top) and SimbaV1 (bottom) on three continuous control benchmarks: MuJoCo (left), HBench (center), and DMC-Hard (right). We show the performance curves of all methods on each benchmark while reporting a normalized IQM return and confidence interval computed on $10$ seeds.
Refer to caption: x6.png

Similarly, MINTO is not restricted to value-based methods. In off-policy actor–critic algorithms, where a dedicated network models the policy, and the $Q$-function (critic) is trained using bootstrapped targets from a target critic network, our proposed update rule can also be applied. There are multiple ways to incorporate MINTO into actor–critic methods, depending in particular on the number of critic networks employed. We defer a discussion of these design choices to Appendix [B](https://arxiv.org/html/2510.02590v1#A2).

To examine the effect of incorporating up-to-date estimates in the actor–critic setting, we adopt two recent architectures that are used in combination with Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), namely SimbaV1 (Lee et al., 2024) and SimbaV2 (Lee et al., 2025). SAC is a widely used actor–critic algorithm that seeks to learn an optimal stochastic policy by encouraging exploration through entropy maximization. In contrast, SimbaV1 and SimbaV2 are recently proposed architectures designed to scale performance with model size by leveraging the concept of simplicity bias, implemented in practice through variants of LN and residual connections. In Alg. [4](https://arxiv.org/html/2510.02590v1#alg4), we highlight the modifications introduced to SAC in order to integrate MINTO.

We benchmark SimbaV1 and SimbaV2 with and without our proposed method, MINTO. The evaluation is conducted on $26$ continuous control tasks, including both manipulation and locomotion, drawn from three different benchmarks: MuJoCo (Todorov et al., 2012), Humanoid Bench (HBench) (Sferrazza et al., 2024), and the DeepMind Control Suite Hard (DMC-Hard) (Tassa et al., 2018). For each actor–critic method, we report the aggregated performance curves on all three benchmarks. Additional details on the experimental setup and hyperparameters are provided in Appendix [C.4](https://arxiv.org/html/2510.02590v1#A3.SS4).

In Fig. [6](https://arxiv.org/html/2510.02590v1#S5.F6), we observe a clear improvement in sample efficiency when applying our method on the MuJoCo and HBench benchmarks. In contrast, performance on DMC-Hard is comparable to the base algorithm (SimbaV1) or slightly lower (as with SimbaV2). Overall, however, MINTO yields a consistent positive impact across all benchmarks and both actor–critic methods, suggesting the value of incorporating recent online estimates when computing regression targets in continuous control and actor–critic settings.

## 6 Conclusion

We introduce MINTO, a simple bootstrapping rule that combines online and target networks by taking their minimum estimate. This design mitigates the overestimation bias that can arise from blindly incorporating online estimates into target computation, thereby alleviating the moving-target problem, while still exploiting recent updates for faster learning. Across online, offline, value-based, and actor–critic methods, MINTO consistently improves sample efficiency and final performance without introducing additional hyperparameters and with only negligible overhead. These results establish MINTO as a practical and effective alternative to conventional target-network designs, pointing toward a promising direction for advancing stable and efficient deep RL.
Although MINTO consistently improves performance across diverse settings, there are natural trade-offs to consider. Relying exclusively on the minimum operator may, in some cases, be overly conservative in low-noise environments, leading to slight underestimation. In addition, by dampening optimistic estimates, MINTO may interact with exploration strategies in ways that are not yet fully understood. These observations open promising avenues for future research, such as adaptive operator selection that dynamically balance online and target estimates based on uncertainty or learning dynamics. Beyond these considerations, a separate line of future work lies in testing MINTO in additional challenging scenarios, such as multi-task and multi-agent reinforcement learning, to assess its scalability and broader applicability.

#### Acknowledgments

This work was funded by the German Federal Ministry of Education and Research (BMBF) (Project: 01IS22078). This work was also funded by Hessian.ai through the project ‘The Third Wave of Artificial Intelligence – 3AI’ by the Ministry for Science and Arts of the state of Hessen. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project b187cb. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683. Calculations for this research were also conducted on the Lichtenberg highperformance computer of the TU Darmstadt and the Intelligent Autonomous Systems (IAS) cluster at TU Darmstadt.

#### Reproducibility Statement

We have taken several measures to ensure the reproducibility of our results. To illustrate the simplicity of our approach, we provide in the appendix both a JAX code snapshot showing how MINTO can be integrated into DQN with minimal changes to the target computation and pseudocode for all algorithms where we integrate our method. We will release the full codebase publicly soon. Detailed descriptions of the experimental setup, architectures, and hyperparameters are provided in Appendix [C](https://arxiv.org/html/2510.02590v1#A3). For theoretical results, we clearly state assumptions and include the complete proof of convergence in Appendix [A](https://arxiv.org/html/2510.02590v1#A1). For empirical evaluation, we report results across multiple random seeds, following standard evaluation protocols (Machado et al., 2018), and provide per-task breakdowns in Appendix [D](https://arxiv.org/html/2510.02590v1#A4). Offline RL experiments use publicly available datasets (RL Unplugged), and we specify all data processing and training procedures in the appendix. Together, these details should facilitate exact replication and further validation of our findings.

#### Large Language Model Usage

A large language model was helpful in polishing writing, improving reading flow, and identifying remaining typos.

## References

- Andrychowicz et al. (2020)
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al.
Learning dexterous in-hand manipulation.
*The International Journal of Robotics Research*, 39(1):3–20, 2020.
- Bellemare et al. (2013)
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.
The arcade learning environment: An evaluation platform for general agents.
*Journal of artificial intelligence research*, 47:253–279, 2013.
- Bellemare et al. (2017)
Marc G Bellemare, Will Dabney, and Rémi Munos.
A distributional perspective on reinforcement learning.
In *International conference on machine learning*, pp. 449–458. PMLR, 2017.
- Bellman (1957)
Richard Bellman.
*Dynamic Programming*.
Princeton University Press, Princeton, NJ, USA, 1 edition, 1957.
- Bhatt et al. (2024)
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters.
Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Bradbury et al. (2018)
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.
JAX: composable transformations of Python+NumPy programs, 2018.
URL [http://github.com/jax-ml/jax](http://github.com/jax-ml/jax).
- Castro et al. (2018)
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare.
Dopamine: A research framework for deep reinforcement learning.
*arXiv preprint arXiv:1812.06110*, 2018.
- Dabney et al. (2018)
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos.
Implicit quantile networks for distributional reinforcement learning.
In *International conference on machine learning*, pp. 1096–1105. PMLR, 2018.
- Espeholt et al. (2018)
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.
In *International conference on machine learning*, pp. 1407–1416. PMLR, 2018.
- Gallici et al. (2025)
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin.
Simplifying deep temporal difference learning.
In *International Conference on Learning Representations*, 2025.
- Gan et al. (2021)
Yaozhong Gan, Zhe Zhang, and Xiaoyang Tan.
Stabilizing q learning via soft mellowmax operator.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2021.
- Graesser et al. (2022)
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro.
The state of sparse training in deep reinforcement learning.
In *International Conference on Machine Learning*, pp. 7766–7792. PMLR, 2022.
- Gulcehre et al. (2020)
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al.
Rl unplugged: A suite of benchmarks for offline reinforcement learning.
*Advances in neural information processing systems*, 33:7248–7259, 2020.
- Haarnoja et al. (2018a)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In *International conference on machine learning*, pp. 1861–1870. Pmlr, 2018a.
- Haarnoja et al. (2018b)
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.
Soft actor-critic algorithms and applications.
*arXiv preprint arXiv:1812.05905*, 2018b.
- Hasselt (2010)
Hado Hasselt.
Double q-learning.
*Advances in neural information processing systems*, 23, 2010.
- Hessel et al. (2018)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In *Proceedings of the AAAI conference on artificial intelligence*, 2018.
- Ioffe & Szegedy (2015)
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing internal covariate shift.
In *International conference on machine learning*, 2015.
- Kim et al. (2019)
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris.
Deepmellow: removing the need for a target network in deep q-learning.
In *Proceedings of the twenty eighth international joint conference on artificial intelligence*, 2019.
- Lan et al. (2020)
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White.
Maxmin q-learning: Controlling the estimation bias of q-learning.
In *International Conference on Learning Representations*, 2020.
- Lee et al. (2024)
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno.
Simba: Simplicity bias for scaling up parameters in deep reinforcement learning.
In *The Thirteenth International Conference on Learning Representations*, 2024.
- Lee et al. (2025)
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo.
Hyperspherical normalization for scalable deep reinforcement learning.
*arXiv preprint arXiv:2502.15280*, 2025.
- Li & Pathak (2021)
Alexander Li and Deepak Pathak.
Functional regularization for reinforcement learning via learned fourier features.
In *Advances in Neural Information Processing Systems*, 2021.
- Lindström et al. (2025)
Alexander Lindström, Arunselvan Ramaswamy, and Karl-Johan Grinnemo.
Pre-training deep q-networks eliminates the need for target networks: An empirical study.
In *The 14th International Conference on Pattern Recognition Applications and Methods (ICPRAM)*, 2025.
- Lu et al. (2025)
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang.
Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning.
*arXiv preprint arXiv:2505.18719*, 2025.
- Machado et al. (2018)
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling.
Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.
*Journal of Artificial Intelligence Research*, 61:523–562, 2018.
- Mnih et al. (2013)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.
Playing atari with deep reinforcement learning.
*arXiv preprint arXiv:1312.5602*, 2013.
- Mnih et al. (2015)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
*nature*, 518(7540):529–533, 2015.
- Nauman et al. (2024)
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan.
Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control.
*Advances in neural information processing systems*, 37:113038–113071, 2024.
- Palenicek et al. (2025)
Daniel Palenicek, Florian Vogt, and Jan Peters.
Scaling off-policy reinforcement learning with batch and weight normalization.
In *Advances in Neural Information Processing Systems*, 2025.
- Piché et al. (2023)
Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan.
Bridging the gap between target networks and functional regularization.
*Transactions on Machine Learning Research*, 2023.
- Puterman (1990)
Martin L Puterman.
Markov decision processes.
*Handbooks in operations research and management science*, 2:331–434, 1990.
- Rudin et al. (2022)
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter.
Learning to walk in minutes using massively parallel deep reinforcement learning.
In *Conference on robot learning*, pp. 91–100. PMLR, 2022.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Sferrazza et al. (2024)
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel.
Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation.
*arXiv preprint arXiv:2403.10506*, 2024.
- Shao et al. (2022)
Lin Shao, Yifan You, Mengyuan Yan, Shenli Yuan, Qingyun Sun, and Jeannette Bohg.
Grac: Self-guided and self-regularized actor-critic.
In *Conference on Robot Learning*, 2022.
- Silver et al. (2016)
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
*nature*, 529(7587):484–489, 2016.
- Silver et al. (2017)
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.
Mastering chess and shogi by self-play with a general reinforcement learning algorithm.
*arXiv preprint arXiv:1712.01815*, 2017.
- Tassa et al. (2018)
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.
Deepmind control suite.
*arXiv preprint arXiv:1801.00690*, 2018.
- Todorov et al. (2012)
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In *2012 IEEE/RSJ international conference on intelligent robots and systems*, pp. 5026–5033. IEEE, 2012.
- Van Hasselt et al. (2016)
Hado Van Hasselt, Arthur Guez, and David Silver.
Deep reinforcement learning with double q-learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 30, 2016.
- Van Hasselt et al. (2018)
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil.
Deep reinforcement learning and the deadly triad.
*arXiv preprint arXiv:1812.02648*, 2018.
- Vincent et al. (2025)
Théo Vincent, Yogesh Tripathi, Tim Faust, Yaniv Oren, Jan Peters, and Carlo D’Eramo.
Bridging the performance gap between target-free and target-based reinforcement learning with iterated q-learning.
*European Workshop on Reinforcement Learning*, 2025.
- Watkins & Dayan (1992)
Christopher JCH Watkins and Peter Dayan.
Q-learning.
*Machine learning*, 8(3):279–292, 1992.
- Zhang et al. (2021)
Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson.
Breaking the deadly triad with a target network.
In *International Conference on Machine Learning*, pp. 12621–12631. PMLR, 2021.
- Zhu & Rigotti (2021)
Rong Zhu and Mattia Rigotti.
Self-correcting q-learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 35, pp. 11185–11192, 2021.

## Appendix

## Appendix A Proof of Corollary 1

To prove the convergence of MINTO, we show that its target operator, $G^{\text{MINTO}}$, satisfies the two conditions of Assumption 1 from the Generalized Q-learning framework Lan et al. (2020).

Assumption 1 (Conditions on G) Let $G:\mathbb{R}^{n\times N\times K}\mapsto\mathbb{R}$ be the target operator where $Q_{s}=(Q_{sa}^{ij})\in\mathbb{R}^{n\times N\times K}$, $a\in\mathcal{A}$, and $|\mathcal{A}|=n$, $i\in\{1,\dots N\}$, $j\in\{0,\dots,K-1\}$, and $s\in\mathcal{S}$ is an arbitrary state. G must satisfy:

- A1.1:
If all input action-values are identical, $Q_{sa}^{ij}=Q_{sa}^{kl},\forall i,k,\forall j,k,$ and $\forall a$, then
$G(Q_{s})=\max_{a}Q_{sa}^{ij}$.
- A1.2:
G is a non-expansion w.r.t. the max norm, $|G(Q_{s})-G(Q^{\prime}_{s})|\leq\max_{a,i,j}|Q_{sa}^{ij}-Q^{\prime ij}_{sa}|$.

The action-value function $Q_{s}$ is represented by a tensor with an ensemble of $N$ action-value functions and for each, $K$ historical time action-values.

The MINTO operator is defined as $G^{\text{MINTO}}(Q_{s})=\max_{a\in\mathcal{A}}\left(\min_{j\in\mathcal{T}}Q_{sa}(j)\right)$ for a given state $s$, where $\mathcal{T}$ is a set of historical time indices, and $j$ is the time index with an abuse of notations. This is a special case of the Generalized Q-learning where $N=1$ and $K=2$, hence dropping the $i$ index. In the analysis, we consider a general variant of MINTO, where $K>1$, by considering a set of historical time values using the indices set $\mathcal{T}$ such that $Q_{sa}=\big(Q_{sa}(t-K),\dots,Q_{sa}(t-1)\big)$ for a given state and action. In practice, $\mathcal{T}$ includes only $(t-1)$ and $(t-K)$, representing the online and target network estimates, respectively.

### A.1 Proof of Condition A1.1

Assume the input Q-values $Q_{sa}(j)$ are identical for all $j\in\mathcal{T}$ and all $a\in\mathcal{A}$.

$$ $\displaystyle G^{\text{MINTO}}(Q_{s})$ $\displaystyle=\max_{a\in\mathcal{A}}\left(\min_{j\in\mathcal{T}}Q_{sa}(j)\right)$ $\displaystyle=\max_{a\in\mathcal{A}}Q_{sa}(j)$ $$

This is the maximum action-value among the inputs. Thus, A1.1 is satisfied.

### A.2 Proof of Condition A1.2

Let $Q_{s}$ and $Q^{\prime}_{s}$ be two distinct sets of historical Q-values for a given state $s$, assuming that $N=1$.

$$ $\displaystyle|G^{\text{MINTO}}(Q_{s})-G^{\text{MINTO}}(Q^{\prime}_{s})|$ $\displaystyle=\left|\max_{a\in\mathcal{A}}\left(\min_{j\in\mathcal{T}}Q_{sa}(j)\right)-\max_{a\in\mathcal{A}}\left(\min_{j\in\mathcal{T}}Q^{\prime}_{sa}(j)\right)\right|$ (5) $\displaystyle\leq\max_{a\in\mathcal{A}}\left|\min_{j\in\mathcal{T}}Q_{sa}(j)-\min_{j\in\mathcal{T}}Q^{\prime}_{sa}(j)\right|$ (6) $\displaystyle\leq\max_{a\in\mathcal{A}}\left(\max_{j\in\mathcal{T}}\left|Q_{sa}(j)-Q^{\prime}_{sa}(j)\right|\right)$ (7) $\displaystyle=\max_{a\in\mathcal{A},j\in\mathcal{T}}|Q_{sa}(j)-Q^{\prime}_{sa}(j)|$ (8) $$

The first inequality holds because the $\max$ operator is a non-expansion. The second inequality holds because the $\min$ operator is also a non-expansion.

The final term is the maximum absolute difference over the subset of Q-values used by the MINTO operator. This maximum is necessarily less than or equal to the maximum over the entire set of all possible Q-values, i.e.:

$$ $\max_{a\in\mathcal{A},j\in\mathcal{T}}|Q_{sa}(j)-Q^{\prime}_{sa}(j)|\leq\max_{a,j}|Q_{sa}(j)-Q^{\prime}_{sa}(j)|$ $$

Therefore, $|G^{\text{MINTO}}(Q_{s})-G^{\text{MINTO}}(Q^{\prime}_{s})|\leq\max_{a,j}|Q_{sa}(j)-Q^{\prime}_{sa}(j)|$. Thus, A1.2 is satisfied.

Conclusion: Since both conditions of Assumption 1 are satisfied, the convergence of MINTO is guaranteed by Theorem 2 of Lan et al. (2020) under the standard stochastic approximation assumptions for learning rate stated in Assumption 2 of Lan et al. (2020).

## Appendix B Algorithmic Details

The pseudocode blocks below present MINTO and its extension to IQN, CQL, and SAC in Alg. [1](https://arxiv.org/html/2510.02590v1#alg1), Alg. [2](https://arxiv.org/html/2510.02590v1#alg2), Alg. [3](https://arxiv.org/html/2510.02590v1#alg3), and Alg. [4](https://arxiv.org/html/2510.02590v1#alg4) respectively. Changes in the pseudocode of the underlying algorithms are colored red. Note that $\lceil{\cdot}\rceil$ denotes the stop-gradient operator. For MINTO we want to explicitly stop the gradient, as the target computation relies on both the online as well as the target parameters. In contrast to other methods, like DoubleDQN, for example, the gradient is not stopped by the $\arg\max$ operator, as the online values are used directly.

Figure: Algorithm 1 DQN+MINTO.

Figure: Algorithm 2 IQN+MINTO

Figure: Algorithm 3 CQL+MINTO

In Alg. [4](https://arxiv.org/html/2510.02590v1#alg4), SAC is adapted to use a single Q-function critic, following the approach taken in Simba (Lee et al. (2024; 2025)) on many benchmarks, which also relies on a single critic. This choice motivated our design, as it simplifies computation while remaining effective when combined with the MINTO operator in the target computation. In practice, we also experimented with a twin-critic setup (see Fig. [13](https://arxiv.org/html/2510.02590v1#A4.F13)-[16](https://arxiv.org/html/2510.02590v1#A4.F16)). However, the largest performance gains were observed with the single-critic variant, since applying the minimum operator on all online and target networks, in the twin-critic case, can lead to conservative updates. On the other hand, the stochastic policy and temperature updates are retained as in standard SAC.

Figure: Algorithm 4 SAC+MINTO.

Overall, the MINTO modifications presented in Alg. [1](https://arxiv.org/html/2510.02590v1#alg1), [2](https://arxiv.org/html/2510.02590v1#alg2), [3](https://arxiv.org/html/2510.02590v1#alg3), and [4](https://arxiv.org/html/2510.02590v1#alg4) are straightforward to implement and can be integrated with minimal changes to the underlying algorithms. This makes MINTO a practical approach for extending existing value-based and actor-critic methods without introducing significant complexity.

## Appendix C Implementation Details

### C.1 Online RL and Discrete Control

For our online reinforcement learning experiments in the discrete-action setting, we use Atari environments as the benchmark domain. The implementations of DoubleDQN, FR-DQN, ScDQN, and MINTO share the same lightweight and reproducible framework for DQN variants written in JAX. The codebase will be shared upon acceptance. All algorithms share the same network architectures and training setups, differing only in the algorithm-specific modifications detailed in Tables [1](https://arxiv.org/html/2510.02590v1#A3.T1) and [2](https://arxiv.org/html/2510.02590v1#A3.T2). MINTO is implemented by modifying the target computation.

Figure: Listing 1: JAX implementation of the MINTO TD target.

**Table 1: Hyperparameter settings for DQN, MaxMinDQN, and MINTO on Atari. Most parameters are shared, with algorithm-specific hyperparameters explicitly listed. Identical values are merged for clarity.**
| Hyperparameter | DQN | MaxMinDQN | MINTO |
| --- | --- | --- | --- |
| Replay Buffer Capacity | 1,000,000 |  |  |
| Batch Size | 32 |  |  |
| Update Horizon | 1 |  |  |
| Discount Factor ($\gamma$) | 0.99 |  |  |
| Learning Rate | $6.25\times 10^{-5}$ |  |  |
| Horizon | 27,000 |  |  |
| Architecture Type | CNN |  |  |
| Features | [32, 64, 64, 512] |  |  |
| Epochs | 100 |  |  |
| Training Steps per Epoch | 250,000 |  |  |
| Data to Update | 4 |  |  |
| Initial Samples | 20,000 |  |  |
| Epsilon End | 0.01 |  |  |
| Epsilon Duration | 250,000 |  |  |
| Target Update Frequency ($T$) | 8,000 |  |  |
| Ensemble Size ($N$) | - | 2 | - |

**Table 2: Hyperparameter settings for the four DQN variants evaluated on Atari, including DoubleDQN, FR-DQN, ScDQN, and MINTO. Most parameters are shared across all algorithms, while the table explicitly lists the algorithm-specific hyperparameters, the regularization parameter $\kappa$ for FR-DQN and the self-correcting parameter $\beta$ for ScDQN. Identical values are merged for clarity.**
| Hyperparameter | DoubleDQN | FR-DQN | ScDQN | MINTO |
| --- | --- | --- | --- | --- |
| Replay Buffer Capacity | 1,000,000 |  |  |  |
| Batch Size | 32 |  |  |  |
| Update Horizon | 1 |  |  |  |
| Discount Factor ($\gamma$) | 0.99 |  |  |  |
| Learning Rate | $6.25\times 10^{-5}$ |  |  |  |
| Horizon | 27,000 |  |  |  |
| Architecture Type | CNN |  |  |  |
| Features | [32, 64, 64, 512] |  |  |  |
| Epochs | 100 |  |  |  |
| Training Steps per Epoch | 250,000 |  |  |  |
| Data to Update | 4 |  |  |  |
| Initial Samples | 20,000 |  |  |  |
| Epsilon End | 0.01 |  |  |  |
| Epsilon Duration | 250,000 |  |  |  |
| Target Update Frequency ($T$) | 8,000 |  |  |  |
| Regularization Parameter ($\kappa$) | - | 1.0 | - |  |
| Self Correcting Parameter ($\beta$) | - | 3.0 | - |  |

### C.2 Distributional RL

For the distributional reinforcement learning experiments, we use Implicit Quantile Networks (IQN). The implementation builds directly on the same JAX codebase as the DQN experiments, ensuring consistency in architecture and training setup. The key algorithm-specific hyperparameters are listed in Table [3](https://arxiv.org/html/2510.02590v1#A3.T3).

**Table 3: Comparison of IQN hyperparameters for $n=1$ using IQN and IQN+MINTO. Update horizon is set to 1.**
| Hyperparameter | IQN | IQN+MINTO |
| --- | --- | --- |
| Replay Buffer Capacity | 1,000,000 |  |
| Batch Size | 32 |  |
| Update Horizon ($n$) | 1 |  |
| Discount Factor ($\gamma$) | 0.99 |  |
| Learning Rate | 5.0 $\times 10^{-5}$ |  |
| Adam $\epsilon$ | 3.125 $\times 10^{-4}$ |  |
| Horizon | 27,000 |  |
| Architecture Type | CNN |  |
| Features | [32, 64, 64, 512] |  |
| Epochs | 100 |  |
| Training Steps per Epoch | 250,000 |  |
| Data to Update | 4 |  |
| Initial Samples | 20,000 |  |
| Epsilon End | 0.01 |  |
| Epsilon Duration | 250,000 |  |
| Target Update Frequency ($T$) | 8000 |  |

### C.3 Offline RL

For the offline reinforcement learning experiments on Atari, we use datasets from RL Unplugged(^1^11[https://github.com/huihanl/rl_unplugged](https://github.com/huihanl/rl_unplugged)) (Gulcehre et al. (2020)), which provide standardized and diverse benchmarks. The implementation is built on a stable and well-tested codebase to ensure reproducibility and fair comparison. The code will be shared upon acceptance. All methods are run with their default hyperparameters, and the most important settings are reported in Table [4](https://arxiv.org/html/2510.02590v1#A3.T4).

**Table 4: Comparison of CQL and CQL+MINTO hyperparameters for the two different network architectures CNN and IMPALA.**
| Hyperparameter | CNN | IMPALA |  |  |
| --- | --- | --- | --- | --- |
|  | CQL | CQL+MINTO | CQL | CQL+MINTO |
| Dataset Size | 5,000,000 |  |  |  |
| Batch Size | 32 |  |  |  |
| Update Horizon | 1 |  |  |  |
| Discount Factor ($\gamma$) | 0.99 |  |  |  |
| Epochs | 100 |  |  |  |
| Learning Rate | $5\times 10^{-5}$ |  |  |  |
| Adam ($\epsilon$) | $5\times 3.125^{-4}$ |  |  |  |
| Training Steps per Epoch | 62,500 |  |  |  |
| Tradeoff Factor ($\alpha$) | 0.1 |  |  |  |
| Target Update Frequency ($T$) | 2000 |  |  |  |
| Layer Norm | no | yes |  |  |

### C.4 Online RL and Continuous Control

For our continuous-control experiments with online reinforcement learning, we adopt SimbaV1 and SimbaV2. The implementation is based on the official SimbaV2 codebase(^2^22[https://github.com/dojeon-ai/SimbaV2](https://github.com/dojeon-ai/SimbaV2)) (Lee et al. (2025)), ensuring consistency with the original work. All experiments use the default hyperparameters provided by the authors, with the most relevant ones summarized in Tables [5](https://arxiv.org/html/2510.02590v1#A3.T5) and [6](https://arxiv.org/html/2510.02590v1#A3.T6).

**Table 5: Comparison of SimbaV1 hyperparameters across DMC-Hard, HumanoidBench, and Mujoco locomotion environments. Identical values are merged.**
| Hyperparameter | SimbaV1,   SimbaV1+MINTO |  |  |
| --- | --- | --- | --- |
|  | DMC-Hard | HumanoidBench | MuJoCo |
| Discount Factor ($\gamma$) | 0.99 | 0.995 |  |
| Learning Rate | $1.0\times 10^{-4}$ |  |  |
| Weight Decay | 0.01 |  |  |
| Target ($\tau$) | 0.005 |  |  |
| Update Horizon ($n$) | 1 |  |  |
| Temperature Initial Value | 0.01 |  |  |
| Temperature Target Entropy | $-0.5\times|\mathcal{A}|$ |  |  |
| Batch Size | 256 |  |  |
| Buffer Max Length | 1,000,000 |  |  |
| Buffer Min Length | 5,000 |  |  |
| Num Train Envs | 1 |  |  |
| Action Repeat | 2 | 1 |  |
| Max Episode Steps | 1000 |  |  |

**Table 6: Comparison of SimbaV2 hyperparameters across DMC-Hard, Humanoid Bench, and MuJoCo. Identical values are merged for clarity.**
| Hyperparameter | SimbaV2,   SimbaV2+MINTO |  |  |
| --- | --- | --- | --- |
|  | DMC-Hard | HumanoidBench | MuJoCo |
| Actor Shift | 3 |  |  |
| Critic Shift | 3 |  |  |
| Critic $v_{\max}$ | 5.0 |  |  |
| Critic $v_{\min}$ | $-5.0$ |  |  |
| Critic Num Bins | 101 |  |  |
| Discount Factor ($\gamma$) | 0.99 | 0.995 |  |
| Learning Rate Init | $1.0\times 10^{-4}$ |  |  |
| Learning Rate End | $5.0\times 10^{-5}$ |  |  |
| Update Horizon ($n$) | 1 |  |  |
| Target ($\tau$) | 0.005 |  |  |
| Temperature Initial Value | 0.01 |  |  |
| Temperature Target Entropy | $-0.5\times|\mathcal{A}|$ |  |  |
| Buffer Max Length | 1,000,000 |  |  |
| Buffer Min Length | 5,000 |  |  |
| Num Train Envs | 1 |  |  |
| Action Repeat | 2 | 1 |  |
| Max Episode Steps | 1000 |  |  |

## Appendix D Individual Results

### D.1 Online RL and Discrete Control

Figure: Figure 7: Individual Results of benchmarking MINTO and DQN on 15 Atari games using the IMPALA architecture with LayerNorm. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x7.png

Figure: Figure 8: Individual Results of benchmarking MINTO and DQN on 15 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x8.png

Figure: Figure 9: Individual Results of benchmarking DoubleDQN, FR-DQN, ScDQN, and MINTO on 15 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x9.png

### D.2 Offline RL

Figure: Figure 10: Individual Results of benchmarking CQL and CQL+MINTO on 15 Atari games using the IMPALA architecture with LayerNorm. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x10.png

Figure: Figure 11: Individual Results of benchmarking CQL and CQL+MINTO on 15 Atari games using the CNN architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x11.png

### D.3 Maxmin DQN vs. MINTO

Figure: Figure 12: Individual Results of benchmarking MaxMinDQN and MINTO on 15 Atari games using the CNN network architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x12.png

### D.4 Online RL and Continuous Control

Figure: Figure 13: Individual Results of benchmarking SimbaV2 and SimbaV2+MINTO on the 5 MuJoCo environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 5 environments.
Refer to caption: x13.png

Figure: Figure 14: Individual Results of benchmarking SimbaV1 and SimbaV1+MINTO on the 5 MuJoCo environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 5 environments.
Refer to caption: x14.png

Figure: Figure 15: Individual Results of benchmarking SimbaV2 and SimbaV2+MINTO on the 14 Humanoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
Refer to caption: x15.png

Figure: Figure 16: Individual Results of benchmarking SimbaV1 and SimbaV1+MINTO on the 14 Humanoid Bench environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 14 environments.
Refer to caption: x16.png

Figure: Figure 17: Individual Results of benchmarking SimbaV2 and SimbaV2+MINTO on the 7 DMC-Hard environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 7 environments.
Refer to caption: x17.png

Figure: Figure 18: Individual Results of benchmarking SimbaV1 and SimbaV1+MINTO on the 7 DMC-Hard environments. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 10 seeds per environment. The last plot (bottom right) shows the IQM of the normalized return over all 7 environments.
Refer to caption: x18.png

### D.5 Ablation on Target Operators

Figure: Figure 19: Individual Results of benchmarking the Minimum operator of MINTO against other potential operators on 15 Atari games using the CNN architecture. Reported metrics are interquartile mean (IQM) scores with 95% confidence intervals across 5 seeds per game.
Refer to caption: x19.png

Figure: Figure 20: Cumulative results of benchmarking the Minimum operator of MINTO against other potential operators on 15 Atari games using the CNN architecture. Both figures show the interquartile mean (IQM) scores with 95% confidence intervals of the final performance of each operator. Left: The final performance of each operator in a bar chart. Right: The performance curve for each operator over 50 million frames.
Refer to caption: x20.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: "$Q$-learning offers a recursive update to approximate the state-action value function $Q$"

Should that last $Q$ be $Q^*$ instead of $Q$?

&nbsp;

does the "preliminaries" section allow for stochastic rewards? if not, is that intentional?

&nbsp;

"The foundation of this success lies primarily in Deep RL, initiated by the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013), which marked the first successful application of deep neural networks in RL."

Neural Fitted Q Iteration was also a success, even though DQN could maybe be considered an even bigger success

&nbsp;

"A problem that presents a challenge and obstacle"

why both?

&nbsp;

"In deep RL, the problem of moving targets is especially evident due to the use of neural networks and the resulting uncontrolled fluctuations in the values of unseen states."

what is this contrasting against? linear RL?

&nbsp;

"For example, Gallici et al. (2025) demonstrated that cleverly using parallel environments eliminates the need for a target network"

some people might consider "eliminates the need for a target network" as too strong of a claim

&nbsp;

"This makes MellowMax orthogonal to our approach."

you might say "somewhat orthogonal" to help avoid confusion

&nbsp;

"making this method orthogonal to our approach"

likewise

&nbsp;

"known as $Q$-function"

typo, should be "known as _a_ $Q$-function" (or "_the_ $Q$-function")

&nbsp;

"(Mnih et al., 2013) introduce a series of algorithmic components, more importantly, is the introduction of the target networks."

typos

&nbsp;

"Despite the success, this results in a slow learning of value function as well as the policy due to relying on out-dated estimates"

typo, should be "learning of _the_ value function"

&nbsp;

"bellman"

typo, should be "Bellman"

&nbsp;

"can we find a practical bellman update rule that results in a stable and fast learning?"

what does "practical" specify here?

&nbsp;

"When implemented in an efficient deep learning framework such as JAX (Bradbury et al., 2018), this overhead is negligible."

I don't understand. does JAX automatically parallelize the additional forward pass with the target network's forward pass?

&nbsp;

"addressing Q1.As"

typo

&nbsp;

why is Fig 1 with 10 Atari games, but Fig 2 with 15 games?

&nbsp;

Fig 2 shows CNN and IMPALA results on 100 vs 50 million frames. why not both 100 or both 50?

&nbsp;

"into both value-based and actor–critic methods"

this is maybe slightly confusing wording because actor-critic methods are partially value-based. maybe "both actor-critic and purely value-based methods" would be slightly clearer.

&nbsp;

"Offline RL aims to learn an optimal policy from a large and static dataset"

offline RL does not need a "large" dataset to be offline RL

&nbsp;

"A central challenge in this paradigm is the distributional shift problem: the learned policy may query the Q-function on state–action pairs absent from the dataset"

offline policy evaluation does not necessarily involve a learned policy. offline policy evaluation is also offline RL, and also may query the value function on state or state-action inputs not present in the dataset
soundness: 3
strengths: applies MINTO on many base algorithms (DQN, IQN, CQL, SAC) and architectures (CNN, IMPALA, SimbaV1, SimbaV2), and fairly consistently shows some improvements in returns

no additional hyperparameters

simple

a good number of fair baseline algorithms -- for ex, includes mean, max, random, and min of {online, target} nets in Fig 1, and later also includes Double DQN, FR-DQN, ScDQN, and Maxmin DQN

novel as far as I am aware

written clearly
summary: for the bootstrapping target in deep Q-learning, uses the minimum of the online net and target net value estimates, instead of using just the target net value estimate

they call their algorithm "MINTO"
weaknesses: I think it would be great if the paper were even more clear about how the hyperparameters were set. while Appendix C.3 says "All methods are run with their default hyperparameters", it's not immediately clear to me how the values in Table 1 were set. are they the default hyperparameters from any existing codebase? if not, why not, and how were they set?

5 seeds is not many seeds. (but the experiments cover so many base algorithms, architectures, codebases, and environment suites that I suspect this is not a big issue)

no tests on toy tasks, along the lines of Baird's counterexample. I think there is a non-negligible chance that MINTO will get "stuck" on some such problems, where both TD and DQN sometimes empirically require their loss to increase before it will decrease. see for ex Fig 1e here: https://openreview.net/pdf?id=j3bKnEidtT. MINTO empirically appears to work on a wide variety of other benchmarks though, so this whole bullet might be irrelevant (even including the possible case that MINTO fails on some toy tasks).

a bit verbose, for ex the paper calls MINTO "simple yet effective" 4 times (once with a comma after simple)

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Suggestions and comments(not directly impacting the score):
- In the ablations, it is reported that taking the mean between the online and target network estimates was not helpful. Do you have any hypotheses why this woudl be the case? Using the mean between two target critics has been found to sometimes an effective alternative to the min between two critics [1] and sometimes underestimation bias has been found to be harmful.


- The proposed method reminds me of the online network trust region method proposed in [2] since they both attempt to make use of the online network safely. It could be beneficial to include it as a baseline or provide some comparisons.

- The algorithm labelled  "Maxmin DQN" in Fig.3. seems to be very similar to the clipped Double-Q learning update proposed in the TD3 paper [3] and I had not seen any mention of this.  


[1] "Scaling for compute and sample-efficient continous control" Nauman et al.

[2] "Human-level Atari 200x faster" Kapturowski et al.

[3] "Addressing function approximation error in actor-critic methods" Fujimoto et al.
soundness: 4
strengths: The paper was easy to follow and the proposed technique is very simple to understand and implement.
The experiments are comprehensive and show that method works in a variety of settings (Q-learning algorithms, actor-critic, distributional RL, continuous-control, atari).
I appreciated testing the variants that would be most relevant such as mean or max of the online and target network estimates. 

In summary, I think the proposed technique would make a nice addition to the deep RL toolbox.
summary: The paper proposes a technique (MINTO) to make better use of the online network when also using a target network for learning the value function. The technique simply changes the update target value to be the minimum between the target network and online network's value estimates.
Experiments on a variety of benchmarks show that this technique is beneficial and can outperform common alternatives such as double Q-learning or using the minimum betweeen two target networks.
weaknesses: I did not identify any major weaknesses.

It could have been nice to include a more detailed analysis of why MINTO may be effective. In particular, thinking through the MINTO updates and comparing to using the target net as usual, the online network value replaces the target net value when the online network's value is lower. This means we can only use the most recent information (online network) if it is lowering the bootstrap target. 
I wonder if you could study the accuracy of the value estimates in a policy evaluation setting where we could accurately estimate true value functions. Perhaps MINTO produces better estimates overall.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 4
questions: **Q1. Missing comparison with Clipped Q-learning and related baselines**
* To fully support the claim, the proposed method should be experimentally compared with other bias reduction methods (e.g., methods introduced in DQN experiment). 
* Among those, I think at least **comparison with Clipped Q-learning [1] is necessary**, which is equivalent to MaxMin DQN with N=2 --- the strongest baseline reported for DQN in the paper, and also widely used.

I would like to emphasize that I believe this paper is strong and has substantial potential. However, this point is a critical concern for me. I currently lean toward a weak reject, but if the authors can provide the requested experimental comparisons (e.g., with Clipped Q-learning), I would be inclined to raise my recommendation, even significantly if the results meet the expectations.
soundness: 2
strengths: **S1. Simplicity of the method**
* The proposed method is conceptually simple and easy to implement; it only modifies the target calculation by taking the minimum between the target and online Q-values. 
* This adjustment can be integrated easily into existing RL frameworks, as demonstrated in Algorithms 1–4. 
Such simplicity makes the method practical and broadly applicable.

**S2. Substantial improvement across diverse RL domains**
* Despite its simplicity, the method demonstrates strong empirical performance. 
* It improves results across various RL domains --- online, offline, and distributional settings --- and across both continuous and discrete action spaces. 
* The reported gains over baseline methods in the DQN setting, suggest that the proposed approach effectively mitigates overestimation bias than other methods.

**S3. Clear organization and presentation**
* The paper is well structured and easy to read.
* Experimental results are clearly presented.
summary: This paper proposes a new method for using the target network in RL. 
Specifically, the authors modify the target computation by taking the minimum of the target Q-network and the online Q-network. 
Albeit simple, it shows substantial improvements across several RL settings, including online RL (DQN, SAC (Simba-V1, Simba-V2)), distributional RL (IQN), and offline RL (CQL).
weaknesses: **W1. Comparison with other regularization methods limited in DQN setting**
* Currently, the comparison with other bias-reduction or regularization methods is limited to the DQN (discrete online RL) setting. 
This restricts the strength of the empirical claims, especially since the method is positioned as a general improvement applicable across multiple RL domains.
* Including comparisons with well-established bias-reduction methods in other settings (e.g., SAC, IQN, and CQL) will clarify whether the observed improvements generalize beyond DQN.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Have you studied the impact of potential underestimation? While you claim that it is slight, is there any theoretical bound to it?

2. The paper briefly mentions possible unexplored interactions between MINTO and exploration strategies. Can the authors clarify how MINTO affects exploratory behavior?
soundness: 3
strengths: 1. The paper is well-motivated and the minimum-operator is easy to integrate into existing RL algorithms.

2. The framework is validated on a wide set of benchmarks (Atari, MuJoCo, offline and online RL) and compared to several baselines, demonstrating consistent gains in both stability and sample efficiency.

3. The modifications require minimal changes to underlying algorithms and introduce only a tiny computational cost, making practical adoption straightforward.
summary: This paper proposes MINTO, a novel update rule for deep reinforcement learning that computes regression targets as the minimum estimate between the online and target networks. MINTO is designed to address the stability-efficiency trade-off inherent in standard RL, where target networks help stabilize learning but slow down convergence, while online networks foster efficiency but risk instability. Experiments on online, offline, discrete, and continuous RL benchmarks such as Atari games and robotic control tasks demonstrate consistent and significant improvements in sample efficiency and final performance over various baselines.
weaknesses: 1. The main evaluation is conducted against SimbaV1 and SimbaV2, which appear to be less strong baselines. This certainly raises the question whether this method can generalize to strong baselines such as Rainbow.

2. The exclusive reliance on the minimum operator can be too conservative, possibly causing underestimation and inhibiting exploration in low-noise or optimistic environments.

3. The code is not attached and will be released upon acceptance. This raises certain concerns.


Decision: Accept (Poster)