Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning
Authors: Ezgi Korkmaz
Venue: NeurIPS 2025 spotlight ()
Keywords: reinforcement learning
sample efficiency
scalable Deep RL
data efficiency
exploration
sample efficient reinforcement learning
temporal difference learning
Primary area: reinforcement_learning

Abstract: Following the pivotal success of learning strategies to win at tasks, solely by interacting with an environment without any supervision, agents have gained the ability to make sequential decisions in complex MDPs. Yet, reinforcement learning policies face exponentially growing state spaces in high dimensional MDPs resulting in a dichotomy between computational complexity and policy success. In our paper we focus on the agent’s interaction with the environment in a high-dimensional MDP during the learning phase and we introduce a theoretically-founded novel paradigm based on experiences obtained through counteractive actions. Our analysis and method provide a theoretical basis for efficient, effective, scalable and accelerated learning, and further comes with zero additional computational complexity while leading to significant acceleration in training. We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs. The experimental results further verify our theoretical analysis, and our method achieves significant performance increase with substantial sample-efficiency in high-dimensional environments.

Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning
Ezgi Korkmaz
Abstract
Following the pivotal success of learning strategies to win at tasks, solely by inter- acting with an environment without any supervision, agents have gained the ability to make sequential decisions in complex MDPs. Yet, reinforcement learning poli- cies face exponentially growing state spaces in high dimensional MDPs resulting in dichotomy between computational complexity and policy success. In our paper we focus on the agent's 's interaction with the environment in a high-dimensional MDP during the learning phase and we introduce paradigm based on experiences obtained through counteractive actions. Our anal- ysis and method provide theoretical basis for efficient, effective, scalable and accelerated learning, and further comes with zero additional computational com- plexity while leading to significant acceleration in training. We conduct extensive experiments in the Arcade Learning Enviromment with high- sigh-dimensional state representation MDPs. The experimental results further verify our theoretical anal- ysis, and our method achieves significant performance increase with substantial sample-efficiency in nigh-dimensional environments.
ceooreiccall-founndee novel
Introduction
Utilization of deep neural networks as function approximators enabled learning functioning policies in high-dimensional state representation MDPs (Mnih et al.) 2015). Following this initial work, the
current line of research trains deep reinforcement learning policies solve highly complex problems
from game solving [Hasselt et al. 2016 Schrittwieser et al.) 2020) to mathematical and scientific reasoning of large language models [Guo al.) [2025]. Yet, there are still remaining unsolved problems restricting the current capabilities of reinforcement learning in exponentially growing state spaces. One of the main intrinsic open problems in deep reinforeement learning research sample complexity and experience collection in high-dimenn diteemmmeniio state representation MDPs. While prior work extensively studied the policy's 's interaction with the environment bandits and tabular reinforcement learning, and proposed various algorithms and techniques optimal to the tabular form or the bandit context [Fiechter] 1994 Kearns and Singh 2002 Brařman and Tennenholtz) 2002] Kakade] 2003) Lu and Roy. 2019), experience collection in deep reinforcement learning remains an open challenging problem while practitioners repeatedly employ quite simple yet effective techniques
(i.c. e-greedy) [Whitehead and Ballard 1991 Flennerhag al. 2022 Hasselt al. 2016 Wang
(et al. 2016 Hammick et al.) 2020. Kapturowski et al. 2023 Korkmaz 2024 Schmied et al. 2025
Krishnamurthy et al. 2024
Despite the provable optimality of the techniques designed for the tabular or bandit setting, they generally rely strongly on the assumptions of tabular reinforcement learning, and in particular on the ability to record tables of statistical estimates for every state-action pair which have size growing with the number of states times the number of actions. Hence, these assumptions are far from what is being faced in the deep reinforcement learning setting where states and actions can be parametrized by nigh-dimensional representations. Thus, in high-dimensional complex MDPs, for which deep
Correspondence to Ezgi Korkmaz: ezgi Magkorkmzmmma1mmmmaii com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

neural networks are used as function approximators, the efficiency and the optimality of the methods proposed for tabular settings do not transfer well to deep reinforcement learning [Kakade] 2003]. Hence, in deep reinforcement learning research still, naive and standard techniques (e.g. 6-greedy) are preferred over both the optimal tabular techniques and over the particular recent experience collection
techniques targeting only high scores for particular games [Mnih et al. 2015 Hasselt et al. 2016
Wang et al. 2016 Bellemare et al. 2017 Dabney et al. 2018 Flennerhag et al., 2022
Korkmaz
2024
Kapturowski et al. 2023].
Sample efficiency still remains to be one of the main challenging problems restricting research progress in reinforcement learning. The magnitude of the number of samples required to learn and adapt continuously is one of the main limiting factors preventing current state-of-the reinforcement learning algorithms from being deployed in many diverse settings from large language model reasoning to the physical world, but most importantly one of the main challenges that needs to be dealt with on the way to building neural policies that can generalize and adapt continuously in non-stationary environments. Hence, given these limitations in our paper we aim to seek answers for the following questions:
deep
How can we construct policies that have the ability to collect novel experiences in high- dimensional complex MDPs without any additional computational complexity?
What is the natural theoretical motivation that can be used to design zero-cost experience collection strategy while achieving high sample efficiency?
To be able answer these questions. in our paper we focus on environment interactions in deep reinforcement learning and make the following contributions:
paradigm for reinforce- ment learning based on state-action value function minimization, which we call counteractive temporal difference learning. Our approach centers on solely reconstituting and conceptually shifting the core principles of learning and as result increases the information gained from the environment interac- tions of the policy in a given MDP without adding computational complexity. We first provide the theoretical analysis in Section 3 explaining why and how minimization will result in higher temporal difference. We then as a first step demonstrate the efficacy of counteractive temporal difference learning in motivating example, i.e. the canonical chain MDP setup, in Section 4 The results in the chain MDP verify the theoretical analysis provided in Section counteractive temporal difference learning increases temporal difference obtained from the experiences. Furthermore, we conduct an extensive study in the Arcade Learning Environment 100K benchmark with the state-of-the-an algorithms and demonstrate that temporal difference learning algorithm CoAct TD learning improves performance by 248% across the entire benchmark compared to the baseline algorithm. We demonstrate the efficacy of our proposed CoAct TD Learning algorithm in terms of sample-efficiency, Our method based on maximizing novel experiences via minimizing the state-action value function reaches approximately to the same performance level as model-based deep reinforcement learning algorithms, without building and learning any model of the environment. Finally, we show that CoAct TD learning a fundamental improvement over canonical methods, it is modular and plug-and-playy method, and any algorithm that uses temporal difference learning can be immediately and simply switched to CoAct TD learning.
Contributions. We introduce a fundamental theoretically
2 Background and Preliminaries
The reinforcement learning problem is formalized as a Markov Decision Process (MDP) [Puterman| 1994] M = (S, A,,, 7, Po. T) that contains a continuous set of states E S, set of actions a E A. P.Po. a probability transition function as, 5') on S x A x S, discount factor 7, reward function r(s, a) S x A + IR with initial state distribution Po. A policy T(s. a) S x A + [0, 1] in an MDP assigns a probability distribution over actions for each state s E S. The main goal in reinforcement learning is to learn an optimal policy π that maximizes the discounted expected cumulative rewards
R Eaysat ()..). (.. (=..or)) E,7'r(s at). In Q-learning [Watkins] 1989) the learned policy parameterized by a state-action value function Q S x A + R, which represents the value of taking action a in state 8. The optimal state-action value function learnt via iterative Bellman update
Q(sr, a + Q(s at) + afr(s at) + max Q(5441. a) - Q(st.

---

where max, ((++1100) = ((s++)). Let a" be the action maximizing the state-action value function, (s(s) = arg max, Q(s, a), in state s. Once the Q-function is learnt the policy determined via taking action a (s). Temporal difference learning [Sutton 1988] improves the estimates of the state-action values in each iteration via the Bellman Operator [Bellman 1957]
(Q*Q)(s a) E - [r(s, a) + max 2(s',a'))
For distributional reinforcement learning, QRDQN is an algorithm that is based on quantile regression [Koenker and Hallock 2001 Koenker 2005 temporal difference learning
(1 a) (8, a) + 72(s', ang max 2(0 (o)))) and 2(s,a) 1 N
(0,e)
d'
where 2n E 2Q maps state-action pairs to probability distribution over values. In deep reinforce- ment learning, the state space or the action space is large enough that it is not possible learn and store the state-action values in tabular form. Thus, the Q-function is approximated via deep neural networks. In deep double-Q learning, two Q-networks are used to decouple the Q-network deciding which action to take and the Q-network to evaluate the action taken
Bc+1 0. + a(r(st, a1) + (Q(++ arg max (= a; 0o): 0.) Q(se, ae: 0z)) Vo, ar; 0t).
Current deep reinforcement learning algorithms use e-greedy during training Wang et al. 2016
Mnih et al. 2015| Hasselt et al., 2016 Hamrick et al.
2020
Flennerhag et al. 2022 Kapturowski
et al.] 2023 Krishnamurthy et al. 2024 Schmied et al. 2025 In particular, the e-rredy [Whitehead
and Ballard 1991] algorithm takes an action 0g ~ U (AA) with probability in a given state 8, i.e.
(s, 02)
T. and takes action a' = arg max, (s, a) with probability 1 1- i.c.
π (s, arg max Q(s, a)) = -
While a family of algorithms have been proposed based on counting state visitations (i.e. the number of times action has been taken in state by time step t) with provable optimal regret bounds using the principal of optimism in the face of uncertainty in the tabular MDP setting, yet incorporating these count-based methods in tigh-dimensionll state representation MDPs requires substantial complexity including training additional deep neural networks to estimate counts or other uncertainty metrics. As result, many state-of-the-art deep reinforcement learning algorithms still use simple, randomized experience collection methods based on sampling a uniformly random action with probability 6 MMih et al.
2015 Hasselt et al. 2016 Wang al., 2016, HHmmrick et al., 2020 Flennerhag al., 2022
Korkmaz 2023] Kapturowski et al.) 2023]. In our experiments, while providing comparison against canonical methods, we also compare our method against computationally complicated and expensive techniques such as noisy-networks that is based on the injection of random noise with additional layers in the deep neural network (Hessel al.) 2018 in Section and count based methods Section and Section 6 We further highlight that our method is a fundamental theoretically motivated improvement of temporal difference learning. Thus, any algorithm that is based on temporal difference learning can immediately be switched to CoAct TD learning.
3 Maximizing Temporal Difference with Counteractive Actions
Seeking experiences that contain high information has long been the focus of reinforcement learning [Schmidhuber] 1991 1999 Moore and Atkeson, 1993) and more particularly the experiences that correspond higher temporal difference MMoore and Atkeson 1993]. In this section will provide
the theoretical analysis for our proposed algorithm counteractive TD learning. Section 5 further
provides the experimental results verifying the theoretical predictions. In deep reinforcement learning
the state-action value function is initialized with random weights [Mnih et al. [2015] 2016 Hasselt
al. 2016 Waag al. 2016, [Schaul et al. 2016 Oh al. 2020 Schritwieser et al. 2020 Hubert
et al. 2021 During large portion of the training prior to convergence, the Q-function behaves as a random function rather than providing an accurate representation of the optimal state-action values while interacting with new experiences in siggh-dimnnsinal MDPs as the learning continues.
In particular, high-dimensional environments a significant portion of the training the Q-funtioon, on average, assigns approximately similar values to states that are similar, and has little correlation with the immediate rewards. Hence, let us formalize these facts on the state-action value function in the following definitions.
3

---

Definition 3.1 (n-uninformed). Let 17 > 0. A Q-function parameterized by weights 0 ~ θ is 7-uniinformed if for any state E with amin arg min, (s a) have
Ee-e(r(st, a ] (())((A))) (r(s), <
Definition 3.2 (d-smooth). Let > 0. A Q-function parameterized by weights ~ è is d-smooth if for any state 8 and action ass, 0) with s' T(s, â. we have
Eg Imax Qe (s, a)] (s.,a, max Qo(s', a)ll <
where the expectation is over both the random initialization of the Q-function weights, and the random transition to state s' ~ T(s, a.
Definition 3.3 (Disadvantage Gap). For state-action value function Qa the disadvantage gap a state s E S given by D(s) = Ea=uu(), Qa(s, am where a a arg min, (s)
The following theorem captures the intuition that choosing counteractive actions, i.c. the action minimizing the state-action value function, will achieve an above-average temporal difference.
Theorem 3.4 (Counteructive Actions Increases Temporal Difference). Let v,, > O and suppose that Q0(s,ar a) is 1-uninformed and 8-smooth. Let 8t S be a state, and let mmin be the action minimizing the state-action value in a given state St, ami = arg min, Qo(se, a). Let gmin T (8,, ammin Then have
for action 0₁ ~ (AA) with St+1 T(st,
E stt am,, ,),00,,)00000)))0) )000 miln) + max (0(s)41, a) Qo(st,a"m
B (r(ss, + max Q0(8:41, Qo(s, a:)l +D(ss)
Proof. Since Qo(s, a) is 8-smooth we have
E min am e (T max Qo Qo(sti Q0(8,, a miin)
n Eg mmax Qo a)] - Egrol (0(s),
- ( ) mmax Q0( a)] 25 Egve Qo a )!
max Q0(s:11, Qo(sn a:)] + D(s.) - 26
where the last line follows from Definition 3.3
Further, because Qo a) is 1-uninformed, Beweir(sst, amin)] (a) ax) 17. Combining with the previous inequality completes the proof.
Theorem 3.4 shows that counteractive actions, i.e. actions that minimize the state-action value function, in fact increase temporal difference. Now we will prove that counteractive actions achieve an increase in temporal difference further in the case where action selection and evaluation in the temporal difference are computed with two different sets of weights and as in double Q-learning.
Definition 3.5 (8-smoothness for Double-Q). Let 0. A pair of Q-functions parameterized by weights 0 ~ ④ and 0 ~ @ are 8-smooth if for any state E S and action à ā(s, 0) e A with T(s, a, +), we have
Qa(s, arg arg (u(,
Poals' arg max Qu(s', |n)] <
E 0.0m0 (',T(s,))))
0.0m o'.T((
where the expectation over both the random initialization of the Q-function weights and and the random transition to state s' T(s ...).
Now we will prove that counteractive actions, i.e. actions that minimize the state-action value instead of maximizing, will lead to increase temporal difference the case of two Q f-fucctions, i.e. Qo and Qÿ- that are alternatively used to take action and evaluate the value of the action.

---

Theorem 3.6. Let 17, > 0 and suppose that Qo and Qi are n-uniformed and 8-smooth. Let 8t E S be a state, and let a ami arg min, Qo(s;, a). Let smin s41 Tst a Then for an action
at U(A) with 8141 T(,,, we have
E
+ 2Q0( min ++1 arg max Qo (0,1, a)) - Qo mm
E
(st, at) QQg(5141, arg max Qa (82+1, a)) Q0(s),
+ +((s)) - 28-
Proof. Since Qu and Qg are d-smooth we have
E "1f T (s ).00 B 64 arg max Qo(s a)) - Qo(sc, amiin)
E
St, arg max Qo(s;, a)) - Qo(sr, a
*4 -T Tsy
E 414 1441~((s
arg arg max Qo 8 a)) - Qo(sc. amminn]] - -26
E E1 (.
ang max Qu (8++1, - Qo(sc, ar]] - D((t)) - -2
where the last line follows from Definition 3.3
Further, because Qo and Qa n-uniiformed
g d~ (r(st, a ] Berru(a)((((s, uz)] 7). Combining with the previous inequality completes
the proof.
Core Counterintuition:
How could minimizing the state-action value function accelerate learning?
At first, the results in Theorem 3.4 and 3.6 might appear counterintuitive. Yet, understanding this counterintuitive fact relies on first understanding the intrinsic difference between the randomly initialized state-action value function, i.e. Qe, and the optimal state-action value function, i.e. Q" In particular, from the perspective of the function Q", the action agin (s) = arg min, Qa a) is uniform random action. However, from the perspective of the function Qe. the action a is meaningful, in that it will lead to a higher TD-error update than any other action; hence the realization of the intrinsic difference between a. (s) and azi (s) with regard to Qo and Q provides valuable insight on how counteractive actions do in fact increase temporal difference. In fact, Theorem 3.4 and 3.6 precisely provide the formalization that the temporal difference achieved by taking the minmum action is larger than that of random action by an amount equal to the disadvantage gap D(s). Experimental results reported in Section 5 further verify the theoretical analysis. Now we will formalize this intuition for initialization and prove that the distribution of the minimum value action in given state is uniform by itself, but is constant once it is conditioned on the weights 0.
Proposition 3.7 (Marginal and Conditional Distribution of Counteractive Actions). Let 0 be the random initial weights for the Q-function. For any state s E let a (s) arg min,' Qo d'). Then for any a E A, P [arg min Qo(s, a] = [A] i.e. the distribution Pa~e (in) is uniform. Simultaneously. the conditional distribution P Powell [a. (s) 0) is constant.
The proof is provided in the supplementary material. Proposition 3.7 shows that in states in which the Q-function has not received sufficient updates, taking the minimum action is almost equivalent to taking a random action with respect to its contribution to the rewards obtained. However, while the action chosen early on in training is almost uniformly random when only considering the current state, it is at the same time still completely determined by the current value the weights 0, as is the temporal difference. Thus while the marginal distribution on actions taken is uniform, the temporal difference when taking counteractive actions, i.e. the minimum action, is quite different than from the case where an independently random action is chosen. In particular, in expectation over the random initialization Ô, the temporal difference is higher when taking the minimum value action than that of random action demonstrated in Section 3
The main objective of our approach to increase the information gained from each environment interaction, and show that this can be achieved via actions that minimize the state-action value
5

---

Algorithm 1 CoAct TD Learning: Counteractive Temporal Difference Learning
Input: In MDP M with 7 E (0, I|, S, a E A with Qs(s, a) function parametrized by 0, dithering parameter, B experience replay buffer, the training learning steps.
Populating Experience Replay Buffer:
Learning:
for st in e do
for n. W do
Sample R U(0,1) if Ni < then amin arg min, Q(s a)
Sample from replay buffer:
(st, at, r(5t, St+1) ~ B
Thus, TD receives update with probability e
st1 ~ (stt, a aminn
TD = r(s,, (,5,,""m) a" + y max, Q(8741 a) Qss, am!i))
B 5 (r(s,, a ) ) St, [+1. amiin)
else
TD receives update with probability
maa = arg max, (1tt, a)
TD = r(st, amax)) + max, Q(5++1.0)
81+1 ~ T(s, a minx
Q(sr, maa))
B x- (r(st, a dmixx) St, S1+1, max)
Gradient step with VL(TD)
end if
end for
end for
function. While minimization of the Q-function may initially be regarded as counterintuitive, Section
3 provides the exact theoretical justification how taking actions that minimize the state-action value function results in higher temporal difference for the corresponding state transitions, and Section 5 provides experimental results that verify the theoretical analysis. Our method is fundamental
theoretically well-motivated improvement on temporal difference learning. Thus, any algorithm in reinforcement learning that is built upon temporal difference learning can be simply switched to CoAct TD learning. Algorithm summarizes our proposed algorithm CoAct TD Learning based on minimizing the state-action value function described in detail in Section 3 Note that populating the experience replay buffer and learning are happening simultancously with different rates. TD receives an update with probability solely due the experience collection.
CoAct TD is modular: It is a plug-and-play method with any canonical baseline algorithm.
4 Motivating Example
To truly understand the intuition behind our counterintuitive foundational method, we consider a motivating example: the chain MDP. In particular, the chain MDP which consists of a chain of states s E S = 21,2,... 2, ....... ..) each with four actions. Each state i has one action that transitions the agent up the chain by one step state i + 1, one action that transitions the agent to state 2, one action that transitions the agent to state 3, and one action which resets the agent to state 1 at the beginning of the chain. All transitions have reward zero, except for the last transition returning the agent the beginning from the n-th state. Thus, when started from the first state in the chain, the agent must learn policy that takes the reward. For the chain MDP, we compare standard approaches in temporal difference learning in tabular Q-learning with our method CoAct TD Learning based minimization of the state-action values. In particular we compare our method CoAct TD Learning with both the e-greedy action selection method, and the upper confidence bound (UCB) method. In more detail, in the UCB method the number of training steps t, and the number of times Ne(s, a) that each action a has been taken in state s by step & recorded. Furthermore, the action a E A selection is determined as follows:
2 consecutive steps up the chain, and then one final step reset and get
log (((s,a)
a LCB arg max ((s, a) + +2 aeA
In a given state s if [(s. from the set of actions a' with ((s,, .') = O. For the experiments reported in our paper the length of the chain is set to n = 10. The Q-function initialized by independently sampling each state-action value from normal distribution with μ 0 and using Q-learning for 100 steps, and then evaluate the reward obtained by the argmax policy using the current f-fuctioon for 100 steps. Note that the maximum achievable reward in 100 steps is 11.
for any action a, then an action is sampled uniformly at random
0.1. In each iteration we train the agent

---

Coect TD Learning
E-greedy
CoAcE Learning
Egreddy
0.115
Human Hamallee Bohh Percentee
0.4
0.150
0.125
0.3-
0.100
0.755
0.-
0.050
0.055
0.3
0.000
(10") Figure 1: Human normalized scores median and 80th percentile over all games in the Arcade Learning Environment (ALE) 100K benchmark for CoAct TD Learning and the canonical temporal difference learning with e-greedy for QRDQN. Left: Median. Right: 80th Percentile.
Ervironment Interactions 210"
Figure 2:
Learning curves in the chain MDP with our proposed algorithm CoAct TD Learning, the canonical algorithm <-greedy and the UCB algorithm with variations in 6.
Figure 2 reports the learning curves for each method with varying c [0.15, 0.25] with step size 0.025. The results in Figure 2 demonstrate that our method converges faster to the optimal policy than either of the standard approaches.
5 Experimental Results
The experiments are conducted in the Arcade Learning Environment (ALE). We conduct empirical analysis with multiple baseline algorithms including Deep Double-Q Network [Hasselt et al. 2016 initially proposed in [van Hasselt 2010] trained with prioritized experience replay [Schaul et al.
2016) without the dueling architecture with its original version (Hasselt et al. [0166), and the QRDQN algorithm that is also described in Section 2 The experiments are conducted both in the 100K
Arcade Learning Environment benchmark, and the canonical version with 200 million frame training Mnihh et al. 2015 Wang et al. 2016). Note that the 100K Arcade Learning Environment benchmark an established baseline proposed measure sample efficiency in deep reinforcement learning research, and contains 26 different Arcade Learning Environment games. The policies are evaluated after 100000 environment interactions. All of the polices in the experiments are trained over random seeds. The hyperparameters, the architecture details, and additional experimental results are reported in the supplementary material. All of the results in the paper are reported with the standard error of the mean. The human normalized scores are computed as, HN = (Score agenz Score / (Scoree Score/umdom)
Figure D reports results of human normalized median scores and 80th percentile over all of the games of the Arcade Learning Environment (ALE) in the low-data regime for QRDQN, while Figure reports results for double Q-learning. The results reported in Figure Tonce more demonstrate that the performance obtained by the CoAct TD Learning algorithm is approximately double the performance achieved by the canonical experience collection techniques. For completeness we also report several results with 200 million frame training (i.e. 50 million environment interactions). Furthermore, we also compare our proposed CoAct TD Learning algorithm with NoisyNetworks as referred to in Sec- reports results of human normalized median scores, 20th percentile, and 80th percentile for the Arcade Learning Environment 100K benchmark. Table [ further demonstrates that the CoAct TD Learning algorithm achieves significantly better performance results compared NeisyNetworks.. Primarily, note that NoisyNetworks includes adding layers in the Q-network to increase exploration. However, this increases the number of parameters that have been added in the training process; thus, introducing substantial additional cost. [ algorithm CoAct TD Learning and the original version of the DDQN algorithm with e-greedy training. In the large data regime we observe that while in some MDPs our proposed method CoAct TD Learn-
tion 2 Table I
the learning curves for our proposed
7

---

-000
Figure 3: Temporal difference for our proposed algorithm CoAct TD Learning and the canonical e-greedy algorithm in the Arcade Learning Environment 100K benchmark. Dashed lines report the temporal difference for the e-greedy algorithm and solid lines report the temporal difference for the CoAct TD Learning algorithm. Colors indicate games.
Figure 4: The learning curves for our proposed method CoAct TD Learning algorithm and canonical temporal difference learning in the Arcade Learning Environment with 200 million frame training. Left: JamesBond. Middlel.eft: Gravitar. MiddleRight: Surround. Right: Tennis.
ing that focuses on experience collection with novel temporal difference boosting via counteractive actions converges faster, in other MDPs CoAct TD Learning simply converges to a better policy, Table T demonstrates that our pro- posed CoAct TD Learning algo- rithm improves on the performance of the canonical algorithm e-greedy by 248% and NoisyNetworks by 204%. The results reported in both of the sample regimes demonstrate that CoAct TD learning results in faster convergence rate and signifi- cantly improves sample-effiienncy in deep reinforcement learning. The large scale experimental analysis verifies the theoretical predictions of Section and further discovers that the CoAct TD Learning algorithm achieves substantial sample-efficiency with zero-additional cost across many algorithms and different compleccoplexxtt regimes over canonical baseline alternatives.
Table 1: Human normalized scores median, 20th and 80th
percentile across all of the games in the Arcade Learning Environment 100K benchmark for CoAct TD Learning, 6- greedy and NetsyNetworks with DDQN.
Method
CoAct TD
r-greedy
NoisyNetworks
Modian
0.0927=0.0050
0.0377:00.031
0.0457=0.0035
20th Percentile 0.0145-0.0003
0.0056=0.0017 0,2942+000233
0.0102=0.0018
80th Percentile
0.3762-0.0137
0.1913=0.0144
6 Investigating the Temporal Difference
In Section we provided the theoretical analysis and justification for collecting experiences with counteractive actions, i.e. the minimum Q-value action, in which counteractive actions increase the temporal difference. Increasing temporal difference of the experiences results in novel transitions, and hence accelerates learning (Andre et al. 1997.. The theoretical analysis from Theorem 33.4 and Theorem 3.6 shows that taking the minimum value action results in an increase in the temporal difference. this section, we further investigate the temporal difference and provide empirical measurements of the TD. To measure the change in the temporal difference when taking the minimum action versus the average action, we compare the temporal difference obtained by CoAct TD Learning with that obtained by several other canonical methods. In more detail, during training, for each batch of transitions of the form (St, at, St+1) we record, the temporal difference TD
E (s 1) A (D(ss, Ge; S141) E ((s,a,))))) a.) + max Qo(5:41,a) Qa a.)].
The results reported in Figure 3 and Figure 6 further confirm the theoretical predictions made via Definition 3.2 Theorem 3.6 and Theorem 3.4 In addition the results for individual games reported in Figure 3 we compute normalized measure of the gain in temporal difference achieved when using CoAct TD Learning and plot the median across games. We define the normalized TD gain to be, Normalized TD Gain = + (TD - TD e-eeggreay))). (/(T )/((TDe.rwaa)))) e.greedy)), where TD and TDr.gressl are the temporal difference for any given learning method and e-greedy respectively. The leftmost and middle plot of Figure [ report the median across all games of the normalized TD gain results for CoAct TD Learning and Networstworks in the Arcade Learning Environment 100K benchmark. Note that, consistent with the predictions of Theorem 3.4 the median normalized

---

Lcoming
Coact Loomiing Ci bely
Egroedy
6.30
0.30
Hamun Narmazzdd
0.25
Normlttdd
0.15
0.15
6300
0.10
6.05
0.00
20
Enstronment interactions 1100)
Ervironment Intoractionn 110f)
Figure 5: Human normalized scores median and 80th percentile over all games in the Arcade Learning Environment (ALE) 100K benchmark in DDQN for CoAct TD Learning algorithm and the canonical temporal difference learning with e-greedy. Left:Median. Right: 80th Percentile.
Chain Lanning
200
Cole TO Leantnng
Ewionmere Farnciinn
Figure 6: Left and Middle: Normalized temporal difference TD gain median across all games in the Arcade Learning Environment 100K benchmark for CoAct TD Learning and NoisyNetworks. Right: Temporal difference TD when exploring chain MDP with Upper Confidence Bound (UCB) method, <-greedy and our proposed algorithm CoAct TD Learning.
temporal difference gain for CoAct TD Learning up to 25 percent larger than that of e-greedy. The results for NoisyNetworks demonstrate that alternate experience collection methods lack this positive bias relative to the uniform random action. Further note that to guarantee that every action has non-zero probability of being chosen in every possible state for guaranteeing convergence of Q-learning, one can additionally introduce noise, and achieve higher temporal difference by CoAct TD learning. Albeit, across all of the experiments introduction of the noise was not necessary, as there sufficient noise in the training process that satisfies the property for convergence. Hence, across all the benchmarks CoAct-TD Learning results in consistently and substantially better performance. CoAct TD Learning is extremely modular, only requires two lines of additional code and can be used as drop-in replacement for any baseline algorithm that uses the canonical methods. The fact that, as demonstrated in Table CoAct TD Learning significantly outperforms noisy networks in the low-data regime is further evidence of the advantage the positive bias in temporal difference confers. The rightmost plot of Figure reports TDD for the motivating example of the chain MDP. As in the large-scale experiments, prior to convergence CoAct TD Learning exhibits notably larger temporal difference relative to the canonical baseline methods, thus CoAct TD Learning achieves accelerated learning across domains from tabular MDPs to large-scale.
7 Conclusion
In our study we focus on the following questions: (i) Is it possible to maximize sample efficiency in deep reinforcement learning without additional computational complexity by solely rethinking the core principles of learning?, (ii) What is the foundation and theoretical motivation of the learning paradigm we introduce that results in one of the most computationally efficient ways explore in deep reinforcement learning? and, (iii) How would the theoretically well-motivated approach transfer to high-dimg high-dimmnsional complex MDPs? To be able to answer these questions we propose novel, theoretically motivated method with zero additional computational cost based on counteractive actions that minimize the state-action value function deep reinforcement learning. We demonstrate theoretically that our method CoAct TD Learning based on minimization of the satteacction value results in higher temporal difference, and thus creates novel transitions with more unique experience collection. Following the theoretical motivation we initially demonstrate in a motivating example in the chain MDP setup that our proposed method CoAet TD Learning results achieving higher sample-efficiency. Then, we expand this intuition and conduct large scale experiments in the Arcade Learning Environment, and demonstrate that our proposed method CoAct TD Learning increases the performance on the Arcade Learning Environment 100K benchmark by 248%.

---

References
D. Andre, N. Friedman, and R. Parr. Generalized prioritized sweeping. Conference in Advances in Neural Information Processing Systems, NeurlPS, 1997.
M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML,, volume 70 of Proceedings of Machine Learning Research, pages 448. PMLR, 2017.
R. E. Bellman. Dynamic programming. In Princeton, NJ: Princeton University Press, 1957.
R. I. Brafman and M. Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 2002.
W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In S. A. Mcllraith and K. Q Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana. USA, February 2-7, 2018, pages 2892-29011 AAAI Press, 2018.
A. de Vries. The growing energy footprint of artificial intelligence. Joule, 2023.
C.-N. Fiechter. Efficient reinforcement learning. In Proceedings of the Seventh Annual ACM Conference on Computational Learning Theory COLT, 1994.
S. Flennerhag, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. Bootstrapped meta-learning. 10th International Conference on Learning Representations, ICLR, 2022.
D. Guo, Yang, H. Zhang, Song. R. Zhang. Xu, Q. Zhu, S. Ma, Wang. X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, Dai, E. Luo,
G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang. H. Ding, H. Xin, H. Gao, H. Qu, H. Li,
Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang. L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang. M. Li, N. Tian, P. Huang, P. Zhang,
Q. Wang, Q Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang. R. J. Chen, R. L. Jin, R. Chen, S. Lu,
S. Zhou,, S. Chen, S. Ye, S. Wang. S. Yu, S. Zhou, S. Pan, and S. S. Li. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 2025.
Hamrick, V. Bapst, A. SanchezGonzalez, Pfaff, T. Weber, L.. Buesing, and Battaglia. Com- bining q-learning and search with amortized value estimates. In 8th International Conference on Learning Representations, ICLR, 2020.
H. v. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. Association for the Advancement of Artificial Intelligence (AAAI), 2016.
M. Hessel, Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018
T. Hubert, J. Schrittwieser, I. Antonoglou, M. Barekatain, S. Schmitt, and D. Silver. Learning and planning in complex action spaces. In Proceedings of the 38th International Conference on Machine Learning. ICML, volume 139 of Proceedings of Machine Learning Research, pages 4476 4486. PMLR, 2021.
S. Kakade. On the sample complexity of reinforcement learning. In PhD Thesis: University College London, 2003.
Kapturowski, Campos, R. Jiang, N. Rakicevic, van Hasselt, C. Blundell, and A. P. Ba- dia. Human-level atari 200x faster. In The Eleventh International Conference Learning Representutions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL. net
10

---

M. Kearns and S. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning. 2002.
R. Koenker. Quantile regression. Cambridge University Press, 2005.
R. Koenker and K. F. Hallock. Quantile regression. Journal of Economic Perspectives, 2001.
E. Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI, pages 8369-8377. AAAI Press, 2023.
E. Korkmaz. Understanding and Diagnosing Deep Reinforcement Learning. In International Conference on Machine Learning, ICML 2024, 2024.
A. Krishnamurthy, K. Harris, J. Foster, C. Zhang, and A. Slivkins. Can large language models explore in-context? Advances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
X. Lu and B. V. Roy. Advances in Neural Information Processing Systems (NeurIPS), pages 2458-2466, 2019.
cnermation-theoretie confidence bounds for reinforcement learning. In
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, a. G. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533 2015.
V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, volume 48, pages 1928-1937, 2016.
A. W. Moore and C. G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13:103-130, 1993.
U. Nations. Ai has an environmental problem. here's what the world can do about that. United Nations Environment Programme, 2024.
J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. van Hasselt, S. Singh, and D. Silver. Discovering rein- forcement learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
M. L. Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley and Sons, Inc, 1994.
T. Schaul, J. Quan, L. Antonogloua, and D. Silver. Prioritized experience replay. International Conference on Learning Representations (ICLR), 2016.
J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. 1991.
J. Schmidhuber, Artificial curiosity based on discovering novel algorithmic predictability through coevolution. In Proceedings of the 1999 Congress on Evolutionary Computation, CEC 1999, Washington, DC, USA July 6-9, 1999, pages 1612-1618. IEEE, 1999. doi: 10. 1991199/ 785467. URL https: //dos::///o.../ org 1109/ CEC. 1999. 785467
T. Schmied, J. Borsschein, Grau-Moya, M. Wulfmeier, and R. Pascan. Lims are greedy agents: Effects of rl fine-tuning on decision-making abilities. 2025.
Schrittwieser, Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. P. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588, 2020.
R. Sutton. Learning to predict by the methods of temporal differences. In Machine Learning, 1988.

---

H. van Hasselt. Double q-learning. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010., pages 2613-2621. Curran Associates, Inc., 2010.
Z. Wang. T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICML., page 1995-2003, 2016.
C. Watkins. Learning from delayed rewards. In PhD thesis, Cambridge. King's College, 1989.
Whitchead and D. Ballard. Learning to percieve and act by trial and error. In Machine Learning. 1991.
A. Zewe. Explained: Generative ai's environmental impact. MIT Technological Review, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The introduction and abstract of our paper describes the contributions of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our paper provides theoretical analysis that introduces the basis on how the minimization of the state-action value function will result in higher temporal difference. Furthermore, we provide experimental results in the canonical training benchmarks, and the results verify the theoretical analysis provided in Section 3 Furthermore, we specifically report the temporal difference obtained by the agents in Section i Figure and Figure These results further verify the theoretical analysis provided in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The theoretical results are stated with full set of assumptions and a complete
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems. The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All the necessary details are explained in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The main paper references the algorithms used and deseribes algorithm with the theoretical analysis. All the necessary details regarding implementation are ex- plained in the supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mtiiss.cc/public//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the necessary details regarding implementation are provided in the supple- mentary material. The standard settings were implemented.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the results reported in the paper contain appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We our method even against some of the compute exhaustive methods which was also explained between Line 261 and 265, Furthermore, we state in Section 6 that CoAct TD Learning only requires two lines of additional code, extremely modular, and can be used as drop-in replacement for any baseline algorithm that uses the canonical methods between Line 310 and 311.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
http://eurriiscccc httperrrip..ppppaleececccaaaaaaa
Answer: [Yes]
Justification:
Our research does not violate the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
16

---

Justification: Our work is foundational research. Yet, we believe it might still have potential societal impacts. Particularly, our paper introduces a foundational drop-in replacement for canonical algorithms that results in more efficient learning without additional cost. With the rise of progress in artificial intelligence and the objectives to scale base algorithms to benefit humanity, the energy consumption required by the hardware regarding Al training is estimated to be at the level of an entire country's 's electricity consumption, i.c. 85 to 134 terawatt hours (Twh) annually, [de Vries 2023 Nations 2024 Zewe 2025 Regarding the carbon footprint of artificial intelligence and the climate crisis, our paper poses a line of contribution trying to minimize the environmental damage and impacts of the algorithms we built by achieving significant improvement on the samples that are required to train them.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
17

---

Answer: [Yes]
Justification: We reference the algorithms, training environments and the metrics that evaluate algorithms throughout our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsppppeaaaaeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our paper does not include any LLM usage in any form.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
19


Reviews:
[{"review_id": "LzFGFqxNMV", "reviewer": "Reviewer_XeS3", "summary": "This paper proposes CoAct TD, an exploration strategy that selects the action that minimizes its current Q-value function to improve the sample efficiency of DRL. The authors provide a theoretical proof that the \"counteractive\" action generates a higher expected TD error, thereby accelerating the learning process. This theory is validated by extensive experiments on the ALE, where the method demonstrates significant performance gains, such as a 248% improvement over a standard baseline on the ALE 100K benchmark. Also, CoAct TD adds no additional computational cost and is designed as a \"plug-and-play\" module for any algorithm that relies on TD learning.", "questions": "- The theory for CoAct TD relies on the Q-function being uninformed, a condition primarily met during early training. What is the method's behavior in late-stage training when this assumption no longer holds? A brief analysis demonstrating that the agent does not get stuck in known undesirable states would increase my confidence in the method's robustness.\n- The paper's claim of being a \"fundamental improvement\" is broad, yet experiments are confined to DQN-family algorithms. Could the authors briefly discuss how the argmin Q principle could be extended to continuous control algorithms (e.g., SAC)? \n- The paper links higher TD error to better exploration. Can the authors provide any direct evidence of improved exploration quality, such as higher state visitation entropy compared to baselines?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- __Originality and Significance__. The core idea of using the action that minimizes the Q-function for exploration is novel. It addresses the critical problem of sample efficiency with a simple, practical solution. The reported 248% performance improvement on the ALE 100K benchmark highlights the method's potential impact.\n\n- __Quality__. The paper has a strong logical structure that flows from theory to practice. It provides a clear theoretical justification (Section 3) for its counterintuitive claim and then rigorously validates this theory with both a simple, motivating example (Section 4) and large-scale, standard benchmarks (Section 5). \n\n- __Clarity__. The paper is well-written and easy to follow.\n\n# Weaknesses\n- The theoretical proof relies on the Q-function being _\"η-uninformed\" and \"δ-smooth\" assumptions_ that hold true during the initial phases of training. However, these assumptions _no longer hold_ during the later stages of training. This raises an unanswered examination regarding the robustness of the method and its long-term convergence properties.\n\n- The experiments are limited to _discrete action_ environments employing DQN-family algorithms. The assertion of being a \"fundamental improvement to temporal difference learning\" is heavily broad; The paper fails to present evidence of its relevance to other crucial domains of reinforcement learning, such as actor-critic methods for continuous control (e.g., SAC, TD3), which also depend on TD learning.", "was_revised": true, "final_justification": "The authors' responses have addressed many of my concerns, so I raise my score to 4", "rebuttal": "Thank you very much for finding the core idea of our paper novel and well-written while addressing a critical problem with a strong logical structure that flows from theory to practice with rigorous validation of the theory in both motivating examples and large-scale standard benchmarks. We highly appreciate your valuable and insightful review. \n\n1. *\"The theory for CoAct TD relies on the Q-function being uninformed, a condition primarily met during early training. What is the method's behavior in late-stage training when this assumption no longer holds? A brief analysis demonstrating that the agent does not get stuck in known undesirable states would increase my confidence in the method's robustness.\"*\n\nThank you for this thoughtful question. Figure 3 reports temporal difference throughout the entire training, i.e. both early-stage training and late stage training. The results reported in Figure 3 demonstrates that CoAct TD learning achieves higher temporal difference **throughout training** compared to canonical methods as has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6. As we investigate the results reported in Figure 3 together with the results in Figure 1 and Figure 5 we see that the agents throughout the entire benchmark robustly overperform compared to the canonical methods. \n\n2. *\"The paper's claim of being a \"fundamental improvement\" is broad, yet experiments are confined to DQN-family algorithms. Could the authors briefly discuss how the argmin Q principle could be extended to continuous control algorithms (e.g., SAC, TD3)?\"*\n\nThank you for this insightful question. The policy update in SAC is performed via taking a gradient step with respect to the loss $J(\\phi) = D_{KL}(\\pi_{\\phi}(\\cdot | s_t) || \\exp(Q_{\\theta}(s_t,\\cdot)/Z_{\\theta}(s_t))$. Here we can take the gradient of the cost function $\\hat{J}(\\phi)$ with $-Q_{\\theta}$ to obtain $\\pi_{\\phi’}$ with probability $\\epsilon$ to obtain the analogue of the argmin $Q$ principle.\n\n3. *\"The paper links higher TD error to better exploration. Can the authors provide any direct evidence of improved exploration quality, such as higher state visitation entropy compared to baselines?\"*\n\nMany thanks for this intriguing question. The rather well-known link between higher temporal difference and better exploration was investigated in quite early work starting from [1], and the main connecting evidence between higher TD error, exploration quality and faster learning in our paper is the illustration of  the significantly improved performance of our method in the ALE benchmark. \n\n[1] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 1993.\n\nWe highly appreciate again that our efforts to produce a rigorous analysis were recognized by you."}, {"review_id": "34hMctYrUg", "reviewer": "Reviewer_mnA4", "summary": "The paper proposes a novel plug-and-play module for online reinforcement learning algorithms, called CoAct. CoAct intentionally selects the action with the lowest action value to collect novel experiences and gain information from the environment. Theoretical analysis shows that sampling the action with the minimum value increases the temporal difference (TD) error compared to uniform action sampling, leading to larger updates. \nEmpirical results demonstrate that the method outperforms $\\epsilon$-greedy, UCB,  and Noisy Networks.", "questions": "* It would be great if the authors could explain the intuition of learning for actions with the lowest value estimation. What the agent needs to learn in the end is the action with the highest value, for deriving a good policy. Explainning the intuition more might be helpful for readers to understand the methodology.\n\n* Could the authors please clarify more about how the algorithm is evaluated? In empirical evaluation, it is not clear whether the reported performance reflects the agent’s performance during online interactions or is based on a a fixed policy rollout, i.e., whether the agent still follows $a_{Q_\\theta}^{\\min}(s)$ as the left side of Algorithm 1 indicates. Given that online actions affect the environment, I think it is important to specify this point. If following $a_{Q_\\theta}^{\\min}(s)$ is part of the learning process, then reporting the return during interactions would be more informative than only evaluating the final policy, as it captures the consequence of taking actions that are considered to be poor choices.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n* The paper provides both theoretical analysis and empirical evaluation to support the intuition and effectiveness of the method.\n\n* The proposed method is novel and easy to use. As a plug-and-play module, CoAct can be integrated into any existing online reinforcement learning algorithm.\n\n* The empirical analysis reports both the 80th percentile and the median, offering a detailed overview of the agent’s performance. The parameter settings are reported for reproducibility. \n\n**Weaknesses**\n* The explanation remains unclear regarding how a larger TD error contributes to more effective learning. While the theoretical analysis demonstrates that selecting the action with the lowest action value leads to a high TD error, the paper lacks further analysis on how a larger TD error affects the value estimation of the true optimal action, which is ultimately what the agent should learn. After asking \"how could minimizing the state-action value function accelerate learning\", the paper states that \"the realization of the intrinsic difference between $a_{Q_{\\theta}}^{\\min}(s)$ and $a_{Q^\\*}^{\\min} (s) $ with regard to $Q_\\theta$ and $Q^\\*$ provides a valuable insight on how counteractive actions do in fact increase temporal difference.\" It is good that the paper attempts to answer this question, but it would be helpful to provide a more detailed explanation of how this insight further contributes to searching for the true $a_{Q^\\*}^{\\max}$.\n\n* The application of the method may be limited by the properties or constraints of the environment. Taking $a_{Q_\\theta}^{\\min}(s)$ may not always be applicable in practice. In online settings, every chosen action affects the environment. Since the action value estimation reflects how good an action is, taking the action with a low estimation introduces concerns about whether it causes irreversible damage to the environment, especially in real-world applications like robotics and autonomous driving. In such domains, collecting data following $a_{Q_\\theta}^{\\min}(s)$ could be unacceptable due to safety or cost concerns.", "was_revised": true, "final_justification": "After reading the authors' response and other reviewers' comments, I would like to raise my score, as the response addresses my concerns regarding the intuition, performance reporting, and the role of $\\epsilon$. However, it would strengthen the paper to include further discussion on the method's sensitivity to $\\epsilon$, which would help readers better understand its practical applicability.", "rebuttal": "We are truly glad to hear that you found our work novel and easy to use, and that our analysis offers a detailed overview of the agent's performance. We are also sincerely glad you highlighted that our paper provides both theoretical analysis and empirical evaluation, supporting the method's intuition and effectiveness. We highly appreciate your positive and valuable feedback.\n\n\n1. *\"It would be great if the authors could explain the intuition of learning for actions with the lowest value estimation. What the agent needs to learn in the end is the action with the highest value, for deriving a good policy. Explainning the intuition more might be helpful for readers to understand the methodology.\"*\n\nThis insightful question is highly related to understanding how higher temporal difference results in effective learning, and the core counterintuition section of our paper that starts in Line 173. In particular, our theoretical analysis shows that taking the lowest value action in fact yields a higher temporal difference compared to canonical, and temporal difference update is essentially the update that leads to learning an optimal policy, i.e. the action with the highest value. Hence, our method shows that by taking action $a_{\\textrm{min}}$ in fact we can increase temporal difference, and hence accelerate learning the highest value action.\n\n2. *”The explanation remains unclear regarding how a larger TD error contributes to more effective learning.”*\n\nThank you for this insightful question. It is well known in the reinforcement learning literature that larger temporal difference will result in effective learning [1]. Although we also cite this work [1] in page 3 we can also further add more explanation regarding larger TD and effective learning. \n\n[1] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 1993.\n\n3. *\"Could the authors please clarify more about how the algorithm is evaluated? In empirical evaluation, it is not clear whether the reported performance reflects the agent’s performance during online interactions or is based on a a fixed policy rollout, i.e., whether the agent still follows $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ as the left side of Algorithm 1 indicates. Given that online actions affect the environment, I think it is important to specify this point. If following $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ is part of the learning process, then reporting the return during interactions would be more informative than only evaluating the final policy, as it captures the consequence of taking actions that are considered to be poor choices.\"*\n\nThe reported performance reflects the agent’s performance when  $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ is taken.\n\n4. *\"The application of the method may be limited by the properties or constraints of the environment. Taking  $a^{\\mathrm{min}}\\_{Q\\_\\theta} (s)$ may not always be applicable in practice. In online settings, every chosen action affects the environment. Since the action value estimation reflects how good an action is, taking the action with a low estimation introduces concerns about whether it causes irreversible damage to the environment, especially in real-world applications like robotics and autonomous driving. In such domains, collecting data following  $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ could be unacceptable due to safety or cost concerns.\"*\n\nWe would like to emphasize that this question is again highly related to the core counterintuition part of our paper starting in line 173. The core counterintuition introduced in our paper is the realization of the critical difference between $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ and $a^{\\textrm{min}}\\_{Q^\\*} (s)$. In particular, from the perspective of $Q^\\*$ in fact the  $a^{\\textrm{min}}\\_{Q\\_\\theta} (s)$ is not the true minimum of the optimal state-action value function $a^{\\textrm{min}}\\_{Q\\_\\theta} (s) \\neq \\textrm{argmin}\\_{a \\in A} Q^\\*(s,a)$. Thus, the damage that can be done to the environment is relevant to the true rewards obtained by the agent and hence $Q^*$ but not $Q_\\theta$. This is indeed very subtle and requires a bit of counterintuitive thinking. We can also provide more textual explanation on this between Line 173 and 187."}, {"review_id": "LwPAR2hN5z", "reviewer": "Reviewer_LB6J", "summary": "The paper introduces a straightforward yet impactful modification to experience collection in deep reinforcement learning: periodically selecting the action with the lowest Q-value. This approach, called CoAct TD Learning, accelerates learning and improves sample efficiency without adding computational overhead. The method is supported by theoretical analysis, is simple to implement, and is empirically validated on various benchmark tasks.", "questions": "Assumptions in Section 3:\n- Line 128-130: The statement that \"During the large portion of the training ... while interacting with new experiences in high-dimensional MDPs as the learning continues\" is accurate for the early training stage. However, it is unclear whether this remains true as training progresses. Is there empirical or theoretical evidence to support this claim beyond the initial phase?\n- Line 131-133: The statement that \"In particular, in high-dimensional environments in a significant portion ... has little correlation with the immediate rewards.\" also needs supporting evidence or citation.\n- Theorem 3.4 depends on assumptions that \" $Q_\\theta(s, a)$ is $\\eta$ -uninformed and $\\delta$ -smooth.\" Can you provide proof or empirical evidence that these assumptions hold, especially beyond the very start of training? It appears these conditions may only be valid at initialization and not throughout training.\n\nExperimental results\n- Figure Order and Baselines: Figure 4 (canonical temporal difference learning baseline) should be presented before Figure 1 (QRDQN results), since the canonical method is a more relevant and familiar baseline for most readers. Additionally, QRDQN, while showing a larger performance gap, is less central to the paper’s main claims.\n- Figure 1 Interpretation: The performance gap between canonical $\\epsilon$ -greedy and CoAct TD emerges midway through training (after about $10^6$ environment steps), which seems at odds with the theoretical foundation that relies on random initialization and uniform action selection. This discrepancy should be addressed.\n- Figure 2 Clarity: The four subplots in Figure 2 lack informativeness; the specific $\\epsilon$ values used are not visible or clear. Furthermore, Figure 2 should appear earlier in the paper, as it is discussed before Figures 1 and 4.\n- Additional Baseline: It would strengthen the empirical evaluation to include a comparison with a purely greedy baseline, where $\\epsilon$ -greedy is not used (i.e., the agent always selects the action with the maximum Q-value).", "limitations": "See weaknesses and questions for more detail.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well written and easy to follow.\n- The idea of leveraging actions with minimum state-action values for experience collection is novel and intriguing. If the theoretical and empirical claims are substantiated, this approach could have significant impact on the field.\n\nWeaknesses\n- Some of the core assumptions underlying the proposed method, particularly those introduced in Section 3, require further explanation and justification.\n- The presentation of the experimental results should be reorganized to improve clarity.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful and in-depth review. We are truly glad you found our work impactful and straightforward, and that you noted our paper is well-written and easy to follow. We are especially glad to hear that you consider our core idea both novel and intriguing, with the potential for significant impact on the field.\n\n1. *“(I) Assumptions: Line 128-130: …It is unclear whether this remains true as training progresses. Is there empirical or theoretical evidence to support this claim beyond the initial phase? (II)Line 131-133: The statement that \"In particular, in high-dimensional environments in a significant portion ... has little correlation with the immediate rewards.\" also needs supporting evidence or citation. (III) Theorem 3.4 depends on assumptions that \" $Q_\\theta(s,a)$ is $\\eta$-uninformed and $\\delta$-smooth.\" Can you provide proof or empirical evidence that these assumptions hold, especially beyond the very start of training? It appears these conditions may only be valid at initialization and not throughout training.”*\n\nAll these questions consolidate the Definitions introduced in Section 3 and its relation to Theorem 3.4 and Theorem 3.6. Yes, there is indeed evidence to support this. In particular, Figure 3 in Section 5 reports the temporal difference throughout the entire training, i.e. both early-stage training and late-stage training. The results reported in Figure 3 demonstrate that CoAct TD learning achieves higher temporal difference throughout training compared to canonical methods as has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6. \n\n2. *”Experimental results”*\n\nWe highly appreciate your insightful suggestions regarding in which places the content and the Figures should appear and the organizational presentation of the experimental results to provide a better reading experience. These suggestions are now indeed reflected for the camera ready version of our paper. \n\n3. *”Additional Baseline: It would strengthen the empirical evaluation to include a comparison with a purely greedy baseline, where $\\epsilon$ -greedy is not used (i.e., the agent always selects the action with the maximum Q-value).”*\n\nAt your suggestion we ran the experiment with a purely greedy baseline. This algorithm completely fails to learn, never achieving a reward above 0.2. In contrast as shown in Figure 2, $\\epsilon$-greedy achieves reward 8 after 100 iterations, and CoAct TD achieves reward 11. We can also add this purely greedy baseline to the results as well."}, {"review_id": "3p4iAWnOhD", "reviewer": "Reviewer_EGYp", "summary": "This paper proposes a novel method called CoAct TD Learning. While conventional reinforcement learning selects actions that maximize the state-action value function, this research demonstrates both theoretically and experimentally that selecting actions that minimize the state-action value function (counteractive actions) can achieve higher TD. The authors show that this approach leads to improved learning efficiency.", "questions": "Q1. To clarify how well the theory holds experimentally, could you plot how long **D(s)−2δ−η>0** continues from the early stages of learning?\n\nQ2. Looking at Figure 2, the learning curve for ε-greedy appears not to have converged. When iterations are increased further, how do the final convergence scores compare between ε-greedy and CoAct TD Learning? While I understand that the proposed method accelerates convergence in early learning, I believe the convergence values are also important in practice.\n\nQ3. As an application area for reinforcement learning, LLM post-training is gaining attention. Post-training starts from a state where the LLM has completed pre-training, but would this method still be effective for improving learning efficiency in such cases? If effective, I believe this would further enhance the value of this research.", "limitations": "**Theory-Practice Gap**: The theoretical results are based on strong assumptions, and it is unclear to what extent these assumptions hold in actual complex environments.\n\n**Bias Toward Short-term Learning**: Experimental evaluation is mainly concentrated on short-term periods (100K interactions), and stability and effectiveness in long-term learning are not sufficiently verified.\n\n**Environment Dependency**: Verification is centered on the specific domain of Atari games, and generalizability to continuous control tasks and other complex domains is unclear.\n\n**Lack of Convergence Guarantees**: No theoretical convergence guarantees for the proposed method are provided, leaving questions about reliability in practical applications.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nS1. **Novelty and Problem Importance**: The paper presents a theoretical method addressing the \"poor initial learning efficiency\" that deep RL researchers commonly experience. The authors construct theory that holds even during the early stages of learning when the Q-function is unreliable (they claim that until convergence, Q behaves almost like a random function with little correlation to immediate rewards), maximizing TD error. Specifically, Theorem 3.4 proves that minimization actions achieve TD that is D(s)-2δ-η higher than average actions.\n\nS2. **High Practicality**: The fact that this method requires zero additional computational cost and can be easily integrated into existing TD learning-based algorithms is extremely valuable in practice.\n\nS3. **Implementation from Theory to High-dimensional Experiments**: The effectiveness of the method is validated across multiple scales, from basic verification in Chain MDP to large-scale experiments in the Arcade Learning Environment. Direct measurement of temporal difference is also conducted.\n\n**Weaknesses**\n\nW1. The assumptions of η-uninformed and δ-smooth are very strong, and verification of whether these hold in actual high-dimensional environments is insufficient. In particular, these assumptions may break down as learning progresses, but the impact of this is not adequately discussed.\n\nW2. No theoretical guarantees for convergence of the proposed method are provided. The selection of minimization actions may bias exploration and potentially hinder convergence to the optimal policy, which requires investigation.\n\n**Minor Comment** The text in the figures is too small and should be enlarged.", "was_revised": true, "final_justification": "As I mentioned in the strengths section, this paper addresses the important problem of poor initial learning efficiency in deep reinforcement learning both theoretically and experimentally, and I strongly recommend the acceptance of this paper.\nThe authors' response has also dispelled concerns about exploration bias.", "rebuttal": "We deeply appreciate your thorough and insightful review, particularly your recognition of the paper's novelty, practicality, and rigorous validation. Thank you very much for finding our work extremely valuable and novel while the effectiveness of our method is validated across multiple scales from theory to high-dimensional large-scale experiments. Your feedback provides clear and sincere insights on how our work offers a significant, practical, and theoretically sound advancement for the deep RL community.\n\n1. *\"To clarify how well the theory holds experimentally, could you plot how long $D(s) - 2 \\delta -\\eta >0 $ continues from the early stages of learning?\"*\n\nRegarding how well the theory holds experimentally, Figure 3 in Section 5 reports temporal difference throughout the entire training, i.e. both early-stage training and late stage training. The results reported in Figure 3 demonstrate that CoAct TD learning achieves higher temporal difference throughout training compared to canonical methods, which has been predicted by the theoretical results in Theorem 3.4 and Theorem 3.6. \n\n2. *\"Looking at Figure 2, the learning curve for $\\epsilon$-greedy appears not to have converged. When iterations are increased further, how do the final convergence scores compare between $\\epsilon$-greedy and CoAct TD Learning? While I understand that the proposed method accelerates convergence in early learning, I believe the convergence values are also important in practice.\"*\n\nThe cumulative return reached by CoAct TD at iteration 84 is the optimal value for the Chain MDP. At your suggestion we ran the experiment for longer and found that $\\epsilon$-greedy converged to this same optimal value at iteration 176.\n\n3. *\"As an application area for reinforcement learning, LLM post-training is gaining attention. Post-training starts from a state where the LLM has completed pre-training, but would this method still be effective for improving learning efficiency in such cases? If effective, I believe this would further enhance the value of this research.\"*\n\nThank you very much for this intriguing and insightful question. Yes, indeed. Because the pre-training of an LLM does not involve the similar structure of reinforcement learning as the post-training. Currently, reinforcement learning is used either to enhance the mathematical and scientific reasoning capabilities of LLMs or in alignment as in RLHF. In both of these cases, post-training the large language model is the start of the reinforcement learning training in a newly defined task, and hence yes indeed we can expect CoAct TD to be effective in improving learning efficiency in both of these cases."}]

Author rebuttal:

Meta-review: This paper introduces an approach to increasing the efficiency of exploration, in which the action with lowest current estimated Q-value is chosen whenever the algorithm explores.  The key insight (lower Q-value actions will tend to have high TD errors) is proven formally, and both the theoretical claims and more rapid learning than epsilon-greedy and UCB benchmarks are demonstrated empirically.

The reviewers (and I) agree that this approach is simple, surprising, and original.  The empirical evaluations are quite convincing.  I believe this work is likely to have significant impact.
Decision: Accept (spotlight)