Title: Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Deep Reinforcement Learning

Causal Reinforcement Learning
Abstract: Deep reinforcement learning (DRL) agents excel in solving complex decision-making tasks across various domains.
However, they often require a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands.
To address these challenges, we introduce a novel theoretical result that leverages the Neyman-Rubin potential outcomes framework into DRL.
Unlike most methods that focus on bounding the counterfactual loss, we establish a causal bound on the factual loss, which is analogous to the on-policy loss in DRL.
This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded.
Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve *up to 383%* higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by *up to 96%*, significantly improving *sample efficiency at a negligible cost*.

Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound

Abstract

Deep reinforcement learning (DRL) agents excel in solving complex decisionmaking tasks across various domains. However, they often require a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands. To address these challenges, we introduce a novel theoretical result that leverages the Neyman-Rubin potential outcomes framework into DRL. Unlike most methods that focus on bounding the counterfactual loss, we establish a causal bound on the factual loss, which is analogous to the on-policy loss in DRL. This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded. Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve up to 383% higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by up to 96% , significantly improving sample efficiency at a negligible cost 1 .

1 Introduction

Deep reinforcement learning agents have demonstrated remarkable success in solving complex decision-making tasks across various areas of interest, such as gaming [1-5], robotics [6-8], healthcare [9-11], and autonomous driving [12-14]. It also has a vital role in enhancing large language models [15, 16]. Despite these achievements, DRL is still challenging due to its high computational and resource demands. For example, AlphaGo Zero was trained for 29 million self-play games [17], and the training experience for OpenAI's Rubik's Cube robotic hand is roughly equated to 13 thousand years [18].

In DRL, we can categorize agents into two types: on-policy agents, like A3C and PPO [19, 20], and off-policy agents, such as DQN and SAC [21, 22]. On-policy agents are generally simpler and often considered first. They interact with the environment, collect experiences using a behavior policy, and once they optimize the target policy, discard all previously collected experiences to ensure that the behavior and target policies remain identical. This consistency leads to more stable convergence but at the cost of sample inefficiency, as each optimization iteration requires collecting new data. Off-policy agents, on the other hand, allow the agent to learn from previously collected experiences, meaning it can have different behavior and target policies. This methodology improves sample efficiency by leveraging past experiences stored in a replay buffer. While this improves sample efficiency, it can also lead to reduced learning stability due to the mismatch between the behavior and target policies. As a result, improving the learning stability typically requires a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands.

Enduro

DemonAttack

Atlantis

ChopperCommand

ChopperCommand

NameThisGame

Kung-Fu Master

BeamRider

UpNDown

Tutankham

Seaquest

H.E.R.O.

Phoenix

Assault

Asterix

Q*Bert

NameThisGame

YarsRevenge

UpNDown

Kangaroo

Atlantis

Assault

Zaxxon

Asterix

Gopher

Q*Bert

Krull

BattleZone

BankHeist

Space Invaders

Kangaroo

RiverRaid

Enduro

IceHockey

SpaceInvaders

Frostbite

YarsRevenge

Tutankham

River Raid

BattleZone

Centipede

Robotank

Amidar

Pong

Alien

Krull

+10%

+100%

Kung-Fu Master

StarGunner

BeamRider

Bank Heist

Breakout

Freeway

Gravitar

Bowling

H.E.R.O.

Pong

+10%

+100%

Pit Fall

Alien

Ice Hockey

Freeway

Amidar

Pit Fall -

Breakout

Frostbite

Robotank

Tennis

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

2.0

2.5

Log Reward Improvement(%)

Log Reward Improvement (%)

SUFT

Baseline

To mitigate these challenges, researchers have proposed numerous strategies designed to reduce computational demands and improve learning stability. A widely adopted approach focuses on enhancing the experience replay buffer sampling, aiming to improve convergence rates by optimizing the selection of experiences [23-26]. While this approach improves convergence, it still involves high computational resources and does not fully address the divergence between on-policy and off-policy methods. Other works have attempted to bridge the divergence by suggesting additional computable terms, e.g., [27]. Although such approaches have improved learning stability, they often require extensive data collection and involve complex techniques.

This work addresses these limitations by introducing a novel theoretical result in causal inference and incorporating it into the DRL framework. Specifically, we establish a causal upper bound on the unobserved factual loss using the observed counterfactual loss and the estimated treatment effect. Within the DRL framework, we incorporate this as an upper bound for the on-policy loss using the standard off-policy loss and an off-policy evaluation (OPE) term, which quantifies the estimated value discrepancy between the behavior policy and the target policy. By incorporating this term into the optimization process, our method explicitly accounts for the causal estimated treatment effect difference between the target policy and the behavior policy. This causal perspective enhances the agent's ability to reason about the differences between the target policy and the behavior policy used to generate the experiences, thereby mitigating the fundamental divergence between on-policy and off-policy learning. Importantly, our method reuses existing data of the behavior policy network values that are usually discarded to improve sample efficiency at a negligible cost .

We evaluate our proposed method, SUFT 2 , on existing off-policy agents such as DQN and SAC, as well as on-policy agents like PPO, which benefit from our method due to the deviation between behavior and target policies during optimization. Throughout comprehensive experiments on the Atari 2600 and MuJoCo domains, agents using the SUFT OPE term outperform the same ones without it. Notably, we achieve up to 383% higher reward ratio, as shown in Figure 2, indicating an improvement in the training convergence rate compared to an identical agent without the additional term. In addition, we reduce the experience replay buffer size by up to 96% while outperforming the baseline agent that uses a 25 times larger buffer by up to 437.05% . These significant improvements highlight the potential of our method to make DRL more computationally efficient and accessible.

450

Atari2600

MujoCo

400

383.19%

(%)

350

Ratio

300

Reward

250

228.12%

226.91%

224.53%

197.34%

200

Mean

150

111.21%

100

50

0

DQN

VanillaDQN

Double DQN

PPO

SAC

PPO

SUFT

Baseline

2 Related Work

2.1 Causal Reinforcement Learning

Causal reinforcement learning (CRL) is an emerging subfield of reinforcement learning (RL). CRL seeks to enhance agents' performance by incorporating causal inference methodologies into the learning process. The causal perspective allows the agent to learn cause-and-effect knowledge, resulting in more informed and effective decision-making [28, 29].

In causal inference, a treatment typically refers to performing a single action, commonly termed as performing an intervention [30, 31]. On the other hand, the majority of works in CRL define treatment as a sequence of actions determined by a dynamic treatment plan [32-35]. In this work, we adopt the latter approach and consider the treatment to be the agent's policy. This aligns with the RL framework, which focuses on evaluating the long-term effects of policies rather than the immediate impact of individual actions. Recent research in CRL covers a variety of areas, such as advancing generalizability [36], promoting safe agents [37, 38], enhancing sample efficiency [39-41], and addressing spurious correlations [42-44]. Our work falls in the field of CRL and primarily contributes to the latter two directions.

CRL for Addressing Spurious Correlations: Spurious correlations in off-policy RL often arise as a result of learning based on experiences from an old behavior policy. These experiences might not accurately reflect the states and actions the target policy is likely to encounter, resulting in deceptive associations between actions and outcomes that limit the agent's ability to make reliable decisions. Several works leverage OPE as a systematic way to evaluate the quality of a policy using off-policy data, which helps mitigate the impact of spurious correlations during policy optimization [45, 46].

CRL for Addressing Sample Inefficiency: Causality provides a valuable perspective for creating sample-efficient agents, with research focusing on three key directions: representation learning [47-49], directed exploration [50, 51], and data augmentation [52, 53]. While many works improve sample efficiency by generating new data, our method utilizes existing discarded data from past experiences, achieving sample efficiency at a negligible cost . This novel strategy sets our work apart, demonstrating that DRL agents can achieve sample efficiency with marginal computational overhead.

2.2 Potential Outcomes Causal Bounds

The Neyman-Rubin potential outcomes framework [54] is a cornerstone of causal inference, providing a robust theoretical basis for estimating treatment effects. Several works have utilized this framework to develop causal bounds [55, 56]. For instance, Shalit et al. [57] establish a generalization-error bound to estimate individual treatment effects from observational data, addressing unobserved

counterfactuals by formalizing a lemma that bounds this loss. Our work expands on this foundational framework and introduces several key changes.

First, while most works apply these bounds within general machine learning (ML) settings [58, 59], we redefine the terms to align with the DRL framework. Second, while other studies consider treatment as a single intervention [60, 61], we address treatments as a sequence of actions within a treatment plan. The primary and novel aspect of our work lies in reversing the traditional lossbounding methodologies. Whereas existing methods typically bound the counterfactual loss [62, 63], our approach does the exact opposite and bounds the factual loss. This pivotal shift is feasible within the RL framework since off-policy methods use counterfactual data while factual data remains unobserved. Finally, whereas other approaches mainly focus on the scenario of a single target treatment and a single control treatment [64, 65], we scale this scenario to a single target treatment and N control treatments. This multiple control treatments scenario allows us to leverage the replay buffer, which contains experiences generated from N behavior policies, enabling effective evaluation of the target policy against a broader spectrum of previous behavior policies.

3 Preliminaries

In this section, we present some fundamental conceptual ideas necessary to understand our method.

Causal and Counterfactual Reasoning: Causal reasoning refers to the process of identifying and understanding the relationships between causes and their effects. Counterfactual reasoning [66], a subfield of causal reasoning, goes a step further by imagining alternative scenarios to answer "what-if" questions, such as "What would have happened if a different treatment had been applied?" as Pearl & Mackenzie [67] elaborate in "The Book of Why". This form of reasoning enables the evaluation of potential outcomes conditioned under different treatments. In RL, incorporating counterfactual reasoning enhances agents' capabilities to evaluate and optimize policies in complex, dynamic environments. By retrospectively analyzing alternative policies and their outcomes, agents can improve decision-making and policy optimization. In this paper, counterfactual reasoning is implemented through the integration of a causal bound into the agent's loss function.

Off-Policy Evaluation: Off-policy evaluation is a fundamental concept in RL [68], which aims to numerically evaluate the performance of a desired policy π using historical data D that was collected under different policies. As outlined in [69], the desired policy being optimized is referred to as the target policy , while the policies used to generate experiences are termed the behavior policies . OPE is particularly important in offline RL, where the agent learns the target policy solely from a static dataset D without interacting with the environment. In contrast, this work focuses on applying OPE to online DRL agents, which continuously interact with the environment while updating the experience replay buffer D .

4 SUFT Causal Bound

In this section, we formalize our theoretical causal result and detail our method.

4.1 Rationale

To illustrate our method and provide a general intuition, we begin with a straightforward example of a greedy off-policy DQN agent. This agent observes the current state of the environment, uses its Q-network to compute Q-values, and selects the action with the highest Q-value for interacting with the environment. As the agent continuously interacts with the environment, it stores the experiences in a replay buffer and periodically updates its Q-network, which determines its target policy.

During Q-network updates, an ideal scenario would be to replace all experiences with fresh samples generated directly from the target policy without any additional cost. This would preserve the sample efficiency from the off-policy approach while resolving the divergence between the behavior and target policies. However, since such a scenario is impractical, we introduce a novel mathematical result and adapt it to the DRL framework to bridge between on-policy and off-policy methods.

Within the causal framework, the on-policy loss corresponds to the factual loss when the treatment being optimized matches the treatment applied for generating the observations. This is analogous to optimizing the target policy using experiences from an identical behavior policy. Similarly, the off-policy loss aligns with the counterfactual loss when the treatment being optimized differs from the one used to generate the observations. This is equivalent to optimizing the target policy using experiences from a different behavior policy.

In this work, we establish a causal upper bound on the unobserved factual loss using the observed counterfactual loss and an additional term, which is the estimated treatment effect of the target treatment compared to N control treatments. After formalizing the upper bound within the causal framework, we seamlessly integrate it into the DRL framework as an upper bound for the on-policy loss using the standard off-policy loss and the estimated treatment effect, which serves as an OPE.

Throughout this paper, we demonstrate the causal definitions alongside corresponding DQN definitions to provide a clear and intuitive representation of the reduction from the causal framework to the DRL framework. Nevertheless, our method is applicable to any DRL agent with a V or Q-value network that influences its policy, such as DQN and Actor-Critic architectures, as long as a divergence exists between the target and behavior policies.

4.2 Setup and Definitions

Here, we present the necessary mathematical definitions required to formalize our causal upper bound. Comprehensive definitions and proofs, along with a detailed reduction from the causal framework to the DRL framework, are provided in Appendix B.

To simplify the equations, we present the binary treatment case, where there is a single target treatment, denoted by t = 1 , and a single control treatment, denoted by t = 0 . In practical DRL applications, we extend the binary case to a multiple control treatment case involving a single target policy and N behavior policies, reflecting the existence of experiences generated from different behavior policies in the replay buffer. The proofs and definitions for the multiple control treatment case are in Appendix B.

We employ the Neyman-Rubin potential outcomes framework and define the following: X ⊂ R d is the observation space. Y ⊂ R is the outcome space. T = { 0 , 1 } is the binary treatment space. Under this framework, there are two potential outcomes depending on the treatment assignment: if t = 1 , the observed outcome is y = Y 1 , and if t = 0 , the observed outcome is y = Y 0 . We assume there exists a joint distribution P ( X,T,Y 0 , Y 1 ) with the following components: q t := Pr( T = t ) is the treatment probability. P t X := P ( X | T = t ) is the conditional distribution of observations. P x Y 1 := P ( Y 1 | X = x ) is the conditional distribution of potential outcomes given the target treatment, and P x Y 0 := P ( Y 0 | X = x ) is the conditional distribution of potential outcomes given the control treatment. We also assume that there are no hidden confounders in the observational data, which makes the estimated treatment effect identifiable. This is formalized by assuming that the standard strong ignorability condition holds: ( Y 0 , Y 1 ) ⊥ ⊥ T | X , and 0 < p ( t = 1 | x ) < 1 for all x . Strong ignorability is a sufficient condition for the estimated treatment effect to be identifiable [70, 57]. To integrate the potential outcomes framework into the ML context, we define a neural network ϕ ( x ; θ t ) : X → Y , parameterized by weights θ t corresponding to the performed treatment t ∈ T .

We now present the analogous definitions within the DRL framework for a DQN agent, assuming the DRL framework is defined as a Markov decision process (MDP).

X := ( S × A ) is the state-action space. Potential outcome y ∼ P ( s,a ) Y t is the accumulated reward over n time steps, starting from state s , taking the initial action a , and following policy π t . The hypothesis ϕ ( x ; θ t ) := Q ( s, a ; θ t ) : ( S × A ) → Y is the Q-values network, which influences the policy π t .

Let L : R × R → R ≥ 0 represent a loss function. We rely on the following inequality assumption, which is a variation of the inverse triangle inequality:

Assumption 4.1.

The proof for the L 1 loss is in Appendix B 3 .

Now, we define the expected factual and counterfactual losses, which are used to formulate the bound.

Definition 4.2. The expected outcome loss:

The corresponding definition in DQN:

Definition 4.3. The expected temporal difference loss:

Note that in the DRL framework, the expected outcome loss refers to the temporal difference loss between the accumulated rewards and the Q-value estimation.

Definition 4.4. The expected factual outcome loss:

To formalize the corresponding definition in the DRL framework, consider a replay buffer D 1 containing samples produced by a behavior policy that matches the target policy. In this context, the on-policy loss is a special case of the expected factual loss, and for DQN, it is defined as:

Definition 4.5. The expected on-policy loss:

Note that this loss is the temporal difference loss with experiences collected on-policy, with identical target and behavior policies from D 1 .

Definition 4.6. The expected counterfactual outcome loss:

To establish the corresponding definition within the DRL framework, consider a replay buffer D 0 that stores experiences generated by a behavior policy that differs from the target policy. In this context, the off-policy loss is a special case of the expected counterfactual loss, and for DQN, it is defined as:

Definition 4.7. The expected off-policy loss:

Note that this loss is the temporal difference loss with experiences gathered off-policy, with different target and behavior policies from D 0 .

Additionally, in DRL applications, we consider the scenario of a replay buffer containing experiences generated from N distinct behavior policies. The definitions for this case are in Appendix B.

We now define our method's additional estimated treatment effect term in both the causal and the DRL frameworks:

Definition 4.8. The estimated treatment effect loss:

The interpretation for the estimated treatment effect in DRL is an OPE, and for DQN, it is defined as:

Definition 4.9. The SUFT OPE term:

This term aligns with the principles of OPE, focusing on quantifying the discrepancy between the target policy and the behavior policy using off-policy data. Once the data is stored in the buffer, it is used as observational data.

4.3 Causal Bound Theorem

Having established the key definitions, we now present our novel factual loss upper bound theorem:

Theorem 4.10. The expected factual outcome loss, ϵ F ϕ , is bounded by the expected counterfactual outcome loss, ϵ CF ϕ , the estimated treatment effect loss, ψ ϕ , and a constant term δ , independent of ϕ :

The interpretation of Theorem 4.10 in the DRL framework establishes an upper bound to the on-policy loss, by the off-policy loss, the SUFT OPE term, and a constant term δ . A certain difference exists in the observational data between the causal and the DRL framework, which is detailed in Appendix B.

Note that since δ is independent of Q , it can be omitted during the gradient-based optimization of Q .

To the best of our knowledge, we are the first to introduce such an upper bound within the causal framework on the factual loss, adapting it to the DRL framework, thereby bounding the on-policy loss. Importantly, based on our causal bound theorem, the adaptation to the DRL framework is not constrained to DQN agents but can be applied to every off-policy agent with V or Q-value networks that influence its policy. This scenario is thoroughly detailed in Appendix B. Furthermore, our method is also applicable to on-policy agents, such as PPO, which benefits from it due to the divergence between the behavior and target policies that arises during epoch optimization.

4.4 Recycling Value Network Outputs

To compute the SUFT OPE term in the causal bound, we need to calculate the Q-network values corresponding to both the target and the behavior policies. While the Q-values for the target policy can be easily computed using the current Q-network, the Q-network values for the old behavior policies are not addressable since storing the network weights for every behavior policy is impractical. Notably, during the agent's training process, the Q-values matching the behavior policies are already calculated when they are used to select the action to perform. These values are calculated for the purpose of exploitation and then discarded once the action is executed. The novel contribution of our approach lies in utilizing and reusing the value network outputs in order to compute the SUFT OPE term. This allows the reuse of data that is overlooked while enhancing the loss optimization with significant causal insights in the form of the estimated treatment effect. Our approach effectively transforms overlooked data into valuable causal insights, metaphorically turning sand data into gold .

To obtain value network outputs in practice, we store them in the experience replay buffer. For agents employing a Q-network, only a single Q-value corresponding to the behavior policy is stored, resulting in an experience tuple of the form ( s, a, r, s ′ , Q ( s, a ; θ behavior )) , instead of the standard ( s, a, r, s ′ ) . For agents using a V-network, the state value output is stored instead, yielding an experience tuple of the form ( s, a, r, s ′ , V ( s ; θ behavior )) , instead of the standard ( s, a, r, s ′ ) .

4.5 Universal Implementation of the SUFT OPE Term

Following the formalization of the SUFT causal bound, in Section 4.3, we adopt the well-established approach of optimizing a bound on the loss, a technique commonly employed in algorithms like expectation-maximization [71] and variational inference [72]. Consequently, rather than directly optimizing the on-policy loss, we focus on optimizing the SUFT causal bound. In addition, to improve

the flexibility and accuracy of the loss optimization, we incorporate a λ TF coefficient into the SUFT OPE additional term. In the case of DQN, the SUFT causal bound objective term is as follows:

In value-based methods such as DQN agents, the SUFT OPE term is seamlessly added to the standard loss function, while in Actor-Critic architectures, it is incorporated into the critic's loss. We provide a detailed pseudocode of our method algorithms for both DQN and PPO in Appendix C.

5 Experiments

In this section, we provide a comprehensive empirical analysis of our proposed method, emphasizing the impact of adding the SUFT OPE term to the agent's standard loss function across a broad and diverse range of environments. In this section, we present results obtained using L 2 loss, allowing us to compare our method to standard DRL agents. The results for L 1 loss are detailed in Appendix D.

5.1 Experimental Setup

We assess our method's performance across two distinct domains: The first consists of 57 Atari games with a discrete action space, which contain a range of decision-making scenarios, including immediate versus long-term strategies, varying image background complexities, and dynamic on-screen objects. The second includes five MuJoCo environments that simulate robotic motions with a continuous action space that provides a robust platform for dynamic physical interactions.

In the Atari domain, we perform a wide range of experiments using DQN [21], which serves as the foundational algorithm for numerous Q-value network agents, Vanilla DQN, and Double DQN [73], as well as PPO [20], which is a widely adopted agent that has become a standard in the field. The Vanilla DQN results are detailed in Appendix D. For the MuJoCo domain, we evaluate our method using SAC [22], a continuous control off-policy actor-critic agent known for its robust performance, and with PPO. Across these settings, we explore different values of λ TF , resulting in more than 10K individual experiments. This extensive and versatile experimentation ensures statistically significant findings, providing confidence in the robustness of the results and enabling us to draw more reliable conclusions about our method's performance. More details about tuning λ TF are in Appendix D.

Training Details: To evaluate SUFT's sample efficiency, we conduct experiments with constrained environment interactions. For the off-policy agents, we use a relatively small experience replay buffer size of 4K. Each agent configuration is independently trained 10 times per environment, with the training process limited to 400K steps in the Atari domain and 1M steps in the MuJoCo domain. Performances are analyzed by taking the median smoothed training reward from the 10-trial set, ensuring a more reliable evaluation while mitigating the influence of randomness on the results. This constrained resources experimentation is designed to demonstrate our method's sample efficiency, and it is a common technique in DRL. [74] Results for a larger buffer size are in Appendix D.

Implementation Details: We use the Gymnasium environments API [75] to train on the Atari [76] (version v5) and MuJoCo [77] (version v4) domains. For training the agents, we are using the default architecture and hyperparameter configurations as implemented in the "Stable-Baselines3" DRL library [78]. In the Atari games, we apply standard preprocessing, which includes resizing observations to 84x84 grayscale images and stacking four consecutive frames. This preprocessing is implemented using the library's default Atari wrapper.

5.2 Controlled Comparison with Baseline Agents

We evaluate the impact of incorporating the SUFT OPE term by comparing agents that include our proposed term to baseline agents without it, ensuring both types share identical architecture and hyperparameters, where the presence of our proposed term is the only difference between them. This controlled experiment is designed to isolate the influence of our method, ensuring fair comparisons.

In addition, we conduct a t -test for each environment against a two-sided alternative using 10 different random seed runs, where the null hypothesis is the baseline result. The p -values provide robust statistical significance results for evaluating the effectiveness of our proposed method.

Assault

Boxing

DemonAttack

JamesBond

1500

20

1250

2000

400

10

Reward

1000

1500

300

750

0

1000

200

500

10

500

100

250

0

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Assault

CrazyClimber

JamesBond

Kangaroo

40000

400

6000

1000

Reward

800

30000

300

4000

600

200

20000

2000

400

100

200

10000

0

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

#steps

# steps

# steps

# steps

SUFT

Baseline

The complete experimental results, the evaluation formulas, and more empirical analysis of our method are detailed in Appendix D.

Atari: On the Atari benchmark, SUFT demonstrates significant improvements for all of the agents. On the DQN, Double DQN, and PPO agents, SUFT surpasses the baseline agent in 52, 48, and 50 games out of the 57 games, respectively, with statistically significant gains ( p -value < 0.05) in 49, 37, and 40 games.

We also analyze the results for the valid Atari games (according to the evaluation formula in Appendix D). For the DQN agent, SUFT outperforms the baseline agent in 28 out of the 29 valid games, achieving a reward improvement of over 100% in 48.3% of them and over 10% in 93.1% of them, as shown in Figure 11. For the Double DQN agent, SUFT outperforms the baseline agent in 35 out of the 40 valid games, achieving a reward improvement of over 100% in 20% of them and over 10% in 72.5% of them, as shown in Figure 1 (Left). For the PPO agent, SUFT surpasses the baseline agent in 39 out of the 42 valid games, achieving a reward improvement of over 100% in 38.1% of them and over 10% in 83.3% of them, as shown in Figure 1 (Right).

Additionally, when evaluating the mean reward ratio, SUFT demonstrates profound gains relative to the baseline agents. For the DQN, Vanilla DQN, Double DQN, and PPO agents, SUFT achieves a mean reward ratio of 383.19%, 228.12%, 226.91%, and 197.34%, respectively, as shown in Figure 2.

Moreover, in several Atari games, SUFT not only surpasses the baseline agent but also achieves higher rewards than an average human, as shown in Figure 3. This achievement highlights the potential of our method to boost agents' performance even above the human level, while keeping constrained environment interactions and resource demands. The human rewards are sourced from [79].

MuJoCo: For the MuJoCo benchmark, SUFT demonstrates improvements in the SAC and PPO agents. For the SAC agent, SUFT surpasses the baseline agent in 4 out of 5 environments, with statistically significant gains in 3 environments. For the PPO agent, SUFT surpasses the baseline agent in 4 out of 5 environments (in the other environment, the score is the same), with statistically significant gains in 2 environments. With regard to the mean reward ratio, SUFT achieves 224.53% for the SAC agent and 111.21% for the PPO agent, as shown in Figure 2.

5.3 Buffer Size Reduction

We also evaluate the sample efficiency of SUFT by comparing its performance to an identical DQN agent utilizing a larger 100K experience replay buffer size. Remarkably, SUFT achieves a mean reward ratio gain of 437.05%, outperforming an identical agent that uses a 25 times larger buffer size.

These results underscore the potential of our approach to obtain high-performance DRL agents with significantly smaller buffers, paving the way for more computationally efficient and accessible DRL methods by enhancing sample efficiency at a negligible cost .

5.4 Ablation Study

We conduct an ablation study to isolate the contribution of the SUFT OPE term to DQN performance. As shown Table 1, we compare SUFT DQN against several alternatives: increasing the buffer size, a Vanilla DQN, and a Double DQN. While enlarging the buffer or adopting more complex DQN variants leads to improvements, incorporating the SUFT OPE term yields the highest Human Normalized Mean of 63.45% , outperforming all other baselines that require higher resources and computational costs. These results highlight the potential of our method to improve the agent's performance by enhancing sample efficiency at a negligible cost , providing a simple yet highly effective method for improving agent performance at a negligible cost .

5.5 Causal Bound Optimization vs Inertia Regularizer

To isolate the effect of our causal mechanism and ensure that the significant improvements of our method stem from the causal bound optimization rather than merely from a regularization effect, we conduct a controlled experiment. Specifically, we evaluate a double DQN agent that adds an inertia regularizer to the standard temporal difference loss function in the form of:

Where Q target network denotes the slowly updated target network, as done in a standard double DQN. The experiment uses the same configurations as both the SUFT double DQN and the baseline double DQN, with the additional regularization term being the only difference from the baseline agent. The results in Table 11 show that the inertia regularization significantly degrades double DQN performance. Resulting in an 83.21% mean reward ratio. This provides evidence that the gains from our method, which result in a 226.91% mean reward ratio improvement, arise from the causal bound optimization, where the old Q-network values used for the OPE term correspond to the behavior policy that generated the experiences, thereby mitigating the divergence between the on-policy and off-policy methods. These results indicate that our method functions not merely as a stability-promoting regularizer, but as a principled causal bound optimization.

6 Conclusion

This paper presents SUFT, a causal upper-bound loss optimization method that enhances DRL sample efficiency and reduces computational demands at a negligible cost . We begin by establishing a provable bound on the factual loss within the Neyman-Rubin potential outcomes framework and seamlessly adapting it to the DRL framework, showing that on-policy loss is bounded by every agent's standard off-policy loss and an additional SUFT OPE term. This term is computable by storing past value network outputs in the experience replay buffer, thereby reusing discarded data to improve agents' performance. Experimental results across diverse environments and agents show that our method leads to significant improvements in convergence rates and reward outcomes. Looking ahead to future work, we see physical robotics as a particularly promising domain, where SUFT's reduced-resource method could make DRL more practical under real-world constraints.

References

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the atari human benchmark. In Proceedings of the 37th International Conference on Machine Learning , pp. 507-517. PMLR, 2020.

Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Gu, S., Holly, E., Lillicrap, T., and Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA) , pp. 3389-3396. IEEE, 2017.

Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., and Levine, S. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research , 40(4-5):698-721, 2021.

Haarnoja, T., Moran, B., Lever, G., Huang, S. H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N. Y., Hafner, R., et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Science Robotics , 9(89):eadi8022, 2024.

Yu, C., Liu, J., Nemati, S., and Yin, G. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

Symeonidis, P., Chairistanidis, S., and Zanker, M. Deep reinforcement learning for medicine recommendation. In 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE) , pp. 85-90. IEEE, 2022.

Shirali, A., Schubert, A., and Alaa, A. Pruning the way to reliable policies: A multi-objective deep q-learning approach to critical care. IEEE Journal of Biomedical and Health Informatics , 2024.

Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., and Shah, A. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA) , pp. 8248-8254. IEEE, 2019.

Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A. A., Yogamani, S., and Pérez, P. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Schier, M., Reinders, C., and Rosenhahn, B. Deep reinforcement learning for autonomous driving using high-level heterogeneous graph representations. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pp. 7147-7153. IEEE, 2023.

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning , pp. 1928-1937. PMLR, 2016.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of The 35th International Conference on Machine Learning , pp. 1861-1870. PMLR, 2018.

Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In ICLR (Poster) , 2016.

Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Zhang, S. and Sutton, R. S. A deeper look at experience replay. arXiv preprint arXiv:1712.01275 , 2017.

Wei, Q., Ma, H., Chen, C., and Dong, D. Deep reinforcement learning with quantum-inspired experience replay. IEEE Transactions on Cybernetics , 52(9):9326-9338, 2021.

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of The 35th International Conference on Machine Learning , pp. 1407-1416. PMLR, 2018.

Deng, Z., Jiang, J., Long, G., and Zhang, C. Causal reinforcement learning: A survey. Transactions on Machine Learning Research , 2023.

Zeng, Y., Cai, R., Sun, F., Huang, L., and Hao, Z. A survey on causal reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Zhang, J., Tian, J., and Bareinboim, E. Partial counterfactual identification from observational and experimental data. In Proceedings of the 39th International Conference on Machine Learning , pp. 26548-26558. PMLR, 2022.

Kasetty, T., Mahajan, D., Dziugaite, G. K., Drouin, A., and Sridhar, D. Evaluating interventional reasoning capabilities of large language models. In Causality and Large Models @NeurIPS 2024 , 2024.

Wang, L., Yang, Z., and Wang, Z. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems , 34: 21164-21175, 2021.

Blumlein, T., Persson, J., and Feuerriegel, S. Learning optimal dynamic treatment regimes using causal tree methods in medicine. In Proceedings of the 7th Machine Learning for Healthcare Conference , pp. 146-171. PMLR, 2022.

Herlau, T. and Larsen, R. Reinforcement learning of causal variables using mediation analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 6910-6917, 2022.

Saghafian, S. Ambiguous dynamic treatment regimes: A reinforcement learning approach. Management Science , 70(9):5667-5690, 2024.

Ding, W., Lin, H., Li, B., and Zhao, D. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems , 35: 26532-26548, 2022.

Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese , 198(Suppl 27): 6435-6467, 2021.

Lin, H., Ding, W., Liu, Z., Niu, Y., Zhu, J., Niu, Y., and Zhao, D. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters , 2024.

Buesing, L., Weber, T., Zwols, Y., Heess, N., Racaniere, S., Guez, A., and Lespiau, J.-B. Woulda, coulda, shoulda: Counterfactually-guided policy search. In International Conference on Learning Representations , 2018.

Lee, T. E., Zhao, J. A., Sawhney, A. S., Girdhar, S., and Kroemer, O. Causal reasoning in simulation for structure and transfer learning of robot manipulation policies. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pp. 4776-4782. IEEE, 2021.

Lan, Y., Xu, X., Fang, Q., and Hao, J. Sample efficient deep reinforcement learning with online state abstraction and causal transformer model prediction. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Gasse, M., Grasset, D., Gaudron, G., and Oudeyer, P.-Y. Using confounded data in latent model-based reinforcement learning. Transactions on Machine Learning Research Journal , 2023.

Pace, A., Yèche, H., Schölkopf, B., Rätsch, G., and Tennenholtz, G. Delphic offline reinforcement learning under nonidentifiable hidden confounding. In The Twelfth International Conference on Learning Representations , 2024.

Ding, W., Shi, L., Chi, Y., and Zhao, D. Seeing is not believing: Robust reinforcement learning against spurious correlation. Advances in Neural Information Processing Systems , 36, 2024.

Namkoong, H., Keramati, R., Yadlowsky, S., and Brunskill, E. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems , 33:18819-18831, 2020.

Shi, C., Uehara, M., Huang, J., and Jiang, N. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In Proceedings of the 39th International Conference on Machine Learning , pp. 20057-20094. PMLR, 2022.

Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

Sontakke, S. A., Mehrjou, A., Itti, L., and Schölkopf, B. Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning. In Proceedings of the 38th International Conference on Machine Learning , pp. 9848-9858. PMLR, 2021.

Lu, C., Hernández-Lobato, J. M., and Schölkopf, B. Invariant causal representation learning for generalization in imitation and reinforcement learning. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality , 2022.

Seitzer, M., Schölkopf, B., and Martius, G. Causal influence detection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems , 34:2290522918, 2021.

Hu, X., Zhang, R., Tang, K., Guo, J., Yi, Q., Chen, R., Du, Z., Li, L., Guo, Q., Chen, Y., et al. Causality-driven hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems , 35:20064-20076, 2022.

Lu, C., Huang, B., Wang, K., Hernández-Lobato, J. M., Zhang, K., and Schölkopf, B. Sampleefficient reinforcement learning via counterfactual-based data augmentation. arXiv preprint arXiv:2012.09092 , 2020.

Pitis, S., Creager, E., Mandlekar, A., and Garg, A. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems , 35:18143-18156, 2022.

Rubin, D. B. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association , 100(469):322-331, 2005.

Oprescu, M., Dorn, J., Ghoummaid, M., Jesson, A., Kallus, N., and Shalit, U. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In Proceedings of the 40th International Conference on Machine Learning , pp. 26599-26618. PMLR, 2023.

Frauen, D., Melnychuk, V., and Feuerriegel, S. Sharp bounds for generalized causal sensitivity analysis. Advances in Neural Information Processing Systems , 36, 2024.

Shalit, U., Johansson, F. D., and Sontag, D. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning , pp. 3076-3085. PMLR, 2017.

Curth, A. and Van der Schaar, M. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pp. 1810-1818. PMLR, 2021.

Athey, S., Keleher, N., and Spiess, J. Machine learning who to nudge: causal vs predictive targeting in a field experiment on student financial aid renewal. Journal of Econometrics , pp. 105945, 2025.

Jesson, A., Mindermann, S., Gal, Y., and Shalit, U. Quantifying ignorance in individual-level causal-effect estimates under hidden confounding. In Proceedings of the 38th International Conference on Machine Learning , pp. 4829-4838. PMLR, 2021.

Wang, H., Fan, J., Chen, Z., Li, H., Liu, W., Liu, T., Dai, Q., Wang, Y., Dong, Z., and Tang, R. Optimal transport for treatment effect estimation. Advances in Neural Information Processing Systems , 36, 2024.

Kennedy, E. H., Balakrishnan, S., and Wasserman, L. Semiparametric counterfactual density estimation. Biometrika , 110(4):875-896, 2023.

Kuzmanovic, M., Hatt, T., and Feuerriegel, S. Estimating conditional average treatment effects with missing treatment information. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pp. 746-766. PMLR, 2023.

Melnychuk, V., Frauen, D., and Feuerriegel, S. Causal transformer for estimating counterfactual outcomes. In Proceedings of the 39th International Conference on Machine Learning , pp. 15293-15329. PMLR, 2022.

Frauen, D., Hatt, T., Melnychuk, V., and Feuerriegel, S. Estimating average causal effects from patient trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pp. 7586-7594, 2023.

Lewis, D. Counterfactuals . John Wiley & Sons, 2013.

Pearl, J. and Mackenzie, D. The book of why: the new science of cause and effect . Basic books, 2018.

Uehara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 , 2022.

Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction . MIT press, 2018.

Pearl, J. Detecting latent heterogeneity. Sociological Methods & Research , 46(3):370-389, 2015.

Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological) , 39(1): 1-22, 1977.

Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning . Springer, 2006.

Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mastering atari games with limited data. Advances in neural information processing systems , 34:25476-25488, 2021.

Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 , 2024.

Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47: 253-279, 2013.

Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pp. 5026-5033. IEEE, 2012.

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stablebaselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning , pp. 1995-2003. PMLR, 2016.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

A Causal Bound Notation Summary

L

(

·

,

·

) :

R

×

R

→

R

0

:

loss function.

q t : the probability of treatment t .

P t X : the conditional distribution of observations.

P x Y 1 : the conditional distribution of potential outcomes given the target treatment.

P x Y 0 : the conditional distribution of potential outcomes given the control treatment.

ϕ ( x ; θ t ) : X → Y : neural network hypothesis parameterized by weights θ t .

ℓ ϕ ( x, t ) : the expected outcome loss.

ϵ F ϕ : the expected factual outcome loss.

ϵ

CF

ϕ

:

the expected counterfactual outcome loss.

ψ ϕ : The estimated treatment effect loss.

δ : a constant term independent of ϕ .

B Theoretical Result

B.1 Definitions and Proofs

In this section, we present the complete definitions and proofs to establish our theoretical result within the causal framework. Let L : R × R → R ≥ 0 represent a loss function.

Assumption B.1.

Proposition B.2. Assumption B.1 holds for L 1 loss.

Proof.

=

|

x

-

x

′

+

y

′

-

y

)

|

Inequality 1 is by x ≤ | x | , equality 2 is by the definition of L 1 loss, inequality 3 is by reverse triangle inequality || x 1 | - | x 2 || ≤ | x 1 -x 2 | , inequality 4 is by triangle inequality | x 1 + x 2 | ≤ | x 1 | + | x 2 | , equality 5 is by absolute symmetry | a | = | -a | , and the definition of L 1 loss.

We start with the binary treatment case and employ the Neyman-Rubin potential outcomes framework to define the following: X ⊂ R d is the observation space. Y ⊂ R is the outcome space. T = { 0 , 1 } is the binary treatment space. Under this framework, there are two potential outcomes depending on the treatment assignment: if t = 1 , the observed outcome is y = Y 1 , and if t = 0 , the observed outcome is y = Y 0 .

We assume a joint distribution P ( X,T,Y 0 , Y 1 ) with the following components: q t := Pr( T = t ) is the treatment probability. P t X := P ( X | T = t ) is the conditional distribution of observations. P x Y 1 := P ( Y 1 | X = x ) is the conditional distribution of potential outcomes given the target treatment, and P x Y 0 := P ( Y 0 | X = x ) is the conditional distribution of potential outcomes given the control treatment. We also assume that there are no hidden confounders in the observational data, which makes the estimated treatment effect identifiable. This is formalized by assuming that the standard strong ignorability condition holds: ( Y 0 , Y 1 ) ⊥ ⊥ T | X , and 0 < p ( t = 1 | x ) < 1 for all x . Strong ignorability is a sufficient condition for the estimated treatment effect to be identifiable [70, 57].

We now present the definitions used to formulate the causal upper bound.

Let t ∈ T be a treatment.

Definition B.3. ϕ ( x ; θ t ) : X → Y is a neural network hypothesis parameterized by weights θ t corresponding to the performed treatment t .

Definition B.4. The expected outcome loss:

Definition B.5. The expected factual outcome loss:

Definition B.6. The expected combined factual outcome loss:

Definition B.7. The expected counterfactual outcome loss:

Definition B.8. The expected combined counterfactual outcome loss:

Definition B.9. The estimated treatment effect loss:

Definition B.10. The combined estimated treatment effect loss:

B.2 Causal Bound Theorem

Similarly, for the opposite case:

Proof.

The proof for the opposite case is the same, replacing y 0 and y 1 .

Theorem B.12. The expected factual outcome loss, ϵ F ϕ , is bounded by the expected counterfactual outcome loss, ϵ CF ϕ , the estimated treatment effect loss, ψ ϕ , and a constant term δ , independent of ϕ :

Where δ is the following:

Proof. We begin by placing the loss definitions:

Now we will use the linearity of expectation and Lemma B.11:

According to Assumption B.1 we will get:

Finally, we are getting the causal bound:

Equality 6 is by Definitions B.6 and B.8, equality 7 is by Definitions B.5 and B.7, equality 8 is by Definition B.4, equality 9 is by linearity of expectations, equality 10 is by linearity of expectations and by Lemma B.11, inequality 11 is by Assumption B.1, equality 12 is by linearity of expectations and because ϕ is a constant on the expectation of Y 0 , Y 1 , equality 13 is by Definition B.9 and δ 1 , δ 0 notation, equality 14 is by Definition B.10 and δ notation.

B.3 Causal Bound for N Control Treatments

We now adjust the definitions and Theorem B.12 proof from the binary treatment scenario to a single target treatment and N control treatments scenario.

We first set the definitions of the Neyman-Rubin potential outcome framework for the N control treatments scenario: T = { 1 , . . . , N +1 } is the treatments space, where t = j : 2 ≤ j ≤ N +1 is the control treatments, and t = 1 is the target treatment compared against each control treatment. Under this framework, there are two types of potential outcomes depending on the treatment assignment: if t = 1 , the observed outcome y = Y 1 is from the target treatment, and ∀ j ∈ { 2 , . . . , N +1 } if t = j , the observed outcome y = Y j is from a control treatment.

We assume joint distribution P ( X,T,Y 1 , Y 2 , . . . , Y N +1 ) with the following components: q t := Pr( T = t ) is the treatment probability. P t X := P ( X | T = t ) is the conditional distribution of observations. P x Y 1 := P ( Y 1 | X = x ) is the conditional distribution of potential outcomes given the target treatment, and ∀ j ∈ { 2 , . . . , N +1 } P x Y j := P ( Y j | X = x ) is the conditional distribution of potential outcomes given a control treatment t = j .

Definition B.13. The expected factual outcome loss for N control treatments:

Definition B.14. The expected combined factual outcome loss for N control treatments:

Definition B.15. The expected counterfactual outcome loss for N control treatments: ∀ j ∈ { 2 , . . . , N +1 } :

Definition B.16. The expected combined counterfactual outcome loss for N control treatments:

Definition B.17. The estimated treatment effect loss for N control treatments: ∀ j ∈ { 2 , . . . , N +1 } :

Definition B.18. The combined estimated treatment effect loss for N control treatments:

All the other definitions and proofs are the same as the binary case.

We now generalize Theorem B.12 from a single control treatment scenario to the N control treatments scenario as follows:

Theorem B.19.

Where δ is the following:

Proof. by induction.

Step 1: Base Case

For a single target treatment and a single control treatment, N = 1 , the following inequality holds:

∀ j ∈ { 2 , . . . , N +1 } :

As we prove in Theorem B.12 where the control treatment is t = 2 instead of t = 0 . Step 2: Inductive Hypothesis

Assume that for N -1 ≥ 2 control treatments, the following inequality holds:

Step 3: Inductive Step

We will now prove the inequality for N control treatments:

Inductive Step proof.

We will get the following inequality:

Finally, we sum 1 N q 1 ϵ 1 F ϕ and get the causal bound for N control treatments:

Causal

DRL

Framework

Framework

EFΦ

Factual

EOn-Policy Q

On-Policy

ECFΦ

Counterfactual

EOff-Policy Q

Off-Policy

Treatment term

YSUFTQ

OPE Term

Inequality 15 is by the inductive hypothesis, Inequality 16 is from the proof of Theorem B.12 where the control treatment is t = N +1 instead of t = 0 .

B.4 Reduction from Causal to DRL Framework

We now formalize a reduction from a causal framework to a DRL framework.

In the DRL framework, we consider the treatment to be the agent's policy, where the target treatment matches the target policy, and the control treatments match the behavior policies.

Within the causal framework, the on-policy loss corresponds to the factual loss when the treatment being optimized matches the treatment applied for generating the observations. This is analogous to optimizing the target policy using experiences from an identical behavior policy. Similarly, the off-policy loss aligns with the counterfactual loss when the treatment being optimized differs from the one used to generate the observations. This is equivalent to optimizing the target policy using experiences from different behavior policies.

To formalize the corresponding definition in the DRL framework, consider a replay buffer D 1 containing samples produced by a behavior policy that matches the target policy. In this context, the on-policy loss is a special case of the expected factual loss.

Similarly, consider a replay buffer D 0 that stores experiences generated by behavior policies that differ from the target policy. In this context, the off-policy loss is a special case of the expected counterfactual loss.

Q-Network Reduction: The definitions for the scenario of an agent using a Q-value network:

X := ( S × A ) is the state-action space. Potential outcome y ∼ P ( s,a ) Y t is the accumulated reward over n time steps, starting from state s , taking the initial action a , and following policy π t . And the hypothesis ϕ ( x ; θ t ) := Q ( s, a ; θ t ) : ( S × A ) → Y is the Q-values network, which influences the policy π t .

Definition B.4 in the DRL framework refers to the temporal difference loss between the accumulated rewards and the Q-value estimation, and it is defined as follows:

Definition B.20. The expected temporal difference loss:

Definitions B.5 and B.13 in the DRL framework assigns the temporal difference loss with experiences collected on-policy, with identical target and behavior policies from D 1 , and it is defined as follows:

Definition B.21. The expected on-policy loss:

Definitions B.7 and B.15 in the DRL framework assigns the temporal difference loss with experiences gathered off-policy, with different target and behavior policies from D 0 , and it is defined as follows:

Definition B.22. The expected off-policy loss:

Definitions B.9 and B.17 interpretation in the DRL framework is an OPE term, and it is defined as follows:

Definition B.23. The SUFT OPE term:

Note that this term evaluates the target policy by measuring the loss between the Q-value estimations of the target policy and the behavior policies, given samples from D 0 .

Theorems B.12 and B.19 interpretation in the DRL framework establishes an upper bound to the on-policy loss, combining the off-policy loss and the SUFT OPE term.

Note that since the constant term, δ , is independent of Q , it is disregarded during the gradient-based optimization of Q , so the SUFT causal bound objective term is as follows:

V-Network Reduction: Now we present the corresponding definitions for the scenario of an agent using a V network:

X := S is the state space. Potential outcome y ∼ P s Y t is the accumulated reward over n time steps, starting from state s , and following policy π t . And the hypothesis ϕ ( x ; θ t ) := V ( s ; θ t ) : S → Y is the V network, which influences the policy π t .

Definition B.4 in the DRL framework refers to the temporal difference loss between the accumulated rewards and the V estimation, and it is defined as follows:

Definition B.24. The expected temporal difference loss:

Definitions B.5 and B.13 in the DRL framework assigns the temporal difference loss with experiences collected on-policy, with identical target and behavior policies from D 1 , and it is defined as follows:

Definition B.25. The expected on-policy loss:

Definitions B.7 and B.15 in the DRL framework assigns the temporal difference loss with experiences gathered off-policy, with different target and behavior policies from D 0 , and it is defined as follows:

Definition B.26. The expected off-policy loss:

Definitions B.9 and B.17 interpretation in the DRL framework is an OPE term, and it is defined as follows:

Definition B.27. The SUFT OPE term:

Note that this term evaluates the target policy by measuring the loss between the V estimations of the target policy and the behavior policies, given samples from D 0 .

Theorems B.12 and B.19 interpretation in the DRL framework establishes an upper bound to the on-policy loss, combining the off-policy loss and the SUFT OPE term.

Note that since the constant term, δ , is independent of V , it is disregarded during the gradient-based optimization of V . Thus, the SUFT causal bound objective term is as follows:

Causal and DRL Framework Alignment: An important alignment needs to be made to address a difference between the static observational data assumed by the causal framework and the dynamic experience replay buffer in the online DRL framework. In the causal framework, all observations are gathered once, so the data is fixed and irrevocably labeled either to be from the target treatment or from the control treatment in the binary treatment case. On the other hand, in the online DRL framework, the observational data changes over time, as we continuously populate the experience replay buffer and update the target policy. On each policy optimization step, all of the samples stored in the buffer that were labeled as the target policy are turned into samples that are labeled as a control policy. This phenomenon of changing the observational data labels when updating the policy in the DRL framework is a gap between the causal and DRL frameworks. In the on-policy case, the experience replay buffer contains only on-policy observations, meaning that q 1 = 1 . When updating the target policy, this data becomes off-policy, and the data that was previously labeled as target policy turns into control policy, shifting the samples' labels together with the treatment probability q 0 new = q 1 old = 1 .

The reduction we have made between the causal and the DRL framework above aligns the on-policy loss and off-policy loss terms with the corresponding terms in the causal framework. The on-policy loss is a special case of the factual loss where the observation data is filled with target treatment samples, corresponding to ϵ 1 F where q 1 = 1 . The off-policy loss is a special case of the counterfactual loss where the observation data is filled with control treatment samples, corresponding to ϵ 1 CF where the labels and the treatment probability are shifted and q 0 new = q 1 old = 1 .

C SUFT Pseudocode

C.1 SUFT DQN Pseudocode

Our method extends the standard DQN algorithm [80] in two main ways: The first is by storing the old Q values Q ( s, a ; θ behavior ) in the replay buffer. The second is by incorporating the additional SUFT OPE term into the objective function λ TF · ( Q ( s j , a j ; θ behavior ) -Q ( s j , a j ; θ target )) 2 , which represents the estimated treatment effect. θ behavior refers to the network weights used for action selection, the behavior policy, and θ target represents the current network weights, which are optimized in the gradient descent step, the target policy.

C.2 SUFT PPO Pseudocode

Algorithm 2

Our method extends the standard PPO algorithm [20] by incorporating the additional SUFT OPE term into the PPO critic loss. Notably, PPO performs multiple epochs of optimization over the same rollout buffer D 1 , leading to a divergence between the behavior and target policies. This divergence is the reason PPO benefits from our method, despite being an on-policy algorithm. θ behavior refers to the network weights used to interact with the environment also referred to in [20] as θ old, the behavior policy, and θ target represents the current network weights, which are optimized, also referred to in [20] as θ , the target policy. During the first epoch, before optimizing the network, the behavior and target policies are identical, so the SUFT OPE term equals zero. After the first optimization step, the target policy diverges from the behavior policy, and the SUFT OPE term now quantifies the estimated treatment effect between the target and behavior policies. In addition, the V ( s t ; θ behavior ) values are already calculated during the calculation of the advantage estimates ˆ A t , and rewards to go ˆ R t . Thus, our method effectively reuses these stored network outputs for computing the SUFT OPE term, similar to the approach used in our SUFT DQN implementation.

D Full Experimental Results

In this section, we present detailed and comprehensive experimental results for both the Atari and the MuJoCo domains, as well as the formulas and criteria used to evaluate the results.

D.1 Negligible Computational Overhead

We quantify the computational and space overhead introduced by computing the SUFT OPE term and storing old Q-values in the experience replay buffer. We run our experiments on a single NVIDIA L4 GPU, and for the Vanilla DQN agent, the additional computation takes approximately 23.77 seconds, compared to a total training time of 2,980 seconds. This corresponds to a 0.8% computational overhead. In terms of space, we append a single 64-bit scalar to each stored experience. For a 4K experience replay buffer, this results in an additional 0.0305 MB, compared to 972.85 MB without it. This is equivalent to a 0.003% space overhead.

D.2 Tuning the λ TF Coefficient

In order to evaluate the impact of the λ TF coefficient on improving the loss accuracy, we conducted experiments using various λ TF values. Adding the λ TF demonstrates a consistent improvement across a range of λ TF values. This indicates that precise hyperparameter tuning is unnecessary, and as long as the SUFT OPE term is added with a coefficient within a valid range of λ TF , the loss optimization improves. The λ TF value can be easily tuned through random search hyperparameter optimization within a valid range. Specifically, we encounter that a sufficient range to search from is between λ TF = 0 . 5 and λ TF = 1 . 5 . The λ TF values that yield the best performance are λ TF = 1 for all the DQN agents, λ TF = 2 . 5 for the PPO agent using the Using L 1 Loss and λ TF = 5 for the PPO agent using the Using L 2 Loss in the Atari domain. In the MuJoCo domain, SAC performs best at λ TF = 0 . 6 and PPO at λ TF = 1 . 8 .

D.3 Evaluation Formulas

To evaluate our method, we train every agent-environment configuration for 10 different runs and report the median result for each setup. Since this is an even-numbered list, there are two central values, and we use the higher of the two as the median. The valid environments used for calculating the improvement percentage and mean reward ratio percentage formulas are those where the numerator and denominator are greater than one. This restriction allows us to exclude cases where the reward is only marginally higher than the random reward, which could otherwise yield arbitrarily large ratios. By doing so, we ensure that the metric remains stable and effectively reflects meaningful improvements.

Improvement Percentage Formula: The improvement percentage for comparing our approach to an identical agent without the additional term is calculated with the following formula:

For better visualization of the results, we display them in log scale with the following formula:

We add 1 to the improvement percentage formula to avoid the log of 0 and fractions, which result in a negative log. This addition has no impact on the results.

Mean Reward Ratio Percentage Formula: The mean reward ratio percentage for comparing our approach to an identical agent without the additional term is calculated with the following formula:

Note that for the Atari domain, the human normalized results are calculated using the above formula, where the baseline reward is replaced with the human reward.

D.4 Assumption 4.1 Empirical Analysis with L 2

DRL agents are often trained using L 2 loss, for which Assumption 4.1 does not hold. To provide intuition for the observed L 2 performance, we empirically examine the validity of the L 2 loss for Assumption 4.1 using synthetic and domain-specific data. For the synthetic data, we generate one billion independent samples from the standard normal distribution, x, y, x ′ , y ′ ∼ N (0 , 1) , and evaluate whether:

This experiment corresponds directly to the inequality form of Assumption 4.1 when applied to the L 2 loss. We find that inequality 17 holds in approximately 82% of random samples, demonstrating that while Assumption 4.1 is not guaranteed theoretically, it is satisfied in the vast majority of cases. To further validate this observation in a domain-specific dataset, we conduct an experiment in the Atari 57 benchmark using a DQN agent trained for 400K steps with a batch size of 32. Since our method operates in an off-policy setting, the left side of the causal bound in Theorem 4.10 is not computable from the observed data, as we don't observe the on-policy experiences (this is the reason we optimize on the causal bound in our method). For this reason, we evaluate only inequality 11 used in the proof of Theorem B.12, considering the off-policy terms on both sides (with q 0 = 1 ):

Because the true values of y 0 and y 1 are not directly observable, since they are the true values from the given state until the end of the episode, we approximate them using the standard Bellman equation for Q-values. Under this setup, we find that inequality 18 holds in approximately 79% of cases across the 57 Atari games. These empirical findings indicate that although Assumption 4.1 is not theoretically guaranteed for L 2 , it holds in the vast majority of the optimization steps in practice, which may explain the strong empirical performance of our method using the L 2 loss.

D.5 Empirical Evaluation of the δ Term Magnitude

In Theorem 4.10 the δ term is ignored during optimization since it is independent of Q . To assess whether this omission introduces significant looseness in the causal bound, we conduct an empirical analysis on the Atari 57 benchmark using a DQN agent trained for 400K steps. Specifically, we plotted the magnitudes of the standard DQN loss, the δ term, and the SUFT OPE term throughout training. As discussed in Appendix D.4, the δ term involves the true values of y 0 and y 1 , which are not directly observable. Thus, we approximate them using the standard Bellman equation for Q-values. The results, shown in Figure 5, indicate that across all of the environments, the δ term remains consistently an order of magnitude lower than the standard DQN loss, and on the same order of magnitude as the SUFT OPE term. These findings support the decision to ignore the δ term during optimization, as it does not meaningfully loosen the causal bound.

MSE Loss MSE Loss MSE Loss

Alien

Amidar

Assault

Asterix

Asteroids

7.5

1.0

2.5

0

OK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0t

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

00

OK

100K

200K

300K

400K

Atlantis

BankHeist

BattleZone

BeamRider

1.0

0.2

0.75

1.0

0.5

0.1

0.1

0.50

0.25

0.5

0.0

100K

200K

300K

400K

0.0

oK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0.00

oK

100K

200K

300K

400K

0.0

oK

100K

200K

300K

400K

Bowling

Boxing

Breakout

Centipede

Chopper Command

0.2

0.4

10

15

0.5

1.0

5

0.0

0t

00

LOSS MSE Loss MSE Loss MSE Loss

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

DemonAttack

100K

200K

300K

400K

OK

Double Dunk

100K

200K

300K

400K

OK

100K

Enduro

200K

400K

1.0

Z'C

30

0.5

0.1

OK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0t

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0

OK

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

10

0.75

0.5

0.25

0.50

0.05

0.10

0.0

OK

100K

200K

300K

400K

0.00

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

0

OK

100K

200K

300K

400K

00°0

OK

100K

200K

...

400K

H.E.R.O.

Ice Hockey

JamesBond

Kangaroo

Krull

0.15

0.10

0.05

0.2

0.1

0.3

MLTL

0.5

1.0

1.5

15

10

品

0

0.00

0.0

0.0

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

400K

Kung-FuMaster

Montezuma'sRevenge

NameThis Game

Phoenix

0.4

0.0002

15

10

10

1.5

1.0

0.0001

0.5

0.0

OK

100K

200K

300K

400K

0.0000

OK

100K

200K

300K

400K

OK

100K

200K

400K

0

OK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

Pit Fall

Pong

PrivateEye

Q*Bert

River Raid

0.1

0.2

0.1

0.2

0.1

10

0.0

OK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0.0

OK

100K

200K

X000

400K

0

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Robotank

Seaquest

Skiing

Solaris

1.5

00E

0.10

MSE I

20

0.2

0.5

1.0

100

200

0.05

0.0

0.0

0

00°0

MSE LosS MSE Loss MSE Loss

OK

SpaceInvaders

100K

200K

300K

400K

Star Gunner

100K

200K

300K

400K

OK

100K

Surround

200K

300K

400K

OK

100K

Tennis

200K

300K

400K

OK

100K

TimePilot

200K

300K

400K

0.2

0.1

0.10

0.05

0.15

0.1

0.2

0.6

0.4

0.2

0

UK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0.00

OK

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0.0

OK

100K

200K

400K

Tutankham

Up N Down

Venture

VideoPinball

WizardofWor

0.06

0.3

0.04

20

0.2

0.02

10

0.1

YarsRevenge

100K

200K

300K

400K

OK

100K

Zaxxon

200K

300K

400K

0.00

OK

100K

#Steps

200K

X000

400K

0

OK

100K

#Steps

200K

300K

400K

00

OK

100K

#Steps

200K

400K

10

0.1

0

OK

100K

#Steps

200K

300K

400K

0.0

OK

100K

#Steps

200K

300K

400K

Standard DQN Loss

Delta Term

SUFT OPE Term

D.6 Atari 2600 57 Games Benchmark

Atari PPO Results (Using L 1 Loss):

Alien

Amidar

Assault

Asterix

Asteroids

Reward

OK

100K

200K

300K

400K

100

200

OK

100K

200K

300K

400K

400

600

200

100K200K300K

400K

400

009

200

OK

100K

（200K300K

400K

600

400

OK

100K

200K300K

400K

Atlantis

Bank Heist

BattleZone

BeamRider

Reward

10000

40000

500

20000

5000

400

400

300

200

100K

Bowling

200K

300K

400K

OK

100K

Boxing

200K

300K

400K

100K

Breakout

200K

300K

400K

OK

Centipede

100K

200K

300K

400K

Chopper Command

OK

100K

200K

300K

400K

Reward

15

2000

40

10

3000

1500

2000

1000

20

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

Double Dunk

Enduro

Reward

2000

40000

4000

006

200

400

000

-20.0

17.5

75

25

2000

100

22.5

oK

100K

200K

300K

400K

OK

100K

200K300K

400K

100K

200K

300K

400K

oK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

Reward

2000

600

400

10

1000

400

200

200

OK

100K

H.E.R.O.

200K

300K

400K

Ice Hockey

100K

200K

300K

400K

JamesBond

100K

200K

300K

400K

OK

100K

Kangaroo

200K

300K

400K

100K

Krull

200K

300K

400K

Reward

f00s

10000

10

5

200

400

2000

4000

4000

2000

15

100K

200K

300K

400K

OK

100K

200K

400K

100K

200K

400K

100K

200K

300K

400K

100K

200K

XOO0

400K

Kung-Fu Master

Montezuma'sRevenge

NameThisGame

Phoenix

Reward

4000

1.0

2000

1500

4000

5000

2000

2000

0.5

1000

2000

3000

1000

0.0

500

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

X000

400K

100K

300K

400K

OK

100K

200K

300K

400K

Pit Fall

Pong

Private Eye

Q*Bert

River Raid

Reward

-10

3000

15

2000

2000

100K

200K

300K

400K

20

oK

100K

200K

300K

400K

500

oK

100K

200K

400K

1000

0

100K

200K

300K

400K

1000

100K

200K

400K

RoadRunner

Robotank

Seaquest

Skiing

Solaris

Reward

750

10000

3000

000S

500

250

20000

2000

100K

200K

300K

400K

100K200K300K

400K

OK

100K

200K

300K

400K

30000

OK

100K

200K

300K

400K

1000

OK

100K

200K

300K

400K

SpaceInvaders

Star Gunner

Surround

Tennis

Time Pilot

Reward

8.5

-10

4000

1000

008

-9.5

9.0

-20

15

3000

100

600

10.0

2000

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

Tutankham

Up N Down

Venture

VideoPinball

WizardofWor

Reward

100

20000

100

15000

1000

50

10000

10000

500

5000

YarsRevenge

100K

200K

300K

400K

OK

100K

Zaxxon

200K

300K

400K

100K

# steps

200K

X000

400K

OK

100K

# steps

200K

300K

400K

100K

#steps

200K

400K

Reward

15000

1500

10000

100

5000

500

OK

100K

#steps

200K

300K

400K

0

oK

100K

#steps

200K

300K

400K

SUFT

Baseline

Up N Down

Enduro

Name This Game

YarsRevenge

Ice Hockey

Asterix

Chopper Command

Pong

Space Invaders

Gopher

Centipede

Atlantis

Bank Heist

JamesBond

BeamRider

Seaquest

Tutankham

Freeway

StarGunner

Assault

BattleZone

Kung-Fu Master

River Raid

+10%

+100%

Pit Fall

H.E.R.O.

Alien

Krull

Tennis

Amidar

Robotank

Breakout

Kangaroo

Q*Bert

Gravitar

Zaxxon

Frostbite

0.0

0.5

1.0

1.5

2.0

2.5

Log Improvement (%)

SUFT

Baseline

Atari PPO Results (Using L 2 Loss):

Alien

Amidar

Assault

Asterix

Asteroids

Reward

200

1000

500

600

400

200

008

500

600

400

OK

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

Atlantis

Bank Heist

BattleZone

Beam Rider

Reward

60000

800

4000-

09

10000

500

600

20

5000

400

00E

200

100K

Bowling

200K

300K

400K

OK

100K

Boxing

200K

400K

100K

Breakout

200K

X000

400K

OK

Centipede

100K

200K

400K

Chopper Command

OK

100K

200K

300K

400K

Reward

60

20

5000

4000

4000

3000-

20

-10

10

3000

2000

2000

1000

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

DoubleDunk

Enduro

Reward

4000

600

800

-21

20000

3000

400

2000

200

OK

100K

200K

300K

400K

oK

100K

200K300K

400K

oK

100K

200K

300K

400K

oK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

Reward

70

3000

600

400

-80-

2000

400

90

10

1000

200

100

OK

100K

H.E.R.O.

200K

300K

400K

Ice Hockey

100K

200K

300K

400K

OK

JamesBond

100K

200K

300K

400K

OK

100K

Kangaroo

200K

300K

400K

100K

Krull

200K

300K

400K

Reward

10000

-10

400

5000

7500

10000

15

200

2500

5000

100K

200K

300K

400K

OK

100K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

400K

Kung-Fu Master

Montezuma'sRevenge

NameThis Game

Phoenix

Reward

4000

2000

2000

1500

4000

1500

2000

1000

3000

1000

500

2000

500

OK

100K

200K

400K

OK

100K

200K

300K

400K

OK

100K

200K

X000

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

Pit Fall

Pong

Private Eye

Q*Bert

RiverRaid

Reward

2000

7500

4000

-200

400

OK

100K

200K

300K400K

20.0

15.0

17.5

100K

200K

400K

1000

0

100K

200K

300K

400K

5000

2500

0

OK

100K

200K

300K

400K

2000

3000

1000

100K

200K

400K

RoadRunner

Robotank

Seaquest

Skiing

Solaris

Reward

1000

10000

3000

5000

500

20000

2000

OK

100K

200K

300K

400K

100K200K300K

400K

OK

100K

200K

300K

400K

30000

OK

100K

200K

300K

400K

1000

OK

100K

200K

300K

400K

SpaceInvaders

Star Gunner

Surround

Tennis

Time Pilot

Reward

400

00o0

1200

8

15

-10

4000

3000

800

-20

100

600

10

2000

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

Tutankham

Up N Down

Venture

VideoPinball

Wizard ofWor

Reward

100

150

5000

7500

100

10000

15000

1000

05

2500

500

5000

Yars Revenge

100K

200K

300K

400K

OK

100K

Zaxxon

200K

300K

400K

100K

# steps

200K

X000

400K

OK

100K

# steps

200K

300K

400K

OK

100K

#steps

200K

X00

400K

Reward

20000

3000

15000

1000

200

5000

1000

100K

# steps

200K

300K

400K

0

oK

100K

#steps

200K

300K

400K

SUFT

Baseline

Enduro

DemonAttack

Chopper Command

Name This Game

YarsRevenge

Asterix

Centipede

Q*Bert

Gopher

Krull

Assault

Zaxxon

Up N Down

Seaquest

Atlantis

Kangaroo

SpaceInvaders

RiverRaid

Ice Hockey

Frostbite

Tutankham

BattleZone

Gravitar

Kung-Fu Master

Pong

Star Gunner

Beam Rider

Bowling

H.E.R.O.

Breakout

JamesBond

BankHeist

Freeway

Alien

Pit Fall

Tennis

+10%

+100%

Robotank

0.0

0.5

1.0

1.5

2.0

2.5

Log Improvement (%)

SUFT

Baseline

Atari DQN Results:

Alien

Amidar

Assault

Asterix

Asteroids

Reward

200

2000

1000

600

100

1000

500

400-

200

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K300K

400K

DK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Atlantis

Bank Heist

BattleZone

Beam Rider

Reward

20000

0 25

75

10000

1000

600

10000

5000

500

400-

200

100K

Bowling

200K

300K

400K

OK

100K

Boxing

200K

300K

400K

0

OK

100K

Breakout

200K

300K

400K

OK

Centipede

100K

200K

400K

Chopper Command

OK

100K

200K

300K

400K

Reward

40

-20

20

0

20

10

2000

4000

3000

1000

1500

20

40

1000

500

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

X000

400K

100K

200K

300K

400K

DemonAttack

Double Dunk

Enduro

Reward

10000

3000

20000

3000

2000

4000

2000

-22

-23

150

100

50

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

oK

100K

200K

300K

400K

24 t

oK

100K

200K

300K

400K

0

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

Reward

1000

750

5001

400

10

500

250

200

100K

H.E.R.O.

200K

300K

400K

Ice Hockey

100K

200K

300K

400K

0

OK

JamesBond

100K

200K

300K

400K

100K

Kangaroo

200K

300K

400K

100K

Krull

200K

300K

400K

Reward

4000

6000

10

400

6000

4000

4000

f0002

15

200

2000

2000

Kung-Fu Master

100K

200K

300K

400K

Montezuma's Revenge

OK

100K

200K

300K

400K

OK

100K

XO0000Z

400K

NameThis Game

100K

200K

300K

400K

100K

Phoenix

200K

400K

Reward

0.10

4000

3000

0.05

1000

2000

2000

500

1000

00°0

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

X000

400K

OK

100K200K

400K

OK

100K

200K

300K

400K

Pit Fall

Pong

Private Eye

Q*Bert

River Raid

Reward

2000

3000

3000

1000

-500

100K

200K

300K400K

-18

20

OK

100K

200K

300K

400K

1000

XO

100K

200K

300K

400K

200

1000

0

100K

200K

300K

400K

2000

1000

100K

400K

RoadRunner

Robotank

Seaquest

Skiing

Solaris

Reward

1000

2

1000

20000

3000

5000

500

-25000

2000

0

100K

200K

300K

400K

2.5

OK

100K

200K

300K

400K

0七

oK

100K

200K

300K

400K

30000

oK

100K

200K

300K

400K

1000

OK

100K

200K

300K

400K

SpaceInvaders

Star Gunner

Surround

Tennis

Time Pilot

Reward

400

1000

8.5

18

4000

200

500

9.0

9.5

-22

2000

10.0

24

Tutankham

100K

200K

300K

400K

Up N Down

100K

200K

300K

400K

OK

100K

Venture

200K

300K

400K

VideoPinball

100K

200K

300K

400K

WizardofWor

100K

200K

300K

400K

Reward

0009

30

750-

4000

20

10000

500

2000

10

5000

250

YarsRevenge

100K

200K

300K

400K

100K

Zaxxon

200K

300K

400K

OK

100K

# steps

200K

300K

400K

OK

100K

#steps

200K

300K

400K

OK

100K

# steps

200K

X00

400K

Reward

3000

2000

2500

1000

100K

# steps

200K

300K

400K

0

oK

100K

# steps

200K

300K

400K

SUFT

Baseline

Kangaroo

Gopher

Atlantis

Frostbite

H.E.R.O.

Seaquest

Enduro

NameThisGame

BeamRider

Q*Bert

Robotank

DemonAttack

Assault

Up N Down

River Raid

Alien

Chopper Command

Amidar

Breakout

Pit Fall

SpaceInvaders

Asterix

Freeway

Centipede

+10%

+100%

Tennis

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Log Improvement (%)

SUFT

Baseline

Atari Vanilla DQN Results:

2

Alien

Amidar

Assault

Asterix

Asteroids

Reward

200

1500

1000

800

100

1000

500

400

100K200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

400K

oK

100K

200K

300K

400K

Atlantis

Bank Heist

BattleZone

BeamRider

Reward

20000

60

15000

1500

600

10000

20

10000

5000

1000

500

400-

200

100K

Bowling

200K

300K

400K

OK

100K

Boxing

200K

300K

400K

0

OK

100K

Breakout

200K

X000

400K

OK

Centipede

100K

200K

400K

Chopper Command

100K

200K

300K

400K

Reward

20

4000

2000

10

3000

1500

0

10

2000

1000

10

1000

500

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

DemonAttack

Double Dunk

Enduro

Reward

40000

20000

4000

3000

2000

2000

3000

1000

-23.0

22.5

23.5

22.0

100

200

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

oK

100K

200K

300K

400K

oK

100K

200K

300K

400K

0

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

Reward

-60

20

750

500-

800

300

200-

80

10

250

100

200

100K

200K

300K

400K

100K

200K

300K

400K

0

OK

100K

200K

300K

400K

100K

200K

X000

400K

100K

200K

300K

400K

H.E.R.O.

Ice Hockey

JamesBond

Kangaroo

Krull

Reward

10000

10

400

4000

4000

5000

200

2000

2000

15

100K

200K

300K

400K

100K

200K

400K

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

Kung-Fu Master

Montezuma'sRevenge

NameThis Game

Phoenix

Reward

4000

1.0

1500

4000

3000

0.5

1000

500

2000

1000

2000

0.0

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K200K

400K

OK

100K

200K

300K

400K

Pit Fall

Pong

PrivateEye

Q*Bert

River Raid

Reward

-16

100

2000

3000

-500

18

1000

2000

20

1000

100K

200K

300K

OK

100K

200K

300K

400K

100K

200K

300K

400K

0

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

RoadRunner

Robotank

Seaquest

Skiing

Solaris

Reward

7.5

1000

20000

15000

2000

10000

500

25000

1000

100K

200K

300K

400K

0.0

OK

100K

200K

300K

400K

0

OK

100K

200K

300K

400K

30000

oK

100K

200K

300K

400K

OK

100K

200K

X00

400K

SpaceInvaders

Star Gunner

Surround

Tennis

Time Pilot

Reward

400

1000

9.0

18

4000

200

500+

750

9.5

-20

2000

Tutankham

100K

200K

300K

400K

250

Up N Down

100K

200K

300K

400K

10.0

OK

100K

Venture

200K

300K

400K

24

VideoPinball

100K

200K

300K

400K

WizardofWor

100K

200K

300K

400K

Reward

100

7500

40

15000

750

50

5000

2500

20

10000

5000

500

250-

OK

YarsRevenge

100K

200K

300K

400K

OK

100K

Zaxxon

200K

300K

400K

100K

# steps

200K

300K

400K

OK

100K

# steps

200K

300K

400K

OK

100K

# steps

200K

300K

400K

Reward

2000

10000

1000

5000

100K

# steps

200K

300K

400K

0

OK

100K

# steps

200K

300K

400K

SUFT

Baseline

Fishing Derby

Zaxxon

Boxing

Gopher

Kangaroo

H.E.R.O.

Tutankham

Star Gunner

Assault

DemonAttack

Name This Game

BeamRider

Seaquest

YarsRevenge

Asterix

Up N Down

Chopper Command

Phoenix

Q*Bert

Robotank

Krull

Bank Heist

Frostbite

Tennis

Enduro

RiverRaid

Space Invaders

Amidar

Centipede

Alien

Breakout

BattleZone

JamesBond

Gravitar

Freeway

Pit Fall

Kung-Fu Master

+10%

+100%

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Log Improvement (%)

SUFT

Baseline

Atari Double DQN Results:

40

Alien

Amidar

Assault

Asterix

Asteroids

Reward

200

1500

1000

100

1000

500

500

600

400

100K200K300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

X000

400K

OK

100K200K300K

400K

OK

100K

200K

300K

400K

Atlantis

Bank Heist

BattleZone

BeamRider

Reward

20000

15000

600-

10000

1000

10000

20

5000

500

200

OK

100K

Bowling

200K

300K

400K

100K

Boxing

200K

300K

400K

0

OK

100K

Breakout

200K

300K

400K

Centipede

100K

200K

400K

Chopper Command

OK

100K

200K

300K

400K

Reward

20

20

4000

1500

2000

25

10

2000

1000

500

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

100K

200K

300K

400K

DemonAttack

Double Dunk

Enduro

Reward

40000

4000

2000

200

20000

3000

1000

23

100

2000

100K

200K

300K

400K

oK

100K

200K

300K

400K

oK

100K

200K

300K

400K

oK

100K

200K

300K

400K

100K

200K

300K

400K

Fishing Derby

Freeway

Frostbite

Gopher

Gravitar

Reward

60

20

600

400

600

200

80

10

200

200

100

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

X000

400K

100K

200K

300K

400K

H.E.R.O.

Ice Hockey

JamesBond

Kangaroo

Krull

Reward

10000

400

4000

4000

005

10

200

2000

2000

Kung-Fu Master

100K

200K

300K

400K

Montezuma'sRevenge

15

100K

200K

400K

0

OK

100K

200K

300K

400K

Name This Game

100K

200K

300K

400K

100K

Phoenix

200K

XOO0

400K

Reward

4000

1.0

1500

4000

2000

2000

0.5

1000

2000

1000

500

0.0

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

X000

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

Pit Fall

Pong

Private Eye

Q*Bert

River Raid

Reward

500

2000

3000

-18

1000

2000

-20

500

1000

100K

200K

300K400K

oK

100K

200K

300K

400K

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

Robotank

Seaquest

Skiing

Solaris

Reward

20000

1000

-20000

2000

10000

2.5

500

25000

1000

100K

200K

300K

400K

OK

100K

200K

300K

400K

0七

OK

100K

200K

300K

400K

30000

oK

100K

200K

300K

400K

OK

100K

200K

300K

400K

SpaceInvaders

Star Gunner

Surround

Tennis

Time Pilot

Reward

400

1000

9.0

18

7500

200

750

500

9.5

22

-20

2500

5000

250

10.0

24

100K

200K

300K

400K

JK

100K

200K

300K

400K

OK

100K

200K

300K

400K

OK

100K

200K

300K

400K

100K

200K

300K

400K

Tutankham

Up N Down

Venture

VideoPinball

WizardofWor

Reward

100

50

5000

60

20

15000

20000

10000

5000

400-

600

200

YarsRevenge

100K

200K

300K

400K

100K

Zaxxon

200K

400K

OK

100K

# steps

200K

300K

400K

OK

100K

# steps

200K

300K

400K

OK

100K

# steps

200K

300K

400K

Reward

10000

150

100

5000

50

100K

#steps

200K

300K

400K

0

OK

100K

# steps

200K

400K

SUFT

Baseline

Atlantis

Phoenix

H.E.R.O.

Seaquest

Assault

BeamRider

Up N Down

Tutankham

NameThisGame

Q*Bert

Chopper Command

Kung-Fu Master

Asterix

StarGunner

Bank Heist

BattleZone

Kangaroo

Enduro

Space Invaders

JamesBond

YarsRevenge

RiverRaid

Krull

Robotank

Gopher

Amidar

Alien

Centipedel

Pong

+10%

+100%

Pit Fall

Ice Hockey

Freeway

Breakout

Frostbite

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Log Improvement (%)

SUFT

Baseline

D.7 MuJoCo Five Environments Benchmark

MuJoCo PPO Results:

Ant

Half Cheetah

Hopper

Reward

2000

1000

1000

1000

-0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.20.4

0.6

0.8

1.0

1e6

1e6

# steps

1e6

Humanoid

Walker2d

Reward

2000

400

1000

200

0.0

0.20.40.6

0.8

1.0

0.0

0.20.40.6

0.8

1.0

# steps

1e6

# steps

1e6

SUFT

Baseline

MuJoCo SAC Results:

Ant

Half Cheetah

Hopper

Reward

2000

7500

2000

AM

5000

1000

2500

1000

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.20.4

0.6

0.8

1.0

1e6

1e6

# steps

1e6

Humanoid

Walker2d

2000

Reward

1000

1000

500

0

0.0

0.20.40.6

0.8

1.0

0.0

0.20.40.6

0.8

1.0

# steps

1e6

# steps

1e6

SUFT

Baseline

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we highlight the main problem, our method, and the empirical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 4, we describe the assumptions of our work, and in particular Assumption 4.1, which does not apply to the L 2 loss. In our experiments (Section 5), we evaluate our method on L 2 loss.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 4, we detailed the relevant assumptions for all theoretical results. We provide all proofs in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use open-source environments and an RL library as detailed in the Implementation Details paragraph in Section 5. Furthermore, we fully detailed all the experimental information in the Training Details paragraph in Section 5. Throughout Section 4 and Appendix B, we thoroughly detail how to implement our method. We also provide all the evaluation formulas in Appendix D. In addition, we also provide all of the code as supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code as supplementary material and fully detail the implementation and training details in Section 5. We also provide all the evaluation formulas in Appendix D.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We fully detailed all the experimental information in the Training Details and Implementation Details paragraphs in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are independently trained for 10 different random seeds per agent-environment configuration, and we take the median result. We report mean performances for all of the environments per domain. We also report environment-specific statistical significance results by conducting a t -test.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We give details on the computational time and resources in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper proposes a method to improve sample efficiency in reinforcement learning. As foundational research, this work has no direct paths to negative societal impacts besides those that can arise from improved reinforcement learning agents.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release high-risk data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all creators whose code we used in our experiments in Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the code as supplementary material and include a README file that explains how to run it.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper uses LLMs only for editing purposes and conforms to the NeurIPS LLM policy.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a novel theoretical result according to which the number of training steps and the experience replay buffer in deep reinforcement learning (DRL) can be significantly reduced.\nThis is very interesting, given the fact that most of DRL papers present ad hoc heuristics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe main strength of the paper lies at its theoretical scope, on such an important topic.\n\nIn more detail:\n\nThe paper introduces a new causal upper bound on the factual loss in reinforcement learning, adapting the Neyman-Rubin potential outcomes framework in a nice way.\nThis conceptual contribution might open new avenues for bridging on-policy and off-policy methods.\n\nThe SUFT OPE term that the authors introduce, leverages computed value network outputs by recycling them in a \"replay buffer.\"\nThis clever reuse of data leads to improvements in sample complexity and buffer size requirements, while achieving good rewards.\n\nThe method the authors introduce is evaluated across a large and diverse set of environments and agents, showing statistically good performance.\n\nWeaknesses:\n\nWhile the causal bound is theoretically novel, it would be nice if the authors could comment more on its tightness and generalizability.\nMoreover, it would be nice to have more comparisons to existing theoretical bounds in OPE :)\n\nAlso, please see Questions below.\n\nMy Impression:\nThe paper merits acceptance :)\n\nQUESTIONS:\nPage 1:\nCan you please elaborate a bit on the comparison between on and off policy agents? :)\n\nPage 2:\nPlease elaborate on the caption of Figure 1.\n\nPage 4:\nIn Preliminaries, can you please also add some general RL background?\n\nPage 5:\nCan you please comment on Assumption 4.1?\n\nPage 6:\nWhy is $a$ included in the arguments of $\\ell$?\nCould it be defined on $s$ and $t_{\\rm target}$ only?\n\nPage 7:\nCan you please further explain the ramifications of Theorem 4.10? :)\n\nPage 8:\nThe experimental setup looks OK!\nFigure 3 needs more comments though, in the caption.\n\nPage 9:\nPlease expand the Conclusion section and add more future work directions.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a causal bound on the factual loss, derived from the potential outcomes framework, to enhance DRL sample efficiency. The method yields substantially higher rewards while drastically reducing the required size of the experience replay buffer at a negligible cost.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Leveraging causal information to enhance sample efficiency in DRL is a significant research direction.\n- The experimental results are extensive and validated across various standard benchmarks.\n- Theoretical results are provided to ground the proposed approach and verify its effectiveness.\n\n**Weaknesses:**\n- The paper is **not well-structured**, which negatively impacts readability. For example, the term \"SUFT\" should be explained when it first appears, not pages later. Similarly, the explanation for Figure 1 is located far from the figure itself (Figure on pp. 2, explanation on pp. 9). Meanwhile, the introduction lacks detail on how the causal bound is incorporated into the DRL framework. It also fails to provide sufficient insight into why adding the proposed term to the loss function yields such significant improvements over conventional methods.\n- Several figures and equations **require formatting adjustments**. For instance, the font size in Figures 4 and 10 is too large, and the equation in line 752 is too long and runs outside the margin. Pages 5 and 6 are overly dense with definitions, many of which could be moved to an appendix. This space in the main body would be better used to detail why the challenge is difficult for previous work to handle and to provide more intuition about the proposed approach.\n- **Theory-Experiment Disconnect**. The main theoretical result (Theorem 4.10) relies on Assumption 4.1, which the authors admit does not hold for the L2 loss—the loss used in their main experiments. This undermines the validity of their theoretical justification. Without the assumption, the proposed method reduces to a heuristic regularization technique, and the claim of optimizing a causal bound does not hold. The brief L1-loss results in the appendix do not adequately support the main claims. What's the main difficulty of providing the theoretical result for the L2 loss?\n- **Fragile and Potentially Misleading Metric**. The reported “2,427% improvement” is based on a reward ratio formula, i.e., (SUFT - Random) / (Baseline - Random) in line 755, which is highly sensitive *when the baseline performs only slightly better than random*. This can produce arbitrarily large values, exaggerating the method’s effectiveness and obscuring its true impact compared to more standard metrics like absolute or human-normalized scores. The soundness of the used metric should be discussed.\n\nQUESTIONS:\n- What is the main difficulty in establishing the theoretical result (Theorem 4.10) for the L2 loss?\n- Could you discuss the soundness of the metric used? Specifically, when the baseline performs only marginally better than a random guess, how can the effectiveness of the ratio be guaranteed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new algorithm for improving the sample efficiency and performance of DRL agents. The algorithm is based on a theoretical result derived from the causal inference literature. Unlike most work that bounds the counterfactual (off-policy) loss, the authors here propose bounding the unobserved factual (on-policy) loss instead, using off-policy samples. This bound is made of the usual off-policy loss plus a term that measures the treatment effect between the target policy and past policies. In practice, the authors suggest to cleverly recycle the value network outputs, which are normally just thrown away after action selection, and instead store them in the replay buffer. The authors conducts extensive experiments on DQN, PPO, and SAC agents across many Atari and MuJoCo environments, demonstrating clear improvements in reward and convergence speed, and a significant reduction in the required replay buffer size.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The paper tackles the crucial problem of sample efficiency in DRL. The method is surprisingly simple, can be used orthogonally with many other algorithms, and its empirical results are exceptionally strong across many different agents and benchmarks. If these results hold up, the method's ability to improve performance and cut down on resource needs would be a major step for the field.\n\n* I also found the main theoretical idea of looking at the problem as a causal bound on the factual loss, instead of the counterfactual loss, to be really original.\n\n* The experimental validation is high quality and very thorough. Using 10 random seeds, all 57 Atari games and 5 MuJoCo environments, with several different DRL agents (DQN, PPO, SAC) really convincingly show the robustness of the advantage of the algorithm.\n\n# Weaknesses\n\n* I really think the paper needs some pseudocode for how to apply the bound to DQN and PPO. Currently, the reader has to guess some of the key implementation details. For instance, it is not clear to me how the existing PPO critique loss would map to the proposed upper bound.\n\n* There's a bit of a gap between the formal theory and the experiments. The proof for the causal bound needs an L1 loss, while the main experiments with the most impressive results are all using an L2 loss. The authors acknowledge this, but this raises the question of why the method is so effective with L2 loss, e.g. it is possible that the additional term act as a regularizer that promotes stability, which benefits the optimization independently of the bound itself\n\nQUESTIONS:\n* The bound has the  $\\delta$ term which is correctly ignored during optimization, since it doesn't depend on the agent's parameters. However it would be good to add a short discussion on potential looseness of the bound. If  $\\delta$ is very large, the bound might be too loose to be useful. Your strong results suggest this isn't a problem, but it would be good to comment on it. In particular, it would be interesting to plot the magnitude of this term over the optimization.\n\n* Coming back to my last point in the \"Weaknesses\": the SUFT loss term, $(V_{behavior} − V_{target} )^2$ , is functionally very similar to known regularization techniques that performs some form of value function smoothing (e.g., in PPO v2). To isolate the effect of your proposed causal mechanism, would it make sense adding a control experiment? For example, it would be interesting to check whether the benefit comes from simply adding inertia to the value function, preventing it from changing too rapidly, rather than tracking a specific behavioral policy's value. One could e.g. add a term ot the TD loss, $L_{total} = L_{TD} + \\lambda_{inertia} \\cdot (Q_{current} - Q_{target})^2$ where $Q_{target}$ is the slowly moving target Q network used to compute the target value in the TD loss, as is usually done in DQN.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a causal approach to an upper bound on-policy loss in RL with off-policy data. The resulting upper bound can be integrated into any on-policy RL algorithm. The authors use empirical evaluations to show that the resulting SUFT version of RL algorithms can achieve better performance with low sample/computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* The paper is overall well-written and easy to follow.\n* The paper is well-motivated, addressing the long-standing challenge in reinforcement learning of using inexpensive off-policy data in place of costly on-policy data.\n* The authors introduce tools from causal inference to RL. The proposed methodology looks natural and intuitive.\n* The empirical evaluations look sound.\n\nWeaknesses:\n* The empirical evaluation introduces a metric termed the reward improvement ratio. This metric appears unconventional and may potentially overstate the improvements achieved by the proposed method.\n* I think the author should try more recent baselines on Atari games and see whether the proposed method works for them. For example the baselines in https://arxiv.org/abs/2112.04145.\n\nSince I am not very familiar with the literature on causal inference, I choose to assign a low confidence score to my evaluation.\n\nQUESTIONS:\nSince Q-learning is inherently an off-policy method, the arguments presented in Section 4.1 are unclear to me. In particular, it is surprising that the SUFT variant leads to such a substantial performance improvement. Could the authors clarify this discrepancy and provide further explanation?\n\nThe theoretical analysis relies on a form of the triangle inequality for the loss function. While the $L_1$ loss satisfies this assumption, the more commonly used $L_2$ loss does not. Nevertheless, SUFT appears to perform even better under the $L_2$ loss in the empirical results, which seems inconsistent with the theoretical framework. I encourage the authors to provide intuitive explanations for this observation.\n\nLIMITATIONS:\nThe authors do not explicitly state the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}