Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
Authors: Tal Fiskus
Uri Shaham
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Causal Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: This paper presents SUFT, a causal upper-bound loss optimization strategy for DRL designed to enhance sample efficiency and reduce computational demands.

Abstract: Deep reinforcement learning (DRL) agents excel in solving complex decision-making tasks across various domains.
However, they often require a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands.
To address these challenges, we introduce a novel theoretical result that leverages the Neyman-Rubin potential outcomes framework into DRL.
Unlike most methods that focus on bounding the counterfactual loss, we establish a causal bound on the factual loss, which is analogous to the on-policy loss in DRL.
This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded.
Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve *up to 383%* higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by *up to 96%*, significantly improving *sample efficiency at a negligible cost*.

Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
Tal Fiskus
Uri Shaham
Department of Computer Science
Department of Computer Science
Bar-Ilan University
Bar-Ilan University
Ramat Gan, Israel f fiskust@biu.c ac.
Ramat Gan, Israel
uri www.sh@biu.uc..mmmmmmmmmm ac.
Abstract
Deep reinforcement learning (DRL) agents excel in solving complex decision- making tasks across various domains. However, they often require a substantial number of training steps and vast experience replay buffer, leading significant computational and resource demands. To address these challenges, we introduce novel theoretical result that leverages the Deyman-Rubin potential outcomes framework into DRL. Unlike most methods that focus on bounding the counter- factual loss, we establish causal bound on the factual loss, which is analogous the on-policy loss in DRL. This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded. Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve up 383% higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by up to 96%, significantly improving sample efficiency at a negligible cost
Introduction
Deep reinforcement learning agents have demonstrated remarkable success in solving complex decision-makee tasks across various areas of interest, such as gaming [[H5]. robotics (6-8), health- care 1 and autonomous driving [2+14.. It also has a vital role in enhancing large language models 15 16]. Despite these achievements, DRL is still challenging due its high computational and resource demands. For example, AlphaGo Zero was trained for 29 million self-play games [17] and the training experience for OpenAl's Rubik's Cube robotic hand is roughly equated to 13 thousand years 18]
In DRI,, we can categorize agents into two types: on-policy agents, like A3C and PPO 19 20 and off-policy agents, such as DQN and SAC 21 22 On-policy agents are generally simpler and often considered first. They interact with the environment, collect experiences using behavior policy, and once they optimize the target policy, discard all previously collected experiences to ensure that the behavior and target policies remain identical. This consistency leads to more stable convergence but at the cost of sample inefficiency, as each optimization iteration requires collecting new data. Off Off-policy agents, the other hand, allow the agent learn from previously collected experiences, meaning can have different behavior and target policies. This methodology improves sample efficiency by leveraging past experiences stored in a replay buffer. While this improves sample efficiency, it can also lead to reduced learning stability due to the mismatch between the behavior and target policies. As result, improving the learning stability typically requires a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands.
Our code is available at h/tttts://shaaaa-aaalt github...h....... woo./urnninnganntooollllddd
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Domon At
Derro Mlacs
King
R5/00
lran kuno
boo 500
Kanto Thx Damp
Cuapper munt
Eang Ru
Up
Banc 4
Gamee
Sparn 20d1
Ioe lidday
(amez
Nonesg
Bange
Sul
Rong
Sin
Road
Basro
AM
ve
+10%
+100%
Tank M400
4100%
Arccar
95
20
1.5
20
25
32
0.0
0.5
15
15
2.0
25
log Reward Improvement (363
log Reward Improwement 196)
SUFT
Baseline
Figure 1: Log- Lcalscaled reward improvements comparison between agents using the additional SUFT OPE term and the baseline agents without it across 57 Atari games. The results demonstrate the superior performance of our method across the majority of games. The red line indicates a 10% improvement, and the green line represents 100% improvement. Left: Double DQN SUFT outperforms the baseline agent 35 out of the valid games; Right: PPO SUFT outperforms the baseline agent in 39 out of the 42 valid games.
To mitigate these challenges, researchers have proposed numerous strategies designed to reduce computational demands and improve learning stability. A widely adopted approach focuses on enhancing the experience replay buffer sampling, aiming to improve convergence rates by optimizing the selection of experiences (23 While this approach improves convergence, it still involves high computational resources and does not fully address the divergence between on-policy and off-policy methods. Other works have attempted to bridge the divergence by suggesting additional computable terms, c.g., [27]. Although such approaches have improved learning stability, they often require extensive data collection and involve complex techniques.
This work addresses these limitations by introducing novel theoretical result in causal inference and incorporating it into the DRL framework. Specifically, we establish causal upper bound on the unobserved factual loss using the observed counterfactual loss and the estimated treatment effect. Within the DRL framework, we incorporate this as an upper bound for the on- on-policy loss using the standard off-policy loss and an off-policy evaluation (OPE) term, which quantifies the estimated value discrepancy between the behavior policy and the target policy. By incorporating this term into the optimization process, our method explicitly accounts for the causal estimated treatment effect difference between the target policy and the behavior policy. This causal perspective enhances the agent's 's ability to reason about the differences between the target policy and the behavior policy used to generate the experiences, thereby mitigating the fundamental divergence between on-policy and off-policy learning. Importantly, our method reuses existing data of the behavior policy network values that are usually discarded to improve sample efficiency negligible cost.
We evaluate our proposed method, SUFT on existing off off-policy agents such as DQN and SAC, as well as on-policy agents like PPO, which benefit from our method due to the deviation between behavior and target policies during optimization. Throughout comprehensive experiments on the Atari 2600 and MuJoCo domains, agents using the SUFT OPE term outperform the same ones without it. Notably, we achieve up to 383% higher reward ratio, as shown in Figure 2 indicating an improvement in the training convergence rate compared an identical agent without the additional term. addition, we reduce the experience replay buffer size by up to 96% while outperforming the baseline agent that uses 25 times larger buffer by up to 437.05% These significant improvements highlight the potential of our method to make DRL more computationally efficient and accessible.
We name this novel theoretical result the SUFT causal bound, representing the initials of Sample efficient, Upper bound, Factual loss, and Treatment effect.
2

---

450
MujoCo
400
383.19%
Mean Reward Rabio (%)
500
228.12%
226.91%
224.53%
197.34%
200
150
111.21%
50
DON
Vanilla DQN
Double DQN
PPO
SAC
PPO
SUFT
Baseline
Figure 2: Mean reward ratio comparison between agents using the additional SUFT OPE term and the baseline agents without across 57 Atari games and five MuJoCo environments, highlighting the profound reward gains across diverse agents and domains.
2 Related Work
2.1 Causal Reinforcement Learning
Causal reinforcement learning (CRL) is an emerging subfield of reinforcement learning (RL). CRL seeks to enhance agents' performance by incorporating causal inference methodologies into the learning process. The causal perspective allows the agent to learn cause-and-effect knowledge,
resulting more informed and effective decision-making [28 (9].
In causal inference, a treatment typically refers to performing a single action, commonly termed as performing an intervention 30 31. On the other hand, the majority of works in CRL define treatment as a sequence of actions determined by dynamic treatment plan 35 In this work, we adopt the latter approach and consider the treatment to be the agent's 's policy. This aligns with the RL framework, which focuses on evaluating the long- sengterm effects of policies rather than the immediate impact of individual actions. Recent research CRL covers variety areas, such as advancing generalizability (36), promoting safe agents [37 38), enhancing sample efficiency [39/41], and addressing spurious correlations [42 Our work falls in the field of CRI, and primarily contributes to the latter two directions.
CRL for Addressing Spurious Correlations: Spurious correlations off-policy RL often arise as result of learning based on experiences from old behavior policy. These experiences might not accurately reflect the states and actions the target policy likely to encounter, resulting in deceptive associations between actions and outcomes that limit the agent's ability to make reliable decisions. Several works leverage OPE as a systematic way to evaluate the quality of policy using oof-policy data, which helps mitigate the impact of spurious correlations during policy optimization 45 46)
CRL for Addressing Sample Inefficiency: Causality provides valuable perspective creating sample-efficient agents, with research focusing on three key directions: representation learning 49,, directed exploration [1), and data augmentation [52] (33. While many works improve data, method utilizes existing discarded data from past experiences, achieving sample efficiency at a negligible cost. This novel strategy sets our work apart, demonstrating that DRL agents can achieve sample efficiency with marginal computational overhead.
sample efficiency by generating
2.2 Potential Outcomes Causal Bounds
The Neyman-Ruuhe Rubin potential outcomes framework is cornerstone causal inference, providing robust theoretical basis for estimating treatment effects. Several works have utilized this framework to develop causal bounds [55] [56]. For instance, Shalit et al. [57 establish generalization-eroo
bound to estimate individual treatment effects from observational data, addressing unobserved

---

counterfactuals by formalizing a lemma that bounds this loss. Our work expands on this foundational framework and introduces several key changes.
First, while most works apply these bounds within general machine learning (MI.) settings 158, 59 redefine the terms to align with the DRL framework. Second, while other studies consider treatment as a single intervention [60] [1], we address treatments as sequence of actions within treatment plan. The primary and novel aspect of our work lies in reversing the traditional loss- bounding methodologies. Whereas existing methods typically bound the counterfactual loss [2] 63 our approach does the exact opposite and bounds the factual loss. This pivotal shift is feasible within the RL framework since off-policy methods use counterfactual data while factual data remains unobserved. Finally. whereas other approaches mainly focus on the scenario of a single target treatment and single control treatment [64] 65], we scale this scenario to a single target treatment and N control treatments. This multiple control treatments scenario allows us to leverage the replay buffer, which contains experiences generated from N behavior policies, enabling effective evaluation of the target policy against a broader spectrum of previous behavior policies.
3 Preliminaries
In this section, we present some fundamental conceptual ideas necessary to understand our method.
Causal and Counterfactual Reasoning: Causal reasoning refers to the process of identifying and understanding the relationships between causes and their effects. Counterfactual reasoning 166), subfield of causal reasoning, goes step further by imagining alternative scenarios to answer "what-if" questions, such as "What would have happened a different treatment had been applied?" as Pearl & Mackenzie [67] elaborate in "The Book of Why". This form of reasoning enables the evaluation of potential outcomes conditioned under different treatments. In RL, incorporating counterfactual reasoning enhances agents' capabilities to evaluate and optimize policies in complex, dynamic environments. By retrospectively analyzing alternative policies and their outcomes, agents can improve decision-making and policy optimization. In this paper, counterfactual reasoning is implemented through the integration of a causal bound into the agent's loss function.
Off-Policy Evaluation: Off-policy evaluation is fundamental concept in RL 168), which aims to numerically evaluate the performance of a desired policy π using historical data that was collected under different policies. As outlined in [69]. the desired policy being optimized is referred to as the target policy, while the policies used to generate experiences are termed the behavior policies. OPE is particularly important in offline RL, where the agent learns the target policy solely from static dataset D without interacting with the environment. In contrast, this work focuses on applying OPE to online DRL agents, which continuously interact with the environment while updating the experience replay buffer D.
4
SUFT Causal Bound
In this section, we formalize our theoretical causal result and detail our method.
4.1 Rationale
To illustrate our method and provide general intuition, we begin with a straightforward example of greedy off-polisy DQN agent. This agent observes the current state of the environment, uses its Q-network to compute Q-values, and selects the action with the highest Q-value for interacting with the environment. As the agent continuously interacts with the environment, it stores the experiences in replay buffer and periodically updates its Q-network, which determines its target policy.
During Q-network updates, an ideal scenario would be to replace all experiences with fresh samples generated directly from the target policy without any additional cost. This would preserve the sample efficiency from the off-policy approach while resolving the divergence between the behavior and target policies. However, since such scenario is impractical, we introduce a novel mathematical result and adapt it the DRL framework to bridge between on-policy and off- off-policy methods.

---

Within the causal framework, the on poliicy loss corresponds to the factual loss when the treaiment being optimized matches the treatment applied for generating the observations. This is analogous to optimizing the target policy using experiences from an identical behavior policy. Similarly, the off- off-policy loss aligns with the counterfactual loss when the treatment being optimized differs from the one used to generate the observations. This is equívalent to optimizing the target policy using experiences from a different behavior policy.
In this work, we establish a causal upper bound on the unobserved factual loss using the observed counterfactual loss and an additional term, which is the estimated treatment effect of the target treatment compared to N control treatments. After formalizing the upper bound within the causal framework, we seamlessly integrate it into the DRL framework as an upper bound for the on-policy loss using the standard off-pooliy loss and the estimated treatment effect, which serves as an OPE.
Throughout this paper, we demonstrate the causal definitions alongside corresponding DQN defini- tions to provide a clear and intuitive representation of the reduction from the causal framework to the DRL framework. Nevertheless, our method is applicable to any DRL agent with a V or Q-value network that influences its policy, such as DQN and Actor-Critic architectures, as long as a divergence exists between the target and behavior policies.
4.2 Setup and Definitions
Here, we present the necessary mathematical definitions required to formalize our causal upper bound. Comprehensive definitions and proofs, along with detailed reduction from the causal framework to the DRL framework, are provided in Appendix B
To simplify the equations, we present the binary treatment case, where there is a single target treatment, denoted by t = I, and a single control treatment, denoted by t = 0. In practical DRL applications, extend the binary case to multiple control treatment case involving single target policy and N behavior policies, reflecting the existence of experiences generated from different behavior policies in the replay buffer. The proofs and definitions for the multiple control treatment case are in Appendix
We employ the Neyman- Rubin potential outcomes framework and define the following: X' C Rd the observation space. y C R the outcome space. T = {0, 1} is the binary treatment space. Under this framework, there are two potential outcomes depending on the treatment assignment: if O, the observed outcome y = Yo. We assume
I == 1, the observed outcome is y = Yi, and there exists a joint distribution P(X, T, Yo. Y1) with the following components: q := Pr(T = t) is the treatment probability. P == P(X = t) is the conditional distribution of observations. P7 := PYY|X 2) is the conditional distribution of potential outcomes given the target treatment, and PT. := P(YoXX x) is the conditional distribution of potential outcomes given the control treatment. We also assume that there are no hidden confounders the observational data, which makes the estimated treatment effect identifiable. This is formalized by assuming that the standard strong ignorability condition holds: (Yo, Y1) TL T, and 0 < p(t = 1|z) < 1 for all æ. Strong ignorability is sufficient condition for the estimated treatment effecc to be identifiable [70] 57). To integrate the potential outcomes framework into the MI. context, we define a neural network p(x;; 0.): X + V/ parameterized by weights 0g corresponding to the performed treatment t E T.
We now present the analogous definitions within the DRL framework for a DQN agent, assuming the DRL framework is defined as a Markov decision process (MDP).
X = (S x A) is the state nate-action space. Potential outcome y (s.a) is the accumulated reward over n time steps, starting from state 8, taking the initial action a, and following policy тe. The hypothesis ((x; Ob) Q(s, a; 6.) (8 A) - y is the Q-values network, which influences the policy ne.
Let L E + Rzo represent a loss function. We rely on the following inequality assumption, which is a variation of the inverse triangle inequality:
Assumption 4.1.
L(x, y) L(x' ./) < L(x, + L(y.v')
5

---

The proof for the L1 loss is in Appendix
Now, define the expected factual and counterfactual losses, which are used formulate the bound. Definition 4.2. The expected outcome loss:
(a(2, t) == EympE [L (.)))
The corresponding definition in DQN: Definition 4.3. The expected temporal difference loss:
as := E yo L (v.Q(s.a;
Note that the DRL framework, the expected outcome loss refers to the temporal difference loss between the accumulated rewards and the Q-value estimation. Definition 4.4. The expected factual outcome loss:
1 == E 1 L 1)]
EP
CF. E px ( 0)].
0
eF == F go
To formalize the corresponding definition in the DRL framework, consider a replay buffer D1 containing samples produced by behavior policy that matches the target policy. In this context, the on-policy loss is a special case of the expected factual loss, and for DQN, it is defined as: Definition 4.5. The expected on-policy loss:
6On Policy (s,a) (&Q a, turgee)]
Note that this loss is the temporal difference loss with experiences collected on-policy, with identical target and behavior policies from Dj. Definition 4.6. The expected counterfactual outcome loss:
&CF, = E. p2 (x; 1)] CCF, = Bynpy (z, 0)].
ECF, = goror, + 91 c
To establish the corresponding definition within the DRL framework, consider replay buffer Do that stores experiences generated by a behavior policy that differs from the target policy. In this context, the off off-policy loss is a special case of the expected counterfactual loss, and for DQN, is defined as: Definition 4.7. The expected off-policy loss:
COrt-Policya Ess leo a, tnaget).
Note that this loss is the temporal difference loss with experiences gathered off off-policy, with different target and behavior policies from Do-
Additionally, in DRL applications, consider the scenario of replay buffer containing experiences generated from N distinct behavior policies. The definitions for this case are in Appendix B
We now define our method's 's additional estimated treatment effect term in both the causal and the DRL frameworks:
Definition 4.8. The estimated treatment effect loss:
15 En [(o(;; (0), cbo;; 01))]-
$ = E,- pt L( 0(1;
1b 9o + q v.
'DRL agents are often trained using La loss, for which Assumption 4.1 does hold. While the assumption
needed to prove Theorem 4.10 similar experimental results were obtained with both Lı and Le losses. To provide intuition for the observed La performance, empirically examined the validity of Assumption 4.] using L2. This analysis is detailed in Appendix D

---

The interpretation for the estimated treatment effect in DRL is an OPE, and for DQN, it is defined as: Definition 4.9. The SUFT OPE term:
VsurT == Da [L (Q(s, (s,,00 0; be Q(s, 0; murget)
This term aligns with the principles of OPE, focusing on quantifying the discrepancy between the target policy and the behavior policy using off-policy data. Once the data is stored in the buffer, used observational data.
4.3 Causal Bound Theorem
Having established the key definitions, we now present our novel factual loss upper bound theorem: Theorem 4.10. The expected factual outcome loss, CFa is bounded by the expected counterfactual outcome loss, cCF the estimated treatment effect loss, Va. and a constant term 6, independent of o:
FF. S ECF, + 20 +
The proof appears in Appendix
The interpretation of Theorem 4.10 loss, by the off ofo-po loss, the SUFT OPE term, and a constant term 6. A certain difference exists in the observational data between the causal and the DRL framework, which detailed Appendix
4.100em/4.10mmm in the DRL framework establishes an upper bound to the on-policy
COn COn-Policy Con + U.SUFT + 5.
Note that since 8 is independent of Q, can he omitted during the gradient-based optimization Q.
To the best of our knowledge, we are the first to introduce such an upper bound within the causal framework on the factual loss, adapting it to the DRL framework, thereby bounding the on-policy loss. Importantly, based on our causal bound theorem, the adaptation to the DRL framework is not constrained to DQN agents but can be applied to every off-policy agent with V or Q-value networks that influence its policy. This scenario is thoroughly detailed in Appendix B Furthermore, our method also applicable to on-policy agents, such as PPO, which benefits from it due to the divergence between the behavior and target policies that arises during epoch optimization.
4.4 Recycling Value Network Outputs
To compute the SUFT OPE term in the causal bound, we need to calculate the Q-network values corresponding to both the target and the behavior policies. While the Q-values for the target policy can easily computed using the current Q-network, the Q-network values for the old behavior policies are not addressable since storing the network weights for every behavior policy is impractical. Notably, during the agent's 's training process, the Q-values matching the behavior policies are already calculated when they are used to select the action to perform. These values calculated for the purpose of exploitation and then discarded once the action is executed. The novel contribution of our approach lies in utilizing and reusing the value network outputs in order compute the SUFT OPE term. This allows the reuse of data that is overlooked while enhancing the loss optimization with significant causal insights in the form of the estimated treatment effect. Our approach effectively transforms overlooked data into valuable causal insights, metaphorically turning sand data into gold.
To obtain value network outputs in practice, store them in the experience replay buffer. For agents employing a Q-network, only a single Q-value corresponding to the behavior policy stored, resulting in experience tuple of the form (s, s', Q(s, a; Obchavior) )). instead of the standard (s, a, s'). For agents using a V-network, the state value output is stored instead, yielding an experience tuple of the form (s, a, s', Vss; bbchavior).. instead of the standard (s, as,
4.5 Universal Implementation of the SUFT OPE Term
Following the formalization of the SUFT causal bound, in Section 4.3 we adopt the well-established approach of optimizing a bound on the loss, technique commonly employed in algorithms like iopectationmmaxiiizan [7T] and variational inference [72]. Consequently. rather than directly optimizing the on-policy loss, focus optimizing the SUFT causal bound. In addition, to improve
7

---

the flexibility and accuracy of the loss optimization, we incopprate a ATF coefficient into the SUFT OPE additional term. In the case of DQN, the SUFT causal bound objective term as follows:
Con q + XTF USUFTO
In value-based methods such as DQN agents, the SUFT OPE term is seamlessly added to the standard loss function, while in CrittcC architectures, it is incorporated into the critic's loss. We provide a
detailed pseudocode of our method algorithms for both DQN and PPO in Appendix C
Experiments
In this section, provide comprehensive empirical analysis of our proposed method, emphasizing the impact of adding the SUFT OPE term to the agent's 's standard loss function across a broad and diverse range of environments. In this section, we present results obtained using L2 loss, allowing us to compare our method to standard DRL agents. The results for L1 loss are detailed in Appendix D
5.1 Experimental Setup
We assess our method's performance across two distinct domains: The first consists of 57 Atari games with a discrete action space, which contain a range of decision makinn scenarios, including immediate versus long-term strategies, varying image background complexities, and dynamic on-screen objects. The second includes five MuJoCo environments that simulate robotic motions with a continuous action space that provides robust platform for dynamic physical interactions.
In the Atari domain, we perform a wide range of experiments using DQN (11]. which serves as the foundational algorithm for numerous Q-value network agents, Vanilla DQN, and Double DQN [73]. as well as PPO [20], which is a widely adopted agent that has become a standard in the field. The Vanilla DON results are detailed in Appendix D For the MuloCo domain, we evaluate our method using SAC [22]. a continuous control off aff-policy actor-critic agent known for its robust performance, and with PPO. Across these settings, we explore different values of Xrr, resulting in more than 10K individual experiments. This extensive and versatile experimentation ensures statistically significant findings, providing confidence in the robustness of the results and enabling us to draw more reliable conclusions about our method's performance. More details about tuning ÀTF are in Appendix D
Training Details:
To evaluate SUFT's sample efficiency, we conduct experiments with constrained environment interactions. For the off-policy agents, we use a relatively small experience replay buffer size of 4K. Each agent configuration is independently trained 10 times per environment, with the training process limited to 400K steps in the Atari domain and 1M steps in the MuJoCo domain. Performances are analyzed by taking the median smoothed training reward from the 10-trial set, ensuring a more reliable evaluation while mitigating the influence of randomness on the results. This constrained resources experimentation is designed demonstrate method's sample efficiency, and it a common technique in DRL. 74 Results for a larger buffer size are in Appendix D
Implementation Details: We the Gymnasium environments API 75 to train on the Atari 76 (version v5) and MuJoCo (77) (version v4) domains. For training the agents, we are using the default architecture and hyperparameter configurations implemented in the DRL library (78). In the Atari games, we apply standard preprocessing, which includes resizing observations 84x84 grayscale images and stacking four consecutive frames. This preprocessing implemented using the library's 's default Atari wrapper.
5.2 Controlled Comparison with Baseline Agents
We evaluate the impact of incorporating the SUFT OPE term by comparing agents that include our proposed term to baseline agents without it, ensuring both types share identical architecture and hyperparameters, where the presence of our proposed term is the only difference between them. This controlled experiment is designed to isolate the influence of our method, ensuring fair comparisons.
In addition, we conduct l-test for each environment against two-sided alternative using 10 different random seed runs, where the null hypothesis is the baseline result. The p-values provide robust statistical significance results for evaluating the effectiveness of our proposed method.
8

---

Assault
Boxing
Demon Attack
James Bond
2005
Rewward
3000
1560
250
-05
0000
Assault
Crazy Climber
James Bond
Kangaroo
3400
Rewarrd
30000
13993
13K%
2008
2006
4654
1004
3006
2006
400
300%
2006
300
400%
130%
2004
3000
4000
0 steps
8 Steps
stepss
17 /steps
SUFT
Baseline
Figure 3: Learning curves comparison between agents using the SUFT OPE and the baseline agents without it across selected Atari games. The red line indicates human-level performance, showing that SUFT not only surpasses the baseline agent but can even exceed human rewards. Top: Double DQN; Bottom: PPO.
The complete experimental results, the evaluation formulas, and more empirical analysis of our method are detailed in Appendix D
Atari: On the Atari benchmark, SUFT demonstrates significant improvements for of the agents. On the DQN, Double DQN, and PPO agents, SUFT surpasses the baseline agent in 52, 48, and 50 games out the 57 games, respectively, with statistically significant gains (p-value 0.05) in 49, 37, and 40 games.
We also analyze the results for the valid Atari games (according the evaluation formula Ap- pendix D). For the DQN agent, SUFT outperforms the baseline agent in 28 out of the 29 valid games, achieving reward improvement of over 100% in 48.3% of them and over 10% in 93. 1 1% of them, as shown in Figure II For the Double DQN agent, SUFT outperforms the baseline agent in 35 out of the 40 valid games, achieving a reward improvement of over 100% in 20% of them and over 10% in 72.5% of them, as shown in Figure (Left. For the PPO agent, SUFT surpasses the baseline agent in 39 out of the 42 valid games, achieving a reward improvement of over 100% 38.15 1% of them and over 10% in 83.3% of them, as shown in Figure [ (Right).
Additionally, when evaluating the mean reward ratio, SUFT demonstrates profound gains relative the baseline agents. For the DQN, Vanilla DQN, Double DQN, and PPO agents, SUFT achieves mean reward ratio of 383.19% 228. 12%, 226.91 226.91% and 197.34%, respectively, as shown in Figure
Moreover, in several Atari games, SUFT not only surpasses the baseline agent but also achieves higher rewards than average human, shown in Figure 3 This achievement highlights the potential of our method to boost agents' performance even above the human level, while keeping constrained environment interactions and resource demands. The human rewards sourced from 779
MuJoCo: For the MuJoCo benchmark, SUFT demonstrates improvements in the SAC and PPO of 5 environments, with
agents. For the SAC agent, SUFT surpasses the baseline agent statistically significant gains in 3 environments. For the PPO agent, SUFT surpasses the baseline agent out of environments (in the other environment, the score is the same), with statistically significant gains in 2 environments. With regard to the mean reward ratio, SUFT achieves 224.53% for the SAC agent and 111. 11.211% % for the PPO agent, as shown in Figure
5.3 Buffer Size Reduction
We also evaluate the sample efficiency of SUFT by comparing its performance to an identical DQN agent utilizing a larger 100K experience replay buffer size. Remarkably, SUFT achieves mean reward ratio gain of 33,005, outperforming identical agent that uses 25 times larger buffer size.

---

Table 1: An ablation study on the DQN agent comparing the SUFT additional term impact against enlarging the buffer size, adding Vanilla DQN, and adding Double DQN. The DQN, Vanilla DQN, Double DQN, and SUFT DQN use a 4K buffer size. All agents are using L2 loss.
Agent
Human Normalized Mean (%)
DQN
25.04%
DQN, buffer 100K
29.29%
Vanilla DQN
41.26%
Double DQN
42.31 %
SUFT DQN
63.45% %
These results underscore the potential of our approach to obtain high-performance DRL agents with significantly smaller buffers, paving the way for more computationally efficient and accessible DRL. methods by enhancing sample efficiency at a negligible cost.
5.4 Ablation Study
We conduct ablation study to isolate the contribution of the SUFT OPE term to DQN performance. As shown Table we compare SUFT DQN against several alternatives: increasing the buffer size, Vanilla DQN, and a Double DQN. While enlarging the buffer or adopting more complex DQN variants leads to improvements, incorporating the SUFT OPE term yields the highest Human Normalized Mean of 63.45%, %, outperforming all other baselines that require higher resources and computational costs. These results highlight the potential of our method improve the agent's 's performance by enhancing sample efficiency negligible cost, providing simple yet highly effective method for improving agent performance at negligible cost.
5.5
Causal Bound Optimization Inertia Regularizer
To isolate the effect of our causal mechanism and ensure that the significant improvements of our method stem from the causal bound optimization rather than merely from a regularization effect, we conduct a controlled experiment. Specifically, we evaluate double DQN agent that adds an inertia regularizer the standard temporal difference loss function the form of:
(Quurentt network Q target network)
Where Quagett network denotes the slowly updated target network, done in standard double DQN. The experiment uses the same configurations as both the SUFT double DQN and the baseline double DQN, with the additional regularization term being the only difference from the baseline agent. The results in Table show that the inertia regularization significantly degrades double DQN performance. Resulting gains from our method, which result in a 226.91% mean reward ratio improvement, arise from the causal bound optimization, where the old Q-network values used for the OPE term correspond to the behavior policy that generated the experiences, thereby mitigating the divergence between the on-policy and off- off-policy methods. These results indicate that method functions not merely mubility-promoone regularizer, but as principled causal bound optimization.
83.21% mean reward ratio. This provides evidence that the
Conclusion
This paper presents SUFT, causal upper-bound loss optimization method that enhances DRL sample efficiency and reduces computational demands at a negligible cost. We begin by establishing provable bound on the factual loss within the Neyman-Rubi potential outcomes framework and seamlessly adapting it to the DRL framework, showing that on-policy loss is bounded by every agent's 's standard off ofo-policy loss and an additional SUFT OPE term. This term is computable by storing past value network outputs in the experience replay buffer, thereby reusing discarded data to improve agents' performance. Experimental results across diverse environments and agents show that our method leads to significant improvements in convergence rates and reward outcomes. Looking ahead to future work, we see physical robotics as a particularly promising domain, where SUFT's reduced-resoss resource method could make DRL more practical under real-world constraints.
10

---

References
[1] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[2] Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[3] Schrittwieser, J., Antonogiou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604 609, 2020.
[4] Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the atari human benchmark. In Proceedings of the 37th International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
[5] Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint ST0.i:::301.040000 2023.
[6] Gu, S., Holly, E., Lillicrap. T., and Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3389-3399 3396. IEEE, 2017.
[7] Ibarz, Tan, J.. Finn, C., Kalakrishnan, M.. Pastor, P., and Levine, S. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698-7211 2021.
[8] Haarnoja, T., Moran, B., Lever, G., Huang, S. H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N. Y., Hafner, et al. Learning agile soccer skills for bipedal robot with deep reinforcement learning. Science Robotics, (89) 2024.
[9] Yu, C.,, Liu, J., Nemati, S., and Yin, G. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
[10] Symeonidis, P.. Chairistanidis. S., and Zanker. M. Deep reinforcement learning for medicine recommendation. In 2022 IEEE 22nd International Conference on Bioinformatics and Bioengi- neering (BIBE), pp. 85-90. IEEE, 2022.
[11] Shirali, A., Schubert, A., and Alaa, A. Pruning the way to reliable policies: A multi-objective deep q-learning approach to critical care. IEFE Journal of Biomedical and Health Informatics, 2024.
[12] Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., and Shah, A. Learning to drive in day. In 2019 international conference robotics and automation (ICRA), pp. 8248-8254. IEEE, 2019.
[13] Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P.. Al Sallab, A. A.. Yogamani, S., and Pérez, Deep reinforcement learning for autonomous driving: survey. IEEE Transactions on Intelligent Transportation Systems, 23(6)4499-4922 2021.
[14] Schier, M., Reinders, C., and Rosenhahn, B. Deep reinforcement learning for autonomous driving using high-level heterogeneous graph representations. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7147-7153. IEEE, 2023.
[15] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[16] Guo, D., Yang. D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint 2rXiv:2501.12948 2025.
11

---

[17] Silver, D., Schrittwieser. J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T. Baker, L., Lai. M., Bolton, A., al. Mastering the game of go without human knowledge. nature, 550(7676):354-55, 2017.
[18] Akkaya, I., Andrychowiez, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 191000710.... 13, 2019.
[19] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukeuoglu, K. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference Machine Learning, pp. 1928-1937. PMLR, 2016.
[20] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.0634 2017.
[21] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529-533 -3333,, 2015.
[22] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of The 35th International Conference Machine Learning, pp. 1861-1870. PMLR, 2018.
[23] Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In ICLR (Poster), 2016.
[24] Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[25] Zhang. S. and Sutton, R. R.S. deeper look at experience replay. arXiv preprint 11X11177177 2017.
[26] Wei, Q., Ma, H., Chen, C., and Dong, D. Deep reinforcement learning with quantum-inspired experience replay. IEEE Transactions on Cybernetics, 52(9):9326-9338, 2021.
[27] Espeholt, L.., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of The 35th International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
[28] Deng, Z., Jiang, J., Long, G., and Zhang, C. Causal reinforcement learning: survey, Transac- tions on Machine Learning Research, 2023.
[29] Zeng, Y., Cai, R., Sun, F., Huang, L., and Hao, Z. A survey on causal reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[30] Zhang, J., Tian, J., and Bareinboim, Partial counterfactual identification from observational and experimental data. In Proceedings of the 39th International Conference on Machine Learning, pp. 26548 26558. PMLR, 2022.
[31] Kasetty, T., Mahajan, D., Dziugaite, G. K., Drouin, A., and Sridhar, D. Evaluating interventional reasoning capabilities of large language models. Causality and Large Models @ @NeurIPS 2024, 2024.
[32] Wang. L., Yang, Z., and Wang. Z. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems, 34: 21 2114-221 2021.
[33] Blumlein, T., Persson, J., and Feuerriegel, S. Learning optimal dynamic treatment regimes using causal tree methods in medicine. In Proceedings of the 7th Machine Learning for Healthcare Conference, pp. 146-171. PMLR, 2022.
12

---

[34] Herlau, T. and Larsen, R. Reinforcement learning of causal variables using mediation analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2022.
[35] Saghafian, S. Ambiguous dynamic treatment regimes: A reinforcement learning approach. Management Science, 7069):5667-5690 2024.
[36] Ding, W., Lin, H., Li, B., and Zhao, D. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems, 35: 26532-26543 2022.
[37] Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198(Suppl 27): 6435 6467, 2021.
[38] Lin. H., Ding, W. Liu, Z., Niu. Y., Zhu, J., Niu, Y., and Zhao, D. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters, 2024.
[39] Buesing, L., Weber, T., Zwols, Y., Heess, N., Racaniere, S., Guez, A., and Lespiau. J.-B. Woulda, coulda, shoulda: Counterfactually-guui policy search. International Conference on Learning Representations, 2018.
[40] Lee, T. E., Zhao, J. A., Sawhney, A. S., Girdhar, S., and Kroemer, O. Causal reasoning in simulation for structure and transfer learning of robot manipulation policies. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4776-4782. IEEE, 2021.
[41] Lan, Y., Xu, X., Fang, Q., and Hao, J. Sample efficient deep reinforcement learning with online state abstraction and causal transformer model prediction. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[42] Gasse, M., Grasset, D., Gaudron, G., and Oudeyer, P.-Y. Using confounded data in latent basel- reinforcement learning. Transactions on Machine Learning Research Journal, 2023.
[43] Pace, A., Yèche, H., Schölkopf, B., Rätsch, G., and Tennenholtz, G. Delphic offline rein- forcement learning under nonidentifiable hidden confounding. In The Twelfth International Conference on Learning Representations, 2024.
[44] Ding, W., Shi, L.,, Chi, Y., and Zhao, D. Seeing not believing: Robust reinforcement learning against spurious correlation. Advances in Neural Information Processing Systems, 36, 2024.
[45] Namkoong, H., Keramati, R., Yadlowsky, S., and Brunskill, E. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33:18819 11888819-8888 2020.
[46] Shi, C., Uehara, M., Huang, J., and Jiang, N. A minimax learning approach to off-policy evaluation confounded partially observable markov decision processes. In Proceedings of the 39th International Conference Machine Learning, pp. 20057-20099 PMLR, 2022.
[47] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, (09(5):612-634 2021.
[48] Sontakke, S. A., Mchrjou, A., Itti, L., and Schölkopf, B. Causal curiosity: RI agents discovering self-supervised experiments for causal representation learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 9848-9858, PMLR, 2021.
[49] Lu, C., Hermández- J. M., and Schölkopf, B. Invariant causal representation learning for generalization in imitation and reinforcement learning. In /CI.20022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[50] Seitzer, M., Schölkopf, B., and Martius, G. Causal influence detection for improving efficiency in reinforcement learning. Advances Neural Information Processing Systems, 34:22905- 22918. 1021.
13

---

[51] Hu, X., Zhang, R., Tang, K. Guo, J., Yi, Q., Chen, R., Du, Z., Li, L., Guo. Q., Chen, Y., al. causality-drive hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35:20064-200777 2022.
[52] Lu, C., Huang, B., Wang, K., Hernández-Lobato J. M., Zhang, K., and Schölkopf. B. Sample- efficient reinforcement learning via counterfactual based data augmentation. arXiv preprins arXiv:2012.09092. 2020.
[53] Pitis, S., Creager, E., Mandlekar, A., and Garg, A. Mocoda: Model- counterfactual data augmentation. Advances in Neural Information Processing Systems, 35:18143, 1143-181111, 2022.
[54] Rubin, D. B. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Aooaodosssoiii Association, 100(469):322-333 2005.
[55] Oprescu, M., Dorn, J., Ghoummaid, M., Jesson, A., Kallus, N., and Shalit, U. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In Proceedings of the 40th International Conference on Machine Leurning, pp. 26599-26618 18. PMLR, 2023,
[56] Frauen, D., Melnychuk, V., and Feuerriegel, S. Sharp bounds for generalized causal sensitivity analysis. Advances in Neural Information Processing Systems, 36, 2024.
[57] Shalit, U., Johansson, F. D., and Sontag, D. Estimating individual treatment effect: generaliza- tion bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning, pp. 1076-3085. PMLR, 2017.
[58] Curth, A. and Van der Schaar, M. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In Proceedings The 24th International Conference on Artificial Intelligence and Statistics, pp. 1810-1818. PMLR, 2021.
[59] Athey, S., Keleher, N., and Spiess, J. Machine learning who to nudge: causal vs predictive targeting in a field experiment on student financial aid renewal. Journal of Econometrics, pp. 105945, 2025.
[60] Jesson, A., Mindermann, S., Gal, Y., and Shalit, U. Quantifying ignorance in individual-level causal-effect estimates under hidden confounding. In Proceedings of the 38th International Conference on Muchine Leurning, pp. 4829-4833 PMLR, 2021.
[61] Wang, H., Fan, J., Chen, Z., Li, H., Liu, W., Liu, T. Dai, Q., Wang, Y., Dong, Z., and Tang, R. Optimal transport for treatment effect estimation. Advances in Neural Information Processing Systems, 36, 2024.
[62] Kennedy, E. H., Balakrishnan, S., and Wasserman, L. Semiparametric counterfactual density estimation. Biometrika, 1 17(4))875-886, 2023.
[63] Kuzmanovic, M., Hatt, T., and Feuerriegel, S. Estimating conditional average treatment effects with missing treatment information. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pp. 746-766. PMLR, 2023.
[64] Melnychuk, V., Frauen, D., and Feuerriegel, S. Causal transformer for estimating counterfactual outcomes. In Proceedings of the 39th International Conference on Machine Learning, pp. 15293-15329. PMLR, 2022.
[65] Frauen, D., Hatt, T., Melnychuk, V., and Feuerriegel, S. Estimating average causal effects from patient trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 758775867555 7594, 2023.
[66] Lewis, D. Counterfactuals, John Wiley & Sons, 2013.
[67] Pearl, J. and Mackenzie, D. The book of why: the new science of cause and effect. Basic books, 2018.
[68] Uchara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.0633 2022.
[69] Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
14

---

[70] Pearl, J. Detecting latent heterogeneity, Sociological Methods & Research, 46(3):370-389, 2015.
[71] Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society; series B (methodological) 39(1):
1-22, 1977.
[72] Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning. Springer, 2006.
[73] Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[74] Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mastering atari games with limited data. Advances in neural information processing systems, 14:25476-25488 2021.
[75] Towers, M., Kwiatkowski, A.. Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
[76] Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
[77] Todorov, E., Erez, and Tassa, Y. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033, IEEE, 2012.
[78] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, Stable- baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8 2021.
[79] Wang. Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1903.2003. PMLR, 2016.
[80] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I,, Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint 1321v=13/ 2013.
15

---

A Causal Bound Notation Summary
L(:-): R Rgo loss function.
: the probability of treatment t.
Pt the conditional distribution of observations.
Py, the conditional distribution of potential outcomes given the target treatment.
Py, the conditional distribution of potential outcomes given the control treatment.
6(2; 0t) + neural network hypothesis parameterized by weights 0z.
(r(rtt) t) the expected outcome loss.
eF, the expected factual outcome loss.
CC the expected counterfactual outcome loss.
V The estimated treatment effect loss.
5 constant term independent of Φ.
B Theoretical Result
B.1 Definitions and Proofs
In this section, we present the complete definitions and proofs to establish our theoretical result within the causal framework. Let L R x RR - Rzo represent loss function.
Assumption B.1.
L(x, y) L(x', 88) - L(x, + L(v.8').
Proposition B.2. Assumption B.1 holds for L1 loss.
Proof.
L(x, - L(x',,/) < [L 4) I(
(1)
|| y) |x' 1'1l
(2)
S Ir y - (a' - -3''l
(3)
Ir x' + 3 -y)l
< |- x' + ///
(4)
L(x, x') L(y v')
(5)
Inequality is by x S |cc,, equality 2 is by the definition L, loss, inequality 3 is by reverse triangle
inequality | |x2|| |an 22 inequality 4 is by triangle inequality (01 22
[xn
+
[22
equality is by absolute symmetry |a| = a. and the definition of L2 loss.
We start with the binary treatment case and employ the Jeyman-Rubll potential outcomes framework define the following: X' C Rdd is the observation space. y C IR is the outcome space. T {0. 1) is the binary treatment space. Under this framework, there are two potential outcomes depending on the treatment assignment: if 1, the observed outcome = Y,, and t 0, the observed outcome is y Yo-
We assume a joint distribution P(X, Yo, Yi) with the following components: qe = Pr(T = is the treatment probability. Px == P(X t) is the conditional distribution of observations. Pr == PYY x) is the conditional distribution of potential outcomes given the target treatment, and 1 = PYY|X = æ) is the conditional distribution of potential outcomes given the control treatment. We also assume that there are no hidden confounders in the observational data, which makes the estimated treatment effect identifiable. This is formalized by assuming that the standard strong ignorability condition holds: (Yo, Y1) 1L T,X, and 0 < p(t 1|z) < for all x. Strong ignorability is a sufficient condition for the estimated treatment effect to be identifiable (7]
We now present the definitions used to formulate the causal upper bound.
Let & T be treatment.
16

---

Definition B.3. ((2;0)) X' + y is neural network hypothesis parameterized by weights 8, corresponding to the performed treatment
Definition B.4. The expected outcome loss:
(0(x, == Py, [L ob 0.)]]
Definition B.S, The expected factual outcome loss:
EP3 := B L (x, 1)].
c. E p Ll (2, 0)]
Definition B.6. The expected combined factual outcome loss:
eF, := f1 + P.
Definition B.7, The expected counterfactual outcome loss:
ccr, == _. (,, 1)].
&CF, EE px ( 0)].
Definition B.8. The expected combined counterfactual outcome loss: CCF, == quc + q
Definition B.9. The estimated treatment effect loss:
1/ P [Ø((;; 0o), oba;; 01 ))]
w6 P3 [L 01), clr; 0o))
Definition B.10. The combined estimated treatment effect loss:
t/ = 2o + 4112
B.2
Causal Bound Theorem
Lemma B.11.
En PP [Ly,, 01))1] Ey Pop-PP [L b(;; 0])]].
Similarly, for the opposite case:
EyouPy Byinp [L(30, o(1; 0o))]l = Eyor [Lvo o(x; 0o)))]
Proof.
E Eygrr (Byorgg (L(,,, (1x; 01))]] J, ( ((n 01)) p(yol=))do)) p(yn (x))
((,1,((;001)) (Sg p(uolr))dy))
kx)dy
L(v olx; 01)) p(uil=)dy
Ey [L(n, pbx; 01)]
The proof for the opposite case is the same, replacing and y1-
Theorem 1..2. The expected factual outcome loss, é F is bounded by the expected counterfactual outcome loss, &CF the estimated treatment effect loss, Va and a constant term 5, independent of oo:
CF. S ECF, + 1/ + 8.
Where 8 is the following:
8²_ = Egppy [ByoP [Ey,~ P;; L y1]l]]-
8 Eanpy pp [Bgipp [3 L(vi, yo]]]]
8 9oodt + :1000
17

---

Proof. We begin by placing the loss definitions:
CF. cCF, tr, + quet c CFF
(6)
p3 (z, 1)] +
((((z,)0) 0)
(7)
go pg 1 (z, 1)] _ p (& (xx
R [ Pi
(vn. (x;; 81 )}] q0 x~ B N0~
(vo, xx;; 66]]]
(8)
P0 [B P.
L (811 (xx; 01))]] 91 11 Px BB P
Lyv. (2x; 0b))]
= 21 (1 rx
E 2/s P.
(v1, (x;;; 0,))1] - B2-r3 IR yo- p%
[voo ((;; 0o))]]
+ go 18 y0 [yo, (2;; 0n)]]] 18 y1 y
(y11, (b:; 01))]
Now will use the linearity of expectation and Lemma B.11
FF ECF,
41 (1 p3 [By
(y1, (x;; 01)) EE yo 10
[Lvo, o (2; 0n))
(9)
+ go (B 12 [B Pr
(yn, (b;; 0o)) - Ey Pp
[Ln (b:; 01 ))ll)
2ı (E 2 E y py
E yo PV,
(y1,, (2;; (1)
L(vo, p(:; 6o) (ol))
(10)
+ go go (B pg Eg y0 Pro
E Eyn~ P3;
[(vo, <<:; Bu))
L811,0(2 0g (1))lI))
According to Assumption
B. we will get:
er, - cCF I 91 (TE
xxpt
E Pp
[vn, 9o) + [(d(;; 01), 0n))
(11)
2o (B 2B2
E y ~ P%
E py:
[yo. y1) L( 60).
91 [1 pp E
E P%
(Bn yollll + -n
(12)
20 (B r
[ No ~ P5
E Ey pV
[vo- vi }]] + rg [[x[:; ((:;
Rr [Ø(:; 00), pbx;; (1)
+ 21 [< (x: 01), 0o))
+ 20 [1E Pi. pP, [L v1llll + q L8 p P p [L1 soll
/ + ğ 2 + 90ōd + 91
(13)
v/ +
(14)
Finally, we are getting the causal bound:
FF S ECF, + v/ +
Equality 6 is by Definitions B.6 and B.8 equality 7 is by Definitions
B.5 and B.7 equality x is by
Definition B.4 equality 9 is by linearity expectations, equality 10 is by linearity of expectations
and by Lemma B.11 inequality ii by Assumption B. equality [2 is by linearity of expectations
and because & is a constant on the expectation of Yo, YI, equality 13 is by Definition B.9 and 8², 80 notation, equality 14 is by Definition and 8 notation.
B.3 Causal Bound for N Control Treatments
We now adjust the definitions and Theorem B.12 proof from the binary treatment scenario single target treatment and N control treatments scenario.
We first set the definitions of the Neyman-Rus Rubin potential outcome framework for the N control treatments scenario: T = {1.... N + 1} is the treatments space, where t = j 2 2 S N ++iis the control treatments, and = 1 is the target treatment compared against each control treatment. Under this framework, there are two types of potential outcomes depending on the treatment assignment:
I == 1, the observed outcome y Yi is from the target treatment, and V/j E (2...., N if = the observed outcome = Yg is from a control treatment.
We assume joint distribution P(X, Y.,,,YY,,..... .,, YN+1) with the following components: 4: Pr(T = t) the treatment probability. P := P(X[T = t) is the conditional distribution of observations. PT P(Y| = z) is the conditional distribution of potential outcomes given the target treatment, and Vj E (2....,A) N + 1} P3 P(Y|IX x) is the conditional distribution of potential outcomes given control treatment t= j.
18

---

Definition B.13. The expected factual outcome loss for N control treatments:
FF R px (2, 1)].
Vj (+1).
c : B= [x).
Definition B.14. The expected combined factual outcome loss for N control treatments:
N+1
CFa == qi + Σ 9
Definition B.15. The expected counterfactual outcome loss for N control treatments: Vj E N N
N
1j E py [Co(2, 1)]. CF
& pp (2,
Definition B.16. The expected combined counterfactual outcome loss for N control treatments:
1i O
N x
CCF,
Definition B.17. The estimated treatment effect loss for N control treatments: E ( 11:
2 == E 1 p L( 0,), 6(2; 01))].
1 pk [< 01), (xx; 0,))].
Definition B.18. The combined estimated treatment effect loss for N control treatments:
N+1
91 2
All the other definitions and proofs the same the binary case.
We now generalize Theorem 1.12 from a single control treatment scenario control treatments scenario as follows:
Theorem B.19.
N+1
N+1 Σ
QiEF.
1 +
Σ
lj CeECr
1 + tavi + 111 4z8hr 1 MIECR
N 7100))
2=2
Where is the following:
81. E
Pawp Byssmpp
E p3
[yg..
8¹ = Egnpy
BBye Pp
[Bg Pi
[
N+1
=
Σ
N 0101))
2=2
Proof. by induction.
Step 1: Base Case
For single target treatment and single control treatment, N 1, the following inequality holds:
n1 + 2
cr 2 I q2 qı 3/3 + q20t.2 + 92
1.2
19

---

As we prove in Theorem B.12 where the control treatment is t 2 instead of t = 0.
Step 2: Inductive Hypothesis
Assume that for N- 1 2 2 control treatments, the following inequality holds:
N
nict + +g
OFECF 15
N - 91 9)F S
1
N
1 N - 9109)
+ N 9 2 + 8 +
1
2=
Step 3: Inductive Step
We will now prove the inequality for N control treatments:
Σ
N+1
1 (a + tgck. - Cc.. N mascra)
1
N+1 Σ 5=2
goobin + N 91 1o. 920tts
+ 1 N 9118
2=2
Inductive Step proof.
1 giepp + +gett DFECF F
1 F + arerr - -geerr Fureoss
+ kicke + + N+1
-F++1
1N+1
N CFF
N+1
1 N 1 9,0h'
1
g +
+ N 9100
(15)
N 1 + C C N+1 Ps
1N+1
N+1
-N++ICFF
CCC
+ NN 9,8h1 + 1
1
N 1180
(16)
1i 1,N+1 + 1N+111
N 4
+
+ + UN+181,N+ 81,N41
+ NION++
1
N+1
ni 9,8hs + 1
= Σ
g3bby +
N 1228)
N
2=2
We will get the following inequality:
N+1
n + tarer
OFer queo +
I
t
15 + v +
1
+ 1 N 49100
Finally, we sum t and get the causal bound for N control treatments:
N+1
N+1
Σ
d
s Σ
1 nnce, + +0 + N nu + tggors
2=2
j=2
20

---

Causal Framework
DRL
Framework
F
Factual On-Policy
POn-Policy Q
Φ
C
Counterfactual Off-Policy
EOff-Policy Q
vo
Treatment term
SUUFT Q
OPE Term
Figure 4: Illustration diagram that demonstrates the reduction from the causal inference framework (Left) to the DRL framework (Right).
Inequality 15 by the inductive hypothesis, Inequality 16 from the proof of Theorem B.12 where the control treatment is N instead of
B.4 Reduction from Causal to DRL Framework
We now formalize reduction from a causal framework to a DRI. framework.
In the DRL framework, we consider the treatment to be the agent's policy, where the target treatment matches the target policy, and the control treatments match the behavior policies.
Within the causal framework, the on-policy loss corresponds to the factual loss when the treatment being optimized matches the treatment applied for generating the observations. This is analogous to optimizing the target policy using experiences from an identical behavior policy. Similarly, the off- off-policy loss aligns with the counterfactual loss when the treatment being optimized differs from the one used to generate the observations. This is equivalent to optimizing the target policy using experiences from different behavior policies.
To formalize the corresponding definition in the DRL framework, consider a replay buffer D containing samples produced by behavior policy that matches the target policy. In this context, the on-policy loss is a special case of the expected factual loss.
Similarly, consider replay buffer Du that stores experiences generated by behavior policies that differ from the target policy. In this context, the off-policy loss is a special case the expected counterfactual loss.
Q-Network Reduction: The definitions for the scenario of agent using Q-value network:
X' := (S x A) is the state- state-action space. Potential outcome y ~ p(a,a) is the accumulated reward over time steps, starting from state taking the initial action and following policy n.. And the hypothesis (xx; 0.) = Q(,, a; 0.) (S A) + y is the Q-values network, which influences the policy π..
Definition B.4 the DRL framework refers the temporal difference loss between the accumulated rewards and the Q-value estimation, and it is defined as follows:
Definition B.20. The expected temporal difference loss:
lo(s, a) por (g.Q(s.
Definitions B.5 and 1.13 in the DRL framework assigns the temporal difference loss with experiences collected on-policy, with identical target and behavior policies from D,, and it is defined as follows:
21

---

Definition B.21. The expected on-policy loss:
COn-Policyg B(,,a)-D, (s, a, furgee))
Definitions B.7 and BB15 in the DRL framework assigns the temporal difference loss with experiences gathered off-policy, with different target and behavior policies from Do. and it is defined as follows: Definition B.22. The expected off-policy loss:
PoftPolicyy = Do Del a, Murget))
Definitions B.9 and B.17 interpretation the DRL framework is an OPE term, and is defined as follows:
Definition B.23. The SUFT OPE term:
@SUFT Do [L (Q(s, Obchavior), Q(s, Ourgee)))
Note that this term evaluates the target policy by measuring the loss between the Q-value estimations of the target policy and the behavior policies, given samples from Do-
Theorems B.12 12 and 19 interpretation in the DRL framework establishes an upper bound to the on-policy loss, combining the off-policy loss and the SUFT OPE term.
EOn-Policy coft-Policy
+ USUFTa + 6.
Note that since the constant term, 6, is independent of Q, it is disregarded during the gradient-based optimization of Q, so the SUFT causal bound objective term is as follows:
<Off-Policy
+ ÀTF USUFTQ
V-Network Reduction: Now we present the corresponding definitions for the scenario of an agent
using a network:
X' = S is the state space. Potential outcome Pe is the accumulated reward over n time steps, starting from state 8, and following policy æ;- And the hypothesis p(>; 0,) = V(s; 0,) S y is the V network, which influences the policy πt-
Definition B. in the DRL framework refers to the temporal difference loss between the accumulated rewards and the V estimation, and it is defined as follows:
Definition B.24. The expected temporal difference loss:
(v L) = Ry-ry [L (v. (s;; 0.))
Definitions B.5 and B.13 the DRL framework assigns the temporal difference loss with experiences collected on on-policy, with identical target and behavior policies from D1, and it is defined as follows: Definition B.25. The expected on-policy loss:
6On-Policy, = E. D. lev (s, 4turges)].
Definitions B.7 B.15 in the DRL framework assigns the temporal difference loss with experiences gathered off-policy, with different target and behavior policies from Do. and it is defined as follows: Definition B.26. The expected off-policy loss:
Cont-Policy == Ex- (ky frarge)).
Definitions B.9 and B.17 17 interpretation the DRL framework is an OPE term, and is defined as follows: Definition B.27. The SUFT OPE term:
dsuuFry = Eg Do LL (V(s; Obchavior), V(s; Ouagget)].
22

---

Note that this term evaluates the target policy by measuring the loss between the V estimations of the target policy and the behavior policies, given samples from Dp-
Theorems B. 12 and B9 interpretation in the DRL framework establishes an upper bound to the on-policy loss, combining the off-policy loss and the SUFT OPE term.
éOn-Policyy Poff.olleyy + USUFTY +
Note that since the constant term, 8, is independent of V, it is disregarded during the gradient-based optimization of V. Thus, the SUFT causal bound objective term is as follows:
COIt-Policyy + XIF USUFTV.
Causal and DRL Framework A Alignment: An important alignment needs to be made to address a difference between the static observational data assumed by the causal framework and the dynamic experience replay buffer the online DRL framework. the causal framework, all observations gathered once, the data is fixed and irrevocably labeled either to be from the target treatment or from the control treatment in the binary treatment case. On the other hand, in the online DRL framework, the observational data changes over time, we continuously populate the experience replay buffer and update the target policy. On each policy optimization step, all of the samples stored in the buffer that were labeled as the target policy are turned into samples that are labeled as a control policy. This phenomenon of changing the observational data labels when updating the policy in the DRL framework is gap between the causal and DRL frameworks. In the on-policy case, the experience replay buffer contains only on-policy observations, meaning that = 1. When updating the target policy, this data becomes off- off-policy, and the data that was previously labeled target policy turns into control policy, shifting the samples' labels together with the treatment probability 40... = -laa 1.
The reduction we have made between the causal and the DRL framework above aligns the on-policy loss and off-policy loss terms with the corresponding terms in the causal framework. The on-policy loss is special case of the factual loss where the observation data is filled with target treatment samples, corresponding to e where n1 I. The off-policy loss is special case of the counterfactual loss where the observation data is filled with control treatment samples, corresponding to e where the labels and the treatment probability are shifted and 90.. Q1au 1.
23

---

C SUFT Pseudocode
C.1 SUFT DQN Pseudocode
Algorithm 1 SUFT Deep Q-learning
Initialize replay memory Do capacity N
Initialize action-value function Q with random weights 0 for episode 1, .M do
Initialize start state $1 for f 1.7 do
With probability P select random action de
otherwise select a; = arg max, Q (st, a; Obehavior)
Execute action in the environment and observe reward Tt and state 81+1
Store transition (st, as rs, St+1, Q(st, ay; (bchavior) in Do
Sample random mini-batch of transitions (sj, "j $1+1, Q(5y, azi Obchavior)) from Do
(T's Set y r + max, Q(85+1; al: Oarge)) Perform gradient descent step with respect the network parameters Ouarge on:
for terminal 8j+1
for non-terminal 88+1
E, [w> Q(s,, ay; (unga)) + AIF (Q(s), aj; bbchavoor) - Q(s,, ay; Bunge)) )²]
end for end for
Our method extends the standard DQN algorithm [8] in main ways: The first is by storing the old Q values (ss, a; (bchavior) in the replay buffer. The second is by incorporating the additional SUFT OPE term into the objective function ÀTP (Q(s), ayi Onehavior) ( Ugi Omrgeet) which represents the estimated treatment effect. Onehavior refers to the network weights used for action selection, the behavior policy, and 0 Ourget represents the current network weights, which are optimized in the gradient descent step. the target policy.
24

---

C.2 SUFT PPO Pseudocode
Algorithm 2 SUFT Proximal Policy Optimization (PPO)
Initialize rollout buffer D1 with horizon T Initialize policy function (actor) and value function (critic) with random weights / for iteration .......C do
for f = 1. do
Run policy Tieerania (8+)
Store transition (St, dt, It,, Mesuiii (at | 82), V ((s;; Onehavior) )) in D₁
end for
Compute advantage estimates AI...., As. and rewards to go Ri.... Re based on the stored values V (ss1 Obchavior)..... V(sri Obcbavior)
for k 1, K epochs do
Sample random mini-batch of transitions (s3, ajz y T Tone (ag 18)), V8y; Obehaviar) ))
from the rollout buffer D1
Compute probability ratio:
15 0r (a, 18))
(0) =
7T Bestusi (() 85))
Optimize actor loss:
Ey min (r)()) clip 1 4;)]
Optimize critic loss:
E, (V (umge)))))) R2 + >TF (V(s); beehavior) Vss; m )²
end for end for
Our method extends the standard PPO algorithm 20 by incorporating the additional SUFT OPE term into the PPO critic loss. Notably, PPO performs multiple epochs of optimization the same rollout buffer D1, leading to a divergence between the behavior and target policies. This divergence the reason PPO benefits from our method, despite being an on-policy algorithm. Onehavior refers to the network weights used to interact with the environment also referred to in [20] as Oold. the behavior policy, and Orarget represents the current network weights, which optimized, also referred to in (20) as 0, the target policy. During the first epoch, before optimizing the network, the behavior and target policies are identical, so the SUFT OPE term equals zero. After the first optimization step, the target policy diverges from the behavior policy, and the SUFT OPE term now quantifies the estimated treatment effect between the target and behavior policies. In addition, the V(s,; Oochavior) ) values are already calculated during the calculation the advantage estimates Åc, and rewards to go R. Thus, our method effectively reuses these stored network outputs for computing the SUFT OPE term, similar to the approach used in our SUFT DQN implementation.
25

---

D Full Experimental Results
In this section, we present detailed and comprehensive experimental results for both the Atari and the MuJoCo domains, as well as the formulas and criteria used to evaluate the results.
D.1 Negligible Computational Overhead
We quantify the computational and space overhead introduced by computing the SUFT OPE term and storing old Q-values in the experience replay buffer. We run experiments a single NVIDIA L4 GPU, and for the Vanilla DQN agent, the additional computation takes approximately 23.77 seconds, compared to total training time of 2,980 seconds. This corresponds to 0.8% computational overhead. terms of space, we append a single 64-bit scalar to each stored experience. For a 4K experience replay buffer, this results an additional 0.0305 MB, compared 972.85 MB without it. This is equivalent to a 0.003% space overhead.
D.2 Tuning the XTF Coefficient
In order evaluate the impact of the ÄTF coefficient on improving the loss accuracy, we conducted experiments using various XT values. Adding the \re demonstrates a consistent improvement across a range of XTF values. This indicates that precise hyperparameter tuning is unnecessary, and as long as the SUFT OPE term is added with coefficient within a valid range of ÄTF, the loss optimization improves. The ÄTF value can easily tuned through random search hyperparameter optimization within a valid range. Specifically, we encounter that sufficient range to search from is between ÄTF 0.5 and XTF 1.5. The XTF values that yield the best performance are XT = for all the DQN agents, ÀTF = 2.5 for the PPO agent using the Using I, Loss and \TF 5 for the PPO agent using the Using L2 Loss in the Atari domain. the MuJoCo domain, SAC performs best at XTF 0.6 and PPO at XTF = 1.8.
D.3 Evaluation Formulas
To evaluate our method, we train every agent-environment configuration for 10 different runs and report the median result for each setup. Since this is even-numbered list, there two central values, and we use the higher of the two as the median. The valid environments used for calculating the improvement percentage and mean reward ratio percentage formulas those where the numerator and denominator are greater than one. This restriction allows us to exclude cases where the reward only marginally higher than the random reward, which could otherwise yield arbitrarily large ratios. By doing so, we ensure that the metric remains stable and effectively reflects meaningful improvements.
Improvement Percentage Formula:
The improvement percentage for comparing approach to an identical agent without the additional term is calculated with the following formula:
Higher reward approach Lower reward approach Lower reward approach Random reward
100%
For better visualization of the results, we display them in log scale with the following formula:
logio [Improvement Percentage Formula + 1]
We add to the improvement percentage formula to avoid the log of 0 and fractions, which result in negative log. This addition has no impact on the results.
The mean reward ratio percentage for comparing approach to an identical agent without the additional term is calculated with the following formula:
Mean Reward Ratio Percentage Formula:
SUFT reward
Random reward
100%.
Valid Envs
Baseline reward Random reward
Note that for the Atari domain, the human normalized results are calculated using the above formula, where the baseline reward is replaced with the human reward.
26

---

D.4
Assumption 4.1 Empirical Analysis with L2
DRL agents are often trained using L2 loss, for which Assumption 4.1 does not hold. To provide intuition for the observed L2 performance, we empirically examine the validity of the L2 loss for Assumption using synthetic and domain-specific data. For the synthetic data, we generate one billion independent samples from the standard normal distribution, U, x. .1/ ~N(0, 1), and
evaluate whether:
- '/)² 2' )
(17)
This experiment corresponds directly to the inequality form of Assumption when applied to the Le loss. We find that inequality [7 holds in approximately 82% of random samples, demonstrating that while Assumption 4. is not guaranteed theoretically, it is satisfied in the vast majority of cases, To further validate this observation a domain-specifte dataset, we conduct an experiment in the Atari 57 benchmark using DQN agent trained for 400K steps with batch size of 32. Since our method operates in an off-policy setting, the left side of the causal bound in Theorem [. 10 is not computable from the observed data, as we don't 't observe the on- on-policy experiences (this is the reason we optimize on the causal bound in our method). For this reason, we evaluate only inequality [I used B.12 considering the off- off-policy terms both sides (with yo 1):
in the proof of Theorem
(E 2o E yo PFo (E [E
E B
[Loo, (x2; 06)) L(v, 0(2; 01 01)) S
(18)
E Pr
Lyyo v1) + L(o(2; 0o), o(x; fIIIIII)
Because the true values of and 1/1 are not directly observable, since they are the true values from the given state until the end of the episode, we approximate them using the standard Bellman equation for Q-values. Under this setup, we find that inequality [8] holds in approximately 79% of cases across the 57 Atari games. These empirical findings indicate that although Assumption [. is not theoretically guaranteed for L2, it holds in the vast majority of the optimization steps in practice, which may explain the strong empirical performance of our method using the I.2 loss.
D.5 Empirical Evaluation of the 8 Term Magnitude
In Theorem 4.10 the å term is ignored during optimization since it is independent of Q. To assess whether this omission introduces sigmificant looseness in the causal bound, we conduct an empirical analysis on the Atari 57 benchmark using DQN agent trained for 400K steps. Specifically, we plotted the magnitudes of the standard DQN loss, the ó term, and the SUFT OPE term throughout training. As discussed in Appendix [D.4 the ō term involves the true values of yo and y1, which are not directly observable. Thus, we approximate them using the standard Bellman equation for Q-values. The results, shown in Figure indicate that across all of the environments, the 8 term remains consistently an order of magnitude lower than the standard DQN loss, and on the same order of magnitude as the SUFT OPE term. These findings support the decision to ignore the ō term during optimization, as does not meaningfully loosen the causal bound.
27

---

MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss MSE Loss
Alien
Amidar
Assault
Asterix
Asteroids
Atlantis
Bank Heist
Battle Zone
Beam Rider
Berzerk
Bowling
Boxing
Breakout
Centipede
Chopper Command
306
XX
10e
208
L0W
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
Di
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Krull
Kung-Fu Fu Master
Montezuma's Revenge
Ms. Pacman
Name This Game
Phoenix
300
5006
2:26
60
Fall
Pong
Private Eye
Q*Bert
River Raid
2006
30c
XX
2016
Road Runner
Robotank
Seaquest
Skiing
Solaris
Space Invaders
Star Gunner
Surround
Tennis
Time Pilot
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
voe Iu
voe
V'e
vie
Yars Revenge
Zaxxon
# Steps
# Steps
# Steps
Steps
Steps
Standard DQN Loss
Delta Term
SUFT OPE Term
Figure 5: Magnitude of the á term, SUFT OPE term, and the standard DQN loss across the Atari 57 benchmark.
28

---

D.6 Atari 2600 57 Games Benchmark
Table 2: Atari 2600 random and average human rewards. The scores are sourced from 79
Game
Random
Human
Alien
227.8
6371.3
Amidar
5.8
1540.4
Assault
222.4
628.9
Asterix
164.5
7536
Asteroids
871.3
36517.3
Atlantis
13463
26575
Bank Heist
14.2
644.5
Battle Zone
3560
33030
Beam Rider
254.6
14961
Berzerk
196.1
2237.5
Bowling
35.2
146.5
Boxing
-1.5
9.6
Breakout
1.6
27.9
Centipede
2090.9
10321.9
Chopper Command
811
7387.8
Crazy Climber
10780.5
32667
Defender
2874.5
14296
Demon Attack
208.3
1971
Double Dunk
-18.6
-16.4
Enduro
0
740.2
Fishing Derby
-77.1
-38.7
Freeway
0.1
25.6
Frostbite
65.2
4202.8
Gopher
257.6
2311
Gravitar
173
3116
H.E.R.O.
1580.3
25839.4
Ice Hockey
-11.2
0,5
James Bond
33.5
302.8
Kangaroo
100
2739
Krull
2109.1
Kung-Fu Master
258.5
20786.8
Montezuma's 's Revenge
25
4182
Ms. Pacman
307.3
6.51.6
Name This Game
1747.8
6796
Phoenix
761.4
6686.2
Pit Fall
-348.8
5998.9
Pong
-20.7
14.6
Private Eye
662.8
64169.1
Q*Bert
163.9
12085
River Raid
1338.5
14382.2
Road Runner
200
6878
Robotank
2.2
8.9
Seaquest
215.5
40425.8
Skiing
-15287.4
-4336.9
Solaris
2047.2
11032.6
Space Invaders
182.6
1464.9
Star Gunner
697
9528
Surround
Tennis
-23.8
-8,3
Time Pilot
3568
5229.2
Tutankham
12.7
138.3
Up Down
707.2
9896.1
Venture
18
Video Pinball
20452
15641.1
Wizard of Wor
804
4556
Yars Revenge
3092.9
47135.2
Zaxxon
32,5
8443
29

---

Atari PPO Results (Using L1 Loss):
Table Training rewards comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without it across 57 Atari games. SUFT surpasses the baseline agent in 40 games, with statistically significant gains in 31 games. Both agents are using L1 loss.
Game
Baseline PPO
SUFT PPO
p-value
Alien
860.0
780.0
N/A
Amidar
184.0
133.0
N/A
Assault
631.0
692.0
9.15e-02
Asterix
346.0
568.0
1.17e-08
Asteroids
483.0
644.0
3.19c-07
Atlantis
36100.0
51300.0
4.50c-06
Bank Heist
37.5
51.5
6.14c-02
Battle Zone
8380.0
8880.0
6.07e-01
Beam Rider
450.0
526.0
2.34e-05
Berzerk
661.0
676.0
4.70e-01
Bowling
41.7
33.8
N/A
Boxing
-10.0
15.1
1.35c-13
Breakout
14.4
10.4
N/A
Centipede
2860.0
3390.0
2.23e-06
Chopper Command
1350.0
1990.0
2.84e-05
Crazy Climber
26700.0
37800.0
4.67e-11
Defender
2850.0
4280.0
8,90e-08
Demon Attack
295.0
306.0
5.37e-01
Double Dunk
-21.7
-18.1
2.19e-03
Enduro
13.1
67.8
3.18c-06
Fishing Derby
-82.9
-79.5
2.95c-01
Freeway
20.2
24.6
6.60c-06
Frostbite
1610.0
261.0
N/A
Gopher
381.0
494.0
2.38e-05
Gravitar
318.0
211.0
N/A
H.E.R.O.
9810.0
8840.0
N/A
Ice Hockey
-9,05
-5.66
2.65c-05
James Bond
324.0
466.0
3.54c-08
Kangaroo
3000.0
1080.0
N/A
Krull
4770.0
4360.0
N/A
Kung-Fu Master
1990.0
2130.0
1.05e-01
Montezuma's Revenge
0.0
0.0
N/A
Ms. Pacman
1210.0
1770.0
1,55e-06
Name This Game
2480.0
4520.0
10e-13
Phoenix
734.0
1700.0
19c-11
Pit Fall
-1.28
-4.73
N/A
Pong
-16.5
-12.0
4,95c-03
Private Eye
84.3
25.7
N/A
Q*Bert
2080.0
689.0
N/A
River Raid
2560.0
2590.0
5.53e-01
Road Runner
2330.0
8600.0
4.66e-03
Robotank
4.88
4.1
N/A
Seaquest
576.0
689.0
4.14e-03
Skiing
-30000.0
-18000.0
1.19--04
Solaris
1850.0
1880.0
6.68c-01
Space Invaders
292.0
406.0
7. 7c126-06
Star Gunner
1030.0
1090.0
7.81c-04
Surround
-9.7
-9.04
1.59e-02
Tennis
-13.3
-15.6
N/A
Tine Pilot
2790.0
3610.0
4.88e-05
Tutankham
82.1
102.0
2.85c-04
Up N Down
3090.0
19100.0
7.83c-06
Venture
16.0
8.0
N/A
Video Pinball
9030.0
14600.0
9,20e-06
Wizard of Wor
508.0
1060.0
2.09e-09
Yars Revenge
$230.0
10700.0
2.87e-12
Zaxxon
1040.0
271.0
N/A
Mean (%)
100.0
164.7
N/A
30

---

Alien
Amidar
Assault
Asterix
Asteroids
Reward
1
Atlantis
Bank Heist
Battle Zone
Bearn Rider
Berzerk
Reward
Bowling
Boxing
Breakout
Centipede
Chopper Command
Reward Reward
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
2013
200
2014
2014
200
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
Reward Reward
100
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Krull
x:00
Kung Fu Master
Montezuma's Revenge
Ms. Pacman
Name This Game
Phoenix
Reward Reward
Pit Fall
Pong
Private Eye
Bert
River Raid
Road Runner
Robotank
Seaquest
Skiing
Solaris
Reward
10C
201
2014
Lib
200
2014 Tennis
2016
Space Invaders
Star Gunner
Surround
Time Pilot
Reward Reward
200
2014
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
J007
Yars Revenge
Zaxxon
steps
steps
steps
Reward
108
# steps
# steps
SUFT
Baseline
Figure 6: Learning curves comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. The shaded area the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using L1 loss.
31

---

Up N Down
Road Runner
Name This Game
Revenge
Ice Hockey
Asterix
Chopper
Pong
Space Invaders
Gopher
Crazy Climber
Contipede
Allantis
Ms. Pacman
Bank Heist
James
Beam Rider
Seequest
Freeway
Star Gunner
Assault
Demon Attack
Battle Zone
Kung-Fu
Berzerk
River Raid
+10%
+100%
Fall
H.E.R.O.
Alien
Tennis
Amidar
Robotank
Breakout
Kangaroo
Gravitar
Zaxxon
Frostbite
0.0
0.5
1.0
1.5
2.0
2.5
Log Improvement (%)
SUFT
Baseline
Figure 7: Log-scaled reward improvements comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without it across 41 valid Atari games. SUFT outperforms baseline agent 28 out the 41 valid games, achieving a reward improvement of over 100% in 24.4% of them and over 10% in 61% of them. The red line indicates 10% improvement, and the green line represents 100% improvement. Both agents are using L1 loss.
32

---

Atari PPO Results (Using Lg Loss):
Table Training rewards comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without it across 57 Atari games. SUFT surpasses the baseline agent in 50 games, with statistically significant gains in 40 games. Both agents are using L2 loss.
Game
Baseline PPO
SUFT PPO
p-value
Alien
860.0
861.0
3.48e-01
Amidar
184.0
184.0
N/A
Assault
631.0
1150.0
3.24e-07
Asterix
346.0
651.0
7.43e-08
Asteroids
483.0
586.0
9.18c-05
Atlantis
36100.0
59100.0
2.57c-09
Bank Heist
37.5
38.8
3.47c-01
Battle Zone
8380.0
11000.0
5,02e-05
Beam Rider
450.0
497.0
7,57e-03
Berzerk
661.0
782.0
3.34e-05
Bowling
41.7
43.2
3.73c-01
Boxing
-10.0
1.75
2.10c-09
Breakout
14.4
17.3
1,90c-04
Centipede
2860.0
4120.0
3,28e-09
Chopper Command
1350.0
2940.0
4.37e-07
Crazy Climber
26700.0
40200.0
1.99e-10
Defender
2850.0
4070.0
3.08e-07
Demon Attack
295.0
634.0
2.31e-06
Double Dunk
-21.7
-21.5
8.58e-01
Enduro
13.1
70.1
1.30c-10
Fishing Derby
-82.9
-73.9
4.32c-06
Freeway
20.2
21.2
1.74c-02
Frostbite
1610.0
2750.0
5,59e-02
Gopher
381.0
547.0
1.71e-06
Gravitar
318.0
370.0
3.63e-01
H.E.R.O.
9810.0
11700.0
1.32c-01
Ice Hockey
-9,05
-7.45
4.09c-04
James Bond
324.0
380.0
6,50c-05
Kangaroo
3000.0
5830.0
1.04e-01
Krull
4770.0
8980.0
1.36e-03
Kung-Fu Master
1990.0
2610.0
3. 7.76-003
Montezuma's Revenge
0.0
0.0
N/A
Ms. Pacman
1210.0
1710.0
3.61e-03
Name This Game
2480.0
4410.0
7.45e-16
Phoenix
734.0
1720.0
1.05c-13
Pit Fall
-1.28
-1.23
2.23c-01
Pong
-16.5
-15.0
6.31c-03
Private Eye
84.3
76.0
N/A
Q*Bert
2080.0
5100.0
1.72e-03
River Raid
2560.0
3540.0
2.84e-11
Road Runner
2330.0
6320.0
1.51e-06
Robotank
4.88
4.29
N/A
Seaquest
576.0
1000.0
1.19e-07
Skiing
-30000.0
-3000000
N/A
Solaris
1850.0
2200.0
3.81e-01
Space Invaders
292.0
387.0
1.31c-04
Star Gunner
1030.0
1130.0
1.60c-08
Surround
-9.7
-8.65
7.33e-04
Tennis
-13.3
-13.6
N/A
Tine Pilot
2790.0
3190.0
1.24e-02
Tutankham
82.1
129.0
1,79c-02
Up N Down
3090.0
6020.0
3,83c-03
Venture
16.0
5.0
N/A
Video Pinball
9030.0
11500.0
3.22e-03
Wizard of Wor
508.0
734.0
6.22e-04
Yars Revenge
$230.0
10100.0
1.36e-04
Zaxxon
1040.0
2310.0
0.88e-03
Mean (%)
100.0
197.34
N/A
33

---

Alien
Amidar
Assault
Asterix
Asteroids
Reward
Atlantis
Bank Heist
Battle Zone
Beamn Rider
Berzerk
Reward
Bowling
Boxing
Breakout
Centipede
Chopper Command
Reward
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
Reward
2013
2014
200
2014
2006
2014
200
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
Reward
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Krull
Reward
3:0
Kung Fu Master
Montezuma's Revenge
Ms. Pacman
Name This Game
Phoenix
Reward
Pit Fall
Pong
Private Eye
Bert
River Raid
Reward
Road Runner
Robotank
Seaquest
Skiing
Solaris
Reward
201
201
200
a01
Space Invaders
Star Gunner
Surround
Tennis
Time Pilot
Reward Reward Reward
2003
200E
2006
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
1900
2013
Yars Revenge
Zaxxon
steps
steps
steps
# steps
# steps
SUFT
Baseline
Figure 8: Learning curves comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. The shaded area the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using La loss.
34

---

Enduro
Chopper Command
Name
Yars Revenge
Road Runner
Asterix
Contipede
Gopher
Assault
Zaxxon
N Down
Seaquest
Atlantis
Kangaroo
Space Invaders
Crazy Climber
River Raid
Hockey
Frostbite
Ms. Pacman
Battle Zone
Gravitar
Kung-Fu Fu Master
Star Gunner
Beam Rider
Bowling
H.E.R.O.
Breakout
James
Bank Heist
Freeway
Alien
Fall
Amidar
Tennis
+10%
+100%
Robotank
0.0
1.0
1.5
2.0
2.5
Log Improvement (%)
SUFT
Baseline
Figure 9: Log-scaled reward improvements comparison between the PPO agent using the additional
SUFT OPE term and the baseline agent without it across 42 valid Atari games. SUFT outperforms the baseline agent 39 out the 42 valid games, achieving a reward improvement over 100% in 38.1 1% of them and over 10% in 83.3% of them. The red line indicates 10% improvement, and the green line represents 100% improvement. Both agents are using Lg loss.
35

---

Table 5: Training rewards comparison between the PPO agent using the additional SUFT OPE term with different XTF F values and the baseline agent without it across Atari games. All agents are using L2 loss.
Agent
ArF
Mean (%)
PPO
0.0
100.0%
SUFT
105.0%
0.2
116.05%
0.3
124.0%
0.4
126.99%
0.5
128.79%
0.6
137.06%
0.7
143.12%
0.8
146.56%
0.9
145,4%
1.0
151.72%
1.5
170.4%
2.0
167.33 33%
2.5
179.01%
3.0
184.66%
5.0
197.34%
10.0
198.28%
36

---

Atari DQN Results:
Table 6: Training rewards comparison between DQN agent using the additional SUFT OPE term and the baseline agent without it across 57 Atari games. SUFT surpasses the baseline agent in 52 games, with statistically significant gains in 49 games. Both agents are using L2 loss.
Game
Baseline DQN, buffer 4K
SUFT DQN, buffer 4K
p-value
Alien
518.0
717.0
3.58c-07
Amidar
121.0
198.0
5.34e-06
Assault
1280.0
2250.0
4.10e-10
Asterix
872.0
1130.0
2.82c-03
Asteroids
294.0
450.0
2.31e-05
Atlantis
14000.0
19300.0
9.56e-05
Bank Heist
12.8
34.4
1.04e-05
Battle Zone
3350.0
4630.0
2.97c-04
Bcam Rider
514.0
1050.0
6.13c-11
Berzerk
507.0
622.0
3.44c-06
Bowling
21.9
31.1
4.56e-02
Boxing
-29.9
17.4
3.70e-15
Breakout
12.8
18.3
5.80e-03
Centipede
3160.0
3220.0
8.00c-01
Chopper Command
1150.0
1380.0
1.91e-03
Crazy Climber
21700.0
36000.0
3.05e-11
Defender
2510.0
2530.0
3.16e-01
Demon Attack
2240.0
4430.0
9.42c-11
Double Dunk
-3.88
-22.9
1. 1.96c-06
Enduro
48.2
169.0
3.24e-16
Fishing Derby
-88,9
-39.8
2.55e-08
Freeway
19.0
21.2
1.39e-03
Frostbite
128.0
607.0
1.03e-03
Gopher
288.0
614.0
1.72c-06
Gravitar
136.0
284.0
2.100-06
H.E.R.O.. O.
2010.0
4110.0
1.56c-05
Ice Hockey
-13.0
-6.62
1.19e-07
James Bond
34.0
363.0
1.24e-10
Kangaroo
460.0
5810.0
5.23e-07
Krull
315.0
3670.0
1.96e-14 14
Kung-Fu Master
6.0
2920.0
2.42e-05
Montezuma's Revenge
0.0
0.0
N/A
Ms. Pacman
1060.0
1270.0
6.08e-05
Name This Game
2540.0
4250.0
6.56e-11
Phoenix
608.0
3120.0
3.82c-16
Pit Fall
-163.0
-78.1
2.47e-03
Pong
20.2
-17.9
2.01e-07
Private Eye
133.0
154.0
7.51e-01
Q*Bert
908.0
2320.0
5.64c-06
River Raid
2410.0
3160.0
2.25c-08
Road Runner
991.0
10400.0
1.59c-10
Robotank
4.9
8.71
2.55e-07
Seaquest
393.0
911.0
2.56e-08
Skiing
-30800.0
30200000
8.17e-04
Solaris
565.0
696.0
2.62e-04
Space Invaders
317.0
371.0
0.5
Star Gunner
519.0
953.0
7.92e-11
Surround
-9.11
-9.17
N/A
Tennis
-18.9
-19.6
N/A
Time Pilot
1210.0
3350.0
4.85c-14
Tutankham
11.2
71.4
7.98e-12 12
N Down
3180.0
$340.0
1.27e-03
Venture
4.0
N/A
Vidco Pinball
9360.0
8890.0
N/A
Wizard of Wor
235.0
421.0
4.94c-09
Yars Revenge
3060.0
8280.0
7.44c-08
Zaxxon
0.0
2710.0
1.23e-07
Mean (%)
100.0
383.19
N/A
37

---

Alien
Amidar
Assault
Asterix
Asteroids
Reward Reward
Atlantis
Bank Heist
Battle Zone
Bearn Rider
Berzerk
Bowling
Boxing
Breakout
Centipede
Chopper Command
Reward
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
Reward
2014 200
Xie
2011 Gopher
1906
201
Fishing Derby
Freeway
Frostbite
Gravitar
Reward Reward
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Krull
Kung-Fi Fu Master
Montezuma's Revenge
Ms. Pacman
Name This Game
Phoenix
Reward 10
Pit Fall
Pong
Private Eye
Bert
River Raid
Reward
2000
Road Runner
Robotank
Seaquest
Skiing
Solaris
Reward
31006
201
Rit
2006
2011 Tennis
2016
Space Invaders
Star Gunner
Surround
Time Pilot
Reward
200
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
Reward
Yars Revenge
Zaxxon
steps
steps
steps
Reward
# steps
# steps
SUFT
Baseline
Figure 10: Learning curves comparison between DQN agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. The shaded area the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using L2 loss.
38

---

Kangaroo
Road Runner
Gopher
Atlantis
Frostbite
Seaquest
Enduro
Name This Game
Beam Rider
Q"Bert
Robotank
Crazy Climber
Demon Attack
Assault
Up N Down
River Raid
Alien
Chopper
Amidar
Breakout
Pit Fall
Space Invaders
Berzerk
Asterix
Freeway
Centipede
+10%
+100%
Tennis
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Log Improvement (%)
SUFT
Baseline
Figure 11: Log-scaled reward improvements comparison between DQN agent using the additional SUFT OPE term and the baseline agent without it across 29 valid Atari games. SUFT outperforms the baseline agent 28 out the 29 valid games, achieving a reward improvement over 100% in 48.3% of them and over 10% in 93.1% of them. The red line indicates 10% improvement, and the green line represents a 100% improvement. Both agents are using Lg loss.
39

---

Table 7: Training rewards comparison between DQN agent using the additional SUFT OPE term with small 4K buffer and the baseline agent without the SUFT OPE term with larger 100K buffer across 57 Atari games. SUFT surpasses the baseline agent in 52 games, with statistically significant gains in 47 games. Both agents are using L2 loss.
Game
Baseline DQN, buffer 100K
SUFT DQN, buffer 4K
p-value
Alien
438.0
717.0
3.14c-09
Amidar
118.0
198.0
2.20e-07
Assault
1120.0
2250.0
6.54e-10
Asterix
914.0
1130.0
4.06e-03
Asteroids
275.0
450.0
4.69c-06
Atlantis
7970.0
19300.0
7.59e-09
Bank Heist
34.4
1.01e-11
Battle Zone
3100.0
4630.0
9.71e-04
Beam Rider
561.0
1050.0
7.44e-09
Berzerk
458.0
622.0
1.55c-05
Bowling
21.8
31.1
4.58c-02
Boxing
-28.7
17.4
2.25c-15 15
Breakout
17.7
18.3
6.94e-01
Centipede
3130.0
3220.0
7.99e-01
Chopper Command
648.0
1380.0
1.30e-09
Crazy Climber
30400.0
36000.0
1.38e-04
Defender
2440.0
2530.0
4.85e-02
Demon Attack
3220.0
4430.0
8,29e-08
Double Dunk
-23.9
-22.9
8.65e-07
Enduro
66.8
169.0
5.22c-16
Fishing Derby
-84.1
-39.8
4.29c-07
Freeway
18.9
21.2
2.32e-03
Frostbite
159.0
607.0
1.95e-03
Gopher
119.0
614.0
3.64e-09
Gravitar
107.0
284.0
4.17c-09
H.E.R.O.
2320.0
4110.0
8.37c-05
Ice Hockey
-13.2
-6.62
2.20c-08
James Bond
43.5
363.0
1.41e-10
Kangaroo
534.0
5810.0
5.25e-07
Krull
336.0
3670.0
2.71e-
Kung-Fu Fu Master
13.0
2920.0
2.17e-05
Montezuma's 's Revenge
0.0
0.0
N/A
Ms. Pacman
1060.0
1270.0
3,69e-03
Name This Game
2990.0
4250.0
3.66c-10
Phoenix
732.0
3120.0
1.12c-15
Pit Fall
-114.0
-78.1
5.41c-01
Pong
-0066
-17.9
7.89e-09
Private Eye
110.0
154.0
6.69e-01
Q*Bert
1130.0
2320.0
3.81e-05
River Raid
2200.0
3160.0
3.60e-10 10
Road Runner
909.0
10400.0
1.04c-10
Robotank
4.44
8.71
4.26c-05
Seaquest
391.0
911.0
1.08e-08
Skiing
-30700.0
33200000
8.64e-03
Solaris
515.0
696.0
2.80e-05
Space Invaders
366.0
371.0
3.07e-01
Star Gunner
351.0
953.0
1.21e-12 1-12
Surround
-8.69
-9.17
N/A
Tennis
-19.6
N/A
Time Pilot
1100.0
3350.0
1.47c-14
Tutankham
9.4
71.4
4.66c-18
Up Down
2160.0
5340.0
6.09e-05
Venture
0.0
N/A
Video Pinball
9890.0
8890.0
N/A
Wizard of Wor
292.0
421.0
2.39e-07
Yars Revenge
3530.0
8280.0
9.75e-08
Zaxxon
2710.0
1.19--07
Mean (%)
100.0
437.05
N/A
40

---

Table 8: Training rewards comparison between DQN agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. SUFT surpasses the baseline agent in 49 games, with statistically significant gains in 44 games. Both agents are using 100k buffer and Lg loss.
Game
Baseline DQN, buffer 100K
SUFT DQN, buffer 100K
p-value
Alien
438.0
686.0
1 1.45e-08
Amidar
118.0
235.0
6,49c-11
Assault
1120.0
2170.0
3.18c-10
Asterix
914.0
1240.0
3.19c-05
Asteroids
275.0
430.0
1.03e-05
Atlantis
7970.0
12400.0
5.88e-05
Bank Heist
9.9
27.7
8.87e-04
Battle Zone
3100.0
4270.0
5,45e-05
Beam Rider
561.0
1010.0
1.61c-09
Berzerk
458.0
625.0
2.38c-05
Bowling
21.8
21.5
N/A
Boxing
-28.7
-1.34
1.10e-10
Breakout
17.7
18.1
5.37e-01
Centipede
3130.0
4820.0
1.47e-02
Chopper Command
648.0
957.0
9.89e-04
Crazy Climber
30400.0
35500.0
6.89e-05
Defender
2440.0
2590.0
3.53e-02
Demon Attack
3220.0
5220.0
1.85c-10
Double Dunk
-23.9
-22.5
1.34c-11
Enduro
66.8
153.0
2,45c-15
Fishing Derby
-84.1
-43.9
1. 1.87e-08
Freeway
18.9
21.1
4. 4.23e-03
Frostbite
159.0
33.3
N/A
Gopher
119.0
666.0
9, 9,22c-07
Gravitar
107.0
286.0
2.30c-04
H.E.R.O.
2320.0
431.0
N/A
Ice Hockey
-13.2
-5,29
6.87e-10
James Bond
43.5
372.0
2.39c-06
Kangaroo
534.0
5670.0
4.1 I le-14
Krull
336.0
1780.0
2.90e-05
Kung-Fu Master
13.0
3190.0
5.18e-09
Montezuma's 's Revenge
0.0
N/A
Ms. Pacman
1060.0
1350.0
9.98c-04
Name This Game
2990.0
3680.0
9.95c-06
Phoenix
732.0
2700.0
4.01c-15
Pit Fall
-114.0
-355.0
N/A
Pong
-20.6
-18.3
4. 4.76e-06
Private Eye
110.0
82.4
N/A
Q*
1130.0
2850.0
1.01e-05
River Raid
2200.0
2950.0
4,95c-11
Road Runner
909.0
9240.0
1.54e-13
Robotank
4.44
8.95
3.82e-05
Seaquest
391.0
574.0
5.20e-04
Skinn
-30700.0
-30200.0
1.39c-02
Solaris
515.0
653.0
1.07e-03
Space Invaders
366.0
421.0
4, 4.64e-06
Star Gunner
3510
387.0
3.01e-01
Surround
-8.69
-8,55
4.83c-01
Tennis
-17.5
-18.5
N/A
Time Pilot
1100.0
3570.0
1,43c-15 15
Tutankham
9.4
15.0
2.77c-01
Up N Down
2160.0
3020.0
2.25e-03
Venture
4.0
10.0
1,46e-02
Video Pinball
9890.0
8970.0
N/A
Wizard Wor
292.0
423,0
4. 1.120-03
Yars Revenge
3530.0
6900.0
7. 7.48e-02
Zaxxon
0.0
1020.0
1.08e-02
Mcan (%)
100.0
413.64
N/A
41

---

Atari Vanilla DQN Results:
Table Training rewards comparison between Vanilla DQN agent using the additional SUFT OPE term and the baseline agent without it across 57 Atari games. SUFT surpasses the baseline agent in 49 games, with statistically significant gains in 33 games. Both agents are using L.2 loss.
Game
Baseline Vanilla DQN, buffer 4K
SUFT Vanilla DQN, buffer 4K
p-value
Alien
657.0
736.0
1.37c-02
Amidar
163.0
201.0
1.95e-04
Assault
904.0
1600.0
7.33e-11
Asterix
680.0
1030.0
8.30c-09
Asteroids
408.0
423.0
5.36e-01
Atlantis
10300.0
19000.0
6.54e-08
Bank Heist
29.4
35.2
8.01e-02
Battle Zone
11600.0
12700.0
1.10c-01
Beam Rider
909.0
1440.0
8.25c-08
Berzerk
613.0
653.0
2.35c-03
Bowling
26.5
25.1
N/A
Boxing
0.31
17.8
1.10e-10
Breakout
16.0
18.0
5.85e-01
Centipede
3430.0
3680.0
2.13c-02
Chopper Command
1390.0
1700.0
1.84c-04
Crazy Climber
38700.0
43500.0
1. 1.76e-05
Defender
3290.0
4000.0
5.24c-04
Demon Attack
1490.0
2790.0
6. 6.15e-12
Double Dunk
-2,99
-23.0
N/A
Enduro
156.0
207.0
3.40c-08
Fishing Derby
-76.0
-60.8
4. 1.17e-05
Freeway
24.9
25.2
9.93e-01
Frostbite
272.0
348.0
1.32e-01
Gopher
362.0
654.0
8.02c-06
Gravitar
221.0
224.0
5.56c-01
H.E.R.O.
4590.0
8700.0
1 1,55c-02
Ice Hockey
-10.8
-9.41
2.94e-02
James Bond
402.0
428.0
1. 1.80e-01
Kangaroo
1780.0
4360.0
1. 1.95e-02
Krull
3650.0
4440.0
8. 8.03e-05
Kung-Fu Master
2880.0
2660.0
N/A
Montezuma's 's Revenge
0.0
0.0
N/A
Ms. Pacman
1560.0
1520.0
N/A
Name This Game
3060.0
4170.0
2.79c-09
Phoenix
2110.0
2720.0
7.66c-07
Fall
-35.2
-42,2
N/A
Pong
-15.6
-16.9
N/A
Private Eye
66.4
72.6
5.77e-01
Q*Bert
1510.0
2100.0
1. 1.69c-03
River Raid
2970.0
3420.0
4.8 4.87c-06
Road Runner
16900.0
17700.0
5. 5.00c-01
Robotank
5.62
7.02
1. 1.02e-04
Seaquest
751.0
1120.0
1. 1.08c-07
Skiing
-30300.0
-30200.0
7. 7.50e-02
Solaris
915.0
683.0
N/A
Space Invaders
374.0
424.0
5.24e-04
Star Gunner
829.0
978.0
1.53e-05
Surround
-9.42
-9.33
5.56c-01
Tennis
-19.5
-18.0
2.67c-02
Time Pilot
2880.0
2910.0
5.70c-01
Tutankham
53.5
106.0
1.03e-07
Up N Down
4640.0
6970.0
8.16e-05
Venture
9.0
18.0
1.29e-01
Video Pinball
13700.0
14200.0
2. 2.84c-01
Wizard of Wor
365.0
424.0
1.79c-03
Yars Revenge
7260.0
10100.0
1.42c-07
Zaxxon
112.0
932.0
1.96c-01
Mean (%)
100.0
228.12
N/A
42

---

Alien
Amidar
Assault
Asterix
Asteroids
Reward Reward
Atlantis
Bank Heist
Battle Zone
Bearn Rider
Berzerk
Bowling
Boxing
Breakout
Centipede
Chopper Command
Reward
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
Reward
2010
2014
201
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
Reward Reward
rike Krull
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Kung- Fu Master
Montezuma's a Revenge
Ms. Pacman
Name This Game
Phoenix
Reward Reward
Pit Fall
Pong
Private Eye
Bert
River Raid
Road Runner
Robotank
Seaquest
Skiing
Solaris
Reward Reward Reward
200
20m
2011 Tennis
2006
Space Invaders
Star Gunner
Surround
Time Pilot
2014
2006
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
1100
Yars Revenge
Zaxxon
steps
steps
steps
Reward
2.m
# steps
# steps
SUFT
Baseline
Figure 12: Learning curves comparison between Vanilla DQN agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. The shaded area is the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using Le loss.
43

---

Fishing Derby
Zexxon
Boxing
Gopher
Defender
Kangaroo
H.E.R.O.
Tutankham
Star Gunner
Assault
Demon Attack
Name This Game
Beam Rider
Seaquest
Yars Revenge
Asterix
Up N Down
Chopper Command
Phoenix
QeBtrt
Robotank
Krull
Bank Heist
Frostbite
Tennis
Enduro
Niver
Space Invaders
Amidar
Contipede
Crazy Climber
Breakout
Battle Zone
Berzerk
James Bond
Gravitar
Road Runner
Freeway
Pit Fall
Kung Fu Master
+10%
+100%
Pong
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Log Improvement (%)
SUFT
Baseline
Figure 13: Log-scaled reward improvements comparison between Vanilla DQN agent using the additional SUFT OPE term and the baseline agent without it across 43 valid Atari games. SUFT outperforms baseline agent in 39 out of 43 valid games, achieving reward improvement of over 100% in 25.6% of them and over 10% in 79.1% of them. The red line indicates 10% improvement, and the green line represents 100% improvement. Both agents are using L2 loss.
44

---

Atari Double DQN Results:
Table 10: Training rewards comparison between the Double DQN agent using the additional SUFT OPE term and the baseline agent without across 57 Atari games. SUFT surpasses the baseline agent 48 games, with statistically significant gains 37 games. Both agents are using L2 loss.
Game
Baseline DDQN, buffer 4K
SUFT DDQN, buffer 4K
p-value
Alien
773.0
806.0
7.88c-01
Amidar
152.0
177.0
8.27e-03
Assault
781.0
1440.0
3.84e-12
Asterix
676.0
939.0
1.03e-08
Asteroids
357.0
365.0
1.30e-01
Atlantis
13900.0
18100.0
1.14e-03
Bank Heist
29.2
36.1
1.30e-03
Battle Zone
10900.0
14000.0
1.16c-04
Beam Rider
719.0
1230.0
1 1.03e-12
Berzerk
567.0
668.0
5.01c-08
Bowling
27.8
27.0
N/A
Boxing
-1.73
18.4
8. 8.10e-11
Breakout
19.3
16.3
N/A
Centipede
3390.0
3430.0
6, 6,14e-01
Chopper Command
1460.0
1870.0
4.47e-05
Crazy Climber
41200.0
42200.0
3,93e-01
Defender
2980.0
4140.0
8.61e-10
Demon Attack
369.0
2240.0
3.09e-15
Double Dunk
-23.6
-23,4
2.41e-02
Enduro
143.0
195.0
3.43e-13
Fishing Derby
-81.4
-67.8
4.39e-07
Freeway
25.2
21.3
N/A
Frostbite
344.0
265.0
N/A
Gopher
595.0
658.0
4.17c-01
Gravitar
100.0
134.0
2.00 2.00c-02
H.E.R.O.
4020.0
8870.0
2.06c-03
Ice Hockey
-7.7
-8.15
N/A
James Bond
316.0
416.0
7.84e-04
Kangaroo
2430.0
3280.0
1. 1.94e-01
Krull
3830.0
4400.0
9. 9.09e-05
Kung-Fu Fu Master
2470.0
3680.0
3.30e-02
Montezuma's 's Revenge
0.0
0.0
N/A
Ms. Pacman
1560.0
1650.0
3. 2.26e-02
Name This Game
3140.0
4140.0
7.7 7.71
Phocnix
1080.0
2220.0
1 1.05e-10
Pit Fall
-14.2
-27.5
N/A
Pong
-15.9
-15.8
6.54e-01
Private Eye
-45.2
49.4
8.98e-01
QeBert
1260.0
1960.0
6.31e-05
River Raid
2960.0
3420.0
1.56c-07
Road Runner
15700.0
17000.0
4,29c-01
Robotank
6.43
7.47
6.65e-03
Seaquest
474.0
938.0
9.91e-12
Skiing
-30300.0
-30200.0
2.87e-01
Solaris
1620.0
599.0
N/A
Space Invaders
377.0
446.0
3. 3.52e-05
Star Gunner
970.0
1100.0
5. 5.04e-06
Surround
-9.62
-9.64
N/A
Tennis
-23.6
-20.3
7.08c-05
Time Pilot
2130.0
2640.0
3.75c-05
Tutankham
64.1
107.0
1.56e-05
Up N Down
3940.0
6950.0
1.23e-04
Venture
18.0
4.54e-02
Vidco Pinball
13400.0
15400.0
1.23e-02
Wizard of Wor
349.0
447.0
3.76e-05
Yars Revenge
8610.0
10400.0
2.31e-06
Zaxxon
4.0
10.0
1.71e-01
Mean (%)
100.0
226.91
N/A
45

---

Allen
Amidar
Assault
Asterix
Asteroids
Reward Reward
Atlantis
Bank Heist
Battle Zone
Bearn Rider
Berzerk
Bowling
Boxing
Breakout
Centipede
Chopper Command
Reward
Crazy Climber
Defender
Demon Attack
Double Dunk
Enduro
Reward
2010
D0E 2014
200
aict
201
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
Reward Reward
7M
H.E.R.O.
Ice Hockey
James Bond
Kangaroo
Krull
Kung-Fu Fu Master
Montezuma's a Revenge
Ms. Pacman
Name This Game
Phoenix
Reward Reward
Pit Fall
Pong
Private Eye
Bert
River Raid
Xie
Road Runner
Robotank
Seaquest
Skiing
Solaris
Reward
Xx
2006
2011 Tennis
2006
Space Invaders
Star Gunner
Surround
Time Pilot
Reward
106
2006
Tutankham
Up N Down
Venture
Video Pinball
Wizard of Wor
Reward Reward
500
Yars Revenge
Zaxxon
steps
steps
steps
# steps
# steps
SUFT
Baseline
Figure 14: Learning curves comparison between the Double DQN agent using the additional SUFT OPE term and the baseline agent without across Atari games. The shaded area the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using L2 loss.
46

---

Demon Attack
Defender
Phoenix
H.E.R.O.
Seaquest
Assault
Beam Rider
Up N Down
Tutankham
QeBt
Chopper Command
Kung-Fu Master
Asterix
Star Gunner
Bank Heist
Battle Zone
Kangaroo
Enduro
Space Invaders
James Bond
Revenge
River Raid
Berzerk
Krull
Gopher
Amidar
Road Runner
Alien
Crazy Climber
Centipede
Pong
+10%
+100%
Pit Fall
Ice Hockey
freeway
Breakout
Frastbite
0.0
0.5
1.0
1.5
2.0
2.5
Log Improvement (%)
SUFT
Baseline
Figure 15: Log-scaled reward improvements comparison between the Double DQN agent using the additional SUFT OPE term and the baseline agent without it across 40 valid Atari games. SUFT outperforms baseline agent in 35 out of 40 valid games, achieving reward improvement of over 100% in 20% of them and over 10% in 72.5% of them. The red line indicates 10% improvement, and the green line represents 100% improvement. Both agents are using L2 loss.
47

---

Table 11: Training rewards comparison between our causal bound optimization method and an inertia regularizer. We compare Double DQN agent with the additional SUFT OPE term to one using an inertia regularizer across 57 Atari games. While SUFT significantly improves the baseline agent's s performance with 226.91 % mean reward ratio, the inertia regularizer harms the baseline agent's performance with only 83.21 % mean reward ratio. All agents are using L2 loss and 4K buffer size.
Game
Baseline DDQN
Inertia Regularizer DDQN
SUFT DDQN
Alien
773.0
762.0
806.0
Amidar
152.0
144.0
177.0
Assault
781.0
670.0
1440.0
Asterix
676.0
666.0
939.0
Asteroids
357.0
278.0
365.0
Atlantis
13900.0
3970.0
18100.0
Bank Heist
29.2
21.8
36.1
Battle Zone
10900.0
11200.0
14000.0
Beam Rider
719.0
707.0
1230.0
Berzerk
567.0
628.0
668.0
Bowling
27.8
28.8
27.0
Boxing
-1.73
-3.91
18.4
Breakout
19.3
8.85
16.3
Centipede
3390.0
3840.0
3430,0
Chopper Command
1460.0
1120.0
1870.0
Crazy Climber
41200.0
42800.0
42200.0
Defender
2980.0
2650.0
4140.0
Demon Attack
369.0
191.0
2240.0
Double Dunk
-23.6
23.6
-2344
Enduro
143.0
107.0
195.0
Fishing Derby
-81.4
-84.9
-67.8
Freeway
25.2
21.2
21.3
Frostbite
344.0
273.0
265.0
Gopher
595.0
376.0
658.0
Gravitar
100.0
138.0
134,0
H.E.R.O.
4020.0
5060.0
8870.0
Ice Hockey
-7.7
-6.6
-8.15
James Bond
316.0
186.0
416.0
Kangaroo
2430.0
1110.0
3280.0
Krull
3830.0
2700.0
4400.0
Kung-Fu Master
2470.0
1520.0
3680.0
Montezuma's ' Revenge
0.0
0.0
0.0
Ms. Pacman
1560.0
1510.0
1650.0
Name This Game
3140.0
2420.0
4140.0
Phoenix
1080.0
640.0
2220.0
Pit Fall
-14.2
25.55
-27.5
Pong
-15.9
-17.3
-15.8
Private Eye
-45.2
-11.9
49.4
QeBett
1260.0
1180.0
1960.0
River Raid
2960.0
2550.0
3420.0
Road Runner
15700.0
11000.0
17000.0
Robotank
6.43
6,51
7.47
Seaquest
474.0
306.0
938.0
Skiing
-30300.0
26550.00
-30200.0
Solaris
1620.0
1860.0
$99.0
Space Invaders
377.0
400.0
446.0
Star Gunner
970.0
980.0
1100.0
Surround
.9622
-9.37
-9.64
Tennis
-23.6
-15.1
-0.3
Time Pilot
2130.0
1920.0
2640.0
Tutankham
64.1
75.8
107.0
Up N Down
3940.0
2930.0
6950.0
Venture
6.0
4.0
18.0
Video Pinball
13400.0
12400.0
15400.0
Wizard Wor
349.0
215.0
447.0
Yars Revenge
8610.0
7990.0
10400.0
Zaxxon
4.0
12.0
10.0
Mean (%)
100.0
83.21
226.91
48

---

D.7 MuJoCo Five Environments Benchmark
MuJoCo PPO Results:
Table 12: Training rewards comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without it across 5 MuloCo environments. SUFT surpasses the baseline agent in 4 of 5 environments, with statistically significant gains in 2 environments. Both agents using Le loss.
Game
Baseline PPO
SUFT PPO
p-value
Ant
1110.0
1570.0
1.65e-02
Half Cheetah
1520.0
1540.0
7.28e-01
Hopper
1880.0
2120.0
3.16c-03
Humanoid
464.0
464.0
N/A
Walker2d
1850.0
1860.0
8.33e-01
Mcan (%)
100.0
111.21
N/A
Ant
Half Cheetah
Hopper
Reward
2000
1000
1000
0.0
3.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
# steps
Humanoid
Walker2d
Reward
2000
200
# steps
le5
# steps
166
SUFT
Baseline
Figure 16: Learning curves comparison between the PPO agent using the additional SUFT OPE term and the baseline agent without it across 5 MuJoCo environments. The shaded area is the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using L2 loss.
49

---

MuJoCo SAC Results:
Table 13: Training rewards comparison between the SAC agent using the additional SUFT OPE term and the baseline agent without it across 5 MuJoCo environments. SUFT surpasses the baseline agent in 4 out of 5 environments, with statistically significant gains in 3 environments. Both agents are using b2 loss.
Game
Baseline SAC, buffer 4K
SUFT SAC, buffer 4K
p-value
Ant
1250.0
1020.0
N/A
Half Cheetah
4440.0
6290.0
3.94e-01
Hopper
978.0
1910.0
1.46e-06
Humanoid
638.0
1100.0
2. 2.166-02
Walker2d
316.0
1680.0
8. 8.16c-11
Mean (%)
100.0
224.53
N/A
Ant
Half Cheetah
Hopper
Reward
2000
5000
1000
2500
0.0
0.2
1.0
0.0
0.2
1.0
0.6
1.0
le6
1e6
# steps
1e6
Humanoid
Walker2d
2000
Reward
1000
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#
steps
# steps
SUFT
Baseline
Figure 17: Learning curves comparison between the SAC agent using the additional SUFT OPE term and the baseline agent without it across MuJoCo environments. The shaded area is the 10% and 90% percentiles with linear interpolation from the 10 different seeds' runs. Both agents are using L2 loss.
50

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we highlight the main problem, our method, and the empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 4 we describe the assumptions of our work, and in particular Assumption [.. which does not apply to the L2 loss. In our experiments (Section 5). we
evaluate method L2 loss.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
51

---

Answer: [Yes]
Justification: In Section we detailed the relevant assumptions for all theoretical results. We provide all proof's in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We use open-source environments and an RL library as detailed the Imple- mentation Details paragraph in Section Furthermore, we fully detailed all the experimental information in the Training Details paragraph in Section Throughout Section and Ap- pendix B we thoroughly detail how to implement our method. We also provide all the evaluation formulas in Appendix In addition, we also provide all of the code as supple- mentary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
52

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the code supplementary material and fully detail the imple- mentation and training details in Section 5 We also provide all the evaluation formulas in
Appendix D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co... cesipss.c/puulic/guuuuc httpa///gguideee/eeeuuubbiaiii
SttSSubnisiioooooiiooo.oooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We fully detailed all the experimental information in the Training Details and Implementation Details paragraphs in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are independently trained for 10 different random seeds per agent-environment configuration, and we take the median result. We report mean performances for of the environments per domain. We also report environment-specifcc statistical significance results by conducting t-test.
53

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We give details on the computational time and resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp////eerrie...cc/p/p///eeas
Answer: [Yes]
Justification:
Our research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
54

---

Justification:
This paper proposes a method to improve sample efficiency in reinforcement learning. As foundational research, this work has no direct paths to negative societal impacts besides those that can arise from improved reinforcement learning agents.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release high-risk data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all creators whose code we used in our experiments in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
55

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code as supplementary material and include a README file that explains how to run it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
56

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper uses LLMs only for editing purposes and conforms the NeurIPS LLM policy.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
57


Reviews:
[{"review_id": "EmrjPu28VC", "reviewer": "Reviewer_zGCa", "summary": "This paper proposes a causal approach to an upper bound on-policy loss in RL with off-policy data. The resulting upper bound can be integrated into any on-policy RL algorithm. The authors use empirical evaluations to show that the resulting SUFT version of RL algorithms can achieve better performance with low sample/computational overhead.", "questions": "Since Q-learning is inherently an off-policy method, the arguments presented in Section 4.1 are unclear to me. In particular, it is surprising that the SUFT variant leads to such a substantial performance improvement. Could the authors clarify this discrepancy and provide further explanation?\n\nThe theoretical analysis relies on a form of the triangle inequality for the loss function. While the $L_1$ loss satisfies this assumption, the more commonly used $L_2$ loss does not. Nevertheless, SUFT appears to perform even better under the $L_2$ loss in the empirical results, which seems inconsistent with the theoretical framework. I encourage the authors to provide intuitive explanations for this observation.", "limitations": "The authors do not explicitly state the limitations of their work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n* The paper is overall well-written and easy to follow.\n* The paper is well-motivated, addressing the long-standing challenge in reinforcement learning of using inexpensive off-policy data in place of costly on-policy data.\n* The authors introduce tools from causal inference to RL. The proposed methodology looks natural and intuitive.\n* The empirical evaluations look sound.\n\nWeaknesses:\n* The empirical evaluation introduces a metric termed the reward improvement ratio. This metric appears unconventional and may potentially overstate the improvements achieved by the proposed method.\n* I think the author should try more recent baselines on Atari games and see whether the proposed method works for them. For example the baselines in https://arxiv.org/abs/2112.04145.\n\nSince I am not very familiar with the literature on causal inference, I choose to assign a low confidence score to my evaluation.", "was_revised": true, "final_justification": "After reviewing the other reviewers’ comments and the authors’ rebuttal, I find that my two primary concerns—evaluation metrics and the inconsistency between theory and empirical findings—have been satisfactorily addressed. I therefore maintain my positive assessment of this paper.", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. \nBelow we address all your questions and concerns:\n\nFirstly, we want to address a misunderstanding in your review.\nIn the summary, you wrote that you understand that our method is applicable to any **on-policy** RL algorithm. Then, in your first question, you asked how our method leads to a substantial performance improvement in **off-policy** Q-learning algorithms.\nIn our paper, we presented a method to enhance any DRL **off-policy** agent with a V or a Q-value network.\n\nSince the off-policy method is a compromise of allowing the agent to act off-policy to improve sample efficiency by leveraging past experiences stored in the replay buffer, it can lead to reduced learning stability due to a mismatch between the behavior and target policies.\nOur method addresses this divergence and bounds the on-policy loss using the standard off-policy loss, along with an additional OPE term, which is the estimated treatment effect.\nTherefore, our method demonstrates significant improvements for off-policy agents, as it bridges the gap between on-policy and off-policy methods using our causal bound.\nNote that our method is also applicable to PPO, an on-policy agent, which benefits from our method due to the divergence between the behavior and target policies during its epoch optimization.\n\n\n$\\underline{L_2 \\text{ loss:}}$\n\nAs mentioned in the paper, the theoretical result relies on Assumption 4.1, which doesn't hold for the $L_2$ loss. We used the $L_2$ loss in our experiments to compare our method with standard DRL agents.\n\nWhile the $L_2$ loss does not theoretically hold this assumption, following your review and to provide an intuitive explanation for the significant improvements in our results, we empirically tested the validity of the $L_2$ loss for Assumption 4.1 using synthetic and domain-specific data.\n\nFor the synthetic data, we created a short Python script (see the code below) that randomly generates 1 billion samples for $x, y, x', y'$, and checks if the inequality holds for $L_2$ loss. \nThe results show that Assumption 4.1 is valid for 82% of randomly sampled values using the $L_2$ loss.\n\nIn addition, we tested Assumption 4.1 using the $L_2$ loss for the Atari games domain using the DQN agent.\nSince we are using an off-policy agent, the left side of our causal bound (Theorem 4.10) is not computable, as we cannot sample observations on-policy (this is why we optimize on the upper bound in our method).\nNevertheless, we return to the part of our proof where we apply Assumption 4.1 (line 595), considering only the parts where the samples are off-policy for both the left and right sides of the inequality in Equations 10 and 11 (using $q_0=1$). Additionally, to approximate the true values of $y_0$ and $y_1$, we use the standard Bellman equation for Q.\n\nWith this setup, we ran the 57 Atari games benchmark using 400K timesteps and a batch size of 32 samples, and calculated the elements of the inequality.\nFor a sanity check, we use the $L_1$ loss to verify that it holds 100% of the time, which it does indeed.\nThe results show that while the assumption may not theoretically hold for $L_2$, it holds in 79% of the cases.\n\nTo summarize, although Assumption 4.1 does not hold for $L_2$ loss, in practice, it holds for the vast majority of the optimization process, which explains the performance improvements of our method in the $L_2$ case.\n\nAdditionally, reviewer MtKE suggested a controlled experiment to isolate the effect of our causal mechanism, which will prove that our method's significant improvements are due to the causal bound optimization, and not just a regularization technique.\nHe suggested adding a regularizer to the standard temporal difference loss function in the form of $(Q_{current}-Q_{target})^2$, where $Q_{target}$ is the slowly moving target network, as done in double DQN.\n\nWe conducted this experiment on 5 Atari games using the DDQN agent on 10 different random seeds.\nThe results are below:\n\n| GAME | DDQN | DDQN with target network regularizer | DDQN using SUFT |\n|-------------|-------|---------------------------|-------------|\n| Assault | 781 | 670 | **1,440** |\n| Asterix | 676 | 666 | **939** |\n| Boxing | -1.73 | -2.41 | **18.4** |\n| Demon Attack | 369 |-17.3 | **2240** |\n| James Bond | 316 | 186 | **416** |\n\nThe results above are conclusive and demonstrate that this kind of regularization harms the actual DDQN performance. It proves that our method has a significant impact due to the causal bound optimization, where the old Q-network values used for the OPE term are of the behavior policy used to generate the experiences, therefore mitigating the divergence between the on-policy and off-policy methods. This shows that our method is not just a regularizer that promotes stability, but a causal bound optimization.\nWe will conduct this experiment on the entire 57 games benchmark and add the results to the appendix.\n\n$\\underline{\\text{Metric:}}$\n\nIn our paper, we used several evaluation metrics and techniques to ensure the robustness and reliability of our results.\nWe conducted controlled experiments with different agents that include our proposed term in comparison to identical baseline agents without it. These controlled experiments were intentionally made in order to isolate and measure the direct impact of our method. To measure our method’s improvement, we compared our agent's reward to the baseline reward using the ratio formula, similar to the one used in [1], which is a highly cited work from a top-tier venue (see Figure 4 in [1]).\n\nWhen calculating the mean reward ratio formula, we have considered only the valid environments where both the numerator and denominator are greater than zero, ensuring the validity of the ratio formula.\nFollowing your review, we will address the areas where the denominator is lower than 1, which leads to large values. We will adjust the valid environment restriction to consider only environments where both the numerator and denominator are greater than 1, thereby preventing arbitrarily large values.\nThe new mean reward ratios for the different agents are:\n\n| Agent | New reward mean ratio | Old reward mean ratio |\n|------|---------------------------|-------------|\n| DQN | 383.19% |  2,427.06% |\n| Vanilla DQN | 228.12% | 230.49% |\n| DDQN | 226.91% | 260.03% |\n| PPO | 197.34% | 203.21% |\n\nThank you for bringing this to our attention, and your intuition about this metric was correct. \nWith our new adjustment to this metric, we are excluding the impact of cases where baseline rewards are only slightly better than the random rewards, and guarantee the effectiveness of this metric.\nThe results with the new adjustment still demonstrate that our method continues to have a significant impact on performance while excluding the slightly better than random environments from the mean calculation, thereby strengthening the effectiveness of the evaluation metric and providing a more accurate assessment of our method’s impact.\n\nFurthermore, please note that the original manuscript contains evaluated results using the standard human-normalized score in an ablation study, as shown in Table 5. In the revised version of the paper, we will relocate this analysis from the appendix to the main body of the paper to strengthen our results.\n\nOverall, we would like to thank you for your feedback. We believe that we have addressed all your questions and concerns and that our paper is now even better.\n\nPython code for the synthetic data:\n\n`import numpy as np`\n\n`size = 1_000_000_000`\n\n`x = np.random.normal(loc=0, scale=1, size=size)`\n\n`y = np.random.normal(loc=0, scale=1, size=size)`\n\n`x_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`y_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`lhs = (x - y)**2 - (x_tag - y_tag)**2`\n\n`rhs = (x - x_tag)**2 + (y - y_tag)**2`\n\n`false_counter = np.sum(lhs > rhs)`\n\n`print(\"Success Percentage:\", 100 - false_counter / size * 100)`\n\nReferences:\n\n[1] Kaiser, Ł., Babaeizadeh, M., Miłos, P., Osiński, B., Campbell, R. H., Czechowski, K., ... & Michalewski, H. (2020). MODEL BASED REINFORCEMENT LEARNING FOR ATARI. In 8th International Conference on Learning Representations, ICLR 2020.‏"}, {"review_id": "4o61sVLMzh", "reviewer": "Reviewer_MtKE", "summary": "This paper presents a new algorithm for improving the sample efficiency and performance of DRL agents. The algorithm is based on a theoretical result derived from the causal inference literature. Unlike most work that bounds the counterfactual (off-policy) loss, the authors here propose bounding the unobserved factual (on-policy) loss instead, using off-policy samples. This bound is made of the usual off-policy loss plus a term that measures the treatment effect between the target policy and past policies. In practice, the authors suggest to cleverly recycle the value network outputs, which are normally just thrown away after action selection, and instead store them in the replay buffer. The authors conducts extensive experiments on DQN, PPO, and SAC agents across many Atari and MuJoCo environments, demonstrating clear improvements in reward and convergence speed, and a significant reduction in the required replay buffer size.", "questions": "* The bound has the  $\\delta$ term which is correctly ignored during optimization, since it doesn't depend on the agent's parameters. However it would be good to add a short discussion on potential looseness of the bound. If  $\\delta$ is very large, the bound might be too loose to be useful. Your strong results suggest this isn't a problem, but it would be good to comment on it. In particular, it would be interesting to plot the magnitude of this term over the optimization.\n\n* Coming back to my last point in the \"Weaknesses\": the SUFT loss term, $(V_{behavior} − V_{target} )^2$ , is functionally very similar to known regularization techniques that performs some form of value function smoothing (e.g., in PPO v2). To isolate the effect of your proposed causal mechanism, would it make sense adding a control experiment? For example, it would be interesting to check whether the benefit comes from simply adding inertia to the value function, preventing it from changing too rapidly, rather than tracking a specific behavioral policy's value. One could e.g. add a term ot the TD loss, $L_{total} = L_{TD} + \\lambda_{inertia} \\cdot (Q_{current} - Q_{target})^2$ where $Q_{target}$ is the slowly moving target Q network used to compute the target value in the TD loss, as is usually done in DQN.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n* The paper tackles the crucial problem of sample efficiency in DRL. The method is surprisingly simple, can be used orthogonally with many other algorithms, and its empirical results are exceptionally strong across many different agents and benchmarks. If these results hold up, the method's ability to improve performance and cut down on resource needs would be a major step for the field.\n\n* I also found the main theoretical idea of looking at the problem as a causal bound on the factual loss, instead of the counterfactual loss, to be really original.\n\n* The experimental validation is high quality and very thorough. Using 10 random seeds, all 57 Atari games and 5 MuJoCo environments, with several different DRL agents (DQN, PPO, SAC) really convincingly show the robustness of the advantage of the algorithm.\n\n# Weaknesses\n\n* I really think the paper needs some pseudocode for how to apply the bound to DQN and PPO. Currently, the reader has to guess some of the key implementation details. For instance, it is not clear to me how the existing PPO critique loss would map to the proposed upper bound.\n\n* There's a bit of a gap between the formal theory and the experiments. The proof for the causal bound needs an L1 loss, while the main experiments with the most impressive results are all using an L2 loss. The authors acknowledge this, but this raises the question of why the method is so effective with L2 loss, e.g. it is possible that the additional term act as a regularizer that promotes stability, which benefits the optimization independently of the bound itself", "was_revised": true, "final_justification": "The performance gain is impressive, and the method simple. The authors addressed my concerns and the proposed revision is satisfactory.", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. \nBelow we address all your questions and suggestions:\n\nFollowing your review, you suggested performing a controlled experiment to isolate the effect of our causal mechanism, which will prove that our method's significant improvements are due to the causal bound optimization, and not just a regularization technique.\n\nAs you suggested, we conducted such an experiment and added a regularizer to the standard temporal difference loss function in the form of $(Q_{current}-Q_{target})^2$, where $Q_{target}$ is the slowly moving target network, as done in double DQN.\nWe conducted this experiment on 5 Atari games using the DDQN agent on 10 different random seeds. The results are below:\n\n| GAME | DDQN | DDQN with target network regularizer | DDQN using SUFT |\n|-------------|-------|---------------------------|-------------|\n| Assault | 781 | 670 | **1,440** |\n| Asterix | 676 | 666 | **939** |\n| Boxing | -1.73 | -2.41 | **18.4** |\n| Demon Attack | 369 |-17.3 | **2240** |\n| James Bond | 316 | 186 | **416** |\n\nThe results above are conclusive and demonstrate that this kind of regularization harms the actual DDQN performance. It proves that our method has a significant impact due to the causal bound optimization, where the old Q-network values used for the OPE term are of the behavior policy used to generate the experiences, therefore mitigating the divergence between the on-policy and off-policy methods. \nThis shows that our method is not just a regularizer that promotes stability, but a causal bound optimization. We will conduct this experiment on the entire 57 games benchmark and add the results to the appendix. \n\nHaving said that we also performed an analysis that explains the success using the $L_2$ loss.\nAs mentioned in the paper, the theoretical result relies on Assumption 4.1, which doesn't hold for the $L_2$ loss.  \nWe used the $L_2$  loss in our experiments to compare our method with standard DRL agents. \n\nWhile the $L_2$ loss does not theoretically hold this assumption, in order to provide an intuitive explanation for the significant improvements in our results, we empirically tested the validity of the $L_2$ loss for Assumption 4.1 using synthetic and domain-specific data.\nFor the synthetic data, we created a short Python script (see the code below) that randomly generates 1 billion samples for $x, y, x', y'$, and checks if the inequality holds for $L_2$ loss. \nThe results show that Assumption 4.1 is valid for 82% of randomly sampled values using the $L_2$ loss.\n\nIn addition, we tested Assumption 4.1 using the $L_2$ loss for the Atari games domain using the DQN agent.\nSince we are using an off-policy agent, the left side of our causal bound (Theorem 4.10) is not computable, as we cannot sample observations on-policy (this is why we optimize on the upper bound in our method).\nNevertheless, we return to the part of our proof where we apply Assumption 4.1 (line 595), considering only the parts where the samples are off-policy for both the left and right sides of the inequality in Equations 10 and 11 (using $q_0=1$). Additionally, to approximate the true values of $y_0$ and $y_1$, we use the standard Bellman equation for Q.\n\nWith this setup, we ran the 57 Atari games benchmark using 400K timesteps and a batch size of 32 samples, and calculated the elements of the inequality.\nFor a sanity check, we use the $L_1$ loss to verify that it holds 100% of the time, which it does indeed.\nThe results show that while the assumption may not theoretically hold for $L_2$, it holds in 79% of the cases.\n\nTo summarize, although Assumption 4.1 does not hold for $L_2$ loss, in practice, it holds for the vast majority of the optimization process, which explains the performance improvements of our method in the $L_2$ case.\n\n$\\underline{\\delta \\text{ term:}}$\n\nThank you for your suggestion to include a discussion on the potential looseness of the bound, specifically regarding the $\\delta$ term, which we ignore during optimization since it is independent of $Q$. You also proposed plotting the magnitude of this term over the optimization.\n\nThe $\\delta$ term involves the true values $y_0, y_1$ (as in Theorem B.12 line 592), which are not observable during training. However, we can approximate these values using the standard Bellman equation for $Q$.\nWe conducted an experiment on 20 Atari games using the DQN agent, plotting the values of the standard off-policy loss, the SUFT OPE term, and the approximation for the $\\delta$ term over the optimization.\n\nFollowing this experiment, we observed that in all of the environments, the $\\delta$ term remained consistently an order of magnitude **lower** than the off-policy loss, and on the same order of magnitude as the SUFT OPE term. \nThis supports the choice of ignoring the $\\delta$ term during optimization, as it does not made the bound too loosen to be useful.\nWe will conduct this experiment on the entire 57 games benchmark, and add the plots showing the $\\delta$ term and the off-policy loss magnitude, along with a short discussion about this term, to the appendix. \n\nIn addition, we accept your suggestion to add pseudocode for how to apply the causal bound in both DQN and PPO, and we will add this to the appendix.\n\nRegarding the integration of the SUFT term into PPO’s critic loss, although PPO is an on-policy algorithm, it performs multiple epochs of optimization over the same buffer of data, leading to a divergence between the behavior and target policies. This divergence is the reason PPO benefits from our method.\n\nIn the PPO paper [1], equation 9 defines the overall PPO loss, where the critic component is expressed as: $L_t^{VF}(\\theta)=(V_{\\theta}(s_t)-V_t^{targ})^2$. \nOur additional SUFT term is implemented to the critic loss in the form of $L_t^{VF}(\\theta)=(V_{\\theta}(s_t)-V_t^{targ})^2 + \\lambda_{TF}(V_{\\theta}(s_t)-V_{\\theta_{old}}(s_t))^2$.\nWhere $V_{\\theta}(s_t)$ is the value outputs of the target policy, and $V_{\\theta_{old}}(s_t)$ is the value outputs of the behavior policy used to generate the experiences.\n\nNotably, during the first epoch, before optimizing the network, the behavior and target policies are identical, meaning that our additional term will be 0.\nAfter the first optimization step, the target policy network has been optimized, and now it differs from the behavior policy, meaning that our additional OPE term now quantifies the estimated treatment effect between the target and behavior policies.\n\nIn addition, the $V_{\\theta_{old}}(s_t)$ is already calculated during the interaction with the environment and the calculation of the $V_t^{targ}$ for each sample collection. This means that our method reuses the old network outputs and stores them along with the sample collection to use them in the calculation of the SUFT OPE term in a similar manner to the DQN implementation.\n\nWe would like to thank you for your positive feedback and suggestions. We believe that we have addressed all your questions and suggestions and that our paper is now even better.\n\n\nPython code for the synthetic data:\n\n`import numpy as np`\n\n`size = 1_000_000_000`\n\n`x = np.random.normal(loc=0, scale=1, size=size)`\n\n`y = np.random.normal(loc=0, scale=1, size=size)`\n\n`x_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`y_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`lhs = (x - y)**2 - (x_tag - y_tag)**2`\n\n`rhs = (x - x_tag)**2 + (y - y_tag)**2`\n\n`false_counter = np.sum(lhs > rhs)`\n\n`print(\"Success Percentage:\", 100 - false_counter / size * 100)`\n\nReferences:\n\n[1] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017"}, {"review_id": "NC4u1eHiZo", "reviewer": "Reviewer_1ska", "summary": "This paper introduces a causal bound on the factual loss, derived from the potential outcomes framework, to enhance DRL sample efficiency. The method yields substantially higher rewards while drastically reducing the required size of the experience replay buffer at a negligible cost.", "questions": "- What is the main difficulty in establishing the theoretical result (Theorem 4.10) for the L2 loss?\n- Could you discuss the soundness of the metric used? Specifically, when the baseline performs only marginally better than a random guess, how can the effectiveness of the ratio be guaranteed?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- Leveraging causal information to enhance sample efficiency in DRL is a significant research direction.\n- The experimental results are extensive and validated across various standard benchmarks.\n- Theoretical results are provided to ground the proposed approach and verify its effectiveness.\n\n**Weaknesses:**\n- The paper is **not well-structured**, which negatively impacts readability. For example, the term \"SUFT\" should be explained when it first appears, not pages later. Similarly, the explanation for Figure 1 is located far from the figure itself (Figure on pp. 2, explanation on pp. 9). Meanwhile, the introduction lacks detail on how the causal bound is incorporated into the DRL framework. It also fails to provide sufficient insight into why adding the proposed term to the loss function yields such significant improvements over conventional methods.\n- Several figures and equations **require formatting adjustments**. For instance, the font size in Figures 4 and 10 is too large, and the equation in line 752 is too long and runs outside the margin. Pages 5 and 6 are overly dense with definitions, many of which could be moved to an appendix. This space in the main body would be better used to detail why the challenge is difficult for previous work to handle and to provide more intuition about the proposed approach.\n- **Theory-Experiment Disconnect**. The main theoretical result (Theorem 4.10) relies on Assumption 4.1, which the authors admit does not hold for the L2 loss—the loss used in their main experiments. This undermines the validity of their theoretical justification. Without the assumption, the proposed method reduces to a heuristic regularization technique, and the claim of optimizing a causal bound does not hold. The brief L1-loss results in the appendix do not adequately support the main claims. What's the main difficulty of providing the theoretical result for the L2 loss?\n- **Fragile and Potentially Misleading Metric**. The reported “2,427% improvement” is based on a reward ratio formula, i.e., (SUFT - Random) / (Baseline - Random) in line 755, which is highly sensitive *when the baseline performs only slightly better than random*. This can produce arbitrarily large values, exaggerating the method’s effectiveness and obscuring its true impact compared to more standard metrics like absolute or human-normalized scores. The soundness of the used metric should be discussed.", "was_revised": true, "final_justification": "While the empirical results still suggest that the theorem might not hold under the $L_2$ setting, most of my concerns have been mitigated, and I am willing to adjust my score accordingly.", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. \nBelow we address all your questions and concerns:\n\n$\\underline{L_2 \\text{ loss:}}$\n\nThe difficulty in establishing the theoretical results for the $L_2$ loss is that in our proof, we rely on Assumption 4.1, which is not satisfied in the $L_2$ case. Having said that, the following analysis shows that in practice, it is satisfied in the vast majority of the optimization process.\n\nFollowing your review and to provide an intuitive explanation for the significant improvements in our results, we empirically tested the validity of the $L_2$ loss for Assumption 4.1 using synthetic and domain-specific data.\n\nFor the synthetic data, we created a short Python script (see the code below) that randomly generates 1 billion samples for $x, y, x', y'$, and checks if the inequality holds for $L_2$ loss. \nThe results show that Assumption 4.1 is valid for 82% of randomly sampled values using the $L_2$ loss.\n\nIn addition, we tested Assumption 4.1 using the $L_2$ loss for the Atari games domain using the DQN agent.\nSince we are using an off-policy agent, the left side of our causal bound (Theorem 4.10) is not computable, as we cannot sample observations on-policy (this is why we optimize on the upper bound in our method).\nNevertheless, we return to the part of our proof where we apply Assumption 4.1 (line 595), considering only the parts where the samples are off-policy for both the left and right sides of the inequality in Equations 10 and 11 (using $q_0=1$). Additionally, to approximate the true values of $y_0$ and $y_1$, we use the standard Bellman equation for Q.\n\nWith this setup, we ran the 57 Atari games benchmark using 400K timesteps and a batch size of 32 samples, and calculated the elements of the inequality.\nFor a sanity check, we use the $L_1$ loss to verify that it holds 100% of the time, which it does indeed.\nThe results show that while the assumption may not theoretically hold for $L_2$, it holds in 79% of the cases.\n\nTo summarize, although Assumption 4.1 does not hold for $L_2$ loss, in practice, it holds for the vast majority of the optimization process, which explains the performance improvements of our method in the $L_2$ case.\n\nAdditionally, you claimed that due to the fact that Assumption 4.1 doesn’t hold for $L_2$, “the proposed method reduces to a heuristic regularization technique, and the claim of optimizing a causal bound does not hold”.\n\nReviewer MtKE suggested a controlled experiment to isolate the effect of our causal mechanism, which will prove that our method's significant improvements are due to the causal bound optimization, and not just a regularization technique.\nHe suggested adding a regularizer to the standard temporal difference loss function in the form of $(Q_{current}-Q_{target})^2$, where $Q_{target}$ is the slowly moving target network, as done in double DQN.\n\nWe conducted this experiment on 5 Atari games using the DDQN agent on 10 different random seeds.\nThe results are below:\n\n| GAME | DDQN | DDQN with target network regularizer | DDQN using SUFT |\n|-------------|-------|---------------------------|-------------|\n| Assault | 781 | 670 | **1,440** |\n| Asterix | 676 | 666 | **939** |\n| Boxing | -1.73 | -2.41 | **18.4** |\n| Demon Attack | 369 |-17.3 | **2240** |\n| James Bond | 316 | 186 | **416** |\n\nThe results above are conclusive and demonstrate that this kind of regularization harms the actual DDQN performance. It proves that our method has a significant impact due to the causal bound optimization, where the old Q-network values used for the OPE term are of the behavior policy used to generate the experiences, therefore mitigating the divergence between the on-policy and off-policy methods. This shows that our method is not just a regularizer that promotes stability, but a causal bound optimization.\nWe will conduct this experiment on the entire 57 games benchmark and add the results to the appendix. \n\n$\\underline{\\text{Metric:}}$\n\nIn our paper, we used several evaluation metrics and techniques to ensure the robustness and reliability of our results.\nWe conducted controlled experiments with different agents that include our proposed term in comparison to identical baseline agents without it. These controlled experiments were intentionally made in order to isolate and measure the direct impact of our method. To measure our method’s improvement, we compared our agent's reward to the baseline reward using the ratio formula, similar to the one used in [1], which is a highly cited work from a top-tier venue (see Figure 4 in [1]).\n\nWhen calculating the mean reward ratio formula, we have considered only the valid environments where both the numerator and denominator are greater than zero, ensuring the validity of the ratio formula.\nFollowing your review, we are willing to adopt your suggestion, which will enhance our method's performance analysis and demonstrate its effectiveness and true impact.\nTherefore, in our revised version of the paper, we will address the areas where the baseline reward is only slightly better than the random reward, specifically when the denominator is lower than 1, which leads to large values. We will adjust the valid environment restriction to consider only environments where both the numerator and denominator are greater than 1, thereby preventing arbitrarily large values.\nThe new mean reward ratios for the different agents are:\n\n| Agent | New reward mean ratio | Old reward mean ratio |\n|------|---------------------------|-------------|\n| DQN | 383.19% |  2,427.06% |\n| Vanilla DQN | 228.12% | 230.49% |\n| DDQN | 226.91% | 260.03% |\n| PPO | 197.34% | 203.21% |\n\nThank you for bringing this to our attention, and your intuition about this metric was correct. \nWith our new adjustment to this metric, we are excluding the impact of cases where baseline rewards are only slightly better than the random rewards, and guarantee the effectiveness of this metric.\nThe results with the new adjustment still demonstrate that our method continues to have a significant impact on performance while excluding the slightly better than random environments from the mean calculation, thereby strengthening the effectiveness of the evaluation metric and providing a more accurate assessment of our method’s impact.\n\nFurthermore, please note that the original manuscript contains evaluated results using the standard human-normalized score in an ablation study, as shown in Table 5. In the revised version of the paper, we will relocate this analysis from the appendix to the main body of the paper to strengthen our results.\n\nRegarding your modification suggestions, we are willing to accept any suggestions for improvement, and in the revised version of the paper, we will modify the following:\n\n1.\tWe will adjust the section in line 40 to add more details to the introduction on how to incorporate the causal bound to the DRL framework, and to provide additional insight into why adding the proposed term to the loss function yields such significant improvements over conventional methods.\n2.\tWe will add further explanation to Figure 1 in the caption. \n3.\tWe will move footnote number 2 that explains \"SUFT\" to page 2.\n4.\tWe will use a smaller font on Figures 4 and 10.\n5.\tWe will adjust the equation on line 752 to fit the margin.\n\nOverall, we would like to thank you for your constructive feedback. We believe that we have addressed all your questions and concerns and that our paper is now even better. \n\nPython code for the synthetic data:\n\n`import numpy as np`\n\n`size = 1_000_000_000`\n\n`x = np.random.normal(loc=0, scale=1, size=size)`\n\n`y = np.random.normal(loc=0, scale=1, size=size)`\n\n`x_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`y_tag = np.random.normal(loc=0, scale=1, size=size)`\n\n`lhs = (x - y)**2 - (x_tag - y_tag)**2`\n\n`rhs = (x - x_tag)**2 + (y - y_tag)**2`\n\n`false_counter = np.sum(lhs > rhs)`\n\n`print(\"Success Percentage:\", 100 - false_counter / size * 100)`\n\nReferences:\n\n[1] Kaiser, Ł., Babaeizadeh, M., Miłos, P., Osiński, B., Campbell, R. H., Czechowski, K., ... & Michalewski, H. (2020). MODEL BASED REINFORCEMENT LEARNING FOR ATARI. In 8th International Conference on Learning Representations, ICLR 2020.‏"}, {"review_id": "sWyFn85962", "reviewer": "Reviewer_KRKD", "summary": "The authors present a novel theoretical result according to which the number of training steps and the experience replay buffer in deep reinforcement learning (DRL) can be significantly reduced.\nThis is very interesting, given the fact that most of DRL papers present ad hoc heuristics.", "questions": "Page 1:\nCan you please elaborate a bit on the comparison between on and off policy agents? :)\n\nPage 2:\nPlease elaborate on the caption of Figure 1.\n\nPage 4:\nIn Preliminaries, can you please also add some general RL background?\n\nPage 5:\nCan you please comment on Assumption 4.1?\n\nPage 6:\nWhy is $a$ included in the arguments of $\\ell$?\nCould it be defined on $s$ and $t_{\\rm target}$ only?\n\nPage 7:\nCan you please further explain the ramifications of Theorem 4.10? :)\n\nPage 8:\nThe experimental setup looks OK!\nFigure 3 needs more comments though, in the caption.\n\nPage 9:\nPlease expand the Conclusion section and add more future work directions.", "limitations": "None.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe main strength of the paper lies at its theoretical scope, on such an important topic.\n\nIn more detail:\n\nThe paper introduces a new causal upper bound on the factual loss in reinforcement learning, adapting the Neyman-Rubin potential outcomes framework in a nice way.\nThis conceptual contribution might open new avenues for bridging on-policy and off-policy methods.\n\nThe SUFT OPE term that the authors introduce, leverages computed value network outputs by recycling them in a \"replay buffer.\"\nThis clever reuse of data leads to improvements in sample complexity and buffer size requirements, while achieving good rewards.\n\nThe method the authors introduce is evaluated across a large and diverse set of environments and agents, showing statistically good performance.\n\nWeaknesses:\n\nWhile the causal bound is theoretically novel, it would be nice if the authors could comment more on its tightness and generalizability.\nMoreover, it would be nice to have more comparisons to existing theoretical bounds in OPE :)\n\nAlso, please see Questions below.\n\nMy Impression:\nThe paper merits acceptance :)", "was_revised": true, "final_justification": "The authors have effectively addressed my comments and concerns.", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to review our paper. \nBelow we address all your questions and suggestions:\n\n$\\underline{\\text{Bound tightness:}}$\n\nFollowing the suggestion from reviewer MtKE, we performed the following experiment to test the bound tightness when ignoring the $\\delta$ term in optimization.\n\nFirstly, the $\\delta$ term involves the true values $y_0, y_1$ (as in Theorem B.12 line 592), which are not observable during training. However, we can approximate these values using the standard Bellman equation for $Q$.\nWe conducted an experiment on 20 Atari games using the DQN agent, plotting the values of the standard off-policy loss, the SUFT OPE term, and the approximation for the $\\delta$ term over the optimization.\n\nFollowing this experiment, we observed that in all of the environments, the $\\delta$ term remained consistently an order of magnitude **lower** than the off-policy loss, and on the same order of magnitude as the SUFT OPE term. \nThis supports the choice of ignoring the $\\delta$ term during optimization, and our significant improvements indicate that the bound is tight enough.\nWe will conduct this experiment on the entire 57 games benchmark, and add the plots showing the $\\delta$ term and the off-policy loss magnitude, along with a short discussion about this term, to the appendix. \n\nQ: Can you please elaborate a bit on the comparison between on and off-policy agents?\n\nA: \n\nRegarding the comparison between on-policy and off-policy agents. \nRL agents interact with the environment using the behavior policy and optimize the target policy. \nOn-policy agents maintain an identical behavior and target policy. They interact with the environment, collect experiences, and once they optimize the target policy, they discard all previously collected experiences to maintain the behavior and target policies identical. This results in a more stable learning convergence, but has a major disadvantage of not being sample efficient since they require new experiences for each optimization step.\nOn the other hand, off-policy agents are a compromise that allows the agent to act off-policy, meaning it can have a different behavior and target polices. This methodology improves sample efficiency by leveraging past experiences stored in the replay buffer, but it can lead to reduced learning stability due to a mismatch between the behavior and target policies. Our method addresses this divergence and bounds the on-policy loss using the standard off-policy loss, along with an additional OPE term, which is the estimated treatment effect.\n\nThank you for your suggestion. We will adjust the introduction (line 23) to elaborate more about the comparison between on-policy and off-policy agents to improve clarity.\n\nQ: Please elaborate on the caption of Figure 1:\n\nA:\n\nRegarding Figure 1, it demonstrates our method's reward improvements in comparison to baseline agents that don’t use our method (not including the additional SUFT OPE term in their loss function).\nThe blue results indicate that our method outperforms the baseline in those environments, while the grey results show the baseline outperforming our method in those environments. We present the improvement percentage for the higher reward method, as calculated in the improvement percentage formula (line 750).\nWe will adjust Figure 1 to include further explanation on this.\n\nQ: In Preliminaries, can you please also add some general RL background?\n\nA: \n\nWe appreciate your suggestion to include additional general RL background in the preliminaries, but we prefer not to expand this section and instead use the space for Section 4, as it is now.\n\nQ: Can you please comment on Assumption 4.1?\n\nA:\n\nRegarding assumption 4.1, it is a variation of the inverse triangle inequality, and it is needed for our causal bound proof. We will include this comment in the revised paper.\n\nQ: Why is $a$ included in the arguments of $\\ell$? Could it be defined on $s$ and $t_{target}$ only?\n\nA:\n\nIn Section 4.2, we present a detailed reduction from the causal framework to the DRL framework using a $Q$-value network for the DQN agent.\nThis agent's stored experiences include the states and the performed actions $(s, a)$. The combination of $Q(s, a, \\theta_t)$ outputs the estimated outcome for this experience, which is analogous to the definitions in the causal framework, as we are interested in potential outcomes. This is why it is needed in the arguments for $\\ell$. For agents that are using a $V$ network, we only use $s$ for $\\ell$. For the full reduction between the causal framework and the DRL framework for both $Q$ and $V$ networks, please see Appendix C, Section B.4. \n\nQ: Can you please further explain the ramifications of Theorem 4.10?\n\nA:\n\nTheorem 4.10 is our causal bound, where we bound the factual using the counterfactual loss, an additional term of the estimated treatment effect, and a delta term, which is independent of the network. This bound interpretation in the DRL framework is of bounding the on-policy loss using the off-policy loss, an additional OPE term, and a delta term, which is independent of the $Q$ network.\nThe ramification of Theorem 4.10 is the implementation of it into any DRL agent with a $V$ or $Q$-value network by adding the SUFT OPE term to the standard off-policy loss, as shown in Section 4.5.\n\nQ: The experimental setup looks OK! Figure 3 needs more comments, though, in the caption.\n\nA:\n\nWe will add additional comments in the caption of Figure 3, further emphasizing that this graph indicates our method's potential to boost agents’ performance even above the human level, while keeping constrained environment interactions and resource demands.\n\nQ: Please expand the Conclusion section and add more future work directions.\n\nA:\n\nWe will expand the conclusion section and provide more future work directions, such as the implementation of our method with physical robotics reinforcement learning, where the need for our reduced resource method is crucial.\n\nWe would like to thank you for your positive feedback and suggestions. We believe that we have addressed all your questions and suggestions and that our paper is now even better."}]

Author rebuttal:

Meta-review: The paper proposes a new objective for off-policy RL based on a novel causal bound on the factual (on-policy) loss, which can be computed using past value network outputs. (As these outputs are usually discarded, the approach turns "sand into gold.") The bound consists of an off-policy component and a term that measures the treatment effect between the current policy and past policies. It's a clever idea, and it appears to actually work too. The paper reports improvements up to 2400% over baselines.

All of the reviews said that the paper was novel and technically sound. They also highlighted two main weaknesses:
1. There is a slight gap between the theory (Assumption 1 says the loss must obey the triangle inequality) and the application (the experiments use $L_2$ loss, which doesn't obey the triangle inequality).
2. The "reward improvement" metric is non-standard, and sensitive to the performance of the baseline.
The authors' rebuttal appears to have mitigated these concerns. They conducted additional analysis of the $L_2$ loss, and updated their reward improvement metric to make it less sensitive. This satisfied the reviewers. I therefore recommend accepting the paper.
Decision: Accept (poster)