Confounding Robust Deep Reinforcement Learning: A Causal Approach
Authors: Mingxuan Li
Junzhe Zhang
Elias Bareinboim
Venue: NeurIPS 2025 poster ()
Keywords: Causal Inference
Reinforcement Learning
Deep Learning
Primary area: probabilistic_methods
TL;DR: This paper proposes a novel deep reinforcement learning algorithm robust to unobserved confounding bias in observed data over complex and high-dimensional domains.

Abstract: A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where \emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

Confounding Robust Deep Reinforcement Learning: A Causal Approach
Mingxuan Li Junzhe Zhang Elias Bareinboim'
Columbia University, Syracuse University (ml,,e))ccs. columbia. edu, "jzhan403Gsyr.ed edu
Abstract
A key task in Artificial Intelligence is learning effective policies for controlling agents unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high- -igeesimmnsi domains where umobserved confounding cannot be ruled out priori. Building on the well-celebrated Deep Q-Network (DQN), we propose novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.
Introduction
Over the last decade, reinforcement learning (RL) has gained sigmificant popularity for solving complex sequential decision making problems, primarily due to its integration with deep learning techniques 24 52 83 This approach, known as deep reinforcement learning (deep RL), par-
decision-making
ticularly effective in high-dimensional state spaces 440
61
70,
84
85
88] Deep RL. addresses the
challenges faced by earlier RL. methods by extracting various abstractions from data in complex domains with minimal prior knowledge. For example, a classic algorithm known as Deep Q-Network (DQN) algorithm can efficiently learn from visual inputs containing thousands of pixels [61], enabling problem-solving capabilities comparable to humans in certain high-dimensional environments, such as Atari games for the first time. These achievements were followed by notable advancements in deep RL, including mastering the game of Go [89], defeating world-class professionals at the game of poker 644. Deep RL has also shown potentials in various real-world applications like robotics [70 (8). autonomous driving [49] and protein design [40]. These advancements eventually
culminated in Sutton and Barto receiving the Turing Award in 2025
[2]]
This paper attempts to leverage the capabilities and insights of DQN, while identifying an important assumption embedded in this algorithm and its variants that does not necessarily hold in the real world. Particularly, we notice that it is often implicitly assumed through the (PO)MDPs (93 framework or explicitly enforced during the data-collection that no unmeasured confounder (NUC, [ 78)) affects the observed action and the subsequent outcomes. When the NUC does not hold, the effect of the target policy is generally not identifible, i.e., the model assumptions are insufficient to uniquely determine the value function from the offline data [72 [II]. On the other hand, partial identification a line of methodologics that enable the derivation of informative bounds on target effects from confounded observations in con-identifiable settings [59]. It has been studied under the rubrics of causal inference
99, and dynamical systems More recently, researchers have been using partial identification methods obtain
7 [15), econometrics [8] 35
75
66 13
20 63
69
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

113. Despite these achievements, significant challenges still exist applying partial identification for policy learning in complex and high-dimensional domains, including images and videos. We refer readers to App. A for a more detailed survey on partial identification and deep reinforcement learning.
reliable off-po evaluation in reinforcement learning [1 39
48
54 67
This paper aims to address these challenges by investigating deep reinforcement learning algorithms from offline data over complex and high-dimensional domains, where the presence of unmeasured confounders could not be assumed away priori. More specifically, our contributions are summarized as follows. (1) We introduce a novel DQN algorithm, which we call Causal DQN, capable of learning robust abstractions from confounded data over complex and high-ding dithesminnsii domains with minimal prior knowledge. (2) We empirically demonstrate that our method significantly improves robustness and generalization under confounded observations and outperforms various DQN baselines across twelve popular Atari games. Due to space constraints, details of the experiment setup and additional experiments are provided in Apps. and Videos of gameplay are included in the supplemental.
Notations. We will consistently use capital letters (V) to denote random variables, lowercase letters (v) for their values, and cursive V to denote the their domains. We use bold capital letters (V) to denote a of random variables and let [V denote its cardinality of set V. Finally, 1 Iz= = is an indicator function that returns if event Z holds true; otherwise, it returns 0.
2 Challenges Due to Unobserved Confounders
We will focus a sequential decision-making problem in the Markov Decision Process (MDP, (77] where the agent intervenes on sequence of actions to optimize subsequent rewards. Standard MDP models focus on the perspective of learners who could actively intervene in the environment. Consequently, confounding is generally assumed away a priori. On the other hand, when considering off off-policy data collected from passive observations, the learner does not necessarily have the liberty to control how the behavioral policy generates the data, giving rise to unobserved confounders in In this paper, we will consider a generalized family of 113 explicitly modeling the presence of unobserved confounders confounded MDPs [4] 44 [12 in the off- policy data generation.
tasks (26 43
51
81 114
Definition 2.1. A Confounded Markov Decision Process (CMDP) M is tuple of (S, X. where (1) S, X, y are, respectively, the space observed states, actions, and rewards; (2) U is the space of unobserved exogenous noise; (3) F is a set consisting of the transition function Jss : S x X' x + S, behavioral policy fx SSl x + X, and reward function fy Sxx xl1 + y; (4) P is exogenous distribution over the domain U.
Throughout this paper, we will consistently assume the action domain X' to be discrete and finite, while the state domain S could be complex and continuous; the reward domain y is bounded a real interval [a, bj C R. Consider demonstrator agent interacting with a CMDP M. generating the off-policy data, For every time step t= 1......,7 T. the environment first draws exogenous noise U, from the distribution P(U);; ; the demonstrator then performs action X, fx (St, U.), receives subsequent reward Y, +- (S,,,, Xx, U2). and moves to the next state S.+1 +- [s, X,, U.). The observed trajectories of the demonstrator (from the leamer's perspective) summarized as the observational distribution P(X1.1, S1r1, Y Y1:)), i.e.,
S
Figure 1:
Causal diagram representing the data-generating mechanisms in a Con- founded Markov Decision Process.
T P(TIT S1T DI.T) P(s1)
II 1=1
I fx. "t m= P(we)
Fig. shows the causal diagram [00] describing the generative process of the off- off-policy data in CMDPs. More specifically, solid nodes represent observed variables X,, St, Y,. and arrows represent the functional relationships fx, : Jss Jy among them. By convention, exogenous variables U, are often not explicitly shown in the graph; bi-directed arrows X, +- Y, and X, <-> Se+1 indicate the presence of an unobserved confounder (UC) U, affecting the action, state, and reward simultaneously. These bi-directed arrows (highlighted in blue) represent the unobserved confounders among action

---

X₁, reward Y,, and state S1+1 in the off-po data, violating the condition of NUC 11 78 violations could lead challenges in off off-policy learning.
Such
Off-Policy Learning. A policy in a CMDP M is decision rule (x; 8x) mapping from state to a distribution over action domain X. An intervention do(z) is an operation that replaces the behavioral policy fx in CMDP M with the policy т. Let M, be the submodel induced by
intervention do(x). The interventional distribution (x(X1.7 I:T, SI.T, Y (IT) is defined as the joint distribution over observed variables in Mn, i.e.,
Pa S (IT) P(s.) II
(xx 8t)T 2t, 8 (1,
(I)
1=1
where the transition distribution T and the reward distribution R are given by, for h = 1...., H,
T(se, Dz, s(+1) 1 (ee=
RR, = 1 P(ue) (2)
For convenience, we write the reward function R(x, x) as the expected value E, (R, v). Fix a discounted factor y E [0. 1]. A common objective for an agent is to optimize its cumulative return R₂ Ei=u y Y'Y+.i We define the optimal action-value function (s, 2) as the maximum expected return obtainable by following any policy æ, after seeing a state s and taking an action r, Q- -) max, Ex,,-z, [Re St s]. One could solve for optimal policy by iteratively evaluating the action-value function using the Bellman Optimality Equation [1] given by.
Q.(8, Ex,-z Vv + max Q- S =
In off-policy evaluation, the agent (i.e., learner) attempts to learn an optimal policy by leveraging the observed data generated by different behavior policy fx (demonstrator). When there is no unmeasured confounder (NUC) introducing spurious correlations between action and subsequent outcomes, one could identify the parameterizations of the transition distribution T and reward function R from the observed data, i.e..
T(st, Zt, 1t+1) = P (s xt)
R(s,, It, Vt) = P (yt St, x1)
When the above identification formula hold, several off- off-policy algorithms have been proposed to estimate the effect of candidate policies from finite observations [38 41 42
655 76 94| 104 [05].
Together with the computational framework deep learning, these methods could be further extended However, NUC could be fragile in practice and does not necessarily hold due to some violations in the generative process. In these situations, applying standard off-policy methods may fail converge to optimal policy, despite using powerful deep learning models. The following example illustrates such challenges in a classic Atari game.
to complex domains 66 70 84
85 88
Example 1 (Confounded Pong). Consider the Pong game in the classic Atari suite. As for the behavior policy, use a pre-trained
actor-critic agent [2] with residual blocks [29] and Long Short-Term Memory (LSTM, (32) layers. Fig. 2a shows a saliency map visualizing the learned policy. Simulation results show that this policy is optimal. For example, this agent can deliver a "kill shot" by directing the ball to a location that the opponent is unlikely to intercept given its current location. We use this agent as the demonstrator in the off off-policy learning task.
We now consider an alternative agent that learns to play Pong by observing the demonstrator's gameplay trajectories. This learning agent has simpler neural network architecture and an impaired sensory capability: it can only observe movements in its nearby surroundings. Fig. 2b shows the learner's visual input; the board's 's left-hand side and the upper side, including the opponent's 's position and score, is now masked. In this case, the opponent's position becomes an unobserved confounder, introducing spurious correlations between the demonstrator's action and observed outcome. For example, the behavioral policy tends to hit the ball toward the center only when the opponent is positioned either corner and unable to return it. As result, center shots appear more effective than they truly are, due to confounding resulted from the masked opponent's position.
To validate whether the standard deep RI. algorithms are robust to confounding biases, we train two DQN learners on masked trajectories from the demonstrator: one the standard convolutional neural network based DQN (Nature DQN, [61)), the second one is an LSTM-based (28). We also include

---

Masked PongNoFramesippuu
Nesurm
Cound cDem
0.0
0.2
24
0.8
3.0
Reviroment Stops,
(a)
bb)
(d)
Figure 2: saliency map of the behavioral policy in Pong that tracks the opponent's 's location and score board; bb) a confounded Pong game where the opponent's 's location and score board is masked; (c) saliency map of the conservative policy focusing on only itself and the ball; (c) the average return of our causal DQN and the standard DQN baselines. Baseline curves are overlapped.
standard DQN that learns directly in the masked Pong game without confounded demonstrations as a baseline. The simulation results, shown in Fig. [ indicate that none of those DQN variants is able to converge to an effective policy. Learning directly with impaired observation in Atari is challenging, while incorporating confounded demonstrations directly also does enhance the convergence of DQN agents; instead, it negatively impacts their learning performance.
3 Confounding Robust Deep Q-Learning
In this section, we will introduce partial identification methods for off-policy learning that are robust to unobserved confounding. Recently, Zhang & Bareinboim [113] extended the well-celebrated Bellman equation to allow one to lower bound the state-action value function (s, x) with closed- form solution Qn (s, z), which can be consistently estimated from the confounded observations. extend this result to obtain a lower bound for the optimal value function.
Proposition 3.1 (Causal Bellman Optimality Equation). For CMDP environment M with reward signals Y, E [a, b) C R, its optimal state-action value function Q. (8, x) 2 Q-(8,)) for any state-action pair (8, x) E x X, where the lower bound Q (8, x)) is given by as follows,
(5,x.) P(# s) ( (x, + ET (s, max (,(5xzz')
(5)
+ +P(-- 8) a aa+ min
min max Qu (s' (s',
(6)
where P(x 8) = P = - and P(-z 8) 1 P(# s): 7 and R are nominal transition distribution and reward function computed from the observational distribution, i.e.,
7
PS s |S x,
(s,
= E S; s, (7)
Prop. 3.1 lower bounds the expected return of optimal policy T² using the return of pessimistic
policy π that optimizes a worst-case CMDP instance M compatible with the observational data. The lower bound is set as the expected return of the pessimistic policy x² in the worst-case CMDP
M, i.e., (s, A Q-- , x;; M). Since x² is optimal in the ground-truth CMDP environment M, we must have (s, x; M) Qu- (s, x;; .M) 2 Qu-(s, x; M). Optimizing the lower bound in Prop. 3.1 leads to a pessimistic policy with a performance guarantee in the ground-truth environment.
Among quantities in the lower bound Prop. 3.1 nominal transition distribution T and nominal reward function R are functions of the observational distribution, and, at least in principle, are consistently estimable from the sampling process. The lower bound Q. (s, x)) can thus be further written as:
(s, 1xx=x (Ye+ max Q+ (St+1,2²)) 1x (a min max Q+ (2)))=))) Ss = (8)
x²

---

Algorithm Causal Deep Q-Learning (Causal-DQN)
1: Initialize replay memory D
2: Initialize action-value function Q-(:0) with random weights 0
3: for episodes = l..... M do
4: Sample initial state $1
5: for fort= 1.... , do
6: Observe an action 24 taken by the demonstrator and subsequent reward yr and state sc+1
7: Store transition (st, Te, Ut, St+1) ) in 'D
8: Sample a minibatch of transitions {(s;, Fs Vi, 81+1)] from D
9:
Set value target w; (z) for every action 2 E X' w.n.t sample (st Us. S141),
yi + max, (+++,,
if
w (a)
(11)
| a + ming max, Q. (5 x²; if o
10: Perform a gradient descent step on E., (w,(x) - (+ (s,, 0)) according to Eq. (10)
end for 12: end for
In the above equation, Y, and St are observed variables drawn from the nominal reward function R and transition distribution T in Eq. ( respectively. 3 a backup diagram illustrating this update step. Like the standard Bellman optimality equation (Eq. 33) Eq. (8) recursively updates the value function based on the current estimates of the optimal value function. On the other hand, Eq. (8 explicitly accounts for the off-y nature of the confounded observations: when the behavior policy takes the same action I; as the target action, the update follows standard Bellman equation and uses the next sampled state 8,; when the sampled action 2, * x differs from the target, our algorithm updates, instead, using the value function associated with the next worst-case best-case state s", corresponding the estimation of the lower bound and upper bound respectively.
(s, x)
S
X
s*
S2
Figure 3: Backup diagram for causal deep Q-learning-
By using the causal Bellman equation of Eq. (8) could apply standard value iteration to obtain a robust policy against the confounding bias in the off off-policy data [113]. In practice, however, this approach could be computationally challenging for complex and high-dimen nigh-dimensional domains. Like many deep reinforcement learning algorithms [61], we will use a neural network with weights 0, called Q-network, to approximate the lower bound over the state-action value function, i.e., (s,s, x;; 0) x (s, r).. We will train Q-network by minimizing sequence of loss functions (0,) at each iteration i. Formally,
iterative update, one
- (W., (,,,, (()x))))
L.(O.)= = Ea-p(-)
where function Wxxx) is defined the right-hand side of the update procedure Eq. and p(s) the state occupancy distribution in the observed Markov chain under the behavioral policy. The parameters from the previous iteration 0;-1 are held fixed when optimizing the loss function. Note that the above loss function attempts to minimize the error of the Q-network bound over all actions. The reason is that, in the causal Bellman update (Eq. (8). the next observed action contains information about the lower bound across all actions, regardless of whether it matches the actual action taken. Differentiating the loss function with respect to the weights, arrive the following gradient,
V L. = Ee-p()
E Σ (Wi(x) Q= 0.)) Vo,
S,
(10)
Details of our proposed algorithm, called Causal Deep Q-Learning (Causal -DQN), are provided in Algo. D Like the standard DQN [61], our algorithm utilizes experience replay (57). Particularly, stores trajectories observed at each time step, represented as (st, Vt. 81-1), in a replay memory DD that pooled from many episodes. During the inner loop of the algorithm, we apply minibatch stochastic gradient descent to samples of experience (st, 24, Br, 5(+1) D, which are randomly
5

---

drawn from the pool of stored samples. On the other hand, our proposed causal algorithm made the following augmentations compared to the standard DQN. First, Causal-DQN is an off-po learning algorithm and does not actively intervene in the environment. At step 6, instead of exploiting the Q-network being trained, it queries the demonstrator to generate confounded transition sample. Second, at Step 9 during the experience replay, Causal-DQN utilizes the causal Q-learning updates of Eq. (T).. which is robust to the potential presence of confounders. Particularly, when the observed action z; is equal to the evaluated action æ, the algorithm follows the standard Q-learning update. Otherwise, performs the update using a lower-bound a over the immediate reward and the value function at the worst-case next state s'. The worst-case state s' is empirically estimated by repeatedly sampling the next possible states at random, and taking the one with the smallest value function estimate. These augmentations improve Causal- Causal-DQN over its non-causal counterpart in terms of robustness and sample efficiency, as it is able to utilize the abundant observational data to improve the evaluation of the state-action value function.
Example 2 (Confounded Pong continued). Consider again the confounded Pong game described in Example | We train a Causal DQN agent with the masked observed trajectories. Fig. 2c shows the saliency map visualizing the learned policy. Our proposed method learns a conservative policy focusing on only tracking the ball location instead of opponent's 's location. Simulation results show that this conservative policy is able to achieve comparable performance to the optimal demonstrator using the full board information. Analyzing the gameplay video reveals that our causal DQN agent learns to proactively place the ball in either corner, where the hard-coded AI opponent is unable to return. See the gameplay video in the supplementary materials for more details.
4 Experiments
In this section, we aim to demonstrate the robustness and performance improvement of our proposed Causal-DQN under confounded settings. For a comprehensive evaluation of Causal-DQN, we choose twelve popular Atari games from the Gymnasium benchmark [100] and design the corresponding confounded versions. See below and also App.[G C for our detailed design of confounded Atari games. For a fair comparison, we also use vanilla DQN with little modifications as the baselines we test. More specifically, the baselines include (1) a CNN-based DQN with confounded demonstrator data (Conf. DQN). (2) an LSTM-based DQN with confounded demonstrator data (Conf. LSTM-DQN), and (3) a CNN-based DQN trained directly under masked observations without confounded data (Interv. DQN). For (1-2), the DQN agent will query the demonstrator for data samples the Causal-DQN does. For (3), the DQN agent uses its own policy to sample environment transitions.
For each game, we train the agent for I million environment steps. We use 20 parallel environments to collect samples. At each parallel environment step, a minibatch is sampled to train the agents, equivalent to an update frequency of 20. We use a batch size of 512, replay buffer of 100K in size, and learning rate of 5e-4 to accelerate convergence. Other hyperparameters are the same as in [T). All results presented in this section are evaluation performances where we test each trained agent in the Atari game with masked observations. Curves in Fig. S are generated by evaluating the agent periodically in separate evaluation environment, not from training returns.
Data Preparation and Model Architecture. We use the standard Atari game preprocessing for the input to the agents except that we resize the input to be 64x64 to align with the size requirement of the demonstrator [2], a competitive actor-critic agent with deep residual blocks and LSTM layers. Other differences in input preprocessing for the demonstrator are that (1) the demonstrator takes a single colored frame as the input per each time step, and (2) the demonstrator has access to the original full screen observation while the learners only has access to a masked partial screen. For all DQN tested, we adopt Double DQN [0I]] to stabilize learning. The CNN version follows the set up of Nature DQN [6] and the LSTM version only replaces the second-to-las linear layer with an LSTM cell. See App. D for detailed model architectures.
Designing the Confounded Atari Games. In the confounded Atari games, we mask out certain areas in each game's 's observation to prevent the agent from using spurious correlated features. To find such spurious visual artifacts used by the demonstrators, we apply a perturbation based approach 25 to visualize saliency maps of both the actor and the critic of the demonstrator 22. As shown in Fig. 2a in the Pong game, the demonstrator is constantly checking the score board and also the opponent's s paddle locations, neither of which is necessary for winning since the opponent Pong has a fixed policy regardless of the current score and as long as the agent shoot back the ball, there is

---

chance to score. Thus, an intuitive optimal policy should only look at the ball location and the agent's own paddle location decide the move. In Fig. [b] we mask out those areas and use the masked out observation as the state input (st) to DQNs while the masked area becomes the confounder which can only be observed by the demonstrator's policy, i.e., 2t f- fx (8t; we).
For the remainder of this section, we will first present few other notable confounded Atari games and discuss the performance of our proposed Causal-DQN in all 12 confounded Atari games. Specifically, despite confounding bias, our proposed causal agent is able to obtain an effective policy under masked observations from the demonstrator's trajectories. The learned policies demonstrate conservative behaviors aligned with human intuitions. Overall, Causal- -DQN consistently dominates its non non-causal vanilla counterparts in all 12 confounded Atari games performance.
original Boxing, the player controls the white agent to punch the black one to score. The party with the higher score when the time runs out, or any party hitting 100 first, wins the game. The demonstrator's policy picks up an aggressive "brawler" style which pres- the opponent and trades blows in the center of the arena. Fig. 4a shows the saliency map the demonstrator's policy.
Confounded Boxing. In
1:54
0153
(b)
(c)
In the confounded Boxing, we mask out the score/remaining time, the outer area of the arena, and the right half of the arena. words, the agent has impaired eyesight and cannot keep track the current score and remaining time. Our Causal-DQN agent picks up conservative focuses on defending its ground on the left-hand side of the arena. Fig. 4clsoowss the saliency map of such conservative policy. Perhaps surprisingly, simulation results, shown in TableTaane Fig. reveal that despite the limited sensory capabilities, Causal-DQU agent can still achieve similar performance the optimal demonstrator, defeating the hard-coded Al opponent.
Figure 4: (a) A saliency map of the demonstrator's policy; (b) confounded Boxing game where only the
left half of the arena is visible; (c) saliency map of Causal-DQN' policy.
boxing style which
Confounded Gopher. In the Gopher game, the player controls a farmer with shovel, tasked with protecting a garden of carrots from a mischievous gopher. The gopher repeatedly attempts to tunnel un- derground to steal the carrots. The player must move horizontally across the screen to block the gopher's 's digging attempts by fill- ing holes. shortcut strategy is to follow the gopher's location underground closely so that the farmer is always close to those newly completed holes. Fig. [a shows the saliency map of the demonstrator' s policy. Our analysis reveals that it manages pick up a "proactive" playing strategy, which actively tracks the gopher's location and uses this information adjust the farmer's position.
(b)
Figure 5: (a saliency map of the demonstrator's policy; (b) confounded Gopher game where the tunnel and score are masked; (c) a saliency map of Causal-DQN's policy.
On the other hand, in the confounded Gopher game, the gophers' locations are now masked, and the agent no longer follows the same "proactive" strategy. Instead, Causal-DQn picks up alternative "reactive" strategy, which will reset the farmer's position around the center and only move when gopher is digging out of the ground. We evaluate the learner's performance and provide them in Table and Fig. Interestingly, simulation results show that the "reactive" strategy is more effective than "proactive" one, and Causal-DQR is able to outperform the demonstrator's policy.
ChopperCommand game, the agent controls a helicopter tasked with defending convoy of trucks from waves of enemy aircraft and helicopters. The agent must navigate across the desert landscape, shooting down enemies while avoiding incoming fire. Successfully protecting the convoy and eliminating threats increases the player's 's score, while
Confounded ChopperCommand.

---

Table 1: Average evaluation returns of agents the 12 confounded Atari games trained with 1M environment steps and aggregated normalized returns concerning the demonstrator's performance. Bold numbers indicate the best-performing methods. All results averaged over seeds except that column Random is from (2]. Causal-DQU significantly outperforms other DQN baselines.
Game
Demonstrator
Random
Interv. DQN
Conf. DQN
Conf. LSTM-DQN
Causal (ours)
Amidar
232.4
5.8
44.0
37.8
59.0
282.6
Asterix
3080.6
210.0
650.0
429.0
479,0
2587.0
Boxing
89.0
0.1
-0.62
-9.8
-6.9
71.5
Breakout
219.2
1.7
2.2
1.2
4,9
131.2
ChopperCommant
1280.0
811.0
1192.0
1076.0
1116.0
1658.0
Gopher
5480.6
257.6
288.8
752.0
485.6
7327.2
KungFuMaster
35400.0
258.5
12416.0
13674.0
6526.0
44196.0
MsPacman
2316.8
307.3
787.4
1747.6
Pong
20.8
-2..7
-20.8
-20.8
-20.4
21.0
Obert
4420.6
163.9
322.5
208.5
253.5
4458,5
RoadRunner
16560.6
1154.0
1168.0
484.0
27414.0
Seaquest
1412.4
68.4
237.2
281.6
164.8
980.0
Normalized Mean (1)
1.00
0.00
0,13
0,10
0.09
1.04
Normalized Median (t)
1.00
0.03
0.13
0.14
0.10
1.01
Normulized IQM (f)
1.00
0.03
0.13
0.13
0.11
1.02
allowing enemy fire to destroy the trucks results in lost points lives. At the bottom of the screen, a mini-map/radar is showing incoming trucks and enemies. Fig. shows the saliency map for the demonstrator's policy. It learns utilize the radar information to "Took ahead."
We next consider a confounded Chopper- Command game where the chopper loses its radar and other sensor devices. As a re- sult, the mini-map area, current score, and remaining lives are masked (as shown Fig. (b). By applying Causal-DQN, the agent picks up more "spontancous" play- ing style, focusing staying alive and eliminating opponents upfront. Fig. [6c de- scribes saliency map of this "spontaneous" policy where only nearby opponents are highlighted. Simulation results in Table I and Fig. g reveal that Causal-DQN out- performs other baselines significantly and even surpasses the demonstrator's policy despite having a simpler neural network architecture.
(c)
Figure 6: (a) saliency map of the demonstrator's
policy; (b confounded ChopperCommand game with the minimap and score/lives masked; (c) saliency map of fausal-DQN policy.
Overall Performance. Table provides the best mean returns for all 12 confounded Atari games across trials, along with mean return normalized by demonstrator's 's perfor- mance and normalized interquantile mean (IQM). We see Causal-DQU consistently outperforming other non-causal baselines by big margin. In 7/12 games, our pro- posed method even surpasses the demon- strator with full observations and way more complex architecture 22. This could be due to the reason that the demonstrator, though powerful in representation learning, may suffer from observational overfitting (90] and rely on spurious visual features to make decisions. Both prior work and our work have empirically verified this by using saliency maps [25). Masking out those spurious features indeed poses non-trivial challenge the DQN. From Table [ we see that the vanilla DQN with interventional data under masked observations (Interv. DQN) hardly achieves any meaningful scores in million environment steps. Even with the help of LSTM cells or confounded data from performing demonstrator, the DQN still cannot learn. Only with the causal bound, the same architecture (Nature DQN) can recover even surpass the demonstrator's performance despite using masked observations and shallow, small CNN as the feature extractor. In Fig. 7 also
Cousel DON rOural
DOK Tiners INto
DON wT Cieet. Dats
LETI DON Oood DMtu
60
03
0.0
Korvalline 10M
Figure 7: Normalized mean and normalized IQM scores. Causal- aaasaN achieves normalized mean return of 1.04 and a normalized IQM 1.02.

---

Xaadoed Amttar////aaeeekppp
Manled Amarss/o//mmeeppp
Maxbed bothggollmmmeeeer
2340
1000
R112
5.0
022
84
Sud
6.
13
12
24
leb
10
2.4
Lesi
9.8
1.
Misked Fronsonk vi
Misked Clogerr/annaddoouurceee
Masked CogherNefancsiip vi
100
1360
1200
1903
Emriboeest Steps, lo5
Enrimoncet Steps. le6
Esstronccot Stops. LoG
40000
36000
1000
20066
10200
940
5000
0.2
055
2.
0G
3.5
Masked ObectNernneskippd
Masked Cac.launnneNoooomslll
Masked Stonppr
3300
3103
4248
2188
22120
(2000
1000
200
5.0
0.2
2.4
Sings, 105
8.
Environent Stegs. Ie6
1.0
Esstroment LeG
0.5
0.8
3.0
Cacaul DON (Oura)
Inteventiooaal
JQN Confounded Data
LSTM--DQN Donfounded Data
Figure 8: Average evaluation performance for 12 confounded Atari games. During training over the 1M environment steps, we evaluate the agent every 100K steps for 10 episodes each time. The curve is further averaged over seeds with one standard deviation across trials as the shaded area.
provide stratified bootstrap confidence intervals for the normalized mean and normalized IQM scores as recommended by Agarwal et al. [ We can see clearly that our proposed Causal-DQa can recover the expert demonstrator's performance even under impaired sensors. And get sense of the sample efficiency of Causal-DQH, in Fig. g we report the evaluation performance during training of all 12 confounded Atari games. Causal- -DNN converges uniformly in less than I million steps in all games. See App. for more results.
Remark on the failure of DQN baselines. We have conducted thorough paper--arameter tuning and used the recommended hyper-parameter settings from the original DQN work [6]] Due to limited time and computational resources, we are only able to finish 1M steps of training for all the environments and seeds. However, we do notice that the interventional DQN baseline can gradually learn the right policy with longer training time. The learning curve usually starts to grow after 3M steps. This corroborates the validity our confounded environment design, that the learning task is now harder to solve, but not totally impossible. With the help of causally aware learner (Causal -DQN), one would be able learn more sample-effiiietlly (3x fewer steps our experiments) than the pure interventional online regime the biased causally unaware offline learners.

---

5
Conclusions
This paper investigates deep reinforcement learning from off-policyy data collected by different be- havior policy through a causal lens. Particularly, we focus on generalized setting where confounding biases cannot be ruled out priori, which poses significant challenges to standard off off-policy evalua- tion algorithms. We first extend the celebrated Bellman equation to the causal Bellman equation that lower bounds the agent's expected return from confounded observations. Building on this extension, we then propose a novel Causal-DQU algorithm that could obtain an effective policy from off off-policy data even when unobserved confounders generally exist. Finally, we evaluate our proposed algorithm in twelve confounded Atari games, showing that the causal approach consistently dominates the standard DQN algorithm with different feature extractors or data sources.
Yet the implications of this work extend far beyond discrete control benchmarks. Unobserved confounding is not an anomaly but a pervasive property of real-world RL. It lurks beneath virtually all forms of observational or off-policc data, from robotic demonstrations to human feedback, silently distorting the mapping between actions, rewards, and outcomes. In an era when the field is increasingly guided by the scaling law, the belief that enlarging models and datasets will automatically yield better intelligence, our findings reveal a critical blind spot: scaling on confounded data does not scale performance. In fact, it may amplify biases, producing policies that are efficient yet misaligned.
Consider large-scale robotic pretraining from internet videos: behavioral policies (human or robot demonstrators) differ sharply from the learner's policy space, introducing systematic unobserved confounding between observations and intended actions. Similarly, in reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs), preference datasets deeply entangled with hidden factors like emotions, social norms, cultural context, temporal inconsistency, and individual baselines that no simple text prompt can fully encode. Without causal understanding of these factors, LLM alignment merely approximates correlations within preference and prompts, not their causal origins. The same problem exacerbated for healthcare, finance, and any domain where observational data conceals the determining forces driving decisions.
Toward the future, we envision causal reinforcement learning as foundation for building confounding-robust, safely-aligned, and generalizable agents. Extending the causal Bellman frame- work to policy-gradient methods, continuous control, RLHE, and multi-agent systems will be crucial steps toward this goal. Ultimately, bridging causal inference and deep RL offers more than robustness. paves the way for agents that reason about interventions and consequences, rather than merely fitting experience. In sum, this work marks an early yet essential stride toward causally grounded agents, a future where RL agents learn not only what do, but why works.
Acknowledgments
This research is supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
References
[1] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare. Deep reinforce- ment learning the edge of the statistical precipice. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[2] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[3] M. Andrychowiez, D. Crow, A. Ray, Schneider, R. Fong, Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. von Luxburg, S. Bengio, M. Wallach, R. Fergus, V. N. Vishwanathan, and R. Garnett, editors, Advances in Neurul Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5048-5058, 2017.
10

---

[4] A. P. Badia, B. Piot, S. Kapturowski. P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the Atari Human Benchmark. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 507-517. PMLR, 2020.
[5] K. Badrinath and D. Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proceedings of the 38th International Conference on Machine Learning, pages 465-474, 2021.
[6] P. Bajari, C. L.. Benkard, and J. Levin. Estimating dynamic models of imperfect competition. Econometrica, 75(5):1331-13700 2007.
[7] A. Balke and J. Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal of the American Statistical Association, (2(43))11722-11 September 1997.
[8] P. J. Ball, L. M. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scar- lett, editors, International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1577-1594, PMLR, 2023.
[9] S. Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta Mathematicae, 3:133-181, 1922.
[10] E. Bareinboim, D. Correa, D. Ibeling, and Icard. On Pearl's 's Hierarchy and the Foundations of Causal Inference, pages 507-556. Association for Computing Machinery, New York, NY, USA. 1 edition, 2022.
[11] E. Bareinboim, J. Zhang, and S. Lee. An introduction to causal reinforcement learning. Technical Report R-65, Causal Artificial Intelligence Lab, Columbia University, December 2024.
[12] R. Bellman. Dynamic programming, Science, 153(3731)334 1966.
[13] A. Bennett and N. Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. Oper. Res., 7233):0011-1086 2024.
[14] A. Bennett, N. Kallus, L. Li, and A. Mousavi. Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders. In International Conference on Artificial Intelligence and Statistics, pages 1999-2007. PMLR, 2021.
[15] S. T. Berry and G. Compiani. An instrumental variable approach to dynamic models. The Review of Economic Studies, 90(4): 2023.
[16] P. Bhargava, R. Chitnis, A. Geramifard, S. Sodhani, and A. Zhang. Sequence modeling is robust contender for offlin reinforcement learning. arxiv, 2023.
[17] D. Bruns-Smith and A. Zhou. Robust fitted-q-evaluatioon and iteration under sequentially exogenous unobserved confounders. arXiv preprint arXiv:2302.00663 2023.
[18] F. A. Bugni. Bootstrap inference in partially identified models defined by moment inequalities: Coverage of the identified set. Econometrica, 78(2):735-753, 2010.
[19] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[20] M.J. Dickstein and F. Morales. What do exporters know? The Quarterly Journal of Economics, 133(4):1553-100,, 2018.
[21] L. Espeholt, H. Soyer, R. Munos, Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: Scalable Distributed Deep- RL with Importance Weighted Actor-Learner Architectures. In G. Dy and A. Krause, editors, Proceedingss of the 35th International Conference Machine Leurning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15 2018 volume 80 of Proceedings of Machine Learning Research, pages 1406-1415. PMLR, 2018
11

---

[22] T. A. for Computing Machinery. Acm a.m. turing award honors two researchers who led the de- velopment of comerstone ai technology. htttps///wward: acm. org/ aabbouut/224-ttttt turing Accessed: 2025-05-15.
[23] M. Ghavamzadeh, M. Petrik, and Y. Chow. Safe policy improvement by minimizing robust baseline regret. In D. D. Lee, M. Sugiyama, U. von Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2298-2306, 2016.
[24] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
[25] S. Greydanus, A. Koul, J. Dodge, and A. Fern. Visualizing and understanding atari agents, In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1787-1796. PMLR, 2018
[26] H. Guo, Q. Cai, Y. Zhang, Z. Yang, and Z. Wang. Provably efficient offline reinforcement learning for partially observable markov decision processes. In International Conference on Machine Learning, pages 8016-8038. PMLR, 2022.
[27] H. Guo, Q. Cai, Zhang, Z. Yang. and Z. Wang. Provably efficient offline reinforcement learning for partially observable markov decision processes. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 8016-8038. PMLR, 2022.
[28] M. J. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12-14, 2015, pages 29-37. AAAI Press, 2015.
[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016
[30] M. Hessel, M. Kroiss, A. Clark, I. Kemaev, J. Quan, T. Keck, F. Viola, and H. van Hasselt. Podracer architectures for scalable reinforcement learning. CoRR, abs/2104.0621 2021.
[31] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In S. A. Mcllraith and K. Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3215-3222. AAAI Press, 2018
[32] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 173: 1780, 1997.
[33] S. Huang, R. F. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mchta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
[34] G. W. Imbens and J. D. Angrist. Identification and estimation of local average treatment effects. Econometrica, 02(2):467-475 1994.
[35] G. W. Imbens and D. B. Rubin. Bayesian inference for causal effects in randomized experi- ments with noncompliance. The annals of statistics, pages 305-327, 1997.
[36] G. N. Tyengar. Robust dynamic programmin...... Mathematics of Operations Research, 30(2):257- 280, 2005.
12

---

[37] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmact In International Conference on Learning Representations, 2017.
[38] N. Jiang and L. Li. Doubly robust off-poliey value evaluation for reinforcement learning. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 022-661,5 New York, New York, USA, 20 20-22 Jun 2016. PMLR.
[39] Joshi, J. Zhang, and E. Bareinboim. Towards safe policy learning under partial identifiability; A causal approach. In M. J. Wooldridge, J. G. Dy, and S. Natarajan, editors, hhirty-Eightt AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference Innovative Applications of Artificial Intelligence, IAAI 2024, Fourleenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 13004-13012. AAAI Press, 2024.
[40] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. 7idek, A. Potapenko, et al. Highly accurate protein structure prediction with 2021.
AlphaFold. Nature, 596(7873)
[41] Y. Jung, J. Tian, and E. Bareinboim. Learning causal effects via weighted empirical risk mini- mization. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[42] Y. Jung, J. Tian, and E. Bareinboim. Estimating identifiable causal effects through double machine learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirly-Thine Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, FAAI 2021, Virtual Event, February 2-9, 2021, pages 12113-12122. AAAI Press, 2021.
[43] N. Kallus and A. Zhou. Confounding-roobe policy improvement. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Gamett,, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 9289-9299, 2018.
[44] N. Kallus and A. Zhou. Confounding-robunn policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems, 15:22293-22300 2020.
[45] B. Kang, Z. Jie, and J. Feng. Policy optimization with demonstrations. In J. G. Dy and Krause, editors, Proceedings of the 35th International Conference on Machine Learn- ing. ICML 2018, Stockholmsmāssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2474-2483. PMIR, 2018
[46] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[47] C. Kausik, Y. Lu, K. Tan, M. Makar, Y. Wang, and A. Tewari. Offline policy evaluation and optimization under confounding. In International Conference on Artificial Intelligence and Statistics, pages 1459-1467 PMLR, 2024.
[48] S. Khan, M. Saveski, and J. Ugander. Off-policy evaluation beyond overlap: partial identifica- tion through smoothness, arXiv preprint arXiv:2305.1 11812, 2023.
[49] B. R. Kiran, Sobh, V. Talpaert, P. Mannion, A. A. A. Sallab, K. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Trans. Intell. Transp. Syst., 23(6):4909-492 2022.
[50] A. Kumar, A. Zhou, G. Tucker, and Levine. Conservative l-aarnnn for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
13

---

[51] D. Kumor, J. Zhang, and E. Bareinboim. Sequential causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 2021.
[52] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436-444, 2015.
[53] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[54] M. Li, J. Zhang, and E. Bareinboim. Automatic reward shaping from confounded offline data. In Forty-second International Conference on Machine Learning, 2025.
[55] S. H. Lim, H. Xu, and S. Mannor. Reinforcement learning in robust markov decision processes. In C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 701-709, 2013.
[56] S. H. Lim, H. Xu, and S. Mannor. Reinforcement learning in robust markov decision processes. Math. Oper. Res., 41(4)::325-1355 2016.
[57] L.J.. Lin. Reinforcement learning for robots using neural networks. Camneii Mellon Univer- sity, 1992.
[58] M. Lu, Y. Min, Z. Wang. and Z. Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OvenRevie,.net, 2023.
[59] C. F. Manski. Nonparametric bounds on treatment effects. The American Economic Review, 80:319-322 1989.
[60] R. Miao, Z. Qi, and X. Zhang. Off-policy evaluation for episodic partially observable markov decision processes under non-parametric models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[61] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-53) 2015.
[62] H. R. Moon and F. Schorfheide. Bayesian and frequentist inference in partially identified models. Econometrica, 80(2):755-782, 2012.
[63] F. Morales, G. Sheu, and A. Zahler. Extended gravity. The Review of economic studies, 86(6):2668-27) 2712, 2019.
[64] M. Moravčík, M. Schmid, N. Burch, V. Lisý, D. Morrill, N. Bard, T. Davis, K. Waugh, M. Johanson, and M. Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-ii poker. Science, 356(6337):508 01388551112000 2017.
[65] R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054- 1062, 2016.
[66] M. Nakamoto. S. Zhai, A. Singh, M. S. Mark, Y. Ma. C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline RL pre-training for efficient online fine-tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[67] H. Namkoong, R. Keramati, S. Yadlowsky, and E. Brunskill. Off- Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances Neural Information Processing Systems, 33:18819- 188889918888 18831, 2020.
14

---

[68] A. Nilim and L. El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[69] A. Norets and X. Tang. Semiparametric inference in dynamic binary choice models. Review of Economic Studies, 81(3):1229-1266 2014.
[70] OpenAI, Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, McGrew, A. Petron, A. Paino, M. Plappert, G. Powell. R. Ribas. J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with robot hand. arxiv, 2019.
[71] K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh. Robust reinforcement learning using offline data. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[72] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, edition, 2009.
[73] M. Petrik and R. H. Russel. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buec Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7047-7056, 2019.
[74] L. Pinto, J. Davidson, Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2817 PMLR, 2017.
[75] J. J. Poirier. Revising beliefs nonidentified models. Econometric theory, 14(4):483-509 1998.
[76] D. Precup, R. S. Sutton, and S. P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766, 2000.
[77] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statisties. Wiley, 1 edition, 1994.
[78] H. Robbins. Some aspects of the sequential design of experiments. In Herbert Robbins Selected Papers, pages 169-177. Springer, 1985.
[79] J. P. Romano and A. M. Shaikh. Inference for identifiable parameters in partially identified econometric models. Journal of Statistical Planning and Inference, 13869)2788-22877, 2008.
[80] A. Roy, H. Xu, and S. Pokutta. Reinforcement learning under model mismatch. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3043-3052, 2017.
[81] K. Ruan, J. Zhang, X. Di, and E. Bareinboim. Causal imitation for markov decision pro- cesses: A partial identification approach. Advances in Neural Information Processing Systems, 37:87592-87620. 2024.
[82] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Y. Bengio and Y. LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[83] J. Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
15

---

[84] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-Din Bigh-Dimensiona Continuous Control Using Generalized Advantage Estimation. In Y. Bengio and Y. LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, Sun Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
[85] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arxiv, 2017.
[86] M. Schwarzer, S. Obando-Ceron, A. C. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro. Bigger, better, faster: Human-lewe level atari with human-level efficiency. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt. S. Sabato, and J. Scarlett, editors, International Confer- ence on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. volume 202 of Proceedings of Machine Learning Research, pages 30365-30380. PMLR, 2023.
[87] C. Shi, M. Uehara, J. Huang, and N. Jiang. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 20057-20094 PMLR, 2022.
[88] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto- cessto--lasticc object manipulation with diverse tools. In J. Tan, M. Toussaint, and K. Darvish, editors, Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pages 662-660. PMLR, 2023.
[89] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge. Nature, 35047676)3344 359, 2017.
[90] Song, Y. Jiang, S. Tu, Du, and B. Neyshabur. Observational overfitting in reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[91] Y. Song, Y. Zhou, A. Sekhari, D. Bagnell, A. Krishnamurthy, and W. Sun. Hybrid RL: using both offline and online data can make RL efficient. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[92] J. Stoye. More on confidence intervals for partially identified parameters. Econometrica, 1794): 2009.
[93] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, second edition, 2018
[94] A. Swaminathan and T. Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pages $14-823. 2015.
[95] A. Tamar, S. Mannor, and H. Xu. Scaling up robust mdps using function approximation. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 181-189 JMLR org, 2014.
[96] E. J. T. Tchetgen, A. Ying, Y. Cui, X. Shi, and W. Miao. An introduction to proximal causal learning, 2020.
[97] P. Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence policy improvement. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 2380-2388, Lille, France, 07-09 Jul 2015, PMLR.
[98] J. Tian and J. Pearl. Probabilities of causation: Bounds and identification. Ann. Math. Artif. Intell., 28( 2000.
16

---

[99] D. Todem, J. Fine, and L. Peng. A global sensitivity test for evaluating statistical hypotheses with nonidentifiable models. Biometrics, 66(2):558-500 2010.
[100] M. Towers. A. Kwiatkowski, J. K. Terry. J. U. Balis, G. D. Cola, T. Deleu, M. Goulão, Kallinteris, Krimmel, A. KG, Perez- Vicente, Pierré, S. Schulhoff, J. J. Tai, H. Tan, and O. G. Younis. Gymnasium: A standard interface for reinforcement learning environments. CoRR, abs/2407. 2024.
[101] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In D. Schuurmans and M. P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100. AAAI Press, 2016.
[102] Y. Wang and S. Zou. Online robust reinforcement learning with model uncertainty. In M. Ranzato, A. Beygelzimer, N. Dauphin, P. Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 7193-7206, 2021.
[103] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. Dueling network architectures for deep reinforcement learning. In M. Balcan and K. Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1995-2003. JMLR.MMR.O.... 2016.
[104] C. J. Watkins and Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.
[105] J. C. H. Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989.
[106] C. C. H. Watkins and P. Dayan. Technical note q-learning, Mach. Learn., 8:279-292, 1992.
[107] W. Wiesemann, D. Kuhn, and B. Rustem. Robust markov decision processes. Math. Oper. Res., 38(1):153-183, 2013.
[108] H. Xu and S. Mannor. Distributionally robust markov decision processes. Mathematics of Operations Research, 37(2):288-300, 2012.
[109] P. Yu and H. Xu. Distributionally robust counterpart in markov decision processes. IEEE Transactions on Automatic Control, 61()):253-225) 253--2533, 2016.
[110] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C. Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. December 6-12, 2020, virtual, 2020.
[111] J. Zhang and E. Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes. Advances in Neural Information Processing Systems, pages 13401-13411, 2019.
[112] J. Zhang and E. Bareinboim. Can humans be out of the loop? In Conference on Causal Learning and Reasoning, pages 1010-1025. PMIR, 2022.
[113] J. Zhang and E. Bareinboim. Eligibility traces for confounding robust off-policy evaluation: A causal approach. In 41st Conference on Uncertainty Artificial Intelligence (UAI), 2025.
[114] J. Zhang, D. Kumor, and E. Bareinboim. Causal imitation learning with unobserved con- founders. Advances neural information processing systems, 33:12263-12274 2020.
[115] J. Zhang, J. Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. International Conference on Machine Learning, pages 26548-26558. PMLR, 2022.
17

---

[116] Y. Zhu, Z. Wang, J. Merel, A. A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In H. Kress-Gazit, S. S. Srinivasa, T. Howard, and N. Atanasov, editors, Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification; All claims are supported by both theoretical proofs (Prop. 3.1 and empirical results on 12 confounded Atari games reported in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in a separate section in the appendix (App. G and shed light possible future works the conclusion section (Sec. For the experiment, we entail the computational requirements in App. Dpp.Daam and the environment setup, data preparation in both Sec. 4 and App. D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The detailed assumptions and proof setups are in Sec. 2 and Sec. The proof detail for Prop. 3.1 is in App.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the necessary details for reproducing our work including preparing confounded Atari games, neural network architectures. training hyper-par prrrmetrreeeers and pseudo-code in Algo. (and a more refined version Algo. 2) in both Sec. 4 and App. D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Our experiments are based on open benchmark (Atari from Gymnasium environments). And the detailed parameters and setups for generating confounded Atari games are also reported in Sec. 4 The demonstrator generating the confounded data is also open sourced model, see https://ggthabccommlllaaooosdddt
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
htlpii//guides//ddeuuumiiisses iociooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. htaplss.ccppublic/uuueeee httpa///gguiiee//eeeuubbiiaiii swt.suubiisiiiooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the full details of training/cvaluution pipeline including model architectures, hyper-parameters data preparation and evaluation metrics Sec. and
App. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We follow the recommended evaluation protocol in 1 We provide both mean and IQM on the returns (Table [ Fig.
Fig. we also report standard deviation as the
shaded area around curves.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computational resources used in each experiment in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We do not foresee any harms caused by the research process since no human subjects are involved in our work, and the environments we use are open-sourced Atari games. We also do not foresee any direct harmful societal impact of our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification: We discuss the motivation and contributions the introduction section (Sec. under a broad picture. We also discuss the social impacts further in App. F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We will only release the trained confounding robust Atari game agents, which we believe won't 't pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For the Atari games we cite the Gymnasium suite in Sec. and for the demonstrator model we also cite the original paer in Sec.
Guidelines:
The answer NA means that the paper does not existing assets.
23

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce a new set of confounded Atari games with demonstrator data. We have listed all details required generate such games in Sec. and App.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human objects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non standaadd components.
Please refer to our LLM policy for what should or should not be described.
25

---

Appendices
Contents
Related Work
26
B Proof Details
27
C Confounded Átari Games Design
29
D Implementation Details and Experiment Setups
30
More Experiment Results
30
Broader Impact
33
G Limitations
33
A
Related Work
Off-Policy Learning. Off-policy learning has long history in RL dating back to the classic
algorithms of Q-learning [105] [06]. importance sampling [3]]44] and temporal difference [65]
76
Recently, people also propose to utilize offline datasets warm start the training [16
50
66
augmenting online training replay buffer [8. 91] or incorporating imitation loss with offline data 45
[16). However, these work rely on a critical assumption that there is no unobserved confounders in the environment. While this assumption is generally true when the off-policy data is collected by an interventional agent, data generated by potentially unknown sources can casily break this assumption [53]. We will introduce more on the confounding robust off- off-policy learning next.
Causal Reinforcement Learning for Off-Policy Learning When the no unobserved confounding assumption does not hold, one would need to either identify the reward and transition distributions before evaluating policy values bound the possible policy values. There is rich line of literature But they usually invoke other critical learning assumptions such as the existence of bridge functions in the line of proximal causal inference literature [96). On the other hand, without further assumptions, one can utilize the bounding method to account for the whole range of possible policy values. Seminal work of Manski [59] developed the first bounds on causal effects in non non-identifiable settings using observational data in the single-stage treatment model with contextual information (i.c., contextual bandit model). These bounds were then expanded to the instrumental variable setting [4]], to partially identify counterfactual probabilities of causation [98], to construct reward shaping functions automatically [54]. This work is inspired by a recent work of Zhang & Bareinboim in partially identifying the policy values via bounding [I3].
in identifying policy values directly from confounded data 13 27
60 87
Robust Reinforcement Learning Unlike reinforcement learning in standard MDPs, robust reinforce- ment learning assumes that the parametrization of the transition probability function is contained in a set of model parameters which called the uncertainty set (36] [66, 68] [7] 107 The goal of the agent is learn a robust policy that performs the best under the worst possible case in the uncertainty set. Similar problems have been studied under the rubrics of safe policy lcarning [23]97] 97]
or pessimistic reinforcement learning (58). Robust RL algorithms with provable guarantees have been proposed in tabular settings or under the assumptions of linear functions [3] 55 80 95 102]
Combined with the computational framework of deep learning, robust RL algorithms have been extended to complex, Fitted Q-Iteration (RFQI) to learn the best possible robust policy from offline data with theoretical guarantees on the performance of the learned policy. Our work differs from robust RL methods since it does not require a pre-specified uncertainty set of model parameters. Instead, we construct the ignorance region over the underlying system dynamics from the confounded observational data
sigh--imensional domains 774, 110.. More recently, [71] proposed Robust
26

---

using partial causal identification. Based on the learned uncertainty set, we then derived closed-form bounds over the optimal interventional Q-value functions.
Model-Free Deep Reinforcement Learning Model-free deep reinforcement learning has seen substantial advancements through the evolution of value-based methods, particularly those building on the foundational Deep Q-Network (DQN) While DQN introduced the use of deep neural networks to approximate Q-values from high-dimensional sensory inputs, it suffered from issues such as overestimation bias, instability, and poor data efficiency. These challenges prompted a series of algorithmic improvements. Double DQN [IO]] addresses overestimation by separating action selection from action evaluation, while Dueling DQN [003] improves representational efficiency by decoupling value and advantage estimation. Prioritized Experience Replay [82) and Hindsight Experience Replay [ enhances data efficiency by sampling more informative transitions. Rainbow DQN 31 effectively combines these ingredients, along with multi-step learning, distributional Q-learning, and noisy networks, resulting in one of the strongest baselines in Atari environments.
To scale value-based methods to more complex tasks and hardware infrastructures, subsequent works introduced distributed and more flexible architectures. IMPALA (Importance Weighted Actor-Learner Architectures) [II tackles the inefficiency of distributed policy learning by decoupling acting and learning processes and correcting the resulting off-policy updates with the V-trace algorithm, enabling efficient parallel training at scale. R2D2 (Recurrent Experience Replay Distributed Reinforcement Learning) [46] enables recurrent architectures to be trained off- off-policy in distributed settings with experience replay, supporting partial observability and long time horizons. Agent57 4 further builds on R2D2, combining exploration bonuses from Random Network Distillation [9, meta-learning of exploration strategies, and population- based training, becoming the first agent to surpass human performance on all 57 Atari games. More recently, the Bigger, Better, Faster (BBF) framework [86] pushes the scalability frontier by optimizing infrastructure, neural network design, and training pipelines, enabling the training of large-scale agents with dramatically improved performance and sample efficiency, These innovations collectively mark significant advancement toward more scalable and generalizable value-based deep reinforcement learning agents.
Note that our proposed Causal Bellman Optimality Equation does not require specific reinforcement learning algorithm implementations. In this work, we choose DQN as the base algorithm only for its simplicity given our goal of showcasing a practical implementation of the proposed result in a straightforward way. It is possible to extending the causal Q-learning update in Eq. (8) to be used with more advanced deep RL algorithms (e.g.. Rainbow [31, IMPALA (21, and BBF (86)) so that one could enable more powerful agents in confounding settings, which we will leave for future work.
B
Proof Details
This section entails the proof for Prop. 1.1 We also prove that our Causal Bellman Optimal Equation (lower bound) has a unique fixed point that is a valid lower bound.
Proposition B.1 (Causal Bellman Optimal Equation (Prop. 3.I.. For CMDP environment M with reward signals Yz E [a, b] C R. its optimal catte-cction value function Q, x) 2 Q= x)) for any state-action pair (s,z) E S x X, where the lower bound Q (s, x) is given by as follows.
(s, = P(# (s) R x)) + +
Σ T 28, s'.x'
8') max (=')
(12)
x²
+P(-x 8) a min min max Q (s's ,x")
(13)
and P(-x 8) 1 - P(x s); T and R are nominal transition distribution and reward function computed from the observational distribution, i.e.
where P(x 1 8) = PXx
T (s. x. 80) P (S 8' S, = X, =),, Rs, = E \Y, S. x, =x] (14)
Proof. Starting from the Bellman Optimal Equation for Q-valuss, the optimal state action value function is given by.
Q'(s = R(s, + T(ss x. s') max Q'(('xz')
(15)
o²
27

---

where (x) E|Y, = s, z). Tr is shorthand for (s, x. s') P(Sh+1 = s'|Si Shh X₁, z) and P(r|s) (X, x Sk s) are estimated from the offline dataset. And is known lower bound on the reward signal, Y, S b. In this step, we lower bound the next state transition by assuming the worst case that for the action not taken with probability Pa(-arla the Note that the actions in the reward and transition functions are done by an interventional agent, which actually (o(x) in the context of CMDP. Due to the confounding nature of those two distributions, we can use the natural bounds to bound the interventional reward (R) and transition distribution (7)
with observational data (R,T) [59]
R(s,x) 2 x(srs) + aP(
(16)
8') (s 2 Σ T(s (z|s) max Q' (s',
P min max Q' (s',
(17)
x
Then have,
("(s,)) R(s, + aP((-als)
Σ 8' P(rl)) max (d', + P(-o)))) min max (6''2=²)
(18)
x'
x²'
agent transits with probability to the worst possible next state, ming" Vitz((")
Then after rearranging terms, have,
Q*(s, 2 P(x 8) (((,0)+
Σ (s, (m) max Q' (s,,)
6'
+ +P( 8) a a min max Q'
Q' (s')
(19)
Optimizing the Q-value function w.r.t this inequality gives us lower bound on the optimal state value. Replace the symbol Q* with Q+ and we have,
Q+(s,)) 2 P(x s) R (2,2)) + ET max max Q+'ss
s'.x'
+ +P(-x 8)
a + m max max Qu (s' (s',
(20)
Next, we will show in Prop. B.2 that this will converge to unique fixed point, which is a valid lower bound of the optimal state-action value function.
Proposition B.2 (Convergence Causal Bellman Optimal Equation in Stationary CMDPs). The Causal Bellman Optimality Equation converges unique fixed point, which is also lower bound on the optimal interventional state values under the assumption that in the observational data P(s,x) 0, Vs, in the given CMDP.
Proof. We will first show that the following Causal Bellman Optimality operator (will denote as "the operator" or 7 below for simplicity) is contraction mapping with respect Banach's 's fixed-point theorem [9]. this operator has unique fixed point, and updating any initial point iteratively will converge it. Then show that this unique fixed point is indeed lower bound of the optimal interventional Q-value.
norm. Then by
Let the operator T be.
(Q* P(z (((,))) +g E
max Qa max x'
s'
+ +P( 8)
a 1 min max Q2(s'
(21)
28

---

For arbitrary Q-value bound, Q. Q. let their initial difference under max-norm be
max (5) QZ(s, 2)) 2 We can bound their difference after one step update by,
S max P(x|s) Σ T(s max Q1(('xx') - 22(6',z'))
max
+P(-xls) max (. - (2' x²)
(22)
Thus, under the operator T, we have non-expansion Q-value differences,
max
(s, - TQ
$ max 8,2
P(als) T(s, s') max Q.(5', x')
-2))
(2(6(, ,²))
+ P(--o())
max 5',2'
Q. (2()'
(23)
$ yemax
P(x|s) Σ
2(), + P(-xls)
(24)
= yc.
(25)
for all Q O satisfying maxs,a (+, (s, O. Thus, T is a contraction mapping with respect to the max nomm. And there exists unique fixed point Q. when we apply this operator T iiteratively an arbitrary Q-value vector till convergence.
We then show that this fixed point indeed lower bound to the optimal interventional Q-value. By the update rule of T (Eq. (21)) (s(s,an x) (8,x)) > Ts),, Thus, for the optimal Q-value, can have (s, x) 2 lim T*x2" Q (s, x) where T denotes applying T iteratively for times. This concludes the proof.
C Confounded Atari Games Design
In this section, we present the detailed design of each confounded Atari games. The core design idea is that we would like to occlude the part of the screens that contains information useful for making decisions but is not significant factor from human players' perspectives. For example, the remaining lives and current scores can be an indicator the difficulty level or even a unique game level identifier. However, such information is not usually exploited intensively in human game plays. Thus, we decide to mask out such regions.
Below is detailed list of confounders design for each game.
The original game screen shows score and remaining life which shouldn't be the major factor affecting the policies.
Amidar:
Asterix: The original game screen shows score and remaining life which shouldn't be the major
factor affecting the policies.
Boxing: The original game screen shows score and remaining time which shouldn's 't be the major factor affecting the policies. The right half of the arena can also be excluded since there is still wining strategy by staying the left hand side (as long the demonstrator has such demonstrations on the left hand side).
Breakout: The original game screen shows score and remaining life which shouldn't 't be the major factor affecting the policies.
ChopperCommanke The original game screen shows score and remaining life which shouldn't t be the major factor affecting the policies. The mini map can be helpful, but without there should also be a good policy.
Gopher: The original game screen shows score and remaining life which shouldn't 't be the major factor affecting the policies. The gopher location, while nice have, be removed to force the learning to focus more on the hole not the gopher.
KungFuMaster: The original game screen shows score and remaining life which shouldn't 't be the major factor affecting the policies.
29

---

MsPacman: The original game screen shows score and remaining life which shouldn't 't be the major factor affecting the policies.
Pong:
The original game screen shows score and remaining life which shouldn't be the major
factor affecting the policies. Moreover, one can simply win the game by shoot back every incoming ball. Thus, we can further block out the opponent's s paddle location.
Qbert: The color itself is already good confounder. The demonstrator model (2) is observing three channel RGB images with LSTM cells. The student model only has grayscale image
stacks. See more details in App. F
RoadRunner: The original game screen shows score and remaining life which shouldn't be the major factor affecting the policies. Also the sky/desert part are mostly
Seaquest: original game screen shows score and remaining life which shouldn't be the major factor affecting the policies.
We show each masked atari games in Fig.
D Implementation Details and Experiment Setups
The input to all the networks consists of a stack of four grayscale frames with frame skipping of four (so the agent observes a stack of four out of sixteen consecutive actual frames), each down sampled to 64x64 resolution, allowing the agent to infer temporal dynamics such as ball velocity. To reduce the inherent flickering from the Atari game, we also apply max pooling over each two consecutive actual frames. We also downsize the input to 64x64 to align with the input size requirement of the demonstrator, diamond (2). The only differences in terms of observations are that (1) the demonstrator
takes single colored frame (also max-pooled) input per each time step, and (2) the demonstrator
has access to the original full screen observation while the learners only has access to a masked partial screen. For the reward, we clip it between [-1,+1] to stabilize training.
For the other demonstrator, we use the sebulba model (30] implemented by the CLeanRL package 33]. Its backbone is from Impala [2I] architecture and the training algorithm is PPO 883 Its input image is in 81x81 but still are stacked grayscale images, the same as what the DQN agents use. Actions are selected the argmax action with the biggest logits after applying the Gumbel softmax trick [37]. The training batch size is also increased to 2048 and is trained with cosine annealing learning rate scheduler. For all runs, the learning rate is set to 5e-4. For the consine annealing learning rate scheduler, the minimum learning rate is 1e-6.
For all CNN based DQN networks in this work, we adopt the nature DQN architecture introduced by Mnih et al. [61] The network comprises three convolutional layers followed by two fully connected layers, outputting Q-values for each discrete action. While for LSTM based ones, we only replace the second to last linear layer in nature DQN with an LSTM cell. For both the linear layers and Istm cells, we use hidden dimension of 512. To mitigate overestimation bias in Q-learning, we further incorporate the Double DQN modification [IOI], which decouples action selection and evaluation in the target update by using the online network to select actions and the target network to evaluate their value. This leads to more stable and accurate value estimates. We also use epsilon greedy for exploration as the standard DQN algorithm. See Algo. 2 for the full pseudo-code.
To train our model, we use an H100 GPU, On average, for each game and each seed, takes around 2 hrs and RAM space of less than 2 GB for using the diamond demonstrator 22) While it takes up to hrs for using the sebulba demonstrator (30 from CleanRL 33
E More Experiment Results
Here we present the experiment results for agents trained with another even stronger demonstrator, sebulba [30]. From Table Fig. T0 we can draw the same conclusion that our Causal-DQN is robust confounded demonstrator demonstrations and is able to extract useful policies out of such demonstrations. And we can see that in most of the games, the agent trained by sebulba demonstrator significantly outperforms the agent trained by diamond demonstrator 2 This can be an empirical evidence that our proposed Causal-DQN is able to scale with the demonstrator's performance. But his also reveals two limitations of our current approach that when the demonstrator generated data has zero support in the unmasked area, it is not possible to learn any meaningful behaviors from such
30

---

Masked Amidar
Masked Asterix
Masked Boxing
Masked Breakout
Masked ChopperCommant
Masked Gopher
Masked Kuster
Masked MsPacman
Masked Pong
Masked Qbert
Masked RoadRunner
Masked Seaquest
Figure 9: All 12 confounded Atari games. Masked areas are shown grey.
demonstrations. In our current setup of the confounded Boxing game, we mask the right half but the sebulba demonstrator has policy of fighting in the right half. Thus, none of the algorithms we tested can learn. To further verify this intuition, we mask out the left hand side of the arena in the Boxing game (App. Ep and rerun all baselines and Causal-DQN. As expected, as shown in App. E our model is now able converge the optimal 100 score matching the demonstrator's 's performance while other baeslines still struggle to learn from the confounded demonstrations. Also, when the demonstrator's policy is distributional and multi-modal, i.e., there could be multiple best actions, the deterministic policy like DQN cannot capture such knowledge very well. In Asterix, due to the non-standard way using Gumbel softmax implemented by CleanRI. 33 there could be different optimal actions for the same state, posing challenge to the deterministic learners.
31

---

8.0
022
Eastroment Stags, lec
1.0
0.8
Envinonent Steps, lef
66
13
00
Erocsinez Staps, le6
38
10
Niccerd Tersooneoofrreeddeett
Maxked Chaperrrineettttrraannnee
Mivded ExpperrStttrreeeeerr
1903
1201
1593
1100
620020
1933
1000
Destroment Stape,
Envimment Sleps, Masked MoleeaanNoheneeee
Dnstrtment Staps,
Masied Kwn.huhaaaarSooooooooooo
Masked PoogNirrameskip..
2001
2103
17808
0200
100
13
115 Endebrocat
14200
51000
1900
12000
11893
22000
3300
300
Eestroment Strre, Left
Envimenent Stace. Ind
Enotesment Stane, 1nd
Intevvenional
Figure 10: Average evaluation performance for 12 confounded Atari games with sebulba 330 the demonstrator. During training over the 1M environment steps, we evaluate the agent every 100K steps for 10 episodes each time. The curve is further averaged over seeds with one standard deviation across trials as the shaded area.
Causal DQN (Ours)
DQN w/ Intery. Data
DQN Conf. Data
LSTM-DQN Conf. Data
0.0
0.3
0.6
0.9
1.2
0.0
0.3
0.6
0.9
1.2
Normalized Mean
Normalized IQM
Figure 11: Normalized mean and normalized IQM scores. Causal-DQN achieves normalized mean return of 0.55 and a normalized IQM of 0.59 with sebulba demonstrator.
32

---

Algorithm 2 Causal Deep Q-Learning (Causal-DQM)
1: Initialize replay memory D
2: Initialize action-value function Q. and target network Q. Q ,0- t
3:
for episodes = l...., M do
4: Sample initial state s and obtain preprocessed Φi o(s1)
5: for 1,...,,,. T do
6:
With probability select random action 2t
7:
Otherwise sample an action from demonstrator, 24 <- fxx ($t, Wt)
8:
Execute action do(æ;) in environment and observe reward yr and state 81+1
9:
Store transition (s,, Fe, Mt, St+1) in !
Sample minibatch of transitions {(s;, T4, Vi, 81+1)] from D
11:
Set value target w/ (2:) for every action E X' w.r.t sample (si, (s,, Us. 8:41),
[ yi max,' (';441,1;;07
if
wy (z)
(26)
a + ming max, Q. (8',''';00 x²; i
12:
Perform a gradient descent step on Ex (w; (अ) ) (s1, according to Eq. (10)
13:
Every Turget steps, update A- +- end for
15: end for
Table 2: Average evaluation returns of agents on the 12 confounded Atari games trained with 1M environment steps and aggregated normalized returns concerning the sebulba demonstrator's performance [30]. Bold numbers indicate the best- best-performing methods. All results are averaged over seeds except that column Random from 2] Causal-DQU significantly outperforms others.
Demonstrator (schulha)
Interv. DQN
Conf. DQN
Conf. LSTM-DQN
Causal DDQ (diamond)
Causal-Doe (sebulba)
Game
Random
Amidar
2148.2
5.8
44.0
22.6
24.6
282.6
462.8
Asterix
250182.0
210.0
650.0
369.0
662.0
2587.0
586.0
Boxing
100.0
0.1
-0.62
-7.6
-13.4
71.5
-6.886
Breakout
771.0
1.7
2.2
0.9
2.9
131.2
408.2
ChopperCommaan
21682.0
811.0
1192.0
1096.0
918.0
1658.0
5410.0
Gopher
3719.2
257.6
288.8
646.4
132.0
7327.2
4008.0
Kunste
46046.0
258.5
12416.0
8468.0
4844.0
44196.0
39222.0
MsPacman
4538.4
307.3
1191.6
963.6
561.4
1747.6
2346.8
Pong
21.0
-20.7
-20.8
-20.8
-20.6
21.0
21.0
Qbert
23484.0
163.9
322.5
283.5
136.5
4458.5
15136.0
RoudRunner
56056.0
11.5
1154.0
1182.0
1108.0
27414.0
49482.0
Seaquest
1797.2
237.2
247.6
101.6
980.0
1350,0
Normalized Mean (
1.00
0.00
0.00
0.00
0.00
0.55
Normalized Median (t)
1.00
0.04
0.02
0.40
0.59
Normllizel IQM (t)
1.00
0.0
0.02
0.02
0.02
0.47
0.59
F Broader Impact
This paper presents work whose goal is to advance the field of Reinforcement Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. One major reason is that our proposed algorithm aims at extracting knowledge from another demonstrator model solving Atari games of which we don't find any profound social impacts worth mentioning here.
G Limitations
Our current derivation applies to single step Q-value functions. For other objectives for critics like multi-step returns, eligibility traces and advantages, we need further extend the Causal Bellman Equation to accommodate those. As we also show in App. E the proposed Causal-DQU cannot learn useful policies when the demonstrator policy no support in the unmasked area. For example, in the boxing game, mask out the right half of the screen while the demonstrator agent's 's winning
33

---

Masked Boxinggoorraesski-xt
110
80
Relurn
50
Causal DQN (Ours) Interventional
DQN Confounded Dota
20
LSTM-DON w/ Confounded Data
-10
0.0
0.2
0.6 Enviroment Steps, 1e6
0.4
0.8
1.0
Figure 12: Left: The confounded Boxing game with left side arena masked. Right: Evaluation returns of Causal-DQU and other baselines. The curve is an average over five random seeds with one standard deviation as the shaded area. The orange curve our Causal-DQN and other colors baselines. Same legend following previous figures.
policy to fight in the right half. In which case, none of the agent under masked observations is able to learn. Furthermore, in games like Asterix where distributional demonstrator has a more stochastic behavior, our Causal- DQN also cannot outperform others. We hypothesis this to be an inherent representational limit of deterministic DQN policies.
34


Reviews:
[{"review_id": "4uxUSt8dli", "reviewer": "Reviewer_czit", "summary": "This paper proposed a DQN algorithm for an off-line RL environment in the presence of an unobserved confounder. The experiment and explanations in the paper primarily address the source of confounding as missing state information during the data collection stage by behavioral policy. This paper also mentions earlier works such as CMDP and the closed-form expression of the Q-function in the presence of an unobserved confounder. The contribution is to extend the optimal value function with a lower bound and propose an off-line learning algorithm. In practice, it can be very challenging to engineer an RL environment with unobserved confounders. This paper shows 12 modified Atari domains based on salience map analysis.", "questions": "Please confirm if the scope of the paper is correct. The decision problem in this paper addresses the off-line RL problem, where an agent has sampled the experience before. The unobserved confounder enters the problem by assuming that the agent that generated the experience might have observability on the state variables that are not present in the collected dataset (Line 60-62).\nTo me, defining CMDP seems like it will address the scenario beyond what's described above. \n\nFigure 1 illustrates that the observed action, the next state variable, the observed action and reward, and finally the reward and the next state variable can be confounded. In Line 77, when the data generation process is explained, it appears that only one unobserved confounder variable ($U_t$) is considered. Do you also cover the factored state and action space MDP?\n\nIn the appendix, is there a proof of Eq. (3) on the Bellman equation for the given setting, or is it from some earlier works?\n\nFigure 2 and Example 1 are too extreme. It removes the opponent in the game and trains a conventional DQN agent that needs to learn a policy by observing the ball only. I don't think this is a realistic example of an unobserved confounder, although it can be used to contrast the performance as shown in Figure 2(d). In this case, how does employing a behavior policy perform in the modified state space?\n\nThe notations in Proposition 3.1 require further elaboration, particularly those with underscores.\nDoes the lower bound hold whether the value function is identifiable or not?\n\n\nIn Algorithm 1, it is an offline method that stores the entire dataset in memory. How to sample transitions in an offline environment? Does the step that samples the initial state $s_1$ work? How can the behavior policy sample an action with a modified state space if it were trained in the original state space? What is the added complexity of the additional sampling in Line 196? In line 204, the opponent is unobserved to the offline learning agent. Does it also see the opponent's location?\n\n\nIn line 243, the optimal policy stipulates that it only needs to see the ball location and its paddle, which is the off-line setting that removes the opponent's visual information and the scoreboard. Then, why did the traditional DQN fail?\n\nBesides, the evaluations were done over 10 episodes across 5 seeds in Figure 8. Would it be possible to enhance the traditional DQN by utilizing different hyperparameters or exploration strategies? I think the evaluation may be favorable to the proposed method.\nFollowing the explanation, the proposed method operates by sampling the transition using both the behavioral policy and the reward. While learning the Q-network, it samples additional points and takes the pessimistic value instead of committing to a single state in Eq. (11).", "limitations": "YES", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\nThe strength of this paper is to handle the ATARI domain in a causal RL setting, which is a non-trivial contribution.\nMany existing benchmarks and algorithms assume the presence of causal variables or causal graphs.\nThe experiment is very novel that I haven't seen in other papers yet.\n\n**Weakness**\nThe role of the unobserved confounder in the problem needs to be discussed in depth. The data generation process doesn't fit well with the data corruption process used in the experiment.\nFrom the RL evaluation viewpoint, the experiment itself is insufficient in terms of the number of trials and hyperparameter optimization.\nThere are supporting theoretical results, such as Proposition 3.1 and the equations in the paper, that need to be elaborated upon in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful feedback. We appreciate your acknowledgment of the significance of our work and have addressed your concerns in the sequel.\n> **W1** “The role of the unobserved confounder in the problem needs to be discussed in depth. The data generation process doesn't fit well with the data corruption process used in the experiment. From the RL evaluation viewpoint, the experiment itself is insufficient in terms of the number of trials and hyperparameter optimization. There are supporting theoretical results, such as Proposition 3.1 and the equations in the paper, that need to be elaborated upon in the paper.”\n\nOn the role of the unobserved confounders and also how the data generation process (definition) aligns with our actual experiments, we will discuss in detail in our replies to **Q1** and **Q2**. Briefly, those unobserved confounders represent information that cannot be observed by the student agent but has been used during off-policy data collection by the behavioral policy. In our experiment, those masked out pixel features together compose a high-dimensional hidden confounder.\n\nRegarding the experiment evaluation, we conducted hyper parameter tuning on a subset of the training parameters and followed the best practice for others. We also tried different DQN variants. For each experiment, we conducted five random trials and reported the normalized mean/IQM/median. We will further explain our evaluation design in reply to **Q8**.\n\nFor elaborations on Proposition 3.1 and related equations, we will provide more details in reply to **Q3** and **Q5**. To summarize, those underscored notations represent their counterparts under the worst case environment that could happen given behavioral trajectories. Proposition 3.1 provides a Bellman equation-like update rule to calculate the pessimistic Q-values with guaranteed convergence.\n\n> **Q1** “Please confirm if the scope of the paper is correct. … defining CMDP seems like it will address the scenario beyond what's described above.”\n\nWe consider a similar off-policy learning setting as the original DQN paper, except that the sample trajectories are not generated by the student DQN being trained, but from a more complex teacher model with better sensor capabilities. Those state variables that cannot be observed by the student agent become unobserved confounders $U_t$ in the CMDP, affecting the teacher model’s action selection, next state transitions, and the rewards. For example, in the Pong game, we masked out the opponent’s paddle location, which will surely affect those factors mentioned above. Thus, we find that defining CMDP is necessary and adequate for our claimed scope of the paper. \n\n> **Q2** “In Line 77, when the data generation process is explained, it appears that only one unobserved confounder variable ($U_t$) is considered. Do you also cover the factored state and action space MDP?”\n\nYes, our framework also applies to factored state/action space MDPs. The confounding variable U can be high-dimensional, supporting the situation where multiple variables in the environment are unobserved by the student agent and thus become unobserved confounders.\n\n> **Q3** “In the appendix, is there a proof of Eq. (3) on the Bellman equation for the given setting, or is it from some earlier works?”\n\nWhen the experimental data is available and the learner can collect data by directly intervening in the environment, the CMDP is reduced to a standard MDP, and the Bellman equation of Eq. (3) follows from the standard RL literature. A detailed survey can be found in [6].\n\n> **Q4** “Figure 2 and Example 1 are too extreme. ... I don't think this is a realistic example of an unobserved confounder, although it can be used to contrast the performance as shown in Figure 2(d). In this case, how does employing a behavior policy perform in the modified state space?”\n\nFirst, we respectfully disagree that the given situation is not realistic, as there are many examples in practice where the agent’s sensors have been damaged, and it cannot perceive information far from its position (e.g., the opponent's location). In this case, one could have two choices: (1) stop the agent entirely, or (2) realign the agent’s current policy to a more conservative, yet still effective one. In the Pong game setting, while the agent could only track the ball location, there exists a winning strategy in the confounded Pong game: one only needs to shoot the ball back every time and wait for the opponent to make a mistake. The strategy itself is even more straightforward than directing the ball smartly to a direction that is further away from the opponent’s current position. When we deploy the behavioral policy in the confounded Pong game, the agent can achieve a score of around 18, which is still less than the optimal score of 21 achieved by a teacher agent who could observe the full state. This means that while the conservative policy of ball tracking is not truly optimal (at least not as in the fully observed case), it can still be effective in practice and is learnable from confounded observations.\n\n> **Q5** “The notations in Proposition 3.1 require further elaboration, particularly those with underscores. Does the lower bound hold whether the value function is identifiable or not?”\n\nYes, the bounds in Prop 3.1 hold for any CMDP, regardless of whether the target is identifiable or not. Specifically, when the value function is not identifiable, our proposed lower bound is tight (unless further assumptions are provided). When the value function is identifiable, our proposed lower bound remains valid because it compensates for the Bellman updates with the worst-case scenario, ensuring that any real transitions under the identifiable case are at least as good as this one. So it’s still a valid lower bound.\n\n> **Q6** “How to sample transitions in an offline environment? Does the step that samples the initial state work? How can the behavior policy sample an action with a modified state space if it were trained in the original state space? What is the added complexity of the additional sampling in Line 196? In line 204, the opponent is unobserved to the offline learning agent. Does it also see the opponent's location?”\n\nThe environment is a standard off-policy environment. The learning agent passively observes the behavioral agent interacting with the unmodified environment, collects the observed trajectories (masked, not including the unobserved confounders), and then saves them into the replay buffer. Those masked trajectories are then used to train our DQN policy. As for Line 196, in implementation, we simply use the observed worst state values during training from the replay buffer as an empirical estimation. So, there is no sampling overhead in practice. In line 204, our causal DQN policy cannot see the opponent’s location because that’s part of the masked area; it only makes a decision based on the observed ball location.\n\n> **Q7** “In line 243, the optimal policy stipulates that it only needs to see the ball location and its paddle, which is the off-line setting that removes the opponent's visual information and the scoreboard. Then, why did the traditional DQN fail?”\n\nTraditional DQN fails to learn since it does not account for the confounding bias. Such bias could make some observed actions appear more effective than they actually are. For example, in the teacher’s trajectories, the behavioral policy only places the ball in the middle when the opponent is located at either corner (top or bottom), and is unable to cover the middle. Therefore, when not observing the opponent’s location, it would appear that placing the ball in the middle is always followed by a positive score, making it a winning strategy. However, if the traditional DQN picks such a policy only based on the observed value function, not accounting for the confounding bias, it would lead to sub-optimal performance, since the opponent could return the ball in the middle in most situations (except when it is at the corners). On the other hand, using the newly proposed causal bound, the agent lowers the weight on the observed behavioral trajectories by its empirical policy distributions. This allows the Causal-DQN agent to learn conservative Q-values without falling into such overly optimistic Q-value traps.  \n\n> **Q8** “Would it be possible to enhance the traditional DQN by utilizing different hyperparameters or exploration strategies? I think the evaluation may be favorable to the proposed method.”\n\nTo the best of our knowledge, it is unlikely due to confounding bias. Specifically, we tuned a subset of training hyperparameters for all DQN variants, like learning rates, batch sizes, and replay buffer sizes (lines 220-226). We keep the comparison fair by using the same set of hyper-parameters for all DQN variants and following the best practices as entailed in the original DQN paper. \n\nWe also explored multiple DQN variants to see if adding more data or adding LSTM layers can help mitigate the learning difficulties. In lines 210-219, we described three different DQN baselines, including: (1) CNN-DQN with trajectories from behavioral policy but observations are masked; (2) LSTM-DQN with the same type of trajectories as (1); and (3) CNN-DQN with masked trajectories collected by itself (the same setting as the standard DQN). However, from Table 1, we see that none of these variants can learn an effective policy in the confounded environments.  \n\nBy and large, we believe the newly proposed, confounded-robust method is significant, since in most practical settings where RL may be used, unobserved confounders and the corresponding trajectories are the rule rather than the exception. There is a natural mismatch between the perceptual capabilities of most agents and the assumptions made in much of the current literature, which either implicitly assumes a perfect match or derives correctness results under that assumption."}, {"review_id": "qOqBBPT8n5", "reviewer": "Reviewer_UUDT", "summary": "This paper proposes Causal DQN, a novel deep reinforcement learning algorithm designed to handle unmeasured confounding in offline, high-dimensional environments. Unlike standard DQN methods that assume the absence of hidden confounders, Causal DQN adopts a partial identification perspective and seeks policies that perform well under the worst-case confounding compatible with the data. The method is empirically validated on twelve confounded Atari games, where it consistently outperforms standard DQN baselines in settings with mismatched behavior and target policies due to hidden biases in visual inputs.", "questions": "Q1: How sensitive is the proposed Causal-DQN to misspecification of the assumed causal graph (DAG), particularly in cases where the unobserved confounder does not simultaneously affect both the action and the reward/transition as modeled in the CMDP framework? Moreover, while the experiments are conducted on Atari games, is there empirical justification or diagnostic evidence that the confounding structures in these games align with the assumed DAG used in the method?\n\nQ2: Is the proposed method limited to environments with discrete action spaces and image-based observations? All experiments are conducted on Atari games with discrete actions and visual inputs.\n\nQ3: The confounding in the experiments is induced via deliberate masking of visual features, which aligns well with the assumed causal graph. However, in real-world scenarios, unobserved confounders may arise from unknown or unrecorded variables rather than masked inputs. Could the authors clarify whether and how the proposed Causal-DQN is expected to mitigate the influence of more general or latent confounding, especially when the nature of confounding is less structured or directly observable?\n\nQ4: If I understand correctly, the causal Bellman lower bound used in this work is based on the theoretical framework proposed by Zhang & Bareinboim (2025), and this paper is the first to integrate it into a practical DQN algorithm with empirical validation in high-dimensional visual environments like Atari. Is this a fair characterization of the novelty and contribution of the work? -> I would say the paper has somewhat weaker motivation and is borderline for NeurIPS.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S1: This paper addresses a novel and important problem in deep reinforcement learning: learning from off-policy data in the presence of unobserved confounders.\nS2: The paper is well-structured.\n\nFor weakness, please see my questions.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your detailed comments and for acknowledging our work’s importance and novelty. We have addressed your questions in the sequel.\n\n> **Q1** “How sensitive is the proposed Causal-DQN to misspecification of the assumed causal graph (DAG), particularly in cases where the unobserved confounder does not simultaneously affect both the action and the reward/transition as modeled in the CMDP framework? Moreover, while the experiments are conducted on Atari games, is there empirical justification or diagnostic evidence that the confounding structures in these games align with the assumed DAG used in the method?”\n\nYes, our method is robust to the possible model misspecification when the unobserved confounder does not simultaneously affect both the action and the reward/transition. Our proposed algorithm is still able to obtain valid bounds over the optimal value function in such settings. Indeed, CMDP is a general class of models that subsumes standard MDP. Therefore, it can handle different confounding situations between those variables. As we don’t restrict how the confounder is used inside transition functions/reward functions, CMDP can handle the situations where those functions are actually not confounded, and the confounder in the function signature serves only as a dummy input. \n\nAlso, the confounders used in our Atari experiments, as entailed in the experiment section, are generated via identifying the part of the screen that the behavioral agent attends to and then masking out a part such that the behavioral agent’s performance would drop. For example, one could verify the significance of masked features (to the behavioral policy) in the saliency map in Figure 2(a). This procedure ensures that the behavioral agent is indeed using those masked areas to make decisions, and they have a meaningful correlation with the state transitions and rewards.\nMore broadly, this is our approach to introducing confounding in an artificial, fully observed setting, acknowledging that confounding is pervasive in real-world scenarios. Our goal is to close the gap between the assumptions made in RL benchmarks -- such as the absence of confounding – and the conditions encountered in practice. \n\n> **Q2** ”Is the proposed method limited to environments with discrete action spaces and image-based observations? All experiments are conducted on Atari games with discrete actions and visual inputs.”\n\nSimilar to the standard DQN, our proposed method handles settings with discrete actions and a continuous state space. Given the popularity of DQN and the fact that it serves as the foundation for a large class of value-based deep RL algorithms, we believe our proposed algorithm covers a wide range of problem cases. That said, to the best of our knowledge, this is the first paper to address such considerations in these settings, and we hope it can initiate a conversation within the community, one that we can build on to expand the general toolbox for more complex decision-making problems.\n\n> **Q3** “The confounding in the experiments is induced via deliberate masking of visual features, which aligns well with the assumed causal graph. However, in real-world scenarios, unobserved confounders may arise from unknown or unrecorded variables rather than masked inputs. Could the authors clarify whether and how the proposed Causal-DQN is expected to mitigate the influence of more general or latent confounding, especially when the nature of confounding is less structured or directly observable?”\n\nYes, our proposed method is able to handle the general form of unobserved confounding since our CMDP model encodes the least amount of assumptions for representing the presence of unobserved confounding in the MDP environment. Specifically, we model unobserved confounding as an exogenous variable $U$ that could be further decomposed into multiple smaller variables and has a rich structure. However, we do not intend to impose any additional structural knowledge in CMDPs, which allows our proposed Causal-DQN to be applicable to any MDP environment where the confounding bias could not be excluded a priori. Also, we want to clarify that masking is a straightforward way to simulate unrecorded variables. For example, in the pong game, we completely mask out the opponent’s location $U_t$, making such information totally unobserved to the learner.\n\n\n> **Q4** “If I understand correctly, the causal Bellman lower bound used in this work is based on the theoretical framework proposed by Zhang & Bareinboim (2025), and this paper is the first to integrate it into a practical DQN algorithm with empirical validation in high-dimensional visual environments like Atari. Is this a fair characterization of the novelty and contribution of the work? -> I would say the paper has somewhat weaker motivation and is borderline for NeurIPS.”\n\nTheoretically, we propose the first confounding-robust off-policy learning algorithm that bounds the target Q-value using function approximation. The derivation of the learning updates and how we combine them into a sample-based learning method like DQN is non-trivial. Also, Zhang & Bareinboim (2025) only consider the policy evaluation problem, and do not provide policy recommendations.  In this work, we study the more challenging policy learning setting, and Causal-DQN is able to obtain an effective conservative policy from confounded observations. \n\nEmpirically, to the best of our knowledge, we are the first to perform confounding robust learning methods for high-dimensional environments (like Atari games) with function approximations, while the previous paper like (Zhang & Bareinboim 2025) only handles small discrete state spaces like very small grid worlds."}, {"review_id": "6DCiUFXXfj", "reviewer": "Reviewer_eDJW", "summary": "Summary: this paper studies deep RL motivated by RL settings with unobserved confounders. In particular, it focuses on scaling up to DQN the (s,a)-rectangular robust bounds of Zhang and Bareinboim. They convert from the probabilistic form of the bounds to an expectation which can be sampled for DQN updates. The paper develops masked versions of 12 Atari games and trains DQN methods with these lower bounds.", "questions": "Questions/suggestions: \n\nI think something that would help the experimental design significantly is 1) compare to methods that explicitly regularize for coverage from the behavior policy, like CQL, or more importantly 2) consider an ablation where the robust bounds from Zhang and Bareinboim are replaced with a uniform lower bound on the reward (i.e, coming from the environment). \n\nI think the more important ablation to run is 2) as it helps isolate whether the PI bounds are being informative in some regions, or whether the method is improving on vanilla DQN because of variance regularization and penalizing policies that stray from the behavior policy; it should also be very runnable given the existing codebase.", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n- There's been a lot of recent work lately on RL with unobserved confounders but few works establishing how these methods do in empirical deep RL environments with modern empirical deep RL methods. They find that this can improve upon naive DQN in terms of total reward. \nWeaknesses: \n\nsignificance: \n\n- The paper pitches itself as being close to RL methods for unobserved confounders, but the method itself is much closer to RL methods that constrain to be close to the behavior policy (like CQL). There is not much mention of these methods that should be a more appropriate comparison. \n\nClaims matching up with experimental design: The paper's main novelty is in extending optimizing RL with lower bounds to the realm of deep RL. However, the experimental design can be improved, which would be important as the new method is a fairly direct extension of prior work on partial identification bounds based on the structure of probability distributions alone, without any additional information. \n\n\n- Are the unobserved confounders chosen here really UCs that affect both state and next action? It's quite confusing from the experimental design whether this is a story of removing extraneous information; or whether the robustness to unobserved confounders is helpful. The scenarios presented in the main text are helpful, but somewhat read as \"anecdata\". The explanations in the appendix are somewhat confusing, most of them say \"The original game screen shows score and remaining life which shouldn’t be the major factor affecting the policies.\" It sounds like irrelevant information is being removed from the environment, and the criterion for the unobserved confounder selection actually seems to choose parts of the environment that are \"action confounders\" but not full confounders. \n\n\"The core design idea is that we would like to occlude the part of the screens that contains information useful for making decisions but is not a significant factor from human players’ perspectives.\" - This selection criterion for unobserved confounders seems like it would also omit bona-fide unobserved confounders that affect both action selection by good policies as well as the transition dynamics. For things like score and # lives, conditional on the observed state and action history, they have no further effect on the transition dynamics. \n\nIt's OK if this is more of a partial observability or irrelevant information story, but I believe that should be precisely presented (rather than emphasizing unobserved confounders).", "was_revised": true, "final_justification": "The ablations that the authors introduced were helpful. I thought the most useful thing was the set of confounded environments. The method flows naturally from prior work, and so the paper would be improved by better connecting to other offline RL frameworks.", "rebuttal": "We thank you for your thoughtful feedback. We appreciate your acknowledgment of the significance of our work and address your concerns in the sequel. \n\n> **W1** “The paper pitches itself as being close to RL methods for unobserved confounders, but the method itself is much closer to RL methods that constrain to be close to the behavior policy (like CQL). There is not much mention of these methods that should be a more appropriate comparison.”\n\nOur method conceptually differs from CQL in that we use pessimism to counter the uncertainty due to unobserved confounders in the offline datasets, while CQL uses pessimism to counter the lack of coverage on the action space, i.e., overly optimistic Q values for actions not visited in the datasets. This means that even when the dataset has full coverage over the action space for each visited state, there could still be unobserved confounders resulting in biased Q-value estimations, which our proposed method can handle.\n\nFor concreteness, we provide below a confounded MDP example that differentiates our proposed method from CQL.\n\n**Example**: For simplicity, we consider a single-step CMDP with all variables binary. At each episode, the state $S$ is sampled from a Bernoulli distribution, $P(S=1)=0.5$. The behavioral policy’s action $X$ is decided by $X = S\\oplus U_X$, if $U=0$ and $X = \\neg S\\oplus U_X$, if $U=1$. And the reward $Y$ is decided by $Y = S\\oplus X$, if $U=0$ and $Y = \\neg S\\oplus X$, if $U=1$. $U$ and $U_X$ are unobservable exogenous variables and $U$ is a confounding variable where $P(U=1)=2/3$ and $P(U_X = 1) = q, q \\in (0,1)$.\n\nTo simplify the discussion, we will focus on the case where state $S=0$ (the following argument can also be easily adapted to the other state). We can calculate the behavioral policy distribution and verify that the behavioral policy has a full coverage over the action space, $\\pi_\\beta(X=0|S=0) = \\frac{1}{3}(1-q) + \\frac{2}{3}q > 0$ and $\\pi_\\beta(X=1|S=0) = \\frac{1}{3}q + \\frac{2}{3}(1-q) > 0$. For $S=0$, the optimal interventional Q-values are $Q^\\*(0, 0) = P(Y=1|S=0, do(X=0)) = \\frac{2}{3}$ and $Q^\\*(0, 1) = P(Y=1|S=0, do(X=1)) = \\frac{1}{3}$. The optimal policy is $do(X=0)$ when $S=0$.\n\nThen, we calculate the CQL Q-values. Given the objective function in equation (4) in the [CQL paper](https://arxiv.org/pdf/2006.04779), we can use Pytorch to calculate the CQL Q-values. When we set $\\alpha=1.0$ and $q=1/4$, the learned CQL Q-values at convergence are $Q(0,0) = 0.23 < Q(0,1)=0.27$ suggesting a suboptimal policy. In fact, no matter how we tune the alpha parameter in the CQL objective, there always exists some $U_X$ distributions that can yield a wrong policy under CQL training. Thus, we have demonstrated how CQL can fail under confounding biases. \n\nInstead, we calculate the causal Bellman lower bound to the optimal Q-values. We have $\\underline{Q_*}(0, 0) = P(X=0|S=0)P(Y=1|S=0,X=0)=\\frac{2}{3}q$ and $\\underline{Q_*}(0, 1) = P(X=1|S=0)P(Y=1|S=0,X=1)=\\frac{1}{3}q$. Clearly, our causal bound correctly captures the optimal interventional policy $do(X=0)$ when $S=0$.\n\nMore broadly, since CQL and our method address different aspects of data bias for off-policy learning, combining those two approaches will yield a more robust offline learning algorithm that can handle both the action space coverage and the unobserved confounding issue.\n\n> **W2** “Are the unobserved confounders chosen here really UCs that affect both state and next action? It's quite confusing from the experimental design whether this is a story of removing extraneous information; or whether the robustness to unobserved confounders is helpful. …It sounds like irrelevant information is being removed from the environment, and the criterion for the unobserved confounder selection actually seems to choose parts of the environment that are \"action confounders\" but not full confounders. … For things like score and # lives, conditional on the observed state and action history, they have no further effect on the transition dynamics. It's OK if this is more of a partial observability or irrelevant information story, but I believe that should be precisely presented (rather than emphasizing unobserved confounders).”\n\nWe would like to reiterate that our unobserved confounders are all chosen from the observation area where the saliency map of the behavioral policy has high values, indicating that the behavioral policy’s action choice is affected by that information. We identified many non-trivial confounders via this approach. For example, in Boxing, we mask out half of the arena and force the agent to stay on the left; in Breakout, we mask out the whole stack of bricks so the agent only learns to shoot the ball back without knowing how many bricks are left; in Choppercommand, we remove the minimap that can reveal incoming enemies out of the main screen; in Gopher, we mask out the gopher location so the agent has to focus on the hole location; in Pong, we mask out opponent’s location; and in Roadrunner, we mask out sky/desert that can indicate levels/incoming obstacles. \n\nRegarding confounders like score and remaining lives, for most of the Atari games we selected (e.g., Kungfu Master, Asterix, Gopher, Pacman), the game difficulty level changes dynamically w.r.t the player’s current score (or remaining time/lives). And conditioning on the limited stacked observations we provide to the agent cannot fully determine the exact level in general. Thus, the observed state transitions are indeed affected by those unobserved confounders. From a human player’s perspective, knowing that having a high score and that the game level is different shouldn’t alter our winning strategy a lot, as the basic game mechanism is still the same, except that it usually requires a faster reaction or more precise execution of the actions. Thus, for DQN agents that can react precisely to the frames (w/o considering sticky actions), we don’t regard this extra information as necessary to learn a good policy. But it doesn’t mean that knowing scores and remaining lives is not helpful. Moreover, given that there could be spurious correlations between remaining time/score/lives and the game mechanism that can be exploited by the DQN, these are genuine confounders. \n\nTo summarize, the confounders we chose may either reveal information on the reward/transition functions or directly affect the future game process. Empirically, if those confounders are indeed irrelevant information, we shouldn’t have observed such a drastic performance drop on those standard baseline DQNs with recommended/tuned hyper-parameters.\n\nHaving said that, our experiments introduce confounding bias in an artificial, fully observed setting, noting that confounding is pervasive in real-world scenarios. Our goal is not to solve harder Atari variants, but to study and close the gap between the assumptions made in RL benchmarks (such as the absence of confounding) and the conditions encountered in practice.  We appreciate the opportunity to clarify this issue and will also revise the wording in the appendix to reduce confusion.\n\n> **Q1**  “I think something that would help the experimental design significantly is 1) compare to methods that explicitly regularize for coverage from the behavior policy, like CQL, or more importantly 2) consider an ablation where the robust bounds from Zhang and Bareinboim are replaced with a uniform lower bound on the reward (i.e, coming from the environment).”\n\nFor Question (1), we have demonstrated in our previous reply to W1 with a detailed example where CQL cannot solve the unobserved confounding case, while the newly proposed Causal Bellman Equation can. More generally, CQL focuses on the challenges of the lack of full action space coverage, called “no overlap” in the causal inference literature, while our work focuses on the challenges due to unobserved confounding. These challenges might seem related, but they model different aspects of data bias in off-policy RL. As we demonstrated in our previous response, confounding bias exists even when the action space has full coverage, and all realizations of actions have been observed.\n\nFor Question (2), we train the model with a uniform lower bound while keeping all other hyperparameters the same as the original Causal-DQN. Across all 12 Atari games we tested, the ablation variants all fall far behind Causal-DQN, indicating that an adaptive lower bound is indeed helpful for learning from confounded data.\n\nenv/algo|Causal-DQN|UniformBound-DQN\n-|-|-\nAmidar          | 282.6             | 32.8\nAsterix         | 2587.0            | 520.0\nBoxing          | 71.0              | 16.0\nBreakout        | 131.2             | 0.1\nChopperCommand  | 1658.0            | 1060.0\nGopher          | 7327.2            | 664.0\nKungFuMaster    | 44196.0           | 14725.0\nMsPacman        | 1747.6            | 486.0\nPong            | 21.0              | 0.0\nQbert           | 4458.0            | 262.0\nRoadRunner      | 27414.0           | 3380.0\nSeaquest        | 980.0             | 108.0"}, {"review_id": "eHqVIdi1oQ", "reviewer": "Reviewer_E4zm", "summary": "This paper introduces Causal-DQN, a novel reinforcement learning algorithm designed to be robust against unobserved confounding in offline, high-dimensional environments. This approach specifically tackles the scenarios when the observed action and reward are confounded. The mothod first incorporates  a Causal Bellman optimality equation that provides a lower bound on the value function, and then designs an algorithm that minimizes this lower bound and demonstrates its effectiveness across 12 carefully Atari environments.", "questions": "1. Did the author discover any chance that the policy behaves too conservative? Given that a lower bound is learned. In particular,  I think it would be helpful to discuss under what scenarios that pessimism is likely to happen or not. \n2. Regarding the assumptions:\n   * The authors first assume the action domain to be discrete while the state domain to be continuous. Could the method be extend to continuous action space? \n   * The authors assume that there's unobserved confounder $U_t$ that affects the action, state and reward simultaneously. Could the authors provide motivation for dealing with this type of confounding? For example, I think in the Pong setting, the confounder (the opponent's position) seems to be a cause for the action and outcome, but the next state should be generated from a transition function that follows physical laws regardless of $U_t$. Why considering the transition function as $S\\times X \\times U \\rightarrow S$ instead of  $S\\times X  \\rightarrow S$. It is also slightly confusing to use three separate dashed double-arrows to represent the same one $U_t$.", "limitations": "Yes, the authors has discussed the limitations about this work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality**\n\nStrengths:\n\n- The theoretical gurantees are sound.\n- The algorithm is well-formulated with careful attention to both theory and implementation.\n- Experiments are comprehensive with 12 Atari games.\n\n  \n\n **Clarity**\n\nStrengths:\n\n- The paper is well written and the logic is easy to follow.\n- The Proposition and Algorithm are clearly presented.\n\nWeaknesses:\n\n- The explanation of some derivations (especially in Proposition 3.1) assumes familiarity with causal RL; a brief intuition would help.\n\n **Significance**\n\nStrengths:\n\n- Tackles an under-explored but important problem in RL: confounding in offline learning.\n- Demonstrates improvements in challenging settings where standard DQNs fail.\n\nWeaknesses:\n\n- The scenarios are limited to Atari games; results on more diverse domains would be interesting to be discussed.\n\n**Originality**\n\nStrengths:\n\n- Extending the Bellman equation to the partially identifiable, confounded setting is novel and impactful.\n\nWeaknesses:\n\n- The framework largely builds on prior causal RL work by Zhang & Bareinboim; the originality lies more in algorithmic application.", "was_revised": true, "final_justification": "I recognize the novelty and contribution of this work for making a first step towards solving confounding RL problems. I think the author's rebuttal further clarifys their method. I'm not completely sure about I understand all the details but I'm inclining to accept.", "rebuttal": "We thank the reviewer for the feedback and appreciate your recognition of the significance of our work. We have addressed each of your concerns in the responses below.\n\n> **W1** “The explanation of some derivations (especially in Proposition 3.1) assumes familiarity with causal RL; a brief intuition would help.”\n\nIntuitively, because of the existence of confounders, we are unsure about whether the worst case can indeed happen in reality or not given solely the observational data. Thus, we compensate the standard Bellman update with the worst case transitions resulting in a pessimistic lower bound to the optimal Q-values. For more details on causal RL, we kindly refer you to [6].\n\n> **W2** “The scenarios are limited to Atari games; results on more diverse domains would be interesting to be discussed.”\n\nWe follow a similar evaluation framework to the original DQN, which has been widely applied in many real-world applications. For other domains like those continuous action environments, it is beyond the scope of this work, which we plan to pursue next.\n\n> **W3** “The framework largely builds on prior causal RL work by Zhang & Bareinboim; the originality lies more in algorithmic application.”\n\nDespite algorithmic applications, we would like to point out that on the theoretical side, we propose the first confounding-robust off-policy learning algorithm that bounds the target Q-value using function approximation. The derivation of the learning updates and how we combine them into a sample-based learning method like DQN is non-trivial. Also, Zhang & Bareinboim (2025) only consider the policy evaluation problem, and do not provide policy recommendations.  In this work, we study the more challenging policy learning setting, and Causal-DQN is able to obtain an effective conservative policy from confounded observations. \n\nEmpirically, to the best of our knowledge, we are the first to perform confounding robust learning methods for high-dimensional environments (like Atari games) with function approximations, while the previous paper like (Zhang & Bareinboim 2025) only handles small discrete state spaces like very small grid worlds.\n\n> **Q1** “Did the author discover any chance that the policy behaves too conservative? Given that a lower bound is learned. In particular, I think it would be helpful to discuss under what scenarios that pessimism is likely to happen or not. ”\n\nThere exist cases where there is a considerable performance gap between our Causal-DQN policy and the demonstrator policy using an LSTM layer. This means that our causal bound could be loose. Still, our conservative policy is able to outperform other standard algorithms that directly use the confounded observations without accounting for the potential confounding bias. It’s possible that we can further improve the bound and be more optimistic with additional domain knowledge or assumptions. But under the general case of CMDP definitions, our Causal Bellman Equation provides a tight lower bound on the optimal Q-values, and thus the policy learned is robust against the worst-case scenarios. \n\nEmpirically, we actually observe that sometimes a conservative policy still works well, like in Pong and RoadRunner. There are instances where our method is inferior to the behavioral policy due to conservatism. But from Table 1, we see that on average, the newly proposed Causal-DQN’s normalized mean/median/IQM scores are even slightly better than the original behavioral policy with full observations.  \n\n> **Q2-1** “The authors first assume the action domain to be discrete while the state domain to be continuous. Could the method be extended to continuous action space?” \n\nYes, it is possible. This work is a first step towards handling confounding robust learning in high-dimensional state space. Learning with the continuous action space is an important next step for generalized confounding robust deep RL; thank you for the note.\n\n> **Q2-2** “The authors assume that there's an unobserved confounder $U_t$ that affects the action, state and reward simultaneously. Could the authors provide motivation for dealing with this type of confounding? For example, I think in the Pong setting, the confounder (the opponent's position) seems to be a cause for the action and outcome, but the next state should be generated from a transition function that follows physical laws regardless of $U_t$. Why consider the transition function as $S\\times X \\times U \\mapsto S$ instead of $S\\times X \\mapsto S$. It is also slightly confusing to use three separate dashed double-arrows to represent the same one .”\n\nWe use U to represent things that are not recorded in the dataset. For example, when the behavioral policy is a higher-end robot with more sensory information, we want to use such trajectories to train a commercial robot with fewer sensors. CMDP can then be used to model such scenarios. In the Pong game, the next state also involves the ball locations, and clearly, the opponent’s paddle location will affect the ball locations when it hits/misses the ball. Thus, in this case, there is a causal relationship between the latent opponent’s location ($U_t$) and the next ball placement ($S_t$), and the transition function is in the form of $S\\times X \\times U \\mapsto S$. \n\nRegarding the usage of three separate dashed double-arrows, we follow the standard notations in Causality [[Pearl (2009)]](https://bayes.cs.ucla.edu/BOOK-2K/) to denote confounders in the causal diagram. That is, we draw a dashed bi-directed edge to represent that two variables share some unobserved confounders as input to their structural equations. \n\nIn general, our experiments introduce confounding bias in an artificial, fully observed setting, noting that confounding is pervasive in real-world scenarios. Our goal is to close the gap between the assumptions made in RL benchmarks -- such as the absence of confounding – and the conditions encountered in practice."}]

Author rebuttal:

Meta-review: Summary of the paper:

The paper proposes a DQN algorithm for an off-line RL environment in the presence of an unobserved confounder. The experiment and explanations in the paper primarily address the source of confounding as missing state information during the data collection stage by the behavioral policy. The contribution is to extend the optimal value function with a lower bound and propose an off-line learning algorithm. In practice, it can be very challenging to engineer an RL environment with unobserved confounders. This paper shows 12 modified Atari domains based on salience map analysis.

Summary of the discussion:

All reviewers agree on acceptance, but some have still some concerns.

One reviewer had some minor comments:

(1) CARL with UC can address more general scenarios, and either authors could sketch some more in the paper (intro/background/future work) or clearly state the boundary of the current work

(2) Some derivations are said to be trivial, but it is always good to add the details. For example, the recursive form of the Bellman equation can be derived from the basic definition of the value. Factored state extension is not as trivial as the authors replied.

(3) The experiment is not of high quality. Authors mentioned that they saw some papers reporting three trials or five trials, but they are some exceptions. In addition, the experiment is not fully persuasive to draw a conclusion. It might be possible that some better hyperparameters would lead to better results than the complete failures that we see for DQNs.

One reviewer still has some concerns remain regarding the realism of the confounding setup and broader applicability.

Recommendation:

All reviewers vote for acceptance. I, therefore, recommend acceptance and encourage the authors to use the feedback provided to improve the paper for its final version, especially, to address the remaining concerns of the reviewers.
Decision: Accept (poster)