Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
Authors: Shuguang Yu
Shuxing Fang
Ruixin Peng
Zhengling Qi
Fan Zhou
Chengchun Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: off-policy evaluation
unmeasured confounding
two-way deconfounder
neural tensor network
Primary area: causal_inference

Abstract: This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which a model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.

Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
Shuguang Yu"
Shuxing Fang"
School of Statistics and Management Shanghai University Finance and Economics Shanghai, China
Department of Applied Mathematics The Hong Kong Polytechnic University Hong Kong, China
Ruixin Peng
Zhengling Qi Department Decision Sciences George Washington University Washington D.C., USA
School Statistics and Management Shanghai University of Finance and Economics Shanghai, China
Fan Zhou
School of Statistics and Management
Shanghai University of Finance and Economics Shanghai, China
Chengehun Shi
Department of Statistics
London School of Economics and Political Science London, UK
Abstract
This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop two-way deconfounder algorithm that devises neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.
Introduction
Before deploying any newly developed policy, it is important to assess its impact. In many high- stakes domains, it is risky or unethical to implement such policies directly for online evaluation. This challenge highlights the essential role of off off-policy evaluation (OPE).
There a vast body of literature on OPE. Most studies assume there are no unmeasured confounders
(NUC), also known unconfoundedness (see e.g., Thomas al. 2015 5 Jiang and Li 2016 Thomas
and Brunskill 2016 Farajtabar et al. 2018 Liu et al. 2018 Irpan et al. 2019 Schlegel et al, 20099 Tang
al. 2019|0X01 et al. 2019
Dai et al. 2020 Chandak al. 202 Hao et
Liao et al. 202 lb
"Equal contribution.
1 "Corresponiing authors: zhoufan@mail www.eaanmaillsuuff....am edu. en, c. shii70le..a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Shi et al. 2021 Chen and Qi 2022 Kallus and Uehara 2022 Liao et al. 2022 Shi et al. 2022b Xie
al. 2023 Zhou et (023a). However, the NUC assumption is restrictive and untestable from the data. It can be violated in various domains such as urgent care [Namkoong et al. 2020). autonomous
driving [Nyholm and Smids] 2020], ride-sharing (Shi et al., [022c]. and bidding (Xu et al., 2023]. Applying standard OPE methods that rely on the NUC assumption in these settings would result in a biased policy value estimator [see e.g., Bennett and Kallus [023 Section 6.
Causal reinforcement learning studies offline policy optimization or OPE in the presence of unmea- sured confounding. Many existing works can be divided into one of the following three groups:
1. Methods under memoryless unmeasured confounding: The first type of methods relies on "memoryless unmeasured confounding" assumption to guarantee that the observed data satisfies
the Markov property in the presence of unmeasured confounders (Zhang and Bareinboim] 2016
Kallus and Zhou, 2020 Li et al. 2021 Liao et al. 202 Wang et al., 202 [ Chen et al. 2022 Fu
et 2022 Shi et al. [022c Yu et al. 2022 Bruns- Smuuh and Zhou| [023 Xu et al. 2023] Many
of these works also require external proxy variables (c.g., mediators and instrumental variables) to
handle latent confounders. In contrast, the method proposed in this article neither relies on the
Markov assumption nor requires external proxies.
POMDP-type methods: The second category employs a partially observable Markov decision process (POMDP) to model unmeasured confounders as latent states, drawing on ideas from the proximal causal inference literature (Tchetgen et al. 2020 to address unmeasured confounding
[Tenenhhllz et al. 2020 Nair and Jiang, 2021 Miao et al. [0022] Shi et al.) 2022a Wang et al.
2022 Bennett and Kallus 2023 HHon et al. 2023 Lu et al. 2023]. However, these works require
restrict mathematical assumptions that are hard to verity in practice [u al. 2018
3. Deconfounding-type methods:
The final category originates from deconfounding methods in causal inference, which leverage the inherent structure within observed data, such as multiple treat- ment dependencies, network structures, and exposure models, to address unmeasured confounding
[Louizos et al. 2017 Tran and Blei 2017 Wang et al. 2018 Veitch et al.
2019
Wang and Blei
Zhang et al. 2019 Bica et al. 2020 Veitch et al. 2020 Shaa et al.
2022 McFowland III
and Shalizii 2023 Shuai et al.] 2023]. Notably, Wang and Bleil [2019] directly estimate latent
confounders for causal inference. However, their algorithm requires the unmeasured confounders
to be a deterministic function of the actions [Ogburn et al. 2020, Wang and Blei 2019]. which has been criticized as being unreasonable (D Amour 2019 Ogburn et al. 2019]: refer to Appendix A.I There have also been several extensions of deconfounding methods to reinforcement learning (RL) [Lu et al. [2018] Hatt and Feuerriegel 2021 Kausik et al.) 2022 2023]. However, these methods either impose a one-way unmeasured confounding assumption, which can be overly restrictive (see Section 2 or require the correct specification of the latent variable (Rissanen and
Marttinen 2021
This paper aims to develop advanced deconfounding-type OPE methodologies, allowing for more flexible assumptions regarding latent variable modeling. Our proposal is inspired by the two-way fixed effects (2FE) model which is widely employed in applied economics MMndlak [1961), Ballagi
and Baltagi [2008] Griliches 1979), Anderson and Hsiao [182), Freyberger [2018] Callaway and
Karamil 2023 and causal inference with panel data Arkhangelsky and Imbens 2022] ,, De Chaise
martin and d H'Hullllelille 2220), Imai and Kim [221], Sant' Anna and Zhao 2020 Athey and
Imbens| 2022]. More recently, Dwivedi et al. [022 applied the 2FE model to counterfactual predic-
tion in contextual bandit setting and Bian et al. [2023] extended the 2FE model to RL. However, their investigations primarily consider an unconfounded setting. Additionally, the model proposed by Bian et al. (2023) imposes restrictive additive assumption requiring the latent factors to influence both the reward and transition functions in a purely additive manner. Furthermore, they rely on linear function approximation to estimate the policy value, which may fail to capture the inherently complex nonlinear dynamics. In contrast, we employ flexible neural networks to model the environment.
In this article, we propose novel wwo-way unmeasured confounding assumption to effectively model latent confounders. This approach categorizes all unmeasured confounders into two distinct groups: those that are time-specific and those that are model's flexibility beyond one-way unmeasured confounding while ensuring that the total number of confounders remains much smaller than the sample size, making them estimable from the observed data. We further develop an original wwo-way deconfounder algorithm that constructs neural tensor network to jointly learn the unmeasured confounders and the system dynamics. Based on the learned
This assumption enhances the

---

model, we construct a model- based estimator to accurately estimate the policy value. Our proposed model for unmeasured confounders shares similarities with latent factor models used capture two-way interactions, such as item-customer interactions in recommendation systems (Hu et al. 2008 He et al. 2017], and relationships between different entities in multi-relational learning [Socher et al.
2013 Nickel et al. 2015 Nguyen et al. 2017 Wang et al. 2017 Ji et al. 2021].
To summarize, our contributions include: (1) the introduction of novel two-way unmeasured confounding assumption; (2) the development of a new two-way deconfounder algorithm for model- based OPE under unmeasured confounding, (3) the demonstration of the effectiveness of our model and algorithm.
Two-way Unmeasured Confounding
In this section, we begin by presenting the data generating process under unmeasured confounding and outlining our objectives. We then introduce the proposed two-way unmeasured confounding assumption and compare it against alternative assumptions.
We consider an offline setting with pre-collected observational data D, containing N trajectories, each consisting of T time points. We use i to index the i-th trajectory and t to index the f-th time point. For each pair of indices (i, t), its associated data is given by the observation-actioo--eww triplet (Oi,t, Ai,,, R.,e). In healthcare applications, each trajectory represents an individual patient where Oi, denotes the covariates of the i-th patient at time I, As, denotes the treatment assigned to the patient, and R,, measures their clinical outcome at time
We investigate a confounded setting characterized by the presence of unmeasured confounders (denoted by Z, that influence both Ai, and (Ri,,, Oi,++1) ) for each pair t). The offline data generating process (DGP) can be described as follows: (1) At each time t, the observation Oz recorded for the i-th trajectory. (2) Subsequently, an action Ai, is assigned for the i-th subject according to a behavior policy Th. such that A,, ~ To ( Z.,)). (3) Next, we obtain the immediate reward Ri. and the next observation O 0+ such that (Re,; Oi,441) P( Ai,r; Oi,t, Zi,z) for some transition function P. (4) Steps 2 and are repeated until we reach the termination time T. See Figure Ta) for illustration.
In contrast, following a given target policy π we wish to evaluate, the data is generated as follows: (1) At each time t, the action Ar,: is determined by the target policy T( O.,)), independent of the unmeasured confounder Zr (2) The immediate reward Ri and next observation 0 are generated according to the transition function Ar,,, O Z,,)). In this setup, the unmeasured confounders affect only the reward and next observation distributions, but not the action. This is primary difference from the offline data generating process. Specifically, whatever relationship exists between the unmeasured confounders and the actions in the offline data, that relationship is no longer in effect when we perform the target policy ा. Our objective lies in evaluating the expected cumulative reward under т, given by
E (Ra,).
NT
where EE (Rt,r) represents the expectation under the target policy n, irrespective of the unmeasured confounders.
To better understand the proposed way unmeasured confounding assumption, we first introduce two alternative assumptions concerning the unmeasured confounders as follows:
(a) Unconstrained unmeasured confounding (UUC): Each pair of indices (i. t) corresponds to an unmeasured confounder Zr,,, and there are no restrictions on these values.
(b) One-way unmeasured confounding (OWUC): For any i, the unmeasured confounders remain the same across time, i.e., H; 2 ZZz
Z
These two assumptions represent two extremes. The UUC assumption offers maximal flexibility without imposing any specific conditions. In contrast, the OWUC assumption restrictive, essentially excluding time-varying confounders'.
3

---

Zo
Ru
Aur1
ALt
Bumi
Auni
(a) UUC
(b) OWUC
(c) TWUC
Figure The directed acyclic graphs data generating processes under different assumptions. (a) :2t, hi. (colored in blue) unconstrained unmeasured confounders. (b) {h₂ J (colored in green) are one-way unmeasured confounders. (c) { ) and {we) (colored in orange) are two-way unmeasured confounders.
The validity of the deconfounder algorithm [Wang and Blei 2019
relies crucially on the consisten estimation of the latent confounders. This is because employs a plug-in method for constructing the average treatment effect estimator, which plugs in the estimated latent confounders into the model.
Unconstrained unmeasured confounding imposes restrictions the latent variables, but requires to estimate a total of NT latent variables, which is equal to the sample size. This make consistent estimation infeasible without resorting the 'deterministic unmeasured confounding' assumption discussed Section T On the other hand, one-way unmeasured confounding only requires estimating N latent variables, but in reality, this assumption is often difficult meet.
In this article, we propose the following two-way unmeasured confounding, which offers a middle ground between the UUC assumption and the OWUC assumption:
(c) Two-way unmeasured confounding (TWUC): There exist time-invariant confounders {U+}a and trajectory-invariant confounders (We) such that Zi, (U, WT)T.
As discussed in the introduction, TWUC requires that all unmeasured confounders belong to one of two groups: sejjectory-specffie time- inverrivrrnnt confounders and time-specific trajectory-invarian confounders. Notably, it excludes confounders that are both trajectory- and time-specific. The U, can be interpreted as individual baseline information (e.g., salary or educational background) that the W,s represent external factors (e.g., weather or holidays) exerting a common influence across all trajectories. This assumption effectively relaxes one-way unmeasured confounding by accommodating time-varying confounders. Meanwhile, the number of latent confounders is confined to N + T, much smaller than the sample size N when both N and T grow to infinity. This ensures the feasibility of consistent estimation. See Figure for graphical visualization of the three assumptions.
remains consistent over time, while
To further elaborate the three modeling assumptions (a) (c). we consider a linear model setup where the conditional means of the next observation and the immediate reward are linear functions of the current observation-actio pair as well as the unmeasured confounders, and summarize the implications of adopting the three assumptions the following corollaries.
Proposition 1 (Inconsistency of the unconstrained model). The least square estimator (LSE) based on the UUC assumption cannot yield consistent predictions. Its mean square error (MSE) remains constant as both N and T increase.
Proposition 2 (Inconsistency of the one-way model under misspecification). When time-varying measured confounders exist, the LSE based on the one-way model cannot yield consistent predictions. Its MSE remains constant as both N and T increase.
Proposition (Consistency of the wwo-way model). The LSE based on the two-way model can yield consistent predictions. Its MSE decays to zero as both N and T increase.
Furthermore, we design linear simulation setting numerically compare the estimators based on these three assumptions and report their MSES in Figure b). The results reveal that the unconstrained
Here, consistency means that the estimators converge their ground truth the sample size increases [see
e.g.. Casella and Berger 2021

---

model tends to overfit the data, as evidenced by its lowest prediction error on the training dataset and higher error in off-policy value estimation. Conversely, the one-way model underfits the data. It achieves the largest errors in both training data prediction and off- value estimation. In contrast, our proposed two-way model strikes a balance, resulting in the lowest error off-policy value estimation.
To conclude this section, we summarize the DGP under the proposed two-way unmeasured confound- ing assumption:
At the initial time, each trajectory independent generates an observed O₂,1 and an unobserved U.- Additionally, latent W, is independently generated.
Next, at each time t, A.. and (Ri,e: 12,4+1) are generated, according to T (or a target policy π) and P. respectively. Moreover, a latent W W4+1 is generated, whose distribution depends only on the past time-specific confounders.
Under this DGP, the two-way unmeasured confounders are policy-agnostic, i.e., unaffected by the actions or policies. Consequently, we can employ a plug-in approach to construct the policy value estimator (see eliminating the need to estimate their distributions.
3 Two-way Deconfounder
We introduce the proposed deconfounder algorithm in this section. We first present the proposed neural network architecture under two-way unmeasured confounding. We next define the loss function used for model training. Finally, we introduce model-based policy value estimator, built upon the estimated model.
Network Architecture.
The proposed model contains the following components: (1) d-dimensional
embedding vectors {u,}; : to model the enjecttory-seccfie latent confounders; (2) d-dimensional embedding vectors (we } to model the time-specific latent confounders; (3) A transition func- tion P( (elaa, a... O, te, we) that takes an action-observation pair (a, o) and a pair of embedding vectors (us, we) input to model the conditional distribution of the wwwrrdnnext-obeerattin pair given the current observation-action pair and the two-way unobserved confounders; (4) An actor network ( (olo, lo, ui, we) that takes and (u,, wx) input to model the behavior policy.
Our objective is to simultaneously learn the embedding representations and the parameters in both the transition and actor networks from the observed data. Toward that end, we treat each pair of embedding vectors (u, wx) as two entities. To accurately capture their joint effects on both the transition function and the behavior policy, we adopt the neural tensor network [NTN, Socheer et al. vectors.
2013), which is known for ability to capture the intricate interactions between pairs of entity
parameterize P(ola, O, U4 w.) via conditional Gaussian model given by Nf (ī (a, U; wx) diag(ō (a, o, ui wc))), where diag(ō (a, o, ug wx)) is a diagonal matrix and ā (a, o, ui, w.) denotes the vector consisting of all diagonal elements, Its conditional mean and variance functions are modeled jointly with the behavior policy, specified by
Specifically,
(7 @T.aTTy TT MLPp NTN Ui wc)). T
U4s we) MLP m, (NTN(o, Us w:)),
where NTN(o, u, wa) denotes the output vector from an NTN, and MLPp, MLP xs are the two multilayer perceptrons that take this output vector as input. The detailed architecture of NTN layer is presented in Appendix B.1 As such, the NTN layer captures the joint information from both the observation o and latent confounders üj ut and shared among the actor and transition networks. We provide an overview of the proposed model architecture in Figure (a).
Finally, we remark that while we model the transition function using a conditional Gaussian like other works in the RL literature [see e.g., Janner el al. 2019 Yu et al. 2020), more complex generative models such transformers or diffusion models are equally applicable.
Loss Function. As mentioned earlier, the proposed model contains both the transition network and the actor network. Accordingly, loss function is given by
L (D; furii (wr)e) =(1 a) LT + a. LA
5

---

Observation
oM my
Wew
Individuals unbudding
Actor network
Neural Tensor Network
Multilayer Perceptron
Au
individual ld time stamp
20
46
Ou-1
Multilayer Perceptron
Ra
T embrdding vectors
Transition network
20
450
Action
Toperty
(a) network architecture
(b) numerical experiments
Figure 2: (a) An overview of the proposed network architecture. (b) The upper panel reports MSEs under different unmeasured confounding assumptions for fitting the observed data whereas bottom panel displays the MSEs for off off-policy value prediction. The unconstrained unmeasured confounding model shows the best fit for the training data, due to overfitting. The OPE estimator under the proposed two-way unmeasured confounding achieves the smallest MSE. More details are referred to Appendix D.1
where T.r is the negative log likelihood of conditional Gaussian model for the transition network, LA is the cross-entropy loss between the actual action and actor network, and a (0, () hyperaarameter that balances the two losses. This loss is optimized to compute both the latent embeddings vectors (u, wr)s and the parameters in the three neural networks: NTN, MLPp p and MLP n4 Further details relegated to Appendix C.I to save space.
Alternative our loss function which involves both the actor and transition networks, one can consider minimizing other losses focused exclusively on either the transition or the actor network, not necessarily both. However, in presence of unmeasured confounding, note that both the behavior policy and transition function are influenced by these latent confounders. Consequently, our joint learning approach is expected to more effectively identify the unmeasured confounders compared to these transition-only actor-only approaches.
essential to
Model-based OPE. Based the estimated model, we develop a model-based estimator learn 77" via Monte Carlo policy evaluation [Sutton and Barto 2018 A key observation is that, since the two-way unmeasured confounders are intermediate estimator for the evaluation target 1
the empirical sum shown below an unbiased
E
Ra., Or,1, (Wo)e
NT
Note that in this formulation, the latent factors in the conditioning set can be substituted with our estimated embedding vectors. Additionally, based on our estimated transition network, the expectation E" can be effectively approximated using the Monte Carlo method. This approach allows to construct a model-based plug-in estimator for Equation (I].
To elaborate, the Monte Carlo simulation begins at t = 1 for each individual i. We sample an action À according to the target policy π T Ôr,)). draw samples for R., and Ô from the learned transition network ( A,,, Ō.,,, Ü., Wx) with the estimated latent factors {Ū.).. (Wehe cc. and iterate this procedure until the terminal time T is reached. Next, we replicate this simulation multiple times to reduce the Monte Carlo error. The final step is to aggregate all the estimated results across these simulations to construct the final OPE estimator.
4 Theoretical Results
In this section, we provide finite-sample error bound for the expected difference between the estimated policy value the ground truth 17th We first introduce some notations. We consider tabular setting where the observation space O, action space A and latent factor spaces U, W are all discrete. Let dp denote the data distribution of quadruples (a, o, u, w)), d o, u, w) = - EIAO.U a, U u, W =w. where the summation ((A.O,W.)ED

---

is carried out over all quadruples in the augmented dataset - D (U U (W containing both the observed data and the latent confounders. Furthermore, let d' represent the visitation distribution under л. Specifically, dt (a, u, w)) denotes the average probability of given quadruple (a, O, u, w) appearing at any time step under и (refer to Appendix B.I for the detailed definition).
We next introduce some assumptions to derive the finite-sample error bound of the proposed policy value estimator.
Assumption (Coverage). The data distribution covers the visitation distribution induced by the d" (e) oc.
larget policy i.c., C = a,,,u (a
Assumption (Boundedness). The absolute values of the immediate reward are upper bounded by some constant Rmax < 00.
for some Ep 0 where 11P alla denotes the total variation distance between P and P (see Appendix B.1 for the detailed definition).
Assumption 3 (Error bound of estimated transition function). EBI -Pllå S
Assumption 4 (Error bound of estimated latent confounders). EISIeN E U/11//N S ei.w and Eisegr B/ W /T ei, for some Su.W 0.
Assumption 5 (Attmmppionssttttoooootttaaaaaa (Autocorrelation)) For any 11,42 1, let p(t1, 12) denote the correlation coefficient between E= [R,,,, (W 1 and ET (R1,t, (We ) There exists some 0 I < such that Eightt5T7P(1 ((4,, = (7))).
We remark that conditions similar to Assumptions 1-2 are widely imposed in the RL literature to e.g.,Chenn simplify the theoretical analysis [see e.g., Chen and Jiang, 2019 Fan et al., 2020 Liu al. 2020 Uehara and Sun [202T]. Assumption 3 is concerned with the estimation error of the transition function.
This error is expected be minimal, since we use neural networks for function approximation [see e.g.. -9_[Schmidt--licbe) 2020 Farrell et al. [021]. Assumptions 4 is concerned with the estimation errors of the estimated two-way unmeasured confounders. According Proposition 3, these errors are negligible under simple models. Finally, Assumption 5 is purely technical. It measures the autocorrelation of the time series E (R1. ... W Here, a indicates independence over time. This condition is automatically satisfied when a = 1.
Theorem 1 (Finite-sample error bound). Suppose the two-way unmeasured confounding assumption holds, and Assumptions 1-5 are satisfied. Then
7 1/ CT Rmaxep cl UmaxEU cBRmaxN-1p3 + climax
for some constant c > 0.
It can be seen from Theorem that the mean absolute error of the proposed policy value estimator involves two components:
1. Estimation errors: The first two terms on the right side of the inequality correspond to the estimation errors of the transition function and latent confounders respectively. Notably, both terms are linear in the time horizon T, due to error accumulation (see Appendix B.5) However, such linear dependence is the best
hope in general (Jiang] 2024 , although is possible
to eliminate the dependence upon 7' under additional ergodicity assumptions [Liao et al. 2022].
Standard deviations: The last two terms measure the standard deviations of the average values across trajectories and time, respectively. These upper bounds decays zero, both N and T approach infinity, provided that the exponent a in Assumption 5 which measures the autocorrelation of the time series E (" )(=z) is strictly smaller than 1.
5 Experiments
In this section, we perform numerical experiments using two simulated datasets and one real-world dataset to demonstrate the effectiveness of the proposed two-way deconfounder (denoted by TWD) in handling unmeasured confounders. We consider two simulated examples in Section (.1] simple dynamic process and a tumor growth example. For each simulated example, the true value of 17" computed based on 10,000 Monte Carlo experiments. We also explore real-worle example using the
7

---

108
1966
1200
3000
100
201
2022
120
180
370
2000
1800
100
2012
(a) The simulated dynamic process
(b) Tumor growth example
Figure 3: Logarithmic MSE and Bias of various estimators for the simulated dynamic process and tumor growth example.
MIMIC-III dataset in Section 5.2 and conduct sensitivity analysis and an ablation study Sections 5.3 and 5.4 respectively. The source code is available on Github: Deconfounder
We use two metrics to evaluate different OPE estimators: the logarithmic mean squared error (LMSE) and bias. Both are estimated based on 20 simulations. Comparison is made between TWD and the following set of baseline methods, which covers wide range of model-based and model-free approaches:
(1) Model-based method (MB) that learns a transition model from the offline data based on the proposal by Yu et al. 2020) and applies the Monte Carlo method to construct the OPE estimator;
(2) Minimax weight learning [MWL, Uchara et al. 2020 that learns a marginalized importance sampling (MIS) ratio from offline data to constructs an MIS estimator for OPE;
(3) Double robust method (DR) that combines the MIS ratio and estimated Q-function computed via minimax learning (Uchara et al. 2020) to enhance robustness OPE;
(4) Partially observable MWL [PO-MWL,, [Shi al. 2022a
POMDP-type method that extends
MWL. to handle unmeasured confounders;
(5) Partially observable DR [PO-DR, [Shi et al. 2022a
another POMDP-type method that extends
DR to handle unmeasured confounders;
(6) Recurrent state-space method [RSSM, Hafner et al.
2019a b) that models unmeasured con-
founders as latent states;
77) Model-free two-way doubly inhomogeneous decision process (TWDIDPI) deconfounding-
type method that the model-free algorithm developed by BBan al.
2023
(8) Model-based two-way doubly inhomogeneous decision process (TWDIDP2) another model- based deconfounding-ly algorithm, also developed by Bian et al. 2023
Notably, the first three methods require the NUC assumption. The next three methods are POMDP- type, whereas the last are deconfounding-yye algorithms. Given our focus on settings without external proxies, we do not compare against methods developed under memoryless unmeasured confounding, which typically rely these proxies, as commented in Section
5.1 Simulation studies
Simulated Dynamic Process. We first consider a dynamic process with four-dimensional obser- vations and binary actions. The data is generated under the proposed TWUC assumption; Appendix D.2 for its detailed DGP. We fix T = 50, and vary the number trajectories from 250 to 2000. As shown in Figure 3(a) our proposed TWD estimator frequently achieves the smallest LMSE with bias closer to 0 in all cases. Additionally, the LMSE of TWD generally decreases as the number of trajectories increases, demonstrating its consistency. In contrast, most other methods under the assumption of NUC or POMDP setting are severely biased, highlighting the risks of ignoring or improperly handling unmeasured confounding.
Tumor Growth Example. We consider tumor growth example and utilize the pharmacokinetic- pharmacodynamic (PK-PD) model for data generation; Appendix D.3 details. The observation

---

10.0
policy
podlie
Number trajectories
Estimated Value
5.0
Madel
2000
3708
MB
0.055
0.252:50.005
0.0
RSSM
0.014
0,329 +-0008
TWDIDP2
0.262=0.008
0.259
-5.0
TWD
0.254
0.251
TWD
RSSM
TWDIDP2
MB
DR
MWL
PD-OR
PO-MWL
Methods
Figure 4: (a) The estimated policy value for four target policies in real-world dataset. (b) Average root MSE and its standard error in the results for predicting immediate reward and next observation. The results are aggregated over 20 runs.
is two-dimensional, including the tumor volume (t)) and the chemotherapy drug concentration C(t). The action space A includes two treatments: radiotherapy (t) and chemotherapy (t). We evaluate two target policies: a random policy (denoted by *A') and a individualized policy that is tailored to the patients' conditions (denoted by 'B'). As shown in Figure [3b] TWD consistently achieves the lowest LMSE and is empirically unbiased in all cases, demonstrating the effectiveness of our method. In contrast, other methods yield biased estimates, resulting in significantly higher LMSEs. Importantly, the performance of these alternative methods does not show significant improvement with increase in the total number of trajectories. These results underscore the applicability of our method in a more realistic scenario.
5.2 Real-world example: MIMIC-III database
In this section, we apply the proposed two-way deconfounder to the medical information mart for intensive care (MIMIC-III) database [Johnson et al. (0166). We extract 3,707 patients with trajectories up to 20 timesteps. Following the analysis of Zhou et al. [2023b],
define 5 x 5 action space and set the reward to the difference between current SOFA score and next SOFA score, so lower reward indicates higher risk of mortality. We also extract 12 covariates as the observation; further details are described in Appendix [D.]] The dataset is likely non-stationary and lacks patient's personal information. Therefore, it might be reasonable employ TWUC model the unmeasured confounders [Bian et al. [023).
Given that this is a real dataset, we do not have access to the true value of the target policy. To compare TWD against other baselines, we employ two approaches. The first approach uses 90% of data for training, and the remaining 10% for evaluating algorithm's prediction error for the reward and next observation. Notice that this approach is applicable to evaluate modell-asee methods only. We report all mean squared prediction errors in Figure 4(b) It can be seen that TWD results in the lowest prediction errors in all cases, demonstrating its effectiveness to infer unobserved confounders.
The second approach assesses each algorithm ability in distinguishing between tailored individualized policies and other random, non-individualized policy. An effective OPE algorithm should consistently rank an individualized policy as the superior policy. In what follows, four policies are evaluated using TWD and other baseline methods: a randomized policy, a non-individualized high dose policy, non-individualized low dose policy and an tailored individualized policy. As shown in Figure ((a) TWD and MB can effectively distinguish the individualized policy from other policies, with the individualized policy consistently achieving the highest estimated value. However, other methods lead to strange conclusions. For example, the result from TWDIDP2 suggest that all policies achieve similar values. Consequently, results from MWL.DR, PO-DR and RSSM suggest that the high dose policy is better than the tailored individualized policy. Additionally. TWDDIPP performs specially poor, rendering it unsuitable for display alongside other methods in this figure. While these results require further validation by medical professionals, they highlight the potential of the proposed method in real-world medical applications.
5.3 Sensitivity analysis
In this section, we investigate how TWD performs when the proposed TWUC assumption is violated. Specifically, vary unmeasured confounders that are both trajectory- and time-specific in the

---

brgeholy
14
82
64
Gertweding Stregh
Confensiigg Slreggh
(a) The simulated dynamic process
(b) Tumor Growth
Figure 5: Sensitivity analysis for the simulated dynamic process and tumor growth experiment.
reward function and behavior policy and we introduce a sensitivity parameter to quantify the extent to which the proposed TWUC assumption is violated. When I = 1, the proposed TWUC assumption holds; as I decreases towards zero, the assumption is increasingly violated. We vary
T= {0.0, 0.3, 0.7, 1.0) in the experiments, and fix the number of the trajectory to 1000. See further details in Appendix [D.5
We focus the two simulated examples. As shown in Figure ((a) in the simulated dynamic process, the performance of the proposed methods remains relatively stable as long as I >> However, when I T= 0 where the TWUC is completely violated TWD loses its superiority. Meanwhile, as shown in Figure [bb] in the tumor growth example, the performance of TWD is very sensitive to r, and TWD performs better than other methods only if 1.0.
5.4 Ablation study
We conduct an ablation study to compare TWD against the following variants:
(1) TWD with transition-only loss function (TWD-TO): This variant employs the proposed TWUC assumption, but removes the cross-entropy loss from the objective function. Consequently, solely uses the transition model to learn the two-way embedding vectors, without modeling the behavior policy during training.
(2) TWD without neural tensor network (TWD-MLP): In this variant, the neural tensor network is replaced with an MLP.
(3) One-way deconfounder without individual embedding (OWD-NI): This variant removes the individual embedding vector, operating under the one-way unmeasured confounding assumption.
(4) One-way deconfounder without time embedding (OWD-NT): This variant removes the time embedding vector.
We fix the number of trajectories to 1000 and report the LMSEs of various estimators in Table [ It can be seen that: (i) OWD-NI and OWD-NT significantly underperform TWD due to their reliance on the one- way unmeasured confounding assumption. (ii) TWD consistently outperforms TWD-MLP, due to the neural tensor network's ability to capture intricate interactions between the trajectory-specific and the time-specific unmeasured confounders. (iii) In the simulated dy- namic process, TWD achieves better performance than TWD-TO. This could be attributed to our proposed joint learning strategy, which simultaneously estimates both the transition function and the behavior policy and enhances the model's capability to infer unmeasured confounders. However, TWD performs worse than TWD-TO in the tumor growth example.
Table 1: Ablation study for variants TWD
Enviromments
TG
Model
A
B
A
TWD
4.23
4.47
4.42
4.33
TWD-TO
4.72
5.14
3.58
3.39
TWD-MLP
5.26
4.56
4.44
OWD-NT
7.50
8.59
6.80
6.92
OWD-NT
6.78
8.92
6.26
6.31
DP: the simulased dynamic precess, TG: the tumar growth example Target policy B: Target policy
In summary, these results demonstrate the effectiveness of the proposed two-way unmeasured confounding model and our joint learning strategy, along with the crucial role the neural tensor network in capturing complex interactions, all contributing to TWD's superior performance.
10

---

References
Theodore Wilbur Anderson and Cheng Hsiao. Formulation and estimation of dynamic models using panel data. Journal of econometrics, 18(1):47-82, 1982.
Dmitry Arkhangelsky and Guido W Imbens. Doubly robust identification for causal panel data models. The Econometrics Journal, 25(3):649-674, 2022.
Susan Athey and Guido W Imbens. Design-based analysis in difference difference-in-differeeeee in-differences settings with staggered adoption. Journal of Econometrics, 226(1):62-79, 2022.
Badi Hani Baltagi and Badi H Baltagi. Econometric analysis of panel data, volume 4. Springer, 2008.
Helmut Bartsch, Heike Dally, Odilia Popanda, Angela Risch, and Peter Schmezer. Genetic risk profiles for cancer susceptibility and therapy response. Cancer Prevention, pages 19-36, 2007.
Andrew Bennett and Nathan Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. Operations Research, 2023.
Zeyu Bian, Chengchun Shi, Zhengling Qi. and Lan Wang. Off-policy evaluation in doubly inhomoge-
neous environments. arXiv preprint 22X.0:2306.88719,
Ioana Bica, Ahmed Alaa, and Mihaela Van Der Schaar. Time series deconfounder: Estimating treatment effects over time the presence of hidden confounders. In International Conference on Machine Learning, pages 884-895. PMLR, 2020.
David Bruns-Smith and Angela Zhou Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders. arXiv preprint arXiv:2302.0066) 2023.
Brantly Callaway and Sonia Karami. Treatment effects in interactive fixed effects models with a small number of time periods. Journal of Econometrics, 133(1):184-22))) 208, 2023.
George Casella and Roger L Berger. Statistical inference. Cengage Learning. 2021.
Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 34: 27475-27490 2021.
Bin Chen and Yongmiao Hong. Testing for the markov property in time series. Econometric Theory, 2801):130-178, 2012.
Jinglin Chen and Nan Jiang. Information- considerations in batch reinforcement learning. CORR, hts/1905.00366 2019.
Xiaohong Chen and Zhengling Qi. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off -olicy evaluation. In International Conference on Machine Learning. pages 3558-35888 PMLR, 2022.
Yutian Chen, Liyun Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando De Freitas, and Arnaud Doucet. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23(302): 100, 2022.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, (111)))CCCC66. 2018.
Carlos Cinelli, Andrew Forney, and Judea Pearl. crash course in good and bad controls. Sociological Methods & Research, 53:00499241222100 1029999 05 2022. doi: 00.11777004912412200999
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off Off-policy confidence interval estimation. Advances in neural information processing systems, 33: 9398 90189941,,211 2020.
Alexander D' Amour. Comment: Reflections the deconfounder, 2019.

---

Clément De Chaisemartin and Xavier d' H'Haultfoeuile. Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9):2964 2996, 2020.
Raaz Dwivedi, Katherine Tian, Sabina Tomkins, Predrag Klasnja, Susan Murphy, and Devavrat Shah. Counterfactual inference for sequential experiments. arXiv preprini arXiv:2202.0689) 2022.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for dynamics and control, pages 486 489. PMLR, 2020.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off- policy evaluation. In International Conference Machine Learning, pages 1447-1456. PMLR, 2018.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):1112113, 2021.
Joachim Freyberger. Non-parametric panel data models with interactive fixed effects. The Review of Economic Studies, 85(3):1824- 1851, 2018.
Zuyue Fu, Zhengling Qi, Zhaoran Wang. Zhuoran Yang. Yanxun Xu, and Michael R Kosorok. Offline reinforcement learning with instrumental variables confounded markov decision processes. 2022.
arXiv preprint
Changran Geng, Harald Paganetti, and Clemens Grassberger. Prediction of treatment response for combined chemo-and radiation therapy for non-small cell lung cancer patients using a bio- mathematical model. Scientific reports, 7(1):13542, 2017.
Zvi Griliches. Issues in assessing the contribution of research and development to productivity growth. The bell journal of economics, pages 92-116, 1979.
Richard Guo, Anton Rask Lundborg, and Qingyuan Zhao. Confounder selection: Objectives and approaches, 2023.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019b.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference Machine Learning, pages 4074 4084. PMLR, 2021.
Tobias Hatt and Stefan Feuerriegel. Sequential deconfounding for causal inference with unobserved confounders. arXiv preprint arXiv:2104. 09323, 2021.
Xiangnan He, Lizi Liao, Hanwang Zhang. Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. Proceedings of the 26th international conference on world wide web, pages 173-182, 2017.
Mao Hong, Zhengling Qi. and Yanxun Xu. A policy gradient method for confounded pomdps. arXiv preprint arXiv:2305. 2023.
Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference on data mining, pages 263-272, leee, 2008.
Kosuke Imai and In Song Kim. On the use of two-way fixed effects regression models for causal inference with panel data. Political Analysis, (9(3)):055415, 2021.
Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via pollcy classification. Proceedings of the 33rd International Conference Neural Information Processing Systems, pages 5437-5448, 2019.
12

---

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions neural networks and learning systems, (3(2):4945544 2021.
Nan Jiang. A note on loss functions and error compounding in model-based reinforcement learning. arXiv preprint arXiv:2404. 240.0994444 2024.
Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. In International Conference Machine Learning, pages 652-661. PMLR, 2016.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):199, 2016.
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research, 70(6):3282-3302, 2022.
Nathan Kallus and Angela Zhou. Confounding-robust policy evaluation in teffintee-hrizoon reinforce- ment learning. Advances in neural information processing systems, 33:22293- -22304,, 2020.
Chinmaya Kausik, Yangyi Lu, Kevin Tan, Maggie Makar, Yixin Wang, and Ambuj Tewari. Offline policy evaluation and optimization under confounding. arXiv preprint arXiv:2211.16588 2022.
Chinmaya Kausik, Kevin Tan, and Ambuj Tewari. Learning mixtures of markov chains and mdps. In International Conference Machine Learning, pages 15970-16017. PMLR, 2023.
Jin Li, Ye Luo, and Xiaowei Zhang. Causal reinforcement learning: An instrumental variable approach. arXiv preprint arXiv:2
2021.
Luofeng Liao, Zuyue Fu, Zhuoran Yang. Yixin Wang, Mladen Kolar, and Zhaoran Wang. Instrumental variable value iteration for causal offline reinforcement learning, arXiv preprint 099907 2021a.
Peng Liao, Predrag Klasnja, and Susan Murphy. Off Off-policy estimation of long term average outcomes with applications to mobile health. Journal of the American Statistical Association, 116(533): 382-391, 2021b.
Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. Annals of statistics, 50(6):3364, 2022.
Bryan Lim. Forecasting treatment responses over time using recurrent marginal structural networks. Advances in neural information processing systems, 31, 2018
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- off-policy estimation. Advances in neural information processing systems, 31, 2018.
Yao Liu, Adith Swaminathan, Alckh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1264 1274. Curran Associates, Inc., 2020.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling, Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
Chaochao Lu, Bernhard Schölkopf. and José Miguel Hernández-Lobato, Deconfounding reinforce- ment learning in observational settings. arXiv preprint arXiv: 2018.
Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Edward McFowland III and Cosma Rohilla Shalizi. Estimating causal peer influence in homophilous social networks by inferring latent locations. Journal of the American Statistical Association, 118 (541):707-718, 2023.
Rui Miao, Zhengling Qi, and Xiaoke Zhang. Off-policy evaluation for episodic partially observ- able markov decision processes under non-parametric models. Advances in Neural Information Processing Systems, 35:593-606, 2022.
Yair Mundlak. Empirical production function free of management bias. Journal of Farm Economics, 43(1):44-56, 1961.
Yash Nair and Nan Jiang. A spectral approach to off- evaluation for pomdps. arXiv preprint arXiv:2109. 10502, 2021.
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy
evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33: 188 18838,,, 2020.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoe Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based convolutional neural network. arXiv preprini arXiv: 171202171 2017.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33, 2015.
Sven Nyholm and Jilles Smids. Automated cars meet human drivers: responsible human-robot coordination and the ethics of mixed traffic. Ethics and Information Technology, 22:335-344, 2020.
Elizabeth L. Ogburn, Ilya Shpitser, and Eric J. J.hhetteen Tchetgen. Comment on "blessings of multiple causes". Journal of the American Statistical Association, 114(528):111116115 October 2019. ISSN 0162- 1459. doi: 10. 0000000621459.2000 21..20091111191..
Elizabeth L Ogburn, Ilya Shpitser, and Eric J Tchetgen Tchetgen. Counterexamples to" the blessings of multiple causes" by wang and blei. arXiv preprint arXiv:2001.0655 2020.
Severi Rissanen and Pekka Marttinen. critical look the consistency of causal estimation with deep latent variable models. Advances in Neural Information Processing Systems, 34:4207-4217, 2021.
Pedro HC Sant' Anna and Jun Zhao. Doubly robust pifference-in-dfffereeee estimators. Journal of Econometrics, 211(1):101-122, 2020.
Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance resampling for off-policy prediction. Advances in Neural Information Processing Systems, 32, 2019.
Johannes Hhamidt-Hiebert Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4), August 2020. ISSN 0090-5364. doi: 10. 1875.
Abhin Shah, Raaz Dwivedi, Devavrat Shah, and Gregory W Woreel. On counterfactual inference with unobserved confounding. arXiv preprint arXiv:2211.08200 2022.
Chengchun Shi, Runzhe Wan, Rui Song, Wenbin Lu, and Ling Leng. Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In International Conference on Machine Learning, pages 8807-8817. PMLR, 2020.
Chengchun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off-policy interval estimation. In International conference on machine learning, pages 9580-9591. PMLR. 2021.
Chengchun Shi, Masatoshi Uehara, Jiawei Huang. and Nan Jiang. A minimax learning approach to off- policy evaluation confounded partially observable markov decision processes. International Conference Machine Learning, pages 20057-20094, PMLR, 2022a.
14

---

Chengehun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):765-793, 2022b.
Chengchun Shi, Jin Zhu, Shen Ye, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, pages 1-12, 2022c.
Kang Shuai, LAn Liu, Yangbo He, and Wei Li. Mediation pathway selection with unmeasured mediator-outcome confounding. arXiv preprint arXiv:2311. 2023.
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. arXiv preprint arXiv; 1910.07186, 2019.
Eric Tchetgen Tchetgen, Andrew Ying. Yifan Cui, Xu Shi, and Wang Miao. An introduction to proximal causal learning, arXiv preprint urXiv:2009.10888 2020.
Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-pollicy evaluation in partially observable envi- ronments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283, 2020.
Philip Thomas and Emma Brunskill. Data-efficient off- off-policy policy evaluation for reinforcement learning. International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Dustin Tran and David M Blei. Implicit causal models for genome-wide association studies. arXiv preprint 1017421710.10742, 2017.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline RL: PAC bounds and posterior sampling under partial coverage. CoRR, abb/210.0062226 2021.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off- policy evaluation. In International Conference on Machine Learning, pages 9659-9668. PMLR, 2020.
Victor Veitch, Yixin Wang, and David Blei. Using embeddings correct for unobserved confounding in networks. Advances in Neural Information Processing Systems, 32, 2019.
Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference. In Conference Uncertainty in Artificial Intelligence, pages 919-928. PMLR, 2020.
Hao Wang, Xingjian Shi, and Yan Yeung. Relational deep learning: A deep latent variable model for link prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jayi Wang, Zhengling Qi, and Chengchun Shi. Blessing from experts: Super reinforcement learning in confounded environments. arXiv preprint 1544442899.1448,, 2022.
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems, 34: 21164 21165221115,,,0111 2021.
Yixin Wang and David M Blei. The blessings of multiple causes. Journal of the American Statistical Association, 114(528): 1574- 1596, 2019.
Yixin Wang, Dawen Liang. Laurent Charlin, and David M Blei. The deconfounded recommender: causal inference approach to recommendation. arXiv preprint
2018.
15

---

Chuhan Xie, Wenhao Yang, and Zhihua Zhang. Semiparametrically efficient off off-policy evaluation in linear markov decision processes. In International Conference on Machine Learning, pages 38227-38257. PMLR, 2023.
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal oft-policy evaluation for reinforce- ment learning with marginalized importance sampling. Advances in Neural Information Processing Systems, 32, 2019.
Yang Xu, Jin Zhu, Chengchun Shi, Shikai Luo, and Rui Song. An instrumental variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pages 38848-38850. PMLR, 2023.
Mengxin Yu, Zhuoran Yang, and Jianqing Fan. Strategic decision-making in the presence of information asymmetry: Provably efficient rl with algorithmic instruments. arXiv preprint arXiv:2208. 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical report, Technical report, Technical Report R-23, Purdue Al Lab, 2016.
Linying Zhang, Yixin Wang, Anna Ostropolets, Jami J Mulgrave, David M Blei, and George Hripesak. The medical deconfounder: assessing treatment effects with electronic health records. In Machine Learning for Healthcare Conference, pages 490-512. PMLR, 2019.
Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu, Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprint arXiv:2309. 13278, 2023a.
Yunzhe Zhou, Zhengling Qi, Chengchun Shi, and Lexin Li. Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach. In International Conference on Artificial Intelligence and Statistics, pages 6704-6721. PMLR, 2023b.
Yunzhe Zhou, Chengchun Shi, Lexin Li, and Qiwei Yao. Testing for the markov property in time series via deep conditional generative learning. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(4): 12(4))11204122.. 2023c.
A Discussion
A.1 Critiques of Wang and Blei 2019
In this section, we discuss the critiques of the paper Wang and Blei 2019 and illustrate how our proposal addresses these criticisms.
Specifically, Wang and Blei 2019] imposed the "consistency of substitute confounders" assumption which requires that the unmeasured confounders Z, can be consistently estimated from the causes A, However, this assumption indeed invalidates the derivations of Theorems 6-8 of Wang and Blei [2019, resulting in the inconsistency of the algorithm. Specifically,
commented by D Amour 20019] the event = a provides perfect measurement Z such that there is some function 2 ((A) such that ((a) Z, then the overlap condition fails. As result, the ATE cannot be consistently identified. Ogburn et al. (0099 expressed similar concerns towards the algorithm.
Unlike Wang and Blei [2019] our algorithm does not require such an assumption. Under the RL setting, the proposed two-way unmeasured confounding assumption effectively limits the number of unmeasured confounders to O(N) + O(T), which facilitates their consistent estimation when both the number of trajectories N and the number of decision points per trajectory grow to infinity, avoiding the need for the unmeasured confounders to be deterministic functions of the actions.
16

---

A.2 Limitations
Although the proposed two-way unmeasured confounding assumption more flexible than the existing one-way unmeasured confounding, it might not adequately handle more complex scenarios involving latent confounders that are both trajectory- and time-specifie, or policy-dependent. In practice, this assumption
be partially evaluated by testing whether including the estimated two- way confounders in the observation leads to a Markovian system [see c.g., Chen and Hong 2012 Shi al. 2020
Zhou et al. 2023c
A.3 Future work
Our proposal requires the existence of observation which, along with the two-way confounders, blocks all backdoor paths between the treatment and both the immediate reward and future observa- tions. Naïvely using pre-treatment variables as observations might lead to biased estimators in certain causal models like the M-graph Cinelli et al. [0022]. While confounder selection algorithms [Guo al., 2026) are designed address the problem, they mainly focus on the bandit setting. Our future work is to extend these algorithms to RL to further improve the practical applicability our method.
B Proofs
B.1 Notations
We first introduce list the notations that will be used our proof.
NTN(o, u, wx) = f(g(u) Wi MT,, where Ti is shorthand for Jo,,,: u, w wT] , is standard linear layer and is the activation function. Within this function, Wil-k] Raxdxk is three-dimensiona. tensor and the bilinear tensor products uf W7L:A) w; result in vector whose m-th entry is given by uu Wiml,,, where Wiml denotes the mn-th slice of the tensor. M E (2d) where do represents the dimension of observation o, and b R is the bias term.
R(a, o, w) En.o rP(r o' (a, O, u, w), which denotes the expectation of reward given obser- vation o, action and latent factors (u, w), and R denotes its estimator.
P(o'la, o, u, w) := -.. o'|a, o, w) denotes the observation transition function, which cal- culates the probability of transitioning from observation to observation o' given the action a and latent factors (u, w). and P denotes its estimator.
Pu( denotes the probability mass function of (0..1U)).
dp denotes the data distribution of quadruples (a, u, w), given by do(a,o,u,,) =
T-1 ""T IPOO,, = o, AI,: a, U1 u, W, =)).
d denotes the distribution of quadruples (a, o, u, w)) in a trajectory of horizon T generated under
n, i.e., d= (a, o, w) = - pa (O1, 0, A a, U1 = u, W, w).
TV aa, w), ( o, w)) = Eno' P(rr, 0'1a, u, P(r, o'Ia, o, w)l denotes the total variation between ( a. o, u, w) and P( a, 0, u, w)
|| Plas == VE TV (P the average estimation error of P on the given offline dataset.
P( (,O,,,)) is used to measure
Additionally, we will use generic constant in the proof, whose value allowed to vary from place to place.
B.2 Proof of Proposition
Proof. Notice that the MSE of the predicted ewward-next-observatioo pair is equal to the sum of the MSE of the predicted immediate reward and that of the predicted next observation. Consequently, suffices to show the MSE of the predicted immediate reward remains constant.
17

---

Toward that end, we consider the following linear model under unconstrained unmeasured confound- ing.
Ra. = (Oi,t. Ast) + Zi,t + Sitt
where Z are one-dimensional, ( (4, (2) = R2a denotes the coefficient vector, and 6
are mean zero i.id. measurement errors with each Et,r being independent of the set of variables H.,, = {[Og,1' Ar,,, Zv) : # or t' I t}.
vector and matrix notations, we define Y = (BI,1...., RI,,, 2 R as the (NT)- dimensional outcome vector, 3 (61. S2, Z1.1...., ZITT.... I (NTT) as the (NT + 2)- dimensional coefficient vector, = (51,1) ,,,,,,.,,,,,77,7 EI,Ts ,N ENTT) RNT as the set of residuals and
O1,1
A1,1
1
O1
A
0
0
x
E R NTX
ON,
AN,T
Thus, we can formulate the linear model as = X3 + E. Since the LSE 3 an unbiased estimator, its predictor y =x3ss MSE can be calculated as follows,
M NT
F B(VIXB)11 = NT XX X) x E1
NT E|E X(X x) NT
E trace X(X x) xT (TEET]]
(2)
NT trace| X(X x) x =
where the conditional expectation E( (33) is taken in componentwise manner, (X x)- denotes the generalized inverse of X X and σ² Var(E;,,). Here, the second last equation holds due to the independence between ε are Higt. Consequently, the MSE is a fixed constant. This completes the proof.
B.3 Proof of Proposition 2
Proof. Similar to the proof of Proposition it suffices to prove that the MSE of the predicted immediate reward remains constant. Toward that end, we assume the immediate reward satisfies the following linear two-way unmeasured confounding assumption,
R = (O,,,, A,,) + U, + W, +
However, one-way unmeasured confounding assumes only the existence of trajectory-speed unmea- sured confounders and ignores time- -mee-fpeciicc confounders, leading to the following model
R1,: (O,.,, H, Ei.
Because are more concerned with estimating fixed effects and also to simplify subsequent symbols, consider scenario that is easier to estimate. Let Ç denote the LSE of Ç based on the one-way model. Since XT X xX = XT,, the LSE of H, satisfies
1 T Σ R4, (O = U; + T
T
7
Σ
w. (O.,,
t=1
Therefore, the MSE of the predicted Ri,, = (Oi,, Art) + A is given by
T Σ w.)
NT Σ AI.O,..A..)++
(0,,,,,,1) -( U, w.l T
B(W, T
For stationary and ergodic time series ( T- Wr is to converge to E(W,) as T 00 and thus, the above equation converge to the variance of W, Consequently, unless each W, degenerate, the MSE will not decay to zero. This completes the proof.
18

---

B.4 Proof of Proposition
Proof. To simplify the proof, we only show the MSE of predicted immediate reward decays zero as both N and T grow to infinity. The two-way model can be similarly expressed as Y = x &, where NT, (Gi, ......, UN, ...... Wr)
Y 1R BI.T.... .N....... RNTT)
R (E1,1...... ,,,,, .. ENT
E RNT and
01
A1
1r
0r
Or
IT
O2
A2
0r
1r
0r
IT
=
E ERRTXX(+++2)
ON
AN
Or
Or
1r
IT
where 1 = (1,.... 1)T E RT Or = 0 17, and IT represents the T x identity matrix.
Similar to Equation 2 the MSE of the predicted reward is equal to
BIg X(X x)- =|== trace(e X T X) O2 + T +)
MSE( NT
NT
NT
which decays zero both N and T increase infinity. The proof is hence completed.
B.5
Proof of Theorem 1
To simplify the proof, we analyze a variant of our estimator computed via sample-spliting and cross-fitting. Specifically, we begin by dividing the entire dataset D into two subsets D and D(2), each containing 2/2 trajectories. Next, algorithm detailed in Section 2 to the two data subsets to learn the transition functions and latent confounders. Denote them by p()), (7()), ) and (WF)), )., respectively, for j 1.2. Finally, we construct two model-based estimators, given by
separately apply the proposed two-way deconfounder
7(11 = NT
Σ iED!)
Σ
RR, 10., 0 (W! -== 22))
(3)
1=1
and
(2
2
Σ iEDG)
Σ
Ras 111, 0(00, ( B
[wi211 M=1
(4)
NT
where RU0) denotes the expectation that uses the transition function p() to approximate E", and the summation ΣieDo is carried over all trajectories in D(3,, We average them construct our final
estimator 77 = (7 1/2
Notice that in both (3) and ) the transition function used to conduct the model-based estimator independent of the initial observation and the estimated latent confounders. This avoids imposing additional VC-class type conditions on the transition network. We remark that sample splitting widely used in statistics and machine learning [see e.g., [Chennozuukvv et al. 2018 Shi et al. 2021
Kallus and Uchara 2022].
Proof. We focus on bounding (1 1/= The same upper bound applies ET v and E E 17" thanks to the triangle inequality. To ease notation, we will remove the superseripts in
(2) ((2), ((") (!))) and present them as R, P. Ü. W. Due the of cross-fitting, E and P are independent Ü.S and W.s. The proof is divided into two steps. In the first step, we upper bound E]G(1) [1) where
T
7
2
Σ
E
(R1, (We)k
NT
SEDU)
t=1
Next, in the second step, we upper bound E[T(1) 1 Finally, combining these two bounds leads to the upper bound for (1) -17"
19

---

Step 1. Recall that
T B(Ri, Ui (Wr)-
77(1)
Σ
NT
iED!
Notice that the difference (7(1)
77(1) can be decomposed into the sum of
U. -( 1E (R,, (Wr))
NT
iDD()
t=1
Σ
(R.,lOO.1. (W.) B(R 02 (.
NT
IED()) 1=1
11 11+1 12.
We first study I1 Since O2,18 and Us are independent of Wis, the first line forms of i.i.d. random variables. Thus, converges to
E (RR,,OO o, U, -
- E(R,,
o, (...) (We.e-))
T
T
xpolo, u) = ΣΣ
1
Σ
Hi,l (W=)oo))o u),
1=1 0,
with the expected error E Til upper bounded by cRmax/VN for some constant by Cauchy-Schwarz inequality.
It remains study Li, equivalently, Hi, for each t. To begin with, consider the case where t 1. follows that
(1, u, Wi)l
Σ [aloo) o, W,) A(a, u, w.)l
Σ -
a.r,a
S Rmax
When = 2, one can show that
11,200, u, (Wa) S Rmax Σ (P (1,,0 (1.
u, Wi))
a1
+ Rmax Σ
02, u, W2), P( a2 02, u ( W1).
02:02:01
Using the same argument, show that
[T,, u, (Welt Rmax
Σ Σ
(aalo II
P(oj,):) W3)
1= ap
XTV(P aa,, Ou, Wx), P(olue, or, u, We)).
As such, we obtain
E (o, (W:))))oo
R (TTTV(((|AA... U,, P(le O U1, W.)).
Using the change of measure theorem, the right-hand-side can be upper bounded by
CT ET O U1 P( Ore, U1, Wo)),
20

---

where C' denotes the single-polige concentration coefficient defined in Assumption 1. By Cauchy- Schwarz inequality. the above expression can be further bounded from above by C'T P.. Using similar arguments, we can upper bound /2] by
Rimax N
N
T
Tax W.) U, W.))]
0,0
1=1
As neural networks are Lipschitz continuous functions of their parameters, the above total variation norm is proportional to |I(Ū, W - (U,", W Ill.. Consequently, under Assumption 4, the above expression is upper bounded by cT U for some constant 0. This completes the proof of Step 1.
Step 2. According to the DGP described in Section 2 (1,)) ... (Un, On,1) i.i.d., n/",
and are independent of }- 17" can thus be represented by 7-1 Et=1 E" (R1,). We further decompose 17722) -7 into the following two components:
17(2)
11"
2
Σ
EET (Rr, Us (Wr):=
1
E
(R1,1)
NT
T
iED() t=1
t=1
Σ
E (Ra, (Wo)t
1
E3 (BI.
NT
T
IEDO)
t=1
1=1
T
T
E (W
- T
Σ E
(R11)
= I3 + 14-
T
Conditional on (Weh.. h. 13 corresponds to a sum of i..d. random variables. An application of Cauchy- Schwarz inequality implies that El3 S VEI < cllma\/ /N,, for some constant >> 0. Similarly, under Assumption 5, we have
Rmax T
EL4 S V E1
Σ
p(ti, t2) (RmaaxT-1))
1Sh.
The proof is hence completed by combining the results in both steps.
C
Implementation.
C.1 Details for loss function
The final loss is defined as follows:
L (D; (ui (we]t)) =(1 ())
1 Σ
[ pi.)] -5- [7 spre + log del E}
i,t
+ Σ
CrossEntropy (ai,s, ui, ui, wx))
10,1) a hyperparameter, Pic (ri,; and (ai, Oi,rs Fi, 11) is sampled from the offline dataset D, E diag(ō), det Σ represents the determinant of Σ, and CrossEntropy the cross-entropy loss.
where loss weighting
C.2 Implementation Details for Two-way Deconfounder Model.
The Two-way Deconfounder Model described in Section was implemented in Pytorch and trained on an NVIDIA GeForce RTX 3090. For every task, we repeat it times and the results are aggregated over 20 runs. The training time of 4 to 12 hours hours (depending on the task) on a single NVIDIA GeForce RTX 3090. Each dataset undergoes 75/25 split for training, validation respectively. Using the validation set, we perform hyperparameter optimization using many iterations of grid
21

---

search to find the optimal values for hyperparameters. The search range for each hyperparameter described as follows, learning rate Ir E [0.005, 0.001], Batch size bs E [2 29, 210 21 ,2 2 weight decay À E [0.01, 0.0001], two-way embedding dimension dew E [2 2 28), Loss weighting [0.0, 0.7.
C.3 Implementation Details for Ablation Study.
Variants of TWD are model-based methods, model-based method with two-way embedding(TWD TO.TWD-MLP) and model-based method with one-way Srbbedddng(WDDN.TWDD..
al.
[2020] Kausik et al. 2022], where the first can be applied to the two-way setting, and the second is originally designed for the one-way case. We adopt the same embedding approach for variants of TWD as our method. TWD-TO can be seen as special case of our method, which only remove the CrossEntropy loss of the behavior policy. TWD-MLP remove the neural tensor network. However, the biggest difference between TWD-NI or TWD-NT and our method is that it use the one-way embedding based on Kausik et al. [022] removing the embedding of trajectory-invariant or time- invariant unmeasured confounders. We provide Change details of those variants compared to TWD the following.
TWD-TO: Because it removes the CrossEntropy loss of the behavior policy, the loss function is as follows,
L (D; (u:4s (we))) EH sa,el Σ-1 [ sP,, + log det ΣJ},
other model settings are the same as TWD.
TWD-MLPP: Its conditional mean and variance functions modeled jointly with the behavior policy,
(T, = -LP M u, wc)):
o, Ug w.) = MIP (MLP(o, u;, w)))
It replaces the neural tensor network with simple MLP and MLP is the multilayer percep- tron that take o, M2 and wi as input.
TWD-NI: Its conditional mean and variance functions are modeled jointly with the behavior policy,
TTT MLP NI(o, w.)). m
wt) = MLP n. (NI(o, wx)),
It removes the individual embedding and NI the multilayer perceptron that take o and we as input.
TWD-NT: Its conditional mean and variance functions are modeled jointly with the behavior policy,
T MLP MLPp NT(o, uz)), ( u:)) = MLP (NT(o, uz),
It removes the time embedding and NT is the multilayer perceptron that take and uq as input.
D Simulation details
D.1 A linear simulation setting
Let O R2 = R. W R. R R. and A {1,0). For evey and t, u, and w, are sampled Normal distribution N (0, 1). The observed data is generated as follows,
The transition model. For each and 1, given (Oi,, Ar,, Us W.), We generate
0 0.70., + A, + 28, + 2W, - 0.5 + c,
22

---

where the random error N (0,1)
The reward model. For each i and I, given (Oi,t, A,,, Uú, Wt), We generate
a.. Oz. + 3A,, + 2U; + 2W; er
where the random error e, N (0,
Behavior policy. For each i and t. given (Os,t: U2 Wt). We generate
p p = 1 Or,,, U4 W;) sigmoid (O,,e + U3 W3)
Target policy. p(A,, = 1) = 0.5
We generate datasets respectively under the behavioral and target policy,. Under the assumptions of one-way unmeasured confounders, two-way unmeasured confounders, and unconstrained unmeasured confounders, respectively,
the least squares method to fit the model parameters. Then, use the learned model to perform the off-policy evaluation. The number of trajectory is N = (200, 300, 400, 500, 600, 700, 800). The length of the horizon is fixed at 50. calculate MSE error in fiuting the observed data and MSE error off-policy evaluation, respectively. Two-way unmeasured confounders outperformed the other two on off off-policy evaluation, as shown the Figure
D.2 Simulated Dynamic Process
We first describe the detailed setting for the simulated data. Let O = Rt = R,W R. R I, and A {1.0}, We let T 50 and the number of data trajectories N (250, 500, 1000, 1500, 2000). We consider a four-dimensional variable Oz, = (O), Ph,,- O Of.)) whose initial distribution given by N IT) where I4 denotes a four-dimensional iden- tity matrix. The distribution N (0, 1). The time-invariant unmeasured confounders [We]e are sampled a Nor- mal distribution N (0, 1). The data generating process is outlined as follows, reward func- tion and transition function are given by Ri, Jr (Oi.e, Ai,, Ui, W., + Gr. Er ~ N (0. 1), and 0 fo (Oi,, A,,, U2, W,, I) + Co. Eo .N' (01, Lt). the behavior policy satisfies F (A,, = Oia, U., W;) = sigmoid (fa (Ota, I"). I' is the confounding strength pa- rameter. We I' 1 in this section. fo(-). fr(-) and (-) are the functions of input variable on the next observation, immediate reward, and action, respectively.
unmeasured confounders (Ur},
sampled Normal
The transition model. For each and t, given (Oi,, Ar,,, , Us, W.) and r, We generate
Oz.41 = O., + + BoJ (Us, Wi) + A0 + b, +
where I = [1, 1, T the random error e, ~ N (0., I,) with I, denoting the 4-by-4 identity matrix,
0.8
0
0
0.8
0
11
0
0.8
0
0
0.8
0.1
0
0
0.1
0
30
0.1
0
0
0.1
A 1,
b. = 0-0.5 -0.5, -.5,, T.
fo (U,, W.) = -[u - Wt, i4 + Wt. - U; - Wt, -; + wi]
The reward model. For each i and t, given (Oi,e, Ai,, Ui, W.) and I, We generate
R = 140, + P 8 (U,, WT) + A, + where the random e A' (0,1), 1), 3, 3.0, X, 2.5.
μ4 [0.25, 0.25, 0.25, 0.25]
23

---

fr ( =
Behavior policy. For each i and I, given (O,,e; ., U,, W.) and r, generate
p p Oz,,, U4 Wi) sigmoid (1, + r Bafa (U,, W.)
where Ba 3,
Na = 10.25, 0.25, 0.25, 0.25] f. (.., - uj Wt + u, + Wt.
Target policy.
Target policy A: p(A, = 1) = 0.3.
Target policy B: p(A,, = 1) = 0.5.
D.3 Tumor Growth simulated data setup
In this section, we give brief description about Tumor Growth environment setting and experiment details. We set up many subsets randomly with different numbers of patients who received treatments sequentially over 60 stages. To be specific, randomized patients into three groups, with each patient having a group label S; E [1,2,3)
The transition model. We apply the unobserved confounding variables to the PK-PD model:
where and K are model parameters sampled for each patient according to prior distributions in Geng et al. 2017 Lim 2018), Bica et al. 2020) and e, ~ N (0, 0.01²). Specifically, S... a and 3 are model parameters sampled for representing specific characteristics which affect with patient's 's response to chemotherapy and radiotherapy according to randomly subclassing patients into different groups S; (1, in Geng al. [2017] Lim [2018). Bica et al. [2020]. which are influenced by genetic factors [Bartsch et al. 2007 and can be regarded as time invariant confounders.
((t) + plog v(r-1) - (aA +
+t, +e)
C(t) 1)/2+ 2 decays over time.
5 +-
(t), which relies the chemo treatment action and exponentially
The reward model. The reward model consists of three parts:
pathological- -eelthhh reward: A negative reward R, for penalising the patient if tumor size is large or accepts too much drugs and radio: 1.5 cxp(-V(tt).
side-effect reward: A positive reward Rd for accepting treatments when take action time ?t)
step t: cxp(-(A'
reward imposed by unmeasured confounders: A reward Rrwaboot the two-way confounder parameter: sS 2) (0.1l..t.)
total reward: R R, + Rp + Rrw + c,, N(.1))
Behavior policy. We introduce two-way unmeasured confounders to treatment assignment policy, For each i and I, We generate,
(t) Bernouli((sigmoid
Te
(D(t) 0.) + (3- sigmoid(S, 2) 0.75 sin (0n(1tt)))
D Diax
A" Bernouuli((igmooid
Yr
(D(t) 0,) + (3 sigmoid(S; 2) 0..7 sin (0)1T)))))
Dmax
where D(t) is turoor diameters, Dmaxis the half maximum size the tumor, 0g and 0, are fixed such that 0. = 0, D Dmax/2 / and Yo and 7, also fixed such that % 7/- 10.0.
Target policy.
Target policy A: A"(), (ernoulli(0.05).
24

---

Target policy B: (tt)) ~ Bernoulli(p), where is related to the conditions of patients:
0.2,
if Y > 88, i/ < Y, S 88, if < Y, 44 otherwize
0.1,
p
(5)
0.05,
0.01,
D.4 real-world example: MIMIC-III
We show how the Two-way Deconfounder can be applied to real-world medical setting using the Medical Information Mart for Intensive Care (MIMIC-III) database [Johnson et al. 2016]. We extract 3,707 patients with trajectories up to 20 timesteps. Following the analysis of Zhou al. [2023b], we define x 5 action space by discretizing both vasopressors fluid A" and intravenous A' fluid interventions into 5 levels. We define the reward R. SOF Ai, SOF 1 as the difference between current SOFA score and next SOFA score, so a lower reward indicates a higher risk of mortality. We extract 12 patient covariates observation space, including Glasgow coma Systolc blood pressure, Weight, SOFA, SIRS, Fraction of inspired oxygen, Blood Urea Transaminase, Partial Pressure of Oxygen Total Bilirubin, Platelet Count. Considering the complexity of the real data, we incorporate the normalization step before modeling them.
Nitrogen, Creatinine, Serum
Target policy.
the randomized policy: P(4" 0) P(A" = p(A" = P(A" 3) = p(
4) = 0.2, P( = 0) = P(A' = 1) = p=1'= =2 = =3) = =p(4' = =4) =02
the high dose polic: P(A" = 3) = =(AA" =1, =3) p(A'= 4-) 0.5.
the low dose polic: P(4" 0) 0.4, P(A-= =) = p(A" = 2) 0.3, P(A' 0)
P(A = 1) p(A' 2) = 0.3.
the tailored policy:
if normalized SOFA score > 0.95, p(A" = = - = 3) = 0.3. p(A" = 2) = 0.2
and P(A" 0) = P(A" = p('' = = = 3) 0.3. =2) =.2
and P(A' = 0) P(A' = 1) = 00
if 0.95 > normalized SOFA score > 0.7, p(A" 4) p(A 3) 0.1,
2) = 0.2, p(A" = =1) = p(A" = 0) = 0.3:p(A' = 4) = p(A' = 3) = 0.1,
2) 0.2, p(A' = =(A' 0) 0.3.
if normalized SOFA score 0.7, (AA 0) 0.8. 0.8. p(A" 1) 0.2andp(A' (A"
2) p(A"= 1) 1)==4A) p(A" = 0) = 0; P(A' = 0) = 0.8, P(A' 1) = 0.2andp((A
2) P(A' = p(A' = =) 0.
D.5 Sensitivity analysis
In two simulated datasets, the modified reward model and behavior policy according to sensitivity parameter I are as follows,
Simulated Dynamic Process.
The reward model: For each and given (Oi,,, Ai, U, W.) and We generate
R p4, 1r, +I 8, (U, Wt) + A, + er where the random emror 6, N' (0, 1), 8, = 3.0. X, 2.5,
n4, 0.225, 0.25, 0.25, 0.25]T.
- f, (U,, Wt) u, Wt.
Behavior policy: For each i and t, given (Oi,,, . U/ Wc) and I. We generate p ( O U, Wi) sigmoid Oi. 4 + r. Bafa (U,, Wt) where 3. = 3,
25

---

Ma = 0.25,.025. 0.25, 0.25] fr (U,, W:) u, + u, + Wt.
Tumor Growth.
The reward model: R R. Rr Rrw cr, where & N (0, 1) and Rrw r (4 siigmidd S; 2) - sin(0.1rt))
Behavior policy: For each i and t, We generate.
(t) Bernooul((sigmoid
No (D(t) - 0e)
Diax
+ (3 sigmoid (S: - 2) - 0.75 - -in(O.1
(T) ~ -Bernouli(ssigoidd 7r (D(I) - 0,)
Diax
D. sigmoid(S, 2) 0.75 sin(0.1
where (() is tumor diameters, maaxis the half maximum size of the tumor, 0. and 0, are fixed such that 0g 0, Dmaxx 2 and Ta and ", are also fixed such that "o = 7r 10.0
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
At the last part of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
All assumptions are provided in Section 2 and Section 4 while the specific
proofs of theoretical results are shown in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The information needed to reproduce the main experimental results are shown in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data is subject to an agreement and cannot be shared due to its sensitivity, but it is publicly available at httts:://pyyiionett org/ wwtpp//wwt.../oooettttteeeeeeeet code is provided in Section
4/ The
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cem
cublic/guidest Staaless/oooSSmbssssioooooo.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
sooolioicciiiiiiiiiooooooooiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details about experiment in Section 5
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide these information in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwrrr...cccccaaaccccaaa
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper merely proposes a new method for off-policy evaluation, and there are no negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "FiFvMAubY8", "reviewer": "Reviewer_wVkG", "summary": "The authors study off-policy evaluation for longitudinal data with hidden confounders, which are accounted for by assuming they are either time-invariant or state-invariant but not both.", "questions": "In Figure 4, why are TWD's estimated values so much more negative? And why is the variance higher?", "limitations": "The authors should discuss limitations in the main text rather than the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea is relatively original, important, and the contribution might be significant. Presentation is somewhat clear.\n * The section studying real-world MIMIC III data shows that this method might be practically relevant.", "weaknesses": "* Despite the term \"deconfounder\" being featured prominently, the authors do not engage with much of the literature on the deconfounder algorithm initially proposed by Wang and Blei (2019). Not much is said beyond the cryptic statement, \"the validity of the deconfounder algorithm relies crucially on the consistent estimation of the latent factors.\" This method is also inspired by two-way fixed effects regression, but since they are drawing connections to the deconfounder algorithm, the authors should discuss its fundamental assumptions in the context of its limitations and criticisms by D'Amour, Ogburn et al., and others.\n\n * The assumption of two-way unmeasured confounding is illuminated with concrete examples starting at line 127, but the Propositions 1-3 that follow are difficult to understand without explicit context. Looking at Appendix A, it seems that additional assumptions need to be made for Proposition 2 or Proposition 3 to be true. \n\nMinor comments\n * Typo on line 117: \"pluggs\"\n * Figure 5 is too small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- __Engagement with the deconfounder literature, limitations and criticisms of [1]__\\\nThis is an excellent comment. A detailed discussion of the criticisms of the deconfounder algorithm also helps better motivate our algorithm. We plan to include the following discussion after our cryptic statement you mentioned, to more clearly elaborate on the limitations of the deconfounder: \n\n  To ensure the latent factors can be estimated precisely, [1] imposed the **Consistency of Substitute Confounders** assumption, requiring to estimate the unmeasured confounders $Z_i$ from the causes $A_i$ with certainty. However, this assumption indeed **invalidates** the derivations of Theorems 6-8 of [1], resulting in the inconsistency of the algorithm. Specifically, as commented by [2], if the event $A=a$ provides a perfect measurement of $Z$ such that there is some function $\\hat{z}(A)$ such that $\\hat{z}(a)=Z$, then the overlap condition fails. As a result, the ATE cannot be consistently identified. [3] expressed similar concerns towards the algorithm. Unlike [1], our algorithm does not require such an assumption. Under the RL setting, the proposed two-way unmeasured confounding assumption effectively limits the number of unmeasured confounders to $O(N)+O(T)$, which facilitates their consistent estimation when both the number of trajectories $N$ and the number of decision points per trajectory $T$ grow to infinity, avoiding the need for the unmeasured confounders to be deterministic functions of the actions. \n- __Clarifications of Propositions 1-3__\\\nLet us first clarify these propositions:\n  - Proposition 1 aims to prove that when **the true model satisfies the unconstrained unmeasured confounding (UUC) assumption**, we cannot obtain consistent estimation even if **we work with the correct model assumption (i.e., UUC)**.\n  - Proposition 2 aims to demonstrate that if **the true model satisfies two-way unmeasured confounding (TWUC)** and **we erroneously assume the model satisfies the one-way unmeasured confounding (OWUC) assumption**, then our estimators would fail to be consistent. It reveals the limitations of OUC in the presence of time-varying unmeasured confounders, thus highlighting the necessity of the proposed TWUC.\n  - Proposition 3 aims to show that when **the true model satisfies TWUC and we correctly specify two-way model**, we can obtain consistent estimation. \n\n  Meanwhile, shall our paper be accepted, we will explicitly list these underlying true model assumptions and imposed model assumptions highlighted in bold, to make these propositions clearer.\n\n  Finally, there are two additional assumptions in the proof of Proposition 2:\n  - The first assumes the oracle $\\zeta$ is known;\n  - The second assumes all $W_t$s are i.i.d.\n \n  Our intention was to make the main text concise and the presentation easy to follow, so we did not introduce them in the statement of Proposition 2. The first condition is not necessary to impose, as an unknown $\\zeta$ would enlarge the estimator's MSE under OWUC. Shall our paper be accepted, we will remove the first condition and explicitly state the second condition in Proposition 2. We hope the aforementioned changes clarify these propositions.\n\n- __Clarification on the experimental results of Figure 4.__\n    - __The issue of higher variance:__ \n   This is another excellent comment. In response, upon checking the code of a related paper [4], we discovered that our procedure omitted a crucial step—data normalization. Given the large state dimension and significant variance among the values within each dimension in the real data, this omission likely contributed to the higher variance observed in TWD's estimated values. During this rebuttal period, we have incorporated the normalization step and re-analyzed the dataset. The updated results, illustrated in Figure 3(a) of the PDF file, demonstrate that the issue with high variance has been addressed effectively, and TWD still performs well. We will modify this figure shall our paper be accepted.\n    - __The issue of negative values:__\n    The negative values arise due to the design of the reward function. In our submitted manuscript, we set the reward to $R_{i,t}=SOFA_{i,t}-SOFA_{i,t+1}$. This choice has been previously used by [5], who also reported similar negative values. During this rebuttual, we have considered two new reward functions $R_{i,t}=-SOFA_{i,t+1}$ and $R_{i,t}=23-SOFA_{i,t+1}$, which were used in [4], and conducted additional experiments under these designs. As shown in Figure 3 of the PDF file, the estimated values of TWD vary from positive to zero to negative, confirming that the appearance of negative values stems from the choice of the reward functions. We will add these discussion shall our paper be accepted.\n- __Discussion of limitations, typos and Figure 5.__\\\nWe will correct the typos and move the limitations of our algorithm to the main text. Figure 5 will be enlarged shall our paper be accepted.\n- __References.__\\\n[1] Wang Y, Blei D M. The blessings of multiple causes[J]. Journal of the American Statistical Association, 2019, 114(528): 1574-1596.\\\n[2] D’Amour A. Comment: Reflections on the deconfounder[J]. Journal of the American Statistical Association, 2019, 114(528): 1597-1601.\\\n[3] Ogburn E L, Shpitser I, Tchetgen E J T. Comment on “blessings of multiple causes”[J]. Journal of the American Statistical Association, 2019, 114(528): 1611-1615.\\\n[4] Yunzhe Zhou, Zhengling Qi, Chengchun Shi, and Lexin Li. Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach. In International Conference on Artificial Intelligence and Statistics, pages 6704–6721. PMLR, 2023.\\\n[5] Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkUqflup0G", "reviewer": "Reviewer_7kZ2", "summary": "This paper studies the problem of confounded OPE. The authors explore a new structural assumption of the data-generating process called the two-way confounding assumption. They also propose an algorithm called the two-way deconfounder that can deal with the new setting they consider. The authors perform theoretical and empirical analyses to validate their proposed assumptions and algorithms.", "questions": "* In the literature of model-based RL, the transition dynamic can be represented with some kind of generative model (for example, [1]). For the two-way confounder algorithm, is it possible to parameterize $P$ as some generative model and thus eliminate the conditional Gaussian restriction?\n\n* Given different confounding assumptions (unconstrained/two-way/one-way confounding), how can we determine which setting fits best with the problem if we do not have full prior knowledge of the data-generating process? Do there exist any hypothesis testing procedures or simple heuristic methods? Can the authors give a short discussion?\n\n* (Minor) What does the term \"relational neural network\" mean?  \n\n[1] Buesing, Lars, et al. \"Learning and querying fast generative models for reinforcement learning.\" arXiv preprint arXiv:1802.03006 (2018).", "limitations": "The authors have addressed the limitations of their paper in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of two-way confounding is interesting and well-motivated. This paper is the first to introduce the two-way confounder assumption into the field of causal RL.\n\n* The two-way deconfounder algorithm is simple and intuitive. And it can deal with challenging causal RL problems with modern functional approximation techniques.\n\n* The authors perform solid theoretical analysis and empirical analysis to evaluate their methodology. The theoretical results include a non-asymptotic error bound of the two-way deconfounder estimator. I checked the proof and found the result is reliable. In the empirical analysis, the authors test their method in both simulated and real-world RL scenarios. They also compare their method with various baselines.", "weaknesses": "* I suggest the author add more explanations about the problem setting to the intro section, making it more readable for readers outside the field of causal inference.\n\n* The algorithm imposes additional modeling restrictions to the MDP by parameterizing $P$ as conditional Gaussians.\n\n* (Minor) It seems that some notations may denote both a variable or a set of variables, which is a little bit confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- __Enhancing clarity of the problem setting in the Introduction.__\\\nThank you for your valuable suggestion. We will include additional explanations about the problem setting in the introduction section of the final version.\n- __Clarifying confusing notations.__\\\nWe apologize for any confusion caused by the misused notations. We will correct these issues in the final version.\n- __Can the transition dynamics be represented using a generative model instead of a conditional Gaussian distribution?__\\\nThis is a good point. We did assume the transition function $\\mathcal{P}$ follows a conditional Gaussian distribution in our implementation. This assumption is common employed in RL, as seen in papers such as [1] and [2]. As you mentioned, it is also possible to parameterize the transition function using a generative model, such as mixture density networks, generative adversarial networks or diffusion models. This alternative modeling approach would not affect our theoretical results, as our theory does not require the conditional Gaussianity assumption. We will add these discussions in the final paper.\n- __How can we determine the best-fitting setting for the problem without prior knowledge of the data-generating process?__\\\nWe greatly appreciate this excellent comment. Our paper covers four confounding assumptions, corresponding to no unmeasured confounding, one-way unmeasured confounding, two-way unmeasured confounding and unconstrained unmeasured confounding. In practice, we may consider the following sequential testing procedure to infer the unknown confounding structure and select the most suitable confounding assumption: \n\n   * **Step 1: Initial testing for the Markov property**. At the first step, we use state-of-the-art test procedures, such as [3], to determine whether the original data satisfies the Markov property. If the null hypothesis is not rejected, we do not have sufficient evidence to believe there are unmeasured confounders. Thus, we stop the procedure and conclude the data is likely does not contain unmeasured confounders. If the null hypothesis is rejected, it suggests the presence of some unmeasured confounders and we proceed to Step 2.\n   * **Step 2: Testing for one-way unmeasured confounding**. We next assume the data contains one-way confounders, estimate them from the data, include the estimators in the observations, and perform the Markov test again using the transformed data. If the null hypothesis is not rejected, we stop the procedure and conclude that the one-way unmeasured confounding assumption is likely to hold. Otherwise, we proceed to Step 3. \n   * **Step 3: Testing for two-way unmeasured confounding**: Finally, we impose the two-way confounding assumption, estimate these confounders from the data and apply the Markov test to the transformed data with estimated two-way confounders incorporated in the observations. If the null hypothesis is not rejected, we stop the procedure and conclude that the two-way unmeasured confounding assumption is likely to hold. Otherwise, we conclude that the unconstrained unmeasured confounding assumption is likely satisfied. \n\n  We will add the related discussions shall our paper be accepted. \n- __Clarification on relational neural network.__\\\nThe term \"relational neural network\" has the same meaning as \"neural tensor network\" [4] mentioned on line 173, known for its ability to capture the intricate interactions between pairs of entity vectors. We apologize for the confusion and will use \"neural tensor network\" consistently shall our paper be accepted.\n- __References.__\\\n[1] Yu T, Thomas G, Yu L, et al. Mopo: Model-based offline policy optimization[J]. Advances in Neural Information Processing Systems, 2020, 33: 14129-14142.\\\n[2] Janner M, Fu J, Zhang M, et al. When to trust your model: Model-based policy optimization[J]. Advances in neural information processing systems, 2019, 32.\\\n[3] Shi, C., Wan, R., Song, R., Lu, W. and Leng, L., 2020, November. Does the Markov decision process fit the data: Testing for the Markov property in sequential decision making. In International Conference on Machine Learning (pp. 8807-8817). PMLR.\\\n[4] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NwNQH5SRu", "reviewer": "Reviewer_GM6x", "summary": "This work proposes a novel technique for performing off-policy evaluation (OPE) in the presence of unobserved confounders that have been classified as \"two-way\" unmeasured confounders, viz., by assuming that there exist both time-invariant and trajectory-invariant confounders, but not time-and-trajectory-invariant confounders. Roughly outlined, their approach is to first use a network architecture that takes embeddings of these different confounder types, uses a neural tensor network to first obtain estimates of transitions from a transition network and actions from an actor network, and then performs OPE in standard Monte Carlo simulation by estimating the expected cumulative reward from many simulated trajectories. Experiments compare their TWD technique's LMSE and LMAE against a host of traditional and modern techniques that span numerous means of dealing with confounding from traditional model-based methods to POMDPs to TWDIDPs. Across several simulation studies using principled data generation (like the PK-PD model for tumor growth generation), the TWD technique is shown to be the most robust and obtain lower LMSE and LMAE compared to the others compared. Ablation studies following these show how the sum of the TWD components are needed to obtain maximal accuracy.", "questions": "** Trivial Points **\n- Although it has its own section explaining it, two-way unmeasured confounding appears in the introduction without even a short explanation of its meaning; it would aid the clarity if it had a brief, intuitive lead before it gets referenced in the text.\n- [39] \"However, these works require restrict mathematical assumptions that are hard to verify...\" -- was \"restrict\" meant to be \"strict\" or \"restricting\"?\n- [117] \"...which pluggs in the estimated latent factors...\" -- typo?\n- [769] \"A illustrative example...\" -- typo\n- [Eqn. 1] Why is the subscript on O_{i,1} rather than O_{i,t}?\n- [236] \"Assumptions 4 is concerned...\" -- typo\n\n** Substantive Questions **\n- [102] \"Our objective lies in evaluating the expected cumulative reward under a given target policy π, which depends solely on the observation and not on the latent confounders.\" The phrasing could be a bit improved here -- one read is that the expected cumulative reward depends solely on the observation and not the latent confounders, but your transition model in the previous paragraph makes it clear that this is not the case. You mean to say that the evaluation itself must be only a function of the observations since we do not have direct access to the UCs?\n- [Figure 1a] What's the significance of the edges emanating from Z_{i,t}? Just to show that OWUC vs TWUC are different ways of modeling the more general unconstrained UC space?\n- One part that you might want to develop a bit more: the intended application of these techniques -- is this primarily suited for artificially generated policies or those employed by human decision-makers like given in the examples?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The review of related work is particularly rich; the authors do a tremendous job at finding recent and relevant work related to the problem space and the various viewpoints in approaching deconfounding.\n- The writing is exceptionally clear, with well-stated assumptions, defined variables, and illustrations of process in each figure.\n- Classifying the types of transience to which UCs belong is interesting and exploited in a nice way within the TWD technique.\n- I'm a fan of the idea of performing MC simulation using the learned transition network to accomplish the OPE. I found the elaboration on line [205] to be particularly compelling.\n- Compared strategies in Section 5 are comprehensive and recent.", "weaknesses": "- [197] One minor point that likely deserves some consideration in general: the causal structure of the system, viz., that certain contexts O should *not* be included in evidence while estimating \\eta^\\pi -- for example, considering the classic confounding M structure (see Cinelli, Forney, & Pearl, 2022), there are cases where an observed context must be omitted from conditioned evidence to obtain an unbiased estimate.\n- [5.1] While I completely agree with the risks of ignoring UCs and that the NUC assumption is usually made for convenience rather than realism, would it not also be fair to run an experiment in an environment *without* UCs to see if there is some tax paid by this paper's technique (which seems to *assume* their presence)? It's possible that looking for confounding where none exists comes with a price that the experiments herein should advertise if present. Still, I think the authors did a good job of putting their technique through its paces within the contexts of Section 5, especially the ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- __Omitting observations for avoiding biases.__\\\nThis is an excellent comment! We have also carefully read [1] that you mentioned and will include it as well as our discussions in our paper, shall it be accepted. This paper discusses whether an additional random variable $Z$ should be included in the regression equation for estimating the average causal effect (ACE) of a treatment $X$ on an outcome $Y$ in causal inference. \n\n  In our current problem, $\\eta^{\\pi}$ represents the expected cumulative reward under a target policy $\\pi$. To  draw a parallel with the causal inference problem, our action $A_{i,t}$ corresponds to the treatment $X$, the expected reward $R_{i,t}$ and the next observation $O_{i,t+1}$ correspond to the outcome $Y$, and the initial observation $O_{i,1}$ corresponds to $Z$. In this case, the causal relationship among $O_{i,1}$, $A_{i,t}$, and $(R_{i,t}, O_{i,t+1})$ aligns with \"Model 1\" in [1], where $A_{i,t} \\rightarrow (R_{i,t}, O_{i,t+1})$ and $A_{i,t} \\leftarrow O_{i,1} \\rightarrow R_{i,t}$. Under these circumstances, controlling for $O_{i,1}$ or including it as an independent variable in the model is beneficial because it blocks the back-door paths. Therefore, when estimating $\\eta^{\\pi}$, there is no need to omit the initial observation $O_{i,1}$.\n- __Settings without unmeasured confounders.__\\\nThis is another excellent comment. We fully agree with the importance to investigate the tax of the proposed two-way deconfounder in settings without unmeasured confounders. Indeed, our sensitivity analysis reported in Section 5.3 compares different algorithms with different degrees of unmeasured confounding, characterized by a sensitivity parameter $\\Gamma$. A large $\\Gamma$ indicates a stronger effect of unmeasured confounding. When $\\Gamma$ decays to zero, no unmeasured confounders exist. As shown in Figure 5:\n  - When $\\Gamma=0$ where there are no unmeasured confounders, the proposed two-way deconfounder (TWD) loses its superiority when compared to standard algorithms that ignore unmeasured confounders, such as MB and TWDIDP2. This demonstrates the prices of looking for confounding where none exists. \n  - When $\\Gamma=0.3$ where the effects unmeasured confounding are weak, TWD achieves similar performance to MB and TWDIDP2.\n  - Finally, when $\\Gamma>0.3$, the effects unmeasured confounding becomes (moderately) strong. As a result, TWD achieves the lowest LMSE and LMAE.\n\n  Inspired by Reviewer 7kZ2, we have outlined a hypothesis testing procedure to test the degree of unmeasured confounding, in order to determine the best algorithm to use. This hybrid procedure is expected to enhance the robustness of the TWD performance across diverse settings.\n- __Minor comments.__\n  - We apologize for the typos and will correct them. \"restrict\" was meant to be \"restrictive\". \n  - Shall our paper be accepted, we will add a short explanation of the proposed two-way unmeasured confounding in the introduction.\n  - In Equation (1), the conditioning set contains variables that are not affected by policies, such as the initial observation. Inclusion of $O_{i,t}$ is problematic because these observations are influenced by the behavior policy, while Equation (1) is intended to compute the expectation under the target policy. \n- __Clarification of the dependence on the latent confounders.__\\\nYou are absolutely correct. The immediate reward indeed depends on the latent confounders, but the target policy we wish to evaluate is irrelevant of these confounders. We plan to adopt the following changes to make this point clearer:\n  - We will rephrase this sentence to emphasize that it is the target policy rather than the immediate reward that does not depend on latent confounders.\n  - Motivated by the comments from Reviewer P4Ev, we will include visualizations of the data generating process of the offline data and that under the  to further illustrate this point; see Figure 2 in the PDF file.\n- __Clarification on Figure 1(a).__\\\nThank you for the constructive comment! The edges indicate that both OWUC and TWUC are special cases of UUC. Specifically, the edges from $\\\\{Z_{i,1}\\\\}\\_{i}$ to $\\\\{H_{i}\\\\}\\_{i}$ indicate that the one-way unmeasured confounders can be viewed as special cases of unconstrained unmeasured confounders that remain the same over time. Similarly, the edges from $\\\\{Z_{i,t}\\\\}\\_{i,t}$ to $\\\\{U_{i}\\\\}\\_{i}$ and $\\\\{W_{t}\\\\}\\_{t}$ indicate that the two-way unmeasured confounders can be viewed as special cases of unconstrained unmeasured confounders that do not involve either over time or across trajectories. \n- __The intended application.__\\\nThank you for your suggestion. Should our paper be accepted, we will further elaborate on the intended application of our method, which is primarily designed for settings involving human decision-makers. In these contexts, decision-makers often rely on critical but unrecorded information when taking actions, leading to confounded datasets. For example, in healthcare or contexts similar to our case study, doctors frequently use visual observations or patient interactions to inform treatment decisions. Such unstructured data can be challenging to quantify and are often omitted  [2]. Similarly, in technology companies where behavior policies involve human interventions, the data can also become confounded [3].\n- __References.__\\\n[1] Cinelli C, Forney A, Pearl J. A crash course in good and bad controls[J]. Sociological Methods & Research, 2022: 00491241221099552.\\\n[2] McDonald, C. J. (1996). Medical heuristics: the silent adjudicators of clinical practice.\\\n[3] Shi, C., Zhu, J., Shen, Y., Luo, S., Zhu, H., & Song, R. (2024). Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119(545), 273-284.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gEU1x2dVnP", "reviewer": "Reviewer_P4Ev", "summary": "This paper considers a setting in which there are observed tuples $(O_{i,t}, A_{i, t}, R_{i, t})$ with $ 1\\leq i \\leq N$, $1 \\leq t \\leq T$; \nHere, $O_{i,t}$ represents observable quantities, $A_{i,t}$ is an action generated by a behavioural policy, and $R_{i,t}$ is the associated reward. \nThe goal of this paper is off-policy evaluation; specifically, the authors aim to estimate the expected cumulative reward under a given policy that differs from the behavioural policy. \n\nThe authors propose a generative model of the observed trajectories. The specific contribution of this work is to consider a two-way confounder, which as I understand a tuple of latent variables $(U_i, W_t)$ representing subject-and time-varying components, respectively.", "questions": "NA", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed two-way confounding model can be considered original. The authors considers separating a confounder into a time- and subject-invariant parts, and model their interaction by a neural network. This may be considered as a low-rank approximation (analogously to nonlinear matrix-factorisation), and reduces the number of the latent variables from $NT$ to $N + T$. \n\nThe empirical evaluation shows that the proposed method performs well. The authors also experimentally demonstrates the sensitivity to the two-way confounder assumption.", "weaknesses": "# Clarity \n\n## Major issues: \n1. Line 104: It is unclear what $\\mathbb{E}^\\pi$ denotes. If the expectation is simply taken with respect to the dynamics generated by $\\pi$, doesn't this have an effect from the confounders? \n2. Introducing the estimator (1) before the model would help the reader to understand the quantities to be estimated and hence the model. \n3. The assumption on the data generating process is not straightforward to understand. In the unconstrained unmeasured confounding (UUC) setting, the trajectories cannot be independent, and the data assumption (Line 89) alone  already excludes the UUC setting. At the same time, for the proposed TWUC setting, the presence of the subject-invariant latent $W_t$ renders the trajectories dependent on each other. Using a graphical model would help clarify the assumption. \n\n## Minor issues: \n1. The $\\cup$ notation is undefined. \n2. Line 187: Gaussian Mixture Model. The transition kernel does not look like a mixture. Why the term? \n3. L 287: forr\n\n# Theoretical justification \nWhile I realise that the focus of the paper is on the empirical performance, the supporting theory is relatively weak. \n\n1. The validity of the estimator (1) is claimed, but there is not reference to this claim. Moreover, the expectation needs to be estimated, and calling (1) an estimator sound odd to me. \n2. I have not been able to read though the proof, but the first paragraph already casts some doubt. Why does estimand $\\eta$ depend on the latent and why are they not marginalised? (this might explain what I don't understand about the assumption). It seems that the only variability comes from the initial observation rather than the whole trajectories (since the confounders are fixed). The consistency result therefore has limited reliability,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- __Clarification of the DGP.__\\\nWe agree with your points and will replace \"independent trajectories\" with \"trajectories\". \"Independent\" was used to indicate conditional independence between trajectories given latent two-way fixed effects, but this was confusing. Following your suggestion, we use different use different graphical models to clarify DGP under different assumptions in Figure 1 of the PDF file. We will include this figure if our paper is accepted. \n- __$\\eta^\\pi$'s dependence on latent variables and proof sketch.__\\\nTo validate our theory and save your time, we provide an outline of our proof below. We first clarify $\\eta^\\pi$'s dependence on latent variables:\n  1. We assume two-way unmeasured confounders are fixed, aligning with existing literature on two-way fixed effects regression. \n  2. While latent factors $U_i$s and $W_t$s are considered fixed, not all observations $O_{i,t}$s are fixed. Thus, **the variability of $R_{i,t}$ comes from the initial observation and all subsequent intermediate observations along the trajectory**, generated randomly by the transition function.\n  3. The estimation error of $\\eta^\\pi$ arises from variability in the initial observation, estimated transition dynamics, latent factors, and distributional shift between behavior and target policy, as highlighted in the first two terms of our error bound in Theorem 1.\n \n  Meanwhile, our theory can also accommodate random two-way unmeasured confounders by bounding the deviation between $(NT)^{-1}\\sum_{i,t} \\mathbb{E}^\\pi[R_{i,t}|U_i,\\\\{W_{t'}\\\\}\\_{t'=1}^t]$ and $T^{-1}\\sum_t \\mathbb{E}^\\pi[R_{1,t}]$. This deviation can be decomposed into two terms:\n  $$\\Big[\\frac{1}{NT}\\sum_{i,t} \\mathbb{E}^\\pi[R_{i,t}|U_i,\\\\{W_{t'}\\\\}\\_{t'=1}^t]-\\frac{1}{T}\\sum_t \\mathbb{E}^\\pi[R_{1,t}|\\\\{W_{t'}\\\\}\\_{t'=1}^t]\\Big]+\\Big[\\frac{1}{T}\\sum_t\\mathbb{E}^\\pi[R_{1,t}|\\\\{W_{t'}\\\\}\\_{t'=1}^t]-\\frac{1}{T}\\sum_t\\mathbb{E}^\\pi[R_{1,t}]\\Big].$$\n  Assuming $U_i$s are i.i.d. and independent from $W_t$s, we apply Hoeffding's inequality to the first term, resulting in an order of $R_{max}\\sqrt{2N^{-1}\\log (2/\\delta)}$ with probability at least $1-\\delta$. The second term requires assuming that the time series exhibits certain mixing properties (e.g., $\\beta$-mixing), allowing us to treat each $R_{1,t}$ as dependent only on recent $W_{t'}$s. Using Berbee’s coupling lemma (Berbee 1987) and approximating this term with sums of i.i.d. random variables, we apply Hoeffding's inequality again to establish the tail inequality for the second term. We are happy to revise our theory and proof to incorporate these changes if our paper is accpeted. \n\n  Finally, to facilitate understanding of Theorem 1, we provide a _proof sketch_. We decompose $|\\hat{\\eta}^\\pi - \\eta^\\pi|$ into two terms, $I_1$ and $I_2$. $I_2$ represents the absolute difference between the mean of the value function and its expectation, given the latent confounders and transition function are true values. The only difference between the two terms is the expectation over the initial state. Based on the boundedness of the value function (Assumption 2) and the conditional independence of each trajectory given all latent confounders, we can apply Hoeffding's inequality. This gives $I_2$ an order of $R_{max}\\sqrt{2N^{-1}\\log (2/\\delta)}$ with probability at least $1-\\delta$. $I_1$ represents the absolute difference between the estimator $\\hat{\\eta}^{\\pi}$ (the mean of the estimated value function) and the mean of the true value function. Using the Bellman equation, we transform the difference into a function of differences between the estimated and true values of the transition function, i.e., a function of $\\operatorname{TV}(\\hat{\\mathcal{P}}(\\bullet|a_t,o_t,\\hat{u}\\_i,\\hat{w}\\_t)-\\mathcal{P}(\\bullet|a_t,o_t,u_i,w_t))$. Furthermore, the function of $\\operatorname{TV}(\\hat{\\mathcal{P}}(\\bullet|a_t,o_t,\\hat{u}\\_i,\\hat{w}\\_t)-\\mathcal{P}(\\bullet|a_t,o_t,u_i,w_t))$ can be decomposed into the sum of the functions of $\\operatorname{TV}(\\hat{\\mathcal{P}}(\\bullet|a_t,o_t,\\hat{u}\\_i,\\hat{w}\\_t)-\\hat{\\mathcal{P}}(\\bullet|a_t,o_t,u_i,w_t))$ (denoted as $I_3$) and $\\operatorname{TV}(\\hat{\\mathcal{P}}(\\bullet|a_t,o_t,u_i,w_t)-\\mathcal{P}(\\bullet|a_t,o_t,u_i,w_t))$ (denoted as $I_4$). Based on Assumption 4 and the Lipschitz continuity of the neural network, we can determine that the order of $I_3$ is $\\varepsilon_{U,W,\\delta}$ with probability at least $1-\\delta$. Finally, by leveraging Assumptions 1 and 3, we find that the order of $I_4$ is $\\varepsilon_{\\mathcal{P},\\delta}$ with probability at least $1-\\delta$. Combining this with all previously derived conclusions, we can then establish the order of $|\\hat{\\eta}^\\pi-\\eta^\\pi|$ with probability at least $1-3\\delta$.\n\n  \n  \n- __Equation (1).__\n  - __Is (1) an estimator?__: As you commented, (1) is not our final estimator, since the expectation needs to be approximated based on the Monte Carlo method detailed on Page 5, Lines 205 to 210. It serves as an **intermediate estimator** that is unbiased to the evaluation target (see our justification of the unbiasedness in the next response). We will make this clear to avoid potential confusion.\n  - __Validity of (1)__: We claimed Equation (1) is unbiased to $\\eta^\\pi$ in Line 200. Demonstrating this unbiasedness is straightforward: Recall that $\\eta^\\pi$ is the average expected reward across $N$ individuals over $T$ time points. Applying the law of total expectation, it can be expressed as $(NT)^{-1}\\sum_{i,t}\\mathbb{E}^\\pi[R_{i,t}|O_{i,1},U_i,\\\\{W_{t'}\\\\}\\_{t'=1}^t]$, which equals Equation (1).\n  - __Introducing (1) earlier__: Following your suggestion, we will introduce (1) before the model if our paper is accepted. \n- __Minor issues.__\\\nWe apologize for the typos. In line 126, we will revise $Z_{i,t}=U_i\\cup W_t$ to $Z_{i,t}=(U_i^\\top,W_t^\\top)^\\top$, to avoid using the undefined notation $\\cup$. Additionally, we will replace 'Gaussian Mixture Model' with 'Gaussian Model' in line 187.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all referees for their valuable and insightful comments. We have addressed all your comments and will incorporate them shall our paper be accepted; please refer to our detailed responses to your review. Below, we would like to briefly clarify the notation $\mathbb{E}^{\pi}$ and our data generating process, in response to the comments raised by Reviewers GM6x and P4Ev.
 
The notation $\mathbb{E}^{\pi}[R_{i,t}]$ represents the expectation under the **interventional distribution** of $R_{i,t}$, assuming all actions are generated solely by the target policy $\pi$, irrespective of the unmeasured confounders. In other words, whatever relationship exists between the unmeasured confounders and the actions in the offline data, that relationship is **no longer in effect** when we perform the intervention according to $\pi$. More specifically, the **interventional distribution** of $R_{i,t}$ can be described as follows:
  - The initial observation is generated according to $\rho_0$;
  - At each time $t$, the action $A_{i,t}$ is determined by the target policy $\pi(\bullet|O_{i,t})$, independent of the unmeasured confounder $Z_{i,t}$;
  - The immediate reward $R_{i,t}$ and next observation $O_{i,t+1}$ are generated according to the transition function $\mathcal{P}(\bullet|A_{i,t},O_{i,t},Z_{i,t})$.  

  In this setup, the unmeasured confounders affect only the reward and next observation distributions, but not the action. This differs from the offline data generating process; see Figure 2 in the PDF file for an illustration. Alternative to $\mathbb{E}^{\pi}$, we can adopt Pearl's do-operator to further clarify the expectation, shall our paper be accepted.

Meta-review: This manuscript concerns off-policy evaluation in the presence of unmeasured confounders. To this end, the authors apply causal reinforcement learning to develop a "deconfounder" to effectively learn the confounding effects and the desired policy value.

The reviewers were broadly initially impressed by the quality of this work and this sentiment was only strengthened during the author-reviewer discussion period. A vigorous discussion among the reviewers ensued following the author-reviewer discussion period. The outcome of this discussion was a consensus that there was sufficient novelty and potential interest in the problems posed by this paper, as well as the proposed solutions. There was some concern that the initial results derived here may not be the strongest possible results, but this sentiment is mitigated by opening up a promising avenue of research.
Decision: Accept (poster)