An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
Authors: Emil Javurek
Valentyn Melnychuk
Jonas Schweisthal
Konstantin Hess
Dennis Frauen
Stefan Feuerriegel
Venue: ICLR 2026 Poster (2026)
Keywords: Causal Machine Learning
Doubly Robust Estimation
Neyman-Orthogonality
Markov Decision Process
Primary area: causal reasoning

Abstract: Predicting individualized potential outcomes in sequential decision-making is central
 for optimizing therapeutic decisions in personalized medicine (e.g., which
dosing sequence to give to a cancer patient). However, predicting potential out-
comes over long horizons is notoriously difficult. Existing methods that break the
curse of the horizon typically lack strong theoretical guarantees such as orthogonality
 and quasi-oracle efficiency. In this paper, we revisit the problem of predicting
 individualized potential outcomes in sequential decision-making (i.e., estimating
 Q-functions in Markov decision processes with observational data) through a
causal inference lens. In particular, we develop a comprehensive theoretical foundation
 for meta-learners in this setting with a focus on beneficial theoretical properties.
 As a result, we yield a novel meta-learner called DRQ-learner and establish
that it is: (1) doubly robust (i.e., valid inference under model misspecification),
(2) Neyman-orthogonal (i.e., insensitive to first-order estimation errors in the nuisance
 functions), and (3) achieves quasi-oracle efficiency (i.e., behaves asymptotically
 as if the ground-truth nuisance functions were known). Our DRQ-learner is
applicable to settings with both discrete and continuous state spaces. Further, our
DRQ-learner is flexible and can be used together with arbitrary machine learning
 models (e.g., neural networks). We validate our theoretical results through
numerical experiments, thereby showing that our meta-learner outperforms state-of-the-art baselines.

Published as a conference paper at ICLR 2026
AN ORTHOGONAL LEARNER FOR INDIVIDUALIZED OUTCOMES IN MARKOV DECISION PROCESSES
Emil Javurek
Valentyn Melnychuk LMU Munich & MCML me 1 nychuk@ 1mu de
Jonas Schweisthal
LMU Munich & MCML emi 1 javureke @1 mm. de
LMU Munich & MCML
Jonas schwei st 1 1mu de
Konstantin Hess
Dennis Frauen LMU Munich & MCML frauen lmu. de
Stefan Feuerriegel
LMU Munich & MCML hess@ 1mu.
LMU Munich & MCML
feuerriegel de
ABSTRACT
Predicting individualized potential outcomes in sequential decision-making is cen- tral for optimizing therapeutic decisions in personalized medicine (e.g., which dosing sequence to give cancer patient). However, predicting potential out- comes over long horizons is notoriously difficult. Existing methods that break the curse of the horizon typically lack strong theoretical guarantees such orthogo- nality and quasi-oracle efficiency. In this paper, we revisit the problem of predict- ing individualized potential outcomes in sequential decision manning (i.e., estimat- ing Q-functions in Markov decision processes with observational data) through a causal inference lens. In particular, we develop a comprehensive theoretical foun- dation for meta-learner in this setting with focus beneficial theoretical prop- erties. As a result, we yield a novel meta-learner called DRQ-learner and establish that it is: (1) doubly robust (i.e., valid inference under the misspecification of one of the models), (2) Neymmn-orthogonal (i.e., insensitive to first-order estimation errors in the nuisance functions), and (3) achieves quasi-oracle efficiency (i.e., behaves asymptotically as if the ground-truth nuisance functions were known). Our DRQ-learner is applicable to settings with both discrete and continuous state spaces. Further, our DRQ-leamer is flexible and can be used together with arbi- trary machine learning models (e.g.. neural networks). We validate our theoretical results through numerical experiments, therehy showing that our meta-learner out- performs ffateoof-the-ar baselines.
INTRODUCTION
Predicting individualized potential outcomes sequential decision-making is central for optimizing
therapeutic decisions in personalized medicine (Feuerriegel et al. 2024). Typical examples
selecting dosage schedules for cancer patients (Zhao et al. 2009 Wang et al. 2012).. scheduling
just-in-time interventions in digital health (Liao et al. 2021 Battalio et al. (021).
determining
treatment schedules for chronic diseases (Shortreed et al. 2011 Matsouaka et al. (014). In recent
years, this problem has been increasingly studied using observational data (c.g.. electronic health records) to avoid "exploration" and leverage the increasing availability of digital patient data (Allam)
al. 202T Bica al. 2021).
Here, we focus predicting individualized potential outcomes in Markov decision processes (MDPs), i.e., estimating the Q-function from observational data. This task has received much atten- tion in off-policy reinforcement learning (c.g.. Liu et al.) 2018 Le et al. 2019 Uehara et al. 2020). where many approaches have focused on delivering new learners, with focus on addressing the curse of horizon. However, comparatively little attention has been given to developing methods in principled way with theoretical guarantees such as orthogonality or quasi-oracle efficiency.
"corresponding author

---

Published as a conference paper at ICLR 2026
Infernce
DRQ--eamer
Markow Decision Processes
Orihogonal Slutisticil
brselise
robuslan
donbly cobust
a Qusction en colimation
orthogmaliyy
Seymat-orthigml
di polcy daia
affictency
quaei crach offiienn
cure of bottzon
Figure 1: Our work located at the intersection of 1 causal inference & orthogonal statistical learning and 2 MDPs. Our problem setup is in 2: we estimate Q-functions in MDPs from off-policy data. Baselines for this task break the curse of the horizon but typically lack strong theoretical guarantees. Our method adopts concepts from D we obtain a novel meta-leamer called DRQ--eamee that is doubly robust, Neyhon-orthogonal, and quasi-oracle efficient.
In this paper, we study the problem of estimating Q-functions in MDPs from observation data through the theoretical lens of causal inference. In particular, we develop a theoretical foundation based on statistical orthogonality theory (Foster& & Syrgkamis 2019), which offers a novel perspec- tive on this task (see Figure IT. For this, we first derive identifiaility results and show that several of the existing baselines correspond to naïve plug-in in learners, which are known to be biased. As remedy, we next derive the efficient influence function of the training loss and use it to construct a debiased second-stage loss that is Neyman-orthogonal
As result, we obtain novel meta-learner for this task, which we call DRQ-learner. The DRQ-learner enjoys several favorable theoretical properties: (1) it is doubly robust, which enables valid inference even under model misspecification; (2) is Neyman-orthogoal, which makes it in- sensitive to first-order estimation errors in the nuisance functions; and
(3) it achieves quasi-oracle
efficiency, meaning it attains the same asymptotic performance as if the ground-trui -ruth nuisance func- tions were known. The DRQ-learner is applicable to settings with both discrete and continuous state spaces. Moreover, the DRQ-learner is flexible and can be used together with arbitrary machine learning models such as neural networks.
Our contributions are three-fold
New theoretical contributions. We provide theoretical framework of causal inference Q- function estimation in MDPs. While causal inference has long been used to address statistical challenges treatment effect estimation from observational data, we extend these ideas for- malize and solve the challenges of estimating Q-functions from observational data. In this setting, interventions induce a distributional shift between behavior and evaluation policies; al- though inverse propensity weighting (IPW) can address this, IPW suffers from exponentially decaying overlap in sequential settings (i.e., the curse of horizon), leading to instability from division by near-zero probabilities and making consistent estimation of potential outcomes im- possible. By leveraging statistical orthogonality theory, we derive a novel meta-learner for valid inference with favorable statistical properties.
New method. We propose the first meta-learner for Q-function estimation that is simulatenously (i) doubly robust, (ii) ods that rely, for example, on IPW and are thus Neyman-orthogonal but fail break the curse of horizon; the DRQ--earmer avoids this issue and achieves all three properties while still ad- dressing the curse of the horizon. Importantly, quasi-oracle efficiency of our method guarantees convergence at the same rate if oracle nuisance functions were known. We thereby aim to make an important contribution to reliable inference in personalized medicine where strong theoretical guarantees are important.
and quasi-oracle efficient. Hence, this is unlike meth-
Empirical performance. The primary objective of our numerical experiments is validate our theoretical results. Hence, we various numerical experiments and show that the DRQ-learner is especially effective for low overlap settings line with our theory. Overall, our results demonstrate state-of-the-arr empirical performance.
Coode http://////thhhhhoooo//aaaaaaaaa
MDPP
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
We group our literature review along streams that are relevant: (1) We review theoretical foundations from causal inference and orthogonal statistical learning to motivate our method, and (2) discuss prior work off-policy Q-function estimation in MDPs. The latter defines our problem setup, while the former shares parallels in terms of the overall methodological approach to formalize causal quantities. We provide an extended literature review in Appendix
Causal inference and orthogonal learning: Both the theory of orthogonal statistical learning (Foster& & Syrgkanis, 2019) and semiparametrie efficiency theory (van der Vaart 1998) have been widely used to construct estimators with strong theoretical properties. Here, a particular focus is on Clltence-fucction-eee
estimators (Kennedy] 2022), with well-known examples such targeted maximum likelihood estimation (TMLE) (Daniel Rubin] 2006). the DoubleMI. framework (Cher- nozhukov et al., 2018), and doubly robust approaches for off-policy policy value estimation (Kallus & Uchara 2022 Shi et al. (02T). These techniques have been extended to the estimation of individ- ualized treatment effects (Foster & Syrgkanis 2019), leading to broad class of orthogonal meta- leamnrrs (Kennedy [020] Nie & Wager [021 Morzywolek et al. 2023). Similarly, meta-learnen, have been proposed for estimating individualized treatment effect estimation over time (Frauen et al.) 2025). However, works on individualized treatment effect estimation over time do not focus on the MDP setting and are well to known to suffer from the curse of horizon (Kallus & Uchara 2022). Im- portantly, similar theoretical framework for individualized potential outcome estimation in MDPs is still missing.
Several methods have been developed for estimating Q- ffnccion from MDPs in off- offlcol settings, that is, using observational data (e.g. Liu et al., 2018 Le et al.]
Off-policy Q-function evaluation:
2019 Uchara et al. 2020). A common theme in these works is to address the curse of horizon (e.g.,
Le et al.
2019 Uehara et al. 2020). We refer to Appendix A for a more detailed overview?
The above works have been developed typically outside of causal inference and thus without plicitly formalizing the underlying estimand as a causal quantity. One of our contributions is to link causal inference and Q-function evaluation from observational data by formalizing the underlying causal estimand. This allows us later to taxonomize prominent works from the literature based on the underlying adjustment strategy. For example, in our framework, existing works correspond to adjustment strategies based on inverse- inverse-propensity-weeee propensity-wiighingg weighting-ii like nuisances (c.g., Q-regression (Liu al., 2018) or implicit adjustment strategies based on (supervised learning) target construction (e.g., FQE (Le et al. (019)). From our causal inference perspective, we later obtain new theoretical insights to understand the failing modes of existing methods. In particular, we show that several ofate-of-the-art methods suffer from so-called plug-in bias (Kennedy (022) and potential instability under model misspecification. To the best of our knowledge, more advanced adjustment strategies, which commonly used in causal inference, missing in the literature on Q Q-function evalua- tion. Consequently, no prior work has developed a Neyman-orthogonn meta-learner for off-policy Q-function estimation.
To the best of our knowledge, a method for Q-function evaluation in MDPs with observational data that enjoys favorable theoretical properties such Neyman-orthogonal and quasi-oracle efficiency is missing. As remedy, we first reframe off-policy Q-function evaluation through the lens of causal inference and then develop new meta-learner called DRQ-learner.
Research gap:
3 PROBLEM FORMULATION
Notation: We denote random variables by capital letters S,A, R and their realizations by small letters 8, a, r from domains S, A, R. Let (P(S) denote a distribution of some random variable S, and let p(S s) be a corresponding density or probability mass function, and let P(S) denote the set of all probability distributions over S. We write E, == Ep, [-] f -dP, to denote expectation with respect to a distribution P,, arising from a stochastic process created by following the MDP with policy π (equivalently, E[- when r ~ P).
Many of these works focus on off- policy evaluation (and thus target scalar average outcomes), where methods for Q-function evaluation often necessary first step (e.g.. Shi et al. (202T) propose a method for interval estimation that yields a Q-function evaluation as byproduct)
3

---

Published as a conference paper at ICLR 2026
Intoricall palcy
cvaluative pulcy
R2
A4
Aa
R4
R
Dr: (Se, Au, Ro, ..,, (So, Ac, Ra Ro....-a.... .)....)
eatimade
(8,, az) - B, tazget aettound
So 50,
Ag
4g
chserntioonal dstruse
Our task: we aim to estimate Qa,, a functional of the unobserved evaluation policy Wa (right), from the observational dataset Dzy from the behavioral policy To (left). A trajectory from time- invariant Markov decision process (MDP) determined by environment dynamics (gray) and by selecting actions according to a policy. We observe the MDP with my (top left), while a potential MDP with mee (top right) is unobserved. Our target estimand Q @T. must thus be estimated from available observational data DT.-
Figure 2:
Data-generating process: We consider the following definition of time-invariant MDP, as is com- mon in the literature and in many practical upplications (Uehara et al. 2020 Shi et al. 2021 Kallus & Uchara 2022). Formally, a time-invariant MDP given by tuple AS, R, Pr. Pa. 7) with: (i) S is the state space that can be discrete or continuous; (ii) A is the action space; (iii) R is the re- ward space, (iv) pr S A + P(R) the reward distribution,
(V) P. : S A + P(S) is the stochastic state transition distribution, and (vi) 7 E (0, 1) is the discount rate for future rewards. A trajectory {(Sr, Ar, Re2 is generated by following a stationary stochastic policy T: at time step L, a mecision-maker in state S,
E A with probability (A, = a St = s), a reward R = r is observed according to the law Rs ~ pr a), and one transitions to new state S s Se+1 (s.
E S selects an action A,
In our data-generating process, we assume (i) that the time- invarinvari MDP model has the Markov property P(S,+1
{S,, Aj, Ry lossst) Ps(s St, As) and (ii) that the conditional mean independence property holds, i.e., E (S). Aj. R. St. As) = As) Together, the assumptions (i) and (ii) guarantee the existence of an optimal stationary policy (Puterman 1994)
and permit us to decompose a dataset of i.i.d. trajectories into one-step transitions, namely
D. = (SS As,,, Ra,,, (x++))osSsST.IS
= {(S). Aj, R2, 1)) N=nT = (0,) W="T (I)
where we use O= (S, A, R, 5) to denote observations.
Key quantities: Given observational dataset from behavioral policy D.,, ma, we are then interested in estimating outcomes under different evaluation policy Te. The target estimand is the state-action value function Q., of Te. which is defined as the 7-discounted expected cumulative reward across trajectories generated according to the policy Te. i.e.,
(s. 2 E
R
S. = s, Ao
1=0
See Figure for a visual illustration of the estimation task. We also define a state value function Us E EAwr(-la)]Q [Q=, (r.(( (s, A)]. We further introduce nuisance functions" of the cumulative and
stationary density ratio via
To (Ax = 0% Sx = sx) Warb(s' ss,a) & E (S, s' S 8, Au a)
(Ax 0g
Sx s8)
Pl:
(3)
po(S= =
3 'Importantly, we can always incorporate any historical information into state variable, simply by concate- nating both and creating an "augmented" state variable where $ = (S. H) where H is any additional historical information we wish store beyond any natural concept of state. The Markov framework will natively apply to such a setup. As such, the MDP setting can also be seen as generalization contextual bandits and some dynamic treatment regime (DTR) setups. Specifically, contextual bandits are one-step MDP with no state transitions (or equivalently, all state transitions are equally possible, i.e., irrelevant). DTR setups usually oper- ate on short time horizons since without assuming Markovianity (=MDP). they invariably also suffer from the curse of horizon. We refer to Appendix [ for extended related work regarding DTR and the relevant overview materials.
'We call nuisance functions all auxiliary functions that are not of primary interest but are required for estimation.
4

---

Published as a conference paper at ICLR 2026
respectively. The subscripts e, b in Pe. Ps used to denote densities arising from following MDP with an evaluation and behavioral policy, respectively. We collect the nuisances in a tuple
7 (p, w./o).
3.1 CAUSAL INTERPRETATION
Objective: Given observational dataset from behavioral policy Dr, Wh. are then inter- ested in estimating outcomes under a different evaluation policy Te Since data following Te is not observed, our target a causal quantity. To formalize this, use the potential outcomes framework (Neyman et al. 1923 Rubin 1974) and denote the potential reward by R[a... i.e., the reward that
would have been observed had action a been selected. Then, M[TT]] 2 E R TT. (a S) is the
potential reward that would have been observed under the policy we (Uehara al., 2022). Hence, we are interested in estimating the potential state-action value had policy been followed:
ST. (s, E
Ro
RRll (-1.)))1))) So s. A.
The causal estimand 5., (s, a) characterizes the expected individualized potential outcomes in se- quential decision-making (e-g., the patient-specific outcome from a dosage schedule of anti-cancer drugs for specific patient trajectory). If identification assumptions hold (see Appendix [B. the causal estimand Eu,, is identified as a statistical estimand Q. and can thus estimated from the observational data (i.e. can be expressed as functional of only the observable distribution from following Tb). Below, we state the identification results in two ways: in Lemma 1 we take obser- vational data at the level of trajectories, whereas, in Lemma [ take the observational data the level of one-step transitions. While the first approach is more straightforward, the second allows us to later break the curse of horizon when develop the DRQ-learner
Lemma 1 (Identification over trajectories). Under Assumptions (1)-33 from above, the causal estimand in Eq. ( is identifiable from the observed data of trajectories via
Ex. (8, a) = 1r. (8, Ero
Ro +
Σ
7 Pipee So =
-, 8,
(5)
Proof. See Appendix D.4
Lemma 2 (Identification over one-step transitions). Under Assumptions (1)-(), the causal es- timand in Eq. (4) is identifiable from the observed data of one-step transitions via (s,
Q=.(s.0) = = a)) where ff is the unique solution (unique up to equality almost everywhere) to the Bellman equation for Te, i.e.,
((,,a) = E R 7E A (13) 5= A
(6)
Proof. See Appendix D.4
While the derivations of the above identifiability results are straightforward, our aim behind these is to cast the target explicitly as a causal estimand. In the following section, we build on these identification Lemmas and recast existing 2T,, estimation algorithms as causal plug-in learners.
4 A ROADMAP TO ORTHOGONAL LEARNING
To derive our method for estimating Qr, from observational data Dr,. we proceed in three steps: ! We first leverage the above identifiability results construct simple plug-in learners (Sec- 4.I.. We show that these plug-in learners recover existing methods from the literature, namely, Q-regression (Liu al. 2018) and FQE (Le et al. 2019). However, plug-in learners have inherent limitations such as so-called plug-in in bias (Kennedy) 2022). This serves two-fold: to formalize the drawbacks of existing methods theoretically (using the lens of the potential outcomes framework) and to motivate alternative estimation strategy. 2 We then sketch out the idea behind design-
losses (Section 4.2 3 Finally, we then ing two-stage meta-learners based on meta-leama called DRQ-learner (Section 5). To do so, present our new leverage semiparametric efficiency theory and derive the efficient influence function. We also show that meta-learner has several favorable theoretical properties, namely, double robustness, Neyman-orthooonalliy and quasi-oracle efficiency. We provide an overview of the different learners Figure
5

---

Published as a conference paper at ICLR 2026
4.1 WHY PLUG-IN LEARNERS ARE OUB-OPTIMAL
The identification results from above (i.e., Lemma and Lemma 2 give immediately rise to two naive plug-in estimators. However, as we show later, each comes with inherent limitations
IPTW plug-in learner: A straightforward way to obtain an estimator of Qr, is to take the iden- (i.e., right-hand side of Eq. (5)) and "plug-in" an estimated
tification result based on Lemma [ cumulative density ratio nuisance Pl:t. This yields
1=1 (
QIPTW as,a)= = Σ
R4,0 +
E''maaa IfSt,0 = A., = a}
(7)
which involves the density ratio Pr:r and thus captures the inverse probability of treatment weighting (IPTW). When then generalize the estimator from tabular point-wise solution to learning the best model y from restricted model class G, we yield
Q = = arg min DEg
Σ Pe,aa (Yi, ((.)), for Yr, = - Rass (8) 220
222
which corresponds exactly to Q-regression (Liu et al. 2018 A specific limitation of the IPTW plug-in learner (=Q-regression) is that it suffers from the curse of horizon consequence of using the cumulative density ratio nuisance #1:1.
Recursive plug-in learner: An alternative is to use the second identification result from Lemma Analogous to the technique used in the identification proof (see Appendix D.4), we can recursively obtain an estimator Qx+1 by "plugging-in in" into the right-hand side of Eq. the previous estimator Qx. Formally, we have
Q lim Qx for Qu=i(s,a)= Σ [1t + (1,
(5., (5) 11S = s, at} (9)
This yields estimated solution to the empirical approximation of Eq. 6 Generalizing this ap- proach to a minimization over a model class G, we yield
N Σ
Q.. = lim OR for 0k+1 = arg min
( (RL+ in dn (S,, A)] o(S, A.) (10)
deg
which corresponds exactly to the FQE baseline et al. 2019)). While the recursive plug-in learner (=FQE) breaks the curse of horizon, its recursive fitting procedure may lead unpredictable failure modes or even divergence (see the problem of deadly triad in, e.g.. Sutton & Barto (2018)).
Fundamental problems of plug-in learn- ers: Both plug-in learners suffer from so-called plug-in bias (Kennedy) 2022): that is, errors in the nuisance estimates directly propagate to the causal estimand. In contrast, we now derive our Nermannorthogonal meta-learner that elim- inates first-order bias from the nuisance func- tions. Hence, bias from nuisance function timates propagates into the final estimand only via higher-order enrors.
[R As. Ru.
[B Sirrord
[Re Rurraa
Ca
aL
Oharrvatioaal ratt So,
Nasuar colizaicos
Leansco
Figure 3: Comparison. After observing the data Di,- the learner-specific nuisance functions are estimated first, followed by the actual estimand. t our DRQ-learner. Learners suffering from plug-in bias marked with X.
4.2 INTUITION
BEHIND TWO-STAGE META-LEARNERS
To resolve issues from plug-in in bias, we later de- velop a two-staged meta-learner (see Fig. 3) The basic idea is: 1 In the first stage, the nuisances
$ For ease of exposition, we adopt the nomenclature for naming different methods based on causal inference literature, but later state the corresponding names of the benchmarks in the literature.
'To see why this is generalization of the tabular QIPTW consider the case of having a free parameter for each possible point evaluation. The learned minimizer is then nothing else than point-wise solution to the estimating equation VoC(0) which will simply equate ((s, D Y., I[S,, 8, ALe a).

---

Published as a conference paper at ICLR 2026
estimated, yielding some estimate ft. 2 In the second stage, the target g with true value g" is estimated by empirical risk minimization (ERM) over a risk via
arg min (f).g). gEG
(11)
Here, one seeks learner (second-stage loss) with small error despite learning 9 with the estimated nuisance carrying first-stage estimation error. However, deriving such second-stage loss non- trivial.
A common feature for the second-stage loss is to employ Neyman-orthogonal loss functions CChe- nozhukov et al., 2018), which (in population) satisfy the property
Dr,D, و)) g. -
(12)
where Dy and D, are directional (Gateaux) derivatives in function space (Foster & Syrgkanis/ 2019). Informally, orthogonality means the gradient of the loss D,C (i.c., the estimating function, or also known the score) is insensitive to small perturbations in the nuisances around their oracle value such as those arising from nuisance estimation error
OUR DRQ-LEARNER
We proceed in three steps: ! We first derive our Neyman-orthogone orthogonal loss. Quasi-oracl efficiency and double-robustness properties of our loss. 3 Finally, we elaborate on the practical implementation.
2 Next, we show the
5.1 THEORETICAL RESULTS
We denote our Neyman-orthogonal loss by L (1,9), which we formally derive in Theorem Therein, we derive and employ the efficient influence function (EIF). With the perspective of classi- cal semiparametric inference, We replace the ERM estimate of the population risk with debiased estimator based on the EIF. Under standard regularity conditions, the resulting population analogue is Neyman-orf orthogonal. Hence, by deriving the EIF of a standard MSE population risk, we obtain our main result, namely, the Neyman-orthoognal loss 1 (1,9)-
Theorem (Neyman--rthhognnaitt) The loss
L. (). Eo'
Σ (a 8*)) (0) a(S' a))² tBorrpp Σ Rela (02 - o(s,a))²
(13)
where
2 I(A' a)
{ It' + (5') (5, (S',aa))
(14)
Ts(A' 5
62 (A'l 18))
| |S") We/b(S' a) a) f yvs, ( (S' (')) { + Qu,(s,a)
(15)
orthogonl w.r.t all the nuisance functions n Te. We/bs Qx.). For intuition on the form of the pseudo-outcome outcomes Φ1 oz we refer to Appendix D.I.I
Proof. We refer the reader to Appendix D.I formal proof. Here, provide intuition, show the efficient influence function of the standard MSE loss, L (n.9), g). which is shown to be
IF (F (n. g). 0')
ne (AA(S))) Σ (a (S', a))² (n.g) 2 2f (t' (8)) (S,
(16)
(AA(S))
(8' g(S' A') Ea (s, o(s
Afterward, we derive the loss 1.3 (17, g) with the debiasing procedure (and some algebraic manip- ulations).
is then proved by taking the necessary derivatives. A formal and detailed derivation is in Appendix D.]
For extended discussion of orthogonal statistical learning, refer Appendix

---

Published as a conference paper at ICLR 2026
Theorem shows that our loss, L3_ (7,9), for estimating QT is Neyman-ortin Neyman-orthogonal and, therefore, robust to nuisance estimation error. Finally, we prove our loss is quasi-orach efficient and doubly robust.
Theorem (Quasi-oracle efficiency). Under standard assumptions (see Foster & Syrgkanis 2019 , La (7,9) achieves quasi-oracle efficiency, specifically, for arg min oeg xx (1.9)
l1g - 011..p0.. S A nh A 11 we/ Qr.
(17)
where x S. y is taken to mean there exists a constant C > such that x S Cy, the Ak operator is defined as k --K, k- for any function k, and g" arg mingeg La_ g). which equals the true Q=. provided the function class G is expressive enough include it. Lastly, the norm weighting pame in p. mirrors that of the loss.
Corollary 1 (Double robustness). The learned approximation 9 doubly robust. Specifically, if either AQx, or Adb + Awefb + then ğ a consistent estimator of g" i.e, asymptotically l1g* 91 =0. 0.
Proof. See Appendix D.3 for the proofs of both the theorem and the corollary.
The bound in Eq. 177) shows that the excess risk of 9 depends only on products of nuisance esti- mation errors. This means that even if one nuisance component (e.g. Q+, converges slowly, the
overall estimator still converges at the fast rate of the better--stimated component. In other words, behaves as if oracles nuisances were used, up to higher-order terms (cf. |ooster & Syrgkanis [2019]
Nie & Wager] 202T). The estimation error is thus shielded from first-order nuisance misspecification and is only impacted through second-order interactions.
Remark: Our above theory is different from [hi et al.
Algorithm Our DRQ-learner for QT.
(021 in the following ways: For the purpose of ob-
Input: Olbserved dataset D -g- class 9 Output: Doubly Robust extimator Q
taining a tight confidence interval for OPE, (hi et al.
202T have derived a point-wise iterative debiasing procedure for (in their view nuisance) QT,, that, when restricted to the discrete state setting with model class G restrictions, corresponds to our learner. We provide more general solution that (i) is applicable to both continuous and discrete state spaces, (ii) able to fit an estimator g E G. and (iii) provides the theory nec- essary to show syyyman-orthoonnanee efficiency. Additionally, our derivation of the efficient influence function means that, for the discrete setting, we show that the estimator is efficien
1: // First stage (nuisance estimation)
2: dgfo, e- Pold - a 18-ma
3: 1h wa,s(o',
Ex Po(e--'' S00
Polo')
4: 4+_ B2r (Eioo'"R "RR//s0 a, do m
5: 11 Second s.age adjustenlt)
6: 4te argmingeg L. (Ft, Pe/bb 42 ).9)
7: Returm: QDR
and quasi- oracle
5.2 IMPLEMENTATION
The pseudocode for DRQ-leamer is in Algorithm I (1) The first stage simply estimates the nuisance functions, namely, n (#,, ue/b,, ²t.). Notably, the nuisances include the target itself Q. (2) The aim of the second stage estimation is to refine the first stage estimate Q1 with a loss designed bring favorable theoretical properties to the second stage refinement. Put differently. we construct a meta-learner that in the first stage accepts any off Qr, estimation method and subsequently refines it. Furthermore, we may choose to restrict the space of solutions g of the second stage and obtain the best projection of true g" & G onto G. for example, if we wish to obtain an interpretable solution.
Pseudocode:
Our DRQ-learner is generally flexible and can be implemented with arbitrary machine learning models for estimating the nuisance functions as well as the second-stage. We provide details about the architectures and fitting process we use in our experiments in Appendix D
Implementation:
$ Note that the approach of Shi al. (2021) cannot readily be extended continuous settings since their pisnt-wise debiasing step includes Dirac delta function on the state. a continuous setting, this either zero or infinite, and thus not directly applicable.
9 Mcaning achieves the semiparametric efficiency hound on asymptotic variance dictated by the EIF.
8

---

Published as a conference paper at ICLR 2026
6 EXPERIMENTS
The primary goal of our experiments is not traditional benchmarking but rather to validate our theo- retical results: ① that our DRQ-leumer oulperforms the plug-in learner;; ? that our DRQ-learner is especially effective in settings that benefit from
such as settings with low overlap: and 3 that our theory is applicable to different function classes including restricted model
classes G.
Settings: We consider the Taxi and Frozen Lake environments from the OpenAl Gym package (Brockman et al., 2016). We set our data- gata-gennrating policy тт and our target evaluation policy Te as -eeeedy policies, N. + E-greedy(Q",, E;) for i E {e, b] and for the optimal Q", which acquire in an online fashion. We generate dataset Dr, of n trajectories following mT. We consider two settings: (A) when the model class G is left unrestricted, and (B) when the model class is restricted to simple linear model. For each setting, we conduct three sets of experiments: (I) We consider a varying dataset size E [2000,.... 6000). (2) By varying the discount factor y we alter the length of the horizon considered. Here, we vary the effective horizor 10 h - 1-7 in the range h [3, 33....,2 [.)]. or in other words, y [0.66..... 0.95]. (3) By varying the greediness & E [0.1,..... .09) of the target evaluation policy, while holding 6, fixed, we can directly vary the degree of overlap between the dataset and the off- off-policy potential distribution whose Q-function we
seek to estimate. We use simple metric Overlap = Σ. min(TT(a) (a))) to quantify the level of overlap.
Metric: We evaluate the performance of all methods using rMSE(Q, Qm.)) the mean (+ standard error) over 5 runs with different seeds.
11Q-Q+. 12 We report 11Q 11
Baselines: As baselines, we implement standard Q=, estimation methods of Q-regression Liu
et al. 2018) and FQE (Le et al., 2019). We have previously shown that these correspond to plug- in methods and should thus be inferior. Additionally, we implement Minimax Q-learning (MQL) (Uchara et al. 2020). For implementation details, see Appendix
Al: varying dataset size
A2: varying length of borizon
A3: varying uverlap
0.57 0.75 053 0.9 1000 052 0as 6,35 047
10
10P
107
10
10-1
30-
7000 2503
1000
3500 1620 1500 5300 9920 6500
10-1
121314
1815
Figure 4: Setting A Taxi environment: Unrestricted model class G. The results confirm the theoretical properties: our DRQQ-lerner in blue is better than the plug-in learners in red/orange, robust for varying lengths of the horizon, and is especially effective for settings with low overlap.
Results for Setting A (unrestricted) are in Fig. Our DRQ-learner performs best across a variety of configurations. In particular, we confirm: our method consistently outperforms the plug-in inn learners. Further, our experiments show our method successfully incorporates the density ratio nuisance without degrading performance in the low overlap scenario (see Fig. A3). Hence, the empirical results confirm our theoretical properties. In particular, we confirm 2 that our DRQ-learner is especially effective for long horizons and for low overlap settings in line with our theory. Results for Setting B (restricted) are in Fig. 5 Our method is highly effective and performs best for many settings, especially with low overlar applicable to restricted model classes.
Results:
Thereby, we confirm 3 that our theory is also
Since Ee 7 = rhy state-action values will have a magnitude of ييبر times that of rewards. Instead of thinking of discounted rewards across an unbounded trajectory, we consider effectively taking a horizon of l steps with undiscounted rewards.
IIntuition:
11 Note about Figure 5 At large e, the target policy becomes nearly random, making Q-regression behave unusually well. This is because the density ratios shrink rather than explode, removing its usual instability. In this casynnuisance regime the robustness and asymptotic henefits of DRQ-leamer matter less, so its finite- sample performance does not strictly dominate.

---

Published as a conference paper at ICLR 2026
B1: varying dutaset size
B2: varying length of harizon
B3: varying overlap
Overlep
6,47
0.75
0:3
5.2
1.00 6,92
10
m85
300
305
2020 2500 3003
3050 4550 4500
5200 3000 6052
2031 CE 30
0.1
Figure 5: Setting B Taxi environment: linear model class G. The results confirm that our theory and thus DRQ leammer (in blue) applicable to different (restricted) function classes.
varying
varying length borizon
varying overlap
Overtap
1.2
201
30"0
10
350
200
10
191 1213
16T9
0.2
0.1
0.5
0.6
07
08
06
Figure 6: Setting A Frozen Lake environment: Unrestricted model class G. The results confirm the theoretical properties: our DRQ-learner in blue better than the plug-in learners in red/orange, robust for varying lengths of the horizon.
Conclusion: In sum, our DRQ-learnr is the first approach jointly achieve double robustness, -yymaa-orthogonalijy and quasi-oracle efficiency. Thereby, we provide principled and flexible foundation for reliable individualized decision-making sequential settings. particular advantage of our DRQ-learner is its flexibility to accommodate real-world constraints such as interpretability or fairness into the solution space G.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This paper is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space. Additionally, this work has been supported by the German Federal Ministry of Education and Research (Grant: 011S24082).
ETHICS STATEMENT
Our work develops a theoretically principled approach, the DRQ-learner, for estimating individu- alized potential outcomes in MDPs from observational data. The primary goal of this work is to improve the reliability of decision-making algorithms, particularly in high-stakes settings such as personalized medicine. To promote reliable decision-mas makiin,, we focus on identifiability results that transparently state the boundary conditions of our method and thus ensure when our method can be safely used.
Potential benefits and risks: Our method aims to enhance safe and effective individualized decision- making by providing more statistically reliable Q-function estimates even under model misspeci- fication and low-overlap conditions. This can ultimately contribute to better treatment policies in healthcare and other critical upplications. However, like any method that can be applied to personal- ized decision-making, misuse in inappropriate or sensitive settings could have unintended negative consequences (e.g., reinforcing biases present in observational data). We therefore emphasize that successful application of our method requires domain expertise to ensure causal assumptions such as unconfoundedness and positivity, which necessary for identifiability. Our work explicitly frames Q-function estimation as a causal inference problem, aligning with recent arguments that reliable algorithmic decision- making must be grounded in causal reasoning ensure valid and trustworthy deployment (cf. Kern et al. (025).
Societal and fairness considerations: While our method model-agnostic and does not impose fairness constraints by design, it can be combined with fairness-aware modeling or post-hoc policy adjustment techniques. We encourage practitioners to monitor for disparate impact across subpopu- lations when deploying systems trained with DRQ-leamner, especially in high-stakes domains.
REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure the reproducibility of our results. All theoretical contribu- tions, including identifiability results, the derivation of the efficient influence function, and the proof of double robustness and quasi-oracle efficiency, are presented in full detail in the main text and rigorously proven in Appendix [ Our algorithm specified formally in Section [.] and summa- rized in pseudocode to facilitate implementation in Section 5.2 Hyperparameters, model classes, and training details are provided in Appendix E For our empirical evaluation, we rely exclusively environments from the OpenAl Gym package, experiments can be exactly reproduced by other researchers. We also provide an anonymized, open- source implementation of DRQ-learner and scripts to reproduce all figures and tables, available https: / / thub. com/ Em 1J IJavured www.eek/aae. sottthhoooal-----10 MDPs Upon acceptance, we will make our code publicly available via GitHub repository. Together, these materials enable in- dependent researchers to fully replicate both the theoretical and empirical results presented in this work.
described in Section 66] which ensures that
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Ahmed Allam, Stefan Feuerriegel, Michael Rebhan, and Michael Krauthammer. Analyzing patient trajectories with artificial intelligence. Journal of Medical Internet Research, 23(12):e29812, 2021.
Samuel L. Battalio, David E. Conroy, Walter Dempsey, Peng Liao, Marianne Menictas, Susan Mur- phy, Inbal Nahum-Shane Tianchen Qian, Santosh Kumar, and Bonnic Spring. Sense2stop: A minro--andommzze trial using wearable sensors optimize aat-i--ttim-adaptive stress manage- ment intervention for smoking relapse prevention. Contemporary Clinical Trials, 109:106534, 2021.
Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihacla van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In ICLR, 2020.
Ioana Bica, Ahmed M. Alaa, Craig Lambert, and Mihaela der Schaar. From real-world patient data to individualized treatment effects using machine learning: Current and future methods to address underlying challenges. Clinical Pharmacology & Therapewtics, 109(1):87-100, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint, 11Xiv:1606.0155 2016.
Bibhas Chakraborty and Erica E. M. Moodie. Statisticul methods for dynamic treatment regimes: Reinforcement learning, causal inference, and personalized medicine. Statistics for biology and health. Springer, New York NY, 2013.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. Econometries Journal, 21(1): 2018.
Daniel Rubin. Targeted maximum likelihood learning. The International Journal of Biostatistics,
(1), 2006.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In ICML, 2018.
Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Ali- cia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958-968, 2024.
Aaron Fisher and Edward H. Kennedy. Visually communicating and teaching intuition for influence functions. arXiv preprint, arXiv:1810.03260, 2018
Dylan J. Foster and Vasilis Syrgkanis. arXiv:1901. 09906,,,6 2019.
Orthogonal statistical learning.
arXiv preprint,
Dennis Frauen, Konstantin Hess, and Stefan Feuerriegel. agnosti meta-learners for estimat- ing heterogeneous treatment effects over time. In ICLR, 2025.
Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Remi Munos. Q(lambda) with off- policy corrections. In ALT, 2016.
Konstantin Hess, Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. G-transformer for conditional average potential outcome estimation over time. arXiv preprint, arXiv:2405.2101, 2024.
Konstantin Hess, Dennis Frauen, Mihaela van der Schaar, and Stefan Feuerriegel. Overlap-weighted orthogonal meta-learner for treatment effect estimation over time. In ICLR, 2026.
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evalu- ation with double reinforcement learning. Operations Research, 70(6):3282-3302 2022.
Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprini, arXiv:2004.1 14497, 2020.
12

---

Published as a conference paper at ICLR 2026
Edward H. Kennedy. Semiparametric doubly robust targeted double machine learning: review. arXiv preprint, arXiv:2203.06469, 2022.
Christoph Ker, Unai Fischer-Abale Abaigar, Jonas Schweisthal, Dennis Frauen, Rayid Ghani, Stefan Feuerriegel, Mihaela van der Schaar, and Frauke Kreuter. Algorithms for reliable decision-making need causal reasoning. Nature Compulational Science, 565):366-300, 2025.
Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. JMLR, 4(Dec): 107-1149, 2003.
Hoang M. Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints, In ICML, 2019.
Greg Lewis and Vasilis Syrgkanis. Double/debiased machine learning for dynamic treatment effects via g-estimation. In NeurIPS, 2021.
Rui Li, Stephanie Hu, Mingyu Lu, Yuria Utsumi, Prithwish Chakraborty, Daby M. Sow, Piyush Madan, Jun Li, Mohamed Ghalwash, Zach Shahn, and Li-wei Lehman. G-net: recurrent network approach to g-computation for counterfactual prediction under a dynamic treatment regime. In ML4H, pp. 282-299, 2021.
Peng Liao, Predrag Klasnja, and Susan Murphy. Off-policy estimation of long-term average out- comes with applications to mobile health. Journal of the American Statistical Association, 116 (533):322-39,, 2021.
Bryan Lim, Ahmed Alaa, and Mihaela van der Schaar. Forecasting treatment responses over time using recurrent marginal structural networks. In NeurIPS, 2018.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- off-policy estimation. NeurlPS, 2018.
Roland A. Matsouaka, Junlong Li, and Tianxi Cai. Evaluating marker-guided treatment selection strategies. Biometrics, 70(3):489 499, 2014.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. ICML, pp. 15293-15329, 2022.
Pawel Morzywolek, Johan Decruyenaere, and Stijn Vansteelandt. On weighted orthogonal learners for heterogeneous treatment effects. arXiv preprint, arXiv:2303.12687, 2023.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off- policy reinforcement learning. In NeurlPS, 2016.
Jerzy Neyman, D. M. Dabrowska, and T. P. Speed. On the application of probability theory to agricultural experiments. essay on principles. section 9, Annals of Agricultural Sciences, 10: 1-51, 1923.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319, 2021.
Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off off-policy policy evalu- ation. ICML, 2000.
Martin L. Puterman. Markov Decision Processes: Discrete stochastic dynamic programming. Wiley series in probability and mathematical statistics. Applied probability and statistics section. Wiley, Hoboken, New Jersey, 1994.
M. Robins, M. A. Hermán, and B. Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology. 11(5):550- 560, 2000.
Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies.
Journal of Educational Psychology, 66(5):688-701 1974.
13

---

Published as a conference paper at ICLR 2026
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. In ICML, 2022.
Chengchun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off-policy interval estimation. In ICML, 2021.
Susan M. Shortreed, Eric Laber, Daniel J. Lizotte. T. Scott Stroup, Joelle Pineau, and Susan A. Mur- phy. Informing sequential clinical decision-makiin through reinforcement learning: an empirical study, Machine Learning, 84(1-2): 109-136, 2011.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning. An introduction: An introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge Massachusetts, second edition edition, 2018
Theresa Blumlein, Joel Persson, and Stefan Feuerriegel. Learning optimal dynamic treatment regimes using causal tree methods in medicine. In ML4H, pp. 146-171, 2022.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In ICML,, 2020.
Masatoshi Uehara, Chengehun Shi, and Nathan Kallus. review of off-po evaluation in rein- forcement learning. arXiv preprint, arXiv:2212.06355 2022.
Lars van der Laan, David Hubbard, Allen Tran, Nathan Kallus, and Aurélien Bibaut. Semiparametric double reinforcement learning with applications to long-term causal inference. arXiv preprint, arXiv:2501.06922 2025.
A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilist Mathe- matics. Cambridge University Press, Cambridge, 1998.
Lu Wang, Andrea Rotnitzky, Xihong Lin, Randall E. Millikan, and Peter F. Thall. Evaluation of viable dynamic treatment regimes in a sequentially randomized trial of advanced prostate cancer. Journal of the American Statistical Association, 107(498):493-500 2012.
Yufan Zhao, Michael R. Kosorok, and Donglin Zeng. Reinforcement learning design for cancer
clinical trials. Statistics in Medicine, 239-33,3 2009.
14

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Here, we provide an extended related work to offer additional context for our work.
Off-policcy Q-function evaluation: Methods targeting the off-policy Q-function from MDPs, as is our goal, are often presented as plug-in off-policy evaluation (OPE) methods. In the OPE literature, Qr is nuisance function 12 where new fitting procedure for Q=. is taken to imply a new ilug-in learner for OPE. An odd consequence of this is that the perfor- mance of Q.. function estimation is often evaluated only via the implied performance of the estimated scalar average off off-policy policy value. Yet, many practical applications have di- rect interest in estimating individualized outcomes such as Qr. to personalize medical deci- sions (Feuerriegel et al.] 2024), and, hence, we focus here on estimating QT., directly. Ex- isting Qn. estimation techniques address the off-policy nature of the problem either explicitly via an inverse-propentee -eenniitty-wightingg like nuisance (Liu et al., 2018 Farajtabar et al., 2018 Uc- hara et al. 22200, Munos et al. 2016), implicitly in the (supervised learning) target struction (Le et al. Lagouddkks & Parr 2003 Precup et al. 2000; Harutyunyan et al. 2016). Finally, mention the work of van der Laan al. (2025), who have developed debiased esti- mator for linear functionals of Qu. While this generalizes debiased estimation from just OPE to all linear functionals of Qr it cannot be applied Qx_ itself.
Potential outcomes in MDPs: Off-policy (potential outcome) estimation in MDPs is commonly encountered in OPE for RL. Here, the goal is estimate the scalar policy value of an evaluation policy different from the one that generated the observed MDP trajectories. Various doubly-robust meta-learning methods have been developed to make the OPE estimate robust to errors in the learned nuisances (Kallus & Uehara 2022. Farajtabar et al., 2018 Shi et al., 2021). Notably, Kallus & Uehara (2022) have derived the efficient influence function of the off-policy policy value and a corresponding efficient DR-learner. For detailed statistical overview of OPE in RL, see [Uehara] jet al. (2022). However, none of these learners are targeted at Q-function estimation, but only target the scalar policy value instead.
Individualized potential outcomes over time: Several methods have been proposed for estimating
individualized potential outcomes in time-series settings (Lim et al. 2018| Bica et al. 2020
Mcl-
nychuk et al., 2022 Li et al., 2021; Hess et al., 2024 Lewis & Syrgkanis 2021 Hess et al. 2026).
These be grouped into both model-based (e.g., adaptations of the transformer architecture for estimating individualized potential outcomes over time, such as (Melnychuk et al. (022) and
meta-learners (e.g., model-agnostic "recipes" for leveraging existing models to perform valid causal inference). Notably. Frauen et al. (2025) have derived a DR-learner and variations thereof. Meth- ods in this stream target the conditional average potential outcome Y > provided the entire history H, up to time t. Hence, while these methods could theoretically be adapted to target the long-term average of future outcomes (our goal), they do not take advantage of the Markov structure of MDPs and thus suffer from the curse of horizon.
DTR: An adjacent field are dynamic treatment regimes (DTR), which are concerned with optimizing (individualized) treatment assignment in time-series setting. For an overview, see Chakraborty & Moodie 2013).. While there are extensions DTRs using machine learning (e.g.. Therrsa Blumlein et al. 20022), these have limitations for our setting. In particular, the DTR literature also typically does not consider the MDP setting, and methods from DTR thus suffer from the same curse of horizon other general time-series methods.
12 QT.: we mean the off-policy Q of evaluation policy that differs from the policy wa that observe data from.
15

---

Published as a conference paper at ICLR 2026
B ADDITIONAL DETAILS
Identification. To be able to estimate this causal quantity from observational data generated with The we need the following standard identification assumptions (Robins et al., 2000 Uehara et al. 2022): (1) Weak positivity: The support of T- 8) is included in the support of T 8) for any
S. (2) Consistency: R₂ R² [.(A,). almost surely. (3) Unconfoundednes:: For any a E A, A
and R[a] are conditionally independent given S. i.e., A, 11 R, a. St. Of note, these assumptions
are standard in the causal inference literature (Lim et al. 2018 Bica et al. 2020: Melnychuk al.)
2022
Seedat et al. 2022 Frauen et al. 2025
Curse of horizon. The curse of horizon refers to the phenomenon that estimation emror in off- policy RL grows exponentially with the time horizon when the estimand depends on long sequences of actions. The reason for this the exponential decay in overlap between trajectories from the observed and evaluation policies. This is inherent difficulty of the setting, for all estimation tasks in the off- off-policy setting and any corresponding approaches. The technical challenge lies in successfully and also efficiently leveraging the time-invariant Markov property of the MDP setting to break this curse. We refer to Kallus & Uehara 2002) for exhaustive treatment of this problem for off policy policy value estimation.
Practical aspects of nuisance estimation. The nuisances required by our second-stage model can in practice be quite complex and difficult estimate. Here, we give several thoughts on this topic:
The second stage is agnostic to choice nuisance estimation model. Apart from the suffi cient convergence guarantees, the practitioner is free to employ any method of their choos- ing to estimate the nuisance. This especially important cases such as ours, where some of the nuisances are more complex.
ii) The Iwo-stage learner can, in principle, only ever improve upon the 1st stage. Since our method requires the estimation of the target Qx, in the first stage nuisance estimation, we are, of course, always free to stick with the nuisance estimate. Our DRQ learner is of use in settings where additional estimation complexity is worth it for the strong theoretical guarantees we provide in return. This motivation is natural for many high-stakes real-world applications such as medical applications listed in the Introduction.
iii) The nuisances are only as complex the underlying setting. Given the principled deriva- tion of our method with the efficient influence function, all nuisance functions present in the loss come from the statistical model inherent the problem setting. is worth pointing out existing papers that propose estimation methods for similar stationary density ratio nui- sances such as ours ( (e.g., the work in [1]). Conversely, methods not using these complex nuisances may be deceptively simple: FQE does not use this nuisance but is well known to have unpredictable failure regimes (see "deadly triad" problem, Sutton & Barto (2018)).
16

---

Published as a conference paper at ICLR 2026
C BACKGROUND ON INFLUENCE FUNCTIONS, ORTHOGONAL LEARNING
In this section, we provide a brief overview of efficient influence functions and orthogonal learning, following the treatment in (Kennedy) 2022).
Efficient influence function (EIF), In semiparametric statistics, estimaation is framed in terms of a statistical model {P P), where P denotes family of probability distributions. We are interested in functional to : p + R. For instance, one might consider (bP) Ep(R|S = s). If vb is sufficiently smooth, admits von Mises (distributional Taylor) expansion:
(PP)~(P)) / o(l, d( P))))(())))))))))))))))(()()))))
P)(() P),
(18)
where R2 P) is second-order remainder term and @(t, P) is the efficient influence function (EIF) th. By definition, the EIF satisfies f o(t, P) dP(() and f ott P)2 dP(t) < co.
Plug-in bias and bias correction. Consider an estimator P of P and the associated plug-in estima- tor (P). The expansion above implies a first-order plug-in in bias:
(PP) - -p)((()) --) (t. P) dP(t) + R2(
(19)
because f ((t P)dP(() O. Intuitively, simply plugging estimated nuisance functions into the identification formula generally leads to a biased estimator. classical way to correct this bias is to estimate the term the hand side and add it back, yielding one-step bias-corrected estimator:
3 = + (OTT,
(20)
This correction removes the leading-order bias, leaving only second-order remainder.
Debiased target loss and orthogonality. While one-step correction works well for finite- dimensional parameters such average treatment effects, it is not directly applicable for infinite- dimensional targets such as conditional treatment effects Tr (X). In such settings, the EIF can still be used to construct debiased loss function rather than directly de-biasing the target parameter. Minimizing this orthogonalized loss leads to estimators that are first-order insensitive to nuisance estimation error, which is the core idea behind Jeyman-orthogome learners.
17

---

Published as a conference paper at ICLR 2026
D PROOFS
D.1 DERIVATION OF OUR Loss
We construct our Neyman-orthogonal loss by debiasing the ERM estimate using the efficient influ- ence function (EIF). We begin by taking the EIF of standard MSE loss L.
STATISTICAL MODEL
First, we must define our statistical model. Let us define a model for observations O = (S, A, R, g via
M- {p p(o) - p(s)p(als)p(r/s ()pps))ccs > 0}.
(21)
We denote the (unknown) true data-generating (observational) distribution e .M and one- dimensional parametrized submodel of distributions by
P. {Pell pe(o) p(o) + e(p' (o) p(o)); e (0, 1)}
(22)
where P. c M. i.e., Pe E M. We take without subscript to be density corresponding to P. i.e., p(als) = ms(als) p(r|s, p-(r s, a), p(5]s.) p=(5)s,
We take the strategy advocated by Kennedy (2022). where, by cleverly choosing the parametric
submodel to represent point-mass deviation from P. i.e. the Dirac delta point O', p' 5(0'
o), the EIF derivation reduces to taking a Gateaux derivative
TF(F(P)) de F(P.)
(23)
We refer to Kennedy (0022; Fisher & Kennedy (2018) for comprehensive tutorials and technical details of efficient influence functions.
TAKING THE EIF OF LT.
With the MSE population risk under the evaluation distribution 4 defined as
T (n, g) Bompe (S, a(S, = Boowps
(Q. (S. -S(s)) {
(24)
we take the EIF via
(T. (n.
(25)
Σ malas)) (S, o((, - -1. (1.
(26)
+ / Σ
g(s,
(27)
To derive as) E. |>in x R = 8, Au = a) Taking the EIF of the individual elements of the sum, we have, sequentially, the EIF for the the null and first conditional expected reward by
we decompose the Qxe via its definition (s,
IF (Ex, Ra so. Ao = o'), 6(su S¹. A') (N' E[RolSo = so. Ao = =a))
(28)
Ps(S s0, A = 0o)
IF (E., Ry so. A0 00) =
(29)
(f (0) ( hinla
(30)
|1, S0 A / (ost s0. = an)
(b(s1 = - mn))
p(s1/0a,
Fe (e a1)r de
(31)
Ó(s) = a' 1')
(8(r) R²) (1 daysd
(32)
Ps(S .',e a1)
6(30 5', ag A')
F., [R = - 12 so. Ao nol
(33)
Po(S s0, A mo)
Pe(S: S = S
Ao = aa)
R' E.,
[Ri = A A']}.
(34)
1s S = 17
18

---

Published as a conference paper at ICLR 2026
We further yield the EIF for the second conditional expected reward by
(Ba, [Ra|So = so, Ao = au). O')=
(35)
=IF (ʃ (801)800000 htgp://a.....a.......aaaaaaaaaaa hwwp://///00..aaa/////ddddaaaada
(36)
/
6(00==5 a00 A')
Pb p s0. = no)
(Ó(s1 5*) ((01ls).00)) ( patp (2)) a2dss dai dezdazdr2
(37)
+ plasleo. { 6(s) pa(S = $1, a1)
[ = -0 AQ (ó(82 = p( p www.aa/wpppaal.....aaaaaaddadadd
(38)
[(s2 [6())=8',az 88,02 1 (8(r2 R²) - -p(2)22.02) rods da dapdaadry
+ //allo..0rraaaliiaaaaaaaaaaaaaaa
phl8 52, A 02)
(39)
5(su ,0 = A')
Es., [Ra]S] = 5*) -x, [Ra |Sn = so, Ao = aol
(40)
Pa(S so, A = uz)
Pe(SI A1 So = 80, Ao ao)
1R 8' - [2 5" S1 =
(41)
Po(5 A=
+ Pe S' A'
Ao an)
R - (Ra = S A2 = A11}
(42)
Pa(S = A A')
Generally, for k 2 (where we abuse the notation with the arrows for readability), we thus have
IF (Bn, [Ra|So = sn, Ao 0n)
= IF ( Pelsi r co)r co) dr.
d 0')
Σ (( Pal01 + Tw s0, av)
a1)) TndS1 - drk
+ / IF(p(ra|sE, ax)) Pe(a) relso, 0o)
radai + dnk
a(
6(00 = 8' = A') Ex [R = 8'1 Ex, [R = Au anl}
po(S so, 1 an)
PS = S' 10 = 50, au) { (E,, (Rx S = S7 E., (RxSS Ar 11}
55 8¹ A A))
(S S' = N = Au = an) {nr ET. = =S', -'1)
(S = S¹ A4 = A
Putting together, we get
IF
B Σ Ri so, Ao ap), O¹
'IF (Fr, [Re = 80; Ao = ao))
0
6(00- 8',a0 au A²) { nt' Σ B', [Re =8
E',,, [ReSa Au av.)
po(8- = A 04)
(pa(8: = S = 1'150
30, Au 20)
{1l
Σ
En, [Re S2 = 5 Σ 1.,,
E., [R(SS: = S" = {1}
Pb(S = S', A)
1=2
7 (S, S', = So su. 0n) {
Σ
7 1 RR1S = 51
Σ
3 - E. 18118 [8111888= = =1
Pn(S S' A")
1=1+1
1=
7 (S; = S A' = A
Ao 20)
R² Rr.
RR = =S",A = =
Pb(S = S'A A²)
19

---

Published as a conference paper at ICLR 2026
for j < k. Now, recognize that, for all second terms in the brackets, yield
En, = R- (++1)
Σ
7 En (RilSo = =8 yym, (5') c0.
(43)
t=
and, analogously, for the final terms, we yield
Σ E. [R]S] = 5' A, = A = Σ
(RelSu A' A - (
as ('),
oo.
(44)
t=
Recognizing that, in the limit, the brackets are equivalent, find the limit of the whole expression to be
IF(Q=, (s0, 00)
= IF
lim Ex, k0:30
R so. Ao aol. aol.
(45)
= -(01x0 8', - (o((S)m
A
Ti. ('))S)))) (5 No. an) { {1' + yuu, (o.. -5.,55,
(A))
(46)
Plugging the result into the EIF of L2,,, obtain
[F (n.9) (),)),0) 0),
(47)
(Qn, (1 a)) L (n.g) 2f10 (8)) (5,()))11))
ne (A'(S))
(48)
(A(11))
(8' A) g(S', A') " Ba (s. g(s.
(49)
DEBIASING THE L.
Applying the EIF to debias the ERM estimate of the population risk, we obtain debiased loss
1 (v. = Boram [Lt (v. IFL (n 9).
(50)
Bor { Σ =e(alss) (Qr, (S', (S',a)
g(S' a))² + Rt' (8') Qx (5', (d(1'))))
(51)
Tn (A'1')
x (rc(5,, - A') F (ea (s,r, g(x, ol)e [s,]]]]
(52)
We complete the squares to obtain a final loss:
12 (n.9)
argmin
1 (n.9)
(53)
(2 (20(A' ( a) {R' + 70x, Qu. (8, A3) )))+ (x, s(8r )1
=po
Σ Te(a|s")
(54)
+EO -pppepps())
Σ ne(als)
(2 " Rt mu Q (5', Qu, -
g(s,a)
(T'''')
(55)
This completes the derivation. The corresponding proof that L is minimized by Qu, can be found in Appendix D.5 We continue with the proof that L is
D.1.1 INTUITION BEHIND Ø1 AND d2
Both Φi and Φ are the respective targets or "pseudo-outcomes" of the MSE subcomponents of the Neyman-oOrthogonal loss. Each contains the Qr, term, which, if alone, would correspond simply to the standard MSE loss without debiasing. This is the non-Neyman-orthognn that we aim to debias order obtain robustness wrt. Nuisance estimation error. The additional debiasing terms of both Φ1 and Φ are two variations of temporal difference error (curly brackets It' 7y (5*) Q- (S', A')) scaled by an importance-sante sawpnnineeaaiingg-g-.... like density ratio.
starting point
While the density ratios here are quite complicated, the overall form of the -othann- loss not. The debiasing being of the form "mean zero error scaled by density ratio" common across
20

---

Published as a conference paper at ICLR 2026
many instances of DR learners in standard causal inference, including but not limited ATE and CATE estimation.
In Φ1, the density-ratie is the simple one-step (first-step) inverse propensity weighting. In d2, the density-ratio is of the conditional stationary state density. Borrowing from Markov chain terminol- ogy, 2 (=(AA")")
(A4187)) We 8, s,a) is the ratio of the likelihood of observing the state-action pair ( following from the stationary distribution the chain induced by following policy Te (conditional on the chain having begun at pair 8, a) versus following from the initial distribution following policy Wb.
D.2 NETMAN-ORTHOGONNAIIY OF 3
First, we state a useful Lemma:
Lemma (Expected TD error zero). The expectation of the temporal difference error Tf, w.r.t to any mea- surable distribution in the model (i... the distribution generated by any policy x), weighted by any (measurable and bounded) function f [S', A') zero.
Ex [(S (H' (orma( (055--Qua. Que (S,
(56)
Proof.
E. ((SS,A') (R' + tvs Qu.,"
(57)
-E. En [(S" [t' + Tun. (5") Qr, (5', (5' sAs
(58)
-E., [(S' (E. Rt' + (8') 5r'''' Qr. (S))
(59)
E., [S(S',A') (S' Qn. (=' [5))] =0
(60)
PROOF OF
Proof. We show the Neyman-orthogonality of our loss 1 We define
Ag(-) g(-) - 9 (.))
(61)
The first (Gateaux) derivative
1. (g.9" 109 9-]=
(62)
2E (m (AsSS a) (2
= a) {et' + tun, (5') - (5. + (S, 9
- (0((
(63)
(2 Te(A'15)) T)(A)()))) (e { (5)) Qn. (8', Qx (8, g (s,
-Bot-pp
Al(s.a)
N2) (8,
(64)
Continuing, take second derivatives with respect all the nuisances 17 (No, We/b, Qtc). First, for ma yield
Dr, DyL2 (/ gi TT)
(65)
am A ()ASs)))AA((((S))254 = {1' + yu,, (S') Qr (5n (-1)
To (A'(s'))
(66)
2E -poo--pppaa---pp0)0.00..
(8, {R' + vrm. Qa ('), (-1)
(A( (A)))))2
(67)
=0.
(68)
21

---

Published as a conference paper at ICLR 2026
We use Lemma 3 to show equality zero. Second, for we/b. we yield
Dwaja Da Py (n - g¹, 10/b - we/6l=
(69)
= 26 O' pp (0 (e(((8))
(((A11)))))))))) {R' + (9) Qr -0. (8', A7] Aus 1
(70)
(A(15))))
=0
(71)
Lastly, for Qu we have
DQ... DyL (n. 9¹, 20. - Q2al=
(72)
= 2o a A0(5", (2 (oo(A(19))
(2(('== (1 (8', - (5', + A AQ. (8', ,a)
(73)
-0000-ppp.....p000000001 (T) " (a) (F ( (8', SQ (8', AA)
(74)
+ (n. (8,0)
(75)
-- (111) AO(S" a) 6(A' = a)
Tr(A'(S') ( (AQ., (.,,
(76)
(²))
(8) (1 A')] 2Q ( .11)
(o (²)
(77)
[ (3
(5',
AQ=, (S', A
5 (S'1s) w ("pp""""" A')
(78)
=0.
(79)
= Since 7 + +5) _ 5", the distribution of S' in the final expectation is (s( 1-m Σ Pep: S'1So = s, An = a). which can be interpreted as a conditional discounted stationary state distribution.
Hence, L is Neyman-orthogonal
22

---

Published as a conference paper at ICLR 2026
D.3 QUASI-ORACLE EFFICIENCY
We prove our loss achieves quasi-oracle efficiency. We write L as
The (7:9) = F0'mppa--,.((18)) [( - a(S" + For tow-spssooottthaaoo ((d2 a(s,,)))],
(80)
where we define
2 8(A' = a)
{I' + -
- (5. (5' ,')) R'+ + yux. (5") (5' A + (s,
.(S)
(81)
(A'(S))
Φ Re (A'(5')
(s(''S)") We as {R'
(82)
Additionally, we repeat the definitions arg T (7, and و" arg sco 1.2 (1, where i are the estimated nuisances and are the (unknown) true oracle nuisances.
So, we now arrive
L (i. 9) E'' pp (
(83)
+ Bo -or ( ((. 9s)
(84)
L (f.9") + 2p [(9 (s. 0(. ()){ + DyL (f) [Aa).
(85)
where we obtain the last line by decomposing the square and recognizing terms. Rearranging, we see
211g* - oll. = Rg -Dy (f)9') (
(86)
where Hg - (9. - L. (9. 9").
We now arrange DyLi, (f) via second-order Taylor approximation to the true n, i.e.,
DyL2. (1.9 [ DyLi. (n. ""agg"""([ [Aál
(87)
D,, (n. [A9, Arl (= by Oetaoon-Ctt )
(88)
1 L. (7. ,') Ar, Arj.
(89)
for some E star(H, 1),, where denotes the star-shaped set with respect to The last term is of the form
12 (7. [AA Ar, An)=
(90)
2E -ppo...ppp...m))s. A)(s " Vmrzan
(91)
Continuing, we then have
2||s* oll..o... = Ry - 2,L3, (n.99) Ag] " (A,
(7.9
(92)
< Rg D2 (n,gg) " An, Ani
(93)
< Rg + 119" 31lpo
Σ
F [(AM] A
(94)
< Rg + 119" - -
Σ F.j.k
8igk
+
E V
dizt
(95)
where we achieve the first inequality by recognizing D (n. [Ag]] 2 0, the second using the Cauchy-Schwarz inequality, and the third using the AM-GM inequality for any constants dijk
23

---

Published as a conference paper at ICLR 2026
such that E. 8esk <. This then finally results the inequality
Rg RR+EEOOSA*H +33A +33 02 [m
2|18* 0112,p07.
-2- 8igk
(96)
$ 2 Eisi 8ejk
Rg+ 11 IICsA 11 11Cra
(97)
2 2 Eing degk
Rg R1
|| A. T. 12 A A 10, 2Q+.ell 112
(98)
where the Ci....,Ca C4 collect all terms that do not contain A terms of difference between estimated and true nuisances. In the last steps, æ & y is taken to mean there exists constant M > 0 s.t. S My. The last inequality is achieved by extracting Ař terms from the C's and noting | < ||Аî since îj lies between f and the oracle ny. For clarity of exposition, the computation of the Hessian terms through which the C's contain Anj terms is postponed to the end of the proof. Lastly, we make use of Hölder's inequality
2||g² 912.
(99)
< 2 Eijja 8igk
Rg +
A A AA.Q. 11ă A| I i0 + 1| /
ll
(100)
A 1 1| m. 11 + || /. + 11 1..11 l1
(101)
This finishes the proof. The double-robustness property is proved trivially by plugging in the condi- tion (either AQx. + 0 or Añ, + A1i./1 + 0) into the here obtained bound.
Note on assumptions: The proof of Quasi-Oracle efficiency holds under the standard assumptions of sample-spliting (first and second stage are fit on separate parts of the dataset), i.i.d. data, well- behaved (convex) risk, sufficient convergence rates of nuisances, and boundedness of first moments. Of specific note is the i.i.d. assumptions. which we assume for ease of exposition, while actually only needing a less strict requirement of the empirical expectation concentrating around the exact population expectation. For Markov chain induced by following the policy (a single trajectory), is enough for it to be ergodic. Less formally but more intuitively, we simply need the effective sample size to be infinite in the asymptote.
For completeness, we write out the Hessians of ø's with respect to = (Te, W.//b, TT.)- These terms are all included in the variables C1,..., C4, since they do not include any differences between estimated and true nuisances. We thus have (=sAts(A1/S)AQ (S",
Dın ((A(ss)
V Vmnd
(102)
D (iss (S',
0
Dm1 45(A'= = a) at' (5)) (S) ('))
(103)
D11 ó(A' a)
(104)
(A(
Dan (A'((s")
D212Am (An)))
Vm
Dre3 (S', A')
D218 (A') (''1S') (8',
Doza (8's (8',
(105)
Da1= = A..
(u { n²' + (5') (s: (A(1'1873
(106)
D212 = (A 5') + yr= (5") Qne (8', A7) (8)) To(Arls5y3
(107)
Dna 2Te1S' "aa== =
(108)
D223 = -2 Ss
(109)
24

---

Published as a conference paper at ICLR 2026
where all the constants elements D are evaluated at 7/
D.4 IDENTIFICATION
PROOF OF THEOREM m
Proof.
Sr, (s, R Ro + Σ 7 Re (-([a))
So Ao
(110)
= RT
7 RR So Ao A0
(111)
Qr (8,
(112)
= Eeg
Ro +
7' Piall So = 8, A,
(113)
The first equality follows by definition, while the second equality is by consistency and unconfoundedness assumptions, and the final equality by the weak positivity assumption.
Technical remark: For the last step, we must assume the rewards are bounded, RR S Rmax such that we can apply the dominated convergence theorem take the infinite sum the expectation, apply importance- sampling style change of distribution element-win wise to each Rd expectation term and then collapse everything into final formula.
PROOF THEOREM
Proof. We prove the identification is valid by showing that Eq. 66) (i) observable and (ii) has unique solution (unique up to equality almost everywhere).
For the question of observability, we first notice that the inner expectation is over known distribution, i.c., the treatment assignment under Te. The remaining randomness is then outer expectation over R, S, condi- tional on S = s, A In the MDP, the reward and transition dynamics are the source of this randomness, meaning this randomness invariant to the policy followed. We can thus freely write the RHS of Eq. 0 as
f(s, a) Br,
A)| s= = A=A=
(114)
And clearly, the RHS is observable.
The uniqueness of the solution of the Bellman equation is well-known result in RL. A rigorous proof of which
available, for example, in (Sutton & Barto 2018). Informally, defining the RHS as the Bellman operator 7 on is shown that this operator is 7-contraction mapping in the space of bounded measure functions on x A. By the Banach fixed-point theorem, this implies that T admits a unique fixed point. Since f = Q satisfies Eq. ( have shown that Qu, is the unique solution (up equality almost everywhere).
O
D.5 PROOF THAT L TARGETS QT
For completeness, prove that I,, is minimized by Que-
Prouf.
We begin by reversing the square completion
(n, argmin 12_ (n.9)
(Q=. (Q. -5) o(S" a))² + 2 Rt' + 7va. (5') Qu (S", A3) (A'lS")
= =Eorap
(A''55))
Q.. (S", g(5", "A'A (ennmm(())aaall))) [[Q (s, - ((s,,)))we/l(555
25

---

Published as a conference paper at ICLR 2026
The proof be completed using Lemma remove the second term from the expectation (using the law of iterated expectations on 5", , arrive L. reversing the construction of L namely that
.A'. With only the first term remaining, we recognize L Alternatively, we can
1 Eo'-ps [Lt, (n.9) + [I (7) g). O`)] Bo'-po [I (17 9)] I. (1,
(115)
since efficient influence functions are mean zero by definition. Finally, showing that Qr, minimizes L. is trivial.
E IMPLEMENTATION DETAILS
https:///itthu: com/ Javurek/ Orthogonal-Q- M Upon acceptance, we move our code to a public GitHub repository. All experiments implemented in the Taxi environment from the OpenAl Gym package (Brockman et al. 2016). Since the focus of our work is on second-stage estimation, we take the ground-truth oracle for the density ratio nuisances, while the first stage Q is estimated for each method. We list all relevant hyperparameters in the following table. All experiments were conducted for runs with different seeds.
Anonymous code
available
Component
Hyperparameter
Value
0.9
Taxi environment
max steps
100
episodes
5000
Online Q control (to construct policies via Q*)
e 0.1
0.05
:
c
0.5
Te
6
0.1
Dxr.
3000
Ground-truth reference QT.. online Expected SARSA prediction
episodes
100000
0.9
oracle
1st Stage
we/b
oracle
Q1.
FQE
DR- DR-learner
iterations
1000
FQE
iterations
50
Q-regression
MQL
iterations
500
Table 1: Hyperparameter settings for experiments in the Taxi environment.
26


Reviews:
[{"review_id": "AvkdAUtJfB", "reviewer": "Reviewer_9GGk", "summary": "The authors used a two-stage approach based on the nature of double machine learning to construct a semi-parametrically efficient estimator for the Q function under the MDP setting, which is helpful for individualized and robust decision making. The authors claim novelty by extending existing work that mainly focuses on value function estimation or is not doubly robust, to the case of doubly robust Q function estimation. The corresponding theory is provided in detail.\n\nOverall, this is a well-structured paper, but I find it lacks theoretical novelty (or at least does not emphasize it clearly) compared with existing papers, given that DML and doubly robust OPE have been extensively studied in the last 5-10 years. I think it is a good practice of applying these theories to Q function estimation, but it may not be solid enough for publication at ICLR. I would vote for a weak reject, and I believe this paper would be a good fit for theory-extension journals or conferences, although I’m open for discussion and won’t hold a very strong opinion.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well structured and carefully written. The flow is clear, and the theoretical results (with a not very detailed look) generally look fine.\n\n2. For estimation construction, the authors propose a two-stage estimation that can get rid of modeling class restrictions as in Shi et al. (2021). I think this part is novel (to my knowledge), and the authors justify the performance and doubly robust property well through simulation studies.", "weaknesses": "1. In general, the contents before Section 4.2 are well established in the related literature. In a standard MDP setting, it is not surprising to see these results. When I first read it, I expected some new perspectives, but it turns out the new contributions mainly start from Section 4.2. For example:\n\n- Line 128–130: The authors claim that \"these works do not focus on the MDP setting and are well known to suffer from the curse of horizon.\" To my knowledge, some of the works listed in this paragraph (such as Shi et al., 2021) are developed under the MDP framework. Maybe the authors meant that 'some' papers in this paragraph are not based on the MDP setting, but the sentence is somewhat misleading and should be rephrased.\n\n- Line 251–256: \"This causal lens, to our knowledge, has not been made explicit in estimating Q-functions.\" I think this causal perspective is already well established in related work, including papers the authors cited such as Uehara (2022), and has been used in many extensions studying settings with causal assumption violations (e.g., confounders, interference, etc). Thus, I don’t think the identification results are new or particularly contributive for this paper under a standard MDP setting.\n\n2. Based on point 1, I think the authors should highlight more clearly the theoretical challenges in establishing semi-parametric efficiency for the Q function compared with existing work (which already has such results for the value function). I don’t think there is a strong theoretical difference, but if there is, it should be emphasized to strengthen the contribution.\n\n3. Regarding estimation, although I appreciate the contribution of proposing a two-stage estimation process that can handle nonparametric function classes for DR-Q learner, I would like to hear more about the challenges when building upon existing work and how the authors address them.\n\n4. For the simulation part, since the authors highlight many comparisons with Shi et al. (2021), it would be great to also show comparisons with their paper on value function estimation. Because the proposed DR-Q learner is more flexible, showing this comparison would further strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e37LAzw4gt", "reviewer": "Reviewer_MFXq", "summary": "The authors address the problem of estimating individualized potential outcomes in sequential decision-making (i.e., estimating Q-functions in Markov decision processes with observational data) through a causal inference lens. They propose a novel meta-learner, termed the DRQ-learner, which enjoys beneficial theoretical properties: double robustness, Neyman orthogonality, and quasi-oracle efficiency. Empirically, the paper demonstrates that the DRQ-learner consistently outperforms baseline Q-regression and Q-learning methods in terms of the rMSE metric across a different settings.", "questions": "1. The authors mention the “curse of horizon” multiple times. Could you provide an explanation of this phenomenon and briefly clarify why it arises in other estimators? Additionally, do other baseline methods in the experiments also suffer from this issue?\n\n2. In Theorem 3, could the authors provide more intuition behind $\\phi_1$ and $\\phi_2$? Moreover, the connections among Eq (13)–(15) and Eq (16) are not clearly explained. The proof here is not clear and requires more interpretation.\n\n3. In Figure B3, DRQ does not perform best when $\\epsilon$ is large. How to explain it?\n\n4. There appear to be a few typos: a misplaced comma in Equation (15); inconsistent indices used in Eq (1): first $T_i$ and subsequent $nT$; and a typographical error (“si” → “is”) in Appendix C.4, line 1283. Besides, Several equations in the Appendix are too long to be fully displayed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Novelty intersection causal inference & orthogonal learning and MDPs.** The paper creatively bridges causal‐inference, orthogonal learning with off‐policy Q‐function estimation in MDPs. By leveraging orthogonal meta‐learner ideas, it provides a fresh perspective on learning in Markov decision processes.\n\n**Theoretical guarantees.** The authors present comprehensive theoretical analysis, including identification from multiple perspectives, derivation of efficient influence functions, and construction of an orthogonal loss. The resulting estimator is shown to possess several theoretical properties.", "weaknesses": "**1. Motivation for the MDP framework.**\nWhile the proposed orthogonal learner is formulated within an MDP setting, the motivation for adopting this framework and the necessity of the Markov structure remain unclear. It would be helpful to clarify what advantages or practical challenges the MDP formulation addresses compared with alternative settings, such as DTR or contextual bandit settings.\n\n**2. Interpretation of theorems.** \nThe paper includes several lengthy and dense formulas. Some theoretical results would benefit from additional intuition and clearer interpretation to help readers understand the key ideas and implications behind the derivations.\n\n**3. Experiments.**\nThe experiments demonstrate that the proposed DRQ estimator outperforms strong baselines. However, several theoretical results, such as double robustness under misspecification of $\\pi_b$ or $w_{e/b}$, are not empirically verified.\n\nSee also questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zhdVsmAgxH", "reviewer": "Reviewer_V1QJ", "summary": "This paper addresses the challenge of estimating individualized potential outcomes in sequential decision-making, specifically the off-policy estimation of Q-functions in Markov decision processes (MDPs) using observational data. The authors introduce a novel meta-learner called the DRQ-learner, which is based on principles of causal inference and semiparametric efficiency theory. The key contribution of this work is a theoretically grounded framework that produces an estimator with three important properties: double robustness (the ability to provide valid inferences even when the behavior policy or the density ratio is mispecified), Neyman orthogonality (resilience to first-order errors in estimating nuisance functions), and quasi-oracle efficiency (asymptotic performance that matches that of an oracle with access to the true nuisance functions). In contrast to previous methods that either experience plug-in bias or struggle with the \"curse of horizon,\" the DRQ-learner utilizes a debiased loss derived from the efficient influence function. This allows for stable and accurate long-horizon estimation, even in scenarios with low policy overlap. Additionally, the method is flexible, capable of handling both discrete and continuous state spaces, and supports various machine learning models, such as neural networks, for estimating nuisance functions.", "questions": "- What are the standard assumptions in Theorem 4?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is strong due to its solid theoretical foundation and its clear synthesis of ideas from causal inference, orthogonal statistical learning, and off-policy reinforcement learning. The authors effectively reframe Q-function estimation as a causal estimand and identify the limitations of existing plug-in approaches, such as Q-regression and FQE, by examining plug-in bias. They also provide formal proofs of the key properties of their method. Empirical results from the Taxi environment support the theoretical claims, demonstrating consistent improvements over baseline methods, especially in scenarios with low overlap and long horizons. The findings confirm the advantages of Neyman orthogonality and the approach's applicability to both unrestricted and restricted model classes.", "weaknesses": "- While the theoretical guarantees are strong, the practical implementation relies on accurate estimation of complex nuisances (e.g., the stationary density ratio), which may be challenging in high-dimensional or continuous settings despite the orthogonality property. \n\n- The experiments are limited to a single, relatively simple environment; broader validation across diverse MDPs (e.g., with partial observability or real-world medical data) would strengthen the empirical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKkNh05oQz", "reviewer": "Reviewer_Pa5y", "summary": "The paper studies off-policy estimation of the full Q-function  from trajectories generated by a different behavior policy. The authors propose  DRQ-learner, a two-stage learner: Stage-1 fits “nuisance” components; Stage-2 minimizes an EIF-based loss to debias the final Q function. Theoretically, they claim Neyman-orthogonality, double robustness, and quasi-oracle efficiency. Empirically, on Taxi-like settings they vary data size, horizon, and overlap and report that DRQ outperforms plug-in baselines.", "questions": "1. Theorem 4 uses $\\Delta k$ without defining $k$; seems like a leftover typo and should be fixed.\n\n2. What exactly is the measure $p_b \\pi_e$ in the theorem 4? Why is this weighting natural for your analysis?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The EIF-based orthogonal loss directly targets $Q^{\\pi_e}$, and cancels 1st-order nuisance error, well-motivated.\n2. The paper is well-organized and clearly written.", "weaknesses": "1. Most results are on Taxi-like setups; no continuous-control tasks.\n2. The paper varies $\\epsilon$ in $\\epsilon$-greedy policies but never provides a simple numerical overlap indicator, so readers cannot tell how low the overlap actually is.\n3. Several results appear to use oracle or near-oracle nuisance components, so the practical robustness of Stage-2 under ordinary estimation errors is not fully demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)