Doubly Robust Uncertainty Quantification for Quantile Treatment Effects in Sequential Decision Making
Authors: Yang Xu
Chengchun Shi
Shikai Luo
Lan Wang
Rui Song
Venue: Accepted by TMLR ()

Abstract: We consider multi-stage sequential decision making, where the treatment at any stage may depend on the subject’s entire treatment and covariate history. We introduce a general framework for doubly robust uncertainty quantification for the quantiles of cumulative outcomes under a sequential treatment rule. While previous studies focused on mean effects, quantile effects offer unique insights into the distributional properties and are more robust for heavy-tailed outcomes. It is known that, doubly robust inference is significantly more challenging and largely unexplored for quantile treatment effects. More importantly, for mean effects, doubly robust estimation does not ensure doubly robust inference. Our approach first provides a doubly robust estimator for any quantile of interest based on pre-collected data, achieving semi-parametric efficiency. We then propose a novel doubly robust estimator for the asymptotic variance, enabling the construction of a doubly robust confidence interval. To overcome the challenges in parameter-dependent nuisance functions, we leverage deep conditional generative learning techniques. We demonstrate advantages of our approach via both simulation and real data from a short video platform. Additionally, we observe that our proposed approach leads to another mean effect estimator that outperforms existing estimators with heavy-tailed outcomes.

Published in Transactions on Machine Learning Research (09/2025)
Doubly Robust Uncertainty Quantification for Quantile Treatment Effects in Sequential Decision Making
Yang Xu Deportment of Statistics North Carolina State University
yru63@ncsu.ddv
Chengehun Shi
c.shi70lsc.co
Deportment Statistics London School of Economics and Political Science
Shikai Luo ByteDance
sluo198122@163 10m..commmmmmmm....cmmm
Lan Wang
lonwang@@mbs.minn www.::/mmmmmmmmmiooomm edu
Miami Herbert Business School Universily of Miana
Rui Song
songray@ggaillccom com
Department Statistics
North Carolina State University
Reviewed on OpenReview: https: //wpr/vveeeeee..eem net/ forum? FOBub se Vus
Abstract
We consider multi-stage sequential decision making, where the treatment at any stage may depend on the subject's entire treatment and covariate history. We introduce a general frame- work for doubly robust uncertainty quantification for the quantiles of cumulative outcomes under sequential treatment rule. While previous studies focused effects offer unique insights into the distributional properties and are more robust for heavy- tailed outcomes. It is known that, doubly robust inference is significantly more challenging and largely unexplored for quantile treatment effects. More importantly, for mean effects, doubly robust estimation does not ensure doubly robust inference. Our approach first pro- vides a doubly robust estimator for any quantile of interest based on pre-collected data, achieving semi-parametric efficiency. We then propose a novel doubly robust estimator for the asymptotic variance, enabling the construction of doubly robust confidence interval. To overcome the challenges in parameterrdeepedeent nuisance functions, leverage deep conditional generative learning techniques. We demonstrate advantages of our approach via both simulation and real data from a short video platform. Additionally, we observe that our proposed approach leads to another mean effect estimator that outperforms existing estimators with heavy-tailed outcomes.
effects, quantile
1 Introduction
Treatment effect evaluation plays an important role in a large variety of real-world applications. In precision medicine (sce c-g., Murphy 2003 Kosorok Laber 2019 Tsiatis et al.] 2019),, physicians wish know the impact of implementing an individualized treatment strategy. In technological companies, decision makers conduct A A/B testing evaluate the performance of newly developed product (see e.g., Bojinov & Shephard.

---

Published in Transactions on Machine Learning Research (09/2025)
2019 Shi et al. 2023 Sun et al. 2024a Wen et al. 2025 In economics and social science, program evaluation methods are widely applied to access the effects of certain intervention policies (see Abadie & Cattaneo 2018
for an overview). In the aforementioned applications, a new treatment decision rule and/or policy need to be evaluated offline before online validation. Over the past docade, offline evaluation of a fixed, data-dependent
or optimal policy has been widely studied in statistics (Zhang et al., 2012 Chakraborty et al.] 2013
Zhang
ct al.
2013: Matsouaka et al., 2014, Luodtke & Van Der Laan [0066] Luckett et al., 2020; Liao et al.
2021:
2022
Shi et al.] 2022 Ramprasad al. 2023] Wang et al.) 2023 Zhou et al. 2023a Shi al.)
2024b
Bian
et a..
2025) and machine learning (Dudik et al.
2011 Jiang & Li 2016
Thomas & Brunskill 2016 Xie
ct al.
2019;
Cai et al. 2020 Dai et al. 2020 Kallus & Zhou 2020
Kallus lus & Uchara
2020
Namkoong
et al.
2020 Shi et al.) 2020;
Hao et al.) 2021 Shi et al. 2021a Chen & Qi 2022
Xu et al.. 2023
Bossens &
Thomas
2024 Yu el. al.
2024 Zhou et al. 2025).
Nonetheless, all the previously mentioned works have primarily focused on the mean outcome, which may not fully capture the distributional effects of a policy, such as its impact on specific quantiles or the entire outcome distribution, are of primary interest. Take our real-data application as an example. The data is collected from a world-leading technological company with one of the largest mobile platforms for production, aggregation and distribution of short-form vidcos. Wc aim to assess the impact of specific advertising policies on user experience, measured by the length of stay on the platform. A particularly important subgroup is the passive users, who exhibit. little responsiveness to any advertising policy and tend to spend minimal time on the platform. These users are concentrated in the lower tail of the engagement distribution, around the 25th percentile. Policy-makery are particularly interested in addressing the needs of this subgroup, since treatments with greater impact on the lower quartile are more likely to attract potential customers who were previously less engaged with the platform.
In addition to short video platforms, evaluating the distributional effects of a policy is crucial in many other applications as well. For instance, in welfare programs, policy-makers may wish to identify interventions that spocifically reduce the cost of living for the lowest 10% individuals, rather than focusing on the average population (Cui & Han 2023). When studying gender discrimination, researchers often care about the entire distribution of wages, social visibility, or media coverage, rather than solely comparing mean differences between men and women (Firpo) 2007). In microbiome rescarch, sequencing reads (Wang et al. 2025), , and in e-commerce, advertisement clicks profit (Wei et al. 2023 are typically non-negative and zero-inflated. In these cases, the mean can be a misleading summary statistic, making it more appropriate and informative to focus on quantiles.
Even when the mean outcome is of interest, traditional sample average estimators can perform poorly in the presence of skewed or heavy-tailed outcomes (Rowland et al. 2023 Zhu et al. 2024 Behnamnia et al. 2025). To illustrate the prevalence of heavy-tailed outcomes in practice, Figure ! visualizes the distribution of the outcome variable in our real data example- user watch time using Q-Q plot and log-log frequency plot. The Q-Q plot reveals clearly right-skewed distribution, while the log-log plot displays an approximately lincar decay in its probability density function, consistent with a power-law tail distribution - such as the I-distribution commonly used to model heavy-tailed outcomes of the form x=a, As we will show later in Section [53] when outcome variables are heavy-tailed, computing quantiles first and then averaging results in a much improved estimator for the mean. This further motivates us to study quantile treatment effects.
This paper focuses on developing doubly robust and semn-parrametrcalll efficient statistical inference frame- work for the quantiles of a target treatment's return using pre-collected datasets from multi-stage studies, which has been less studied in literature. In causal inference, the estimation of the causal effect often depends on two nuisance parameters, one for the propensity score and another for the outcome regression model. An estimator is doubly robust if it is consistent. for the causal effect of interest. when any one of the two nui- sance parameters is consistently estimated. While doubly robust estimation has been extensively studied for mean-based estimators, our work addresses two key difficulties that set it apart from existing studies: (1) estimating quantiles rather than means, and (2) conducting rigorous statistical inference beyond point estimation. These additional complexities make our approach both novel and technically demanding.
The challenges of conducting treatment effect inference quantiles are twofold. Firstly, constructing a doubly robust confidence interval requires doubly robust asymptotic linearity, which is more stringent than
2

---

Published in Transactions on Machine Learning Research (09/2025)
QQ plot
Log Log frequency plot
12.5
Normal Raference Line
Sample Quanitas
10.0
12
7.5
10
2.5
0.0
-25
5.0
.75
-2
0
3
-1.5
-1.0
-0.5
0.0
0.5
1,0
1.5
2.0
2.5
Theoretical Quantiles
Log of values
Figure 1: Left: Q-Q plot that compares the sample quantiles of user watch time in our real dataset to the theoretical quantiles standard normal distribution. Right: Log-log plot showing the count of data points at each user watch time value, grouped by percentiles. Both plots highlight the heavy-tailed nature of the outcome distribution in our dataset.
doubly robust consistency and has been less studiod. Benkeser et al. (2017) provided thorough discussion of the challenges associated with doubly robust inference for the average treatment effect at single time point. In particular, they revealed from careful theoretical analysis that when flexible, data-adaptive estimators of the nuisance parameters are used, doubly robust estimation does not readily extend to doubly robust inference. They showed that some natural alternatives are inappropriate for this purpose and further proposed a doubly robust inference procedure via targeted minimum loss-based estimation, which, however, requires additional nuisance parameters. Secondly, the efficient estimating equation for the quantile value is nonsmooth and involves parameler-deeennden conditional mean function estimation which requires learning the conditional mean function for various values of the target parameter (Kallus et al. 2019). Directly applying supervised learning algorithms to construct the efficient estimating equation is very computationally intensive. This problem becomes more challenging when considering multiple-stage scenarios with history- dependent treatment.
Our contributions can be summarized as follows: (i) We propose a comprehensive doubly-robust uncortainty quantification procedure for quantile treatment effect in sequential decision making. In particular, we develop a doubly-robust quantile value estimator and a doubll-robust estimator for its asymptotic variance. Semi- parametric efficiency established to guarantee the performance of our quantile estimator, To the best of our knowledge, this is the first work that rigorously investigates uncertainty quantification for target policy's quantile value in multi-stage studies. (ii) We propose utilizing existing state-of-the-art deep condit nuisance function estimation, which circumvents
generative learning methods to handle parameter-dependeee the error aggregation commonly encountered in standard iterative estimation methods prevalent in sequential decision making. (iii) We further develop a tail-robust doubly-robust estimator for the mean outcome by aggregating the proposed quantile estimator at multiple quantile values. Our simulation study demonstrates that the mean squared error (MSE) of the proposed estimator is around 15%-50% less than that of the standard doubly-robust mean effect estimator with heavy-tailed responses.
1.1 Related work
In the literature, several methods have been proposed for evaluating treatment effects beyond the mean. One line of work focuses quantile distribution regression (Machado & Mata 2005 Chernozhukov et al., 2013). Another line employs inverse probability weighting, using propensity scores to reweight each outcome in order to estimate off-policy quantile treatment effects (QTE, Firpo 2007) or cumulative distribution functions (Donald & Hsu 2014). More recently, Kallus et al. (2019) proposed a localized debiased machine learning approach for QTE estimation, while Muandet et al. [2021) introduced a nonparametric method basod on kernel mcan embeddings. Huang et al. (2021) further extended prior analyses from mean outcomes distributional metrics in contextual bandit settings. Wu et al. (2023a) proposed a non-crossing neural network architecture for learning distributional individualized treatment. effects. A recent concurrent work
3

---

Published in Transactions on Machine Learning Research (09/2025)
by Cheng et al. (2024) also investigated the quantile treatment effect under marginal structural quantile model. Our work differs from theirs by further establishing doubly robust estimator for the variance.
A related but different line of work focuses on policy learning rather than treatment effect estimation. For example, Wang et al. (2018a) and Qi et al. (2019) studied optimal policy learning to maximize distributional quantities other than the mean outcome. Later, Leqi & Kennedy (2021) considered optimizing the average of conditional quantiles to promote "across-group fairness". More recently, Cui & Han (2023) proposed minimax policy learning that robust to model uncertainty. Despite these developments, both strands of research whether focused on policy evaluation policy learning have largely been confined to single-stage decision making settings, limiting their applicability to real-world problems that require sequential decision making.
Some recent. works have explored distributional evaluation and policy optimization in sequential decision making and reinforcement learning (see e.g., Dabney et al. 2018 Zhou et al. 2020 Huang et al. 2022 Li et al.] 2022 Wu al.] 2023b Zhang et al. 2023 However, most of the alorementioned work focuses on point estimation of target. quantiles, with limited attention to inference procedures. Our contribution to this line of research is to provide a doubly robust inference procedure for quantile-based policy evaluation.
al. 2024 Sun et al. 2024b Shen et al.) 2025)
1.2 Paper Organization
The rest of the paper is organized as follows. In Section 2 we introduce the doubly-robust estimation and inference procedure for single-stage decision making. In Section g we formally establish the framework to the multi-stage setting with detailed algorithm and estimation procedures. In Section 4 we study the asymplotic properties and semi-parametric efficiency of the proposed procedure. Simulation ion studies and real data analysis are conducted in Sections 5 and G respectively. Finally, we conclude our paper Section All the proofs are presented the appendix.
2 Single- Decision Making
2.1
Preliminaries
We first describe the observed data. Let X E X' denote the d-dimensional vector that summarizes the baseline information (e.g., age, gender) a given subject before treatment assignment, let E A = -I. mb denote the treatment assigned to that subject where m denotes the number of treatment options, and let R R denote the subject's outcome (the larger the better by convention). The obscrved data can thus be summarized N i.i.d. Dvarriales-rreattent--oocco triplets (X,, Ai, R.) where N denotes the number of subjects.
We next introduce the potential outcomes. For any 1 5 a S m, let (a) denote the potential outcome had the subject received the treatment. a. Let : X x A + [0 denote the behavior policy that generates the data, Le., b(a]z)= = IP(A- = a|Xx = x). Likewise, let π : X A + [0, 1] denote the target policy we wish to evaluate. Notice that b is allowed to be unknown, as in observational studies.
Finally, introduce the following standard assumptions:
(C1) Consistency: 12" (A).
(C2) No unmeasured confounders [Rosenbaum & Rubin 1983 A (R"(1), .., (I
(C3) Positivity:
There exists some constant e <> 0, such that P(b(A[X) 2) 1
We remark that (C1)-(C3) are commonly imposed in causal inference and individualized treatment regimes literature (soe c.g., Zhang et al., 2012 Wang et al.) 2018a Nic Wager 2021). They allow us to infer the potential outcome distribution from the observed data. (C2) and (C3) are automatically satisfied in randomized studies where the behavior policy usually strictly positive function independent of x.
4

---

Published in Transactions on Machine Learning Research (09/2025)
2.2 Quantile Treatment Effect evaluation
Let R²(z) denote the potential outcome variable following the target policy и. For given quantile level < 7 1, our problem of interest to infer the rth quantile of (x) marginalized over the initial state
distribution X E G, denoted by V/- Specifically, we have P(R (T) S 1)r) Ex-s [P(R (Tt S nr [)][X)] For given target policy the 7th quantile of (T) can be decomposed to a mixture of {R* (a) 1 S a S m} with weights specified by æ. As such,
(x) ST|x)
Σ =(a Sr|)).
(1)
Under (CI))(C3), the conditional potential outcome distribution PR- (a)(x is equal to the conditional distri- bution observed outcome PRIX,A- This together with Equation T allows us to identify the quantile value from the observed data. In the following, we first introduce the direct method and the inverse probability weighting method, and then combine the two for efficient and robust estimation of the quantile treatment effect.
2.2.1 Direct Method (DM)
The first estimator is provided by a direct method, where we integrate quantile regression with Equation to devise an optimization- based approach for quantile estimation. The methodology is conceptually equivalent the g-computation formula. used in causal inference (Daniel et al., 2011 Snowden et al. 2011 Chatton & Rohrer 2024). Let pr pr(u)= u. (7 1--77-1f0 1(u 0}) denote the quantile loss function. By the law of iterated expectation, 7/ arg min, E[p, (R n)] arg min, B[B[p, (R)()) (i) []] Under (C1)-(C3), it follows
that
MD arg min
Σ
B[T(a|X) Blp ( 10001aa..
= XHI
(2)
where E denotes the empirical expectation calculated from the observed data. DM first estimates the con- ditional expectation within the square brackets for each n7 and then plugs in this estimator in the above optimization problem to learn 1/--
As discussed in the introduction, it remains challenging to compute the conditional mean estimator for each y.. We will later introduce our proposal to address this issue.
2.2.2 Inverse Probability Weighting (IPW) method
An inverse probability weighting estimator, akin to the off-policy mean outcome estimation in classical causal literabure, can be developed by the change of measure theorem based on Equation
T(A ((A[X)
P+ - (n)
NPw arg min E
(3)
When the behavior policy b is unknown, it can be estimated using any proper supervised learning algorithm with a mild convergence rate. The precise requirements are provided in Assumption (A1) and Remark 1.
2.2.3 Doubly Robust (DR) Estimator
In causal inference, an estimator is doubly robust if it is consistent for the causal effect of interest. when any one the two muisance functions, i.e. propensity score or outcome regression function, is consistently estimated. As a result, the doubly-robust method combines the DM and IPW procedures and alleviates the risk of model misspccification by only requiring one of the nuisance functions to be correctly specified. Based on the classical DR estimator for the mean outcome, DR quantile estimator be formulated by
5

---

Published in Transactions on Machine Learning Research (09/2025)
incorporating the fundamental concept of quantile regression as follows:
NDR arg min ? N
L (n; X,, A2, R2, E)
(4)
(AA())) b(A))X,)) Pr (R,
T(A()X))) ((A()),) +1-66A1A))) (R: A, ~ ~m7
(1
arg min 1
In the above formula, 5 represents the estimated behavior policy. [E т denotes the conditional mean functional that needs to be estimated later. Here, the reward R3 in the conditional expectation is random variable by executing the target policy x under baseline information X;.
The first term of the second line in Equation g is identical to the IPW estimator. The second part is an augmentation term which offers additional protection against the model misspecification of the behavior policy while improving the estimation efficiency. When the conditional mean function is correctly specified, the expectation of our DR estimator is equivalent to that of the DM estimator. When the behavior policy is correctly specified, the estimator is unbiased to the IPW estimator and thus also consistent. This verifies that MDR a combination of DM and IPW estimators, indeed satisfies the doubly-robustnees property.
Similar to DM, DR requires estimating the conditional mean function for each 7, We propose employing a deep conditional generative learning method to compute E[- |J. The main idea is utilizing deep neural networks to learn a conditional sampler that takes the covariato-treatment pair and some random noises as input and outputs pseudo outcomes whose conditional distribution is similar to that of PRJ... By using the Monte Carlo method, these pscudo outcomes can be further used to approximate the conditional mean functions for any 7). This addresses the challenge of parameter-dependent nuisance function estimation. As powerful approach for generating random samples from complex and high-dimensional conditional distributions, deep conditional generative learning methods have been successfully implemented in a large variety of applications including computer vision, imaging processing, epidemiology, time scries analysis and
machine learning (Yan et al. 2016 Shu et al. 2017 Wang et al. 2018b Bellot & van der Schaar
2019 Davis
et al.
2020
Jo et al.] [2021] Shi et al. 2021b
Zhou et al., 2023b Shi et al. 2024a Zhang et al. 2024 Yang
ct al.
2025
We detail our estimation procedure in the next section.
2.3
Estimation and Inference Procedures
Our estimation procedure consists four steps: 1) sample splitting, 2) muisance functions estimation, 3) doubly-robust quantile value and variance estimation, 4) tail-robust mean value estimation. Pseudocode summarizing the proposed algorithm is given Algorithm
2.3.1 Step 1: Sample Splitting
Step 1 of the proposed algorithm is to randomly divide samples [1....,N} into S disjoint subgroups of equal size. We define n = N/S and assume n E Z without the loss of generality. This step incorporates Equation with cross-fitting in Step 3 to reduce the bias the nuisance function estimators obtained by the aforementioned supervised and generative learning methods. For any subgroup and I, denote its complement. Sample splitting allows us to use the data in Iς compute b addition, we can get full efficiency by
S S, let I, denote the sth
and [{.), and those in I, to construct the estimating equation. aggregating the estimating equations over different 8. Such a technique has been implemented for policy evaluation in recent literature (Chernozhukov al. 2018 Shi al.) 2021a Kallus & Uehara 2022).
2.3.2 Step 2: Nuisance Functions Estimation
Step 2 of the proposed algorithm is to use each data subset in Ie to estimate the behavior policy and the conditional distribution of R given A and X.
Firstly, notice that estimating the behavior policy is essentially regression problem. As such, we can apply any supervised learning algorithm designed for categorical response variables to compute b. In
6

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1 Pseudo Code of Quantile Treatment Elfect Evaluation in Single Stage
1: Input: {X,, A,, Ra triplets; quantile levels r; number folds S. Output: doubly robust quantile estimator R tail-robust DR mean estimator JDRR
procedure A. DOUBLY ROBUST QUANTILE ESTIMATOR
Randomly split the data into S folds. For any S, denote I, the sth subgroup. for all E S do Estimate b by GBDT with the data in Ig: Estimate R (X, A) by MDN with the data in I;:
6:
Generate {R2 2,0 (X = Fz. = 1 for any a E A and E I,:
end for
10: Solve the optimization problem in formula (4) to obtain 7/n.
11: end procedure
12: procedure B. TAIL-ROBUST DR MEAN ESTIMATOR
13: For all of the quantile levels of our interest, calculate 7r by Procedure A. Averaging among all quantiles
yields the tail-robust DR. mean estimator ĀDD.
14:
end procedure
implementation, employ Gradient Boosting Decision Tree (GBDT) algorithm and find that it works well in our numerical examples.
Secondly, we need to estimate the conditional expectation one approach is to apply supervised learning to approximate the conditional expectation for any n. However, such a method is computationally intensive when the search space of rj is large, since is unrealistic and super inefficient to posit scparated model for each possible value of 17. To facilitate the computation, wo leverage the state-of-the-art deep conditional generative learning methods to learn the conditional reward distribution. This type of approach includes but not limited to Generative Adversarial Networks (GAN) and Mixture Density Network (MDN), which provide promising performance in real applications.
A a}. As discussed earlier,
In the subsequent sections, we employ Mixture Density Networks (MDN) as an example for illustrative pur- poses. It's worth noting that our framework adaptable to the implementation of any conditional distribu- tion approxination methodology, provided that the convergence rate satisfics a readily attainable assumption (Assumption (A1)), which will be elucidated in Section
For brief introduction to MDN, it integrates the Gaussian mixture model with deep neural networks by assuming that the conditional density of It given X and A is mixture of J conditional Gaussian distributions. That is, the probability density function of R)(X,A) satisfies
f(r(z,a) = Σ (2, V2Ta,(z,a) exp { (r - (x-,((x,))²
J
(z) }
with weights ())( E [0, 1]. conditional means {uz (z,,))) and the corresponding variances {a, parameterized via deep neural networks. The universal approximation property of deep neu- ral nets allows approximate any smooth or non-smooth weight, conditional mean and variance functions (Tmaizumi & Fukumizu 2019). This together with the universal approximation property of Gaussian mixture models allows ff to approximate any smooth conditional density functions (see e.g., Kostantinos 2000
Finally, we discuss how to compute Elp, (R- x, A= - a). The idea to sample i.idd. pseudo outcomes [Reh] 1 j < M) based on the estimated conditional distribution generalor for [2| A) and approximate the expectation via. Monte Carlo, i.e,
M Pr (RU) j=1
1
Elp 7) A= a)
17)
M
The number of Monte Carlo samples M represents a trade-off. Notice that the bias of the estimator is the same for any M. A large M improves the accuracy of the estimator at the cost of increasing the computation
7

---

Published in Transactions on Machine Learning Research (09/2025)
time. In theory, shown in Section require M - to ensure the resulting value estimator achieves the minimum asymptotic variance. In our numerical implementation, we set M to 50 to achieve a good balance between estimation accuracy and computation cost.
2.3.3 Step 3: Doubly-Robus Robust Quantile Value and Variance Estimation
Step 3 of the proposed algorithm is to employ cross-fitting to ostimate the quantile value and construct the associaled confidence interval. In particular, we develop two doubly-robust estimators, one for the quantile value itself, and another for the asymptotic variance of the quantile estimator.
First, let bg and E, denote the estimated behavior policy and conditional mean function, respectively. We use cross-fiuting to construct the following estimating equation to compute the doubly-robust quantile estimator, i.e.
PP arg min
Σ ieT
Vnn W V,,,,,E.))
where is defined in Equation and W, is a shorthand for the triplet (,, Ra).
Second, as shown in Theorem (see Section 4). the asymptotic variance of f is given by the sandwich formula Jo Var V (nr; W,, b, where Jo (m) (1++) equals the probability density function (pdf) of the potential outcome R' (T) evaluated at 7). While the variance term Var(a, M (n+; W b, E)) can be consistently estimated via the sampling variance estimator, the pdf is more difficult to learn from the data.
A key observation is that, SR (x) (1z) is not only the marginal pdf of the reward following x and the baseline distribution G, but the expectation of a conditional pdf for fn-(=) x ( (X) as well. This motivates us to consider the following two estimators for [R-(x)
(i) IPW estimator for Jo: for those subjects following the target policy म, natural estimator for Jo can be obtained by applying kernel density estimation (KDE). This idea, combined with the change of measure theorem, yields the following IPW estimator for Jo:
S
T. TA
R, TJRR
JPP
K h
(AIX.))
TET.
where is some kernel function and is a pre-specified bandwidth.
(ii) IPW estimator for Jo: In Step 2, we have obtained the conditional pdf of R A via some deep conditional generative model. Let fa denote the conditional probability density function obtained from the data zg. This yields the following direct method estimator for SR- (m) (1=);
8 ΣΣΣ
JDM = fn
Σ
(aa))) X,) = N
Σ
Σ
F' (m), (T), (fDR|X.)
8=1 1E
#=1 IE
Combining both (i) and (ii) yields doubly-robust variance estimator, given by (iii) DR estimator for Jo:
S JOR
x(AIX))) (,(A h
1 K
ÎDR R h
x(IXX),) J (TDRR Xx) + -111-,(41
(R)
(5)
8=1 SEI,
As such, Wald-type confidence interval can thus be derived for 7r, which is detailed in Equation of the theory section multi-stage general form.
8

---

Published in Transactions on Machine Learning Research (09/2025)
2.3.4 Step 4: Tail-Robust Mean Value Estimation
For some studies where the mean outcome is still of interest, Step 4 of the proposed algorithm provides a tail-robust mean value estimator based on the quantile values we estimated from Step 3. Since the target policy's mean value can be represented as an average of the quantile values, i.c, Jo nrdT, this motivates us to consider the following plug-in estimator So d for the mean value. The integral can be approximated numerically based on midpoint rule, trapezoidal rule or Simpson's 's rule.
Compared to the standard doubly-robust estimator, the proposed estimator is tail robust in that its con- sisteney requires very mild moment conditions. Specifically, it only requires well-defined mean value, i,e, E|R] oc. On the contrary, the standard doubly-robust estimator requires the reward to have at least. a finite second-order moment. In our numerical studies, we find that the proposed tail-robust estimator achieves a much smaller MSE when reward is heavy-tailed or has a quasi-infinite variance.
3
Sequential Decision Making
3.1
Preliminaries
In this section, we extend our framework to more general sequential decision making problems with K stages. The data trajectory for single subject is given by
(X Ar, X2, A2, XK, Ak, Rg),
(6)
where (Xx, Ak, R&) denotes the Dovariates-rreammen--oo K denotes the horizon (e.g., number of decision stages).
triplet observed at the kth decision stage and
A (bistory-dependent) policy = (TI,... ... Tk) maps the observed data history to probability distribution over the action space at each time. Specifically, let H₂ denote the historical data collected prior to the assignment of AR, i.e., H2 = AI, R1, ... .x).
Following ग, the agent will set A& according to the probability mass function (TH-HHb each time k. In addition, we use b = (b,,... .bx) to denote the behavior policy that generates the data, which consists of N i..dd realizations of equation 6 denoted by
{X (,,,,,A1, A1,s 11,,,2,11 A2,1, Re. XK,, AK,, RK,)) N).
(7)
Note that the i.idd. assumption is imposed across trajectories (i.e., over index i), rather than across time steps within trajectory. That is, we do not assume (X,, Ar, R2) are i.d. over k, which would be unrealistic in practice.
Usually, the final outcome can be defined as a function of rewards at all stages, i.e. f(Ri,..., Rx). However, in most cases when we periodically evaluate a specific kind of outcome of interest, it is natural to infer the cumulative reward Σx_ R3 following the target policy æ. Note that in the following sections, although wo focus on the estimation and inference of Σ Rr at the 7th quantile, the whole framework can be easily extended to any reward function fR,,,.... Rx).
Similar to Section 2 we assume versions of (C1') consistency, (C2') no unmeasured confounders, which is better known as the sequential randomization assumption, and (C3') positive assumptions hold in multiple stage settings. An explicit version of these assumptions are provided in Appendix A.I (see e.g., Zhang et al., 2013, for the detailed definitions of these assumptions).
3.2 Quantile Treatment Effect Evaluation
In multi-stage scttings, our objective is to infer the rth quantile of the sum of potential outcomes at all stages under a fixed target policy л. Let Rt())) denote the potential outcome one would observe under policy sequence āg (a1,...,ak). Similarly, define (T) the patential outcome under policy म, where
9

---

Published in Transactions on Machine Learning Research (09/2025)
Rd (T) Es (T) is the cumulative outcome of interest. It then follows that
K II k=1
K PP
P Σ [T( $ x - E
(ax (b)
Σ
Rg $ sr(xi)
(8)
(or D
where the summation over {ax) k=1 can take all possible treatment combinations in |A|K and the expectation E is taken w.r.t. possible sequences of {H&, R& RiJISKK. Under (CI')-(C3'', formula S can be represented using the observed data distribution. For any given quantile т, we solve the optimization problem
-RA-
7/ arg min ExinG Pr
RR Σ kml
(9)
where Ex, т denotes the expectation with respect the history data (K,.Ak Rk), assuming the initial covariates X1 were generated according to G and all subsequent observations were to follow the target policy In the following sections, will first introduce the direct method and inverse probability weighting estimator, and then combine the above two to derive the doubly robust estimator.
3.2.1
The DM estimator
Under (C1')-(C3'), follows from that
K
K R - (An = {ax ( X1
MM = arg min
Σ
TA (anl H2) BBpp
( k=1
(10)
{a]K km
Similar to the single-stage setting, we first estimate the conditional probability function for Σ (HK, Ax) and then leverage MC approximation to calculate the conditional expectation in Equation 10 Details will provided in Section 3.3
k=1 R given
3.2.2 The IPW estimator
According to the the change of measure theorem,
II T (A.)Ha) (As HA)
Σ Rr m) k=1
frw arg min E
Pr
Notice that when the number of stages increases, the proportion of data observe that aligns with target policy will decrease drastically. This may lead to poor IPW estimator. Akin to the single-stage setting, one can implement any supervised learning algorithm to learn b, that satisfies some mild convergence rate assumptions specified in Assumption (A1) of Section
3.2.3 The DR estimator
By incorporating the DM and IPW estimators, one can derive a doubly robust estimator for K-stage quantile treatment effect:
N 1 (n: W, E)
TDR = arg min
N
im1
arg min
I Tee ( (A Ho))) Ber HW
Pr
Σ km
K
Ric - n)
(11)
ET,
K
Te (AK HKr Ty (Akr, (Hrr))
TA (Ak, (K.,))
La (HHh.))
+
by (AR.e)HH..)
k=
10

---

Published in Transactions on Machine Learning Research (09/2025)
where W, is shorthand for the observed data trajectory
(X1,6 A1,s Ri X2,1, A2, R2n..... X XK, AK, Rx,,),
and (Hx) pr [pppp +.... R Rs + +.... ing that in (H&), the expectation is conditioning on HR. in which Ri,..., Ra-1 are directly obtained from the observed data, and R,,..., Rx are the rewards estimated from some data generating model under treatment (Tk:...,TK).
+ Rx - J {Ax J ~ n, Xi It is worth not-
Equation 11 is the average of the sum of K + terms. The first term serves as IPW estimator, where the coefficient - TH is the probability of a subject's actions at all K stages being consistent with policy IT== b.r
n. The last K terms, shown in the second line, denote the augmentation terms estimating the quantile loss originated from the kth level of missing data. The kth augmentation term is the multiplication of two main components: the product of the two policy ratios in parenthesis, and a direct estimator (HHH).. It aims to leverage subjects whose observed actions are consistent with π during the first k- 1 stages, but inconsistent starting from the kth stage. The product of the two policy ratios in this case roughly describes the probability of this inconsistency occurring. (H2) is the expected quantile loss estimated by combining the observed reward before stage with the estimated rewards from stage & onwards. Using this DR estimator, one can maximize the utilization of the observed information related policy æ. The estimation details are described in Section 3.3
The proposed estimator ÍDR is doubly robust in the sense that if the propensity score functions (br (AxxHH))) or the outcome regression models (T&(H))) are consistently estimated at each stage, then ÎJR is a consistent estimator of f/r, Compared with the DM and IPW estimators, the DR estimator TIR tends to be more robust w.r.t. model misspecification. This also enables us to implement more flexible machine learning methods and leverage their advantages in estimating propensity score and outcome models. Later Section 4 we will also prove the semi-parametric efficiency of fiDR, which provides further thcoretical guarantee for the performance of our estimator.
3.3 Estimation details
In this subsection, we focus on the estimation procedure under general K-stage settings. Similar to single stage, our approach contains four steps in sequential decision making process: (1) sample splitting, (2)
nuisance functions estimation, (3) doubly-robust quantile value and (4) variance estimation, tail-robust
mean value estimation. The pseudo code for multi-stage quantile estimation is summarized in Algorithm 2
Steps 1 and 4 are essentially the same as the single-stage case described in Sections 2.3.1 and 2.3.4 and are therefore omitted here. Below, we focus on the differences in Steps 2 and 3.
In Step 2, the estimation involves two sets of functions: propensity score functions {bx (H&) and the conditional expectation functions (Lx) The estimation of (Hx)) is more involved. First, the conditional distribution functions for [Re( A&)] can be obtained by some conditional generative modeling process. After getting these Monte Carlo approximation. As long as the number of replicates M is large enough, (HH)))) will converge to the true value if the model is correctly specified. Specifically.
at each stage, (Ha) can be estimated through a
[u(lss) = E (R) ..... + -R + Rr + .... Rx - -1--,n,) ( Ja HR ~
(A 1k-1
1 M
Pr + RR + 12(00) +... + n7)), M
where (7(() ...., 7o دوا are obtained from the reward generator for ( ) estimated from deep conditional generative learning, and {A K J in this case is randomly generated from target policy Note that the whole procedure for estimating Tx (H&) only involves the information in the observed H
11

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 2 Pseudo Code of Quantile Treatment Elfect Evaluation in Multiple Stage
Input Data trajectories {(11,,, A1,6, R1,s, X2, Ag, R2....... XK,s AK, RK,) isisi NN quantile levels t; number of folds S; large integer M.
Output doubly robust quantile estimator o tail-robust DR mean estimator ÅDR.
procedure A. DOUBLY ROBUST QUANTILE ESTIMATOR
Randomly split the data into S folds. For any S, denotes I, the sth subgroup. for all E S do Estimate {b by GBDT with the data in 25; Estimate (Re Ax)] by MDN with the data in I 6: Generate {R) (HH = Ha, A = a}}} for any a E A and E Iz:
4:
end for
Solve the optimization problem in formula end procedure
to obtain ĀDR
10: procedure B. TAI--ROBUST DR MEAN ESTIMATOR
11: For all of the quantile levels of our interest, calculate fr by procodure A. Averaging among all quantiles yields the lail-robust DR mean Estimator JIDR-
12: end procedure
The level of information utilize in estimating R (π)) depends how well the observed actions are in line with the target policy, which aligns with the core idea of double robustness.
In Step 3, the main difference from the single-stage case lies in the extended form of the est immmoos for Jn in the multi-stage. Define fn=(=), as the conditional pdf of R (T) Hy obtained by some conditional generative learning approach from the data in Zg Then Jo can be estimated by
(i) DM estimator for Jo:
JoM (r) Σ Σ
(TD. H1,)).
s=liET,
(i) IPW estimator for Jo: Following the same logic in Equation 3 we have:
K
Te (AL. K
Ek=1 Ri, ?D
JPW
N
by (Ax. (
h
(i) DR estimator for Jo:
I The (AN (w K
Ex= Ri, - MD
JDR
N
ber Aw HN) h
h
$=1
IEZ.
(12)
1-1
The (Aw (kxss)) by (A Hk)s))
)
(1- (1-
The Ta(Ak,
FREZ (NDR Hr.))
(A Hk.e)
where (m) () is the estimated conditional pdf of (T((T) Σk 12 (T) given Hy and A n obtained by MDN.
A Wald-type confidence interval can thus be constructed based on our DR quantile estimator 7PR and its DR variance estimator obtained from formula 12) We will detail this in the corollary of Section
4 Theory
In this section, provide asymptotic guarantees for the proposed doubly robust quantile estimator CPR
12

---

Published in Transactions on Machine Learning Research (09/2025)
Define (WW, a) W,, B, which is the derivative of the objective function we trying to optimize. Similarly, we define / (WW)) = W, 1) where the nuisance functions for and are replaced by their population functions
Solving the optimization problem in 11 thus equivalent to obtaining the solution 7p to the following estimating equation:
1
(W.n,, 0,
(13)
N =E
where åg is the collection of nuisance parameters in estimating 6 and E using data in Iç.
4.1 The asymptotic properties of n
Before we proceed, let's introduce the assumptions that will be used in this section.
Assumption 1. We assume the following conditions hold:
(A1) [[A(AA)))))) - (u.(As) P.2
|8 (FRAG))A FR FR 1|pa = o(nn11/2) (l,...,K)
(A2) fnoGr)i (r(Hk) is uniformly bounded in r and HE, v KE fI....,,K} K}.
(A3) 3 constant C1 > 0, such that fn- 2 C₁ holds for all n in neighbor of fr-
(A4) 3 <>>>> O, s.t. P(c S (A)())))) (1- 1, v Hg and le (1....,K)
(A5) 0, (m) is bounded.
Remark 1. Assumptions (A1) measures the accuracy in estimating propensity score and outcome at each stage. Specifically, we require the product of the convergence rates of both the propensity score model and the outcome regression model to be at least o(n-1/2), implying that each one of them must be o(n11/1) if both share the same convergence rate. This assumption is mild and can be achieved by various methods. For instance, estimators constructed via lookup tables typically attain the standard o(n convergence rate individually, yielding a product rate of o(n- which satisfies Assumption (A1). Additionally, several machine-learning-basse approaches, such the "lassoed" gradient boosted tree algorithm (Schuler et al. 2022 for estimating the behavior policy, and mixture density networks (MDNs, see Zhouu et al. 2022 Theorem 3) for estimating conditional distribution functions, also meet this requirement. Similar ptioons are commonly imposed in the literature (see, e.g. Chernozhukov et al.] 2018 Farrell et al. 2021 Kallus & Cehara 2022 In Assumption (A2), we require the conditional pdfs to be uniformly bounded, which aims to guarantee the continuity of the edf of the corresponding reward functions. Assumption (A3) ensures the identifiability of the quantile 17r of our interest, and Assumption (A4) is an adjunctive condition on the estimated propensity score, which is commonly assumed in related literature (Chernozhukov et al.) 2018). Assumption (A5) requires the derivative of (r) to be bounded only at the true value 7. Since the true value is unknown to us, we may need a stronger condition that requires uniform boundedness on the support of 77.
We first introduce the two lemmas that guarantees the double robustness of both the quantile estimator and its variance estimator.
Lemma 1. (Double Robustness of the Quantile Estimator)
Suppose that Assumptions (A8)-(A4) are satisfied. h.PR consistent estimator of nr, as long as one of the following two parts of models is consistently estimated:
(1) The propensity score functions at each stage, i.e. (Hx))
The explicit expressions for s:WW; 1) and (Ws:n) are given Appendix
13

---

Published in Transactions on Machine Learning Research (09/2025)
(2)
The conditional expectation functions at each stage, i.e. (Lx(H&)) or equivalently the data generating models for (Rel(HK, A
Lemma (Double Robusiness of the Variance Estimator)
In kernel density estimation, suppose the kernel function K is a real-valued, Borel-measuruble function in L (R) which satisfies lim,-+oo [x)] = lim (x)) 0. Define the bandwidth as hy, which can be function of the somple size. We assume N- NN = 0, and lim Nhx xc.
We claim that DR is a consistent estimator of o², as long as one of the sollowing two parts of models is consistently estimated:
(1) The propensity score functions at each stage, i.e. (H&))
(2) The conditional expectation functions at each stage, i.c. (Hx)) or equivalently the data generating models for ( Ax)]
The proofs of Lemma and Lemma along with other auxiliary theoretical results are provided in Appendix C Next, we will derive the asymptotic properties of our DR quantile estimator 7p.
Define lll the (P(P) norm which satisfies Il( (L) 14 Let F(F,,F2) de- note the total variation distance between two probability measures, where Fi and F2 are the CDF of the two distributions. Define (R and ( Hx) as the true conditional pdf and odf of (T) Ex= (T) given the historical data H&, respectively. Similarly, define BRAGEME (r)Hs) and FRI (r) H+) as the conditional pdf and edf of (T) estimated by conditional distribution generator, respectively.
Theorem 1. (Asymptotic Normality)
Suppose that Assumptions (A1)-(A5) are satisfied. When M - 00, n. is asymptotically normal. Specifically,
N b (Wis)t) + op(1) 2, N,,,,,
V 1T) = o JJ
(14)
where
o² J (W: (W;n>11055>) and Jo 0, (E (W:n)l
(15)
Furthermore,
Jo 0, (Ew -/=" (W; (Wn)llb = EninG [[R [H)] (1)
(16)
The results still hold when x² is replaced by its doubly robust estimator TbR given by
s L ( 7o. ac)l (JpR)²
RD = (JOR)
(17)
S
where En,s denotes the empirical expectation calculated with the data in fold s, and JoR is estimated under the doubly robust frumework, as we' 've discussed in formada and 12
Estimation details are provided in Section 2.3.3 and Section 3.3 for both single-stage and multiple-stage settings.
Corollary According to Theorem 1 a Wald-type a-level confidence interval of nh can be constructed by
CI = 7 +
Φ (1 - (1-22) 6DD
(18)
where & is the edf of the standard normal distribution. The confidence interval constructed in asymptotic coverage rale 1 a.
18
achieves
14

---

Published in Transactions on Machine Learning Research (09/2025)
As illustrated in Lemma 1 and Theorem our quantile estimator o/. satisfies the double robustness property and is semi- pa-aaet efficient. in both single stage and sequential decision making problems. The proof of Theorem I is summarized in Appendix C.4
Theorem 2 below ensures that our DR quantile estimator is constructed from the efficient influence function (EIF) and thus enjoys semi-parametric efficiency.
Theorem 2. Paramerrrci Ffficiency)
Define C(n) = - (x) Under the some conditions as proposed in Theorem 1
the EIF of thr is given by
EIF, C(+)) EE
II==1 TM (A (Hx)
(1
Σ
K
Rs
< m m
= bw (Aw HHr)
x=1
(19)
K
The (AN HK))
(1
(A( Hx)
F Fr ())
by [Aw HHx))
(x(AgHHp)
which alignes with the DR estimator we proposed in Equation
11
The efficiency bound for DR variance is
given by Equation 77
The proof of semi-parametric efficiency is summarized in Appendix C.5
5 Numerical Results
In this section, we will justify the performance our DR quantile estimator in both single-stage and multi- stage settings under different levels of heavy-tailness of the reward distribution. Throughout this paper, we use student-t distributions with different degrees of freedom to measure the level of heavy tail. All source code and supplementary experimental results are available on our Github page
In the following sections, we will first illustrate how close our estimated quantile is to the true reward distribution, and then report the empirical coverage probability of our estimator. To show the performance of quantile aggregation in estimating the mean outcome the heavy-tailed cases, we compare our quantile- based DR mean estimator Rquantile, with the DR mean estimator Amean used in common literature The details of the lata-generating mechanism and additional numerical results are given in Appendix B.1.1
Before procced, let's discuss how to choose the number of folds in cross fitting. Although larger value of S' may yield intuilively belter performance in quantile estimation, according to some comparison results in existing work (Chernozhukov et al. 2018), there is no strong relationship between the number of folds and estimation accuracy in the context of double machine learning. To balance the computational complexity and the potential advantage of a large number of S, we set S S= 5 throughout the simulation and real data analysis. For ease of calculation, we first compute the DR quantile estimator under each fold and average over all folds to obtain our final estimator.
5.1 Quantile Estimation Performance
Figure 2 shows the estimation accuracy of our DR quantile estimator. As we can see, our DR quantile estimator performs quite well in evaluating the true reward distributions under the target policy of our interest.
5.2 Coverage Probability
According the estimation details elaborated in Section 2.3 and 3.3 doubly robust variance estimator 2h by formula (17). For each quantile level 7, repeat the estimation procedure 500 times to calculate the empirical coverage probability, which is shown in Figure
we calculate the quantile 7PR and its
The expression of the DR mean estimator summarized in Appendix
15

---

Published in Transactions on Machine Learning Research (09/2025)
Quantlee Estimation in Single-Stage
Stage Setting
Quantile Estimation 2 2-Stage Setting
Empinical
1.0
Empinical
Estimated quantiles
Estimated quarttles
0.8
0.8
Quantile level
Quantielevel
0.0
0.2
0.0
0.0
Outcome
Dutcome
Figure 2: Quantile estimation result in single-stage (left) and two-stage (right) settings. The blue curve denotes the true reward distribution (the CDF under target policy x), and red stars denote the estimated quantiles at 20 equally spaced quantile levels. Details about the data generating processes are introduced in Appendix B.1.1
Notice that when calculating the asymptotic variance, it is required to select proper bandwidth in the kernel function. After comparing several commonly used bandwidth selection methods, finally choose fixed bandwidth under this heavy-tailed circumstance due to its robustness in preventing the over-smoothing issue in kernel density estimation 3
Crse Hosalilly for bandaitthee.
1.00 Cowarage Hobabiltt for 1333, bandwadttee 15
Crawrage Robaliilly baritAl, handsitted
1.000
1.000
3.75
0515
3990
0146
2.45
4125
2.00
8.00
4000
2.15
3250
8:8K
0250
3505
8:05
0025
3.00
8,00
0.08
Ounill Lond
Figure 3: Coverage Probability under noise level t(2), t(3), and t(4). The red dots correspond the empirical coverage probability estimated from 500 times of replicates, and the blue area denotes the confidence band at each quantile level. It is clear from the plot that 0.95 falls into almost all of the confidence bands, which means that our CI achieves nominal coverage in most cases.
5.3 Comparison between Rquantile and Rmean
To compare our quantile-based mean DR estimator Rquantile with the classical DR mean estimator Rmean, calculated the MSE for both estimators under different heavy-tailed levels of the reward distributions. Detailed comparisons are summarized in Table and Table 2 As we can see from both tables, Rquantile al- ways yields smaller MSE and standard deviation than Rmean. The more heavy-tailed the reward distribution is, the more powerful our method tends to be.
See Appendix B.1.3 for details about bandwidth selection.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Comparison between Rquantile and Rmean
Rtrue
Rquantile
Estimated outcome
2.
20
80
Number of trials
Figure 4: Comparison when the noise follows (11.2) distribution. The blue line denotes the true value of mean reward, red stars denote the estimated value of Rmean, and green stars denote the estimated value of Rquantile. For comparisons across all levels of reward heavy-tailedness, please refer to fhis Github page
Table 1: Single-stage comparison between Rquantile and Rmean. Each cell reports the MSE (with standard deviation in parentheses). We use 3-folds cross fooss-iittig with N 2500. All the results are obtained from 100 times of replicates.
Heavy-tailed level
(1.2))
b((.5)
t(1.8)
t(2)
6(2.5)
1(3)
t(3.5)
t(4)
0.00600
0.00169
0.00090
0.00099
0.00055
0.00043
0.00031
0.00037
MSE of Rquantile
(0.009771)
(0.002250)
(0.001190)
(0.010338)
(0.000724)
(0.00U557)
(0.000355)
(0.0005447)
0.59478
0,03105
0.00203
0.00186
0,00062
0.00045
0.00032
0,00038
MSE of Rmean
(32.11465)
(2.852596)
(0.044305)
(0.043518)
(0.004603)
(0.003373)
(0.003202)
(0.002741)
Table 2: Two-stage comparison between Rquantile and Rmean. Each cell reports the MSE (with standard deviation in parentheses). We use 5-folds cross-fitting with N = 2500. All the results are obtained from 100 times of replicates.
Heavy-tailed level
t(2)
t(4)
t(6)
t(8)
N 1)
0.006708
0.002729
0.002447
0.002427
0.001549
MSE of Rquantile
(0. (0.007954)
(0.003865)
(0.003007)
(0.002709)
(0.001541)
0.027780
0.003945
0.003558
0.003710
0.002062
MSE of Rmean
(1.648467)
(0.027229)
(0.024007)
(0.023832)
(0.002024)
To illustrate the advantage of our estimator more clcarly, we visualize the estimation results of 100 times of replicates for both Rquantile and Rmean when the reward follows ((1.2) distribution, which is shown in Figure
It is obvious from Figure that under a super heavy-tailed setting, Rquantile outperforms Rmean in estimat- ing the true reward expectation. There are some "outliers" when estimating Rmean, which performs much worse than other points. This is because the estimated propensity scores of some subjects are quite close 0, leading to extremely unstable estimators for Rmeon. On the contrary, our approach divides the mean estimation procedure into many quantile levels, and solving each estimating equation would affect much what we may have in obtaining Rmean. This intuition explains the tail robustness of estimator.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Since each quantile we estimate also enjoys the double robustness property, it's reasonable that Rquantile achieves better results than Rmean.
6 Real Data Analysis
In this section, we use a dataset collected from an advertisement experiment conducted at a world-leading tech company. The company plans to add an advertising position with three candidate choices. This experiment serves as an exploration applied to three groups of customers (180,000 users per group) randomly selected from daily active users. The company conduct causal inference on this data, and come up with four different personalized intervention policies. The key response of interest the watch time (the larger, the better), which has heavy-tailed distribution shown by the histogram, Q-Q plot and log-log plot in Figure !
We demonstrate that our method can (i) reliably infer the quantiles of revenues under the four policies, and (ii) accurately estimate their expected revenues using only a small subset of the full dataset. Toward that end, we repeat the following procedure 1,000 times: in each iteration, we randomly sample 1% of users from each group and apply our method to estimate 999 quantiles from 0.001 0.999 of the outcome distribution for each policy. Based on these estimators, we construct confidence intervals for the first, second, and third quartiles, and use the mean of the 999 quantiles as the estimated expected value. The ground-truth values of the four policies are obtained from online A/B tests. The mixture density network consists of two hidden layers with eight neurons each, and four mixture components.
Table shows that the coverage probabilities of the 25%, 50% and 75% quantiles of the reward distribution of the four personalized policies are all close to the 95% nominal level. The bias, standard deviation and root MSE are presented in Table Our doubly robust quantile- based mean estimator outperforms the traditional doubly robust mean estimator on this heavy-tailed real dataset. Furthermore, we also compare
the performance among DM, IPW and DR estimators. Details are deferred to Appendix B.2 to save space.
Table 3: Coverage probabilities of the first, second, third quartiles of the proposed confidence intervals under the four personalized policies.
S1
S2
Sy
S4
25%
0.926
0.944
0.933
0.911
50%
0.961
0.958
0.957
0.948
75%
0.949
0.926
0.942
0.915
Table Comparison of biases, standard deviations and root MSEs of the expected revenue estimators between mean-based and the proposed quantile-basen approaches. S1, S2, S3, and S₁ denote the four per- sonalized policies.
Bias
STD
RMSE
Rmean
Rquantile
Rmean
Rquantile
Rmean
Rquantile
S1
0.006
-0.013
0.076
0.069
0.076
0.071
S2
-0.008
-0.026
0.060
0.055
0.060
0.061
S3
0.007
-0.014
0.066
0.059
0.067
0.061
S4
0.001
-0.017
0.077
0.068
0.077
0.070
7 Conclusion and Discussions
In this paper, we conducted comprehensive research on the doubly robust off policy evaluation procedure for the entire reward distribution in sequential decision making problems. We constructed a doubly robust quan- tile estimator with theoretical guarantees, and provided an algorithm based on deep conditional generative learning, allowing a broad class of machine-learning- based tools to be utilized in estimating propensity scores and reward functions at each stage. Furthermore, doubly robust. variance estimator was also proposed to im-
18

---

Published in Transactions on Machine Learning Research (09/2025)
prove the stability of the estimated confidence interval. Our quantile estimator enjoys asymptotic normality and semi-pamametric efficiency, which is well-illustrated from both simulation and real data analysis.
Based on our estimated DR quantiles, we proposed a tail-robust doubly-robust mean estimator which signif- icantly outperforms the classical DR mean estimator used in the earlier literature. Both simulation and real data analysis illustrate the power of our estimator in decreasing the MSE when the reward is heavy-tailed.
The quantile-based mean estimator has demonstrated significant advantages over classical mean estimators in various contexts, including single-stage (Wang & Zhou, 2010) and infinite-horizon scenarios such re- inforcement learning (RI, Rowland et al., 2023). Given the promising results observed in both simulated and real-world data, further investigation into the theoretical properties of the proposed tail-robust doubly robust (DR) mean estimator would be valuable for future research. Additionally, it's quite interesting ing lo explore the potential of developing optimal treatment regimes using the doubly robust quantile estimator introduced in this study. Another direction to extend the binary actions to discrete or continous action space, which would allow the entire framework to be applied to a wider range of real-life scenarios.
Acknowledgement
Shi's research was partly supported by an EPSRC grant EP/W014971/1
Data Availability
Due to the sensitive nature of the dataset, it cannot be made publicly available.
References
Alberto Abadic and Matias D Cattanco. Econometric methods for program evaluation. Annual Review of Economics, 10(1), 2018.
Armin Behnammia, Gholamali Aminian, Alireza Aghaei, Chengehun Shi, Vincent YF Tan, and Hamid R Ra- bice. Log-sum-exponential estimator for off-policy evaluation and learning. In Forty-second International Conference on Machine Learning, 2025.
Alexis Bellot and Mihaela van der Schaar. Conditional independence testing using generative adversarial networks. Advances in neural information processing systems, 32, 2019.
David Benkeser, Marco Carone, MJ Van Der Laan, and Peter Gilbert. Doubly robust nonparametric inference on the average treatment effect. Biometrika, 104(4):863 880, 2017.
Zeyu Bian, Chengehun Shi, Zhengling Qi, and Lan Wang. Off-policy evaluation in doubly inhomogeneous environments. Journal of the American Statistical Association, 120(550):1102-1100 2025.
Tavor Bojinov and Neil Shephard. Time series experiments and causal estimands: exact randomization tests and trading. Journal of the American Statistical Association, 111((228):6655-1662 2019.
David M. Bossens and Philip S Thomas. Low variance off-policy evaluation with state-based importance sampling. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp. 871-883. IEEE, 2024.
Hengrui Cai, Chengehun Shi, Rui Song, and Wenbin Lu. Deep jump q-evaluation for oflline policy evaluation in continuous action space. arXiv preprint arXiv:2010.155: 15963, 2020.
Bibhas Chakraborty, Eric B Laber, and Yingqi Zhao. Inference for optimal dynamic treatment regimes using an adaptive m-out-offn bootstrap scheme. Biometrics, 69(3))714 723, 2013.
Arthur Chatton and Julia M Rohrer. The causal cookbook: Recipes for propensity scores, g-computation, and doubly robust standardization. Advances in Methods and Practices in Psychological Science, 7(1): 251524592412361 2261922222349999 2024.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Xiaohong Chen and Zhengling Qi. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. In International Conference Machine Learning, pp. 3558-3588
PMLR, 2022.
Chao Cheng, Liangyuan Hu, and Fan Li. Doubly robust estimation and sensitivity analysis for marginal structural quantile models. Biometrics, s0je)u4,, 2024.
Victor Chernozhukov, Iván Fernández-Vall Val, and Blaise Melly. Inference counterfactual distributions. Econometrica, (1(6):2205-28) 2268,, 2013.
Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, 2244:11644-1577 2014.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural paramcters, 2018.
Yifan Cui and Sukjin Han. Policy learning with distributional welfare. arXiv preprint arXiv:2311.1597 2023.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Seoond AAAT Conference Artificial Intelligence, 2018.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off- policy confidence interval estimation. Advances in neural information processing systems, 33:9398 9411, 2020.
Rhian M Daniel, Bianca L De Stavola, and Simon N Cousens. gformula: Estimating causal effects in the presence of time- varying confounding or mediation using the g-computation formula. The Stata Journal, 11(4):479 517, 2011.
Christopher N Davis, T Deirdre Hollingsworth, Quentin Caudron, and Michael A Irvine. The use of mixture density networks in the emulation of complex epidemiological individual-based models. PLoS computational biology, pp. e1006869, 2020.
Stephen G Donald and Yu-Chin Hsu. Estimation and inference for distribution functions and quantile functions in treatment effect models. Journal of Econometrics, 178:383-335 397, 2014.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601 2011.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 19(1):181-213 2021.
Sergio Firpo. Efficient semiparametric estimation of quantile treatment effects. Econometrica, 75(1):259 276, 2007.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning, pp. 4074-4084. PMLR, 2021.
Oliver Hines, Oliver Dukes, Karla Diaz- Bazz-rrazz, and Stijn Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician, 76(3):292 304, 2022.
Audrey Huang, Liu Leqi, Zachary C Lipton, and Kamyar Avizadeneshelli Off-policy risk assessment in contextual bandits. arXiv preprint arXiv:2104.08977, 2021.
Audrey Huang, Lin Leqi, Zachary Lipton, and Kamyar Arizzadenesheli. Off-policy risk assessment for markov decision processes. In International Conference on Artificial Intelligence and Statistics, pp. 5022-5050. PMLR, 2022.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Masaaki Imaizumi and Kenji Fukumizu. Deep neural networks learn non-smooth functions effectively. In The 22nd international conference on artificial intelligence and statistics, pp. 869-878. PMLR, 2019.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652 661. PMLR, 2016.
Younghyun Jo, Sejong Yang, and Seon Joo Kim. Srftow-da: Super-resolution using normalizing flow with deep convolutional block. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguition, pp. 364 372, 2021.
Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. J. Mach. Learn. Res., 21:167-1, 2020.
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in ofl-policy evaluation with double reinforcement learning. Operations Research, 2022.
Nathan Kallus and Angela Zhou. Confoudding-oouus policy evaluation in aufinit-hhorizon reinforcement learning. arXiv preprint arXiv:2002.04511, 2020.
Nathan Kallus, Xiaojie Mao, and Masatoshi Uchara. Localized debiased machine learning: Efficient inference quantile treatment effects and beyond. arXiv preprint arXiv:1912.12945, 2019.
Edward Kennedy. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.06469. 2022.
Michael Rt Kosorok and Eric B Laber. Precision medicine. Annual review of statistics and its application, 6: 263, 2019.
N Kostantinos. Gaussian mixtures and their applications signal processing. Advanced signal processing handbook: theory and implementation for radar, sonar, and medical imaging real time systems, pp. 3 3-1, 2000.
Liu Leqi and Edward H Kennedy. Median optimal treatment regimes. arXiv preprint arXiv:2103.01802 2021.
Jonathan Levy. Tutorial: Deriving the efficient influence curve for large models. arXiv:1903.0177 2019.
ar preprint
Ting Li, Chengehun Shi, Zhaohna Lu, Yi Li, and Hongtu Zhu. Evaluating dynamic conditional quantile treatment effects with applications in ridesharing. Journal of the American Statistical Association, 119 (547):1736 1750, 2024.
Xiaocheng Li, Huaiyang Zhong, and Margaret L Brandeau. Quantile markov decision processes, Operations research, 70(3):1428-1447, 2022.
Peng Liao, Predrag Klasuja, and Susan Murphy. Off-policy estimation of long-term average outcomes with applications to mobile health. Journal of the American Statistical Association, 116(533):382-35) 2021.
Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. The Annals of Statistics, 50(6):3364-333) 2022.
Daniel Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Flizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal the American Statistical Association, 115(530):692-706, 2020.
Alexander R Luedtke and Mark J Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics, 44(2):713, 2016.
José AF Machado and José Mata. Counterfcctal decomposition of changes in wage distributions using quantile regression. Journal of applied Econometrics, 20(4):445-4655 2005.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Roland A Matsouaka, Junlong Li, and Tianxi Cai. Evaluating marker-guided treatment selection strategies. Biometrics, 70(3):489-45) 2014.
Krikamol Muandet, Motonobu Kanagawa, Sorawit Saengkyongam, and Sanparith Marukatat. Counterfactual mean embeddings. Journal of Machine Learning Research, 22(162):1-71, 2021.
Susan A Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(2):331 355, 2003.
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evalua- tion for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 13:18819-18881, 2020.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108 (2):299-319, 2021.
Emanuel Parzen. On estimation of a probability density function and mode. The annals mathematical statistics, (3(3):1065 1076, 1962.
Zhengling Qi, Jong- Shi Pang, and Yufeng Liu. Estimating individualized decision rules with tail controls. arXiv preprint ar 190: 003660 2019.
Pratik Ramprasad, Yuantong Li, Zhuoran Yang, Zhaoran Wang, Will Wei Sun, and Guang Cheng. Online bootstrap inference for policy evaluation in reinforcement learning. Journal of the American Statistical Association, 118(544):2901-22) 2991,,,4,,, 2023.
Paul R. Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55) 1983.
Mark Rowland, Yunhao Tang, Clare Lyle, Rémi Munos, Mare G Bellemare, and Will Dabney. The statistical benefits of quantile temporal-difference learning for value estimation. In International Conference on Machine Learning, pp. 19210-29231. PMLR, 2023.
Donald B Rubin. Comment: Neyman (1923) and causal inference in experiments and observational studies. Statistical Science, 5(4):472-45)) 1990.
Alejandro Schuler, Yi Li, and Mark van der Laan. Lassoed tree boosting. arXiv preprint arXiv:2205.10697, 2022.
Guohao Shen, Runpeng Dai, Guojun Wu, Shikai Luo, Chengehun Shi, and Hongtu Zhu. Deep distributional learning with non-crossing quantile network. arXiv preprint arXiv:2504.08215, 2025.
Chengehun Shi, Wenbin Lu, and Rui Song. Breaking the curse of nonregularity with bernaging--iffrrrn of the mean outcome under optimal treatment regimes. Journal of Machine Learning Research, (176)) 1 67, 2020.
Chengehun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off-policy interval cstimation. In International Conference on Machine Learning, pp. 9580-9591. PMLR, 2021a.
Chengchun Shi, Tianlin Xu, Wicher Bergsma, and Lexin Li. Double generative adversarial networks for conditional independence testing. Journal of Machine Learning Research, 22(285):1-32, 2021b.
Chengehun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for rein- forcement learning in infinite-hrriznn settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):765 793, 2022.
Chengehun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye, and Rui Song. Dynamic causal effects evaluation in a/b testing with a reinforcement learning framework. Journal of the American Statistical Association, 118(543):2059 20701-2777, 2023.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Chengchun Shi, Yunzhe Zhou, and Lexin Li. Testing directed acyclic graph via structural, supervised and
generative adversarial learning. Journal of the American Statistical Association, (19(547):18331184) 2024a.
Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119 (545):273-285) 284, 2024b.
Rui Shu, Hung H Bui, and Mohammad Ghavamzadeh. Bottleneck conditional density estimation. In International Conference on Machine Learning, pp. 3164 3172. PMLR, 2017.
Jonathan M Snowden, Sherri Rose, and Kathleen M Mortimer. Implementation of g-computation on simulated data set: demonstration of causal inference technique. American journal of epidemiology, 173 (7):731-738, 2011.
Ke Sun, Linglong Kong, Hongtu Zhu, and Chengehun Shi. Arma-desigu: Optimal treatment allocation strategies for a/b testing in partially observable time series experiments. arXiv preprint arXiv:2408.05342 2024a.
Ke Sun, Yingnan Zhao, Wulong Liu, Bei Jiang, and Linglong Kong. Distributional reinforcement learning with regularized wasserstein loss. Advances in Neural Information Processing Systems, 37:63184 63221, 2024b.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pp. 2139 2148. PMLR, 2016.
Anastasios A Tsiatis, Marie Davidian, Shamnon T Holloway, and Eric B Laber. Dynamic treatment regimes: Statistical methods for precision medicine. Chapman and Hal//CRC, 2019.
Aad W Van Der Vaart, Aad van der Vaart, Adriamus Willem van der Vaart, and Jon Wellner. Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media, 1996.
Huixia Judy Wang and Xiao-Hua Zhou. Estimation of the retransformed conditional mean health care cost studies. Biometrika, 97(1):147 158, 2010.
Jiayi Wang, Zhengling Qi, and Raymond KW Wong. Projected state-action balancing weights for offline reinforcement learning. The Annals of Statistics, 51(4):1639 1665, 2023.
Lan Wang, Yu Zhou, Rui Song, and Ben Sherwood. Quantilo-optimal treatment regimes. Journal of the American Statistical Association, 113(523):1243 1254, 2018a.
Ting-Chun Wang, Ming-Yi Yu Liu, Jun- Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings the TEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
Zirui Wang, Wodan Ling, and Tianying Wang. A semiparametric quantile regression rank score test for zero-inflated data. Biometrics, (1(2):uja0050, 2025.
Haoyu Wei, Runzhe Wan, Lei Shi, and Rui Song. Zero-inflated bandits. arXiv preprint arXiv:2312.15595, 2023.
Qianglin Wen, Chengehun Shi, Niansheng Tang, Hongtu Zhu, et al. Unraveling the interplay between carryover effects and reward autocorrelations in switchback experiments. In Forty-second International Conference on Machine Learning, 2025.
Guojun Wu, Ge Song, Xiaoxiang Lv, Shikai Luo, Chengchun Shi, and Hongtu Zhu. Dnet: distributional network for distributional individualized treatment effects. In Proccedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 5215 5215-5224, 2023a.
23

---

Published in Transactions on Machine Learning Research (09/2025)
Runzhe Wu, Masatoshi Uchara, and Wen Sun. Distributional offline policy evaluation with predictive error guarantees. arXiv preprint M.Ni::33020094550 2023b.
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019.
Yang Xu, Jin Zhu, Chengchun Shi, Shikai Luo, and Rui Song. An instrumental variable approach to con- founded off- off-policy evaluation. In International Conference on Machine Learning, pp. 2023.
PMLR,
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Aurrbuteeinage: Conditional image generation from visual attributes. European Conference on Computer Vision, pp. 176-791. Springer, 2016.
Yanfeng Yang, Shuai Li, Yingjie Zhang, Zhuoran Sun, Hai Shu, Ziqi Chen, and Renming Zhang. Conditional diffusion models based conditional independence testing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22020 22028, 2025.
Shuguang Yu, Shuxing Fang, Ruixin Peng, Zhengling Qi, Fan Zhou, and Chengchun Shi. Two-way de- confounder for off-policy evaluation in causal reinforcement learning. Advances in Neural Information Processing Systems, 37:78169-78200, 2024.
Baqun Zhang, Anastasios A Tsiatis, Eric B Laber, and Marie Davidian. robust method for estimating optimal treatment regimes. Biometrics, 1100)1100- 2012.
Baqun Zhang, Anastasios Tsiatis, Eric B Laber, and Marie Davidian. Robust estimation optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100(3):681 694, 2013.
Liangyu Zhang, Yang Peng, Jiadong Liang, Wenhao Yang, and Zhihua Zhang. Estimation and inference in distributional reinforcement learning. arXiv preprint arXiv:2309. 2023.
Yi Zhang, Linjum Huang, Yun Yang, and Xiaofeng Shao. Doubly robust conditional independence testing with generative neural networks. arXiv preprint arXiv:2407.17694 2024.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforce- ment learning. Advances in neural information processing systems, 18:15909-159111 2020.
Hongyi Zhou, Josiah P Hanna, Jin Zhu, Ying Yang, and Chengchun Shi. Demystifying the paradox of importance sampling with an estimated history-dependent behavior policy in off-policy evaluation. Forty-second International Conference on Machine Learning, 2025.
Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu. Distributional shift-aware off-policy interval esti- mation: A unified error quantification framework. arXiv preprint #XXiv22399.32278 2023a.
Yunzhe Zhou, C Shi, Li, and Q Yao. Testing for the markov property in time series via deep conditional generative learning. Manuscript submitted for publication, 2022.
Yunzhe Zhou, Chengehun Shi, Lexin Li, and Qiwei Yao. Testing for the markov property in time series via deep conditional generative learning. Journal the Royal Statistical Sociely Series B: Statistical Methodology, 85(4):1204 1222, 2023b.
Jin Zhu, Runzhe Wan, Zhengling Qi, Shikai Luo, and Chengehun Shi. Robust offline reinforcement learning with heavy-tailed rewards. International Conference on Artificial Intelligence and Statistics, pp. 541 549. PMLR, 2024.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
A Assumptions and Notations
In this section, we aim to claborate more on the explicit expressions of some assumptions and formulas mentioned in the main paper.
A.1 Assumption (CI')-(C3')
In multi-stage setting, we also accept. the potential outcome framework causal inference.
(C1') Consistency (or SUTVA) Rubin 19900:
For any k E fl....,K], K].
R4 = (Ax) = Σ
Rº (ax))((A āz}:
A.E.A
When 2 2,
X₂ X((Aa-1)=
E
[(A az-1}.
IEA:
(C2') Sequential Randomization Assumption (SRA) [Rosenbaum & Rubin 1983): Given any treatment
sequence bx, we have
As T (Ra(br), XR+I(be), Re41(bu+ (bu+1), Rx (bx)) Vk E (I,...,,K]
(C3') Positivity Assumption: For any & E {l...., K}, there exists constant > S.t. s.t. P(e S (A(A)HH)) S c c) = for any A& E 10.
Note that (C1')-(C3') are just natural extensions of (C1)-(C3) in multi-stage setting, which is also widely assumed in causal inference literature.
A.2 (WW.n, a) and o (WW;))
The estimating equation of our interest is defined as
((WWi,,,)
(W U,
5
K T The (AR (A (K.)) (If Σ
K
Σ
Ras < m -T)
N
I B (Ak (H
(20)
1=1
kml
Te (Aw Hw:s) by (Av, Hw
(1- (1- Te(A (Hr.))
To (H(HE))
+
ba(A HK.e)
where (H₂,) = A I(R] R + R2 +... + Rx < n) - 7[{A [[AA)) 1k-1 (Aw ~ nt, x, in which Ru,..., Rx are the estimated rewards obtained by the data generating models following treatment (Tg...., TK) starting from stage k.
When the nuisance functions are replaced by the true models, we have
3 (W;n) =
1
Te
(I
Σ Ric < nn -7)
N
TE
I -
(e
Aw (a)
k=l
(21)
K
Tee (AK HK)s)
(1
Ta (Ak. HK.s)
4e (H(sss))
+
by (Aw Hw)
by (Ax, Hy.s)
k=
25

---

Published in Transactions on Machine Learning Research (09/2025)
In the rest of this appendix, will focus on the case K 2. Therefore, the expressions become
(W (t) (.
T1 (A..::HH,i))) A2, H2
(I(R1,; + R2 < <n) T)
(A1, (A,,i|H,,,,, (b2 12, 6)
Ti (A1,a H..i)
(22)
E[[LRR] + R2 < <n) -)) (.1,,,((111111 A2,,) ~T]
b1 (A1.)HHH.)
n (AI,
(2 H2,1)
B[ + n2 <n) 7) H2 A2,1 ~
+
(A, 122,s)
and
Tii (AI. H 12 (22, (2,s)
(W;:))
(I( + R2 < <n) - a + R2 <n)
(23)
11, (A A2,) ~ m| + (L + R2 < < r) H2 A2, ~ n|
where we define oj and až as
ai = (1
Ti 11 H1,s)
Mi (A1s))H
(1 (A(, H2,s)
(24)
b (A1, H1,1)
b1 (A1,1) H1,s) (2,, H2,5)
A.3 Rmean
In simulation studies, we compare our quantile-based DR mean estimator with the traditional DR mean estimator used in common literature, named as Rmnaan In single-stage setting, Rmean is estimated from
Rmean 1
Ti (AA1,HHH,, R1, (1- Ti (Ai, (A.i(HH.))) B|RiX X1, A1 ~ ~ (A1, (1,1)
(25)
(H,,)
In two-stage settings,
T (A1,1 (AA1:1/pp (1.)) (A2, H221)
Rmean
(Bi,: + R2.s)
by (A1, 14 1 3, 162 (A2, H2,s)
Ti (A (11
E((Ri + R2) H1, (AI, n|
(26)
by (A1 H1,x)
Ti (A1,i
T2(A2, H2.5)
B[(RI, + H2 H2,, A2 T]
(A1,i H4,s)
(2( H2.))
B More on Simulation and Real Data Analysis
B.1
Simulation
B.1.1 Data Generating Process
We first introduce the data generating processes in both single stage and multiple stage seltings.
In single stage quantile estimation problem, we generate the data as follows:
X G = NN(,
A = b(X), where b(X) = T(X 4 > 0):
(27)
R= = (1 - X + 21X) c'),
4
26

---

Published in Transactions on Machine Learning Research (09/2025)
where and e' are noise terms with different levels of heavy tail. The more heavy-tailed and e' are, the harder it would be to estimate the reward distribution. As mentioned in the main paper, we use the student- distribution with different degrees of freedom to generate heavy-tailed distributions, To illustrate the performance of our estimator under different levels of heavy tail, we let e and c' follow t distribution t(1.5), ((1.8), 1(2), (2.5), (()), 13..5), and u(4) respectively. The larget policy here defined as π (AIX) = T(X > 0}.
Let's take t(3) as an example. Figure shows the PDF of reward (x) when and follows distribution with df 3. From the probability plot shown in Figure [6] the rewards generated in this setting, shown by the blue dots, aligns very well with the baseline distribution t(3) which is shown by the red line. Therefore,
the heavy tail level of and é indeed represents the heavy tail level of the reward distribution.
Histogram of R1 under t(3)
Probability Plot
06
200
0.4
100
Probabiity
0.3
Ordered
0
-50
0.1
-100
-150
-100
100
R1
Theoretical quantiles
Figure 5: PDF of (T) when E ~ t(3)
Figure 6: Probability plot of R (T) compared with bascline distribution t(3), shown by the red line. The reward distribution indeed heavy-tailed.
In multiple stage settings, we will only consider the case when K = 2 for illustration purpose. The observa- tional covariatos-acioo--eeaad triplet (X1, A1, R1, X2, A2, R2) generated as follows:
X1 G~~G= = =W(0 1);
1
4/X, (X1), where (X1) 1(X, 61 > 0};
1
R1((X, A1) (1 X1 211x)) (2))
1 ~=x1= _ 1 X2|(X1, A1) ~
(28)
A2|(X1, A1, X2) = bz (H22, where (2(22) = I(X2 + 7 > 0):
1
[2| A1, X2, A2) (1 0.5,, 4x81 X2 31,X2) + (6),
where we define c (61, 62, 63, E,, Es) the noise terms which denote student-t distributions with different degrees of freedom to control the heavy tail level of the cumulative reward distribution. The target policy sequence at each stage is defined
(Ai|X)) = T(X; > 0},
(29)
72 A2 X1, AI, X2) I[X2 > 0}.
Consider the cases when follows t(2), t44,, t(6), t(8) and A/ 1). Unlike the single-stage setting, here we use relatively larger degrees of freedom in distribution since the estimation is harder when the number of decision stages increases.
The PDF R (T) (*)x) R²(x) )) is shown in Figure 7 and the probability plot of R (T) comparing with t(2) distribution is shown in Figure The true reward distribution R (TT) indeed quite heavy-tailed.
27

---

Published in Transactions on Machine Learning Research (09/2025)
Histogram R, +R
Probability Plot
1500
1000
0.3
Ordered
-500
-1000
-750
-500
-250
250
500
750
Figure 7: PDF of R² (T) when ~ 1(2)
Figure Probability plot of Rº (T) compared with baseline distribution t(2), shown by the red line.
B.1.2
Quantile Estimation Bias and Variance
Quantile estimation bias
Quantile estimation Variance
0.100
0100
0.000
0.500
0.055
0.025
Varisnce
0.000
0.000
-0.025
-0.075
-0.075
-0100
-0 100
0.0
0.2
0.4
0.8
10
0.0
0.4
0.8
10
Quantile level
Quantile level
Figure 9: Quantile estimation bias and variance at different quantile levels. The estimation bias and variance are quite close to at most the quantile levols in the middle, and the performance tends to become unstable when 7 approaches or The poor performance at extreme quantile levels is reasonable because of the lack of data points, which makes the statistical inference even harder to handle.
B.1.3 Bandwidth Selection
Since the estimation of JpR involves KDE, the choice of bandwidth h is also important problem to deal with. In Figure 10 we tried three commonly used methods: cross-validation, Scott's method, and fixed bandwidths where = 0.10, 0.15 and 0.20, to compare their performances in estimating the standard error of 7PRR_ Surprisingly, both cross-validation and Scott's method encountered some over-smoothing issues, which is potentially caused by the heavy tail of the reward distribution. On the contrary, a reasonable fixed bandwidth lends to stabilize the estimation of JPR, yielding a smaller MSE the standard error estimated all quantile levels. Since the performances under different fixed bandwidths are all pretty well, we will fix h 0.15 in simulation studics.
B.1.4 Comparison between DM, IPW and DR
As we can see from the comparisons in Figure I1 the performance of DM clearly worse than IPW and DR. The performance of IPW estimator was not badly influenced due the similarity of behavior policy and target policy under this specific simulation setting. As is expected to us, our DR estimator always yields the best result.
28

---

Published in Transactions on Machine Learning Research (09/2023)
hh- 30
14-
311
80
09
0x
00
0.4
1.9
10
00
04
Oasttlolios
Figure 10: The comparison between the MSE of the standard error at cach quantile level 7 with noise distribution t(2), t(3), and t(4).
Quantile estimation
Quantile estimation MSE
IPW DM
IPW
DM
DR
-8-
035
Figure 11: The comparison between DM, IPW, and DR estimators with heavy tail level t(4). Logarithm transformation was laken to better distinguish the performance of the three methods.
B.2 Real Data Analysis
In this section, we add the comparison between DM, IPW and DR in our real dataset.
As show in Figure 12 our doubly robust approach performs the best in terms of bias, standard deviation and root mean squared errors, for all the four policies considered.
C More on Theory Section
In this section, we provide several lemmas and proofs that are omitted in the main paper.
C.1 Lemma Some Basic Results
Lemma Suppose W = {(XR, As, Re)) the full data with baseline information X1 G, and f/r is the tth quontile of the potential cumulative reward function Rº (*).. Then have
Ew (w Sw(w"(W;n+)) O.
(30)
Proof. For the brevity of content, we'll only show the proof when can derived using the same logic.
2. The results in K stage settings
To prove Lemma we first separate vi (WWW:))) into three parts. Let (W";,, a b + where
(A2 (1( + Rg < 11 -T); a b1(A, H1 bby (A2) H2)
(31)
b of É[([ R R₂ k x) (A1, T].
c = ab [[(I Rı + Ru
< <
H)H A2 ~ T],
29

---

Published in Transactions on Machine Learning Research (09/2025)
Policy
Policy
Policy
1.29
Inetacs
Methock
0.10
0.03
3.99
3.04
0.30
3.54
0320
G4
490
0330
1.82
001
0.35
9.05
OlaB
Quarties
Querd
0.06
0.0
2.00
0.66
0.55
2.54
3.34
0.04
2.54
0.02
0.43
3.02
0.02
040
0350
Ourr
Our Kih
Ouil
Queii
3.55
0.00
Hethocs
Mechods
0.07
3.22
0.55
0465
3.54
0.4
2.56
0.0
0.22
1.22
0.00
259
30
50
53
30
43
70
Qkareiicc
Quartiar
Figure 12: Bias, standard deviation, and root mean squared error of quantiles over 100 replicates. The four columns correspond to four target policies, and each plot shows the comparison of three methods: DM-direct method, IPW-inverse probability weighting method and DR-doubly robust method.
where of and ag are defined in Formula 24 1. E[a] = 0. By change of measure theorem,
Now it suffices to show E[a]
E[c] = 0.
Ti H2)
Bia] = E
b1 (1))2 (1(R) + R6 <m))-)
= (T) - 0.
2. B[b]- O.
n. (AilHi)
EbD E
B[[[[R] + Ro < (AI, A2) ~T]
by (A,
T1 (A1 (H1)
(32)
E
E
Hh
([T RS(T) < n- -T]
(A(A)H
E 10- E (11 + (T) < +) 0.
3. E[c] = 0. Similarly, since
( (Ai (AI())H)))
(1- (2 [12)
Elag| H2] = E
H2
[11 H1)
(2 H22)
(33)
= E
( (mn(A1)HA)
(b((A,{H)))
E.
(1
(2(A2 H2)
H2
= E
(A, -0] = 0,
[TA H1)
:0]
(2 H2)
30

---

Published in Transactions on Machine Learning Research (09/2025)
then we have
[|c] = E
42
[ + Rg < - 7) m5]
=
H2 ([( R((()) < <n)
T) H2, A2 ~ π
(34)
E|0 ([( + (x n) 7) H2, A2 m] 0.
The proof is thus complete.
C.2 Proof of Lemma 1 Double Robustness of our quantile Estimator
For illustration purpose, we will focus on the case where K = 2 for simplicity.
Proof. Suppose without the loss of generality that the sample size According to our estimating procedure, the quantile estimator f/ is the solution to the following estimating equation:
S Σ En,a (W,,,, a.)] 0,
(35)
where W, denotes the data in Ig, and En,, is the empirical expectation over sth subgroup.
Suppose that A, is sequence of positive numbers that converge to O. We define Na,, (7>) {1 1 s } as shrinking neighborhood of n7,, According to Uniform Law of Large Numbers (ULLN), it suffices to show the following three conditions hold:
1. Sua (n-) 3 Eim Bn,, (W: a)] [ (W:))] (1(;
2. Ve 0, inf ( (W; m)] 1|7 7 2 = E [b (W:n)]]
3. Ee En,s (W; PR a)] = (1).
Condition 2 can be casily proved by the identifiability of nr According to Assumption (A3), there exists posilive constant C1, such that
a, (W; (W;n)l fn=(m) 2 C1
(36)
for all 1) E NNs (nr). According to Lemma 1, dr is the solution to Ew (V* nW;n)] = 0. Therefore, the claim of condition 2 follows.
Condition guarantecs the estimated 7P to be close the solution to the empirical estimating equation, which naturally holds under valid estimating procedure. To complete the consistency proof, it remains to establish Condition 1.
Define the empirical process Gn,s (Ws: az)] as a lincar operator on mcasurable functions sp via
Gn,, (b((W,nn vn
(WWn,a - E(o
Since
sup nENa. (1+)
5 En (WWnn)))) - (W;))]
8
sup (1-)
ICn,a (WWW; d.))l +
sup vENS, (n-)
E[ 6.)] - EE 166 (W; mi]l
#=1
31

---

Published in Transactions on Machine Learning Research (09/2025)
it suffices to show that for any 8 E {l,..., S), the two terms on the RHS of the above inequality are oy()). For the brevity of content, we omit the subscript 8 in Gn,s and og in the following proof to illustrate the general results that hold for any
Step 1: Prove that SunnENan (n-) GG,, (W n, &)]l = op((n).
Define F (W; à): 7pR E NA.. (M=) F is VC Class. By Conditioning on (Wa) á can be regarded as fixed value. Therefore, according to the Maximal Inequality in VC type classes (Chernozhukov et al. 2014 2018). there exist sufficiently large constant C, such that with probability 1 o1))
FEF sup (n)
C. (nm CC (ra (2/rr(((///rr))))))))) +n 1/241/9 logn).
(37)
where rn SupnEN Na. (n.) specific,
|o n, p:2 p.2+ and q 222 2 is an integer that be arbitrarily large. To be more (W 1, d)
In =uunnENa (n+)
(I(A1,)H1, | A2, (2,35
=suPnENAm (n-)
(I(R1,5 R2, < <n} 7)
(A1, H1 a2 (A2 H2,,,,,,00 (t)
T1 H1.)
[[ + R2 n/ r) (A,,5;42,)) ~M m|
by (A1,)
(A1,1HH1))
(2(A2.1HH211
B[ + R2 <n} T)1,1, A2,
61 (A
(2 H2,
P.2
m1(A (A1, 1 /22 (22, H2.:)
<. SsupyENan (n+)
[|I(B11 + R2 < <n} - =11
(A1. H1,5, az)b (A2, H2 ④c)
P.2
mi (1,))
[[( + R2 < n)
X1,i, (A1,5; A2,1) ~ n|
bi (A
P.2
P.2
m. (A1))
1 T2
[[[([[[L + Re < <n] - -) H2 A2, T]
+
To (AI,))H)))))
b2(A2.1H2.
P.2
P.2
1 < " 2+ 2+ = 2
< ~²
where the second last incquality is obtained from the Positivity Assumption (C3) and Assumption (A4). Therefore, Ta = 0(1,, and we have sup (G) s (1()) = =(vn).
Step 2: Prove that SuPnENA, (n-) E (W 77, 4)] E j (W:n)ll = o,.)
In the following proof, we use È to denote the error aggregated by Monte Carlo Method, and use Rs to denote the estimated reward obtained by MDN. Likewise, (T) is to denote the estimated potential outcome under policy æ at the kth stage. When M + oo, no error comes from the Monte Carlo method, and thus È = E. Let's first start with E [WW d.].
[bb m, a]] E
T (A1,5 (1)))) (1, H1.
To (2 H2,1)
E
(I(R1,, + R2, <n) -T)
7zl (A2,1 H2,))
nı (A1. H1,s)
E0 [i + Ro < n - 7|X1,1 (AI.,, A2,1) ~M
(A1, (
ni (A1,i (1.s)
T2 H2,5)
1
B I + Ro <n} - -/ A2, ~
7, (A,, (...)
(2 H2,))
32

---

Published in Transactions on Machine Learning Research (09/2025)
which is equivalent to
Ti (A H1,))
(e(AA,.HH22)
(BIT(RI,s+ + R((T) < n) H22 - [I RT(TT) < n) H2.ll)
E
(A (H)
(22,2 H22))
mi H1.))
B[I[R1, + (1(
< m) - -
(38)
b, (H,,)
Tı (A H1.))
([((RT(+)+ (T) nu [[
= L1 + L2 + L3,
(A1, H1.s)
where we define the last three lines in (8
L1, L2 and L3. Then
Ii (A1. (1.
L2 =E
B[ + (T) < n -/112
b1(A1, (H))))
T (Ar, (())))
T2 (A2 112,5)
BTT( + PT((T) < <n) - =/H2,
b2(A2,1 H22.))
ni (
(2 (22,))
[E[[ RG(T)) < n H2,1 R + R5(x) < ())
81 (A1,)
b2(A2, H2,1)
Ti (AA.(HHH.)))
E[I( + RZ(T) < <n) H2.1]
(A11)(H.))
where we add and subtract a term in the last equality to maintain the equivalence of the formula.
Plug in L2 to Formula 38 we have
E (W: :n, 17,6)]
M1 (A,.i (
(A2, H22))
) (BI(RL:+ R5(()) n) H2.1 EII(R1, + R)(T) <n) H2.l)
-E
(A1, (
(A,, H2,))
(Ai H1.))
(2(,2,
H2,s) (H2ss)
[Bu + RG(() < m - E + (T) < n H2.l)
(A1.) H1.))
(2 H22s)
Ti (
BII + (T) < n) T H2.1
(A1, ())
[[( + RT( < n) - H1.1]
(Ai. H1,))
mi (A..(HHH.
(A2, H2,))
T2(A2,1 H2,5)
-E
(2 H2,)
b2(A H2.s)
(E]TTRR, + (T) < n/}
H2,
[[ + ++)((T)) < n) H2.l)
(H,, (A,,
EI(RR.: + RT(T) < <n) T H2,l (1- by (A1. H1,))
t (Ar, H2,))
([((R((=) R.((T) < n -- H|H
Also, similar to the logic in Lemma 1, it's easy to show that the augmentation terms in vi* (W;n) are 0. Therefore,
nz (Ar H1,)) H2,))
(I((B) + R2 < <n)>> <7)]
E B
(A H1.1) (A2,,)))22A))))))) H222)
= (T) < <n) - -]
33

---

Published in Transactions on Machine Learning Research (09/2025)
Then we have
(1;, ;7, - E ( (W; = (W:
17, 6 - -8 ((() -1
n (AI,c)HH.)) T2(A2,1|H2.)
72 H2.s)
8, (AI.)) (11,) (2(A2,
bz(A2. H2,,))
(E]T(RI, + (T( < n7) H2
E[[(R1 R + RT(T)) < 7) H2.l)
(39)
Ti (AI.
BiI(RI + [Z(T)) < <n) - H2.1
ni (AI,sHHH,,) [[](R((T)) + (T) < イ - 7 H1. - [TT(( + (T) < <n) H1H
(AI,))))).))
Let's then consider the last two lines in formula 39
E
Ti
B[T(RR. + RT(T)) < v) 7 H2.1
61 (Ai,i) (1.,))
2i (A1, (+)
(T) + (T))) < n) -
+ #(()) <n) - - r|H
(H,,)
(40)
(
TI (A,,)|HH.
(BLL(R) (T) + [5) - + [T()) < <nH
(A1,HH1.))
+ (2 + (T) - + < Had]
Ti (Ai.l
E
((I(()(T)) + (5(+) -
R5(a) < H1.1)
(AI.))
n1 (A1,i (1.
[T) + RT(+) m) n) H1.1 - R + (T) nt (..ll
bi (A(A,,i/11,)
= E [ (I(Al
(i(A))HH.))
H1 - ii + (T) < <n]len.i))
+ (T( ((()
(A
(A,.)
Combining the result of Formula
39 and Formula 40
we have
E [WWWW; ; 17, a)] - E [1) (W:;))
TI (HI,))
T A2. H2,2)
(A2 H22i)
E
(AI,)
b2 (A,,, H2
b2( H2,1)
(EII(RI. + RS(T) 7/} H2.1
[[[(RI, + R2(TT) n (nHHH.1))
(A
(B[I(R((T) + R5( < < H -[ +)())) (m)
E
G. (AI. (An, H1,))
(D2,A,,21/22 - (2,, (222))
| (Prgt))a ( )
3
P.2
P.2
+ 2 (A1.)) - (A1. H
1
||0 (FReG))), FR-()) P2
P2
(41)
where the last inequality holds by condition (C3') and Assumption (A4).
34

---

Published in Transactions on Machine Learning Research (09/2025)
Therefore, as long as the propensity score models are consistently estimated, or the outcome regression models for Ri and R2 obtained by MDN are consistently estimated, we have
E (WW a)] B [v3 (W; o(1)
according to formula 41
To be more specific, we only need one of the following conditions to be satisfied:
(A(())))) bi(Ai)Hi) P.2 o(1), and [ (2 H2)
o(1);
P.2
2. 10 (E
o(1), and |o (FRAGO))))) FRAGO)H.) o(1)
The proof of Lemma. 2 is thus complete.
Notice that Assumption (A1) is not necessary prove the double robustness. This assumption is imposed further prove the asymptotic normality of our quantile estimator in Appendix C.4
C.3 Proof of Lemma 2: Double Robustness of our Variance Estimator
Proof. Firstly, we show that when lim hn = 0, and lim Nhx co, the kernel density estimator for f(r)= == (r) attains weak consistency.
Let's first define (r) as the KDE of f(r). By Chebyshev's Inequality, at each point of continuity r of f and for any c 0,
P((SN(r) - s()) 2)S [[(s(r)- - [(r)) Var[/N(r)] + Biass[[N[[
2²
2
According to the result in Parzen 1962.. limn +o0 Bias [fx()) = 0 when lim by 0. That is, fx(r) is asymplotically unbiased when lim AN = 0.
According to the mild assumptions on stated Lemma] is given by
2 the asymptotic variance of the density estimator
f(r) [ K² oc.
Therefore, both the asymptotic bias and variance go to 0 as - oc. This illustrates the asymptotic consistency of the kernel density estimator in ōbR.
The rest of the proof is trivial. One can follow the idea in Lemma 2 to similarly prove the double robustness of our variance estimator. For the brevity of content, we omit the details for this part.
C.4 Proof of Theorem 1 Asymptotic Normality of our quantile estimator
Proof. same as the proof Lemma 1-3, we will only illustrate the case when K 2 for the brevity of content. We will give a brief sketch in Step 1, and leave the rest of the details in Step 2-5 accordingly.
Based our estimating procedure in theory section, have
s
En.s (WW: a.)l "1 IEZ.
(W r.i 7 (..)) = 0;
(42)
El (W = Ela + b bbbc = 0 for any (I,..., S}.
Step 1:
35

---

Published in Transactions on Machine Learning Research (09/2025)
First, apply Taylor series expansion [l("((Wn) around 1/.-
= (Wy [1-] + Jo(n 1/-) + 1
2 Ho(T - 11=) + +(ll1 - n=113)
(43)
Jo 1/-)
1
Hn 1² + o(In 1/112),
2
where Jo 8 (E (W* (W; n)]) 1y and Hp = 0g Ew (00" (W; (Wi))Hler
Suppose that E s
(W RP
-I 3/- (W ワー + (We will prove it in Step 4). Since
(WW:n))) = 0, by plugging in the definition of empirical operator, got
Vn
B Bv si DR .a,) - 2² (Wain+))
Gn M7 DD a.s) 30 (Wa:n+)) + vn
Bl ds)l
(44)
8
s
Gn. ( si 9
PDR (2)) 3" (Wsn:)))) vn
Σ
(W, W. + (1).
Let 1/ in formula
43 and combine it with formula 44
Then
s
Vn
Jo 17
DR
11) + vn Σ
E En 16 (WWs:)) =
G G (W. a.,) -00 >'Wwin=)
s=1
s=1
(45)
1
Ho
DR 17
n,, 2= vn - o( 17, 112) o1().
2
8=1
Therefore,
s
VS.
VN
Jo - 11)) +
En, 1 (Ws:17) 1]]]
S
8
Jo(
DH
-m)) + Σ
En, (+ (W (Woi 5
Σ
Gn, (W.i DR ,.,) - +/- (Wan
a=1
v Σ
Ho (Tr
7)2 + Vn o( ( nr 11 + (1).
2
To finish the proof of this theorem, it suffices to show that
VN
Jo 1T) + S
1 Σ Bu (W =1p(1),
(46)
which is satisfied if we can prove that all terms on the RHS of the inequality in formula (46) are (1). That is, we only need to show
1. Σ Ge. 7 a.) - 10*h (W (Wain=>1 = op(,), which follows directly from triangle inequality we can prove for any s E SI,
...)) - (W (W.i = (1).
2. ||TPR 7/ || (1/221
36

---

Published in Transactions on Machine Learning Research (09/2025)
3. B| (Wa; RD. á.]] = E (Ws: )] + o(n o/2)) which follows naturally if we can prove for any 8 E (1, (1,...,,)) S}
(Wv PR a..) Blo""W,;; Rp + o(n 1/2).
4. Ho is bounded.
Detailed proofs of these four conditions provided in Step 2-5. Assuming condition 1-4 hold, have
1 5
N
J0
DR n7)
B W (Wain>)
= o,)
(47)
then
1 8 VN - 11) -0
N 3 (WWn))++pll
Σ
En, ( +z))
5
The claim this theorem thus follows.
Next, we will prove that all of these four conditions are satisfied in Step 2, Step 3, Step and Step 5 rospectively.
In the following proof, when the result holds for any fold E (I,...,,S) S}, the subscript in Ws. Css Gn,s and En,s will be omitted to avoid the redundancy of notations.
Step 2: Prove that (W::: RPR a) - & (W: - o1))
Define No. (++) {n: : ||17 - 7l S Tn} as neighborhood of п with size Tui and define F= (W(W: a) (WW () :n E Nr., (n>),, where Tn is a sequence of positive constants converging to zero (According to Step 3, Tn can be any sequence with order Tn ((n-1/2)).
Since
GnIW(W; AP a) 2 (Wi x sup [ (W: a)ll NENT, (n+)
(48)
+ [[ 6) - 3/2 (Win>)lls
It sulfices to show both terms on the RHS of Formula 48 is (1))
Step 2.1: Prove that suppeNs, (v=) ||G, (WW a) - (WW &)]ll = (1()).
First, it can be shown that F0 (w> 77, 6) : Nr,, (n=)) is a VC Class with VC-index V(F)). Also, for
some q > 2, the entropy is bounded by some constant C2, i.e. || C2. According to Lemma 2.6.18
in Van Der Vaart et al. [1996), FF is also a VC Class. By Conditioning on (Wi)ees, á can regarded as a fixed value. By applying the Maximal Incquality specialized to VC type classes (Chernozhukov et al., 2014). there exists sufficiently large constant C, such that with probability 1 --o(1),
G, Gn())) S C. (rx Jowlog" (31r/((1/rN) + 1/2+1/1 10 FEF
sup
(49)
where
FN
sup
[\,(W,, +W;
(50)
1y- ITN
If we can show that rx = (1), then sup (G,(f) = o(1), and the claim of Step 2.1 follows.
To prove this, we split sb into three parts and show that each part is indeed o(1). Let (WW n, 6) a(n,
b(n, â) + c(n, where
a(1, ()) Ti (A (T(R: R2 < <n) -T).
(A (1 (Ag
(51)
b(n, &) aj (ā) [1 Rg n - -1) A2) ~ n]]
c(1, @) = (σ) (I + â < n) - H2, A2 x],
37

---

Published in Transactions on Machine Learning Research (09/2025)
and a1 (ā), aá) are defined
(i())= ( (AI,i)HH.)) bi (Ai,))Hi,,)
[AI, (1,,))
(1
T2( H22s)
a2(x)
(52)
(bi
b2 (A, H2,a)
Specifically, when the estimating equation is + (W;n), we let 1 (WWnn) = a" (n) + (b((()) + c (n).
Step 2.1.a: Prove that supneN,. (n-) |a(n, a(),, a o(1).
sup nEN-, (n=)
|a(n, â) a(Trr a)]l
(A H2) (1[B) + R2 < <n) -
sup veN. (n+)
1R + R2 <
17
(02 H2)
P.2
(T(A)|Hi)TT( A2 H2) If R1 + R2 E [min(0>> 7), max 11))
sup veN.,,
;i(Ai|H,) (12
P.2
mi (A1 (22
$
(A H1 122)02(241 H2)
sup NEN (n.)
E [( E (m:n n1), max(n 7)] )13)
(53)
m1(A]HH))) (A2 H22)
E [B[T(R' R (T) E [min((- n), max(mr m))3
sup neNT (n+)
(A1
H2)
P.2
sup nENT, (n-)
(F) (1) (z)
EHinG [FR-(s)),, (1) - (R) (nr (9r]H))]
sup JEN.mm
2²
where (1) is the marginal cumulative density function of random variable (*) = (T()) + (T).
Notice that the first incquality in (53) holds because of Cauchy-Schware incquality, and the second inoquality holds by Assumption (A4).
According to Assumption (A2), there exists constant C3, such that for all r E I and H1, fZ (r( S C3. Thus,
(1) H,) - (R- (n-1Hi)) s C3 ||n nrll-
(54)
Then we have
sup
||a(n, â) - a(T,, a))l
n,,, (1+)
1
EHING (n) FR-(x))M ( [H]]|
sup
(55)
yEN, (0+)
1
mm)) - o(1) = o(1),
< σ
sup
yEN= (m=)
which finishes the proof of this Step 2.1.a.
38

---

Published in Transactions on Machine Learning Research (09/2025)
Step 2.1.b: Prove that sup (n-) [b(n,, b(1r,0)lp2 o(1).
sup nENs, (9-)
[16(7, a)) - b(Tr,, Pp2
1 (A1. (1.))
(T)
+ Rz(T) < (z) (T((T) H1
sup neWi (9,)
bi (A1, (1,2)
P.2
= sup
1
ni (A,,))HH
(Frocm) (n
a(H)) (n:H11 FRe(F) n. (1+i (ft:
(56)
neN. (9.)
bn
P.2
mi (A1,i (1,))
(R FRI (1: (1:
sup (n-)
by (A1,i H1,s)
P2
P.2
2
F (r (mā)HH) FRIGII (1ri ( P.2
sup
nENT, (n+)
where FR (r: (r: is the estimated edf of (Ra + R2) given baseline information H1 and policy n. Also,
we have
|| (n Fr)(F)) H, (m::
< (n (m. Fre (=)) (n) H1)
P.2
(57)
+ (e (H) FR (0)8, (1-
+ ([ (R) (
P.2
Now let's analyze the three terms on the RHS of formula 57
On the one hand, according to Assumption (A1), for any
(1: PR (1: (m; (
$ |/0 (FR (R) = o(1)
(58)
The first and second terms in 57 thus be bounded.
On the other hand, by assuming (A2) and utilizing the same proof as in formula 54
sup neN, (nx)
F (m)), (1)) H2) - (n = o(1).
(59)
Combining the result of 58
59
with formula 57
we have
sup
|lb(n. à)- b(nr.))lla a)ll
(n.),,
S
sup (n=)
(1 - F (1s:)
-((1 = o(1).
The proof of Step 2.1.b is thus complete.
Step 2.1.c: Prove that suP,eN, (n) ||c(n, à) - c(n d) o(1).
Similar to Slep 2.1.b, it follows from Condition (C3') that
sup |,(1, a) - c(1r, a) neN,, (n+)
(z((a) [[ + RT((T) n) MR PT((T) < nH H2.
sup neNr, (n.)
P.2
(60)
(2((a) N8 [ R1 + (T) [min()); m)) max(1 m) H2 P.2
sup (nN...
< <11aa(ā)llr.2" (
Fegem)H R1; R1 H2x) FRgeo)u, - -R;
sup
P2
39

---

Published in Transactions on Machine Learning Research (09/2025)
where (R ((2))) is the estimated edf of R2 given historical data H2 and policy и. We know that
(o((A) (
(1 (2(22, 2
[aa((a)]
(61)
(22)))H22))
2
P.2
Also,
F (n R1; FRI(H))HH R1; â H2,1)
P.2
Fngtt))lu R1; & FRz(M))M R1 H2,
P.2
(62)
|| [RRGGO)H (11 - R1; H2.)) Fnz(z) - R₁ 2,
P.2
(RGGO))HS( - (2.,) Ha - R1|H2221) P2
where according to Assumption (A1), it holds for any v that
PRgie) ( Ri; a(H2,)) FRI())HA Rí (21H
S ( o,(), (63)
P.2
and by using Assumption (A2) and the proof in formula
54
sup [Fng())1 (1 R2 (22,s) (nz R1 H2 o(1). N,,, (m-)
(64)
By summarizing the result formula 63 and 64 have
the LHS formula 62 is thus (1)) Therefore, we
sup |c(n, () c(ur, 6) nEN., (n+)
< [1a2(())llp;) p.2 yeN,, (1-)
sup
(n H2.5 Frgere)ua - RR; &
(65)
P.2
2 2 o(1) = o(1).
Combining the results in Step 2.1.a-2.1.,, we have
|G, [G,,)(WW (WW; 4)]l = (1).
sup MEN. (n))
The claim of step 2.1 thus follows.
Step 2.2: Prove that [G,[vWW;;) nr. â) 50 (W:n+)ll = op().
Notice that under each Fold 8, independent with W. By Chebyshev's Inequality, suffices to show that
Tar[G,[W(W" nr, a) (W; n-)]] = Varls:WW; à) 3 (W; = o(1).
(66)
According to definition in formula 51 it follows from triangular incquality that
(Wa *² " < (Var(a(),; â) (n2) + (Varlb(n,; 55 (1+))) + (Varic(nt. a) - 12(
(67)
40

---

Published in Transactions on Machine Learning Research (09/2025)
in order to show the LHS is (1(,), we only need to prove that the three parts on the RHS are all 0p (1).
(Var[a(1r: a)- a" (1)+ 1/2 < |a( (:) a" (1)+)l
Ti (A₂ H1) (A2 (4g H2)
(1 H1 (12 H2)
(I(RI + R2 <m²} -T)
by (Ag Hz)
(A H1) A2 H2)
P.2
(A,lH H2)
(1 H1 (1 (A2 H2)
[[[(R] + Rg < < T =1p
P.2
(68)
(A H2) (A, H, bb (A2) H2)
P.2
2
(A(1FT)))) )
(Ta1))))T((A421 (22 H2)
b2 (22 H2) }
+
p.2
P.2
2
|10 (AI1HT) - (4, 1 P.2 +
b₂(A21 H2)
o(1);
P.2
where we used Condition (C3'), Assumption (A4), triangle incquality, and last equality holds by Assumption (A1).
Likewise,
()) -) (arbbn,.)--b()))))""""(000000 =)
[a))) (T) -)] - -a [[([( + RG(=) <m} 7)1
P.2
< (a) [[ + (z))) ()) H P.2
RT(+) < 7r } H
P2
{ (a) - ai}
+
(69)
Ti (1).)
[| + (5( < (T) [T(T)) < |
2
(H)
P.2
P.2
m (A1. H1.)
(A H
[[000) + + < < - 1)
(AL. (..))
by (Ai,a H1,))
P.2
122
2 || [ & - Fn)( (n- H1)
2 (1 - by (AI H1.
+ P2
= o(1)
2
122
Under Assumption (A1) and (A4), the last equality in Formula
69 holds because
F (=) (7-111)
$ |00
=()).
o(1).
R.2
Similarly,
(Varlc(n.. a) - (n-))) 1/2 < [|c(T,, Q) - c² (n+))
|o)(a) [[ (T) < nx) -) ob [[( + R²(x) < < 7) H2]
P.2
(a) [BIL(R: + (T(T)) < m - I + (T(TT) < n ) H2]
P.2
+ | (a) - ai} [ + R5()) < 7/} - H2) Hz
P.2
2
BIL(RI + RG(T) < < } IRR - (T)(TT) < n. HH²
< _
P.2
+ 3
4
{ || - (A
+ (Aib(AA)HH)) - (e
P.2 }
P.2
2.
|PRIG)IMM - R1; &H22)H2) FRI (nr R1HH2)
+
3 o()),
4
2
P22
41

---

Published in Transactions on Machine Learning Research (09/2025)
which is thus bounded by
2 ||o
(ngee)) + d11 d1),
P2
where we used Assumption (A1), (A4) and Positivity assumption (C3').
The proof of Step 2.2 is thus complete.
Step 3: Prove that || 17r Op(n-1/3)
Similar to (W,,;,)),) we split the original optimization problem ((W.,,))) into three parts.
V(W;n,;)) A( + B(n, () C),,
(70)
where
A(7, a) T1 (A H1 (22 H2)
Pr (R1 - RR - -n),
(A, H1 b2 (2 H2)
(71)
B(n, â) aá) Blp, (R: + 1) (A1, ~ T], H2, C(n, () a2(á) [lp + -2) A2 T).
and a1 (â),, az( are defined in formula 52 Specifically, when the muisance paramcters á is replaced by the true values, we let V/" (W;;)) = (r) + B (n) + (²(()) in the same manner.
(1)
Furthermore, we denote (n a) = ( 7,, a)] as the empirical average of our optimization function, and (n, d)) (W 77, 6)]. When the models with nuisance parameters are correctly specifiod, we define (n) = E., V (W; and M (1) E (W;:
According to Theorem 3.2.5 of Van Der Vaart et al. 1996). to finish the proof of this step, it suffices to show that for every n and sufficiently small ö, the following two conditions hold:
sup Mn(n 6) (²) - Mn a) + M( S Vn:
There exists constant C4, such that Mn) M (n+)) 2 C4 || 17, 112
*Note: This is an extension to the original proof of Theorem 3.2.5. The result of Theorem 3.2.5 does not change after involving muisance parameter å in this case. Morcover, notice that the two conditions listed above hold for any s E S}, and we omitted the subscript to avoid redundancy.
Step 3.1: Prove E sup S [M,(n. -M - Ma(n,, a) + M (1+)) s VR-
Since
E
sup
[M, a) (n Mn(n,, M (1+)
|| ||--nl1l
<
E
sup
[M, (t) M(n, Mn(1,, + M(T. a)]
(72)
||
[1, M(n. a)) (M -
sup |<6
1--n
to finish the proof of this step, it suffices to show that the first term on the RHS of Formula. (2) is less than v and the second term is actually o(n & || - 1=ll). We will prove them in Step 3.1.a and Step
or equal to à
3.1.b accordingly.
Step 3.1.a: Prove that E sup (M, @(n Mn(n. a) + M(.. a)l <
5 vn.
Thus, it is equivalent to proving that
I sup (n, M(), Ma(Tr;)) M (1T, ((r,
S 5O(nn112 1| -011).
(73)
"
42

---

Published in Transactions on Machine Learning Research (09/2025)
Let F3 (V(W:n.d) n.: ||n- 17 |l - 6) ,nd J, (V) 1n 53.
We claim that for any and 7/2 satisfying ||7 7/|| - 6, there exist square-integrabl function M (W;á),
such that
WW (W; 122 a)) < M(W; Ilm - -2ll-
(74)
To prove this claim, we split formula
74 into three parts:
NWWW; m, d)) (W 72, a)] S IA(W; m, 12;
(75)
+ BBWW m. a) BWW 72, x)] + (C(W:n, a) 12))
Without the loss of generality, we assume that 7n < 1/2 Recall that R² (Tr) is the conditional cumulative reward given H1. Since
R (T) T
IJ 12
- (2a)) < m2) (R' m)]((R(() (T) < <m>>
['( 12
71
R (T) < m2
(76)
0,
R" (x) > 1/2
we have (T) - (2 (T) < n2) (R) 1 [( < <mll s 11 m ll-
Thus,
A(WW,1100 - 12;
m(Ai)H))myAA)lE (p- (pr + R2 m) pr (R1 + RR -2))
(A1 HinHHH (22 H2)
1
1 2
(p-(R" (5) - m) (R" (अ) - 722
(77)
$ 2
1
{
(I -m)l((t (T) < m2 (T) m)] <m3l+ 1172 -m
+1 l
<
|1/22 mll-
2
Similarly,
BBW: a) - B(W; ((W:7,,)) ,2,
= (G) (R,, + Ri2 -m]) -P+(fi + R2 - -11) Hi
(78)
- i pr Ri + R₂ - - -p-(-R + R₂ - -mz) Hi $
2(T +
1|122 - mill Il-
e
and
CCW m/, à)) C(W: 722 a)l
(a) E (1 + R2 - mm) - Pr (Ri + RS 720 n22)(12]
(79)
s 2(T + 1)
p (Pi Ra + R₂ - -m) - Pr R2 + Rs 12) H2
||12 mll-
Combining the result of formula
77
and 79 we can simply set M(W; = 5(7 + 1)/2, so that
the condition in Formula (74) is satisfied. More specifically, VWW; â) is Lipschitz continuous of order 1.
Therefore, F& has envelope function F := 8- ((;;)) Based on Theorem 2.7.11 and Lemma 2.7.8 of Van
Der Vaart et al. (1996), its bracketing number satisfics
C8
(2e ,F, 11-1) < N(e, (m= 1Ill 1 s 01. -
(80)
Hence, according to the maximal incquality with bracketing numbers, we have
= /
[FIII Vlog (i, de 3
E [[VMnMM, - M(),))))
(81)
43

---

Published in Transactions on Machine Learning Research (09/2025)
Plugging in the definition of || ||,,, it shows that
8
E sup (7, a) - ((n. à) Mn â) + M(n. a)] S 0O(n <02n-12110 17 - /ni
Thus, the claim of Step 3.1.a follows.
Step 3.1.b: Prove sup (M, a) (1). .)) - (M - M* o(n ||m 17r Il)
By using exactly the same strategy as deriving formula (41 in Lemma 2, we can obtain similar results for (n;;
M(1, a) = E/W( - E[L"
Ti (A)
T2(A2|H2)
(2(Ag|H2)
(Bip, (R1 + Rt(T) - 17) H2] - Elp, (R1 + RT(TT) 71 Hal)
=E
(AH
(22
(2(A2)112)
(A)
(A, (h)
(R)) +R - (TT + - mm)
+E
bi (Ai)(A)) (A)
(82)
Similarly, we get the expression for M (1), -M* (m+) by substituting 77, for 1j in Formula. 82
Therefore,
M Q) - (T) -
a) - ( (n+))
Ti (Ai Hh) (1)
(T2(A2)
(22) Elp + RE(TT) 1)
E
bi (A,,
(12)
b2(
(R1 + R(((T) - 1) Pr(Ri + RT(T) 17) Pr(R1 + - n H2l
(83)
TT (AI n (A, H11)
E
-p(p-(Ri(T) + + -)) 1))
(i(A))))))))))
(11 H1)
(R) (m) - -
+ -))
( (R(=)))))+))))))))))((((((()))() n- (1))[11))
Let's first focus on bounding the last term in formula
83
E ps + + - 7) - p. (fG + (T( - 11)
Pr
+ RT(T) (
nT) + p. ( (अ) + (m) - n:>) H,
/ (p+(r - 77) - - (r (((--)) n)
(84)
(fR(()) (r|H1) fRI( (r(H1) dr
/ (p-(r - 77) - p+(r 7) (1) dr.
JreR
By mean value theorem, for any fixed r, there exists constant Th E (min(n, nr J. max(n, 7r t), such that
pr(r nr) - pr(r- - n)= = (r - I(r < k) -
(85)
Therefore, for any r, holds that
le-(r 17) pr(r 1))l 1(7 Thr < 7))) |1> 7-ll S 1|n n
(86)
44

---

Published in Transactions on Machine Learning Research (09/2025)
Under mild conditions and Assumption (A1),
E (R) +
+ (T) 7) p- (R) RT((T) 11) (R) [T(()) 7/) + p (T) ) RT(TT) (
$ / p-(r- - 11-) --((r - [)]l [.. (r|H)) ( (r) (r) dr,
(87)
FER
$ ||1] 1/ll / (m)ia (i) dr
= || 7r |l 26 (FN-GR) Fn-(=))....
Similarly, we can prove by Assumption (A1) that
E (R1 + (T) 7)) - Pr (R,) + (T) -)) - (r + RG(T) n:) + Pr Pr + RM(T) m Ha2
1 / (pr(Rr + - 17)) - Pr (R1 + r 111 (r fRi(m)) (r( dr
PER
(88)
[ JTER (T)(,)) (2) fr dr
$ ||17 mrll
|||
11| 28 (PRz FRG(H)HH.)
By summarizing the result of Formula
87
and
88
we have
M 6) (1) (M(T,: () M (n+))
Ti (A1HHH)
T2(A2 H2)
(A2()HH2)
-2|l - 1 110 (Fngen))):
(A H1)
(2 H2)
(2(22
P.2
(A)
(A4
H+)
2 ll 7/ ||o Fn
(89)
(AI(H,)
H1)
P.2
Fr (22 (2(22 H2)
2/ 1/1 ||o
(FRgG))Hs PR((s)) P.2
3
P.2
2 (A(HH) by (1) |
2 2
nl 8 (e) p.2
onnnt |17 77 7711),
P.2
where the last inequality holds according to Assumption (A1).
Step 3.2: Prove that Mn) (n+) 2 Ca - ||n 111??.
According to Taylor series expansion, we have
(n) M (n+) + EEU (W:m nx) 1 (W: n²)² +lln By 11113).
(90)
in which BWW>(W;ww) [-)] = 0, and
(Ailf))) (T(R) + R2 < <n} -7)
F (W:: v) O,
+ +,
[a H.
(BT) - -
(91)
+ +,
E
[
B(LL)) < -)
= En (1) + (R) 20
45

---

Published in Transactions on Machine Learning Research (09/2025)
where (=) n, (n) is the pdf of random variable R[(T)) + R2(T) given baseline information H1, and (x) is the marginal pdf of ([()) + (T) given baseline distribution H1 ~ G.
Therefore, by Assumption (A3),
1 (n) - (m)) - (m) n2 + +(117 n,113) 2 C4 ||7 n m11
(92)
where C4 can be defined /2 - 8 with small enough å such that C4 > 0. Therefore, the proof of Step 3 is complete.
Step 4: Prove that 9 (W; n. â) 2 (W:n))l = onn-n1/1
According the derivation we did in formula 41
sup 11-1-11
(W 17, () 4²" (W;
T1(A1,i 42(A2, H2,))
T2(A2, H2,5)
|
sup
11:6
E
(A2, H2,s)
b2(A2, H2,s)
(EII(RI. + (T)
m)| H2.1
B[,( + < H2.1)
(A H1.0)
(93)
E
bi(Ai,s) H1.s)
b (AA (1,1)
(BII(RT (T) + (T)
< n H1,1 - + R²(x) < n) H1.H
[A2 H2) bz (22))H22)
P:2 ||0 (PREGN)) (R
FR(60)H)))
P2
1 + 2 [((A1111) - (A, (4))
P2 ||5 (FR- Hy. FR)()))))
= o(n - 1/2),
p.2
where the order in the last equality comes from Assumption (A1).
Step 5: Prove that Hy is bounded.
In Formula 91 we obtained that 0 (Wan)) = EHInE (f) Hz]] = fn)(x)(1). Therefore,
Ho = (W 1=
= (0) カ=
(94)
This condition holds naturally according to Assumption (A5).
By combining the results in Step 2-5, one can follow the structure in Step and the whole proof of Theorem thus complete.
C.5 Proof of Semi- Paramearaietiic Efficiency for 1)
In this subsection, we proof the semi-parametric efficiency of the DR estimator 7,RR proposed in Equation There are several ways of finding the efficient influence function (EIF) (see e.g., Kennedy 2022 Hines ct al. 2022 Levy 2019). In our proof, we focus on deriving the explicit expression of the Gatcaux derivative, and the EIF thus the unique mean zero function whose inner product with the score function equals the pathwise derivative of ())
46

---

Published in Transactions on Machine Learning Research (09/2025)
Recall that W {(X,, R+)] is the full data with baseline information X2 G. For the brevity of notation, define function (n, ()) as a) Ew (W, 17, a)], i.e.
K
K II
(n, = (1) - Σ
Jm (1{
Th <1)
-r)
The (a(ax)ha)
(ax kmm
k=1
(95)
In Mn (Tal@x (g) S lāg a. (nx M j Jdlra
Notice that when & L, manually set {AR-11 HR-1} o, yielding fú, 1o, =((x) G. Therefore, by definition, U(b,,a") a²) Ew (WW T7, a")] = 0. The efficient influence function (EIF) of 1/ should satisfy
a
T (a)
(V(WW s(W)],
(96)
da
a=ar
where s(W) is the score function of the entire data sequence W,, given by
K
s sW)
3 ((W) Da
Σ
(o log(b(A,, + Hal
da
Our objective is to derive the explicit expression for V(W; a²), which the EIF of Tr-
To do so, we start from the LHS of Equation 0G For simplicity, we will omit the subscript T mr when necessary, and denote 7/ as the true quantile value under the given target policy æ. Since Unr, aa) 0, we have
0 (a) a=ar = - - U(naa))
UU a) ]=ar] da
(97)
da
On
m=
On the RHS of Equation
97 the first term
8 a')
(U)
can easily simplified as
Bn
1=
(n, a²)
(F) T "= = (n=())(1)),,
On
"=n"
which is positive constant for any given quantile 17' (Notice that the positiveness is guaranteod by As- sumption (A3)). Next, focus deriving the second term of Equation 97 i.e. da U a))
Taking
the derivative with respect to a, we can decompose da U( a) a=a"
into two components:
i
U(n a)
1 (K)
(
(98)
da
a=a"
where
K Srurttte (K
K
((() =
Σ
II W=l
Th Fulaw (bwr) (If
Σ
Th < 11 -7)
JRalAe He (rn lāx, bu; (i)
Fix
k==1
II
JRalAn (rwlag law, ax
II SHolAe- (hw lww hw Jdw
K II k
K
Σ
ma(av (hrr) (I{En)
rw < <n] -7) Za {H, 1ag-1, (he la hx)
k==1
K
K II
II
Fnwlinw (rwllag, hw: a)
Smolāv nit (hw aar /ww /d fdfhor
47

---

Published in Transactions on Machine Learning Research (09/2025)
Step 1: For T we can further derive
k-1 1
Tge (Aw HK))
k'zk Seee Ahber b l==k
K
((K) = EB HH
Ax. R
II xx(ax (x(a)(lb)))
da J Rul x
by (Aur (Hwr)
===l
low Fi -
(1{
Σ
K
rke < |1 -r)
II
K
(BullAn }
K II 1o=
{ [Heldw- H. } (dlr
===1
=-=
Since
= da (log FRilAn, fnulAe,
da Bolde,
= B[s( Hy: Ag. Ral E[s(W) As, Re-il SRelAr,He
we can plug in the result above to the expression of T which yields
T
E
The (Akr (H)) by (Ax Hx)
II
K az(aw (gg) - (If Σ Tw < nn m
fau =
B[s((W)|HH H AR, Rul] B(s(W) Ae, Rr 11 (drk))dttk))
Tee (Aw (Hx))
E. Me Ror {1f
Σ
K
Rs ]1] -r) E[s(WW] H Ar. Rul
E
b (AN (
==1
The (AN HKr) k:
{11
K
RN < nn [n] -T) E[s(W) Av,, Ru Ra-.1) 111
EA MN
bue (Aur (
Wml
E
The (Aw (Ww) by (Aw (Hw)
E
(I
Σ
Re < kn]-n)
Hes Ak: Rx (((W)
The Thr (Ax)) II k==1 b (Aw li
k==1
The (Av [Hw). by (Aw [Wx)
{
(Fr-tt)) (1) F F s(m)]
E
where the first expectation after all the equality signs above is taken over (Hg, Ak, Rk-1). The derivation above works for all k E (1...., K 1}. When k K, special case,
K
Th (AN (HN b (Awrl (r)))
7(K))
Σ
RN < <1) - -r)((W)
#=1
K
The (Akl Hx:)
E
(I{
Σ
K
Ry <n} -r) H Ak. Rx- Ru-1 )01r))
(
(99)
by (Aw H
Wml
E
Ter (Aw (K)) be (Awl Hx)
{11
Σ
K
Rw < m -7) FF H.A. s(W)
k==1
Note that in the first line of Equation 99 the expectation outside of the indicator function vanishes because all observational data (Hg, Ar, R&) were given when k = K.
48

---

Published in Transactions on Machine Learning Research (09/2025)
Step 2: Similarly, for T have
hh = -EHI-T At Rx-1
The (Aw (KW)
K22k (ow
II The (aw (a lo
K
dx Hel 1,--1 x
by (Ax+l (Hw)
k==1
tan Na
(u{
Σ
K
< 1) -T)
(mulldrtthe }. k'=k
K II
II k=
K
{ fHulde- (dtw)ddhhes
Tk'
==11
Since
i da
f
Hal 18-1
da (l1g folsHHine-.Hee H He
[[]s((W)) As-1, Re-1] B[s(WW) Ar-1, Ra-1l3
Snulde M. 1
we can plug in the result above to the expression of T which yields
E
The (Apr (Hw) bw (Aw (Ww)
Σ
M26
K II kr=k
(aw)))))) (I{
Σ k=1
Th < <n -r)-
II
(aw b
Stu Agr ([s((W))Hr Ak-1, Rx-1l E[s(WW) Ax- RA df
Teg (Awl HK)
(1f
Σ
Ry < 2n] -7) 1[s((W) H4 -
Ax Rs Ra--l1
E
E A He. Re 22
bye (Anel Hw))
W=11
TTy (Apil Hw))
BA BA H Re 22 (T{
Σ
K
Rv < =)]-)) [E(s(W)) Au RI--1)
by (Ak. HKr)
k=1l
Tee (Ax' H by (ARr H
K
E
(1{
Rs < 11 -r)
H Ag-1, Rx-1 ((WW)
1
The (AN Hw) A
B
{1{11
Σ
Rw <1) -T) H Rr- }o(w))
byr (Ax Hxr)
#=1
Ter (Ax Hw). by (Akl Hx)
n. (n) F FRI())), n. R. (m) s(W) {FR-CR))MM
where the first expectation after all the equality signs above is taken over HHE11,A-1 RA-1).
Similar to (k), there is a slight exception when 1. This happens when the target policy is inconsistent with the behavior policy from the initial stage, which corresponds to the DM estimator component in the final expression of the EIF:
K R < ] <n]-r)
-r) Hi ((W) = E. (FR-))))))))) - -r] s(W))
T
E
I
49

---

Published in Transactions on Machine Learning Research (09/2025)
Step 3: By combining the expression for T/ and T22 have
derived in the previous two steps with Equation
① Un" xx)
T
(R)
K
Tplk)
T(()
K-1
T
(k) + T
} +72 T,
2
da
a=a"
K sWW)
Te (Ax Hx:)
{
{I{
Σ
K
R < ns -)) -Fn- (n)
by (Ax. Hxy)
An
k'ml
The (Aw (HW) bw (Av Hw))
(FR (n) (T)
k= ===1
F He -- - 1s (n)} (20))+(FR(CC (FR-()))n())) - +}
The (Ak HK.)
{ Ri n - - -. (n)
E
sW)
by (Ax Hxx)
K- k
Tw (A [kw) by (Ax
(F (1) -F (
k=1 k==
+ (Fe())),, (1) - -7]
where the last equation holds since Ae (e (1)) given that Th known.
Define C(n²*) = (n) as a constant independent of v,. According to Equation 97 the EIF of r is given by
EIF, C(1")
The (AgHHH))
(
(T
K
R < n) FR (
by (A)
K
The Th (Aw Hw)
(Fn-(o))8a. (1) F Frr
(100)
by (Awr [Hxx)
2= ==11
क A Hh)
b
(A, H1)
By breaking the bracket and re-arranging terms via propensity scores, this further gives us
Th Th (Ax (I
K Σ
EIF, C(n") E
Ra
< m) -7)
K I bw (Aw (Hw)
k=1
(101)
Ter (Akl (AN|HE)) bur (Aw HR.)
(1-
(e (r. Ha)
F FR-(x) (
The expression above aligns perfectly with the quantile DR estimator we proposed in Equation after taking the derivative with a constant-wise difference. As the optimization result remains the same after multiplying a constant, the EIF we derived above demonstrates the semi-parametric efficiency of our DR estimator.
50


Reviews:
[{"review_id": "hlAwilTyKs", "reviewer": "Reviewer_3nDJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I think the paper is very well motivated and the extension to sequential decision making problems is a good one. It is easy to see such a method could be usefully applied. \n- The theoretical and experimental results are extensive.\n- The paper is in general clear, easy to follow, and well written.\n\nWeaknesses:\n -  Currently the related work section is very small (only a paragraph) which makes it very hard to situate the work within the broader literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a double robust estimator for the quantile loss function, which they apply to get the quantiles of the reward of a policy in decision-making settings. They extend this to the sequential decision making setting, where the goal is to get the quantiles of the reward of a policy for multiple rounds of decisions simultaneously. Here again they propose a doubly robust estimator of the quantile loss function, which now depends on the estimation of the outcome and propensity for each round. They provide extensive theoretical analysis with normal doubly robust style results and experimentally demonstrate improvement over baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the authors include a more detailed related work section, focused around causal estimation on distributional qualities. As an example of some starting points please consider the following three papers and references therein:\n\nInference on Counterfactual Distributions. Victor Chernozhukov, Ivan Fernandez-Val, Blaise Melly\n\nCounterfactual Mean Embeddings. Krikamol Muandet, Motonobu Kanagawa, Sorawit Saengkyongam, Sanparith Marukatat\n\nQuantifying Aleatoric Uncertainty of the Treatment Effect: A Novel Orthogonal Learner. Valentyn Melnychuk, Stefan Feuerriegel, Mihaela van der Schaar.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTEAwNFyQT", "reviewer": "Reviewer_QX8u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength \n- The novelty of the paper is clearly highlighted.\n\n- The paper is comprehensive—covering a clear problem formulation, a well-defined solution framework, theoretical guarantees, and both synthetic and real-world experiments.\n\n- The claims are well supported by a range of empirical and theoretical evidence.\n\n\nWeakness\n- The paper could benefit from including more motivating examples or practical scenarios where the quantile treatment effect is of genuine interest.\n\n- In the real-world experiments, the paper defines the quantity of interest as the mean of the 0.999 quantile. A natural question arises: why use this definition instead of directly estimating quantiles?\n\n- The extension from one stage to multiple stages does not appear to introduce significant technical challenges—or perhaps this aspect is under-explained. Clarifying the difficulty and novelty of this extension would strengthen the contribution.\n\n- Although Assumption 1 is justified in the paper, it seems relatively strong. For instance, if Gradient Boosting Decision Trees (GBDT) are used—as claimed in the experiments—does the assumption still hold? Clarification would be helpful.\n\n- The assumption of i.i.d. triplets in the multi-stage setting also feels strong and may limit practical applicability. For example, when it comes to sequential decision making, there could be correlations between subsequent steps, e.g. Markovian. \n\n- The algorithm follows a plug-in approach, and the core methodological novelty remains somewhat unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies quantile treatment effect estimation using doubly robust estimators, potentially in multi-stage settings. It first defines the quantity of interest as the quantile of the potential outcome under a given policy, based on observational data. The paper then proposes an estimation procedure for this quantity and compares inverse probability weighting (IPW) and doubly robust (DR) methods, initially under the assumption of a single-stage setting. It subsequently extends the approach to the multi-stage case and establishes a corresponding estimation framework. Theoretical guarantees are provided by proving both asymptotic convergence to the standard distribution and bounds on sample complexity. Finally, the paper validates the proposed methods through simulation and real-world experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Paper Organization \n- The paper could benefit from adding more motivating examples or scenarios where the quantile treatment effect is of genuine interest.\n\n- The single-stage and multi-stage settings appear to have significant overlap. A more concise and unified presentation of these two parts would improve clarity and readability.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c8renp0VXY", "reviewer": "Reviewer_ULxt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is theoretically sound, and clearly written providing an interesting new method for practical applications.\n\nHowever, the experiments are very insufficiently developed and presented. Although the multi-stage seems to be an entirely new contribution, other methods have been proposed for the single stage quantile estimation, and the authors should review prior works on the topic, and clarify exactly their contribution compared to this prior work. In particular, the work of Firpo, Sergio. \"Efficient semiparametric estimation of quantile treatment effects.\" Econometrica 75.1 (2007): 259-276, seems very relevant, and to have similarities in the proposed solution and its properties. The presentation of the results on simulated and real data should include this baseline, as well as other baselines (some results are shown in supplementary methods, but not all). There are also a number of presentation issues (see the requested changes section).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present the development of a doubly robust estimator for quantile treatment effect, for the case of a single intervention, and for the multi-stage sequential setting. They provide strong theoretical proofs, showing the robustness properties, the convergence, and the semi-parametric efficiency of the proposed estimators. Further, they evaluate the proposed method on simulated and real data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Regarding the conduct of the experiments\n- Regarding the comparison of the Rquantile and Rmean, maybe an additional baseline to consider would be Rmean, but with simple transformation of the outcome, like log? Also, the reported metric is the MSE, is it the MSE of the outcome, or the MSE of the estimated effect? In those tables 1 and 2, the standard deviation could also be reported, in addition to the mean, to give some insight about the significance of the difference between the two methods.\n- For the results presented in Figure 3, it would be interesting to represent the same result for other levels of heavy-tailed, only the worst case of the Rmean approach is represented, which does not fully account for all the results in tables 1 and 2. Additionnally, the results are presented only for N=2500, and the number of sample for Figure 3 is not specified.\n- The histogram of figure 4 is very badly presented, why this range of the x-axis, while it does not allow to visualize well the data distribution? there is no axes labels on this plot, and the legend is incomplete\n- figure 7, the x-axis label is R1, while the text says it represents R*.\n- it would be very interesting to provide more simulations with cases where the DM approach works and not the IPW\n- the DM name is surprising, does it correspond to a form of g-computation or plug-in g-formula?\n- the plots in Figure 1 don't have axes labels.\n- regarding the GDBT approach for estimation, what is the used loss? the authors mention that it should be proper, is it?\n\nOverall, to reach a good level the authors should provide more experiments, more baselines, a better figure presentation, and caption so that they are easy to understand, and clarify the state of the art and previous methods. It is also very important for the impact and potential to reuse the method that the code for the estimators is shared, and the code to reproduce all the experiments.", "broader_impact_concerns": "I don't have concerns about the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)