Conditional Outcome Equivalence: A Quantile Alternative to CATE
Authors: Josh Givens
Henry Reeve
Song Liu
Katarzyna Reluga
Venue: NeurIPS 2024 poster (2024)
Keywords: Heteregenous Treatment Effect
Conditional Average Treatment Effect
Conditional Quantile Treatment Effect
Quantile Regression
Primary area: causal_inference
TL;DR: We introduce a new estimand, the conditional quantile comparator to compete with both the CATE and the CQTE.

Abstract: The conditional quantile treatment effect (CQTE) can provide insight into the effect of a treatment beyond the conditional average treatment effect (CATE). This ability to provide information over multiple quantiles of the response makes the CQTE especially valuable in cases where the effect of a treatment is not well-modelled by a location shift, even conditionally on the covariates. Nevertheless, the estimation of the CQTE is challenging and often depends upon the smoothness of the individual quantiles as a function of the covariates rather than smoothness of the CQTE itself. This is in stark contrast to the CATE where it is possible to obtain high-quality estimates  which have less dependency upon the smoothness of the nuisance parameters when the CATE itself is smooth. Moreover, relative smoothness of the CQTE lacks the interpretability of smoothness of the CATE making it less clear whether it is a reasonable assumption to make. We combine the desirable properties of the CATE and CQTE by considering a new estimand, the conditional quantile comparator (CQC). The CQC not only retains information about the whole treatment distribution, similar to the CQTE, but also having more natural examples of smoothness and is able to leverage simplicity in an auxiliary estimand. We provide finite sample bounds on the error of our estimator, demonstrating its ability to exploit simplicity. We validate our theory in numerical simulations which show that our method produces more accurate estimates than baselines. Finally, we apply our methodology to a study on the effect of employment incentives on earnings across different age groups. We see that our method is able to reveal heterogeneity of the effect across different quantiles.

Conditional Outcome Equivalence: A Quantile Alternative to CATE
Josh Givens University of Bristol
Henry W Reeve University of Bristol
josh. josh.givens@bra givens@bristol.ccoomm ac. uk
henry. reary.reeveebristoll.co
Song Liu University of Bristol
Katarzyna Reluga
University of Bristol
song. Long.Liu@rristol.co
retarayya ac.uk
Abstract
The conditional quantile treatment effect (CQTE) can provide insight into the ef- feet of a treatment beyond the conditional average treatment effect (CATE). This ability provide information CQTE especially valuable in cases where the effect of treatment is not well- modelled by a location shift, even conditionally on the covariates. Nevertheless, the estimation of the CQTE is challenging and often depends upon the smoothness of the individual quantiles of the CQTE itself. This is in stark contrast to the CATE where it is possible to obtain high-quality estimates which have less dependency upon the smoothness of the nuisance parameters when the CATE itself is smooth. Moreover, relative smoothness of the CQTE lacks the interpretability of smoothness of the CATE making it less clear whether it is a reasonable assumption to make. We combine the desirable properties of the CATE and CQTE by considering new estimand, the conditional quantile comparator (CQC). The CQC not only retains informa- tion about the whole treatment distribution, similar to the CQTE, but also having more natural examples of smoothness and is able to leverage simplicity in aux- iliary estimand. We provide finite sample bounds on the error our estimator, demonstrating its ability to exploit simplicity. We validate ical simulations which show that our method produces more accurate estimates than baselines. Finally, employment incentives on earnings across different age groups. We see that our method is able to reveal heterogeneity of the effect across different quantiles.
multiple quantiles of the response makes the
function of the covariates rather than smoothness
theory in numer-
apply our methodology to study the effect of
Introduction
In many real world scenarios such as personalised treatment allocation and individual level policy decisions, understanding the effect of treatment/intervention
an individual level is invaluable in providing bespoke care. The field which aims to understand a treatment's effect given certain covariates is referred to heterogeneous treatment effect (HTE) estimation and has seen popularity across many applications [II] 24, 20]. Within HTE, the conditional average treatment effect (CATE) has proved itself to be a popular target of study in this area due to its simplicity and inter- pretability 2 13 28 A key limitation of the CATE however, is that it fails to paint full picture of the differences between distributions the two responses. In addition, can be sensitive to outliers, with extreme values leading to biased outcome. As such, the conditional quantile treatment effect (CQTE), an estimand which compares the conditional quantiles the distributions in the treated and untreated populations, has established itself as a popular alternative []
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While the CQTE offers more information than the CATE and is more robust to outliers, it lacks some of the CATE's desirable estimation properties. Specifically, CQTE estimation involves estimating the quantile functions for the two marginal outcomes. This harms the estimation procedure in cases where estimation of marginal quantile functions is more challenging than estimation of the CQTE itself. An example of this is when the marginal quantile functions are less smooth as a function of the covariates than the CQTE. This aligns with a recurring idea within HTE estimation that the effect of treatment may be simpler than the marginal outcomes. In contrast to the CQTE, there are many CATE estimators which aim to learn the CATE directly allowing them to exploit its relative simplicity. These include the X-leamer [17, R-learner [23], and Doubly Robust (DR) learner [[5]. Before estimating the CATE, these procedures require estimation of intermediary estimands (nui- sance parameters) which condition the covariates such the average marginal outcomes and the propensity score (the probability of being assigned to treatment group). These nuisance parameters then used aid the estimation of the CATE. With the DR learner specifically, it has been shown that it can still achieve optimal convergence rates even when estimation of the nuisance parameters is worse than estimation of the CATE itself. This notion is referred to as double robustness, as estimation is robust to sub-optimal estimation of both of the nuisance parameters.
with a key work being that of Kallus and Oprescu [14]. In this they provide an extension of the double robustness property to the CQTE, creating an estimation procedure that can achieve strong convergence even when nuisance parameters are more difficult to estimate. Unfortunately, one of the nuisance parameters which must be estimated is the reciprocal of the conditional densities over the response. These are highly difficult to estimate and risk the errors blowing up in low density regions which could potentially nullify the desirable estimation rates they achieve even with the dependence on the estimation accuracy of these nuisance parameters being less strong. Furthermore it is still unclear how one can interpret relative smoothness in the CQTE compared to the individual quantiles with their being relatively little discussion of this within the literature. In general there is distinct lack of illustrative examples; which are present for the CATE. To our knowledge no other works specifically aim to tackle this double robustness phenomenon for the CQTE other quantile based treatment effect estimands.
Some attempts have been made to improve CQTE estimation B4 33
We introduce a novel estimand called the "conditional quantile comparator" (CQC). The CQC gives the outcome for treated individual in the same quantile given outcome for an untreated indi- vidual, conditional on covariates. This relates to the conditional Quantile- Quanttle (QQ) plot for the treated and untreated outcomes as demonstrated in Figure [ Similarly to the CQTE, our new esti-
121
x=0
[vix 0)
1.0
X-0.5
Treated Response
x=1
= 0.5)
0.4
0.8
0.2
0.6
0.0
0.8
0.6
0.1
0.0
0.2 0.6
YJK = x, 0
x
0.0
Untreated Response
Figure 1: The left plot gives the CQC surface which takes in covariates (x) and an untreated response (y) and returns the treated response of the equivalent quantile (g" (g]z)). The right plot a QQ-plot of the responses (Y) in the untreated (A 0) vs treated (A = 1) population conditional on various covariates (X- 0. 0.5, 1). These conditional QQ-plots correspond "slices" of the CQC surface, as shown by the coloured lines in the left plot. The plot is best viewed in colour.
mand, the CQC, allows us to compare equivalent quantiles while working exclusively in the response landscape, making a more interpretable tool. This allows us to construct canonical examples of the CQC being smoother than various nuisance parameters, the CQTE, and the CATE; adding to this interpretabilityy In addition, using the pseudo-outcome framework presented in Kennedy [15) we can leverage CATE estimation procedures to estimate the CQC in doubly robust way, mentioned

---

above. Crucially, the CQC can keep the valuable quantile-level information previously offered by the CQTE while building on much of the CATE literature acquire its desirable robustness properties and interpretability. Our contributions are as follows:
Introduce new estimand for HTE analysis: the conditional quantile comparator (CQC).
Propose an estimation procedure which we prove to be doubly robust.
Demonstrate better estimation accuracy especially when the CQC is smooth but individual conditional cumulative distribution functions are not.
Provide insights into real-world datasets on employment intervention and medical treat- ment.
2 Set-up
We now introduce the standard HTE set-up in our notation. Let Z denote the random triple XY, A) with Y random variable (RV) on ) S R, X' a RV on X' S Rd and A a RV on {0, 1}. We treat Z as representing an individual and interpret the components as
y: Outcome/Response
X Observed covariates
A Treatment assignment
Remark 1. We could view our setting as coming from a potential outcome framework (27]. Under this framework we assume there exists RVs Yo, Y1 on y representing the outcome with and without treatment and that Y Ya. YI-A would then be umobserved/unknow for each individual.
We define the propensity score x : X (0, by
n (x) = P(A=11X- x))
in other words, π denotes the conditional probability of being assigned to treatment given the co- variates. We shall assume that æ is continuous and bounded away from and 1. From potential outcomes perspective, this means that each individual could potentially be assigned to either treat- ment. We also define
Fa(ul=)== P(Y S v/X = x, 2. A=
F. = inf E (u 2a),
(2)
and refer to them as the conditional cumulative distribution function (CCDF) and the quantile func- tion respectively. We also refer F--1 in
the generalised inverse of Fa-
We can now define the CATE and CQTE to be given by T X R and Tq [0. 1] X - with
T(x) == Y X = x, A = 1 E|Y x, A 0].
Ty (a/x)) (x) = FT- (a [=)==) Fo (a|x))
We let = ( = {(Y, X,, A,) for n N be IID copies of representing our data sample with indexing the individual. We assume an even number of samples for notational conve- nience. For a E {0, 1). we take La- = (i]A, = 1}. the indices individuals on treatment a. We can then define D. == (ZE))) } and n,, = |4a the dataset and sample size of those treatment a.
For E N, let [n] = (1...., n.. For a vector u E Rp let wy to represent the jth component of w and let ||]]] be the Fuclidean norm unless otherwise specified. We also take ||w]| the 1-norm and | i= maxje(>) w,yl- We keep summary table of all notation used in Appendix A.1
2.1 Introducing the quantile comparator
Our aim to find "equivalent quantiles" " between the treated and non-treated distributions condi- tional on the covariates. Specifically, for each yo E V, æ E X' aim to find y such that
F (In (x) Fo (zol2z)
This now allows to define our primary estimand of interest, the conditional quantile comparator.
Definition 1 (Conditional quantile comparator (CQC)). our triple (Y,X, A), the conditional quantile comparator is the measurable function g" y x X - y such that, for all y E y, X,
F9 (y|z) xx) Fo(y)z))
3

---

We then simply define y1 g" (yolz). The name conditional quantile comparator derives from the fact that it returns the value y/ in the equivalent quantile of Y|X x, as the quantile of Y|X = A = O that yo is in.
Remark 2. For simplicity and to ensure such a function is well defined, we will assume that Y|X 2, A = a is continuous RV for any given X, E {0, with strictly positive density its support. We will however allow the support Y = Σ, A = a to vary in both a and a.
We now introduce another estimand which will serve as a useful stepping stone our estimation.
Definition 2 (CCDF contrasting function). The CCDF contrasting function is defined to be h" ; x x + [-1,1] given by
ht (Vo: v1 ax) = (v [z) Fo (z))
This estimand allows following alternative definitions for g" which help its interpretation and esti- mation (detailed later). We take ht - representing the inverse of h- with respect to the 2nd argument.
g" x)) h- hh~1 0|z) F (Fo(so/zz)(zz).
(4)
The second equality still holds if replace the inverses in the above with generalised inverses. The equality (4) falls straight from the definition of each object and shows how we can use h to estimate g* Moreover, the equality (4) allows to generalise g* discontinuous (or pdfs with non-trivial 0 density regions inside the support).
2.2 Exploring the CQC
We specifically focus on the CQC, g". as we feel it gives insightful information allowing comparison of the two distributions (Y),,, A = 1) and (Y A 0).
The CQC allows us to compare the two distributions beyond simply single point estimate such as that given by the CATE. This is especially valuable in cases where the two distributions differ beyond just a shift. For example, the effect some treatments varies greatly between individuals with the same similar covariates. An example of this is antidepressants, where some patients respond positively while others may have adverse reactions leading to a worse outcome than no treatment whatsoever. Another example is the use of opioids painkillers where some patients have an increased tolerance making them less effective [12 22).
As well as being of interest on its own, the quantile comparator relates closely other estimands of interest. For example, if we take A " x> = g (go|zz yo then A tells us whether the equivalent quantile in the treated distribution is higher lower. This estimand then serves as a heuristic for whether the treatment is beneficial at that untreated response value.
Furthermore, CQTE can be written as
(a(a)> = A" (a) ()) g" (z))) - (o)
(5)
linking the quantile comparator back to the CQTE. This equivalence highlights the perspective that the CQC can be seen as rephrasing the input of the CQTE in terms of the outcome space.
A key idea within CATE literature is the notion that the CATE itself may be a simpler estimand to study than the marginal treatment outcomes Y A = a)) may be individually. One can exploit this feature improve the CATE's estimation. A similar concept exists with the CQC as we will see in the example below.
Example 1 (Illustrative Example). Suppose that
=|== =
N(sin(10x), 12),
- = xx A= N(2sin(10r), 29).
Then we have g" g x)]xx) 2y which does not depend on and does not include the sine term present the individual CDFs. Interestingly E| A =
E| A 0) = sin(x) hence the CATE is still non-constant in this case (the same also holds for the CQTE). Additionally, we have A (y]x:) g" (y|xr - y = suggesting the intervention is beneficial for positive y and detrimental for negative y. We now show 3D plots of CCDF, the CQC, and the CQTE in Figure 2

---

04
X
(a) Fy (x)e)
(b) 9 (y) (z)
(c) Tq (a/x)
Figure 2: Surface plots for CCDF (panel (a)), CQC (panel (b)) and CQTE (panel (c)). We can see that CCDF, and CQTE have high-frequency change in æ while the CQC does depend on
Example 2 (General Smoothness Case). Suppose YA. Now also suppose that Y1 = =YYo, for some transformation
in the potential outcomes framework so
that Yo, Y1 exist with increasing in Yo for each X. Then gives the CQC (i.e. = g²) meaning that smoothness of the CQC can be seen as smoothness of 6. This gives a generalisation of the CATE case where smoothness is present when Yi Yo (X) with smooth
A specific example could be a treatment which halves all individuals blood pressure. In this case 6" (y.z) = 9 ()) l/ and so the CQC smooth but the CQTE and CATE would not be if the individual responses are non-smooth.
Estimation procedure
We now describe our estimation procedure for the CQC which motivated by equation (4. At
high level our approach for estimating g" (yolæ) will be the following:
Estimate (yo- (x) using pseudo-outcome - specified proxy response computed from XY, which we will regress against.
Find the value ú1 which makes our estimate of h -aoo-la:)) closest to O.
Section 3.1 and Algorithm I give our h" estimation procedure while Section 3.3 and Algorithm
give our estimation procedure.
3.1 Estimating the CCDF contrasting function h"
We focus estimating h* primarily for its two nice properties:
1. Similar to g", ht can exhibit smoothness even when the individual CCDFs are not smooth.
2. The estimation ht* can be re-framed CATE problem.
The first property is important the smoothness of ht determines the best estimation rate that can be achieved when using non-parametric regression, setting a target for our approach. In particular, smoother functions have better estimation rates. The second property is important as it gives us a method for attaining this target rate. By re-framing the estimation as a CATE problem. we can leverage existing results to build robust estimator which achieves the target estimation accuracy rate even when the rate estimating nuisance parameters sub-optimal. We demonstrate this robustness later using finite sample bounds the estimation accuracy (Proposition & Theorem 2}.
First, we show how the estimation of h² can be solved using CATE estimator. Note that
h" (yo. y1 (x) [Y < villX x, A 1] EY yollx x, A=
Hence, for given No. V/, we define the WT Woon := 1(Y < ya then estimating hs" (jo. V1 is equivalent to estimating the CATE with W yo..,, replacing Y as the response. To perform this estimation, we turn a resent method developed by Kennedy [15]. They propose to write the CATE conditional expectation of function of Z called pseudo-outcome. A robust estimator

---

is then obtained by regressing this aseudo-outcone against X. In our setting. the ostudo- with the new response Wi yo. for a sample (y/ x. a) is given by
(②)
Pyo.a1 (y,
{1 ya Fa Fa(yala))) + Fi (1112)) (x0
(6)
7 (z) - (2))
((x)
{1 ya Fa(yalar) h² (yo, yn 11)).
(x) x x)))
Since ht" (yo, y1 |æ) = am (2) X = x² (Proposition 5 Appendix
C.I
regressing (Pue.nn (Z) on
X provides an estimate for h²"
As we do not know the CDFs nor the propensity score, we need to replace them in (6) with estimates. We define मt, Fo, Fi to be estimates of т, Fo, Pi respectively. We then construct P in the same way as yo-ar but using estimated quantities म. F. instead. Pyo.y can now serve as the pseudo- outcome in our regression. We also use sample splitting to de-correlate the propensity score and CDF estimates from the h* estimate. This helps make estimator doubly robust, the following theory.
will see in
We are now ready to define our Doubly Robust (DR)-learner estimate ht in Algorithm D
Algorithm 1 DR estimation procedure for the CCDF contrasting function h"
Require: No. /1 E V Data D, regressor (e.g. linear smoother)
1: Define I = mn] J = {m + 1, 1..... .2.] and split D into Dr {Zi)izz Dy = (Zibier.
2: Using Dz to estimate मं, Fo(yol-), Pi (yn by regressing 1|AA= = 1), S sm), 11 s y respectively against X.
3: Use these estimates to obtain (Zz)) for j E Dy using equation
4: Using Dy regress against X to obtain estimate h(yo, 31l-) of h" (yo, V1l-).
Remark 3. Cross-fitting can be implemented by repeating the procedure with the roles of and J switched and then averaging the two estimates of".. We could also perform this procedure multiple times with different random splits of the data to improve our estimator's 's potential stability.
Note that our algorithm is not specific on which form of regression to use allowing for any parametric or non-parametric procedure. Further to this, it can also be easily adapted use other pseudo- outcome procedures such as the R-learner of Nie and Wager 23 or a standard inverse propensity weighting approach which we describe in Appendix A.3
3.2 Finite sample bound of h* estimator
In this section, we prove the estimation accuracy of h, which will play important roles in the follow- ing notation and assumptions
need for estimating g" These accuracy statements will be made for an arbitrarily fixed æ E X. For our theoretical and experimental results we use linear smoothers for the final regression. This means our estimate and oracle estimate h'' are of the form
Myo 3n 31(0)
Σ JEJ
1o3& vo.ii (Zz)
h (yoo sn 31(=))
Σ iET
WylPuo.n (Z₁)
where the weights wy = wy(x Xy) are constructed using Xg = {Xyleez with wy 2 0 and ||w]| Linear smoothers encompass a broad class of estimation techniques used in both low and high-dimensional settings. Examples include k-NN regression [8] []] 91 kernel ridge regression [29],
generalised forests (3) and Mondrian forests [8] Additionally linear smoothers have been shown
to adapt to intrinsic low dimensionality in regression problems in higher dimensions 16], making them an apt estimator for our purposes.
For Φ : + R treated as deterministic and p > 1, we also define the norms
| lollw>>>
Σ
Ew) X,]
V ||=0|11
.E.
and take [| 11 Note that this norm is random the weights depend upon X

---

We now aim to show that we are able to exploit smoothness in ht even when the CCDFs and propensity score are less smooth. We introduce the notion of smoothness through Hölder functions.
Definition 3 (Hölder functions). We say that a function f X + is (7, CY-Hölder for E (0, 10,1,,C2 if for any x', x" EX,,
f(xx) - f(x"((())))))))))) S C x x"11
Here, larger represents a smoother function which can be estimated at faster rates.
Assumption 1. For any you 91 E 3 5 0, {0,
(a) There exists & (0,1/2) such that π (()), #(x') E [s, for all x' X.
(b) With probability at least 6, 12 Poo.n I1w-5 (p(n, d).
(c) With probability at least - 5, म - x Tll (a 8) and Fa - Fa lw ea(n, 6).
(d) For E (0, 1]. C 1, h² (yo, vil is (r. -)Höldr
Assumption Taa exists ensure for any covariate value, neither treatment assignment has too low a probability. Assumption [bb controls the convergence of the estimated pseudo-outcome to the true pseudo-outcome. Assumption [[c) sets up the smoothness of the propensity score and CCDFs alongside the convergence rates of their estimators as Ea: Eg respectively. Assumption [d) sets up the smoothness of h" which will control the convergence rate of the oracle estimation procedure. Assumptions I (c) and (d) control the accuracy of our estimator in the following result.
Proposition 1. Suppose that Assumption holds and let h be a linear smoother estimated as in Algorithm 7
Then for any 91, yo YY, 8 (0, 2/e] and our x X, with probability at least 1 1- 8,
|Cy0 21 v1(x)) h² (vo, (vo.vilz)) 01|z)| (n,
Here, for each E (0, 2/e] we have
en 8) == /2lo(380)//nep( 8/4) + E. (n, 8/4)e 8/4) + en(n, 8/4), (n, = [E[h' (yo' y1 k1 - h (vo. yn + 22oog(2/8)/n n | 10|||
Dz]1
hyyo, 81 ||=ll + 20/g////
We have that Ey gives an upper bound on the accuracy the oracle estimation and so acts as a target for our estimation procedure. If (n, - 0 then the first term in E₁, is guaranteed to be (n, 8)) for fixed 6. Hence the first and last terms converge at oracle rates with respect to n. As the Ea and Eg terms multiplied together is because both 6.x and 6g can converge to 0 slower than Ey while their product & Eg converges quicker. This provides the desired double robustness even when convergence for the nuisance parameters slower. Now that we have this we can convert our estimate of ht', into an estimate of g"
obtain better rates than either of them individually have. This
estimation can converge oracle rates
3.3 Estimating the conditional quantile comparator g"
In order to obtain estimate of g" (yolæ) a fixed yo, we need to obtain estimates of h (Mo, v1(=)) at various values of y1- As hi* is monotonic, we would then like to search for a monotonic function which aligns with these estimates. This monotonicity is especially important because it allows to bound the estimation accuracy of h uniformly over all y1 at similar rate pointwise accuracy. With this, we can easily translate the estimation accuracy in h (ob- tained in proposition I into the accuracy in ÿ. Additionally, it simplifies the process of inverting h. In general our method in Algorithm will not produce monotonic h (this is in contrast to some other approaches such as an IPW ostudo-outcome, see Appendix (A.3). or separately estimating the CCDFs). We can however obtain monotonic estimate of h" using isotonic projection.
Definition 4 (Isotonic Projection). We define the isotonic projection of RRP as follows:
P(a') == argmin|/a -a1 a''l a<Tsa(pp)
where Iso(p) {ax E R2/ay 0441 Vl e 1p- 1]}. the set of all isotonic vectors in RR.
7

---

Remark 4. We can use the Pool Adjacent Violators Algorithm (PAVA) [6] which performs isotonic projection and is implemented the SeotonicRegressee class of sci-kit learn Python [25].
Hence, for a fixed value of (yo, x) and a set predictions ãj h(yo: yi ()) with y0 S y can take ā Piso(p)) (á) and use these to obtain new monotonic estimate of h'. Furthermore, by a result in Yang and Barber [32], & will be at least as accurate as & in the worst case. We now describe our approach for estimating the CQC using this projection approach in Algorithm
Algorithm 2 DR estimation procedure for the CQC g"
Require: Data D; test point (yo, x); sorted evaluation points
He
1: Apply Algorithm [ to obtain estimate ht* given by h.
2: Define â = h(oo: ( (z) for / E [p]-
3: Isotonically project & using PAVA to obtain à with ā; S ā.+1-
4: Take ((yol 1) yld with 1* == ergmingglp](ay!
Remark 5. For the case where h is a step function, these steps can serve as our evaluation points while for continuous h one could take these candidate y1 points at small evenly spaced intervals. Empirically we also find that i is already close to isotonic and so step 3. of the algorithm is mostly for the theoretical justification of our approach.
While it may seem inefficient to be estimating the CQC via the CCDF contrasting function and then inverting, due to the monotonicity of h", we actually pay very small cost in estimation accuracy for having to estimate the CCDF contrasting function over all y1. We make this notion more explicit in the following section.
3.4 Finite sample bound of the CQC estimator
We now provide the accuracy of our estimate I obtained by Algorithm 2 when used in conjunction with linear smoothers. We assume that Fa are also fit using linear smoothers of the form Falulz', a)
wr, Xz, IEI
with wy way'') = waya'' Xz, Ar) > O, 1 (x Ill 1. We will also require the following assumptions.
Assumption 2. For RV X, any y V, x' 6
(a) There exists some 8, 1) > 0 such that (1/ 2 for all y E (y)g)))
(b) W.p. at least - 8, maxje.7 wj S Sum(n,, 8) and maxiez WF S (n,
Assumption 2 a) is a mild assumption which allows us to convert the h accuracy into 9 accuracy while Assumption bb) bounds the rates of decay of the weights in our linear smoothers.
Theorem 2. Let 9 be estimated as using Algorithm with (Y; YYie,,, sorted and then used as our for
evaluation points and linear smoothers used for regressions in Algorithm [ Then provided Assump- tions] & 2 hold we have that for E (0.e and sufficiently large n, w.p. at least 1 - ò,
[g(y g" (y]=)) (17 (m, 0/ (2n) +
ew e 8 6/(n))
From this result we see that if our weights decay at rate faster than En (n, 8) then this error will be dominated by the En term. We believe this hold in most cases and show that it does comfortably when using Nadaraya-Watson Watson (NW) estimation 22 31 with box kernel in Appendix C..4 Further- more, if the dependence on d in both terms is of the form (1/8) for some e > 0 then obtain the same rate of estimation as for h² up to polylog factors. This means we translate our desirable double
robustness h over to ÿ. We also obtain finite sample bounds on (l]((((=)) g" (Y [|])) A = 0,
with high probability and present this in Appendix C.3
Numerical experiments
We now apply our approach to series of simulated and real data scenarios in order to demonstrate the utility of our estimand and the effectiveness of our estimation procedure. For these, we use NW estimation as our regression procedure throughout. See appendix A.2 for details on NW estimation.
8

---

4.1 Simulated experiment
In this section, we test our method's performance in terms our estimator's mean absolute error under simulated scenarios | In each scenario we test against separate estimator which estimates the two CCDFs separately and simply takes their difference, an IPW pseudo-outcome estimator detailed in Appendix A.3 the CQTE estimator of Kallus and Oprescu [14], and the oracle DR estimator where & is replaced with ş (i.e. exact æ, Fa are used). In this experiment, we return back to the set-up of example [ We now change the frequency of the sine term by taking
Y|X = 2; A = 0 ~ N(sin(yyx:; 12),
Y|X = I, A = N(2sin(77z), 2²),
T (x) (.4 + 0.5.
for y E 10] so that increasing y imitates decreasing smoothness of our nuisance parameters. In our experiments half the samples are used to estimate the propensity score and CCDFs and the other half are used to regress against the pseudo-outcome, Our estimate ÿ is then compared against g" using hold-out testing set. This process is repeated 500 times with new training data each run. From this, a Monte-Carlo estimate of Eg [x alongside 95% confidence intervals (CIs). In our first experiment, let 2n. 1000 and vary 7 in [0, 10). In our second experiment, we let ry = 6 and 2n vary in [200, 5000]. The results of this are shown in Figure
- g (IIX) is produced
1.4
Saparate Estimator
Scparate Estimator
Estimator
Estimatoo
1.2
COTE Estimater (Kalls at
COTE Estimrar (Kallus et
DR Entimator (Ours)
DR Estiaator (Ours)
1.0
Exact DR Estimator (Oraci)
2.0
Exact Estimator (Cracle)
0.0
0.6
0.4
6.5-
0.2
0.0
2000
4000
6000
2n
Figure 3: Mean absolute error with 95% CIs for various estimators. The left plot has fixed sample
size (2n- = 1000) and increasing 7. The right plot has 7 = and increasing sample size.
We can see that the average error decreases as the sample size 2n grows and mildly increases as y grows. This is expected as increasing makes estimating the nuisance parameters more challenging. The result shows that the proposed DR method achieves the best performance compared with the Separate and IPW estimators and is only marginally outperformed by the oracle estimator. Addi- tionally we see that the method of Kallus and Oprescu [14] is much more affected by the increase in y. This is because unlike the CQC, the CQTE has a complexity that depends on the frequency term (7). We also observe much better performance as the sample size increases. We hypothe- sise that the plateau in the CQTE approach due the difficulty of estimating the reciprocal of the PDE, which causes the estimation to be unstable irrespective of sample size. Further simulated experiments including 10-dimensional ær and linear CQC are given in Appendix B.
4.2 Real world employment example
To show the performance in the real-world scenarios, we use dataset on employment programme which has been studied in various prior works 44 [ 26). Within the programme, some participants were given job placements or temporary help jobs while others received no intervention. Partici- pants' earnings were then monitored over the next 8 quarters following their enrolments. We take their net carnings our response (Y) and the employment intervention the treatment (A 1). We use each participant's 's age at their entry to the study as our covariate (X). We our quantile comparator function on 2,000 participants. Figure 4 shows our estimate of A (y(a)) (y](ar) for various values of 2)).
We see that participants around age 23 and between ages 32-37 benefit most from this scheme, as indicated by the darker colour on the heat map. Additionally, the lower quantile of the income distri- bution (wage < $7500) shows the least change, indicating that wage improvements primarily occur
Code implementation be found githlconnnieeeraaaooootooonoeeee

---

15000
12500
10000
7500
15000
Eariigs (V)
5000
5000
12500
(y(x)
2500
10000
2500
5500
Earnings (y)
15000
10000
-0000
5000
Age (x)
30
35
40
22.5
25.0
30.0
Age
32.5
35,0
37.5
Figure 4: Surface and heat plot of A (yylaz) for our employment data with X =Age, IIcoome.
for the higher income group. For participants aged 40, there appears to be little change in outcomes overall. To demonstrate our approach in a medical setting, we apply our method to evaluate the effectiveness of colon cancer treatment, as detailed Appendix B.3 For comparative purposes, we also provide an estimate of the CQTE for this data in Appendix B.2
5
Limitations
The theory provided here gives a strong foundation and motivation for framing our problem this particular manner. There is however great deal more be explored in this area from theoretical perspective. For example, one immediate improvement would be to give a more general case where weight decay (in Assumption 2 (b) for Theorem 2 is sufficiently fast. In addition
work needs to be done exploring the relationship between the smoothness of g and the smoothness ft". Smoothness in h" appears to be a stronger condition ideally we would like make theoretical statements directly on the smoothness of g". Interestingly our experiments on synthetic data do seem to suggest that the complexity of g which drives the estimation rate as in these experiments h" increases in complexity while g* remains constant. Additionally, changing our experiment in Section 4.1 have uniform response so that both h* gg are constant rather than just g", seems to give no material improvement to the performance of our estimator (see Appendix B. 8.1.3
Another limitation of current estimation procedure is that it requires learning estimand and then inverting to obtain our final estimator. While this process is relatively simple, it could be streamlined and made similar to the DR-Learner for CATE [5] or the CQTE estimator in Kallus and Oprescu 14].
computationally efficient if could produce more direct estimator
Finally, while we have been able to provide more concrete examples of smoothness for the CQC these are still limited to case a deterministic treatment effect which we would like to ex- pand upon. This is closely related to more general limitation with quantile-based estimands, the CQC included, in that they lack meaningful interpretability for the individual. While the CATE can be viewed the expected difference in individual's outcome such individual-level interpretation exists for the CQC or indeed any other estimand trying to learn higher level distributional information than the mean. As result the CATE is still a more naturally interpretable estimand. facilitate this interpretation however, one still needs to make assumptions about lack of confounding between the treatment assignment and the potential outcomes, which are only verifiable in certain restrictive scenarios [30].
and off the treatment,
6 Conclusion
In this paper we have introduced a new treatment effect estimand, the conditional quantile com- parator and demonstrated its efficacy both in terms of its doubly robust estimation, and its ability to provide valuable data insights. This promising direction as allows quantile-based treatment effect exploration to "keep up" with the CATE in terms of estimation quality offering more flexibil- ity as which estimand can be used to best describe the data. For these reasons, we see the CQC as an exciting and worthwhile new direction within the HTE framework.
10

---

Acknowledgments and Disclosure of Funding
Josh Givens was supported by a PhD studentship from the EPSRC Centre for Doctoral Training in Computational Statistics and Data Science (COMPASS).
References
[1] Abadie, A., Angrist, J., and Imbens, G. (2002). Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica, 70(1):91-117.
[2] Abadie, A. and Imbens, G. W. (2002). Simple and bias-corrected matching estimators for av- erage treatment effects. Working Paper 283, National Bureau of Economic Research. Series: Technical working paper series.
[3] Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests. The Annals of Statistics, 47(2): 148 1178. Publisher: Institute of Mathematical Statistics.
[4] Autor, D. H. and Houseman, S. N. (2010). Do temporary-hell jobs improve labor market out- comes for low-skilled workers? Evidence from "Work First". American Economic Journal: Applied Economics, 2(3):96-128.
[5] Autor, D. H., Houseman, S. N., and Kerr, S. P. (2017). The effect of work first job placements the distribution of earnings: An instrumental variable quantile regression approach. Journal of Labor Economics, 35(1):149-190.
[6] Barlow, R. E., Bartholomew, D. J., Bremner, M., and Brunk, H. D. (1972). Statistical Infer- ence under Order Restrictions (The Theory and Application of Isotonic Regression). Wiley Series Probability and Statistics. John Wiley & Sons Ltd.
[7] Boucheron, S., Lugosi, G., and Massart, P. (2013). Concentration inequalities: a nonasymptotic theory of independence. Oxford University Press.
[8] Chen, G. (2019). Nearest neighbor and kernel survival analysis: Nonasymptotic error bounds and strong consistency rates. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th international conference machine learning, volume 97 of Proceedings of machine learning research, pages 1001-1010. PMLR
[9] Chen, P., Dong, W., Lu, X., Kaymak, U., He, K., and Huang. Z. (2019). Deep representation learning for individualized treatment effect estimation using electronic health records. Journal of Biomedical Informatics, 100:103303 103303.
[10] Collins, F. S. and Varmus, H. (2015). A new initiative on precision medicine. The New England journal of medicine, 372(9):793-79) -955. Place: United States.
[11] Hirano, K. and Porter, J. R. (2009). Asymptotics for statistical treatment rules. Econometrica, 77(5): 1183-7711 Publisher: [Wiley, The Econometric Society].
[12] Huynh, P., Villaluz, J., Bhandal, H., Alem, N., and Dayal, R. (2021). Long-Term Opioid Therapy: The Burden of Adverse Effects. Pain Medicine, 22(9))2128-2130
[13] Imbens, G. W. (2004), Nonparametric estimation of average treatment effects under exogene- ity: a review. The Review of Economics and Statistics, 86(1):4-29.
[14] Kallus, N. and Oprescu, M. (2023). Robust and agnostic learning of conditional distributional treatment effects. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of the 26th international conference on artificial intelligence and statistics, volume 206 Proceedings of machine learning research, pages 6037-6060. PMLR.
[15] Kennedy, F. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2)30088 3049. Institute of Mathematical Statistics and Bernoulli Society.
[16] Kpotufe, S. (2011). k-NN regression adapts to local intrinsic dimension. In Shawe-Taylor J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances neural information processing systems, volume 24. Curran Associates, Inc.

---

[17] Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Bin Yu (2019), Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 1 11((1))4156-4166
[18]
Lakshminarayanan, B., Roy, M., and Teh, Y. W. (2014). Mondrian forests: Efficient online random forests. In Ghahramani, Z.,, Welling, M., Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in neural information processing systems, volume 27. Curran Associates, Inc.
[19] Laurie, A., Moertel, C. G., Fleming, T. R., Wieand, H. S., Leigh, J. E., Rubin, J., McCor- mack, G. W., Gerstner, J. B., Krook, J. E., and Malliard, J. (1989). Surgical adjuvant therapy of large-bowel careinoma: an evaluation of levamisole and the combination of levamisole and flu- orouracil. The North Central Cancer Treatment Group and the Mayo Clinic. Journal of clinical oncology, 7(10):444--1456 Place: United States.
[20] Lei, L.. and Candès, E. J. (2021). Conformal inference of counterfactuals and individual treat- ment effects. Journal of the Royal Statistical Society, Series B, 83(5):911-938)
[21] Nadaraya, E. (1965). On non non-parametric estimates of density functions and regression curves. Theory of Probability & Its Applications, 10(1):186-190.
[22] Nadeau, S. E., Wu, J. K., and Lawhern, R. A. (2021). Opioids and chronic pain: An analytic review of the clinical evidence. Frontiers in Pain Research, 2.
[23] Nie, X. and Wager, S. (2020). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-311
[24] Obermeyer. Z. and Emanuel, E. (2016). Predicting the Future Big Data, Machine Learning, and Clinical Medicine. The New England journal of medicine, 375(13) Place: United States.
[25] Pedregosa, F,, Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay. E. (2011). Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
[26] Powell, D. (2020). Quantile Treatment Efffetts in the Presence of Covariates. The Review of Economics and Statistics, 102(5):994- 1005.
[27] Rubin, D. B. (2005). Causal inference using potential outcomes. Journal of the American Statistical Association, 100(469):322-331.
[28] Semenova, and Chernozhukov, V. (2021). Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal, 24(2):264-289,
[29] Singh, R., Xu, L., and Gretton, A. (2023). Kernel methods for causal functions: dose, heterogeneous and incremental response curves. Biometrika, 111(2)497-556. https://academmcc.ogg oup.
tex.eprint:
[30] VanderWeele, T. J. (2008). The sign of the bias of unmeasured confounding. Biometrics. Journal of the International Biometric Society, 64(3):702-706. Publisher: [Wiley, International Biometric Society).
[31] Watson, G. S. (1964). Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, pages 359-372.
[32] Yang, F. and Barber, R. F. (2019). Contraction and uniform convergence of isotonic regression. Electronic Journal of Statistics, 13(1):646 - 677. Publisher: Institute of Mathematical Statistics and Bernoulli Society.
[33] Ying Zhang, Lei Wang, M. Y. and Shao, J. (2020). Quantile treatment effect estimation with dimension reduction. Statistical Theory and Related Fields, 4(2):202-213, Publisher: Taylor & Francis tex.eprint: https://doo.orr/1100000777733730
1696446.
[34] Zhou, T., Carson, W. E. t., and Carlson, D. (2022). Estimating Potential Outcome Distributions with Collaborating Causal Networks. Transactions on machine learning research, 2022. Place: United States.
12

---

A
Additional details
A.1 Notation table
Table Table of notation.
Notation
Definition
Description
Y
RV on y R
Response
X
RV X C Rdd
Covariates
A
RV on {0.
Treatment
Z
AY
((x)
P(A = 1 1AA=1X-00
Propensity Score Conditional CDF (CCDF)
Fa(y(z)
P(Y S =/==
F. (a/z)
inf (y F))((/))) 2a}
Conditional quantile func- tion Conditional average treat- ment effect (CATE) Conditional quantile treat-
T(x)
E|y =. - x,A=0) 0]
Ty(a(z)
FT (alx) Fo (a/z)
ment effect (CQTE)
D
[
IID data sample Treatment index
1=
Ta
E [2n] |4, 1}
Da
{ kiea
Tae
Lal
m]
m
V E ot
Euclidean Norm
3 ا
1-norm
max Reyl
co-norm
9" (v/=)
FT (Fo(gl=)(z)
Conditional quantile com- parator CQC
h. (Jo. yn (1))
(1) - ((
CCDF contrasting function
A (y/2z
g" (v(xx) y
Quantile difference function
f
Estimate f/f*
[()||~ [क) [1 S Va (valk)))
((x))
9y (u, x0, a)
The Pseudo-Outcome
Fo Fo x)) s0 with n, F. replaced by n, F.
Fi
vu (v, ax, a) I,J
1....,n. fn +,... 2n]
Sample splitting indices
DI, DJJ
ZiJiEz, (2,he
Data split
un TR."
" = (r)(m) m))
i estimation weights
h(yo, 31 31(x))
> JEJ w M (Zz)
ht estimate
W(Vo 31 (x)
jEg wy (2(x))p (2,)
Oracle h" estimate
|| 1.EE wy 2( X = X,)
n (ar') [K,, - for all x' Ex
(n, 8)
4 S (n, w.p. 1-d
c error bound w.h.p.
e (n, 6)
T w (n w.p. 1-8
error bound w.h.p. Fa error bound w.h.p.
(n,
(vl.))) Fa(yl.))l (n 5) w.p. 1
Smoothness V1
Elh (vo, 3/1 (z) (Vo- V1 |x) Dz]l
(1,
+ / log (1/8)// | |1ye- h(yo, 3n 1-)l)
h' error bound w.h.p.
2 (
2
ST,, (1,,)
(o0 (n, 0)
T, (xx) error bound w.h.p.
ST, (n, 5/4) + (Ea. (n) 8/4)+ e, 8/4)
En (n,
h error bound w.h.p.
Iso(p)
{a E R' aj < 0441Vle E 1}}
Set of isotonic vectors Isotonic projection
Piso(p))
argmin a a'll
WF. 6 R"
(F = Wr u (x', X7)
f. estimation weights
13

---

Notation
Definition
Description
Fa(y/ax')
Eggg wr, {Y; S y Fy x) > v E (z(())
CCDF estimate
Density lower bound
(n, 6)
w.p. 1- 8 max [ /w(a)) leoe /FF (X)
Variance lower bound w.h.p.
Appendix Notation
k( x')
NW estimation kernel
m
E]f(Z)]==2
my (x)
Σjer (y(z)f(Z,)
b(x)
m (x)
(a(
max log (1/8)/11))))))) n, 1/n] 10.1,..., n
Inlo ((Ay)
limaco f'y) - f(y-e) - 8)
Step size of f at y
A.2 Nadaraya-Watae Watson estimation
Throughout, we use NW estimation our standard non-parametric regression technique. For kernel k: X' x X' - [0, x)), IID data sample D == {(Y,, X:) and æ E X' the NW estimate of B|Y|X = ar] is given by
k(x, X;) Ej k(,,, Xj)
Y.
Applying this to our pseudo-outcome regression Algorithm we get that
huyo v1 11)= = Σ Xxx
Poo.1r (Y,, X,, AA)
(7)
JETT k(I,, Xj)
iET
k(a, Xi)
A; 1 -Xx,)
(8)
Σ EJeg k( X;)
- (xx))(1=((X))
EEJ
{I( sun:) (uA,
(m - Po (2ol = Xi)
where for any x' Ex a E 10
k(a', Xi)
*(x') = Σ
1{A =1
(9)
E jEZ k(x', 10',,9 X,)
VET.
k( Xi)
Falya (2)) = Σ VET.
Ejez k(æ' X, 1 1| 1o111f4 = a}.
(10)
Note that for theoretical results do not specify how nuisance parameters are estimated and the results only depend on the accuracy of our estimation of the nuisance parameters.
Remark 6. We can re-write (8) as:
h(yo, 811x)
Σ
k(x, X.)
[1(Y,5M) (n) - A((ilx)))
k(x, Xi)
((X.)
SEZ,
+ +i(lx - Fo(eo)x.))
Σ EEI
k(x, Xi)
1 *(X,) {1(VS S yo! - Fo(volx.))
E k( Xi)
+ Fu(yol - Fe(l1
Which can be helpful in terms of the practical implementation.
14

---

A.3 IPW pseudo-outeome estimator
Another pseudo-ouccome one can use for estimating the treatment effect is the based on inverse propensity weighting. Specifically can take our pseudo-outcome to be
A' xXx) yy (Y', X', ,A, x(X'(1 x(x)) 1(Y' S yar).
(11)
we regress against it using NW estimation, and also use NW estimation for our nuisance parameter estimation as in (9) & (TO,, our estimate will be increasing in y/1 and decreasing in Yo meaning we do not need perform any isotonic projection.
A.4 Additional experimental details
As the method of Kallus and Oprescu [1 one for estimating the CQTE, we transform into an estimator of the CQC (which denote by g) using the following formula where f-(-l.) is our CQTE estimator
0(yxx) = -. (Fy1x,()(z))az)
using the true CCDF, Fr|x,0-
This is the inverse of equation 5 which defines the CQTE in terms of the CQC. Conversely we also tested transforming all our CQC estimators into CQTE estimators (using exactly equation ( with I replacing g²) and testing the accuracy on this space and found similar results.
Each experiment took no longer than hour to run on a single 4 core CPU with 8GB of RAM.
The bandwidths the kernels for the NW estimation of the nuisance parameters were chosen by limited grid search on additional simulated data by validating against the true value of the nuisance parameters. While this unrealistic practice it was done to make estimation of the nuisance parameters strong as possible. This was to err on the side of caution strong nuisance parameter estimation would naturally favour the baseline approaches.
Code to implement our approach alongside Jupyter notebooks running our numerical experiments can be found in the supplementary materials. The code to implement the kernels is adapted from com cot////aaaaaaat///eeeeeeee-e gof/ which is free to use under the MIT Licence.
B Additional results
B.1
Simulation results
We run additional simulated experiments in order to further test and explore our approach. Through- out, overall experimental set-up is the same in Section 4.1
B.1.1
10-dimensional example
To test our problem higher dimensions we ran experiments where X was 10-dimensional with X uniform on [-1, 1)"0. That is each component X, independent with X, U--1, 1). We then took
Y| x, A = Beein~~Nsssniiii ((sin(TTT( x) n (xx) 0.4 x)) 0.5
where 33 was randomly sampled from N(, (0.2)²) and 7 E [0, 3). This gave the maximum gradient multiplied by 0.5 diami))(2)) = vd close to making the rate of change of the CDFs and propensities similar I-dimensional examples.
In our first experiment we take 2n = 1000 and E 0.5, 1, 1.5, 2, 2,2.5,3) 3). The results of this are shown Figure 5a In our second experiment we take ? and 2n E {200, 500, 1000, 2000, 5000). The results of this are shown in Figure
As we can see the approach performs the best with it performing close oracle for low sample size and low frequency. As the frequency increases the DR estimator deviates from the oracle.
15

---

Separate Estimator
Separate
IPW Estimator
IPW Estimator
COTE Estmator (Kallus
Estimetor
Estimator (Ours)
0.8
DR Estimator (Ours)
eerror
Exact DR Estimator (Oracle)
error
0.8
0.6
0.4
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
2000
4000
6000
8000
10000
V
2n
(a) Average error as y increases with 95% CIs (b) Average error as sample size n increases with various approaches.
95% Cls for various approaches.
Figure 5:
Estimation accuracy of 10-dimensional CQC estimation procedures for highly varying CCDFs and constant CQC with respect to x.
B.1.2 Varying CQC
In this experiment our set-up
sin()TTr)
Y|X, A
0.5x + 1.5"
Y = 1 (sin()7x) + 0.255 + 0.75, (0.5x + +.5)))
(z) 0.4 + 0.5
for y [0, 10). This gives g* (y + 00.5z 1.5) which is still simpler than the individual CCDFs but now does depend upon
2, 4, 6, 8, 10). The results of this shown in figure 5a our second experiment we take y = and 2n E {200, 500, 1000, 2000, 5000}. The results of this shown in Figure 5b
In our first experiment take 2n 1000 and 2,
2.00
Separate Estimetor
Separate Estimator
IPW Estimator
Estimator
1.75
COTE Estimator (Kallus et al.)
COTE Estimator (Kallus et al.)
1.50
Estimator (Ouns)
Estimator (Ours)
eerror
Exact DR Estimator (Oracle)
á erroo
1.5
Exact DR Estimator (Oracie)
1.25
1.00
1.0
0.75
2n
(a) Average error as n increases with 95% Cls for (b) Average enror increases with 95% CIs for various approaches.
various approaches.
Figure 6: Estimation accuracy of CQC estimation procedures for highly varying CCDFs and linear CQC with respect æ.
B.1.3 Constant h**
In previous examples, while g" has been simple and constant, h² has actually included the high frequency sine term. In this experiment we adjust our original illustrative example to give constant h" well. Specifically we take
Y = 0~ Unif(sin(77xx)) siny) + 1),
Y Y = -,
(nif(( 2 + 2).
for 7 E [0, 10). We also take the propensity to be <<x) = 0.4 (i4sin(yyaz) + 0.5. In this case h" (yo, yı (z) V1 yo which does not depend on for any yo. v1 V
16

---

In our first experiment we take 2n 1000 and (0, 6, 10). The results of this are shown in figure 5a our second experiment we take ? and 2n {200, 500, 1000. 2000, 5000). The results of this are shown in Figure 5b As we can see we obtain very similar results to original
1.2
Separate Estimator
Separate
IPW Estimator
IPW Estimator
COTE Estmator (Kallus al.)
Estimator (Durs)
Exact DR Estimator
Serror
Exact DR Estimator (Oracle)
@ eerron
Estimetor
0.8
0.6
0.6
0.4
0.4
0.2
0.0
10
2000
4000
6000
8000
10000
2n
(a) Average error as n increases with 95% Cls for (b) Average error as increases with 95% Cls for various approaches.
various approaches.
Figure Constant h"
Figure 8: Estimation accuracy of CQC estimation procedures for highly varying CCDFs and stant ht and CQC.
illustrative example suggesting that our method effectively exploiting simplicity in g rather than in h²
B.2 Employment example: Comparing to the CQTE
To further explore the interpretability of our estimator we provide an estimate of the CQTE for comparison. This can be seen in Figure 9
15000
10000'g
125002
8.
10020
7500,
Eariing Quantile
84
5002
Earnings Quantile (d)
85
8.4
-9000
Age
25.0
27.3
30.0
35.0
13000
Age
Figure 9: Surface and heat plot of the CQTE, Ty (alaz), for our employment data with X =Age, o=Income Quantile.
In these plots of the CQC. This is because, for each covariate value, the quantile value corresponds to a different untreated response making comparisons between various values of the covariates less direct. From this plot we do still see that higher income quantiles are associated with a greater increase in wages. Interestingly the value of the CQTE plummets 90% the highest quantile for individuals of age 40. This is likely an estimation error due to the limited data at higher quantiles as and higher ages.
that the interpretation of the CQTE is less immediate than for the case
B.3 Colon cancer treatment
To demonstrate the effectiveness of our approach in medical settings, we apply it to trial on the effect of colon cancer treatment on survival time/time to remission. This dataset was originally introduced in Laurie et al. 19 and can be found in the "survival" package in R and loaded with the line data (colon, Dackage="survivess Placebo or Levamisole. They are then followed up for period of up to 3329 days and the time till
this dataset 929 are randomised to either receive
17

---

either death or recurrence of their cancer. For our analysis we take our response (Y) as the time to first of death or recurrence. For simplicity, when an individual makes the end of a trial without an event and is censored we take that to be the time of their event. Looking at the data censoring times are mostly at around 2,000-3,000 days while over 90% of death or recurrence events that do occur, occur within 1 1,500 days. Therefore censored individuals will still have better responses than those who had events, as we would want. As our covariate, we took the patients' ages when joining the trial. We show 3D plot and heat plot of the estimate of A (v x)) =g" (z((() in Figure I0
1250
1000
1500
1000
750
800
Time to event (v)
500
4
500
600
Time to event
800
1000
400
-500
600
200
400
200
55 60 65 70
-1500
Age (x)
75
35
60
65
Age
Figure 10: Surface plot and heat plot of (yy over y, for colon cancer trial data with X =Age, ==Time to Event.
Interestingly from Figure 10 we see that the treatment appears to have little effect on patients aged 53-57. For the reaming patients, can that there relatively little effect on the time event for events of 400 days or less while at around 500 days the difference in time to event jumps to 1000 days. This jump takes the time to evidence that rather than the treatment delaying the time to event, it increases the proportion of the patients who have no event during the trial essentially fully treating those patients for the duration of the trial. This shows the value of the CQC it was able to reveal meaningful information about the treatment beyond simply positive effect. On top of this, we are able to identify the censoring within our results without explicitly controlling for it in any way.
(TTE) to the time when censoring begins. This gives
C Additional theory
C.1 Proof of h accuracy
We first define some additional notation. For an output which depends upon = (v, a) define my (x) (Z) æ²) and
(a( = Σ wzj((Z),
JEJ
with wy defined as before so that mz( our estimator and m. (x) is the oracle estimator. Additionally define
((x) = mz)
in other words the bias in our estimate of the pseudo-outcome for given æ. We will also work with m)(x) where we view b as being function of
Finally for n E N, E (0, 1), define
<s(n) = (log(2/0)/n
if (0,2/e]
{1/n
otherwise.
Theorem (Stability Result). For å E (0, 1) we have that with probability at least 1 - (x)(x;) (2)) E( (x: m. (a:) X Dz]1
(12)
2|1er|1aces =: =:B()).
+ 225(n)) (w0l ||p h(vo vn lll
3
18

---

Moreover, with probability at least 6 we have
lm (mpp=))mppp2))) mmp()))== (p) [ / (2 (n)][wlllllll -llw== (6).
(13)
Proof. To prove 12 we first observe that since w is Dr, Me-muuurable we have
Elm D2, Xyl
= Σ
wf -
D2
jEJ
2/ almost surely. Note also that { (5 E I are conditionally independent given Dz, Xg. Hence, the bound (12)
Note also that have max {p(Zz) (i
Elp(Zz)|
follows from applications Bernstein's inequality Boucheron al. Theorem 2.10], applied
conditionally on Dr, X
Next note that by the triangle inequality we have
E (m) - mx)))), Dz. X H
(x) m2) DI. X + (lmp(x)) (m)())) Dz, XyHl
Σ wy (ج, <p jEJ
Dz. Dz:
-B((m mp(x) Xz]
||rigl |I wl (x)) (2)) XxH
Moreover, since (lo(Z,)) Xy1 is the projection of p(2z) onto the subspace of Dz. Xy- measureable functions we have
||m112 lime m (x)))
Σ w - - X1J1,,
=
JEG
$
Σ
wf E((((())) - E(((2))),,.X))))):2....
sE|m]
= Σ ((((((() - +(2,)) + (+(2,) -11
sE(n]
Σ
wf (B(()(Z)) - +(2,)) Dz, (2 Xx) B((p(z,) "b) Xyl)
sEin]
|
l| {(Iim mm +1 mpp m
< u 1 {(I)mn 1w +| - mpllwa>) )
Hence, to deduce (3) we apply the first bound with & in place of yo to obtain the following bound
with probability at least
(x) - (z)x)) S (1)(m)()) mz(x)) Dr. X + v (25((n) |1ll2 Imm mpllww
+ 25 || es (n)/3
1 (m mm Dr, X
+ 22, u 2 ((1iin - mpllum + m llo
+ 2 || Es (n)/3
B(() + ||mmg] w. 2cs (n) /lll lim mipliw
-B+(p),
as required.
We apply the following result from Kennedy 155
19

---

Proposition 4 (Proposition 2 from Kennedy [5]). Suppose that b(zz) = by (zz))2(z)) (α) then
[m)((a)) = || 1 1 12
Proof.
Follows from the Cauchy-Schwartz inequality
Proposition 5. Our pseudo-outcones is conditionally unbiased,
El (2) x hh" (yo. y1 (z))
Proof. We have
El (2) - = A= 1|= T(x)
[B[I(Y s <n = x, A 1|- F((1)(z)) -F(vll)))) + h" (yu. (x))
h1" (yo. 11)x)),
and similarly Blp (Z)[X = x. A = 1| = h" (yo, 3/1 lz] The result now follows by the law of total expectation.
C.1.1 Proof of Proposition
1
Proof of Proposition L Fix you V1, x Let mp(x) be the regression of X against the estimated pseudo-outcont 2o (Z) evaluated at æ (i.e. h(yo, y1 (z)), let m. be the same but with the esti- mated pseudo-outcome replaced with the exact pseudo-outcome Pao..n Finally take
mm (x) = El (2) = x] = P S vılx = x A= 1) - P(Y s 31 X = x, A-A=0) 0).
Theorem 3 gives us that with probability. at least 1 -
(x) m2 S (n, 6) +
S en 6) + [=
with 8((v. x, ;= =
- - = æ².
To bound 6 we have that from Proposition 5
E Eippoo.n vn (Z)lX = æ] = P(Y S y1 = x, x, = - P(Y S yol = x, 22,==00 = =).
Additionally, by splitting over the events {A- = {A = 0} and noting that P(A = 1|X = xx)
क (ax) we have
T(Φ)
E ( (2)lX = =x]= =
(P(Y S = x, A = 1) 1) PP s v = x, A A-1)
[(x))
1 T(x)
(P(Y S = 0) pe S 5ol X = x, 1 - =0)
1 - #(x)
+ P( S w1 X x, A= 11) - S vnl = x, A=
Hence
b(xz)=
T T④x)
(P(Y s snii- x, 1= PP S 2n1i = x, 1-1==))
7(x))
- π (2x)
-1) (PIV s volx = A = 0) - P:y - vol = ==)
- # (xx)
20

---

We have that using Proposition
(x)(()) S |
11
P(Y S 91 X, X P(Y < 31 X,
π
P(Y & X A= 0) P(Y & 80 XA - 0)
P(Y S ya X, A = a) P(Y S ya X, a
80
Now simply matter of bounding all the relevant terms which we do through the following events
Earacle = {Im m
m
(x))
2||2||/ sel }
E2 = {ll +llu> S (n, 6/4)}
E, {mp m I (n, 8/4)]
Eg || klw S Ea (n, (/4))4)]
E3,0 = IIP(Y yo! X, A 0) PP(Y S yol X, = 0) ga(n, 8/4)
E0,1 = [IŘ(Y &
1) PPYY S 91 X A 1) 3 (n 8/4)
According to our assumptions and previous results, Eerack Eg. E,, E. n E3,0 n E3,1 separately occur w.p. at least 8/4. Then by the union bound, the intersection of all these events holds w.p. least 1 and under these events
| mup + myp mmp S er, (n, 5/4) (n, (,,n,,6/)ss(()) 8/4) + (0 (n, (1/444)
Where the first term comes from Proposition the second from Proposition 4 and the final term from smoothness of g"
C.2 Proof of ý accuracy
Proposition 6 (Maximum step-size bound). Suppose that both the outer regression and estimation of CCDFs fit using linear smoothers that
hyyo y1 y1( = Σ
Pa(yl=') = Σ
WF [a S
JEJ
TEI
with wy(xx, wi(x') Additionally suppose with probability at least 1 8
max w/ (x), max W (x,:(X) s (n)
max
JEJ
NEI
Then with probability at least 1
max [h(vo: y(1) (x) hlyo: y-(+2)) xx) (m, JE(m +1)
where [n 11o = [n + 1 {0}, h(yo (10) -1, -1, h(yo, y ((m=-1) |z)), and fiE)) the
sorted version of (Yi)ien.
Proof. We immediately have via union bounds that with probability at least - 5
max (x),,, max wF (X,) S sw 6/n).
max
JEJ
JEJ
Now try bound the jump size of our function. To do for step function y - R and a step points y E y we define
f(Ag) == lim f f(y =).
eL0
21

---

iec the size of the step at y. From the definition of our pseudo-outcome, the jumps occur in different (177, Note that we are assuming Y continuous so these
forms at {Y2) (E.T, (1,))), points are all distinct a.s.
For Y3 with j I and A, 1. We have
Myo. x))l=)= = Was #(x,)
112 & AY,
Wip
S
For with A L, we have that
1 F(AYI))) + -((XI)))
|/ AY(22)) =
Σ wj -1f4, = *(X,)
IE.J
max Fi (AY;)X;)
< JEJ
max W :N (X)) AY)
JEJ
max wr.; (Xz).
{
JEJ
Hence the maximum jump size is less than
max max ( max WF :EIJJT
[JE]
Therefore combining this with our previous bounds gives that with probability at least 5
max [100 (( h(yo. ((+) E- "w (n, 8/n) Ae(m++1pp
Fi We also have result ensuring the step size of the projected function is even smaller. Proposition 7. Let Ep and ā Piso(p) (a). Then la a:41 1 |ā: ăه+1 for all I e [p].
Proof. We will prove this by first giving an algorithm to compute the projection. Define
{{0
a
if an ou+1
(a((a)
= = a1
actor11
ortor11
am
if aq +
2
2
Now define a (0.0) a, (m,i) (a 1) for Ip 1] and a (m.0) (n-11.p-) for N.
Finally define (t) (lttn/)..mmmtlt Then by Yang and Barber 32 we know that lim (!)
Piso())(a))
Now if (n.+1) (n,141))> (nn) aj (n.)), Then 0141 has be moved by th+1. Hence
(n,++)
(n.++) laj+1 aj
(n.). For this increase the distance then must have a141 <a141
(m.+1) <a((tt)
(n,+1)
01
<a aq
Therefore as aj
(m+1,1-1) 2 aj m111 we have that
(n+1.)) (n+1.1)
a1+1 aj
By similar (simpler) argument (a (n./-1)
di+1 (n la (u(ul12) 0
(n.l-2) then (n.)) = a1+1
(n.))
Hence if any iteration moves adjacent points further apart, later iteration will make them equal meaning that convergence they will be equal. Hence have proved our claim.
22

---

We now have the result which justifies our choice to take the isotonic projection of the data which is take from Yang and Barber [32].
Proposition 8 (Theorem from Yang and Barher 32 Let 2", & E RP with x" E [so(p). Then for = (iso(p) (2),
max|= - max 2f -
We now use this isotonic projection to obtain supremum bounds the accuracy of our estimator.
Proposition 9 (Supremum bound on h accuracy). Fix a E X, yo and let h be our original )m to be a potentially random set of points in y in increasing estimate of h. For m E N, take (Y R=1 order.
Now define a E R by og = Myo. (0 and â == Piso(m)(( (a)) Finally, define to be the piecewise constant right continuous function with h(yo, (())xx))) = āg.
Suppose for any y1 E V, n E N. and 6, 8' > that
P ([ v1lz) h- (yo- vnlar)] 2 en (n,,)) 5
IP
max /E(ml]
i(yo, (())2)) - h(vo y((+ (20) 2 Estep m, 8) 5
where we take [m/] = [m]] u (0), N(vo, (()) 1, and h(yo, Y y(m+l) (x) == 1.
Then for any 5,5' > 0, N,
IP sup BIEY
Myo. (1 (z) he (vo. m1/ S En 8/m) + (n, m, 2 1 - 8
Proof. For each E [m] define the event
E = { [16000 x101z)
(x) h- (Bo. 1==12))| s (n(mm66m)
and take E2 = ni=1 Fn. Additionally define the event
Estep
max [E(m]o
h(v0 (101 x)) h(yo, ((()))z))) (m,
Then En and Estep hold w.p.s at least - 8 and 1 - 8' respectively. Also under En, Estops by Propositions 7
we have that
max
T(yo, 1 (ax) 1 (yo, y("h y("|a=) (m))
s en (n,
LE 1m)mm)
max IE (m]
h(yo, y())))) - h(vo, (())))a))))
< Estop m, 8)
We then have that for any y1 E y there exists [m] such that (1-1) v1 S 11
Hence by monotonicity, we have the following 2 inequalities
[yyo vn (x) h" (vo, (x) h(yo, Y, (1 (1-11) (x)) 11- (20. x1 (x)
h(yo y1 (x) h-" h"'v,. (z) h(yo, 11()(0) (x) h" (vo. r, 111-1)
Now for the first inequality we have
A h(vo, X1" ((1-1) //) (vo. (1 xx) + h(vo, (101(=) h2" 11(20. 11
2 (n. 8) Escep (n, m. 8').
23

---

With the final inequality coming from Eh,--1 and our definition of h. By similar argument we get from En, that A2 En 6) + Estep m, 8'). Again we can use the same approach get that under 1 Fh,i, Estep
(n, 8) Estep (n, m, 8') ht (yo. 91(x) - hyyo 31120) S En(nn, Estep m, 8').
Hence for our specific yn under FR,1 Fh,1-1, Pistep
hh" (yo+ y1 [1)=)) hyyo y1 xz) S (T) - Estep(n, m,
Now as this inequality holds for arbitrary yn under FA n Estep (an event which not depend our choice of y/) and the intersection of these two events holds w.p. least - 8 by the union bound we have our result.
Our final key result before piece them all together will be obtain accuracy in g from accuracy in h**
Theorem 10 (Single point accuracy bound). Fix æ. Yo- assume that h, h are strictly monotonic in Y1 and suppose that
suplh (10,31l0)) - h(vo un /x)
91
Additionally let = Fo a²) and assume that I= (x)) > t7 for all y (Be(()))))
where here we are taking FT- (Alx)) to be the pre-image in y of the set A for fixed x. Then
(yul xx) - ((vbl==))
where 9" (yolæ) is the unique y1 such that ht" (Yo. V1 xx) O.
Proof. Let yi == 9" (yolz)) and ún ((volz) so that Fi Fi (z) From our accuracy assumption on the set-up of the Theorem we have
PF (v) Fi æ>]
|/t (yos víl (x) h" (yo, ò1 [z)]
|h" (yo. 01 [x)]
1h* (Vo, in (x)
h(vo- un +11 in [z)
I 1h" (Yo, d1 [x) h(yo (=)=)) vi [x)]
< |h* (yo. 01 (x) h(v0 011=)) h vi vi(=) h- (yo. vil=)
2s,
where the second and fifth line come from the definition of y and the fourth from the definition of 91.
we define f to be the derivative with respect to the t argument of f then we have
(y0 xx) g" (yo [=)|=>]|
(FI (x)))) Fr (Fi (0)
(i
0 (f)=))
FL(bull)))
f((F) (B(x)))) d3 1
F. (a))
25
max
81 E(viii))) f(y1lx)
2s
24

---

C.2.1 Proof of Theorem 2
We can finally combine all these results to prove Theorem 2
Proof of Theorem
From proposition 1 we have that
[[L00 y1 v1(=) - h² (v0. 81 (z)) - En(n, 6)) 2 -
with
En (n, 5) = EI, (n, + tEann ((((m.6/4))) ( 8/4) E,(n, 8/4).
Now define ( to be the sorted version of {Yi) Then by Proposition we have
P max [[yo, r [x)) 1t" 120000 Y, [>)> En-step S 6.
jE(nla
Plugging this into Proposition with Entep(n, m, 8) replaced by { (n, 8/m]) and en en 8) re- placed with en (n, 8/2) gives
P sup
Th l=)) he (2o. vil0)) S {en(n,8/(2n) + ( (m, 8/n)) 2 5.
on Ey
Now let vi g" (Volæ)) Then for any y1 E y if 1/1 - w/o> this implies that (vi (x)
Fi (vf [>]>> by our lower bound on the density in (vi) Hence by the contrapositive, if (F (x) Fi (wilan) [ 87 then M - vil< s.
Now as
(1 - (1()))) sn + ví E (Be
+ v
F Fi (B., (Fo()o)))
Hence if n is sufficiently large so that := (n, (2m) + E e- ew (n, d/n)) TJ5 then satisfy the bounded density condition of Theorem I0. Therefore we can plug our bound into theorem I0 gives
P (10()olæ)) - g" (vol=)) |x) < 2 2 Em, (m(m))))) +5- (n 2 11
C.3 Extension to expectation
Proposition 11. Let Yo. D be RVs on y, 2" respectively (with D representing data used to fit model and Yo representing the point where the model fit). Now take i(Yo, D) be a non-negative bounded loss so that 1(Y,, D) < Imax a.s. Suppose that for any ö > 0. all y E y
P(l(y, D) > e(n, 5)) <
Then for any I, 80 E [0, 1] and p,q [1, oo] such that 1/p + 1/9 =
P (EI(YY, D)ID] < t D)"" 1/0 + E(n, bo)) 2
E(n,
60
In particular if D) < Imax a.s. then taking q = 1.p = 0o yields
P (E7(Y, D)ID) < Ummx + s(n, 56)) 2
7
Proof. First fix 6, e [0, 1] We will first bound the probability that the number of y which don't satisfy our bound isn't 't too large. We do this by defining the event
A = (1(%, D) > E(n, 60)}
and now aim to bound the probability that P) (this probability is just w.r. Yo treating as fixed.)
25

---

By Markov's inequality.
P(P(A > 1) []]
- Yo)l
by Fubini's Theorem
[116ol t
60
Now define B = (P(AAD) t) so that P(B) =1 50. Then under B
Bi((6, DJ - Ei1 A D) + E(((Y, D)IA", D)P(A)))
E A D] (n, 6o) by definitions of A < [[[A|D] D) E on) by Holder's inequality
- // D)? ((
As we are assuming B.
Corollary 12. Assume that F1(y(x) > for all y1 E V. Then, our fixed x E X and E (0, !), w.p. at least.
8 diam (3)
2n, 6/2n) + 6-1Ew(t 8/n2))*
E - (Y|=)) A= 0.9 s 4 En + sm
where En(ō,n) = ET, (n. 8/4) + In particular for (n 8/n) + 26 sn 8/n)) S (z)( For 8 < 1/e we have that w.p. at least ô
|0(((z)) - g" (Y [x)|
A= 0,9
s 12 (a)
Proof. Plugging the result of theorem into proposition
11
noting that l0(volz) 9" ()(11))
diam ()) and taking
2 diam())
(1 (n, + (-m (n,)))/))))
yields that w.p. at least
8 (diam())
lnn 8//) { -l (n,8//)))))
en(n, +2 (n, Following from this if 2 (n-1 5/n) + (m (mm)) < have
E [V]])) (Y ab)=)) A=
Then we
(B(10(V1a) g
A A= 0,g (2)gg(e(m))0))//2e0
8 ddiam())))(()
log(es (n)/5)e
for 8 1/e have
8 < 1<1/ 8 < exp - ( )
1
log(1/8) >
log(ei (n)))())) 1
(n))g(e
+ 28 >
log(ei (n)/0)a²
26

---

Hence
P E [0 [x) 9g (Y())) A 0,9 2 2lgg(s(m))/99/ee(mn) 28 diam()))ey()).
Finally 8 = 23 diam(())ez(m)) gives
P
1 gY(xxl) A 0.0 2 220o2cdaam00/ee//0/////////991/0 88.
C.4 Application and justification of NW estimation with box kernel
We aim show that the box kernel satisfies some of our conditions. Specifically conditions 2 & from Proposition
[ We first start by bounding the step size.
Proposition 13 (Effective sample size). Suppose that for our E X there exists Co 0 such that for any r E (0,ro)
PP(X E Br(x)) 2 Cor
Now for E ro, lake our kernel to (2, (x, = 1[1a | - æ'1s r). Then w.p. at least 6
E (,(x)) 2 nCorr 2n log(1/0) 10g) - log(1/6)/3)
JET
Proof. We have 1[X, (x)) and [[ e = B (x)x)) Cor Therefore by one sided Bernstein's inequality with log(1/6) L/6) we getet
P
Σ 1(2 Br(x)) S nCord 22nCor log(1/6) 5.
JEJ
This gives our desired result.
Note that ΣJEr 11X, B,(x)) is also the effective sample size of estimation it is the number of samples used in the average.
Now that we have the effective sample size result in terms of our kernel radius r, need obtain the optimal rate of decay of for our estimation.
Proposition 14 (NW estimation with box kernel). let ( be the NW estimation of my(()) = [[f(Z) = x using IID copies (Z,);= of Z. and assume f(Z)) < M. For a fixed r E (0,r), kernel ky- as defined above and suppose the same assumptions hold. Suppose that (x) is smooth for a S (i.e. a-Holder continuous.) Then for sufficiently large n depending on Co a and 5 < 2e w.p. least 1 8.
(m)(x)) - -mf(2)l5
2M +
log(2/0)n 24//0
Co
Proof. With our kernel define I, = {i E [m]]k Xi) = 1) and ny = [Tl. Now define the event
Frn (n, 2 Cour (oGordnlo((2// log(2/5)/3).
27

---

Then by Proposition 13 this event oceur w.p. at least 1-8/2. Now if we define E, == -((z)) (X,)
then we have Ele, [A,] Also, we have
1
( m)(()))
J(Z,) my(xz)
FT iEI,
1
( t my(x))
nr İET.
(mx) - m
1
Σ
Ei
Mhr SEI.
KEI,
1
S pd
+
6
nr
IET
With the final equality coming by our smoothness condition and definition of I,. Now by Hoeffding bounds we have that
2 log( (2/6)4))
8
IP
E
n4
2
KEI
Hence by combining this event and En then we have that with probability at least
2 (2g(2)
( - -my(z)]
S +1
Br
2 M
S r
V Courd / 120onnd - log(2/8)/3
Now for sufficiently large n
Conr
Conr 2Courd log(2/8) log(2/6)/3 2log(2/5)
This in turn gives
( m2(x)) a 2log(2/))
Vonrd
Then the optimal choice of is such that
a 1
V
20ea
V V
Hence plugging this in we get that for sufficiently large n depending on Co. we have that for 8 < 2e-1 w.p. at least
2M +
()(()) - my(x)1=
log(2/0)n 24d70
Co
Proposition 15 (Final weight decay). Suppose that for our & E X there exists Co, ro > 0 such that for any E r)
P(X E Bx)x)) 2 Cord.
28

---

Suppose that we are performing NW estimation of an smooth function at points æ E X using l| r) with Tn decaying optimally, Now define kernel kr,, x') 1[ x
krr, (xt, Xi)
w = Ejer Kr (x, Xz.)
the weight of the 1th component. Then with probability least 6
log log(2/0) 24d/0.
max wy = J
Co
Proof. We have
max wy S JEJ
Σjeg k( X,)
Ejer 1(X, B,(x))
Hence by proposition
we have w.p. at least 5
wj JEJ S Coorr"
/2Conrd log(1/6) 1og(1/8)/3)
We know from Proposition 14 that the optimal radius decay gives upd into our result gives that
24077 Plugging this
Con /2C (ogg)o((18)) 1og(1/6)/3))
max wy JEJ
log(1/5)
For sufficiently large depending on Co-
Cun 3+077
Note that for y smooth function the MSE is C-1 24//7, Hence the box kernel comfortably satisfies our weight decay condition 2 in Assumptions 2
Additionally if have smooth function and assume that for any x' X there exists Co(x') such that for all r E (0, ro) P(X E (x(x)) 2 Cor Then we have that w.p. at least 6,
log(2/5) n 24070 Cu(X)
Wr.s (X, <
Thus if we assume that there exists C" 0 such that w.p. at least - 8, Co(T] S C" (log((/))) then get that w.p. at least 5
WF, (X,X²) C" log 10g8/2(3/8)n ,
This then gives our condition 2 in Assumptions We now try to bound
Eser (X)(Xx)) B{E (x)();))
Corollary 16 (Accuracy under NW estimation with box kernel). Suppose that our linear smoother is NW estimation with the box kernel and optimally decaying radius additionally assune that:
Ep(n, 6) = e.(nn) (1o))/ooo (l0((1/)) with e1 = o(1).
Ea(n, 6) = Ca /og((1 1/8)n at/eo
E3(n,0) Co / log(1/0]n 24573.
29

---

'B> 2(14d/7) Then for any 6. for sufficiently large n. the following events each separately hold w.p. at least - 8,
d
(1/6)
h(vo. b1)x) h² (vo, v1 31/
I Ch
(n)
Cg log (ng")((n/0)
(0(y 9" [1(12))
n
(n
Cg diam (V) log 3/2 (e2(n)n/0)
E - A 0,9
En
e2(n)
with On. Cg. Ca depending on Ca. Cp. C² C" (where C², C" are the constants define in Proposition
Proof. For any NW estimator we have that | 1, 2202 1 a.s. meaning we can take E (n. 6) = 1. We also have that (n, 5) C. log (og(1/0)m 11275. Hence ST, (2es(n)es(()) v /og(11) 1/0). This then gives
h(20. (x) (100081113)) 111)))
s C, log (7/0))" sthr7 + CCC's log(7/5)n (atdia t4435))
+ (mm))(1))esm))
Ch (log(160)))))) ेाेन + log(1/0)n -(x d7 +21d70) log(1/5)
(n)
for < e- and C₁ = 7(C, + CaCo + v2) giving our first result.
Using our weight decay results for NW estimation and plugging into Proposition 6 we get that
Sh-step = = C- l/g
(n) (n/8).
Hence plugging into 2
[Ø(v|=) 9" (u|=)) < C₁ C,, log(n/8)
+ 40" log (n/2)() शगेØア
(n(n)
C log (
S En (n)
with C. CAv2+ 4C" giving second result. Finally by Corollary 12 for 5 e- w.p. least
1 6,
Cg diam()) log 3/2 (nen (n))8)
E [((()) - (Y(==) |A A- 0.0)
E1
(n)
Note that both ĒT, and step actually both o(E, (n, e)), thus if we were allowed to take sufficiently large depending upon then we would have all 3/2 terms replaced with log terms and the dependence on C", C removed.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce new estimand, the CQC. We claim this maintain the distributional information of the CQTE which we show through the definition and also on real world scenarios in Section 4 We claim that our method able obtain double robustness which we show in proposition [ and Theorem22 We also demonstrate its strong performance empirically on simulated data scenarios in Section and Appendix B.]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have dedicated limitations section where discuss limitations in theoretical results regarding both assumptions of sufficiently quickly decaying weights in linear smoother and requiring smoothness in h" rather than in g". We also discuss the limitations in the interpretability of quantile hased approaches general when exploring treatment effect well limitations in the interpretability of our assumptions within theory. Finally we briefly address the methodological limitations of our approach acknowl- edging this potential area for improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
31

---

If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The two main results of the paper, Proposition I and Theorem 2 are proved in sections C.1 and C.2 The assumptions for these results are given in Assumptions with the implications of these assumptions discussed.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper gives detailed information about the experimental set-up and ex- plicit algorithms for methods introduced. The specific use of algorithm [ with NW esti- mation is also described Appendix A.2 In addition code is provided the supplementary materials with "SimulatedExperimennn..k provide link to public github repository containig this in a footnote on page 9.
Justification:
reproducing all of the experiments. We also
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on contribution, reproducibility can accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specifie model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with
32

---

the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibiltty can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Code is provided in the supplementary materials with each function docu- mented as well as in linked public github repository. Jupyter Notebooks provided to reproduce all of the experiments and real world data settings used within the paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: wetggddde//oeeuuubeeeeeee
onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
33

---

6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Most of the experimental details are provided alongside the experimental results Section I with additional details provided in Appendix [..4 In addition code to replicate of the experiments provided in the supplmentary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
For our experimental results we explain that 500 replications of the entire fitting procedure (including newly generated data making each run entirely independent) was run and that 95% confidence intervals are provided for the mean absolute error of each estimator.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
34

---

Answer: [Yes]
Justification:
Information on runtime and compute resources is provided in Appendix A..4 Overall computational resources were very low with all experiments run on personal laptop and each one taking less than an hour even with a large number of replications.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ftctpa://neuriig cccpppllic/thiicccceeeellllll
Answer: [Yes]
Justification: No outsourcing was done for this project/paper and all datasets used are pub- licly available and completely anonymised. Additionally we feel there is essentially no scope for our work to cause negative societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive impact of our work in terms of further em- powering HTE analysis. We do not for see any potential negative impacts of our work as is simply furthering the field of HTE and quantile based treatment effect analysis. To knowledge, there no suggestion that these fields could be used maliciously or unfairly beyond the ways that any regression technique analysis could.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
35

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
There are no new datasets released in our work. Our models use standard non- parametric regression techniques and adapt these to best estimate our estimand, the CQC meaning there is no real risk of misuse beyond the misuse of any regression technique.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are publicly available and properly referenced. Proprietary code used is also open source and used in accordance with its licence. The licence for this code can also be found in code itself. The model we use is our own and so no licensing is required.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pattasets.paprsswitthe paperswithcodee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: The only asset is the code which is implemented in our method which is provided in the supplementary materials alongside a licence.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can cither create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects is used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
37

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "BID2hv1GJU", "reviewer": "Reviewer_bJi9", "summary": "The authors propose a new estimand called the Conditional Quantile Comparator (CQC), which computes the quantiles of the outcome distributions of the treatment effect, hence improving on the (usual) conditional average treatment effect. The CQC is defined as a measurable function that maps an untreated outcome to the equivalent treated outcome in the same quantile, conditional on covariates. The authors introduce a doubly robust estimation procedure for CQC, where the estimation is re-framed as a CATE problem using the pseudo-outcome framework. The authors present finite sample bounds on the estimator's error, and provide numerical simulations and result on a real employment dataset.", "questions": "See the weaknesses section above.", "limitations": "The authors do address some limitations and implications of their approach.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow\n- The paper provides a thorough theoretical analysis for the finite sample bounds and proofs of convergence rates for CQC and proposed estimation algorithms\n- The simulations and real use-case considered provide a good intuition to the reader on the benefits of the proposed approach", "weaknesses": "First of all, I'd like to say I am not very familiar with the causal inference literature, so some of my comments might not be as relevant.\n\n- Complexity of using quantiles: can the authors comment in terms of error bounds and convergence rates what is the price to pay to estimate quantiles as opposed to CATE? Some of these results, including the use of pseudo-outcomes and isotonic projections, appear a little bit unusual, as usually quantiles are estimated through the pinball loss. As usual what is important is assessing the left tail or right tail (i.e., whether the treatment effect crosses the zero or not), I would expect the right/left tail quantiles to be the most important ones -- is the error in estimating all the quantiles the same across the entire distribution?\n\n- Dimensionality of the input $\\textbf{x}$: both the simulation and the real dataset use a very low-dimensional $\\textbf{x}$, and the comments on using the Nadaraya-Watson estimator and kernel smoothing are really only applicable when X is generally low-dimensional. How does in practice your method behave as function of the input dimension $\\textbf{x}$, and would your comments still be applicable even in higher-dimensional settings?\n\n- Estimating quantiles vs entire distribution: how does this approach compare to other approaches that estimate the entire conditional density, such as [1]? Is the use of a doubly robust framework guarantee the optimal error rates?\n\n\n[1] Zhou, T., Carson IV, W. E., & Carlson, D. (2022). Estimating potential outcome distributions with collaborating causal networks. Transactions on machine learning research, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and salient comments and aim to address them now.\n___\n# Complexity of using quantiles\nRegarding the indirect estimation of the quantiles: due to the monotonicity of the CCDFs and $h^*$, we incur only a trivial increase in our error bounds for estimating the CCDF and then inverting when compared to directly estimating the quantiles.\nSpecifically, our rate only gains a $\\log(n)$ factor. This trivial cost is greatly outweighed by the benefit of being able to properly exploit the smoothness of our estimand.\n\nThe isotonic projection step itself is a fairly minor technical detail to ensure our desirable theoretical results. Empirically, we find the function to be close to monotonic before the projection. In Figure 1, we plot our approach with and without isotonic projection (taking the generalized inverse in both cases) and see no meaningful difference in the outcome (the green and purple lines).\n\nWhen it comes to estimating specific tails of the distribution, our method allows us to specify an input $y_0$ for our CQC, enabling us to focus on the tails if desired. We do have to estimate across all $y_1$ values, however as mentioned, this does not pose an issue regarding our estimation accuracy. \n___\n# Dimensionality of the Input\nWe appreciate that NW estimation may not generalise to high dimensions quite as well, however, our theory is for the more general class of linear smoothers. These encompass many methods (k-NN regression, Mondrian forests, Generalised forests), including methods associated with high-dimensional data such as kernel ridge regression (see global response for references and more detail). We use NW estimation as a simple illustrative example as we focus more on the general idea of the pseudo-outcome regression rather than attempting to optimise the regression technique itself. Furthermore, as we are assuming simplicity of the $h^*$, we can exploit the fact that some linear smoothing approaches can adapt to simple/low-dimensional manifolds in our covariates even when the data itself is high-dimensional.\n\nFinally, while our theory is for the case of linear smoothers, in practice, it can extend beyond this, and other high-dimensional regression techniques can be used.\n___\n# Estimating quantiles vs Entire Distribution\nThank you for bringing this work to our attention, we will make sure to mention it in our paper.\nIn Zhou et al., the output is separate estimates of the two conditional distributions ($Y|X,A=0$ and $Y|X,A=1$).  As such, the optimal rates for estimating these distributions will depend on the smoothness of the marginal quantiles. In our approach, we show that we are able to obtain rates that are better than the estimation rates for the marginal quantiles, and thus necessarily improve upon the estimation rates obtained in their work.\n\nWe have also added a comparison to another work on robust CQTE estimation (Kallus et al., 2023). This can be seen in Figure 1, where we achieve better performance (green/purple line) compared to the robust CQTE method (brown line). This improvement is due to the smoothness present in the CQC, which is not present in the CQTE or the marginal distributions.\n___\n# References\nKallus, N. and Oprescu, M. (2023). Robust and agnostic learning of conditional distributional treatment effects. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of the 26th international conference on artificial intelligence and statistics, volume 206 of Proceedings of machine learning research, 6037–6060. PMLR.\n\nZhou, T., Carson, W. E. t., and Carlson, D. (2022). Estimating Potential Outcome Distributions with\nCollaborating Causal Networks. Transactions on machine learning research, 2022. Place: United States.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7BPDBZi3e", "reviewer": "Reviewer_nAy3", "summary": "The authors propose the Conditional Quantile Comparator (CQC), a function that maps an outcome from the control group in a binary treatment setting to an outcome in the treatment group, such that they represent the same conditional quantiles in their respective distributions. The estimation procedure consists of two stages: first, estimating a conditional contrasting function that examines the difference between conditional CDFs, followed by isotonic regression. The contrasting function is estimated using pseudo-outcome regression, which confers the algorithm with double-robust properties in finite samples. This estimator addresses some of the shortcomings of the Conditional Quantile Treatment Effect (CQTE), which is not robust to errors in the quantiles. The CQC is supported by a theoretical analysis of finite sample convergence rates under smoothness assumptions, as well as by empirical simulations.", "questions": "See weaknesses section.", "limitations": "The authors have adequately adressed the limitations of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors propose a novel quantile-based treatment effect estimation procedure tailored for skewed outcome distributions. The double-robustness properties in the first stage of the algorithm are desirable as they improve the rate dependence on the nuisance functions ( propensity and local CDF estimates). The paper is clearly written, with sound theoretical results and promising empirical evidence.", "weaknesses": "* The motivation for the estimator is somewhat weak.  Estimators for CQTEs with double-robustness properties have been already proposed (see [1, 2]). These estimators have a second order dependence on the rate of quantile estimation, as desired. The authors should contextualize their work with respect of exiting literature (which seems to overall be missing in the paper) and compare their algorithm with the existing techniques. \n* Moreover, the pseudo-outcome  estimation technique in [1] is somewhat simpler (see their Appendix B) since they only require one pseudo-outcome regression. There seem to be some tradeoffs, e.g. the CQC requires a two stage procedure that includes a prost processing step (the isotonic regression), as well as for Algorithm 1 to be run several times for each value of $y_1$ considered (because the conditional CDFs need to be estimated at each value). On the other hand, the robust CQTE algorithm from [1] require estimating the conditional PDF at several points. Overall, the CQC seems to be a more computationally (and statistically) intensive estimation procedure and the authors should at least discuss the tradeoffs. \n* Furthermore,  [1] provides rates for more general classes a functions outside of the linear smoothers framework in [3]. I suggest the authors try to provide similar guarantees in order to generalize their results beyond linear smoothers which, to the best of my knowledge, are not often used in practice. This would be particularly useful since the rate results in section 3.4 require all nuisances to be estimated using linear smoothers. \n* Empirical evaluations should also include these existing methods. \n\nOverall, I tend towards soft rejection. This estimator might be useful in its own right, but the authors should contextualize and compare with existing work. I am willing to update my score upon further discussion.  \n\n[1] Kallus, Nathan, and Miruna Oprescu. \"Robust and agnostic learning of conditional distributional treatment effects.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n[2] Leqi, Liu, and Edward H. Kennedy. \"Median optimal treatment regimes.\" arXiv preprint arXiv:2103.01802 (2021).\n\n[3] Kennedy, Edward H. \"Towards optimal doubly robust estimation of heterogeneous causal effects.\" Electronic Journal of Statistics 17.2 (2023): 3008-3049.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and salient comments and aim to address them now.\n___\n# Justification of our DR estimator in the context of other estimators\nWe thank the reviewer for pointing us towards these works and acknowledge that they do provide robust methods for estimating the CQTE. We will make sure to include these works in our paper and discuss the tradeoffs between them.\n\nWe see the main strength of our work being the fact that the CQC is a more desirable object to target than the CQTE. This is because the smoothness of the CQC appears to be a more intuitive and naturally occurring phenomenon than the smoothness of the CQTE. Example 1 in our paper illustrates this: the CQTE still contains high-frequency oscillation terms, while the CQC does not. We have attached Figure 2, showing the \"non-smoothness\" of the CQTE, to the PDF and will include it in the final paper.\n\nIn general, there seems to be limited intuition about what smoothness of the CQTE represents (for example, there is not much discussion of this in Kallus et al. (2023) paper). This contrasts with the CQC, where we believe there is a more intuitive explanation. Specifically, if we are in the potential outcome framework and $Y_1=\\phi(Y_0,X)$ with $\\phi$ being some increasing function in $Y_0$, then $\\phi$ is the CQC. Thus, smoothness in this $\\phi$ represents smoothness in the CQC. This provides a generalisation to the CATE case, where we observe smoothness of the CATE when $Y_1=Y_0+\\phi(X)$ with $\\phi(X)$ being smooth. We naturally therefore see the CQC as the more general extension of the CATE when comparing two distributions (note: here the CATE transformation is also monotonic in $Y_0$.) \nA simple example of this would be a treatment that halves the response for an individual (e.g. a blood pressure medication which halves an individual's blood pressure.) In this case the CQC would be $g^*(y,x)=0.5y$ whereas the CQTE would depend upon the distribution of $Y|X,0$ making it non-smooth if this marginal distribution is non-smooth. \n\n\nOutside this potential outcome framework we can also think of smoothness in the CQC as representing a smooth change in the conditional QQ-plot between $Y|X=x, A=0$ and $Y|X=x,A=0$ as you vary the covariates ($x$).\nOverall, we feel the CQC is a natural way to characterise the relationship between two distributions, having many useful alternative definitions. For example, the CQC is the unique monotonic function that maps one distribution to the other. This monotonicity seems an intuitive restriction, as it ensures that healthier patients without treatment are mapped to healthier patients under treatment.\n___\n# Comparing estimation procedure tradeoffs\nWe appreciate your comment about discussing the trade-offs between our approach and that of Kallus et al. (2023) and will make sure to do so. We agree that our approach is more statistically intensive due to the need to invert our CCDF contrasting function. In terms of complexity, while our procedure does have more steps, it has the benefit of estimating fewer nuisance parameters. Specifically, we do not need to estimate the reciprocal conditional PDFs, which can be difficult to estimate in low-density regions where errors can blow up. Even with the desirable second-order error terms that this estimation incurs, it could still have a strongly negative impact on the estimation accuracy.\n___\n# Generalising beyond Linear Smoothers\nWe appreciate the reviewers' suggestion to extend our results beyond linear smoothers. In the context of Kallus et al. (2023), our result represents a different assumption about the class of regression functions, rather than a more restrictive one.\n\nLinear smoothers encompass a broad range of methods, including kernel ridge regression, k-NN regression, Mondrian forests, and generalised random forests (refer to the global response for details and references). Additionally, Theorem 3 of Kallus et al. (2023) incorporates Kennedy's (2023) stability condition (acknowledged in their proof in Appendix A4). Since stability has primarily been demonstrated for linear smoothers (in Kennedy (2023)), this result is comparable to ours.\n\nOur requirement for CCDF estimators to be linear smoothers in Section 3.4 is solely to control the maximum step size. We are open to replacing this with a more general condition on the step size, which would mean only requiring the outer regression to be a linear smoother.\n\nWhile Kallus et al. (2023) provide Theorem 2 with desirable convergence rates beyond linear smoothers, their assumptions on bracketing entropy restrict function classes one can use significantly. Therefore, we view this as an alternative result rather than a more general one. Since submitting the paper, we have obtained preliminary results in this framework and are prepared to include them in the supplementary material.\n\nFinally, a key advantage of our result is the provision of pointwise accuracy bounds, as opposed to bounds in expectation over responses and covariates. This allows our accuracy results to adapt to local smoothness and avoids penalties for the worst points.\n___\n# Comparing estimation results\nWe have taken your suggestion and compared our method against theirs in our empirical results. This can be seen in Figure 1 of the PDF document.\n\nThe results show that our approach (green/purple) clearly outperforms the CQTE estimation approach (brown) as the frequency term ($\\gamma$) increases in the left plot. This is because the CQTE has a complexity that depends on the frequency term, while the CQC does not. We also observe much better performance as the sample size increases. We hypothesise that the plateau in the CQTE approach is due to the difficulty of estimating the reciprocal of the PDF, which causes the estimation to be unstable regardless of sample size.\n___\n# References\nKennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7Ci8KE3Gy", "reviewer": "Reviewer_CQR9", "summary": "The paper studies an important problem -- treatment effects often have distributional effects and considering conditional mean outcomes is often not the correct objective for evaluating prescriptive interventions. One popular approach to consider heterogeneous effects across a population which allows researchers to determine treatment effects at different quantiles of the study distribution to determine how treatments might effect the worst-off populations. Unfortunately, as the authors note obtaining quantile treatment effects often relies strict smoothness assumptions on the marginal quantile functions and lacks the double robustness properties of CATE estimators. To this end, the authors propose a new estimand to evaluate heterogenous treatment effects that they show too be doubly robust and apply this estimator to both simulational and real-world data.", "questions": "In remark 3 you mention you use a cross-fitting method to improve stability, why is this the case? Duplicating the data yields non-independence?\nHow do i interpret the earnings for groups with the highest/lowest benefits in Figure 4?", "limitations": "The authors have addressed some limitations of their work but have not acknowledged possible negative societal impact. From my perspective, work that emphasizes the importance of considering distributional effects is generally societally beneficial. The main limitation that the authors address are primarily on the interplay between the assumption on h and if they could be relaxed to assumptions on g. However, there are several other assumptions and limitations that should be made clear in this work. The authors should potentially consider including limitations in their analysis of the two real-world data sets which might give practitioners a better understanding of how to use this method. Examples of what could be added are given in the weakness section especially with regards to interpretability of results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a clean mathematical framework for their estimator and proves useful finite sample properties that should be theoretically convincing for users\n- Example 1 is a nice simulational example to showcase why typical conditional quantile estimators may provide instable results when the author's CQC estimator is more robust\n- Proposition 1 provides the double robustness result which has been a useful benefit in finite sample analyses and helps to make the case for using this estimator\n- Numerical experiments in section 4 help to show case the benefits of this procedure. In particular the second graph in Figure 3 provides good evidence for using this methodology.\n- Finally, the application to two real-world data sets is helpful to showcase the viability of this methodology for practioners.", "weaknesses": "- I understand that the CQTE can be written as a function of $\\Delta^*$ but this does not help me to understand how I should interpret this quantity in relation to the typical CQTE. The authors indicate that this it is a \"rephrasing\" but and provide the mapping. but its not obvious how this translates in actual examples. Since the authors are proposing a non-standard estimand they should provide more details linking this to the standard analysis. One place where this would be very beneficial is in the interpretation of the real-world data sets. I consider this to be the main weakness of the work because without a clear interpretation for the estimand practioners will not adopt this methodology. Without this I doubt the impact of the paper beyound showcasing a technical deficiency in the CQTE estimand due to smoothness requirements.\n- I found the algorithms a little difficult to understand especially because of the reindexing where you duplicate the data. This seems purely notational and seems like you could simplify the notation here.\n- The method does not require smoothness of the marginal quantiles, however h must be Holder-smooth. Certainly in many cases smoothness of the marginal may effect smoothness of h. Since the reason for introducing this method is non-smoothness of the marginal quantiles it might be useful to expand on the class of problems where this is an easier problem beyound the example given in Example 1. How does this interact with smoothness of g?\n- To that end, could you also provide real-world examples where you might expect the marginals to be poorly behaved but the CQC to be well be haved?\n- Finally, in the the final paper in paper it would be good to have the CQTE estimates in the real world examples so you can draw a contrast between the methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and salient comments and aim to address them now.\n___\n# Interpreting the CQTE\nWe appreciate how crucial it is to properly justify and explain a new estimate and will make sure to provide more intuition and interpretation for our estimator in the final document.\n\nAs a brief insight, we argue that the CQC is inherently a more intuitive object overall than the CQTE, as it more naturally characterises the effect of a treatment. To compare with CATE, consider the canonical example in the potential outcome framework where we assume $Y_1=\\phi(X)+Y_0$ for some $\\phi$. In this case, $\\phi(X)$ is the CATE, and so a smooth CATE is given by a smooth $\\phi$. The CQC generalises this notion beyond additive treatment effects to the case $Y_1=\\phi(Y_0,X)$, where $\\phi$ is any function that increases in $Y_0$ (the CATE case is a subset of this). If we want to express this in the CATE form, we can write it as $Y_1=\\Delta^*(Y_0,X)+Y_0$.\n\nA simple example of this might be a treatment that halves the response for any individual (e.g., a blood pressure medication that decreases an individual's blood pressure by 50\\%), in which case the CQC is $g^*(Y,X)=0.5Y$. Here, the CQC is constant with respect to the covariates, whereas the form of the CQTE will depend on $Y|X,A=0$, which may lead to a complicated dependence on $X$, hampering its interpretability (as will also be the case for the CATE).\n\nFrom another perspective, we frame our approach as a conditional QQ-plot. QQ-plots are already an established tool for comparing two unconditional 1-dimensional distributions. Our CQC function provides the QQ-plot conditioned on each possible value of our covariates, allowing us to apply this QQ-plot tool at the individual patient level, which is central to HTE studies.\n\nOverall, we see our approach as the most intuitive way of comparing the treated and untreated response across the entire distributions.\n___\n# Algorithm clarity\nWe appreciate that the reindexing in our algorithms adds complexity, making it more difficult to understand. We will find a more intuitive way to present this sample-splitting concept and potentially introduce it textually before the algorithm to avoid notation overload.\n___\n# Smoothness of $h$ vs smoothness of $g$ and CCDFs\nWe know that the smoothness of $h^*$ is a weaker assumption than the smoothness of the marginal distributions. For example, if $Y|X,A=0$ are both uniformly distributed with a fixed width (but with a non-smoothly varying centre with respect to $X$), and $Y_1=\\phi(Y_0,X)$, then the CCDFs will be non-smooth, but the differences will be smooth.\n\nWe believe that the smoothness of $h^*$ and $g^*$ are technically distinct concepts (with neither implying the other). However, in most practical examples (as illustrated above), when $h^*$ is smooth, so is $g^*$. Overall, we acknowledge that the smoothness of $h^*$ is less interpretable than the smoothness of $g^*$, and we consider extending to smooth $g^*$ as a goal for future work, as mentioned in our Limitations section.\n___\n# Including CQTE in real world examples\nThank you for the suggestion. We have added CQTE plots in the PDF, as shown in Figure 3, and will use this as an opportunity to highlight the relationship between the CQC and the CQTE in our paper. We will ensure to discuss the benefits and drawbacks of each approach with respect to interpretability here as well.\n\nIn these plots, we can see that the interpretation of the CQTE is less obvious. This is because, for each covariate value, the quantile value corresponds to a different untreated response. We believe this makes direct comparisons between different covariate values less intuitive.\n___\n# Questions\n> In remark 3 you mention you use a cross-fitting method to improve stability, why is \nthis the case? Duplicating the data yields non-independence?\n\nIn this context, improving stability means minimising the impact of our data split on the estimator. Cross-fitting will also maximise sample utilisation which we would expect to improve the estimation overall. Cross-fitting maximises sample utilisation and is therefore also expected to improve overall estimation accuracy. Using the same data for both inner and outer regressions would lead to dependence between the estimation of the outcome and the nuisance parameters, which would affect the theoretical results.\n\n> How do i interpret the earnings for groups with the highest/lowest benefits in Figure 4?\n\nBy groups I assume you are referring to covariate groups. For the covariates, the groups with the most significant impact appear to be the youngest (those close to age 23), and individuals aged 30-36.\n\nFor the youngest group, there is overall improvement, but most of this improvement occurs at the highest earnings. This suggests that the intervention increases the pay disparity between individuals, likely because those already earning more benefit the most. However, as with the CQTE, we cannot make strict assertions about individual-level impacts.\n\nIn the 30-36 age group, there is a more even increase in incomes across the board. This indicates that the intervention shifts the entire income distribution up for this particular age group.\n\nOverall, those with the least benefit seem to be individuals with low non-intervention earnings and those aged 38-40. They experience minimal or no benefit from the intervention, with the value of $\\Delta$ being close to 0.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3EXvmwdT1S", "reviewer": "Reviewer_zMNr", "summary": "The paper introduces a new estimator to obtain conditional quantiles of a treatment effect and argues that it is superior to the pre-existing \"conditional quantile treatment effect\" (CQTE) estimator. The main difference is that accurate CQTE estimates require accurate conditional quantile models even though the latter can be hard to fit, while the introduced CQC estimator is more robust to the CDF estimators it relies on.", "questions": "See above", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Introduces the doubly robust property to estimating conditional quantile treatment effects, a property that many CATE estimators already have.\n\nDetailed theory work supporting the claim of robustness of CQC to its component estimators.\n\nExploits existing work on doubly robust CATE estimation by reframing CCDF estimation as a CATE-like problem.", "weaknesses": "In 4.1 you test against estimating the CCDFs separately and taking their difference. Do you mean estimating the inverse CDFs / quantile functions and taking their difference i.e. CQTE? If not, what exactly is the procedure here and why not compare against the difference of two quantile estimators.\n\nCCDFs are arguably harder to estimate than quantile functions in my experience (they require the additional 'feature' y, have to work with classification losses instead of regression losses, and are more prone to monotonicity violations). So even if the final estimator here is more robust to bad CCDF estimation I'm not convinced it will necessarily be better in practice to the CQTE approach of fitting quantile functions. \n\nThis is especially true with modern day neural network based quantile models. NW estimators are unlikely to be as good as those on larger / higher dimension datasets, and it's not clear how the theory would hold in cases like that.\n\nIsotonization can be risky and create large flat zones for $\\hat{g}$. Did you notice any in your experiments? Are there other ways to get monotonicity here, similar to the various strategies to get monotonicity in multi-quantile regression?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and salient comments and aim to address them now.\n ___\n# Clarification on the separate estimator\nWe appreciate that we did not provide enough information on this separate estimator and will ensure to include its explicit form in the paper.\nWe do not quite take the inverse CDFs and subtract them, as that would provide an estimate of the CQTE, which is different from our CQC estimand. \n\nOur separate estimator can be thought of in two ways. For both we have estimates $\\hat F_{Y|X,1},\\hat F_{Y|X,0}$ of the CCDFs. We either use:\n\n1. $\\hat g=\\hat F_{Y|X,1}^{-1}(\\hat F_{Y|X,0}(y_0|x)|x)$ \n2. $\\hat g=\\hat h^{-1}(y_0,0|x)$ with $\\hat h(y_0,y_1|x)=\\hat F_{Y|X,1}(y_1|x)-\\hat F_{Y|X,0}(y_0|x)$ and the inverse of $\\hat h$ being w.r.t. the second argument. \n\nBoth methods are actually equivalent. The first method may be more intuitive, while the second method involves estimating $\\hat h$ using the classic plug-in estimator (which is equivalent to the T-learner in CATE) and then inverting to obtain $\\hat g$, as we also do in our DR procedure.\n\nReturning to the separate estimator you mentioned (i.e., taking the difference of the two inverse CCDFs), if we were to compute this and then transform it using our estimated CCDFs to obtain the CQC (as detailed in the transformation between the CQC and CQTE in Section 2.2), it would be numerically very close to the two approaches described above. Any differences would arise from the fact that the inverses of step functions are not true inverses but generalised inverses.\n___\n# Estimating the CCDFs vs the quantiles\nWe don't see personally see evidence to suggest that CCDFs are more difficult to estimate than quantile functions. In fact, a common non-parametric kernel approach for estimating quantile functions involves first estimating the CCDFs and then inverting this estimate. Additionally, due to the monotonicity of the true CCDF, the price we pay for estimating over all $y$ rather than a single point is very small. This is reflected in our theoretical results, which show only a $\\log(n)$ slower convergence rate compared to the equivalent results for CATE estimation.\n\nRegarding the ensuring of monotonicity in our estimate, we use the method from Yang and Barber (2019). This approach allows us to initially obtain a non-monotonic estimator and then improve it by monotonically projecting. Therefore, we do not need to be overly concerned about monotonicity in our initial estimator.\n___\n# NW estimation and issues with high dimensionality\nWhile we agree that NW estimation may be impractical for high-dimensional problems, our theory applies to the case of linear smoothers, which is a broad class that includes generalised random forests, k-NN regression, and methods often associated with higher-dimensional data, such as kernel ridge regression (see the global response for references and more detail). We use NW estimation as an illustrative regression technique since this paper focuses on the estimation of CQC rather than on the regression techniques themselves. Additionally, our pseudo-outcome regression could be readily extended to incorporate neural networks or other high-dimensional regression techniques.\n___\n# Issues with isotonization\nThank you for flagging this issue. We agree that isotonization can cause problems when applied to functions that significantly violate the monotonicity assumption. In practice, however, we find that our approach produces functions that are already very close to isotonic, meaning the projection has minimal effect. This is illustrated in Figure 1, where we compare the method with isotonic projection to the method without isotonic projection, taking the inverse to be the first point where it exceeds zero. As shown, the results are indistinguishable (red and purple lines).\n\nWe focused specifically on a post-processing approach because it allows us to preserve the accuracy of our original, unmodified estimator. While other methods could be used to guarantee monotonicity, they would be incompatible with our theoretical results. Given that our monotonicity violations are relatively small, post-processing feels the most logical approach.\n___\n# References\nYang, F. and Barber, R. F. (2019). Contraction and uniform convergence of isotonic regression. Electronic\nJournal of Statistics, 13(1):646 – 677. Publisher: Institute of Mathematical Statistics and Bernoulli Society.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of the reviewers for their thoughtful questions and valuable feedback. We address most points in the reviewer-specific responses but will highlight a few key points here as well.
___

# Comparisons to other approaches
In line with feedback from multiple reviewers, we have included a comparison to a CQTE estimation approach in our empirical results. This new comparison can be seen in Figure 1 of the attached PDF. The approach from Kallus et al. (2023) is represented in brown in the figure. As shown, it performs significantly worse than our approach, which we believe is due to the increased complexity of the CQTE in this example compared to the CQC. To compare this estimator with CQC estimators, we used the transformation highlighted in Section 2.2 of our paper to produce a CQC estimator. This transformation was applied using the exact CDFs to ensure that it was not the transformation affecting performance.
___

# Linear Smoothers and NW estimation
Multiple reviewers mentioned the potentially poor performance of NW estimation in higher dimensions. While this is absolutely true, we are using NW estimation as a simple example of a linear smoother for illustrative purposes in our experiments. Our theory itself applies to all linear smoothers. Despite their simplistic definition, linear smoothers encompass a deceptively broad class of estimation techniques used in both low and high-dimensional settings. Examples include k-NN regression (Chen, 2019; Chen et al., 2019), kernel ridge regression (Singh et al., 2023), generalised forests (Athey et al., 2019), and Mondrian forests (Lakshminarayanan et al., 2014). Additionally, even in high-dimensional cases, linear smoothers have been shown to adapt to intrinsic low dimensionality, which relates to our notion of the CQC being simpler or smoother than the marginal quantities (Kpotufe, 2011).
___

# References
Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests. The Annals of Statistics,
47(2):1148 – 1178. Publisher: Institute of Mathematical Statistics.

Chen, G. (2019). Nearest neighbor and kernel survival analysis: Nonasymptotic error bounds and strong
consistency rates. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th international
conference on machine learning, volume 97 of Proceedings of machine learning research, pages 1001–1010. PMLR.

Chen, P., Dong, W., Lu, X., Kaymak, U., He, K., and Huang, Z. (2019). Deep representation learning for in-
dividualized treatment effect estimation using electronic health records. Journal of Biomedical Informatics,
100:103303.

Kallus, N. and Oprescu, M. (2023). Robust and agnostic learning of conditional distributional treatment
effects. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of the 26th international
conference on artificial intelligence and statistics, volume 206 of Proceedings of machine learning research,
pages 6037–6060. PMLR.

Kpotufe, S. (2011). k-NN regression adapts to local intrinsic dimension. In Shawe-Taylor, J., Zemel, R.,
Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances in neural information processing systems,
volume 24. Curran Associates, Inc.

Leqi, L. and Kennedy, E. H. (2022). Median optimal treatment regimes. arXiv: 2103.01802 .

Singh, R., Xu, L., and Gretton, A. (2023). Kernel methods for causal functions: dose,
heterogeneous and incremental response curves. Biometrika, 111(2):497–516.

Meta-review: This work proposes a new estimand for comparing the quantiles between control and treatment groups. The Conditional Quantile Comparator (CQC) maps an outcome from the control group to an outcome in the treatment group, such that they represent the same conditional quantiles in their respective distributions. The paper is motivated by the shortcomings of the Conditional Quantile Treatment Effect (CQTE), which is not robust to errors in the quantiles. 

The authors propose an doubly-robust estimation procedure which estimates a conditional contrasting function that examines the difference between conditional CDFs, and then performs isotonic regression. The contrasting function is estimated using pseudo-outcome regression, which provides double robustness. 

Several experts reviewed the paper and raised concerns on interpretability and lack of appropriate comparisons to prior work. The authors have addressed these issues during the discussion period, which should be incorporated in the camera-ready version of the paper.
Decision: Accept (poster)