Conditional Outcome Equivalence: A Quantile Alternative to CATE

Authors: josh  givens
henry w j reeve
song  liu
katarzyna  reluga
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:01:19

Abstract: The conditional quantile treatment effect (CQTE) can provide insight into the effect of a treatment beyond the conditional average treatment effect (CATE). This ability to provide information over multiple quantiles of the response makes the CQTE especially valuable in cases where the effect of a treatment is not well-modelled by a location shift, even conditionally on the covariates. Nevertheless, the estimation of the CQTE is challenging and often depends upon the smoothness of the individual quantiles as a function of the covariates rather than smoothness of the CQTE itself. This is in stark contrast to the CATE where it is possible to obtain high-quality estimates  which have less dependency upon the smoothness of the nuisance parameters when the CATE itself is smooth. Moreover, relative smoothness of the CQTE lacks the interpretability of smoothness of the CATE making it less clear whether it is a reasonable assumption to make. We combine the desirable properties of the CATE and CQTE by considering a new estimand, the conditional quantile comparator (CQC). The CQC not only retains information about the whole treatment distribution, similar to the CQTE, but also having more natural examples of smoothness and is able to leverage simplicity in an auxiliary estimand. We provide finite sample bounds on the error of our estimator, demonstrating its ability to exploit simplicity. We validate our theory in numerical simulations which show that our method produces more accurate estimates than baselines. Finally, we apply our methodology to a study on the effect of employment incentives on earnings across different age groups. We see that our method is able to reveal heterogeneity of the effect across different quantiles.

Main paper: 
 introduction: 
 In many real world scenarios such as personalised treatment allocation and individual level policy decisions, understanding the effect of a treatment/intervention at an individual level is invaluable in providing bespoke care. The field which aims to understand a treatment's effect given certain covariates is referred to as heterogeneous treatment effect (HTE) estimation and has seen popularity across many applications [11, 10, 24, 20] . Within HTE, the conditional average treatment effect (CATE) has proved itself to be a popular target of study in this area due to its simplicity and interpretability [2, 13, 28] . A key limitation of the CATE however, is that it fails to paint a full picture of the differences between distributions of the two responses. In addition, it can be sensitive to outliers, with extreme values leading to a biased outcome. As such, the conditional quantile treatment effect (CQTE), an estimand which compares the conditional quantiles of the distributions in the treated and untreated populations, has established itself as a popular alternative [1, 5, 26] .While the CQTE offers more information than the CATE and is more robust to outliers, it lacks some of the CATE's desirable estimation properties. Specifically, CQTE estimation involves estimating the quantile functions for the two marginal outcomes. This harms the estimation procedure in cases where estimation of marginal quantile functions is more challenging than estimation of the CQTE itself. An example of this is when the marginal quantile functions are less smooth as a function of the covariates than the CQTE. This aligns with a recurring idea within HTE estimation that the effect of a treatment may be simpler than the marginal outcomes. In contrast to the CQTE, there are many CATE estimators which aim to learn the CATE directly allowing them to exploit its relative simplicity. These include the X-learner [17] , R-learner [23] , and Doubly Robust (DR) learner [15] . Before estimating the CATE, these procedures require estimation of intermediary estimands (nuisance parameters) which condition on the covariates such as the average marginal outcomes and the propensity score (the probability of being assigned to treatment group). These nuisance parameters are then used to aid the estimation of the CATE. With the DR learner specifically, it has been shown that it can still achieve optimal convergence rates even when estimation of the nuisance parameters is worse than estimation of the CATE itself. This notion is referred to as double robustness, as our estimation is robust to sub-optimal estimation of both of the nuisance parameters. Some attempts have been made to improve CQTE estimation [34, 33] with a key work being that of Kallus and Oprescu [14] . In this they provide an extension of the double robustness property to the CQTE, creating an estimation procedure that can achieve strong convergence even when nuisance parameters are more difficult to estimate. Unfortunately, one of the nuisance parameters which must be estimated is the reciprocal of the conditional densities over the response. These are highly difficult to estimate and risk the errors blowing up in low density regions which could potentially nullify the desirable estimation rates they achieve even with the dependence on the estimation accuracy of these nuisance parameters being less strong. Furthermore it is still unclear how one can interpret relative smoothness in the CQTE compared to the individual quantiles with their being relatively little discussion of this within the literature. In general there is a distinct lack of illustrative examples; which are present for the CATE. To our knowledge no other works specifically aim to tackle this double robustness phenomenon for the CQTE or other quantile based treatment effect estimands.We introduce a novel estimand called the "conditional quantile comparator" (CQC). The CQC gives the outcome for a treated individual in the same quantile as a given outcome for an untreated individual, conditional on covariates. This relates to the conditional Quantile-Quantile (QQ) plot for the treated and untreated outcomes as demonstrated in Figure 1 . Similarly to the CQTE, our new esti- Figure 1 : The left plot gives the CQC surface which takes in covariates (x) and an untreated response (y) and returns the treated response of the equivalent quantile (g * (y|x)). The right plot is a QQ-plot of the responses (Y ) in the untreated (A = 0) vs treated (A = 1) population conditional on various covariates (X = 0, 0.5, 1). These conditional QQ-plots correspond to "slices" of the CQC surface, as shown by the coloured lines in the left plot. The plot is best viewed in colour.mand, the CQC, allows us to compare equivalent quantiles while working exclusively in the response landscape, making it a more interpretable tool. This allows us to construct canonical examples of the CQC being smoother than various nuisance parameters, the CQTE, and the CATE; adding to this interpretability. In addition, using the pseudo-outcome framework presented in Kennedy [15] , we can leverage CATE estimation procedures to estimate the CQC in a doubly robust way, as mentioned above. Crucially, the CQC can keep the valuable quantile-level information previously offered by the CQTE while building on much of the CATE literature to acquire its desirable robustness properties and interpretability. Our contributions are as follows:• Introduce a new estimand for HTE analysis: the conditional quantile comparator (CQC).• Propose an estimation procedure which we prove to be doubly robust.• Demonstrate better estimation accuracy especially when the CQC is smooth but individual conditional cumulative distribution functions are not.• Provide insights into real-world datasets on employment intervention and medical treatment. 
set-up: 
 We now introduce the standard HTE set-up in our notation. Let Z denote the random triple (Y, X, A) with Y a random variable (RV) on Y ⊆ R, X a RV on X ⊆ R d , and A a RV on {0, 1}. We treat Z as representing an individual and interpret the components as Y : Outcome/Response X : Observed covariates A : Treatment assignment Remark 1. We could view our setting as coming from a potential outcome framework [27] . Under this framework we assume there exists RVs Y 0 , Y 1 on Y representing the outcome with and without treatment and that Y ≡ Y A . Y 1-A would then be unobserved/unknown for each individual.We define the propensity score π : X → (0, 1) by π(x) := P(A = 1|X = x), in other words, π denotes the conditional probability of being assigned to treatment given the covariates. We shall assume that π is continuous and bounded away from 0 and 1. From a potential outcomes perspective, this means that each individual could potentially be assigned to either treatment. We also defineEQUATIONEQUATIONand refer to them as the conditional cumulative distribution function (CCDF) and the quantile function respectively. We also refer to F -1 a in (2) as the generalised inverse of F a . We can now define the CATE and CQTE to be given by τ : X → R and τ q : [0, 1] × X → R with τ (x) := E[Y |X = x, A = 1] -E[Y |X = x, A = 0], τ q (α|x) := F -1 1 (α|x) -F -1 0 (α|x).We letD := {Z i } 2n i=1 ≡ {(Y i , X i , A i )} 2n i=1for n ∈ N be IID copies of Z representing our data sample with i indexing the individual. We assume an even number of samples for notational convenience. For a ∈ {0, 1}, we take I a := {i|A i = 1}, the indices of individuals on treatment a. We can then define D a := {Z i } {i∈Ia} and n a := |I a | as the dataset and sample size of those on treatment a.For n ∈ N, let [n] := {1, . . . , n}. For a vector w ∈ R p let w j to represent the j th component of w and let ∥w∥ be the Euclidean norm unless otherwise specified. We also take ∥w∥ 1 as the 1-norm and ∥w∥ ∞ := max j∈[p] |w j |. We keep a summary table of all notation used in Appendix A.1.Our aim is to find "equivalent quantiles" between the treated and non-treated distributions conditional on the covariates. Specifically, for each y 0 ∈ Y, x ∈ X we aim to find y 1 such thatF 1 (y 1 |x) = F 0 (y 0 |x).This now allows us to define our primary estimand of interest, the conditional quantile comparator.Definition 1 (Conditional quantile comparator (CQC)). For our triple (Y, X, A), the conditional quantile comparator is the measurable function g * : Y × X → Y such that, for all y ∈ Y, x ∈ X ,F 1 (g * (y|x)|x) = F 0 (y|x).We then simply define y 1 as g * (y 0 |x). The name conditional quantile comparator derives from the fact that it returns the value of y 1 in the equivalent quantile of Y |X = x, A = 1 as the quantile of Y |X = x, A = 0 that y 0 is in.Remark 2. For simplicity and to ensure such a function is well defined, we will assume that Y |X = x, A = a is a continuous RV for any given x ∈ X , a ∈ {0, 1} with strictly positive density on its support. We will however allow the support of Y |X = x, A = a to vary in both x and a.We now introduce another estimand which will serve as a useful stepping stone in our estimation.Definition 2 (CCDF contrasting function). The CCDF contrasting function is defined to be hEQUATIONThis estimand allows following alternative definitions for g * which help its interpretation and estimation (detailed later). We take h * -1 representing the inverse of h * with respect to the 2 nd argument.EQUATIONThe second equality still holds if we replace the inverses in the above with generalised inverses. The equality (4) falls straight from the definition of each object and shows how we can use h * to estimate g * . Moreover, the equality (4) allows us to generalise g * to discontinuous Y (or pdfs with non-trivial 0 density regions inside the support).We specifically focus on the CQC, g * , as we feel it gives insightful information allowing comparison of the two distributions (Y |X, A = 1) and (Y |X, A = 0).The CQC allows us to compare the two distributions beyond simply a single point estimate such as that given by the CATE. This is especially valuable in cases where the two distributions differ beyond just a shift. For example, the effect of some treatments varies greatly between individuals with the same or similar covariates. An example of this is antidepressants, where some patients respond positively while others may have adverse reactions leading to a worse outcome than no treatment whatsoever. Another example is the use of opioids as painkillers where some patients have an increased tolerance making them less effective [12, 22] .As well as being of interest on its own, the quantile comparator relates closely to other estimands of interest. For example, if we take ∆ * (y 0 |x) := g * (y 0 |x) -y 0 then ∆ * tells us whether the equivalent quantile in the treated distribution is higher or lower. This estimand then serves as a heuristic for whether the treatment is beneficial at that untreated response value.Furthermore, CQTE can be written asEQUATIONlinking the quantile comparator back to the CQTE. This equivalence highlights the perspective that the CQC can be seen as rephrasing the input of the CQTE in terms of the outcome space.A key idea within CATE literature is the notion that the CATE itself may be a simpler estimand to study than the marginal treatment outcomes (E[Y |X = x, A = a]) may be individually. One can exploit this feature to improve the CATE's estimation. A similar concept exists with the CQC as we will see in the example below.Example 1 (Illustrative Example). Suppose thatY |X = x, A = 0 ∼ N(sin(10x), 1 2 ), Y |X = x, A = 1 ∼ N(2 sin(10x), 2 2 ).Then we have g * (y|x) = 2y which does not depend on x and does not include the sine term present in the individual CDFs. Interestingly,E[Y |X, A = 1] -E[Y |X, A = 0] = sin(x)hence the CATE is still non-constant in this case (the same also holds for the CQTE). Additionally, we have ∆ * (y|x) = g * (y|x) -y = y suggesting the intervention is beneficial for positive y and detrimental for negative y. We now show 3D plots of a CCDF, the CQC, and the CQTE in Figure 2 . Example 2 (General Smoothness Case). Suppose we are in the potential outcomes framework so that Y 0 , Y 1 exist with Y ≡ Y A . Now also suppose that Y 1 = ϕ(Y 0 , X) for some transformation ϕ increasing in Y 0 for each X. Then ϕ gives the CQC (i.e. ϕ = g * ) meaning that smoothness of the CQC can be seen as smoothness of ϕ. This gives a generalisation of the CATE case where smoothness is present whenY 1 = Y 0 + ψ(X) with ψ smoothA specific example could be a treatment which halves all individuals blood pressure. In this case ϕ * (y, x) = g * (y|x) = 1 2 y and so the CQC is smooth but the CQTE and CATE would not be if the individual responses are non-smooth. 
estimation procedure: 
 We now describe our estimation procedure for the CQC which is motivated by equation (4) . At a high level our approach for estimating g * (y 0 |x) will be the following:• Estimate h * (y 0 , .|x) using a pseudo-outcome -a specified proxy response computed from (Y, X, A) which we will regress against.• Find the value ŷ1 which makes our estimate of h * (y 0 , .|x) closest to 0.Section 3.1 and Algorithm 1 give our h * estimation procedure while Section 3.3 and Algorithm 2 give our g * estimation procedure.We focus on estimating h * primarily for its two nice properties:1. Similar to g * , h * can exhibit smoothness even when the individual CCDFs are not smooth.2. The estimation of h * can be re-framed as a CATE problem.The first property is important as the smoothness of h * determines the best estimation rate that can be achieved when using non-parametric regression, setting a target for our approach. In particular, smoother functions have better estimation rates. The second property is important as it gives us a method for attaining this target rate. By re-framing the estimation as a CATE problem, we can leverage existing results to build a robust estimator which achieves the target estimation accuracy rate even when the rate of estimating nuisance parameters is sub-optimal. We demonstrate this robustness later using finite sample bounds on the estimation accuracy (Proposition 1 & Theorem 2).First, we show how the estimation of h * can be solved using a CATE estimator. Note thath * (y 0 , y 1 |x) = E[1{Y ≤ y 1 }|X = x, A = 1] -E[1{Y ≤ y 0 }|X = x, A = 0].Hence, for a given y 0 , y 1 , if we define the RV W y0,y1 := 1{Y ≤ y A }, then estimating h * (y 0 , y 1 |.) is equivalent to estimating the CATE with W y0,y1 replacing Y as the response. To perform this estimation, we turn to a recent method developed by Kennedy [15] . They propose to write the CATE as a conditional expectation of a function of Z called a pseudo-outcome. A robust estimator is then obtained by regressing this pseudo-outcome against X. In our setting, the pseudo-outcome with the new response W y0,y1 , for a sample (y, x, a) is given byφ y0,y1 (y, x, a) : = a -π(x) π(x)(1 -π(x)) {1{y ≤ y a } -F a (y a |x)} + F 1 (y 1 |x) -F 0 (y 0 |x) (6) = a -π(x) π(x)(1 -π(x)){1{y ≤ y a } -F a (y a |x)} + h * (y 0 , y 1 |x).Since h * (y 0 , y 1 |x) = E[φ y0,y1 (Z)|X = x] (Proposition 5, Appendix C.1), regressing φ y0,y1 (Z) on X provides an estimate for h * .As we do not know the CDFs nor the propensity score, we need to replace them in (6) with estimates. We define π, F0 , F1 to be estimates of π, F 0 , F 1 respectively. We then construct φy0,y1 in the same way as φ y0,y1 , but using estimated quantities π, Fa instead. φy0,y1 can now serve as the pseudooutcome in our regression. We also use sample splitting to de-correlate the propensity score and CDF estimates from the h * estimate. This helps make our estimator doubly robust, as we will see in the following theory.We are now ready to define our Doubly Robust (DR)-learner to estimate h * in Algorithm 1. ) by regressing 1{A = 1}, 1{Y ≤ y 0 }, 1{Y ≤ y 1 } respectively against X. 3: Use these estimates to obtain φ(Z j ) for j ∈ D J using equation (6) 4: Using D J to regress φ against X to obtain estimate ĥ(y 0 , y 1 |.) of h * (y 0 , y 1 |.). Remark 3. Cross-fitting can be implemented by repeating the procedure with the roles of I and J switched and then averaging the two estimates of h * . We could also perform this procedure multiple times with different random splits of the data to improve our estimator's potential stability.Note that our algorithm is not specific on which form of regression to use allowing for any parametric or non-parametric procedure. Further to this, it can also be easily adapted to use other pseudooutcome procedures such as the R-learner of Nie and Wager [23] or a standard inverse propensity weighting approach which we describe in Appendix A.3.In this section, we prove the estimation accuracy of ĥ, which will play important roles in the following notation and assumptions we need for estimating g * . These accuracy statements will be made for an arbitrarily fixed x ∈ X . For our theoretical and experimental results we use linear smoothers for the final regression. This means our estimate ĥ and oracle estimate h ′ are of the form ĥ(y 0 , y1 |x) = j∈J w j φy0,y1 (Z j ) h ′ (y 0 , y 1 |x) = j∈J w j φ y0,y1 (Z j )where the weights w j ≡ w j (x, X J ) are constructed using X J := {X j } j∈J with w j ≥ 0 and ∥w∥ 1 . Linear smoothers encompass a broad class of estimation techniques used in both low and high-dimensional settings. Examples include k-NN regression [8, 9] , kernel ridge regression [29] , generalised forests [3] , and Mondrian forests [18] . Additionally linear smoothers have been shown to adapt to intrinsic low dimensionality in regression problems in higher dimensions [16] , making them an apt estimator for our purposes.For ϕ : Z → R treated as deterministic and p > 1, we also define the norms∥ϕ∥ w s := 1 ∥w s ∥ 1 i∈J w s j E [ϕ(Z) 2 |X = X i ]and take ∥ϕ∥ w := ∥ϕ∥ w 1 . Note that this norm is random as the weights depend upon X J .We now aim to show that we are able to exploit smoothness in h * even when the CCDFs and propensity score are less smooth. We introduce the notion of smoothness through Hölder functions.Definition 3 (Hölder functions). We say that a function f :X → R is (γ, C)-Hölder for γ ∈ (0, 1], C ≥ 1 if for any x ′ , x ′′ ∈ X , |f (x ′ ) -f (x ′′ )| ≤ C∥x ′ -x ′′ ∥ γ .Here, larger γ represents a smoother function which can be estimated at faster rates.Assumption 1. For any y 0 , y 1 ∈ Y, δ > 0, a ∈ {0, 1}:(a) There exists ξ ∈ (0, 1/2] such that π(x), π(x′ ) ∈ [ξ, 1 -ξ] for all x ′ ∈ X . (b) With probability at least 1 -δ, ∥ φy0,y1 -φ y0,y1 ∥ w 2 ≤ ε φ(n, δ) . (c) With probability at least 1 -δ, ∥π -π∥ w ≤ ε α (n, δ) and ∥F a (y a |.) -Fa (y a |.)∥ w ≤ ε β (n, δ). (d) For γ ∈ (0, 1], C ≥ 1, h * (y 0 , y 1 |.) is (γ, C)-Hölder.Assumption 1(a) exists to ensure for any covariate value, neither treatment assignment has too low a probability. Assumption 1(b) controls the convergence of the estimated pseudo-outcome to the true pseudo-outcome. Assumption 1(c) sets up the smoothness of the propensity score and CCDFs alongside the convergence rates of their estimators as ε α , ε β respectively. Assumption 1(d) sets up the smoothness of h * which will control the convergence rate of the oracle estimation procedure. Assumptions 1 (c) and (d) control the accuracy of our estimator in the following result.Proposition 1. Suppose that Assumption 1 holds and let ĥ be a linear smoother estimated as in Algorithm 1. Then for any y 1 , y 0 ∈ Y, δ ∈ (0, 2/e] and our x ∈ X , with probability at least 1 -δ, ĥ(y 0 , y1 |x) -h * (y 0 , y 1 |x) ≤ ε h (n, δ).Here, for each δ ∈ (0, 2/e] we haveε h (n, δ) := 2 log(8/δ)/n ε φ(n, δ/4) + ε α (n, δ/4)ε β (n, δ/4) + ε γ (n, δ/4), ε γ (n, δ) := |E[h ′ (y 0 , y 1 |x) -h * (y 0 , y 1 |x)|X J , D I ]| + 2 log(2/δ)/n ∥w∥∥φ -h(y 0 , y 1 |.)∥ w 2 +2∥w∥ ∞ log(2/δ)/(3ξ).We have that ε γ gives an upper bound on the accuracy of the oracle estimation and so acts as a target for our estimation procedure. If ε φ(n, δ) → 0 then the first term in ε h is guaranteed to be o(ε γ (n, δ)) for fixed δ. Hence the first and last terms converge at oracle rates with respect to n. As the ε α and ε β terms are multiplied together we can obtain better rates than either of them individually have. This is because both ε α and ε β can converge to 0 slower than ε γ while their product ε α • ε β converges quicker. This provides the desired double robustness as our estimation can converge at oracle rates even when convergence for the nuisance parameters is slower. Now that we have this we can convert our estimate of h * , into an estimate of g * .In order to obtain an estimate of g * (y 0 |x) at a fixed y 0 , x, we need to obtain estimates of h * (y 0 , y 1 |x) at various values of y 1 . As h * is monotonic, we would then like to search for a monotonic function which aligns with these estimates. This monotonicity is especially important because it allows us to bound the estimation accuracy of ĥ uniformly over all y 1 at a similar rate to our pointwise accuracy. With this, we can easily translate the estimation accuracy in ĥ (obtained in proposition 1) into the accuracy in ĝ. Additionally, it simplifies the process of inverting ĥ. In general our method in Algorithm 1 will not produce monotonic ĥ (this is in contrast to some other approaches such as an IPW pseudo-outcome, see Appendix A.3), or separately estimating the CCDFs). We can however obtain a monotonic estimate of h * using isotonic projection.Definition 4 (Isotonic Projection). We define the isotonic projection of α ′ ∈ R p as follows:P (α ′ ) := argmin α∈Iso(p) ∥α -α ′ ∥ where Iso(p) := {α ∈ R p |α j ≤ α l+1 ∀l ∈ [p -1]}, the set of all isotonic vectors in R p .Remark 4. We can use the Pool Adjacent Violators Algorithm (PAVA) [6] which performs isotonic projection and is implemented in the IsotonicRegression class of sci-kit learn in Python [25] .Hence, for a fixed value of (y 0 , x) and a set of predictions αl = ĥ(y 0 , y1 |x) with y (l) ≤ y (l+1) , we can take α := P Iso(p) ( α) and use these to obtain a new monotonic estimate of h * . Furthermore, by a result in Yang and Barber [32] , α will be at least as accurate as α in the worst case. We now describe our approach for estimating the CQC using this projection approach in Algorithm 2.Algorithm 2 DR estimation procedure for the CQC g * Require: Data D; test point (y 0 , x); sorted evaluation points {y (l) } p l=1 1: Apply Algorithm 1 to obtain estimate of h * given by ĥ. 2: Define αl := ĥ(y 0 , y (l) |x) for l ∈ [p]. 3: Isotonically project α using PAVA to obtain α with αi ≤ αi+1 . Remark 5. For the case where ĥ is a step function, these steps can serve as our evaluation points while for continuous ĥ one could take these candidate y 1 points at small evenly spaced intervals. Empirically we also find that ĥ is already close to isotonic and so step 3. of the algorithm is mostly for the theoretical justification of our approach.While it may seem inefficient to be estimating the CQC via the CCDF contrasting function and then inverting, due to the monotonicity of h * , we actually pay a very small cost in estimation accuracy for having to estimate the CCDF contrasting function over all y 1 . We make this notion more explicit in the following section.We now provide the accuracy of our estimate ĝ obtained by Algorithm 2 when used in conjunction with linear smoothers. We assume that Fa are also fit using linear smoothers of the formFa (y|x ′ , a) = i∈I w Fa;i (x ′ ; X I , A I )1{Y i ≤ y} with w j (x ′ ) ≡ w j (x ′ , X I , A I ) > 0, ∥w(x ′ )∥ 1 = 1.We will also require the following assumptions.Assumption 2. For our RV X, any y ∈ Y, x ′ ∈ X , δ < e -1 :(a) There exists some s, η > 0 such that F 1 (y ′ |x) ≥ η for all y ′ ∈ B s (g * (y|x)).(b) W.p. at least 1 -δ, max j∈J w j ≤ ε w (n, δ) and max i∈I w Fa;i (X) ≤ ε w (n, δ).Assumption 2(a) is a mild assumption which allows us to convert the ĥ accuracy into ĝ accuracy while Assumption 2(b) bounds the rates of decay of the weights in our linear smoothers.Theorem 2. Let ĝ be estimated as using Algorithm 2 with {Y i } i∈I1 , sorted and then used as our evaluation points and linear smoothers used for regressions in Algorithm 1. Then provided Assumptions 1 & 2 hold we have that for δ ∈ (0, e -1 ) and sufficiently large n, w.p. at least1 -δ, |ĝ(y|x) -g * (y|x)|≤ 2 η -1 ε h (n, δ/(2n)) + ξ -1 ε w (n, δ/(2n)) .From this result we see that if our weights decay at rate faster than ε h (n, δ) then this error will be dominated by the ε h term. We believe this to hold in most cases and show that it does comfortably when using Nadaraya-Watson (NW) estimation [21, 31] with a box kernel in Appendix C.4. Furthermore, if the dependence on δ in both terms is of the form log c (1/δ) for some c > 0 then we obtain the same rate of estimation as for h * up to polylog factors. This means we translate our desirable double robustness ĥ over to ĝ. We also obtain finite sample bounds onE[|ĝ(Y |x) -g * (Y |x)| | A = 0, ĝ]with high probability and present this in Appendix C.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: The two main results of the paper, Proposition 1 and Theorem 2 are proved in sections C.1 and C.2. The assumptions for these results are given in Assumptions 1 & 2 with the implications of these assumptions discussed.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
numerical experiments: 
 We now apply our approach to a series of simulated and real data scenarios in order to demonstrate the utility of our estimand and the effectiveness of our estimation procedure. For these, we use NW estimation as our regression procedure throughout. See appendix A.2 for details on NW estimation.In this section, we test our method's performance in terms of our estimator's mean absolute error under simulated scenarios. 1 In each scenario we test against a separate estimator which estimates the two CCDFs separately and simply takes their difference, an IPW pseudo-outcome estimator detailed in Appendix A.3, the CQTE estimator of Kallus and Oprescu [14] , and the oracle DR estimator where φ is replaced with φ (i.e. exact π, F a are used). In this experiment, we return back to the set-up of example 1. We now change the frequency of the sine term by takingY |X = x, A = 0 ∼ N(sin(γπx), 1 2 ), Y |X = x, A = 1 ∼ N(2 sin(γπx), 2 2 ), π(x) = 0.4 sin(γπx) + 0.5.for γ ∈ [0, 10] so that increasing γ imitates decreasing smoothness of our nuisance parameters.In our experiments half the samples are used to estimate the propensity score and CCDFs and the other half are used to regress against the pseudo-outcome. Our estimate ĝ is then compared against g * using a hold-out testing set. This process is repeated 500 times with new training data on each run. From this, a Monte-Carlo estimate ofE ĝ [E X [E Y |X,A=0 [|ĝ(Y |X) -g * (Y |X)|]]] is produced alongside 95% confidence intervals (CIs). In our first experiment, we let 2n = 1000 and vary γ in [0, 10]. In our second experiment, we let γ = 6 and 2n vary in [200, 5000] . The results of this are shown in Figure 3 . We can see that the average error decreases as the sample size 2n grows and mildly increases as γ grows. This is expected as increasing γ makes estimating the nuisance parameters more challenging.The result shows that the proposed DR method achieves the best performance compared with the Separate and IPW estimators and is only marginally outperformed by the oracle estimator. Additionally we see that the method of Kallus and Oprescu [14] is much more affected by the increase in γ. This is because unlike the CQC, the CQTE has a complexity that depends on the frequency term (γ). We also observe much better performance as the sample size increases. We hypothesise that the plateau in the CQTE approach is due to the difficulty of estimating the reciprocal of the PDF, which causes the estimation to be unstable irrespective of sample size. Further simulated experiments including 10-dimensional x and linear CQC are given in Appendix B.1.To show the performance in the real-world scenarios, we use a dataset on an employment programme which has been studied in various prior works [4, 5, 26] . Within the programme, some participants were given job placements or temporary help jobs while others received no intervention. Participants' earnings were then monitored over the next 8 quarters following their enrolments. We take their net earnings as our response (Y ) and the employment intervention as the treatment (A = 1).We use each participant's age at their entry to the study as our covariate (X). We fit our quantile comparator function on 2,000 participants. Figure 4 shows our estimate of ∆ * (y|x) = g * (y|x) -y for various values of (y, x).We see that participants around age 23 and between ages 32-37 benefit most from this scheme, as indicated by the darker colour on the heat map. Additionally, the lower quantile of the income distribution (wage ≤ $7500) shows the least change, indicating that wage improvements primarily occur for the higher income group. For participants aged 40, there appears to be little change in outcomes overall. To demonstrate our approach in a medical setting, we apply our method to evaluate the effectiveness of a colon cancer treatment, as detailed in Appendix B.3. For comparative purposes, we also provide an estimate of the CQTE for this data in Appendix B.2.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: The paper gives detailed information about the experimental set-up and explicit algorithms for methods introduced. The specific use of algorithm 1 with NW estimation is also described Appendix A.2. In addition code is provided in the supplementary materials with 'SimulatedExperiments.ipynb' reproducing all of the experiments. We also provide a link to a public github repository containig this in a footnote on page 9.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitations: 
 The theory provided here gives a strong foundation and motivation for framing our problem in this particular manner. There is however a great deal more to be explored in this area from a theoretical perspective. For example, one immediate improvement would be to give a more general case where our weight decay (in Assumption 2 (b) for Theorem 2) is sufficiently fast. In addition more work needs to be done exploring the relationship between the smoothness of g * and the smoothness h * . Smoothness in h * appears to be a stronger condition so ideally we would like to make theoretical statements directly on the smoothness of g * . Interestingly our experiments on synthetic data do seem to suggest that it is the complexity of g * which drives the estimation rate as in these experiments h * increases in complexity while g * remains constant. Additionally, changing our experiment in Section 4.1 to have uniform response so that both h * , g * are constant rather than just g * , seems to give no material improvement to the performance of our estimator (see Appendix B.1.3).Another limitation of our current estimation procedure is that it requires learning an estimand and then inverting it to obtain our final estimator. While this process is relatively simple, it could be streamlined and made more computationally efficient if we could produce a more direct estimator similar to the DR-Learner for CATE [15] or the CQTE estimator in Kallus and Oprescu [14] .Finally, while we have been able to provide more concrete examples of smoothness for the CQC these are still limited to the case of a deterministic treatment effect which we would like to expand upon. This is closely related to a more general limitation with quantile-based estimands, the CQC included, in that they lack meaningful interpretability for the individual. While the CATE can be viewed as the expected difference in an individual's outcome on and off the treatment, no such individual-level interpretation exists for the CQC or indeed any other estimand trying to learn higher level distributional information than the mean. As a result the CATE is still a more naturally interpretable estimand. To facilitate this interpretation however, one still needs to make assumptions about a lack of confounding between the treatment assignment and the potential outcomes, which are only verifiable in certain restrictive scenarios [30] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code is provided in the supplementary materials with each function documented as well as in a linked public github repository. Jupyter Notebooks are provided to reproduce all of the experiments and real world data settings used within the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper we have introduced a new treatment effect estimand, the conditional quantile comparator and demonstrated its efficacy both in terms of its doubly robust estimation, and its ability to provide valuable data insights. This is a promising direction as it allows quantile-based treatment effect exploration to "keep up" with the CATE in terms of estimation quality offering more flexibility as to which estimand can be used to best describe the data. For these reasons, we see the CQC as an exciting and worthwhile new direction within the HTE framework.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Most of the experimental details are provided alongside the experimental results in Section 4 with additional details provided in Appendix A.4. In addition code to replicate all of the experiments is provided in the supplmentary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: For our experimental results we explain that 500 replications of the entire fitting procedure (including newly generated data making each run entirely independent) was run and that 95% confidence intervals are provided for the mean absolute error of each estimator.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: A.1 Notation table Table 1: Table of notation. Notation Definition Description Y RV on Y ⊆ R Response X RV on X ⊆ R d Covariates A RV on {0, 1} Treatment Z (Y, X, A) π(x) P(A = 1|X = x) Propensity Score F a (y|x) P(Y ≤ y|X = x, A = a) Conditional CDF (CCDF) F -1 a (α|x) inf {y ∈ Y|F a (y|x) ≥ α} Conditional quantile func- tion τ (x) E[Y |X = x, A = 1] -E[Y |X = x, A = 0] Conditional average treat- ment effect (CATE) τ q (α|x) F -1 1 (α|x) -F -1 0 (α|x) Conditional quantile treat- ment effect (CQTE) D {Z i } 2n i=1 IID data sample I a {i ∈ [2n]|A i = 1} Treatment a index D a {Z i } i∈Ia n a |I a | [n] {1, . . . , n} ∥x∥ 1 d d i=1 x 2 i Euclidean Norm ∥x∥ 1 1 d d j=1 |x j | 1-norm ∥x∥ ∞ max j∈[d] |w j | ∞-norm g * (y|x) F -1 1 (F 0 (y|x)|x) Conditional quantile com- parator CQC h * (y 0 , y 1 |x) F 1 (y 1 |x) -F Y |X,0(y0|x) CCDF contrasting function ∆ * (y|x) g * (y|x) -y Quantile difference function f Estimate of f /f * φ y0,y1 (y, x, a) a-π(x) π(x)(1-π(x)) {1{y ≤ y a } -F a (y a |x)} + F 1 (y 1 |x) -F 0 (y 0 |x)The Pseudo-Outcome φy0,y1 (y, x, a) φ with π, F a replaced by π, Fa I, J {1, . . . , n}, {n + 1, . . . , 2n} Sample splitting indicesD I , D J {Z i } i∈I , {Z j } j∈J Data split w ∈ R nw j ≡ w j (x) ĥ estimation weights ĥ(y 0 , y 1 |x) j∈J w j (x) φy0,y1 (Z j )h * estimate h ′ (y 0 , y 1 |x) j∈J w j (x)φ y0,y1 (Z j ) Oracle h * estimate ∥ϕ∥ w s ∥w s ∥ -1 1 i∈J w s j E [ϕ(Z) 2 |X = X i ] ξ π(x ′ ) ∈ [ξ, 1 -ξ] for all x ′ ∈ X ε φ(n, δ) ∥ φ -φ∥ w 2 ≤ ε φ(n, δ) w.p. 1 -δ φ error bound w.h.p. ε α (n, δ) ∥π -π∥ w 2 ≤ ε α (n, δ) w.p. 1 -δ π error bound w.h.p. ε β (n, δ) ∥ Fa (y|.) -F a (y|.)∥ w 2 ≤ ε β (n, δ) w.p. 1 -δ Fa error bound w.h.p. γ Smoothness of h * (y 0 , y 1 |.) ε γ (n, δ) E[h ′ (y 0 , y 1 |x) -h * (y 0 , y 1 |x)|X J , D I ]| + 2 log(1/δ)/n ∥w∥∥φ -h(y 0 , y 1 |.)∥ w 2 + 2ξ -1 ∥w∥ ∞ ε δ (n)/3 h ′ error bound w.h.p. ε Tn (n, δ) 2 log(1/δ)/nε w (n, δ)ε φ(n, δ) T n (x) error bound w.h.p. ε h (n, δ) ε Tn (n, δ/4) + (ε α • ε β )(n, δ/4)+ ε γ (n, δ/4) ĥ error bound w.h.p.Iso(p) {α ∈ R p |α l ≤ α l+1 ∀l ∈ [p -1]} Set of isotonic vectors P Iso(p) )(α ′ ) argmin α∈Iso(p) ∥α -α ′ ∥ Isotonic projection w Fa ∈ R n w Fa;i (x) ≡ w Fa;i (x ′ , X I ) Fa estimation weights Notation Definition Description Fa (y|x ′ ) i∈J w Fa (x)1{Y j ≤ y} CCDF estimate η F 1 (y ′ |x) > η ∀, y ′ ∈ B s (g * (y|x)) Density lower bound ε w (n, δ) max{∥w(x)∥ ∞ , ∥w Fa (X)∥ ∞ } w.p. 1 -δ Variance lower bound w.h.p. Appendix Notation k(x, x ′ ) NW estimation kernel m f (x) E[f (Z)|X = x] mf (x) j∈J w j (x)f (Z j ) b(x) m φ-φ (x) ε δ (n) max{log(1/δ)/n, 1/n} [n] 0 {0, 1, . . . , n} f (∆y) lim ε↓0 f (y) -f (y -ε)Step size of f at y A.2 Nadaraya-Watson estimation Throughout, we use NW estimation as our standard non-parametric regression technique. For a kernel k :X × X → [0, ∞), IID data sample D := {(Y i , X i )} n i=1 , and x ∈ X the NW estimate of E[Y |X = x] is given by n i=1 k(x, X i ) n j=1 k(x, X j ) Y i .Applying this to our pseudo-outcome regression in Algorithm 1, we get that ĥ(y 0 , yEQUATIONEQUATION• 1{Y i ≤ y Ai } -FY |X,Ai (y Ai |X i ) + Fa (y 1 |X i ) -F0 (y 0 |X = X i )where for anyEQUATIONFa (y a |x ′ ) :EQUATIONNote that for our theoretical results we do not specify how our nuisance parameters are estimated and the results only depend on the accuracy of our estimation of the nuisance parameters.Remark 6. We can re-write (8) as:ĥ(y 0 , y 1 |x) := i∈I1 k(x, X i ) n i=1 k(x, X i ) 1 π(X i ) 1{Y i ≤ y 1 } -F1 (y 1 |X i ) + F1 (y 1 |X i ) -F0 (y 0 |X i ) - i∈I0 k(x, X i ) n i=1 k(x, X i ) 1 1 -π(X i ) 1{Y i ≤ y 0 } -F0 (y 0 |X i ) + F0 (y 0 |X i ) -F1 (y 1 |X i )Which can be helpful in terms of the practical implementation.Another pseudo-outcome one can use for estimating the treatment effect is the based on inverse propensity weighting. Specifically we can take our pseudo-outcome to beEQUATIONIf we regress against it using NW estimation, and also use NW estimation for our nuisance parameter estimation as in ( 9) & (10), our estimate ĥ will be increasing in y 1 and decreasing in y 0 meaning we do not need to perform any isotonic projection.As the method of Kallus and Oprescu [14] is one for estimating the CQTE, we transform it into an estimator of the CQC (which we denote by ĝ) using the following formula where τq (.|.) is our CQTE estimatorĝ(y|x) = τq F Y |X,0 (y|x)|x + y. using the true CCDF, F Y |X,0 .This is the inverse of equation ( 5) which defines the CQTE in terms of the CQC. Conversely we also tested transforming all our CQC estimators into CQTE estimators (using exactly equation ( 5) with ĝ replacing g * ) and testing the accuracy on this space and found similar results.Each experiment took no longer than 1 hour to run on a single 4 core CPU with 8GB of RAM.The bandwidths of the kernels for the NW estimation of the nuisance parameters were chosen by a limited grid search on additional simulated data by validating against the true value of the nuisance parameters. While this is unrealistic in practice it was done to make estimation of the nuisance parameters as strong as possible. This was to err on the side of caution as strong nuisance parameter estimation would naturally favour the baseline approaches.Code to implement our approach alongside Jupyter notebooks running our numerical experiments can be found in the supplementary materials. The code to implement the kernels is adapted from https://github.com/wittawatj/kernel-gof/ which is free to use under the MIT Licence.We run additional simulated experiments in order to further test and explore our approach. Throughout, our overall experimental set-up is the same as in Section 4.1To test our problem in higher dimensions we ran experiments where X was 10-dimensional with X uniform on [-1, 1] 10 . That is each component X j was independent with X j ∼ U (-1, 1). We then took In our first experiment we take 2n = 1000 and γ ∈ {0, 0.5, 1, 1.5, 2, 2.5, 3}. The results of this are shown in Figure 5a In our second experiment we take γ = 1 and 2n ∈ {200, 500, 1000, 2000, 5000}.Y |X = x, A = a ∼ N (sin(γπ(β ⊤ x), 1) π(x) = 0.The results of this are shown in Figure 5b .As we can see the DR approach performs the best with it performing close to oracle for low sample size and low frequency. As the frequency increases the DR estimator deviates from the oracle. In our first experiment we take 2n = 1000 and γ ∈ {0, 2, 4, 6, 8, 10}. The results of this are shown in figure 5a In our second experiment we take γ = 6 and 2n ∈ {200, 500, 1000, 2000, 5000}. The results of this are shown in Figure 5b . for γ ∈ [0, 10]. We also take the propensity to be π(x) = 0.4 sin(γπx) + 0.5. In this case h * (y 0 , y 1 |x) = 1 2 y 1 -y 0 which does not depend on x for any y 0 , y 1 ∈ Y.In our first experiment we take 2n = 1000 and γ ∈ {0, 2, 4, 6, 8, 10}. The results of this are shown in figure 5a In our second experiment we take γ = 6 and 2n ∈ {200, 500, 1000, 2000, 5000}. The results of this are shown in Figure 5b illustrative example suggesting that our method is effectively exploiting simplicity in g * rather than in h * .To further explore the interpretability of our estimator we provide an estimate of the CQTE for comparison. This can be seen in Figure 9 . Figure 9 : Surface and heat plot of the CQTE, τ * q (α|x), for our employment data with X =Age, α=Income Quantile.In these plots we can see that the interpretation of the CQTE is less immediate than for the case of the CQC. This is because, for each covariate value, the quantile value corresponds to a different untreated response making comparisons between various values of the covariates less direct. From this plot we do still see that higher income quantiles are associated with a greater increase in wages. Interestingly the value of the CQTE plummets 90% at the highest quantile for individuals of age 40. This is likely an estimation error due to the limited data at higher quantiles as and higher ages.To demonstrate the effectiveness of our approach in medical settings, we apply it to a trial on the effect of colon cancer treatment on survival time/time to remission. This dataset was originally introduced in Laurie et al. [19] and can be found in the "survival" package in R and loaded with the line data(colon, package="survival"). In this dataset 929 are randomised to either receive Placebo or Levamisole. They are then followed up for a period of up to 3329 days and the time till either death or recurrence of their cancer. For our analysis we take our response (Y ) as the time to first of death or recurrence. For simplicity, when an individual makes it to the end of a trial without an event and is censored we take that to be the time of their event. Looking at the data censoring times are mostly at around 2,000-3,000 days while over 90% of death or recurrence events that do occur, occur within 1,500 days. Therefore censored individuals will still have better responses than those who had events, as we would want. As our covariate, we took the patients' ages when joining the trial. We show a 3D plot and heat plot of the estimate of ∆(y|x) = g * (y|x) -y in Figure 10 . Interestingly from Figure 10 we see that the treatment appears to have little effect on patients aged 53-57. For the reaming patients, we can see that there is relatively little effect on the time to event for events of 400 days or less while at around 500 days the difference in time to event jumps to 1000 days. This jump takes the time to event (TTE) to the time when censoring begins. This gives evidence that rather than the treatment delaying the time to event, it increases the proportion of the patients who have no event during the trial essentially fully treating those patients for the duration of the trial. This shows the value of the CQC it was able to reveal meaningful information about the treatment beyond simply its positive effect. On top of this, we are able to identify the censoring within our results without explicitly controlling for it in any way.C.1 Proof of ĥ accuracy We first define some additional notation. For an output f which depends upon z = (y, x, a) definem f (x) = E[f (Z)|X = x] and mf (x) = j∈J w j f (Z j ),with w j defined as before so that m φ(x) is our estimator and mφ (x) is the oracle estimator.b(x) := m φ-φ (x),in other words the bias in our estimate of the pseudo-outcome for a given x. We will also work with mb (x) where we view b as being a function of z.Finally for n ∈ N, δ ∈ (0, 1), defineε δ (n) := log(2/δ)/n if δ ∈ (0, 2/e] 1/n otherwise..). For δ ∈ (0, 1) we have that with probability at leastEQUATION+ 2ε δ (n)∥w∥∥φ -h(y 0 , y 1 |.)∥ w 2 + 2∥w∥ ∞ ε δ (n) 3ξ =: B(φ).Moreover, with probability at least 1 -δ we haveEQUATIONProof. To prove (12) we first observe that since w is D I , X J -measurable we haveE[ mφ (x) 2 |D I , X J ] -E[ mφ (x)|D I , X J ] 2 = j∈J w 2 j E[(φ(Z j ) -E[φ(Z)|X = X j ]) 2 |D I , X J ]Note also that we have max j∈J |w j {φ(Z j ) -E[φ(Z j )|D I , X J ]}|≤ ∥w∥ ∞ 2ξ -1 almost surely. Note also that {φ(Z j )} j ∈ J are conditionally independent given D I , X J . Hence, the bound (12) follows from two applications Bernstein's inequality Boucheron et al. [7, Theorem 2.10] , applied conditionally on D I , X J .Next we note that by the triangle inequality we have|E{ m φ(x) -m φ (x), | D I , X J }| ≤ |E{ m φ(x) -mφ (x) | D I , X J }| + |E{ mφ (x) -m φ (x) | D I , X J }| ≤ j∈J w j E [ φ(Z j ) -φ(Z j )|D I , X J ] + |E{ mφ (x) -m φ (x) | D I , X J }| = ∥ mb ∥ w,1 +|E{ mφ (x) -m φ (x) | D I , X J }| Moreover, since E[ φ(Z j )|D I , X J ] is the projection of φ(Z j ) onto the subspace of D I , X J - measureable functions we have ∥w∥ 2 2 ∥ m φ -m φ(x)∥ 2 w 2 = j∈J w 2 j E{( φ(Z) -E{ φ(Z)|X = X j }) 2 |D I , X J } ≤ i∈[n] w 2 j E{( φ(Z j ) -E{φ(Z j )|D I , X J }) 2 |D I , X J } = i∈[n] w 2 j E({( φ(Z j ) -φ(Z j )) + (φ(Z j ) -E{φ(Z j )|D I , X J })} 2 |D I , X J ) = i∈[n] w 2 j (E{( φ(Z j ) -φ(Z j )) 2 |D I , X J } + E{(φ(Z j ) -E{φ(Z j )|D I , X J }) 2 |D I , X J }) = ∥w∥ 2 2 {(∥ m φ -mφ ∥ 2 w 2 +∥ mφ -m φ ∥ 2 w 2 } ≤ ∥w∥ 2 2 {(∥ m φ -mφ ∥ w 2 +∥ mφ -m φ ∥ w 2 } 2 .Hence, to deduce (13) we apply the first bound with φ in place of φ to obtain the following bound with probability at least 1 -δ,| mφ (x) -m φ (x)| ≤|E{ m φ(x) -m φ (x) | D I , X J }| + 2ε δ (n) ∥w∥ 2 ∥ m φ -m φ∥ w 2 + 2ξ -1 ∥w∥ ∞ ε δ (n)/3 ≤∥ mb ∥ w,1 +|E{ mφ (x) -m φ (x) | D I , X J }| + 2ε n (δ)∥w∥ 2 {(∥ m φ -mφ ∥ w 2 +∥ mφ -m φ ∥ w 2 } + 2ξ -1 ∥w∥ ∞ ε δ (n)/3 = B(φ) + ∥ mb ∥ w,1 + 2ε δ (n) ∥w∥ 2 ∥ m φ -mφ ∥ w 2 =B + ( φ),as required.We apply the following result from Kennedy [15] . Proposition 4 (Proposition 2 from Kennedy [15] ). Suppose that b(x) = b1 (x) b2 (x) then| mb (x)| = ∥w∥ 1 ∥ b1 ∥ w ∥ b1 ∥ w .Proof. Follows from the Cauchy-Schwartz inequality.Proposition 5. Our pseudo-outcome is conditionally unbiased,E[φ y0,y1 (Z)|X = x] = h * (y 0 , y 1 |x).Proof. We haveE[φ y0,y1 (Z)|X = x, A = 1] = 1 π(x) E[1{Y ≤ y 1 }|X = x, A = 1] -F 1 (y 1 |x) + h * (y 0 , y 1 |x) = h * (y 0 , y 1 |x),and similarly E[φ y0,y1 (Z)|X = x, A = 1] = h * (y 0 , y 1 |x). The result now follows by the law of total expectation.Proof of Proposition 1. Fix y 0 , y 1 , x Let m φ(x) be the regression of X against the estimated pseudo-outcome φy0,y1 (Z) evaluated at x (i.e. ĥ(y 0 , y 1 |x)), let mφ be the same but with the estimated pseudo-outcome replaced with the exact pseudo-outcome φ y0,y1 . Finally takem φ (x) := E[φ y0,y1 (Z)|X = x] = P(Y ≤ y 1 |X = x, A = 1) -P(Y ≤ y 1 |X = x, A = 0).Theorem 3 gives us that with probability. at least 1 -δ| m φ(x) -m φ (x)| ≤ ε γ (n, δ) + | mb (x)|+ 2ε δ (n)∥w∥ 2 ∥ φ -φ∥ w 2 ≤ ε γ (n, δ) + | mb (x)|+ 2ε δ (n)∥ φ -φ∥ w 2 with b((y, x, a)) := E[ φ(Z) -φ(Z)|X = x].To bound b we have that from Proposition 5,E[φ y0,y1 (Z)|X = x] = P(Y ≤ y 1 |X = x, A = 1) -P(Y ≤ y 0 |X = x, A = 0).Additionally, by splitting over the events {A = 1}, {A = 0} and noting thatP(A = 1|X = x) = π(x) we have E[ φy0,y1 (Z)|X = x] = π(x) π(x) P(Y ≤ y 1 |X = x, A = 1) -P(Y ≤ y 1 |X = x, A = 1) - 1 -π(x) 1 -π(x) P(Y ≤ y 0 |X = x, A = 0) -P(Y ≤ y 0 |X = x, A = 0) + P(Y ≤ y 1 |X = x, A = 1) -P(Y ≤ y 0 |X = x, A = 0). Hence b(x) = π(x) π(x) -1 P(Y ≤ y 1 |X = x, A = 1) -P(Y ≤ y 1 |X = x, A = 1) - 1 -π(x) 1 -π(x) -1 P(Y ≤ y 0 |X = x, A = 0) -P(Y ≤ y 0 |X = x, A = 0) .We have that using Proposition 4| mb (x)| ≤ ∥w∥ 1 π π -1 w P(Y ≤ y 1 |X, A = 1) -P(Y ≤ y 1 |X, A = 1) w + 1 -π 1 - π -1 w P(Y ≤ y 0 |X, A = 0) -P(Y ≤ y 0 |X, A = 0) w ≤ 1 ξ 1 a=0 ∥π -π∥ w P(Y ≤ y a |X, A = a) -P(Y ≤ y a |X, A = a) w .Now it is simply a matter of bounding all the relevant terms which we do through the following eventsE oracle := | m φ -m φ |≤ | mφ (x) -m φ (x)|+| mb (x)|+ 2ε δ (n)∥w∥ 2 ∥ φ -φ∥ w 2 E φ := {∥ φ -φ∥ w 2 ≤ ε φ(n, δ/4)} E γ := { mφ -m φ ≤ ε γ (n, δ/4)} E α := ∥π -π∥ w,2 ≤ ε α (n, δ/4) E β,0 := ∥ P(Y ≤ y 0 |X, A = 0) -P(Y ≤ y 0 |X, A = 0)∥ w,2 ≤ g β (n, δ/4) E β,1 := ∥ P(Y ≤ y 1 |X, A = 1) -P(Y ≤ y 1 |X, A = 1)∥ w,2 ≤ ε β (n, δ/4)According to our assumptions and previous results,E oracle , E φ, E γ , E α ∩ E β,0 ∩ E β,1 separately occur w.p. at least 1 -δ/4.Then by the union bound, the intersection of all these events holds w.p. at least 1 -δ and under these events| m φ -mφ + mφ -m φ | ≤ ε Tn (n, δ/4) + 2 ξ ε α (n, δ/4)ε β (n, δ/4) + ε γ (n, δ/4)ε w (n, δ/4)Where the first term comes from Proposition 3, the second from Proposition 4 and the final term from smoothness of g * .C.2 Proof of ĝ accuracy Proposition 6 (Maximum step-size bound). Suppose that both the outer regression and estimation of CCDFs is fit using linear smoothers so that ĥ(y 0 , y1 |x) = j∈J w j (x; X J ) φ(Z j ) Fa (y|x ′ ) = i∈I w Fa;i (x ′ ; X J )1{Y i ≤ y} with w j (x), w i (x ′ ) Additionally suppose with probability at least 1 -δ max max j∈J w j (x), max i∈I w Fa;i (X) ≤ ε w (n, δ).Then with probability at least 1 -δEQUATION|x), and{Y (i) } i∈I1 is the sorted version of {Y i } i∈I1 .Proof. We immediately have via union bounds that with probability at least 1 -δ max max j∈J w j (x), max j∈J w Fa;i (X j ) ≤ ε w (n, δ/n). Now we can try to bound the jump size of our function. To do so for a step function f : Y → R and a step points y ∈ Y we definef (∆y) := lim ε↓0 f (y) -f (y -ε),i.e. the size of the step at y. From the definition of our pseudo-outcome, the jumps occur in different forms at {Y j } {j∈J ,Aj =1} , {Y (i)} {i∈I|A (i) =1} . Note that we are assuming Y continuous so these points are all distinct a.s. . For Y j with j ∈ J and A j = 1. We have ĥ(y 0 , ∆Y j |x) = w φ,j 1 π(X j )1{Y j ≤ ∆Y j } ≤ w φ,j ξ .For i ∈ I with A (i) = 1, we have that| ĥ(y 0 , ∆Y i |x)| = j∈J w j -1{A j = 1} 1 π(X j ) F1 (∆Y i |X j ) + F1 (∆Y i |X j ) ≤ 1 ξ max j∈J F1 (∆Y i |X j ) ≤ 1 ξ max j∈J w Fa;i (X j )1{Y i ≤ ∆Y i } = 1 ξ max j∈J w Fa;i (X j ).Hence the maximum jump size is less thanEQUATIONw Fa;i (X j ) .Therefore combining this with our previous bounds gives that with probability at least 1 -δmax i∈[n1+1]0 | ĥ(y 0 , Y (i) |x) -ĥ(y 0 , Y (i+1) |x)|≤ ξ -1 ε w (n, δ/n) F 1We also have a result ensuring the step size of the projected function is even smaller. Proof. We will prove this by first giving an algorithm to compute the projection. Defineψ l (α) := α if α l ≤ α l+1 α 1 , . . . , α l +α l+1 2 , α l +α l+1 2 , α m if α l > α l + 1. Now define α (0,0) = α, α (n,i) = ψ i (α (n,i-1) ) for i ∈ [p -1] and α (n,0) = α (n-1,p-1) for n ∈ N.Finally define α (t) = α (⌊t/p⌋,mod(t,p) . Then by Yang and Barber [32] we know that lim t→∞ α (t) = P Iso(p) (α).Now if |α (n,l+1) l+1 -α (n,l+1) l |> |α (n,l) l+1 -α (n,l) l |. Then α (n,l+1) l+1has be moved by ψ l+1 . Hence α(n,l+1) l+1 < α (n,l)l+1 . For this to increase the distance then we must haveα (n,l+1) l+1 < α (n,l+1) l . Therefore as α (n+1,l-1) l ≥ α n,l+1 l , we have that α (n+1,l) l+1 = α (n+1,l) l .By a similar (simpler) argument |α(n,l-1) l -α (n,l-1) l+1 |> |α (n,l-2) l -α (n,l-2) l+1 | then α (n,l) l = α (n,l) l+1 .Hence if any iteration moves adjacent points further apart, a later iteration will make them equal meaning that on convergence they will be equal. Hence we have proved our claim.We now have the result which justifies our choice to take the isotonic projection of the data which is take from Yang and Barber [32] .Proposition 8 (Theorem 1 from Yang and Barber [32] ). Let z * , ẑ ∈ R p with z * ∈ Iso(p). Then for z := P Iso(p) ( ẑ),max l |z * l -zl |≤ max l |z * l -ẑl |.We now use this isotonic projection to obtain supremum bounds on the accuracy of our estimator.Proposition 9 (Supremum bound on ĥ accuracy). Fix x ∈ X , y ∈ y 0 and let ĥ be our original estimate of h * . For m ∈ N, take {Y (l) } m l=1 to be a potentially random set of points in Y in increasing order. Now define α ∈ R m by α l := ĥ(y 0 , Y (l) |x) and α := P Iso(m) (α). Finally, define h to be the piecewise constant right continuous function with h(y 0 , Y (l) |x) := αl . Suppose for any y 1 ∈ Y, n ∈ N, and δ, δ ′ > 0 thatP h(y 0 , y 1 |x) -h * (y 0 , y 1 |x) ≥ ε h (n, δ) ≤ δ P max l∈[m]0 ĥ(y 0 , Y (l) |x) -ĥ(y 0 , Y (l+1) |x) ≥ ε step (n, m, δ) ≤ δwhere we take [m] 0 := [m] ∪ {0}, ĥ(y 0 , Y (0) |x) := -1, and ĥ(y 0 , Y (m+1) |x) := 1.δ, δ ′ > 0, n ∈ N, P sup y1∈Y ĥ(y 0 , y 1 |x) -h * (y 0 , y 1 |x) ≤ ε h (n, δ/m) + ε step (n, m, δ ′ ) ≥ 1 -δ -δ ′ . Proof. For each l ∈ [m] define the event E h,l := ĥ(y 0 , Y (l) 1 |x) -h * (y 0 , Y (l) 1 |x) ≤ ε h (n, δ/m)and take E h := n l=1 E h,l . Additionally define the eventE step := max l∈[m]0 ĥ(y 0 , Y (l) |x) -ĥ(y 0 , Y (l+1) |x) < ε step (n, m, δ)Then E h and E step hold w.p.s at least 1 -δ and 1 -δ ′ respectively. Also under E h , E step , by Propositions 7 & 8, we have thatmax l∈[m] h(y 0 , Y (l) 1 |x) -h * (y 0 , Y (l) 1 |x) ≤ ε h (n, δ/m) max l∈[m]0 h(y 0 , Y (l) |x) -h(y 0 , Y (l+1) |x) < ε step (n, m, δ)We then have that for any y 1 ∈ Y there exists i ∈ [m] such that y(l-1) 1 ≤ y 1 ≤ y (l)1 . Hence by monotonicity, we have the following 2 inequalities ĥ(y 0 , y1 |x) -h * (y 0 , y 1 |x) ≥ h(y 0 , Y (l-1) 1 |x) -h * (y 0 , Y (l) 1 |x) Λ1 ĥ(y 0 , y 1 |x) -h * (y 0 , y 1 |x) ≤ h(y 0 , Y (l) 1 |x) -h * (y 0 , Y (l-1) 1 |x) Λ2 .Λ 1 = h(y 0 , Y (l-1) 1 |x) -h * (y 0 , Y (l) 1 |x) + h(y 0 , Y (l) 1 |x) -h * (y 0 , Y (l) 1 |x) ≥ -ε h (n, δ) -ε step (n, m, δ ′ ).With the final inequality coming from E h,l-1 and our definition of h. By a similar argument we get from E h,l that Λ 2 ≤ ε h (n, δ) + ε step (n, m, δ ′ ). Again we can use the same approach to get that underE h,l-1 , E h,l , E step -ε h (n, δ) -ε step (n, m, δ ′ ) ≤ h * (y 0 , y 1 |x) -h(y 0 , y 1 |x) ≤ ε h (n, δ) + ε step (n, m, δ ′ ).Hence for our specificy 1 under E h,l , E h,l-1 , E step h * (y 0 , y 1 |x) -h(y 0 , y 1 |x) ≤ ε h (n, δ) + ε step (n, m, δ ′ )Now as this inequality holds for arbitrary y 1 under E h ∩ E step (an event which not depend on our choice of y 1 ) and the intersection of these two events holds w.p. at least 1 -δ -δ ′ by the union bound we have our result.Our final key result before we can piece them all together will be to obtain accuracy in g from our accuracy in h * Theorem 10 (Single point accuracy bound). Fix x, y 0 , assume that h, h are strictly monotonic in y 1 and suppose thatsup y1 |h * (y 0 , y 1 |x) -h(y 0 , y 1 |x)|< ε.Additionally let α := F 0 (y 0 |x) and assume that f Y |X,A=1 (y 1 |x) > η for all y 1 ∈ F -1 1 (B ε (α)|x) where here we are taking F -1 1 (A|x) to be the pre-image in Y of the set A for fixed x. Then|g * (y 0 |x) -ĝ(y 0 |x)|≤ 2ε η ,where g * (y 0 |x) is the unique y 1 such that h * (y 0 , y 1 |x) = 0.Proof. Let y * 1 := g * (y 0 |x) and ŷ1 := ĝ(y 0 |x) so that F 1 (y * 1 |x). From our accuracy assumption on ĥ in the set-up of the Theorem we have|F 1 (y * 1 |x) -F 1 (ŷ 1 |x)| = |h * (y 0 , y * 1 |x) -h * (y 0 , ŷ1 |x)| = |h * (y 0 , ŷ1 |x)|≤ |h * (y 0 , ŷ1 |x) -ĥ(y 0 , ŷ1 |x)|+| ĥ(y 0 , ŷ1 |x)| ≤ |h * (y 0 , ŷ1 |x) -ĥ(y 0 , ŷ1 |x)|+| ĥ(y 0 , y * 1 |x)| ≤ |h * (y 0 , ŷ1 |x) -ĥ(y 0 , ŷ1 |x)|+| ĥ(y 0 , y *1 |x) -h * (y 0 , y * 1 |x)| ≤ 2ε,where the second and fifth line come from the definition of y * 1 and the fourth from the definition of ŷ1 .If we define ∂ k f to be the derivative with respect to the k th argument of f then we have|ĝ(y 0 |x) -g * (y 0 |x)| = |F -1 1 (F 1 (y * 1 |x)|x) -F -1 1 (F 1 (ŷ 1 |x)|x)| = F1(ŷ1|x) F1(y * 1 |x) ∂ 1 F -1 1 (β|x)dβ = F1(ŷ1|x) F1(y * 1 |x) 1 f ((F -1 1 (β|x)|x) dβ ≤ 2ε max y1∈(y * 1 ,ŷ1) 1 f (y 1 |x) ≤ 2ε η .By Markov's inequality,P(P(A|D) > t) ≤ 1 t E[P(A|D)] = 1 t E[P(A|Y 0 )] by Fubini's Theorem < 1 t E[δ 0 ] = δ 0 t . Now define B := {P(A|D) ≤ t} so that P(B) = 1 -δ0 t . Then under B E[l(Y 0 , D)|D] = E[1 A l(Y 0 , D)|D] + E[l(Y 0 , D)|A c , D]P(A c |D) ≤ E[1 A l(Y 0 , D)|D] + ε(n, δ 0 ) by definitions of A ≤ E[1 A |D] 1/q E[l(Y 0 , D) p |D] 1/p + ε(n, δ 0 ) by Holder's inequality ≤ t 1/q E[l(Y 0 , D) p |D] 1/p t + ε(n, δ 0 )As we are assuming B.Corollary 12. Assume that F 1 (y|x) > η for all y 1 ∈ Y. Then, for our fixed x ∈ X and δ ∈ (0, e -1 ), w.p. at least,1 - δ diam(Y) 2 (η -1 ε h (n, δ/2n) + ξ -1 ε w (n, δ/n))) , E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≤ 4 η -1 ε h (n, δ/n) + 2ξ -1 ε w (n, δ/n))where ε h (δ, n) := ε Tn (n, δ/4) + ε α (n, δ/4)ε β (n, δ/4) + ε γ (n, δ/4).In particular for 2 η -1 ε h (n, δ/n) + 2ξ -1 ε w (n, δ/n)) ≤ log(e 1 (n)/δ) a /e 2 (n). For δ < 1/e we have that w.p. at least1 -δ E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≤ log(2 diam(Y)e 2 (n)e 1 (n)/δ) a /e 2 (n)Proof. Plugging the result of theorem 2 into proposition 11, noting that |ĝ(y 0 |x) -g * (y|x)|≤ diam(Y) and takingt = 2 diam(Y) η -1 ε h (n, δ/(2n)) + ξ -1 ε w (n, δ/n))yields that w.p. at least1 - δ diam(Y) 2 (η -1 ε h (n, δ/n) + ξ -1 ε w (n, δ/n))) , E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≤ 4 η -1 ε h (n, δ/n) + 2ξ -1 ε w (n, δ/n)) . Following from this if 2 η -1 ε h (n, δ/n) + 2ξ -1 ε w (n, δ/n)) ≤ log(e 1 (n)/δ) α /e 2 (n).Then we haveP E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≥ 2 log(e 1 (n)/δ) a /e 2 (n) ≤ δ diam(Y)e 2 (n) log(e 1 (n)/δ) a for δ < 1/e we have δ < 1/e ⇒ δ < exp -( 1 2 ) 1/a ⇔ log(1/δ) a > 1 2 ⇒ log(e 1 (n)/δ) a > 1 2 ⇒ 2δ log(e 1 (n)/δ) a > δ ⇔ 2δ > δ log(e 1 (n)/δ) a . Hence P E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≥ 2 log(e 1 (n)/δ) a /e 2 (n) ≤ 2δ diam(Y)e 2 (n). Finally δ ′ = 2δ diam(Y)e 2 (n) gives P E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≥ 2 log(2 diam(Y)e 1 (n)e 2 (n)/δ ′ ) a /e 2 (n) ≤ δ ′ .We aim to show that the box kernel satisfies some of our conditions. Specifically conditions 2 & 3 from Proposition 1. We first start by bounding the step size.Proposition 13 (Effective sample size). Suppose that for our x ∈ X there exists C 0 , r 0 > 0 such that for any r ∈ (0, r 0 )P(X ∈ B r (x)) ≥ C 0 r d .Now for r ∈ (0, r 0 ) take our kernel to be k r (x, x ′ ) := 1{∥x -x ′ ∥≤ r}. Then w.p. at least 1 - This gives our desired result.δ j∈J 1{X j ∈ B r (x)} ≥ nC 0 r d -2nC 0 r d log(1/δ) -log(1/δ)/3) -1 . Proof. We have 1{X j ∈ B r (x)} ≤ 1 and E[1{X j ∈ B r (x)} 2 ] = E[1{X j ∈ B r (x)} 2 ] = P(X j ∈ B r (x)) = C 0Note that j∈J 1{X j ∈ B r (x)} is also the effective sample size of our estimation as it is the number of samples used in the average. Now that we have the effective sample size result in terms of our kernel radius r, we need to obtain the optimal rate of decay of r for our estimation.Proposition 14 (NW estimation with box kernel). let mf (x) be the NW estimation of m f (x) := E[f (Z)|X = x] using IID copies (Z i ) n i=1 of Z. and assume |f (Z)| ≤ M . For a fixed r ∈ (0, r), use kernel k r as defined above and suppose the same assumptions hold. Suppose that m f (x) is α smooth for α ≤ 1 (i.e. α-Holder continuous.) Then for sufficiently large n depending on C 0 , α and δ ≤ 2e -1 w.p. at least 1 -δ,| mf (x) -m f (x)|≤ 2M + 1 C 0 log(2/δ)n -1 2+d/α .Proof. With our kernel define I r := {i ∈ [n]|k r (x, X i ) = 1} and n r = |I r |. Now define the eventE n := {n r ≥ C 0 nr d -2C 0 r d n log(2/δ) -log(2/δ)/3}.Then by Proposition 13 this event occur w.p. at least 1-δ/2. Now if we defineε i := f (Z i )-m f (X i ) then we have E[ε i |X i ] = 0. Also, we have | mf (x) -m f (x)| = 1 n r i∈Ir f (Z i ) -m f (x) = 1 n r i∈Ir m f (X i ) + ε i -m f (x) ≤ 1 n r i∈Ir |m f (X i ) -m f (x)| + 1 n r i∈Ir ε i ≤ r α + 1 n r i∈Ir ε i .With the final equality coming by our smoothness condition and definition of I r . Now by Hoeffding bounds we have thatP   i∈Ir ε i ≥ 2 log(4/δ)M 2 n r   ≤ δ 2 .Hence by combining this event and E n then we have that with probability at least 1 -δ This in turn gives| mf (x) -m f (x)| ≤ r α + 2 log(2/δ)M 2 n r ≤ r α + 2 log(2/δ)M 2| mf (x) -m f (x)| ≤ r α + 2 log(2/δ)M C 0 nr d .Then the optimal choice of r is such thatr α = 1 √ nr d ⇔ r α+ d 2 = n -1 2 ⇔ r = n -1 2α+d ⇔ r α = n -1 2+d/α = 1 √ nr d .Hence plugging this in we get that for sufficiently large n depending on C 0 , α we have that for δ ≤ 2e -1 w.p. at least 1 -δ| mf (x) -m f (x)|≤ 2M + 1 C 0 log(2/δ)n -1 2+d/α .Proposition 15 (Final weight decay). Suppose that for our x ∈ X there exists C 0 , r 0 > 0 such that for any r ∈ (0, r)P(X ∈ B r (x)) ≥ C 0 r d .Suppose that we are performing NW estimation of an α smooth function at points x ∈ X using kernel k rn (x, x ′ ) := 1{∥x -x ′ ∥≤ r} with r n decaying optimally. Now definew j := k rn (x, X j ) j ′ ∈J k rn (x, X j ′ )the weight of the j th component. Then with probability at least 1 -δmax j∈J w j ≤= log(2/δ) C 0 n -2 2+d/α .Proof. We havemax j∈J w j ≤ 1 j∈J k(x, X j ) = 1 j∈J 1{X j ∈ B r (x)}.Hence by proposition 13 we have w.p. at least 1 -δmax j∈J w j ≤ C 0 nr d -2C 0 nr d log(1/δ) -log(1/δ)/3) -1.We know from Proposition 14 that the optimal radius decay gives 1 nr d = n -2 2+d/γ . Plugging this into our result gives thatmax j∈J w j ≤ C 0 n 2 2+d/γ -2C 0 log(1/δ)n 1 2+d/γ -log(1/δ)/3) -1 ≤ log(1/δ) C 0 n 2 2+d/γFor sufficiently large n depending on γ, C 0 .Note that for a γ smooth function the MSE is C -1 n -1 2+d/γ . Hence the box kernel comfortably satisfies our weight decay condition 2 in Assumptions 2.Additionally now if we have an α smooth function and assume that for any x ′ ∈ X there exists C 0 (x ′ ) such that for all r ∈ (0, r 0 ) P(X ∈ B r (x ′ ) ≥ C 0 r d . Then we have that w.p. at least 1 -δ, w Fa;i (X,X I ) ≤ log(2/δ) C 0 (X) n -2 2+d/α .Thus if we assume that there exists C ′′ > 0 such that w.p. at least 1 -δ, 1 C0(X) ≤ C ′′ log(1/δ) then we get that w.p. at least 1 -δ w Fa;i (X,X I ) ≤ C ′′ log 3/2 (3/δ)n -2 2+d/α .This then gives our condition 2 in Assumptions 2.We now try to boundj∈J w 2 j σ(Xj ) E[ j∈J w 2 j σ(Xj )] .Corollary 16 (Accuracy under NW estimation with box kernel). Suppose that our linear smoother is NW estimation with the box kernel and optimally decaying radius additionally assume that:• ε φ(n, δ) = e φ(n) log(1/δ) with e 1 = o(1). • ε α (n, δ) = C α log(1/δ)n -1 2+d/α . • ε β (n, δ) = C β log(1/δ)n -1 2+d/β . • β > d 2(1+d/γ). Then for any δ, for sufficiently large n the following events each separately hold w.p. at least 1 -δ, ĥ(y 0 , y 1 |x) -h * (y 0 , y Proof. For any NW estimator we have that ∥w∥ 1 , ∥w∥ 2 ≤ 1 a.s. meaning we can take ε w (n, δ) ≡ 1.1 |x) ≤ C h log 3/2 (1/δ) e h (n) |ĝ(y|x) -g * (y|x)| ≤ C g ηξ log 3/2 (n/δ) e h (n) E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≤ C g,2 diam(Y) ξηWe also have that ε γ (n, δ) = C γ log(1/δ)n -1 1+d/γ . Hence ε Tn = 2ε δ (n)e φ(n) log(1/δ). This then gives ĥ(y 0 , y 1 |x) -h * (y 0 , y1 |x) ≤ C γ log 1/2 (7/δ)n -1 2+d/γ + C α C β log(7/δ)n -( 1 2+d/α + 1 2+d/β ) + 2ε δ/4 (n)e φ(n) log(7/δ) ≤ C h log(1/δ)n -1 2+d/γ + log(1/δ)n -( 1 2+d/α + 1 2+d/β ) ≤ C h log(1/δ) e h (n) for δ < e -1 and C h = 7(C γ + C α C β + √ 2)giving our first result.Using our weight decay results for NW estimation and plugging into Proposition 6 we get thatε h-step = C ′′ ξ -1 log 3/2 e h (n) (n/δ).Hence plugging into Theorem 2 givesEQUATION+ 4C ′′ giving our second result. Finally by Corollary 12 for δ < e -1 w.p. at least1 -δ, E |ĝ(Y |x) -g * (Y |x)| A = 0, ĝ ≤ C g diam(Y) ξη log 3/2 (ne h (n)/δ) e h (n) .• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There are no new datasets released in our work. Our models use standard nonparametric regression techniques and adapt these to best estimate our estimand, the CQC meaning there is no real risk of misuse beyond the misuse of any regression technique.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets are publicly available and properly referenced. Proprietary code used is also open source and used in accordance with its licence. The licence for this code can also be found in code itself. The model we use is our own and so no licensing is required.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Code implementation can be found at: github.com/joshgivens/ConditionalOutcomeEquivalenceNote that both ε Tn and ε h-step are actually both o(ε γ (n, c)), thus if we were allowed to take n sufficiently large depending upon δ then we would have all log 3/2 terms replaced with log terms and the dependence on C ′ , C ′′ removed. 


Decision: This work proposes a new estimand for comparing the quantiles between control and treatment groups. The Conditional Quantile Comparator (CQC) maps an outcome from the control group to an outcome in the treatment group, such that they represent the same conditional quantiles in their respective distributions. The paper is motivated by the shortcomings of the Conditional Quantile Treatment Effect (CQTE), which is not robust to errors in the quantiles. 

The authors propose an doubly-robust estimation procedure which estimates a conditional contrasting function that examines the difference between conditional CDFs, and then performs isotonic regression. The contrasting function is estimated using pseudo-outcome regression, which provides double robustness. 

Several experts reviewed the paper and raised concerns on interpretability and lack of appropriate comparisons to prior work. The authors have addressed these issues during the discussion period, which should be incorporated in the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: The authors propose a new estimand called the Conditional Quantile Comparator (CQC), which computes the quantiles of the outcome distributions of the treatment effect, hence improving on the (usual) conditional average treatment effect. The CQC is defined as a measurable function that maps an untreated outcome to the equivalent treated outcome in the same quantile, conditional on covariates. The authors introduce a doubly robust estimation procedure for CQC, where the estimation is re-framed as a CATE problem using the pseudo-outcome framework. The authors present finite sample bounds on the estimator's error, and provide numerical simulations and result on a real employment dataset.
strength_weakness: strength_weakness: First of all, I'd like to say I am not very familiar with the causal inference literature, so some of my comments might not be as relevant.

- Complexity of using quantiles: can the authors comment in terms of error bounds and convergence rates what is the price to pay to estimate quantiles as opposed to CATE? Some of these results, including the use of pseudo-outcomes and isotonic projections, appear a little bit unusual, as usually quantiles are estimated through the pinball loss. As usual what is important is assessing the left tail or right tail (i.e., whether the treatment effect crosses the zero or not), I would expect the right/left tail quantiles to be the most important ones -- is the error in estimating all the quantiles the same across the entire distribution?

- Dimensionality of the input $\textbf{x}$: both the simulation and the real dataset use a very low-dimensional $\textbf{x}$, and the comments on using the Nadaraya-Watson estimator and kernel smoothing are really only applicable when X is generally low-dimensional. How does in practice your method behave as function of the input dimension $\textbf{x}$, and would your comments still be applicable even in higher-dimensional settings?

- Estimating quantiles vs entire distribution: how does this approach compare to other approaches that estimate the entire conditional density, such as [1]? Is the use of a doubly robust framework guarantee the optimal error rates?


[1] Zhou, T., Carson IV, W. E., & Carlson, D. (2022). Estimating potential outcome distributions with collaborating causal networks. Transactions on machine learning research, 2022.
questions: questions: See the weaknesses section above.
limitations: limitations: The authors do address some limitations and implications of their approach.
paper_summary: paper_summary: The authors propose the Conditional Quantile Comparator (CQC), a function that maps an outcome from the control group in a binary treatment setting to an outcome in the treatment group, such that they represent the same conditional quantiles in their respective distributions. The estimation procedure consists of two stages: first, estimating a conditional contrasting function that examines the difference between conditional CDFs, followed by isotonic regression. The contrasting function is estimated using pseudo-outcome regression, which confers the algorithm with double-robust properties in finite samples. This estimator addresses some of the shortcomings of the Conditional Quantile Treatment Effect (CQTE), which is not robust to errors in the quantiles. The CQC is supported by a theoretical analysis of finite sample convergence rates under smoothness assumptions, as well as by empirical simulations.
strength_weakness: strength_weakness: * The motivation for the estimator is somewhat weak.  Estimators for CQTEs with double-robustness properties have been already proposed (see [1, 2]). These estimators have a second order dependence on the rate of quantile estimation, as desired. The authors should contextualize their work with respect of exiting literature (which seems to overall be missing in the paper) and compare their algorithm with the existing techniques. 
* Moreover, the pseudo-outcome  estimation technique in [1] is somewhat simpler (see their Appendix B) since they only require one pseudo-outcome regression. There seem to be some tradeoffs, e.g. the CQC requires a two stage procedure that includes a prost processing step (the isotonic regression), as well as for Algorithm 1 to be run several times for each value of $y_1$ considered (because the conditional CDFs need to be estimated at each value). On the other hand, the robust CQTE algorithm from [1] require estimating the conditional PDF at several points. Overall, the CQC seems to be a more computationally (and statistically) intensive estimation procedure and the authors should at least discuss the tradeoffs. 
* Furthermore,  [1] provides rates for more general classes a functions outside of the linear smoothers framework in [3]. I suggest the authors try to provide similar guarantees in order to generalize their results beyond linear smoothers which, to the best of my knowledge, are not often used in practice. This would be particularly useful since the rate results in section 3.4 require all nuisances to be estimated using linear smoothers. 
* Empirical evaluations should also include these existing methods. 

Overall, I tend towards soft rejection. This estimator might be useful in its own right, but the authors should contextualize and compare with existing work. I am willing to update my score upon further discussion.  

[1] Kallus, Nathan, and Miruna Oprescu. "Robust and agnostic learning of conditional distributional treatment effects." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.

[2] Leqi, Liu, and Edward H. Kennedy. "Median optimal treatment regimes." arXiv preprint arXiv:2103.01802 (2021).

[3] Kennedy, Edward H. "Towards optimal doubly robust estimation of heterogeneous causal effects." Electronic Journal of Statistics 17.2 (2023): 3008-3049.
questions: questions: See weaknesses section.
limitations: limitations: The authors have adequately adressed the limitations of their work.
paper_summary: paper_summary: The paper studies an important problem -- treatment effects often have distributional effects and considering conditional mean outcomes is often not the correct objective for evaluating prescriptive interventions. One popular approach to consider heterogeneous effects across a population which allows researchers to determine treatment effects at different quantiles of the study distribution to determine how treatments might effect the worst-off populations. Unfortunately, as the authors note obtaining quantile treatment effects often relies strict smoothness assumptions on the marginal quantile functions and lacks the double robustness properties of CATE estimators. To this end, the authors propose a new estimand to evaluate heterogenous treatment effects that they show too be doubly robust and apply this estimator to both simulational and real-world data.
strength_weakness: strength_weakness: - I understand that the CQTE can be written as a function of $\Delta^*$ but this does not help me to understand how I should interpret this quantity in relation to the typical CQTE. The authors indicate that this it is a "rephrasing" but and provide the mapping. but its not obvious how this translates in actual examples. Since the authors are proposing a non-standard estimand they should provide more details linking this to the standard analysis. One place where this would be very beneficial is in the interpretation of the real-world data sets. I consider this to be the main weakness of the work because without a clear interpretation for the estimand practioners will not adopt this methodology. Without this I doubt the impact of the paper beyound showcasing a technical deficiency in the CQTE estimand due to smoothness requirements.
- I found the algorithms a little difficult to understand especially because of the reindexing where you duplicate the data. This seems purely notational and seems like you could simplify the notation here.
- The method does not require smoothness of the marginal quantiles, however h must be Holder-smooth. Certainly in many cases smoothness of the marginal may effect smoothness of h. Since the reason for introducing this method is non-smoothness of the marginal quantiles it might be useful to expand on the class of problems where this is an easier problem beyound the example given in Example 1. How does this interact with smoothness of g?
- To that end, could you also provide real-world examples where you might expect the marginals to be poorly behaved but the CQC to be well be haved?
- Finally, in the the final paper in paper it would be good to have the CQTE estimates in the real world examples so you can draw a contrast between the methods.
questions: questions: In remark 3 you mention you use a cross-fitting method to improve stability, why is this the case? Duplicating the data yields non-independence?
How do i interpret the earnings for groups with the highest/lowest benefits in Figure 4?
limitations: limitations: The authors have addressed some limitations of their work but have not acknowledged possible negative societal impact. From my perspective, work that emphasizes the importance of considering distributional effects is generally societally beneficial. The main limitation that the authors address are primarily on the interplay between the assumption on h and if they could be relaxed to assumptions on g. However, there are several other assumptions and limitations that should be made clear in this work. The authors should potentially consider including limitations in their analysis of the two real-world data sets which might give practitioners a better understanding of how to use this method. Examples of what could be added are given in the weakness section especially with regards to interpretability of results.
paper_summary: paper_summary: The paper introduces a new estimator to obtain conditional quantiles of a treatment effect and argues that it is superior to the pre-existing "conditional quantile treatment effect" (CQTE) estimator. The main difference is that accurate CQTE estimates require accurate conditional quantile models even though the latter can be hard to fit, while the introduced CQC estimator is more robust to the CDF estimators it relies on.
strength_weakness: strength_weakness: In 4.1 you test against estimating the CCDFs separately and taking their difference. Do you mean estimating the inverse CDFs / quantile functions and taking their difference i.e. CQTE? If not, what exactly is the procedure here and why not compare against the difference of two quantile estimators.

CCDFs are arguably harder to estimate than quantile functions in my experience (they require the additional 'feature' y, have to work with classification losses instead of regression losses, and are more prone to monotonicity violations). So even if the final estimator here is more robust to bad CCDF estimation I'm not convinced it will necessarily be better in practice to the CQTE approach of fitting quantile functions. 

This is especially true with modern day neural network based quantile models. NW estimators are unlikely to be as good as those on larger / higher dimension datasets, and it's not clear how the theory would hold in cases like that.

Isotonization can be risky and create large flat zones for $\hat{g}$. Did you notice any in your experiments? Are there other ways to get monotonicity here, similar to the various strategies to get monotonicity in multi-quantile regression?
questions: questions: See above
limitations: limitations: Yes

Comments: comment: Thanks for the response! Appreciate the extra info and I feel more confident in my already positive score.
Thank you for the reply. We will make sure to include the comparisons with Kallus et al (empirically, methodologically, and theoretically) in the final paper.
Thanks for your response. My concernes have been addressed. I have raised my score with the expectation that the authors will include this discussion (especially the comparisons to Kallus et al. (2023)) in their final version.
Thank you for the reply. We will make sure to include a further discussion on the overall estimation difficulty of the CCDF when compared to quantile estimation in the final paper.
I thank the authors for their comments. In particular, please include the discussion about the complexities of using quantiles in the paper, as it was not clear to me that inverting the CCDF is actually not that bad in terms of an idea in practice and in terms of error rate. Due to the author points in this response and in the global rebuttal, I increase my score within the limits of my understanding and experience with the subject.