Title: Conditional Outcome Equivalence: A Quantile Alternative to CATE
Venue: NeurIPS 2024
Accepted: yes
Keywords: Heteregenous Treatment Effect

Conditional Average Treatment Effect

Conditional Quantile Treatment Effect

Quantile Regression
Abstract: The conditional quantile treatment effect (CQTE) can provide insight into the effect of a treatment beyond the conditional average treatment effect (CATE). This ability to provide information over multiple quantiles of the response makes the CQTE especially valuable in cases where the effect of a treatment is not well-modelled by a location shift, even conditionally on the covariates. Nevertheless, the estimation of the CQTE is challenging and often depends upon the smoothness of the individual quantiles as a function of the covariates rather than smoothness of the CQTE itself. This is in stark contrast to the CATE where it is possible to obtain high-quality estimates  which have less dependency upon the smoothness of the nuisance parameters when the CATE itself is smooth. Moreover, relative smoothness of the CQTE lacks the interpretability of smoothness of the CATE making it less clear whether it is a reasonable assumption to make. We combine the desirable properties of the CATE and CQTE by considering a new estimand, the conditional quantile comparator (CQC). The CQC not only retains information about the whole treatment distribution, similar to the CQTE, but also having more natural examples of smoothness and is able to leverage simplicity in an auxiliary estimand. We provide finite sample bounds on the error of our estimator, demonstrating its ability to exploit simplicity. We validate our theory in numerical simulations which show that our method produces more accurate estimates than baselines. Finally, we apply our methodology to a study on the effect of employment incentives on earnings across different age groups. We see that our method is able to reveal heterogeneity of the effect across different quantiles.

Conditional Outcome Equivalence: A Quantile Alternative to CATE

Abstract

The conditional quantile treatment effect (CQTE) can provide insight into the effect of a treatment beyond the conditional average treatment effect (CATE). This ability to provide information over multiple quantiles of the response makes the CQTE especially valuable in cases where the effect of a treatment is not wellmodelled by a location shift, even conditionally on the covariates. Nevertheless, the estimation of the CQTE is challenging and often depends upon the smoothness of the individual quantiles as a function of the covariates rather than smoothness of the CQTE itself. This is in stark contrast to the CATE where it is possible to obtain high-quality estimates which have less dependency upon the smoothness of the nuisance parameters when the CATE itself is smooth. Moreover, relative smoothness of the CQTE lacks the interpretability of smoothness of the CATE making it less clear whether it is a reasonable assumption to make. We combine the desirable properties of the CATE and CQTE by considering a new estimand, the conditional quantile comparator (CQC). The CQC not only retains information about the whole treatment distribution, similar to the CQTE, but also having more natural examples of smoothness and is able to leverage simplicity in an auxiliary estimand. We provide finite sample bounds on the error of our estimator, demonstrating its ability to exploit simplicity. We validate our theory in numerical simulations which show that our method produces more accurate estimates than baselines. Finally, we apply our methodology to a study on the effect of employment incentives on earnings across different age groups. We see that our method is able to reveal heterogeneity of the effect across different quantiles.

1 Introduction

In many real world scenarios such as personalised treatment allocation and individual level policy decisions, understanding the effect of a treatment/intervention at an individual level is invaluable in providing bespoke care. The field which aims to understand a treatment's effect given certain covariates is referred to as heterogeneous treatment effect (HTE) estimation and has seen popularity across many applications [11, 10, 24, 20]. Within HTE, the conditional average treatment effect (CATE) has proved itself to be a popular target of study in this area due to its simplicity and interpretability [2, 13, 28]. A key limitation of the CATE however, is that it fails to paint a full picture of the differences between distributions of the two responses. In addition, it can be sensitive to outliers, with extreme values leading to a biased outcome. As such, the conditional quantile treatment effect (CQTE), an estimand which compares the conditional quantiles of the distributions in the treated and untreated populations, has established itself as a popular alternative [1, 5, 26].

While the CQTE offers more information than the CATE and is more robust to outliers, it lacks some of the CATE's desirable estimation properties. Specifically, CQTE estimation involves estimating the quantile functions for the two marginal outcomes. This harms the estimation procedure in cases where estimation of marginal quantile functions is more challenging than estimation of the CQTE itself. An example of this is when the marginal quantile functions are less smooth as a function of the covariates than the CQTE. This aligns with a recurring idea within HTE estimation that the effect of a treatment may be simpler than the marginal outcomes. In contrast to the CQTE, there are many CATE estimators which aim to learn the CATE directly allowing them to exploit its relative simplicity. These include the X-learner [17], R-learner [23], and Doubly Robust (DR) learner [15]. Before estimating the CATE, these procedures require estimation of intermediary estimands (nuisance parameters) which condition on the covariates such as the average marginal outcomes and the propensity score (the probability of being assigned to treatment group). These nuisance parameters are then used to aid the estimation of the CATE. With the DR learner specifically, it has been shown that it can still achieve optimal convergence rates even when estimation of the nuisance parameters is worse than estimation of the CATE itself. This notion is referred to as double robustness , as our estimation is robust to sub-optimal estimation of both of the nuisance parameters.

Some attempts have been made to improve CQTE estimation [34, 33] with a key work being that of Kallus and Oprescu [14]. In this they provide an extension of the double robustness property to the CQTE, creating an estimation procedure that can achieve strong convergence even when nuisance parameters are more difficult to estimate. Unfortunately, one of the nuisance parameters which must be estimated is the reciprocal of the conditional densities over the response. These are highly difficult to estimate and risk the errors blowing up in low density regions which could potentially nullify the desirable estimation rates they achieve even with the dependence on the estimation accuracy of these nuisance parameters being less strong. Furthermore it is still unclear how one can interpret relative smoothness in the CQTE compared to the individual quantiles with their being relatively little discussion of this within the literature. In general there is a distinct lack of illustrative examples; which are present for the CATE. To our knowledge no other works specifically aim to tackle this double robustness phenomenon for the CQTE or other quantile based treatment effect estimands.

We introduce a novel estimand called the 'conditional quantile comparator' (CQC). The CQC gives the outcome for a treated individual in the same quantile as a given outcome for an untreated individual, conditional on covariates. This relates to the conditional Quantile-Quantile (QQ) plot for the treated and untreated outcomes as demonstrated in Figure 1. Similarly to the CQTE, our new esti-

0.0

0.2

0.4

0.6

0.8

1.0

x

0.0

0.2

0.4

0.6

0.8

1.0

y

0.0

0.2

0.4

0.6

0.8

1.0

g

*

(

y

|

x

)

0.0

0.2

0.4

0.6

0.8

1.0

Y

|

X

=

x

,

A

=0

Untreated Response

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Treated Response

Y

|

X

=

x

,

A

=1

g

*

(

y

|

x

=0)

g

*

(

y

|

x

=0.5)

g

*

(

y

|

x

=1)

x=0

x=0.5

x=1

mand, the CQC, allows us to compare equivalent quantiles while working exclusively in the response landscape, making it a more interpretable tool. This allows us to construct canonical examples of the CQC being smoother than various nuisance parameters, the CQTE, and the CATE; adding to this interpretability. In addition, using the pseudo-outcome framework presented in Kennedy [15], we can leverage CATE estimation procedures to estimate the CQC in a doubly robust way, as mentioned

above. Crucially, the CQC can keep the valuable quantile-level information previously offered by the CQTE while building on much of the CATE literature to acquire its desirable robustness properties and interpretability. Our contributions are as follows:

Introduce a new estimand for HTE analysis: the conditional quantile comparator (CQC).

Propose an estimation procedure which we prove to be doubly robust.

Demonstrate better estimation accuracy especially when the CQC is smooth but individual conditional cumulative distribution functions are not.

Provide insights into real-world datasets on employment intervention and medical treatment.

2 Set-up

Wenowintroduce the standard HTE set-up in our notation. Let Z denote the random triple ( Y, X, A ) with Y a random variable (RV) on Y ⊆ R , X a RV on X ⊆ R d , and A a RV on { 0 , 1 } . We treat Z as representing an individual and interpret the components as

Y

:

Outcome/Response

X : Observed covariates

A : Treatment assignment

Remark 1. We could view our setting as coming from a potential outcome framework [27]. Under this framework we assume there exists RVs Y 0 , Y 1 on Y representing the outcome with and without treatment and that Y ≡ Y A . Y 1 -A would then be unobserved/unknown for each individual.

We define the propensity score π : X → (0 , 1) by

in other words, π denotes the conditional probability of being assigned to treatment given the covariates. We shall assume that π is continuous and bounded away from 0 and 1 . From a potential outcomes perspective, this means that each individual could potentially be assigned to either treatment. We also define

and refer to them as the conditional cumulative distribution function (CCDF) and the quantile function respectively. We also refer to F -1 a in (2) as the generalised inverse of F a .

We can now define the CATE and CQTE to be given by τ : X → R and τ q : [0 , 1] ×X → R with

We let D := { Z i } 2 n i =1 ≡ { ( Y i , X i , A i ) } 2 n i =1 for n ∈ N be IID copies of Z representing our data sample with i indexing the individual. We assume an even number of samples for notational convenience. For a ∈ { 0 , 1 } , we take I a := { i | A i = 1 } , the indices of individuals on treatment a . We can then define D a := { Z i } { i ∈ I a } and n a := | I a | as the dataset and sample size of those on treatment a .

For n ∈ N , let [ n ] := { 1 , . . . , n } . For a vector w ∈ R p let w j to represent the j th component of w and let ∥ w ∥ be the Euclidean norm unless otherwise specified. We also take ∥ w ∥ 1 as the 1 -norm and ∥ w ∥ ∞ := max j ∈ [ p ] | w j | . We keep a summary table of all notation used in Appendix A.1.

2.1 Introducing the quantile comparator

Our aim is to find 'equivalent quantiles" between the treated and non-treated distributions conditional on the covariates. Specifically, for each y 0 ∈ Y , x ∈ X we aim to find y 1 such that

This now allows us to define our primary estimand of interest, the conditional quantile comparator .

Definition 1 (Conditional quantile comparator (CQC)) . For our triple ( Y, X, A ) , the conditional quantile comparator is the measurable function g ∗ : Y × X → Y such that, for all y ∈ Y , x ∈ X ,

We then simply define y 1 as g ∗ ( y 0 | x ) . The name conditional quantile comparator derives from the fact that it returns the value of y 1 in the equivalent quantile of Y | X = x , A = 1 as the quantile of Y | X = x , A = 0 that y 0 is in.

Remark 2. For simplicity and to ensure such a function is well defined, we will assume that Y | X = x , A = a is a continuous RV for any given x ∈ X , a ∈ { 0 , 1 } with strictly positive density on its support. We will however allow the support of Y | X = x, A = a to vary in both x and a .

We now introduce another estimand which will serve as a useful stepping stone in our estimation.

Definition 2 (CCDF contrasting function) . The CCDF contrasting function is defined to be h ∗ : Y × Y × X → [ -1 , 1] given by

This estimand allows following alternative definitions for g ∗ which help its interpretation and estimation (detailed later). We take h ∗-1 representing the inverse of h ∗ with respect to the 2 nd argument.

The second equality still holds if we replace the inverses in the above with generalised inverses. The equality (4) falls straight from the definition of each object and shows how we can use h ∗ to estimate g ∗ . Moreover, the equality (4) allows us to generalise g ∗ to discontinuous Y (or pdfs with non-trivial 0 density regions inside the support).

2.2 Exploring the CQC

Wespecifically focus on the CQC, g ∗ , as we feel it gives insightful information allowing comparison of the two distributions ( Y | X,A = 1 ) and ( Y | X,A = 0 ).

The CQC allows us to compare the two distributions beyond simply a single point estimate such as that given by the CATE. This is especially valuable in cases where the two distributions differ beyond just a shift. For example, the effect of some treatments varies greatly between individuals with the same or similar covariates. An example of this is antidepressants, where some patients respond positively while others may have adverse reactions leading to a worse outcome than no treatment whatsoever. Another example is the use of opioids as painkillers where some patients have an increased tolerance making them less effective [12, 22].

As well as being of interest on its own, the quantile comparator relates closely to other estimands of interest. For example, if we take ∆ ∗ ( y 0 | x ) := g ∗ ( y 0 | x ) -y 0 then ∆ ∗ tells us whether the equivalent quantile in the treated distribution is higher or lower. This estimand then serves as a heuristic for whether the treatment is beneficial at that untreated response value.

Furthermore, CQTE can be written as

linking the quantile comparator back to the CQTE. This equivalence highlights the perspective that the CQC can be seen as rephrasing the input of the CQTE in terms of the outcome space.

A key idea within CATE literature is the notion that the CATE itself may be a simpler estimand to study than the marginal treatment outcomes ( E [ Y | X = x , A = a ] ) may be individually. One can exploit this feature to improve the CATE's estimation. A similar concept exists with the CQC as we will see in the example below.

Example 1 (Illustrative Example) . Suppose that

Then we have g ∗ ( y | x ) = 2 y which does not depend on x and does not include the sine term present in the individual CDFs. Interestingly, E [ Y | X,A = 1] -E [ Y | X,A = 0] = sin( x ) hence the CATE is still non-constant in this case (the same also holds for the CQTE). Additionally, we have ∆ ∗ ( y | x ) = g ∗ ( y | x ) -y = y suggesting the intervention is beneficial for positive y and detrimental for negative y . We now show 3D plots of a CCDF, the CQC, and the CQTE in Figure 2.

-1.0

-0.5

0.0

0.5

1.0

x

-3

-2

-1

0

1

2

3

y

0.2

0.4

0.6

0.8

FY

|

X

=

x

, 0

(a)

F

Y

|

X,

0

(

y

|

x

)

-1.0

-0.5

0.0

0.5

1.0

x

-3

-2

-1

0

1

2

3

y

-3

-2

-1

0

1

2

3

g

*

(

y

∣

x

)

(b)

g

∗

(

y

|

x

)

-1.0

-0.5

0.0

0.5

1.0

x

0.0

0.2

0.4

0.6

0.8

1.0

α

-3

-2

-1

0

1

2

3

τq

(

α

|

x

)

(c)

τ

∗

q

(

α

|

x

)

Example 2 (General Smoothness Case) . Suppose we are in the potential outcomes framework so that Y 0 , Y 1 exist with Y ≡ Y A . Now also suppose that Y 1 = ϕ ( Y 0 , X ) for some transformation ϕ increasing in Y 0 for each X . Then ϕ gives the CQC (i.e. ϕ = g ∗ ) meaning that smoothness of the CQC can be seen as smoothness of ϕ . This gives a generalisation of the CATE case where smoothness is present when Y 1 = Y 0 + ψ ( X ) with ψ smooth

A specific example could be a treatment which halves all individuals blood pressure. In this case ϕ ∗ ( y, x ) = g ∗ ( y | x ) = 1 2 y and so the CQC is smooth but the CQTE and CATE would not be if the individual responses are non-smooth.

3 Estimation procedure

We now describe our estimation procedure for the CQC which is motivated by equation (4). At a high level our approach for estimating g ∗ ( y 0 | x ) will be the following:

Estimate h ∗ ( y 0 , . | x ) using a pseudo-outcome - a specified proxy response computed from ( Y, X, A ) which we will regress against.

Find the value ˆ y 1 which makes our estimate of h ∗ ( y 0 , . | x ) closest to 0 .

Section 3.1 and Algorithm 1 give our h ∗ estimation procedure while Section 3.3 and Algorithm 2 give our g ∗ estimation procedure.

3.1 Estimating the CCDF contrasting function h ∗

We focus on estimating h ∗ primarily for its two nice properties:

Similar to g ∗ , h ∗ can exhibit smoothness even when the individual CCDFs are not smooth.

The estimation of h ∗ can be re-framed as a CATE problem.

The first property is important as the smoothness of h ∗ determines the best estimation rate that can be achieved when using non-parametric regression, setting a target for our approach. In particular, smoother functions have better estimation rates. The second property is important as it gives us a method for attaining this target rate. By re-framing the estimation as a CATE problem, we can leverage existing results to build a robust estimator which achieves the target estimation accuracy rate even when the rate of estimating nuisance parameters is sub-optimal. We demonstrate this robustness later using finite sample bounds on the estimation accuracy (Proposition 1 & Theorem 2).

First, we show how the estimation of h ∗ can be solved using a CATE estimator. Note that

Hence, for a given y 0 , y 1 , if we define the RV W y 0 ,y 1 := 1 { Y ≤ y A } , then estimating h ∗ ( y 0 , y 1 | . ) is equivalent to estimating the CATE with W y 0 ,y 1 replacing Y as the response. To perform this estimation, we turn to a recent method developed by Kennedy [15]. They propose to write the CATE as a conditional expectation of a function of Z called a pseudo-outcome. A robust estimator

is then obtained by regressing this pseudo-outcome against X . In our setting, the pseudo-outcome with the new response W y 0 ,y 1 , for a sample ( y, x , a ) is given by

Since h ∗ ( y 0 , y 1 | x ) = E [ φ y 0 ,y 1 ( Z ) | X = x ] (Proposition 5, Appendix C.1), regressing φ y 0 ,y 1 ( Z ) on X provides an estimate for h ∗ .

As we do not know the CDFs nor the propensity score, we need to replace them in (6) with estimates. We define ˆ π, ˆ F 0 , ˆ F 1 to be estimates of π, F 0 , F 1 respectively. We then construct ˆ φ y 0 ,y 1 in the same way as φ y 0 ,y 1 , but using estimated quantities ˆ π, ˆ F a instead. ˆ φ y 0 ,y 1 can now serve as the pseudooutcome in our regression. We also use sample splitting to de-correlate the propensity score and CDF estimates from the h ∗ estimate. This helps make our estimator doubly robust, as we will see in the following theory.

We are now ready to define our Doubly Robust (DR)-learner to estimate h ∗ in Algorithm 1.

Algorithm 1 DR estimation procedure for the CCDF contrasting function h ∗

Require: y 0 , y 1 ∈ Y , Data D , a regressor (e.g. linear smoother)

1: Define I := [ n ] , J := { n +1 , . . . , 2 n } and split D into D I := { Z i } i ∈I , D J := { Z j } j ∈J .

2: Using D I to estimate ˆ π, ˆ F 0 ( y 0 | . ) , ˆ F 1 ( y 1 | . ) by regressing 1 { A = 1 } , 1 { Y ≤ y 0 } , 1 { Y ≤ y 1 } respectively against X .

3: Use these estimates to obtain ˆ φ ( Z j ) for j ∈ D J using equation (6)

4: Using D J to regress ˆ φ against X to obtain estimate ˆ h ( y 0 , y 1 | . ) of h ∗ ( y 0 , y 1 | . ) .

Remark 3. Cross-fitting can be implemented by repeating the procedure with the roles of I and J switched and then averaging the two estimates of h ∗ . We could also perform this procedure multiple times with different random splits of the data to improve our estimator's potential stability.

Note that our algorithm is not specific on which form of regression to use allowing for any parametric or non-parametric procedure. Further to this, it can also be easily adapted to use other pseudooutcome procedures such as the R-learner of Nie and Wager [23] or a standard inverse propensity weighting approach which we describe in Appendix A.3.

3.2 Finite sample bound of h ∗ estimator

In this section, we prove the estimation accuracy of ˆ h , which will play important roles in the following notation and assumptions we need for estimating g ∗ . These accuracy statements will be made for an arbitrarily fixed x ∈ X . For our theoretical and experimental results we use linear smoothers for the final regression. This means our estimate ˆ h and oracle estimate h ′ are of the form

where the weights w j ≡ w j ( x , X J ) are constructed using X J := { X j } j ∈J with w j ≥ 0 and ∥ w ∥ 1 . Linear smoothers encompass a broad class of estimation techniques used in both low and high-dimensional settings. Examples include k-NN regression [8, 9], kernel ridge regression [29], generalised forests [3], and Mondrian forests [18]. Additionally linear smoothers have been shown to adapt to intrinsic low dimensionality in regression problems in higher dimensions [16], making them an apt estimator for our purposes.

For ϕ : Z → R treated as deterministic and p > 1 , we also define the norms

and take ∥ ϕ ∥ w := ∥ ϕ ∥ w 1 . Note that this norm is random as the weights depend upon X J .

We now aim to show that we are able to exploit smoothness in h ∗ even when the CCDFs and propensity score are less smooth. We introduce the notion of smoothness through Hölder functions.

Definition 3 (Hölder functions) . We say that a function f : X → R is ( γ, C ) -Hölder for γ ∈ (0 , 1] , C ≥ 1 if for any x ′ , x ′′ ∈ X ,

Here, larger γ represents a smoother function which can be estimated at faster rates.

Assumption 1. For any y 0 , y 1 ∈ Y , δ > 0 , a ∈ { 0 , 1 } :

(a) There exists ξ ∈ (0 , 1 / 2] such that π ( x ) , ˆ π ( x ′ ) ∈ [ ξ, 1 -ξ ] for all x ′ ∈ X .

(b) With probability at least 1 -δ , ∥ ˆ φ y 0 ,y 1 -φ y 0 ,y 1 ∥ w 2 ≤ ε ˆ φ ( n, δ ) .

(c) With probability at least 1 -δ , ∥ ˆ π -π ∥ w ≤ ε α ( n, δ ) and ∥ F a ( y a | . ) -ˆ F a ( y a | . ) ∥ w ≤ ε β ( n, δ ) .

(d) For γ ∈ (0 , 1] , C ≥ 1 , h ∗ ( y 0 , y 1 | . ) is ( γ, C ) -Hölder.

Assumption 1(a) exists to ensure for any covariate value, neither treatment assignment has too low a probability. Assumption 1(b) controls the convergence of the estimated pseudo-outcome to the true pseudo-outcome. Assumption 1(c) sets up the smoothness of the propensity score and CCDFs alongside the convergence rates of their estimators as ε α , ε β respectively. Assumption 1(d) sets up the smoothness of h ∗ which will control the convergence rate of the oracle estimation procedure. Assumptions 1 (c) and (d) control the accuracy of our estimator in the following result.

Proposition 1. Suppose that Assumption 1 holds and let ˆ h be a linear smoother estimated as in Algorithm 1. Then for any y 1 , y 0 ∈ Y , δ ∈ (0 , 2 /e ] and our x ∈ X , with probability at least 1 -δ ,

Here, for each δ ∈ (0 , 2 /e ] we have

We have that ε γ gives an upper bound on the accuracy of the oracle estimation and so acts as a target for our estimation procedure. If ε ˆ φ ( n, δ ) → 0 then the first term in ε h is guaranteed to be o ( ε γ ( n, δ )) for fixed δ . Hence the first and last terms converge at oracle rates with respect to n . As the ε α and ε β terms are multiplied together we can obtain better rates than either of them individually have. This is because both ε α and ε β can converge to 0 slower than ε γ while their product ε α · ε β converges quicker. This provides the desired double robustness as our estimation can converge at oracle rates even when convergence for the nuisance parameters is slower. Now that we have this we can convert our estimate of h ∗ , into an estimate of g ∗ .

3.3 Estimating the conditional quantile comparator g ∗

In order to obtain an estimate of g ∗ ( y 0 | x ) at a fixed y 0 , x , we need to obtain estimates of h ∗ ( y 0 , y 1 | x ) at various values of y 1 . As h ∗ is monotonic, we would then like to search for a monotonic function which aligns with these estimates. This monotonicity is especially important because it allows us to bound the estimation accuracy of ˆ h uniformly over all y 1 at a similar rate to our pointwise accuracy. With this, we can easily translate the estimation accuracy in ˆ h (obtained in proposition 1) into the accuracy in ˆ g . Additionally, it simplifies the process of inverting ˆ h . In general our method in Algorithm 1 will not produce monotonic ˆ h (this is in contrast to some other approaches such as an IPW pseudo-outcome, see Appendix A.3), or separately estimating the CCDFs). We can however obtain a monotonic estimate of h ∗ using isotonic projection.

Definition 4 (Isotonic Projection) . We define the isotonic projection of α ′ ∈ R p as follows:

where Iso( p ) := { α ∈ R p | α j ≤ α l +1 ∀ l ∈ [ p -1] } , the set of all isotonic vectors in R p .

Remark 4. We can use the Pool Adjacent Violators Algorithm (PAVA) [6] which performs isotonic projection and is implemented in the IsotonicRegression class of sci-kit learn in Python [25].

Hence, for a fixed value of ( y 0 , x ) and a set of predictions ˆ α l = ˆ h ( y 0 , y ( l ) 1 | x ) with y ( l ) ≤ y ( l +1) , we can take ˜ α := P Iso( p ) ( ˆ α ) and use these to obtain a new monotonic estimate of h ∗ . Furthermore, by a result in Yang and Barber [32], ˜ α will be at least as accurate as ˆ α in the worst case. We now describe our approach for estimating the CQC using this projection approach in Algorithm 2.

Algorithm 2 DR estimation procedure for the CQC g ∗

Require: Data D ; test point ( y 0 , x ) ; sorted evaluation points { y ( l ) } p l =1

1: Apply Algorithm 1 to obtain estimate of h ∗ given by ˆ h .

2: Define ˆ α l := ˆ h ( y 0 , y ( l ) | x ) for l ∈ [ p ] .

3: Isotonically project ˆ α using PAVA to obtain ˜ α with ˜ α i ≤ ˜ α i +1 .

4: Take ˆ g ( y 0 | x ) := y ( l ∗ ) with l ∗ := argmin l ∈ [ p ] | ˜ α l | .

Remark 5. For the case where ˆ h is a step function, these steps can serve as our evaluation points while for continuous ˆ h one could take these candidate y 1 points at small evenly spaced intervals. Empirically we also find that ˆ h is already close to isotonic and so step 3. of the algorithm is mostly for the theoretical justification of our approach.

While it may seem inefficient to be estimating the CQC via the CCDF contrasting function and then inverting, due to the monotonicity of h ∗ , we actually pay a very small cost in estimation accuracy for having to estimate the CCDF contrasting function over all y 1 . We make this notion more explicit in the following section.

3.4 Finite sample bound of the CQC estimator

We now provide the accuracy of our estimate ˆ g obtained by Algorithm 2 when used in conjunction with linear smoothers. We assume that ˆ F a are also fit using linear smoothers of the form

with w j ( x ′ ) ≡ w j ( x ′ , X I , A I ) > 0 , ∥ w ( x ′ ) ∥ 1 = 1 . Wewill also require the following assumptions.

Assumption 2. For our RV X , any y ∈ Y , x ′ ∈ X , δ < e -1 :

(a) There exists some s, η > 0 such that F 1 ( y ′ | x ) ≥ η for all y ′ ∈ B s ( g ∗ ( y | x )) .

(b) W.p. at least 1 -δ , max j ∈J w j ≤ ε w ( n, δ ) and max i ∈I w F a ; i ( X ) ≤ ε w ( n, δ ) .

Assumption 2(a) is a mild assumption which allows us to convert the ˆ h accuracy into ˆ g accuracy while Assumption 2(b) bounds the rates of decay of the weights in our linear smoothers.

Theorem 2. Let ˆ g be estimated as using Algorithm 2 with { Y i } i ∈ I 1 , sorted and then used as our evaluation points and linear smoothers used for regressions in Algorithm 1. Then provided Assumptions 1 & 2 hold we have that for δ ∈ (0 , e -1 ) and sufficiently large n , w.p. at least 1 -δ ,

From this result we see that if our weights decay at rate faster than ε h ( n, δ ) then this error will be dominated by the ε h term. We believe this to hold in most cases and show that it does comfortably when using Nadaraya-Watson (NW) estimation [21, 31] with a box kernel in Appendix C.4. Furthermore, if the dependence on δ in both terms is of the form log c (1 /δ ) for some c > 0 then we obtain the same rate of estimation as for h ∗ up to polylog factors. This means we translate our desirable double robustness ˆ h over to ˆ g . We also obtain finite sample bounds on E [ | ˆ g ( Y | x ) -g ∗ ( Y | x ) | | A = 0 , ˆ g ] with high probability and present this in Appendix C.3.

4 Numerical experiments

We now apply our approach to a series of simulated and real data scenarios in order to demonstrate the utility of our estimand and the effectiveness of our estimation procedure. For these, we use NW estimation as our regression procedure throughout. See appendix A.2 for details on NW estimation.

4.1 Simulated experiment

In this section, we test our method's performance in terms of our estimator's mean absolute error under simulated scenarios. 1 In each scenario we test against a separate estimator which estimates the two CCDFs separately and simply takes their difference, an IPW pseudo-outcome estimator detailed in Appendix A.3, the CQTE estimator of Kallus and Oprescu [14], and the oracle DR estimator where ˆ φ is replaced with φ (i.e. exact π, F a are used). In this experiment, we return back to the set-up of example 1. We now change the frequency of the sine term by taking

for γ ∈ [0 , 10] so that increasing γ imitates decreasing smoothness of our nuisance parameters. In our experiments half the samples are used to estimate the propensity score and CCDFs and the other half are used to regress against the pseudo-outcome. Our estimate ˆ g is then compared against g ∗ using a hold-out testing set. This process is repeated 500 times with new training data on each run. From this, a Monte-Carlo estimate of E ˆ g [ E X [ E Y | X,A =0 [ | ˆ g ( Y | X ) -g ∗ ( Y | X ) | ]]] is produced alongside 95% confidence intervals (CIs). In our first experiment, we let 2 n = 1000 and vary γ in [0 , 10] . In our second experiment, we let γ = 6 and 2 n vary in [200 , 5000] . The results of this are shown in Figure 3.

̂

0

2

4

6

8

10

γ

0.2

0.4

0.6

0.8

1.0

1.2

1.4

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

0

2000

4000

6000

8000

10000

2

n

0.0

0.5

1.0

1.5

2.0

2.5

3.0

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

̂

We can see that the average error decreases as the sample size 2 n grows and mildly increases as γ grows. This is expected as increasing γ makes estimating the nuisance parameters more challenging. The result shows that the proposed DR method achieves the best performance compared with the Separate and IPW estimators and is only marginally outperformed by the oracle estimator. Additionally we see that the method of Kallus and Oprescu [14] is much more affected by the increase in γ . This is because unlike the CQC, the CQTE has a complexity that depends on the frequency term ( γ ). We also observe much better performance as the sample size increases. We hypothesise that the plateau in the CQTE approach is due to the difficulty of estimating the reciprocal of the PDF, which causes the estimation to be unstable irrespective of sample size. Further simulated experiments including 10-dimensional x and linear CQC are given in Appendix B.1.

4.2 Real world employment example

To show the performance in the real-world scenarios, we use a dataset on an employment programme which has been studied in various prior works [4, 5, 26]. Within the programme, some participants were given job placements or temporary help jobs while others received no intervention. Participants' earnings were then monitored over the next 8 quarters following their enrolments. We take their net earnings as our response ( Y ) and the employment intervention as the treatment ( A = 1 ). We use each participant's age at their entry to the study as our covariate ( X ). We fit our quantile comparator function on 2,000 participants. Figure 4 shows our estimate of ∆ ∗ ( y | x ) = g ∗ ( y | x ) -y for various values of ( y, x ) .

We see that participants around age 23 and between ages 32-37 benefit most from this scheme, as indicated by the darker colour on the heat map. Additionally, the lower quantile of the income distribution (wage ≤ $7500) shows the least change, indicating that wage improvements primarily occur

25

30

35

40

Age (

x

)

0

5000

10000

15000

20000

Earnings (

y

)

2500

5000

7500

10000

12500

15000

Δ *

(

y

|

x

)

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

Age (

x

)

0

2500

5000

7500

10000

12500

15000

17500

20000

Earnings (

y

)

-15000

-10000

-5000

0

5000

10000

15000

Δ *

(

y

|

x

)

for the higher income group. For participants aged 40, there appears to be little change in outcomes overall. To demonstrate our approach in a medical setting, we apply our method to evaluate the effectiveness of a colon cancer treatment, as detailed in Appendix B.3. For comparative purposes, we also provide an estimate of the CQTE for this data in Appendix B.2.

5 Limitations

The theory provided here gives a strong foundation and motivation for framing our problem in this particular manner. There is however a great deal more to be explored in this area from a theoretical perspective. For example, one immediate improvement would be to give a more general case where our weight decay (in Assumption 2 (b) for Theorem 2) is sufficiently fast. In addition more work needs to be done exploring the relationship between the smoothness of g ∗ and the smoothness h ∗ . Smoothness in h ∗ appears to be a stronger condition so ideally we would like to make theoretical statements directly on the smoothness of g ∗ . Interestingly our experiments on synthetic data do seem to suggest that it is the complexity of g ∗ which drives the estimation rate as in these experiments h ∗ increases in complexity while g ∗ remains constant. Additionally, changing our experiment in Section 4.1 to have uniform response so that both h ∗ , g ∗ are constant rather than just g ∗ , seems to give no material improvement to the performance of our estimator (see Appendix B.1.3).

Another limitation of our current estimation procedure is that it requires learning an estimand and then inverting it to obtain our final estimator. While this process is relatively simple, it could be streamlined and made more computationally efficient if we could produce a more direct estimator similar to the DR-Learner for CATE [15] or the CQTE estimator in Kallus and Oprescu [14].

Finally, while we have been able to provide more concrete examples of smoothness for the CQC these are still limited to the case of a deterministic treatment effect which we would like to expand upon. This is closely related to a more general limitation with quantile-based estimands, the CQC included, in that they lack meaningful interpretability for the individual. While the CATE can be viewed as the expected difference in an individual's outcome on and off the treatment, no such individual-level interpretation exists for the CQC or indeed any other estimand trying to learn higher level distributional information than the mean. As a result the CATE is still a more naturally interpretable estimand. To facilitate this interpretation however, one still needs to make assumptions about a lack of confounding between the treatment assignment and the potential outcomes, which are only verifiable in certain restrictive scenarios [30].

6 Conclusion

In this paper we have introduced a new treatment effect estimand, the conditional quantile comparator and demonstrated its efficacy both in terms of its doubly robust estimation, and its ability to provide valuable data insights. This is a promising direction as it allows quantile-based treatment effect exploration to 'keep up" with the CATE in terms of estimation quality offering more flexibility as to which estimand can be used to best describe the data. For these reasons, we see the CQC as an exciting and worthwhile new direction within the HTE framework.

Acknowledgments and Disclosure of Funding

Josh Givens was supported by a PhD studentship from the EPSRC Centre for Doctoral Training in Computational Statistics and Data Science (COMPASS).

References

Abadie, A., Angrist, J., and Imbens, G. (2002). Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica , 70(1):91-117.

Abadie, A. and Imbens, G. W. (2002). Simple and bias-corrected matching estimators for average treatment effects. Working Paper 283, National Bureau of Economic Research. Series: Technical working paper series.

Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests. The Annals of Statistics , 47(2):1148 - 1178. Publisher: Institute of Mathematical Statistics.

Autor, D. H. and Houseman, S. N. (2010). Do temporary-help jobs improve labor market outcomes for low-skilled workers? Evidence from "Work First". American Economic Journal: Applied Economics , 2(3):96-128.

Autor, D. H., Houseman, S. N., and Kerr, S. P. (2017). The effect of work first job placements on the distribution of earnings: An instrumental variable quantile regression approach. Journal of Labor Economics , 35(1):149-190.

Barlow, R. E., Bartholomew, D. J., Bremner, J. M., and Brunk, H. D. (1972). Statistical Inference under Order Restrictions (The Theory and Application of Isotonic Regression) . Wiley Series in Probability and Statistics. John Wiley & Sons Ltd.

Boucheron, S., Lugosi, G., and Massart, P. (2013). Concentration inequalities: a nonasymptotic theory of independence . Oxford University Press.

Chen, G. (2019). Nearest neighbor and kernel survival analysis: Nonasymptotic error bounds and strong consistency rates. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th international conference on machine learning , volume 97 of Proceedings of machine learning research , pages 1001-1010. PMLR.

Chen, P., Dong, W., Lu, X., Kaymak, U., He, K., and Huang, Z. (2019). Deep representation learning for individualized treatment effect estimation using electronic health records. Journal of Biomedical Informatics , 100:103303.

Collins, F. S. and Varmus, H. (2015). A new initiative on precision medicine. The New England journal of medicine , 372(9):793-795. Place: United States.

Hirano, K. and Porter, J. R. (2009). Asymptotics for statistical treatment rules. Econometrica , 77(5):1683-1701. Publisher: [Wiley, The Econometric Society].

Huynh, P., Villaluz, J., Bhandal, H., Alem, N., and Dayal, R. (2021). Long-Term Opioid Therapy: The Burden of Adverse Effects. Pain Medicine , 22(9):2128-2130.

Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: a review. The Review of Economics and Statistics , 86(1):4-29.

Kallus, N. and Oprescu, M. (2023). Robust and agnostic learning of conditional distributional treatment effects. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of the 26th international conference on artificial intelligence and statistics , volume 206 of Proceedings of machine learning research , pages 6037-6060. PMLR.

Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics , 17(2):3008 - 3049. Institute of Mathematical Statistics and Bernoulli Society.

Kpotufe, S. (2011). k-NN regression adapts to local intrinsic dimension. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances in neural information processing systems , volume 24. Curran Associates, Inc.

Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Bin Yu (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences , 116(10):4156-4165.

Lakshminarayanan, B., Roy, D. M., and Teh, Y. W. (2014). Mondrian forests: Efficient online random forests. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in neural information processing systems , volume 27. Curran Associates, Inc.

Laurie, J. A., Moertel, C. G., Fleming, T. R., Wieand, H. S., Leigh, J. E., Rubin, J., McCormack, G. W., Gerstner, J. B., Krook, J. E., and Malliard, J. (1989). Surgical adjuvant therapy of large-bowel carcinoma: an evaluation of levamisole and the combination of levamisole and fluorouracil. The North Central Cancer Treatment Group and the Mayo Clinic. Journal of clinical oncology , 7(10):1447-1456. Place: United States.

Lei, L. and Candès, E. J. (2021). Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society, Series B , 83(5):911-938.

Nadaraya, E. (1965). On non-parametric estimates of density functions and regression curves. Theory of Probability & Its Applications , 10(1):186-190.

Nadeau, S. E., Wu, J. K., and Lawhern, R. A. (2021). Opioids and chronic pain: An analytic review of the clinical evidence. Frontiers in Pain Research , 2.

Nie, X. and Wager, S. (2020). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika , 108(2):299-319.

Obermeyer, Z. and Emanuel, E. J. (2016). Predicting the Future - Big Data, Machine Learning, and Clinical Medicine. The New England journal of medicine , 375(13):1216-1219. Place: United States.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830.

Powell, D. (2020). Quantile Treatment Effects in the Presence of Covariates. The Review of Economics and Statistics , 102(5):994-1005.

Rubin, D. B. (2005). Causal inference using potential outcomes. Journal of the American Statistical Association , 100(469):322-331.

Semenova, V. and Chernozhukov, V. (2021). Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal , 24(2):264-289.

Singh, R., Xu, L., and Gretton, A. (2023). Kernel methods for causal functions: dose, heterogeneous and incremental response curves. Biometrika , 111(2):497-516. tex.eprint: https://academic.oup.com/biomet/article-pdf/111/2/497/57467664/asad042.pdf.

VanderWeele, T. J. (2008). The sign of the bias of unmeasured confounding. Biometrics. Journal of the International Biometric Society , 64(3):702-706. Publisher: [Wiley, International Biometric Society].

Watson, G. S. (1964). Smooth regression analysis. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 359-372.

Yang, F. and Barber, R. F. (2019). Contraction and uniform convergence of isotonic regression. Electronic Journal of Statistics , 13(1):646 - 677. Publisher: Institute of Mathematical Statistics and Bernoulli Society.

Ying Zhang, Lei Wang, M. Y. and Shao, J. (2020). Quantile treatment effect estimation with dimension reduction. Statistical Theory and Related Fields , 4(2):202-213. Publisher: Taylor & Francis tex.eprint: https://doi.org/10.1080/24754269.2019.1696645.

Zhou, T., Carson, W. E. t., and Carlson, D. (2022). Estimating Potential Outcome Distributions with Collaborating Causal Networks. Transactions on machine learning research , 2022. Place: United States.

A Additional details

A.1 Notation table

A.2 Nadaraya-Watson estimation

Throughout, we use NW estimation as our standard non-parametric regression technique. For a kernel k : X × X → [0 , ∞ ) , IID data sample D := { ( Y i , X i ) } n i =1 , and x ∈ X the NW estimate of E [ Y | X = x ] is given by

Applying this to our pseudo-outcome regression in Algorithm 1, we get that

where for any x ′ ∈ X , a ∈ { 0 , 1 }

Note that for our theoretical results we do not specify how our nuisance parameters are estimated and the results only depend on the accuracy of our estimation of the nuisance parameters.

Remark 6. We can re-write (8) as:

Which can be helpful in terms of the practical implementation.

A.3 IPW pseudo-outcome estimator

Another pseudo-outcome one can use for estimating the treatment effect is the based on inverse propensity weighting. Specifically we can take our pseudo-outcome to be

If we regress against it using NW estimation, and also use NW estimation for our nuisance parameter estimation as in (9) & (10), our estimate ˆ h will be increasing in y 1 and decreasing in y 0 meaning we do not need to perform any isotonic projection.

A.4 Additional experimental details

As the method of Kallus and Oprescu [14] is one for estimating the CQTE, we transform it into an estimator of the CQC (which we denote by ˆ g ) using the following formula where ˆ τ q ( . | . ) is our CQTE estimator

using the true CCDF, F Y | X, 0 .

This is the inverse of equation (5) which defines the CQTE in terms of the CQC. Conversely we also tested transforming all our CQC estimators into CQTE estimators (using exactly equation (5) with ˆ g replacing g ∗ ) and testing the accuracy on this space and found similar results.

Each experiment took no longer than 1 hour to run on a single 4 core CPU with 8GB of RAM.

The bandwidths of the kernels for the NW estimation of the nuisance parameters were chosen by a limited grid search on additional simulated data by validating against the true value of the nuisance parameters. While this is unrealistic in practice it was done to make estimation of the nuisance parameters as strong as possible. This was to err on the side of caution as strong nuisance parameter estimation would naturally favour the baseline approaches.

Code to implement our approach alongside Jupyter notebooks running our numerical experiments can be found in the supplementary materials. The code to implement the kernels is adapted from https://github.com/wittawatj/kernel-gof/ which is free to use under the MIT Licence.

B Additional results

B.1 Simulation results

We run additional simulated experiments in order to further test and explore our approach. Throughout, our overall experimental set-up is the same as in Section 4.1

B.1.1 10-dimensional example

To test our problem in higher dimensions we ran experiments where X was 10-dimensional with X uniform on [ -1 , 1] 10 . That is each component X j was independent with X j ∼ U ( -1 , 1) . We then took

where β was randomly sampled from N (0 , (0 . 2) 2 ) and γ ∈ [0 , 3] . This gave the maximum gradient multiplied by 0 . 5 diam( X ) = √ d close to 1 making the rate of change of the CDFs and propensities similar to our 1-dimensional examples.

In our first experiment we take 2 n = 1000 and γ ∈ { 0 , 0 . 5 , 1 , 1 . 5 , 2 , 2 . 5 , 3 } . The results of this are shown in Figure 5a In our second experiment we take γ = 1 and 2 n ∈ { 200 , 500 , 1000 , 2000 , 5000 } . The results of this are shown in Figure 5b.

As we can see the DR approach performs the best with it performing close to oracle for low sample size and low frequency. As the frequency increases the DR estimator deviates from the oracle.

̂

̂

0.0

0.5

1.0

1.5

2.0

2.5

3.0

γ

0.2

0.4

0.6

0.8

1.0

1.2

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

0

2000

4000

6000

8000

10000

2

n

0.0

0.2

0.4

0.6

0.8

1.0

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

̂

(a) Average error as γ increases with 95% CIs for various approaches. (b) Average error as sample size n increases with 95% CIs for various approaches.

B.1.2 Varying CQC

In this experiment our set-up is

for γ ∈ [0 , 10] . This gives g ∗ = ( y + 0 . 5)(0 . 5 x + 1 . 5) which is still simpler than the individual CCDFs but now does depend upon x .

In our first experiment we take 2 n = 1000 and γ ∈ { 0 , 2 , 4 , 6 , 8 , 10 } . The results of this are shown in figure 5a In our second experiment we take γ = 6 and 2 n ∈ { 200 , 500 , 1000 , 2000 , 5000 } . The results of this are shown in Figure 5b.

̂

0

2

4

6

8

10

γ

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

0

2000

4000

6000

8000

10000

2

n

0.5

1.0

1.5

2.0

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

(a) Average error as n increases with 95% CIs for various approaches. (b) Average error as n increases with 95% CIs for various approaches.

B.1.3 Constant h ∗

In our previous examples, while g ∗ has been simple and or constant, h ∗ has actually included the high frequency sine term. In this experiment we adjust our original illustrative example to give a constant h ∗ as well. Specifically we take

for γ ∈ [0 , 10] . We also take the propensity to be π ( x ) = 0 . 4 sin( γπx ) + 0 . 5 . In this case h ∗ ( y 0 , y 1 | x ) = 1 2 y 1 -y 0 which does not depend on x for any y 0 , y 1 ∈ Y .

In our first experiment we take 2 n = 1000 and γ ∈ { 0 , 2 , 4 , 6 , 8 , 10 } . The results of this are shown in figure 5a In our second experiment we take γ = 6 and 2 n ∈ { 200 , 500 , 1000 , 2000 , 5000 } . The results of this are shown in Figure 5b. As we can see we obtain very similar results to original

̂

0

2

4

6

8

10

γ

0.2

0.4

0.6

0.8

1.0

1.2

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator (Oracle)

0

2000

4000

6000

8000

10000

2

n

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

g

error

Separate Estimator

IPW Estimator

DR Estimator (Ours)

Exact DR Estimator

̂

(a) Average error as n increases with 95% CIs for various approaches. (b) Average error as n increases with 95% CIs for various approaches.

illustrative example suggesting that our method is effectively exploiting simplicity in g ∗ rather than in h ∗ .

B.2 Employment example: Comparing to the CQTE

To further explore the interpretability of our estimator we provide an estimate of the CQTE for comparison. This can be seen in Figure 9.

25

30

35

40

Age (

x

)

0.2

0.4

0.6

0.8

Earnings Quantile (

α

)

0

2500

5000

7500

10000

12500

15000

τ

*

q

(

α

|

x

)

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

Age (

x

)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Earnings Quantile (

α

)

-15000

-10000

-5000

0

5000

10000

15000

τ

*

q

(

α

|

x

)

In these plots we can see that the interpretation of the CQTE is less immediate than for the case of the CQC. This is because, for each covariate value, the quantile value corresponds to a different untreated response making comparisons between various values of the covariates less direct. From this plot we do still see that higher income quantiles are associated with a greater increase in wages. Interestingly the value of the CQTE plummets 90% at the highest quantile for individuals of age 40. This is likely an estimation error due to the limited data at higher quantiles as and higher ages.

B.3 Colon cancer treatment

To demonstrate the effectiveness of our approach in medical settings, we apply it to a trial on the effect of colon cancer treatment on survival time/time to remission. This dataset was originally introduced in Laurie et al. [19] and can be found in the 'survival" package in R and loaded with the line data(colon, package="survival") . In this dataset 929 are randomised to either receive Placebo or Levamisole. They are then followed up for a period of up to 3329 days and the time till

either death or recurrence of their cancer. For our analysis we take our response ( Y ) as the time to first of death or recurrence. For simplicity, when an individual makes it to the end of a trial without an event and is censored we take that to be the time of their event. Looking at the data censoring times are mostly at around 2,000-3,000 days while over 90% of death or recurrence events that do occur, occur within 1,500 days. Therefore censored individuals will still have better responses than those who had events, as we would want. As our covariate, we took the patients' ages when joining the trial. We show a 3D plot and heat plot of the estimate of ∆( y | x ) = g ∗ ( y | x ) -y in Figure 10.

45

50

55

60

65

70

75

Age (

x

)

0

200

400

600

800

1000

Time to event (

y

)

0

250

500

750

1000

1250

Δ *

(

y

|

x

)

45

50

55

60

65

70

Age (

x

)

0

200

400

600

800

1000

Time to event (

y

)

-1500

-1000

-500

0

500

1000

1500

Δ *

(

y

|

x

)

Interestingly from Figure 10 we see that the treatment appears to have little effect on patients aged 53-57. For the reaming patients, we can see that there is relatively little effect on the time to event for events of 400 days or less while at around 500 days the difference in time to event jumps to 1000 days. This jump takes the time to event (TTE) to the time when censoring begins. This gives evidence that rather than the treatment delaying the time to event, it increases the proportion of the patients who have no event during the trial essentially fully treating those patients for the duration of the trial. This shows the value of the CQC it was able to reveal meaningful information about the treatment beyond simply its positive effect. On top of this, we are able to identify the censoring within our results without explicitly controlling for it in any way.

C Additional theory

C.1 Proof of ˆ h accuracy

We first define some additional notation. For an output f which depends upon z = ( y, x, a ) define m f ( x ) = E [ f ( Z ) | X = x ] and

with w defined as before so that ˆ m ( x ) is our estimator and ˆ m ( x ) is the oracle estimator.

j ˆ φ φ Additionally define

in other words the bias in our estimate of the pseudo-outcome for a given x . We will also work with ˆ m b ( x ) where we view b as being a function of z .

Finally for n ∈ N , δ ∈ (0 , 1) , define

Moreover, with probability at least 1 -δ we have

Proof. To prove (12) we first observe that since w is D I , X J -measurable we have

Note also that we have max j ∈J | w j { φ ( Z j ) -E [ φ ( Z j ) | D I , X J ] }|≤ ∥ w ∥ ∞ 2 ξ -1 almost surely. Note also that { φ ( Z j ) } j ∈ J are conditionally independent given D I , X J . Hence, the bound (12) follows from two applications Bernstein's inequality Boucheron et al. [7, Theorem 2.10], applied conditionally on D I , X J .

Next we note that by the triangle inequality we have

Moreover, since E [ ˆ φ ( Z j ) | D I , X J ] is the projection of ˆ φ ( Z j ) onto the subspace of D I , X J -measureable functions we have

Hence, to deduce (13) we apply the first bound with ˆ φ in place of φ to obtain the following bound with probability at least 1 -δ ,

as required.

We apply the following result from Kennedy [15].

Proposition 4 (Proposition 2 from Kennedy [15]) . Suppose that ˆ b ( x ) = ˆ b 1 ( x ) ˆ b 2 ( x ) then

Proof. Follows from the Cauchy-Schwartz inequality.

Proposition 5. Our pseudo-outcome is conditionally unbiased,

Proof. We have

and similarly E [ φ y 0 ,y 1 ( Z ) | X = x , A = 1] = h ∗ ( y 0 , y 1 | x ) . The result now follows by the law of total expectation.

C.1.1 Proof of Proposition 1

Proof of Proposition 1. Fix y 0 , y 1 , x Let ˆ m ˆ φ ( x ) be the regression of X against the estimated pseudo-outcome ˆ φ y 0 ,y 1 ( Z ) evaluated at x (i.e. ˆ h ( y 0 , y 1 | x ) ), let ˆ m φ be the same but with the estimated pseudo-outcome replaced with the exact pseudo-outcome φ y 0 ,y 1 . Finally take

Theorem 3 gives us that with probability. at least 1 -δ

with ˆ b (( y, x , a )) := E [ ˆ φ ( Z ) -φ ( Z ) | X = x ] .

To bound ˆ b we have that from Proposition 5,

Additionally, by splitting over the events { A = 1 } , { A = 0 } and noting that P ( A = 1 | X = x ) = π ( x ) we have

Hence

We have that using Proposition 4

Now it is simply a matter of bounding all the relevant terms which we do through the following events

According to our assumptions and previous results, E oracle , E ˆ φ , E γ , E α ∩ E β, 0 ∩ E β, 1 separately occur w.p. at least 1 -δ/ 4 . Then by the union bound, the intersection of all these events holds w.p. at least 1 -δ and under these events

Where the first term comes from Proposition 3, the second from Proposition 4 and the final term from smoothness of g ∗ .

C.2 Proof of ˆ g accuracy

Proposition 6 (Maximum step-size bound) . Suppose that both the outer regression and estimation of CCDFs is fit using linear smoothers so that

with w j ( x ) , w i ( x ′ ) Additionally suppose with probability at least 1 -δ

Then with probability at least 1 -δ

where [ n 1 +1] 0 := [ n 1 +1] ∪ { 0 } , ˆ h ( y 0 , Y (0) 1 | x ) = -1 , ˆ h ( y 0 , Y ( n 1 +1) 1 | x ) , and { Y ( i ) } i ∈ I 1 is the sorted version of { Y i } i ∈ I 1 .

Proof. We immediately have via union bounds that with probability at least 1 -δ

Now we can try to bound the jump size of our function. To do so for a step function f : Y → R and a step points y ∈ Y we define

i.e. the size of the step at y . From the definition of our pseudo-outcome, the jumps occur in different forms at { Y j } { j ∈J ,A j =1 } , { Y ( i ) } { i ∈I| A ( i ) =1 } . Note that we are assuming Y continuous so these points are all distinct a.s. .

For Y j with j ∈ J and A j = 1 . We have

For i ∈ I with A ( i ) = 1 , we have that

Hence the maximum jump size is less than

Therefore combining this with our previous bounds gives that with probability at least 1 -δ

F 1 We also have a result ensuring the step size of the projected function is even smaller.

Proposition 7. Let α ∈ R p and ˜ α := P Iso( p ) ( α ) . Then | α l -α l +1 |≥ | ˜ α l -˜ α l +1 | for all l ∈ [ p ] .

Proof. We will prove this by first giving an algorithm to compute the projection. Define

Now define α (0 , 0) = α , α ( n,i ) = ψ i ( α ( n,i -1) ) for i ∈ [ p -1] and α ( n, 0) = α ( n -1 ,p -1) for n ∈ N .

Finally define α ( t ) = α ( ⌊ t/p ⌋ , mod ( t,p ) . Then by Yang and Barber [32] we know that lim t →∞ α ( t ) = P Iso( p ) ( α ) .

Now if | α ( n,l +1) l +1 -α ( n,l +1) l | > | α ( n,l ) l +1 -α ( n,l ) l | . Then α ( n,l +1) l +1 has be moved by ψ l +1 . Hence α ( n,l +1) l +1 < α ( n,l ) l +1 . For this to increase the distance then we must have

Therefore as α ( n +1 ,l -1) l ≥ α n,l +1 l , we have that

By a similar (simpler) argument | α ( n,l -1) l -α ( n,l -1) l +1 | > | α ( n,l -2) l -α ( n,l -2) l +1 | then α ( n,l ) l = α ( n,l ) l +1 . Hence if any iteration moves adjacent points further apart, a later iteration will make them equal meaning that on convergence they will be equal. Hence we have proved our claim.

We now have the result which justifies our choice to take the isotonic projection of the data which is take from Yang and Barber [32].

Proposition 8 (Theorem 1 from Yang and Barber [32]) . Let z ∗ , ˆ z ∈ R p with z ∗ ∈ Iso( p ) . Then for ˜ z := P Iso( p ) (ˆ z ) ,

We now use this isotonic projection to obtain supremum bounds on the accuracy of our estimator.

Proposition 9 (Supremum bound on ˆ h accuracy) . Fix x ∈ X , y ∈ y 0 and let ˆ h be our original estimate of h ∗ . For m ∈ N , take { Y ( l ) } l m =1 to be a potentially random set of points in Y in increasing order.

Now define α ∈ R m by α l := ˆ h ( y 0 , Y ( l ) | x ) and ˜ α := P Iso( m ) ( α ) . Finally, define ˜ h to be the piecewise constant right continuous function with ˜ h ( y 0 , Y ( l ) | x ) := ˜ α l .

Suppose for any y 1 ∈ Y , n ∈ N , and δ, δ ′ > 0 that

where we take [ m ] 0 := [ m ] ∪ { 0 } , ˆ h ( y 0 , Y (0) | x ) := -1 , and ˆ h ( y 0 , Y ( m +1) | x ) := 1 . Then for any δ, δ ′ > 0 , n ∈ N ,

Proof. For each l ∈ [ m ] define the event

and take E h := ⋂ n l =1 E h,l . Additionally define the event

Then E h and E step hold w.p.s at least 1 -δ and 1 -δ ′ respectively. Also under E h , E step , by Propositions 7 & 8, we have that

We then have that for any y 1 ∈ Y there exists i ∈ [ m ] such that y ( l -1) 1 ≤ y 1 ≤ y ( l ) 1 . Hence by monotonicity, we have the following 2 inequalities

Now for the first inequality we have

With the final inequality coming from E h,l -1 and our definition of h . By a similar argument we get from E h,l that Λ 2 ≤ ε h ( n, δ ) + ε step ( n, m, δ ′ ) . Again we can use the same approach to get that under E h,l -1 , E h,l , E step

Hence for our specific y 1 under E h,l , E h,l -1 , E step

Now as this inequality holds for arbitrary y 1 under E h ∩ E step (an event which not depend on our choice of y 1 ) and the intersection of these two events holds w.p. at least 1 -δ -δ ′ by the union bound we have our result.

Our final key result before we can piece them all together will be to obtain accuracy in g from our accuracy in h ∗

Theorem 10 (Single point accuracy bound) . Fix x , y 0 , assume that h, ˜ h are strictly monotonic in y 1 and suppose that

Additionally let α := F 0 ( y 0 | x ) and assume that f Y | X,A =1 ( y 1 | x ) > η for all y 1 ∈ F -1 1 ( B ε ( α ) | x ) where here we are taking F -1 1 ( A | x ) to be the pre-image in Y of the set A for fixed x . Then

where g ∗ ( y 0 | x ) is the unique y 1 such that h ∗ ( y 0 , y 1 | x ) = 0 .

Proof. Let y ∗ 1 := g ∗ ( y 0 | x ) and ˆ y 1 := ˆ g ( y 0 | x ) so that F 1 ( y ∗ 1 | x ) . From our accuracy assumption on ˆ h in the set-up of the Theorem we have

where the second and fifth line come from the definition of y ∗ 1 and the fourth from the definition of ˆ y 1 .

If we define ∂ k f to be the derivative with respect to the k th argument of f then we have

C.2.1 Proof of Theorem 2

We can now finally combine all these results to prove Theorem 2.

Proof of Theorem 2. From proposition 1 we have that

with

Now define { Y ( i ) } n i =1 to be the sorted version of { Y i } n i =1 . Then by Proposition 6 we have

Plugging this into Proposition 9 with ε step ( n, m, δ ) replaced by ξ -1 ε w ( n, δ/n )) and ε h ( n, δ ) replaced with ε h ( n, δ/ 2) gives

Now let y ∗ 1 := g ∗ ( y 0 | x ) . Then for any y 1 ∈ Y if | y ′ 1 -y ∗ 1 | > s this implies that | F 1 ( y ′ 1 | x ) -F 1 ( y ∗ 1 | x ) | > sη by our lower bound on the density in B s ( y ∗ 1 ) . Hence by the contrapositive, if | F 1 ( y ′ 1 | x ) -F 1 ( y ∗ 1 | x ) |≤ sη then | y ′ 1 -y ∗ 1 | < s .

Now as

Hence if n is sufficiently large so that ε := ε h ( n, δ/ (2 n )) + ξ -1 ε w ( n, δ/n )) ≤ ηs then we satisfy the bounded density condition of Theorem 10. Therefore we can plug our bound into theorem 10 gives

C.3 Extension to expectation

Proposition 11. Let Y 0 , D be RVs on Y , Z n respectively (with D representing data used to fit a model and Y 0 representing the point where the model is fit). Now take l ( Y 0 , D ) to be a non-negative bounded loss so that l ( Y 0 , D ) < l max a.s. . Suppose that for any δ > 0 , for all y ∈ Y

Then for any t, δ 0 ∈ [0 , 1] and p, q ∈ [1 , ∞ ] such that 1 /p +1 /q = 1

In particular if l ( y, D ) < l max a.s. then taking q = 1 , p = ∞ yields

Proof. First fix t, δ 0 ∈ [0 , 1] We will first bound the probability that the number of y which don't satisfy our bound isn't too large. We do this by defining the event

and now aim to bound the probability that P ( A | D ) > t (this probability is just w.r.t Y 0 treating D as fixed.)

By Markov's inequality,

Now define B := { P ( A | D ) ≤ t } so that P ( B ) = 1 -δ 0 . Then under B

Corollary 12. Assume that F 1 ( y | x ) > η for all y 1 ∈ Y . Then, for our fixed x ∈ X and δ ∈ (0 , e -1 ) , w.p. at least,

where ε h ( δ, n ) := ε T n ( n, δ/ 4) + ε α ( n, δ/ 4) ε β ( n, δ/ 4) + ε γ ( n, δ/ 4) . In particular for 2 ( η -1 ε h ( n, δ/n ) + 2 ξ -1 ε w ( n, δ/n )) ) ≤ log( e 1 ( n ) /δ ) a /e 2 ( n ) . For δ < 1 /e we have that w.p. at least 1 -δ

Proof. Plugging the result of theorem 2 into proposition 11, noting that | ˆ g ( y 0 | x ) -g ∗ ( y | x ) |≤ diam( Y ) and taking

yields that w.p. at least

Following from this if 2 ( η -1 ε h ( n, δ/n ) + 2 ξ -1 ε w ( n, δ/n )) ) ≤ log( e 1 ( n ) /δ ) α /e 2 ( n ) . Then we have

for δ < 1 /e we have

Hence

Finally δ ′ = 2 δ diam( Y ) e 2 ( n ) gives

C.4 Application and justification of NW estimation with box kernel

We aim to show that the box kernel satisfies some of our conditions. Specifically conditions 2 & 3 from Proposition 1. We first start by bounding the step size.

Proposition 13 (Effective sample size) . Suppose that for our x ∈ X there exists C 0 , r 0 > 0 such that for any r ∈ (0 , r 0 )

Now for r ∈ (0 , r 0 ) take our kernel to be k r ( x , x ′ ) := 1 {∥ x -x ′ ∥≤ r } . Then w.p. at least 1 -δ

Proof. We have 1 { X j ∈ B r ( x ) } ≤ 1 and E [ 1 { X j ∈ B r ( x ) } 2 ] = E [ 1 { X j ∈ B r ( x ) } 2 ] = P ( X j ∈ B r ( x )) = C 0 r d . Therefore by one sided Bernstein's inequality with ε = log(1 /δ ) we get

This gives our desired result.

Note that ∑ j ∈J 1 { X j ∈ B r ( x ) } is also the effective sample size of our estimation as it is the number of samples used in the average.

Now that we have the effective sample size result in terms of our kernel radius r , we need to obtain the optimal rate of decay of r for our estimation.

Proposition 14 (NW estimation with box kernel) . let ˆ m f ( x ) be the NW estimation of m f ( x ) := E [ f ( Z ) | X = x ] using IID copies ( Z i ) n i =1 of Z . and assume | f ( Z ) | ≤ M . For a fixed r ∈ (0 , r ) , use kernel k r as defined above and suppose the same assumptions hold. Suppose that m f ( x ) is α smooth for α ≤ 1 (i.e. α -Holder continuous.) Then for sufficiently large n depending on C 0 , α and δ ≤ 2 e -1 w.p. at least 1 -δ ,

Proof. With our kernel define I r := { i ∈ [ n ] | k r ( x , X i ) = 1 } and n r = | I r | . Now define the event

Then by Proposition 13 this event occur w.p. at least 1 -δ/ 2 . Nowif we define ε i := f ( Z i ) -m f ( X i ) then we have E [ ε i | X i ] = 0 . Also, we have

With the final equality coming by our smoothness condition and definition of I r . Now by Hoeffding bounds we have that

Hence by combining this event and E n then we have that with probability at least 1 -δ

Now for sufficiently large n

This in turn gives

Then the optimal choice of r is such that

Hence plugging this in we get that for sufficiently large n depending on C 0 , α we have that for δ ≤ 2 e -1 w.p. at least 1 -δ

Proposition 15 (Final weight decay) . Suppose that for our x ∈ X there exists C 0 , r 0 > 0 such that for any r ∈ (0 , r )

Suppose that we are performing NW estimation of an α smooth function at points x ∈ X using kernel k r n ( x , x ′ ) := 1 {∥ x -x ′ ∥≤ r } with r n decaying optimally. Now define

the weight of the j th component. Then with probability at least 1 -δ

Hence by proposition 13 we have w.p. at least 1 -δ

We know from Proposition 14 that the optimal radius decay gives 1 nr d = n -2 2+ d/γ . Plugging this into our result gives that

Note that for a γ smooth function the MSE is C -1 n -1 2+ d/γ . Hence the box kernel comfortably satisfies our weight decay condition 2 in Assumptions 2.

Additionally now if we have an α smooth function and assume that for any x ′ ∈ X there exists C 0 ( x ′ ) such that for all r ∈ (0 , r 0 ) P ( X ∈ B r ( x ′ ) ≥ C 0 r d . Then we have that w.p. at least 1 -δ ,

Thus if we assume that there exists C ′′ > 0 such that w.p. at least 1 -δ , 1 C 0 ( X ) ≤ C ′′ √ log(1 /δ ) then we get that w.p. at least 1 -δ

This then gives our condition 2 in Assumptions 2.

We now try to bound ∑ j ∈J w 2 j σ ( X j ) E [ ∑ j ∈J w 2 j σ ( X j ) ] .

Corollary 16 (Accuracy under NW estimation with box kernel) . Suppose that our linear smoother is NW estimation with the box kernel and optimally decaying radius additionally assume that:

ε α ( n, δ ) = C α √ log(1 /δ ) n -1 2+ d/α .

ε β ( n, δ ) = C β √ log(1 /δ ) n -1 2+ d/β .

Proof. We have

β > d 2(1+ d/γ ) . Then for any δ , for sufficiently large n the following events each separately hold w.p. at least 1 -δ ,

with C h , C g , C g, 2 depending on C α , C β , C ′ , C ′′ (where C ′ , C ′′ are the constants define in Proposition 15.

Proof. For any NW estimator we have that ∥ w ∥ 1 , ∥ w ∥ 2 ≤ 1 a.s. meaning we can take ε w ( n, δ ) ≡ 1 . We also have that ε γ ( n, δ ) = C γ log(1 /δ ) n -1 1+ d/γ . Hence ε T n = √ 2 ε δ ( n ) e ˆ φ ( n ) √ log(1 /δ ) . This then gives

for δ < e -1 and C h = 7( C γ + C α C β + √ 2) giving our first result.

Using our weight decay results for NW estimation and plugging into Proposition 6 we get that

Hence plugging into Theorem 2 gives

with C g = C h √ 2 + 4 C ′′ giving our second result. Finally by Corollary 12 for δ < e -1 w.p. at least 1 -δ ,

Note that both ε T n and ε h -step are actually both o ( ε γ ( n, c )) , thus if we were allowed to take n sufficiently large depending upon δ then we would have all log 3 / 2 terms replaced with log terms and the dependence on C ′ , C ′′ removed.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We introduce a a new estimand, the CQC. We claim this to maintain the distributional information of the CQTE which we show through the definition and also on real world scenarios in Section 4. We claim that our method is able to obtain double robustness which we show in proposition 1 and Theorem 2. We also demonstrate its strong performance empirically on simulated data scenarios in Section 4 and Appendix B.1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a dedicated limitations section where we discuss limitations in our theoretical results regarding both assumptions of sufficiently quickly decaying weights in our linear smoother and requiring smoothness in h ∗ rather than in g ∗ . We also discuss the limitations in the interpretability of quantile based approaches in general when exploring treatment effect as well as limitations in the interpretability of our assumptions within our theory. Finally we briefly address the methodological limitations of our approach acknowledging this as a potential area for improvement.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The two main results of the paper, Proposition 1 and Theorem 2 are proved in sections C.1 and C.2. The assumptions for these results are given in Assumptions 1 & 2 with the implications of these assumptions discussed.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper gives detailed information about the experimental set-up and explicit algorithms for methods introduced. The specific use of algorithm 1 with NW estimation is also described Appendix A.2. In addition code is provided in the supplementary materials with 'SimulatedExperiments.ipynb' reproducing all of the experiments. We also provide a link to a public github repository containig this in a footnote on page 9.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with

the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided in the supplementary materials with each function documented as well as in a linked public github repository. Jupyter Notebooks are provided to reproduce all of the experiments and real world data settings used within the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most of the experimental details are provided alongside the experimental results in Section 4 with additional details provided in Appendix A.4. In addition code to replicate all of the experiments is provided in the supplmentary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For our experimental results we explain that 500 replications of the entire fitting procedure (including newly generated data making each run entirely independent) was run and that 95% confidence intervals are provided for the mean absolute error of each estimator.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information on runtime and compute resources is provided in Appendix A.4. Overall computational resources were very low with all experiments run on a personal laptop and each one taking less than an hour even with a large number of replications.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No outsourcing was done for this project/paper and all datasets used are publicly available and completely anonymised. Additionally we feel there is essentially no scope for our work to cause negative societal impact.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential positive impact of our work in terms of further empowering HTE analysis. We do not for see any potential negative impacts of our work as it is simply furthering the field of HTE and quantile based treatment effect analysis. To our knowledge, there is no suggestion that these fields could be used maliciously or unfairly beyond the ways that any regression technique or analysis could.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no new datasets released in our work. Our models use standard nonparametric regression techniques and adapt these to best estimate our estimand, the CQC meaning there is no real risk of misuse beyond the misuse of any regression technique.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets are publicly available and properly referenced. Proprietary code used is also open source and used in accordance with its licence. The licence for this code can also be found in code itself. The model we use is our own and so no licensing is required.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only asset is the code which is implemented in our method which is provided in the supplementary materials alongside a licence.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects is used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new estimator to obtain conditional quantiles of a treatment effect and argues that it is superior to the pre-existing \"conditional quantile treatment effect\" (CQTE) estimator. The main difference is that accurate CQTE estimates require accurate conditional quantile models even though the latter can be hard to fit, while the introduced CQC estimator is more robust to the CDF estimators it relies on.\n\nSTRENGTHS:\nIntroduces the doubly robust property to estimating conditional quantile treatment effects, a property that many CATE estimators already have.\n\nDetailed theory work supporting the claim of robustness of CQC to its component estimators.\n\nExploits existing work on doubly robust CATE estimation by reframing CCDF estimation as a CATE-like problem.\n\nWEAKNESSES:\nIn 4.1 you test against estimating the CCDFs separately and taking their difference. Do you mean estimating the inverse CDFs / quantile functions and taking their difference i.e. CQTE? If not, what exactly is the procedure here and why not compare against the difference of two quantile estimators.\n\nCCDFs are arguably harder to estimate than quantile functions in my experience (they require the additional 'feature' y, have to work with classification losses instead of regression losses, and are more prone to monotonicity violations). So even if the final estimator here is more robust to bad CCDF estimation I'm not convinced it will necessarily be better in practice to the CQTE approach of fitting quantile functions. \n\nThis is especially true with modern day neural network based quantile models. NW estimators are unlikely to be as good as those on larger / higher dimension datasets, and it's not clear how the theory would hold in cases like that.\n\nIsotonization can be risky and create large flat zones for $\\hat{g}$. Did you notice any in your experiments? Are there other ways to get monotonicity here, similar to the various strategies to get monotonicity in multi-quantile regression?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an important problem -- treatment effects often have distributional effects and considering conditional mean outcomes is often not the correct objective for evaluating prescriptive interventions. One popular approach to consider heterogeneous effects across a population which allows researchers to determine treatment effects at different quantiles of the study distribution to determine how treatments might effect the worst-off populations. Unfortunately, as the authors note obtaining quantile treatment effects often relies strict smoothness assumptions on the marginal quantile functions and lacks the double robustness properties of CATE estimators. To this end, the authors propose a new estimand to evaluate heterogenous treatment effects that they show too be doubly robust and apply this estimator to both simulational and real-world data.\n\nSTRENGTHS:\n- The paper provides a clean mathematical framework for their estimator and proves useful finite sample properties that should be theoretically convincing for users\n- Example 1 is a nice simulational example to showcase why typical conditional quantile estimators may provide instable results when the author's CQC estimator is more robust\n- Proposition 1 provides the double robustness result which has been a useful benefit in finite sample analyses and helps to make the case for using this estimator\n- Numerical experiments in section 4 help to show case the benefits of this procedure. In particular the second graph in Figure 3 provides good evidence for using this methodology.\n- Finally, the application to two real-world data sets is helpful to showcase the viability of this methodology for practioners.\n\nWEAKNESSES:\n- I understand that the CQTE can be written as a function of $\\Delta^*$ but this does not help me to understand how I should interpret this quantity in relation to the typical CQTE. The authors indicate that this it is a \"rephrasing\" but and provide the mapping. but its not obvious how this translates in actual examples. Since the authors are proposing a non-standard estimand they should provide more details linking this to the standard analysis. One place where this would be very beneficial is in the interpretation of the real-world data sets. I consider this to be the main weakness of the work because without a clear interpretation for the estimand practioners will not adopt this methodology. Without this I doubt the impact of the paper beyound showcasing a technical deficiency in the CQTE estimand due to smoothness requirements.\n- I found the algorithms a little difficult to understand especially because of the reindexing where you duplicate the data. This seems purely notational and seems like you could simplify the notation here.\n- The method does not require smoothness of the marginal quantiles, however h must be Holder-smooth. Certainly in many cases smoothness of the marginal may effect smoothness of h. Since the reason for introducing this method is non-smoothness of the marginal quantiles it might be useful to expand on the class of problems where this is an easier problem beyound the example given in Example 1. How does this interact with smoothness of g?\n- To that end, could you also provide real-world examples where you might expect the marginals to be poorly behaved but the CQC to be well be haved?\n- Finally, in the the final paper in paper it would be good to have the CQTE estimates in the real world examples so you can draw a contrast between the methods.\n\nQUESTIONS:\nIn remark 3 you mention you use a cross-fitting method to improve stability, why is this the case? Duplicating the data yields non-independence?\nHow do i interpret the earnings for groups with the highest/lowest benefits in Figure 4?\n\nLIMITATIONS:\nThe authors have addressed some limitations of their work but have not acknowledged possible negative societal impact. From my perspective, work that emphasizes the importance of considering distributional effects is generally societally beneficial. The main limitation that the authors address are primarily on the interplay between the assumption on h and if they could be relaxed to assumptions on g. However, there are several other assumptions and limitations that should be made clear in this work. The authors should potentially consider including limitations in their analysis of the two real-world data sets which might give practitioners a better understanding of how to use this method. Examples of what could be added are given in the weakness section especially with regards to interpretability of results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the Conditional Quantile Comparator (CQC), a function that maps an outcome from the control group in a binary treatment setting to an outcome in the treatment group, such that they represent the same conditional quantiles in their respective distributions. The estimation procedure consists of two stages: first, estimating a conditional contrasting function that examines the difference between conditional CDFs, followed by isotonic regression. The contrasting function is estimated using pseudo-outcome regression, which confers the algorithm with double-robust properties in finite samples. This estimator addresses some of the shortcomings of the Conditional Quantile Treatment Effect (CQTE), which is not robust to errors in the quantiles. The CQC is supported by a theoretical analysis of finite sample convergence rates under smoothness assumptions, as well as by empirical simulations.\n\nSTRENGTHS:\nThe authors propose a novel quantile-based treatment effect estimation procedure tailored for skewed outcome distributions. The double-robustness properties in the first stage of the algorithm are desirable as they improve the rate dependence on the nuisance functions ( propensity and local CDF estimates). The paper is clearly written, with sound theoretical results and promising empirical evidence.\n\nWEAKNESSES:\n* The motivation for the estimator is somewhat weak.  Estimators for CQTEs with double-robustness properties have been already proposed (see [1, 2]). These estimators have a second order dependence on the rate of quantile estimation, as desired. The authors should contextualize their work with respect of exiting literature (which seems to overall be missing in the paper) and compare their algorithm with the existing techniques. \n* Moreover, the pseudo-outcome  estimation technique in [1] is somewhat simpler (see their Appendix B) since they only require one pseudo-outcome regression. There seem to be some tradeoffs, e.g. the CQC requires a two stage procedure that includes a prost processing step (the isotonic regression), as well as for Algorithm 1 to be run several times for each value of $y_1$ considered (because the conditional CDFs need to be estimated at each value). On the other hand, the robust CQTE algorithm from [1] require estimating the conditional PDF at several points. Overall, the CQC seems to be a more computationally (and statistically) intensive estimation procedure and the authors should at least discuss the tradeoffs. \n* Furthermore,  [1] provides rates for more general classes a functions outside of the linear smoothers framework in [3]. I suggest the authors try to provide similar guarantees in order to generalize their results beyond linear smoothers which, to the best of my knowledge, are not often used in practice. This would be particularly useful since the rate results in section 3.4 require all nuisances to be estimated using linear smoothers. \n* Empirical evaluations should also include these existing methods. \n\nOverall, I tend towards soft rejection. This estimator might be useful in its own right, but the authors should contextualize and compare with existing work. I am willing to update my score upon further discussion.  \n\n[1] Kallus, Nathan, and Miruna Oprescu. \"Robust and agnostic learning of conditional distributional treatment effects.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n[2] Leqi, Liu, and Edward H. Kennedy. \"Median optimal treatment regimes.\" arXiv preprint arXiv:2103.01802 (2021).\n\n[3] Kennedy, Edward H. \"Towards optimal doubly robust estimation of heterogeneous causal effects.\" Electronic Journal of Statistics 17.2 (2023): 3008-3049.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nThe authors have adequately adressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new estimand called the Conditional Quantile Comparator (CQC), which computes the quantiles of the outcome distributions of the treatment effect, hence improving on the (usual) conditional average treatment effect. The CQC is defined as a measurable function that maps an untreated outcome to the equivalent treated outcome in the same quantile, conditional on covariates. The authors introduce a doubly robust estimation procedure for CQC, where the estimation is re-framed as a CATE problem using the pseudo-outcome framework. The authors present finite sample bounds on the estimator's error, and provide numerical simulations and result on a real employment dataset.\n\nSTRENGTHS:\n- The paper is well written and easy to follow\n- The paper provides a thorough theoretical analysis for the finite sample bounds and proofs of convergence rates for CQC and proposed estimation algorithms\n- The simulations and real use-case considered provide a good intuition to the reader on the benefits of the proposed approach\n\nWEAKNESSES:\nFirst of all, I'd like to say I am not very familiar with the causal inference literature, so some of my comments might not be as relevant.\n\n- Complexity of using quantiles: can the authors comment in terms of error bounds and convergence rates what is the price to pay to estimate quantiles as opposed to CATE? Some of these results, including the use of pseudo-outcomes and isotonic projections, appear a little bit unusual, as usually quantiles are estimated through the pinball loss. As usual what is important is assessing the left tail or right tail (i.e., whether the treatment effect crosses the zero or not), I would expect the right/left tail quantiles to be the most important ones -- is the error in estimating all the quantiles the same across the entire distribution?\n\n- Dimensionality of the input $\\textbf{x}$: both the simulation and the real dataset use a very low-dimensional $\\textbf{x}$, and the comments on using the Nadaraya-Watson estimator and kernel smoothing are really only applicable when X is generally low-dimensional. How does in practice your method behave as function of the input dimension $\\textbf{x}$, and would your comments still be applicable even in higher-dimensional settings?\n\n- Estimating quantiles vs entire distribution: how does this approach compare to other approaches that estimate the entire conditional density, such as [1]? Is the use of a doubly robust framework guarantee the optimal error rates?\n\n\n[1] Zhou, T., Carson IV, W. E., & Carlson, D. (2022). Estimating potential outcome distributions with collaborating causal networks. Transactions on machine learning research, 2022.\n\nQUESTIONS:\nSee the weaknesses section above.\n\nLIMITATIONS:\nThe authors do address some limitations and implications of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}