Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Authors: Minqin Zhu
Zexu Sun
Ruoxuan Xiong
Anpeng Wu
Baohong Li
Caizhi Tang
JUN ZHOU
Fei Wu
Kun Kuang
Venue: ICML 2025 poster (2025)
Keywords: Uplift Modeling; Causal Ranking; Qini Curve; Marketing
Primary area: general_machine_learning->causality

Abstract: Uplift modeling is crucial for identifying individuals likely to respond to a treatment in applications like marketing and customer retention, but evaluating these models is challenging due to the inaccessibility of counterfactual outcomes in real-world settings.
In this paper, we identify a fundamental limitation in existing evaluation metrics, such as the uplift and Qini curves, which fail to rank individuals with binary negative outcomes accurately.
This can lead to biased evaluations, where biased models receive higher curve values than unbiased ones, resulting in suboptimal model selection.
To address this, we propose the Principled Uplift Curve (PUC), a novel evaluation metric that assigns equal curve values of individuals with both positive and negative binary outcomes, offering a more balanced and unbiased assessment. 
We then derive the Principled Uplift Loss (PUL) function from the PUC and integrate it into a new uplift model, the Principled Treatment and Outcome Network (PTONet), to reduce bias during uplift model training.
Experiments on both simulated and real-world datasets demonstrate that the PUC provides less biased evaluations, while PTONet outperforms existing methods. The source code is available at: https://github.com/euzmin/PUC.

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Minqin Zhu'
Zexu Sun?
Ruoxuan Xiong²
Anpeng Wu' Baohong Li
Caizhi Tang
Jun Zhou
Fei Wu'
Kun Kuang'
Abstract
Principal Strata Analysis
Notation Explain
Persuadable
Sleeping Dog (50.c0))
TP: Treated 1) with Pesrivv outcomess 1)
Uplift modeling is crucial for identifying indi- viduals likely to respond to treatment in appli- cations like marketing and customer retention, but evaluating these models is challenging due to the inaccessibility of counterfactual outcomes in real-world settings. In this paper, we identify fundamental limitation in existing evaluation metrics, such as the uplift and Qini curves, which fail to rank individuals with binary negative out- comes accurately. This can lead to biased evalua- tions, where biased models receive higher curve values than unbiased ones, resulting in subopti- mal model selection. To address this, we propose the Principled Uplift Curve (PUC), a novel eval- uation metric that assigns equal curve values of individuals with both positive and negative binary outcomes, offering a more balanced and unbi- ased assessment. We then derive the Principled Uplift Loss (PUL) function from the PUC and integrate into new uplift model, the Principled Treatment and Outcome Network (PTONet), to reduce bias during uplift model training. Experi- ments on both simulated and real-world datasets demonstrate that the PUC provides less biased evaluations, while PTONet outperforms exist- ing methods. The source code available at: https: /ttp://gtttttbttt comm/uuzmi//PSS
(PE, 0)
Negatie ouscomes (Y 0)
Pg"
sDiT ra)=
SD
TN: Treined with Negative outcontes
rd)-1 1
Y
CP: Custrolled (T- B) with Fucilive ouneomas (Y
E Highest priecity @ Lower, prictty Neutal Priority
Curve Analysis from Ralanced Perspoctive
convestional uplift
Evalustion using
I Lead to Unbalanced priority:
Pgtte
SgCP
Bux
Pg 5p
7
Becssae they are undervalued by conpentional uplin curves
patt higher than Pydy sD higher than SD
Incorreut ranking order: 1, 2.5 PY 4. S.
Evalussion using the principled uplill curve (Ours):
PgTP pgd SD sper
Load
Balanced priority:
Lead
pg equal to padss spCr equal to Sprv
Correct making order: 1, p6TT 3, 3. 4,5
Figure I. A halanced perspective analysis conventional versus principled curves in evaluating CATE-based principal strata.
individuals most likely to benefit from a treatment (Yamane et al., 2018; Devriendt et al., 2020; Belbahri et al., 2021; Liu et al., 2023). these scenarios, uplift models estimate the Conditional Average Treatment Effect (CATB) of binary treatment variable T on binary outcome variable Y for each individual with covariate X = X. The CATE T(x) represents the difference in expected outcomes for individu- als with covariate X = x assigned to the treatment group (T' = 1) versus the control group (T' = 0). Based on CATE values, individuals are divided into two principal strata: per- suadables (PF, where ((x) > 0) and sleeping dogs (SD, where T(x) < 0). Each stratum
be further divided into two subgroups according to the outcome value, as illustrated in the principal strata analysis in Figure 1. Uplift modeling aims to rank individuals based on CATE, prioritizing per- suadables over others, particularly over sleeping dogs. This indicates the core objective of uplift modeling lies in the ranking of CATE values rather than their precise estimation.
1. Introduction
Uplift modeling is widely applied in fields such as market- ing, advertising, and customer retention, where the goal is to support personalized decision- decision-making by identifying
'College of Computer Science and Technology, Zhejiang Uni- versity, China "Gaoling School of Artificial Intelligence, Ren- min University of China, China 'Depariment of Quantitative The- ory and Methods, Emory University, USA * AAnt Group. Zhejjang, China. Correspondence to: Kun Kuang <kunkuang@ @ zju.edu.cn>>.
However, CATE is counterfactual quantity and cannot be directly observed in real-world data, posing challenges in its accurate ranking evaluation and using it to make informed decisions (Morgan & Winship, 2015; Xiong et al., 2024). Previous studies have proposed various methods to estimate the CATE, commonly using uplift and Qini curves as evalua-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
tion metrics to assess model performance (Shalit et al., 2017:
ing methods (Shalit et al., 2017: Shi et al., 2019; Devriendt et al., 2020; Ke et al., 2021; Zhong et al., 2022; Liu et al., 2023). A notable study by Renaudin & Martin (2021) cau- tions against using conventional uplift and Qini curves for evaluating observational data, but suggests applying them in randomized controlled trials (RCT). However, we demon- strate that these curves can still produce biased evaluations even with RCT data. To address this limitation, we propose novel evaluation metric mitigate this concern. The most relevant to ours is Devriendt et al. (2020), which designs an uplift helper function to guide the uplift modeling. Building on this, we propose novel loss functions to demonstrate how uplift models optimized to maximize conventional curves can lead to biased estimates. In contrast, our method elim- inates these biases, providing more accurate ranking of CATE. A more detailed discussion of related work is pro- vided in Appendix A.
Yamane et al., 2018; Künzel et al., 2019; Shi et al., 2019; Ke et al., 2021; Zhang et al., 2021; Zhong et al., 2022; Liu et al., 2023). These curves evaluate uplift models by quan- tifying the cumulative gain among ranked individuals, but crucially, only individuals with positive outcomes (Y = 1) are explicitly included in the calculation of the uplift and Qini curves. Specifically, these individuals are ranked in descending order according to their estimated CATE, and model performance is measured by calculating the cumula- tive difference between the number of treated (T = 1) and control (T' = 0) individuals within this ranked subset.
In this paper. we argue that conventional uplift and Qini curves fail to accurately measure the CATE ranking per- formance of uplift models, largely because they overlook individuals with negative outcomes in their calculations. As illustrated in Figure 1, these metrics focus exclusively on individuals with positive outcomes (PETP and SDCP) in the ranked data, while neglecting those with negative out- comes (PEON and SD As a result, the distinction between persuadables and sleeping dogs among individuals with negative outcomes is overlooked, allowing sleeping dogs in the treated group with negative outcomes (SDTN) to be assigned equal even higher priority than persuad- ables in the control group with negative outcomes (PECN) in the model ranking. This imbalance enables biased uplift models to attain artificially inflated curve values simply by ranking treated individuals with positive outcomes (I' P) above others, potentially surpassing unbiased models that more accurately differentiate between these subgroups.
2. Preliminaries
2.1. Problem Setup
In uplift modeling, particularly in the context of randomized controlled trials (RCTs), we observe n units in the dataset
D = {(x,, &4, y:))== where x, E X C I29 represents base- line covariates (e.g., product features), t; E T = {0, 1} denotes binary treatment assignment (e.g.. whether ad- vertisement is delivered), and y/ E ) = {0, 1} indicates the binary outcome (e.g., whether the product is purchased), Here, q is the dimension of covariates.
Following the Neyman-Rubin potential outcome framework (Rubin, 1974; Rosenbaum & Rubin, 1983), we denote Y:()) as the outcome for unit i under treatment T t. The individual treatment effect (ITE) is defined as r; = (1) (0). Since only one of the potential outcomes, (1) and (0), can be observed, the ITE is unobservable. Instead, uplift models estimate the Conditional Average Treatment Effect (CATE), defined as: T T(x) = (1) - (0) | X; = z/. We adopt standard causal assumptions (Imbens & Rubin, 2015) to ensure the identifiability of the CATE. Based the CATE value, individuals can be classified into four strata:
To address this limitation, we propose the Principled Uplift Curve (PUC), a novel evaluation metric that assigns equal importance to individuals with both positive and negative outcomes. As shown in Figure 1, the PUC achieves a bal- anced assessment by assigning equal value to PETP and PECN as well as to SDCP and SDTN This can then pro- vide more accurate evaluation than conventional uplift and Qini curves. To directly leverage the PUC in model training, we introduce the Principled Uplift Loss (PUL.) function and incorporate it into our uplift modeling framework, the Prin- cipled Treatment and Outcome Network (PTONet). Specif- ically, PTTONet employs three-headed neural network as its backbone and incorporates the targeted regularizer from Shi et al. (2019) alongside our proposed principled uplift loss function to enhance the model's ability to rank CATES effectively. We conduct extensive experiments on simulated data, real-world Criteo data, and real-worid Lazada data to demonstrate the superior performance of the Principled Uplift Curve and PTONet.
Persuadable (r(z) > 0) Y = 1 and Y Y()) 0; Sure thing (T(x) = 0): Y(1) = and Y(0) = 1;
Lost cause (r(x) = 0) : Y()) = and Y(0) = 0; Sleeping dog (r(x) < 0): Y = 1 and Y(1) = 0.
We denote the persuadable, sure thing, lost cause, and sleep- ing dog strata as PE, ST. LC, and SD. respectively. As shown in Figure 1, uplift modeling primarily focuses on the PE and SD strata, as these groups have zero CATE val- ues. Superscripts TP, TN, CP, and CN denote treatment (T 1) and control (T = 0) groups with positive (Y = 1)
Related Work. Recent research in uplift modeling includes meta-learners (Künzel et al., 2019), tree-based methods (Rzepakowski & Jaroszewicz, 2010; 2012), and deep learn-
2

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
or negative (Y~ 0) outcomes. For example, PETP rep- resents the persuadable individuals in the treatment group (T' = 1) with positive outcomes (Y = 1).
Based on this definition, the value function for the com- monly used Separate Uplift Curve (SUC) (Diemert Eustache et al., 2018) is defined as:
After estimating the CATE for all individuals, we rank them in descending order. An unbiased uplift model should rank persuadable individuals above others, especially ahead of sleeping dogs. Optimizing and evaluating this CATE rank- ing the primary objective of uplift modeling.
VsUO(k, S) = (D,)) |2]
RG(D, k) |C|
(3)
where the notations [T] and |C] represent the total number of individuals in the treatment and control groups, respectively.
Without loss of generality, we use SUC as the represen- tative of the conventional curve. Other commonly used metrics include the Separate Qini Curve (SQC) (Radeliffe, 2007; Radcliffe & Surry, 2011), 1), Joint Uplift Curve (JUC) (Gutierrez & Gérardy, 2017), and Joint Qini Curve (JQC) (Devriendt et al., 2020), The specific formulations of these metrics are provided in Appendix B.
2.2. Definitions of Uplift and Qini Curves
Since the true CATE values are unknown in real-world datasets, the evaluation of uplift models depends user- provided ranking rules. These rules can be formulated by CATE estimates, hard-coded heuristics, or other methods (Yadlowsky et al., 2024). We define the ranking rule as:
Definition 2.1. A ranking rule is defined by score function S x x + R, where samples i = I,..., n are ranked in descending order of S(D:). A higher value of (())) indicates higher priority for treatment. Let i(j) denote the sample corresponding to rank j. and π (ki, S) = ((0))= represent the top k individuals based on ranking score S.
To quantify the performance of uplift models, we further define the Area metric as follows:
Area(S) - - /11 1) 1(0) (V(K,5) + V(K+ (4)
where [4] is the total number of indexes in L. This Area metrie represents the integral of the curve (kk, over the intervals [kk). Then, we employ the Area Under the Uplift or Qini Curve (AUUQC) (Gutierrez & Gérardy, 2017; Zhang et al., 2021), defined as follows:
The uplift and Qini curves are commonly used to evaluate the individual rankings of uplift models. During evalua- tion, three key ranking rules are considered:
Model ranking rule: SModel (D.) = =f(x)). based on the predicted CATE from the uplift model +(x.):
AUUQC(S) Area -
(5)
Area(SMax) Area(Saaraaaa (SRandom)
Random ranking rule: SRandom (D:) = Random(D) obtained by randomly permuting the dataset D;
AUUQC represents the ratio of the area enclosed by the curves of the Model ranking rule and the Random ranking rule to the area enclosed by the curves of the Max ranking rule and the Random ranking rule.
Max ranking rule: (D;; I(y; I)(T(t 1) I[tt, = 0)), which assigns highest priority to individuals who experienced positive outcome under treatment and penalizes those with positive outcomes under control.
3. Can Uplift and Qini Curves Truly Assess Causal Effect Rankings?
Given the ranking rules, we define distinct value indices of the uplift and Qini curves as follows:
Conventional uplift and Qini curves are commonly applied in uplift model evaluation because they approximate the true Average Treatment Effect (ATE) when the CATE ranking is accurate (Yadlowsky al., 2024). In the absence of true ATE, the magnitude of these curves is commonly used as proxy for the true ATE, with higher curve values interpreted as indicative of better CATE ranking performance.
1= {i 5(D++1) A S(D,), e ((n 1,5) n0. (1)
The inclusion of {0, n ensures full coverage of the curve's endpoints. The set ideetifies the positions where the uplift score changes (breakpoints), enabling efficient computation of the curve's cumulative values. We denote I(k: as the index of the k-th district value.
Despite their widespread use, a key question arises: Do higher values on conventional uplift and Qini curves truly re- flect more accurate CATE rankings? To explore this, we em- ploy simulated data with known individual treatment effects to compare rankings produced by SModel, SRandom. and SMax: evaluating these rankings with conventional curves.
Following Devriendt et al. (2020), we define the cumulative number of individuals with positive outcomes in treatment and control groups based on the ranking rule S as follows:
Σ 1(k)
(D(D,k)=
E
==1(1) T(4 1)T(v= 1),
As shown in Figure 2, the worst and best cases emerge when ranking PECN and SDTN using SMax from SUC, where these two strata are assigned equal priority and thus appear
(2)
Σ 1(K)
(D(
(=) I[4: 0)I(m= 1).
3

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Table The contribution of difterent individuals to the value functions of various uplift and Qini eurves. Columns 7 and C represent the total contributions from the treatment and control groups, respectively. The contribution of PET exceeds that of PE and SDTN contributes more than SDCP, Additionally, the column has a greater impact than the C' column.
Curve
PFTP
PFON
STTP
STOP
LCTN
L.CN
SD
SDCP
T
C
SUC
1
0
1
0
0
0
2
[
T
|0|
0
71
[C|
PUC
1
1
1
-1
1
-1
-1
0
0
Max Curve Evaluation
Individual Rankings of Conventional Curve (index represents the ranking order)
section). The results show that the AUUQC values for the biased model, f(2), surpass those of the unbiased model, {(), across all conventional curves."
Worst ranking case 2. PTTP3. SDTN LCCN
T.
1.PE" TT P.P LCCW 4. TM SD 7. ST
To understand why the conventional curves fail, we quan- tify the contribution of individuals to each curve's value function. For example, in the SUC value function, the con- tribution of treatment group individual with a positive outcome is dVsuc((.S) [ x x 0 . and the total treatment group contribution and control groups [T and .ল. A similar approach applies to SQC, JUC, JQC and PUC.? As shown in Table I, in the SUC row, the
Best ranking case
Model
140
Random
Max
120
100
Curve Values
80
60
40
20
Table 2. Case study of two uplift models, ((1) and f(2) with their estimated CATE values for eight individuals, labeled Dı Da. each drawn from different subgroups.
0
100
200
300
400
500
Ranking of individuals
Uplitt Model
Dr PgCN
D2 LC
PE
STOP
STCP
LOTN
8D
SD
Figure The worst best cases for individual ranking using SMax in SUC, which arise from the randomness in ranking equally prioritized individuals and reflect the lowest and highest consis- tency with the true CATE ranking. The axes show individual index and curve values. The blue, yellow, and green lines represent the Model curve (based on SModel), the Random curve (based on SRandom). and the Max curve (based on SMax).
20 #
0
0
Table 3. AUUQC values for different curves across eight samples
ranked by two uplift models. Bold indicates optimal performance,
while underlined values denote suboptimal performance. The
symbol (1) indicates that higher AUUQC values are desirable.
in random order. The ranking most consistent with the true CATE, which represents the best case, is achieved when all PECN individuals are placed ahead SDTN whereas the least consistent case, representing the worst case, occurs when this order is reversed. Results show that even in the best case, the conventional curves fail achieve the correct ranking, and in the worst case, they may rank sleeping dogs ahead of persuadable individuals.
Uplift Model
SUC (?)
SQC (T)
JUC (1)
JQC (1)
PUC (1)
f(L)
0.500
0.500
0.667
0.500
0.613
((2)
1.000
1.000
0.833
1.000
0.484
TP individuals contribute more to the conventional curves than others, while CP individuals contribute least. This focus on individuals with positive outcomes undervalues the PECN relative to PETP while overvaluing SD'N' relative to SDCP, ultimately making the value of PEON compara- ble to that of S SD'N This imbalance even leads to a higher overall contribution from the individuals in the treatment group compared to those in the control group. Thus, an up- lift model that maximizes conventional curves will rank TP individuals at the top, CPP. individuals at the bottom, and all other individuals in between. This ranking strategy can lead
To further validate the phenomenon illustrated in Figure 2, we conduct a case study using two uplift models. Table presents the estimated CATE values for eight individuals, labeled D, through Da. each drawn from different stratum. unbiased estimator, whereas (2) is
Here, f(1) serves biased-it captures incorrect causal effects but coincidentally aligns with the ranking rule SMax of conventional curves. Table displays the AUUQC values of the conventional curves (SUC, SQC, JUC and JQC) and the principled uplift curve (PUC, which will be formally introduced in the next
For detailed calculation process of the conventional and prin- cipled curves, please refer to Appendix C.
FFor details, please refer to Appendix D.

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
incorrect rankings, such as placing PECN 'ndividuals below SDTN individuals.
Max Curve Evaluation
Individual Rankings of Principled Uplift Curve (index represents the ranking order)
1.
P.
5.5D
7.
This leads to an important conclusion: an incorrect ranking can cause the AUUQC of conventional curves to exceed that of a correct ranking. In some cases, it may even produce the maximum AUUC value. Beyond the case studies, this con- clusion is further supported by analyzing the conventional curve value functionn. For sufficiently large RCT sample sizes, the shared components A (D, I:) of conventional curve value functions can be expressed as:
Worst ranking case
2.
4.P
6.SDCP
8.
1.
3.STiP
5.
7.SD
Best ranking case
2. P.
4.1.CCN
6.
8.
Model
250
Random Max
200
Curve Values
(D),)) = (1),k) RC k))
100
(6)
Σ
P(t: 1, y =
P(t+ = 0, 0. =xx=1 1).
50
100
200
300
400
500
Although this difference in probabilities is related to the true ATE, it is important to note that in the context of CATE ranking, large value of E/(k) P(4, 1, 3/ = 1) leads to an increase in the conventional curve value. Consequently, ranking STTP and PETP ahead of other individuals cre-
Ranking of individuals
Figure 3. The worst case and best case for individual ranking of SMax in PUC. The meaning of the coordinates, legends and table headers similar to that in Figure 2.
ates an incorrect but inflated Σi=(1) P(t, 1, y; = 1) in the top (kk individuals, thus exaggerating the conven- tional curve value. Similarly, ranking all treatment group individuals above control group individuals also inflates the conventional curve value. This inflation effect is further amplified when calculating the AUUQC.
To address this imbalance, we propose the Principled Uplift Curve (PUC), which assigns equal curve values to individ- uals with both positive and negative outcomes, ensuring a more balanced evaluation. The ranking rule that maximizing PUC value is defined as follows:
This reveals fundamental limitation of conventional curves: they are susceptible to manipulation by suboptimal ranking strategies, inflating the performance of biased uplift models. Although the inclusion of denominators such as [T] and |C| in the conventional curve value function partially mitigates this issue, Table shows that remains insufficient. This explains why the biased model 4 shows higher AUUQC than the unbiased model +(1) in Tables 2 and 3. In (2)), the higher ranking of STTP and PETP increases the treatment group's contribution, inflating the conventional curve value.
SMax((i) = I(y = (I =1)-I(t: = =0)
(8)
+ I( (I(t 0) I(t = 1)).
This ranking rule achieves optimality when all samples be- longing to the TP and CN groups are ranked ahead of those belonging the TN and CP groups
Correspondingly, the value function of the principled uplift curve is formulated as follows
Vpuc 5) =
(D,
4. Principled Uplift Curve and PTONet
(D,A) -R (D,
Based on the above analysis, we conclude that conventional uplift and Qini curves primarily focus individuals with positive outcomes, while undervaluing the potential value of individuals with negative outcomes in both the treatment and control groups. The two terms can be formulated as:
The intuition behind this formulation is that persuadable individuals are primarily found in the TP and CN groups, whereas sleeping dogs are mostly in the TN and CP. Con- sequently, T and CN are ranked above TN and CP,, with both pairs having equal priority.
(D, = Σ I(k)
(=)()) I(44 = 1)I((; = 0),
The above ranking rule and value function ensure accurate evaluation of uplift models by equitably valuing individuals with positive and negative outcomes.
Σ T(k)
N (D.)
(0)) I(t: 0)I(n, 0).
Proposition 4.1 (Properties of PUC). The principled uplift curve exhibits the following properties: It differentiates
This undervaluation leads to an imbalance in the contribu- tions of individuals from different strata to the curve values, resulting in biased evaluations of existing metrics. For in- stance, the value of PFTP exceeds that of P PFON
We explore the relationship of PUC and conventional curves in Appendix F and detail eight PUC variants in Appendix
5

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Representation leumnngg
Propensity score estimator
by uplift models. This encourages the model to assign higher values of (1x) to samples in the TP and CN groups, and lower values to those in the TN and CP groups.
=X0
Treatment reconstructor
,,, 7)
L
We propose the Principled Uplift Loss (PUL) function to implement this idea, defined as follows:
x
(X)
Outcome catimator
C (D) (P) E (C ((;)).
by (X, T)
(11)
T
Here, BCE((,)) .) is binary cross entropy loss function, and (z)) I+xxp(-f(x)) is the sigmoid function of the esti- mated CATE f(x). The PUL function guides the model to assign higher CATE values to persuadable individuals (comprising PETP and PE than sleeping dog indi- viduals (comprising SDTN and SDCP), thereby ranking persuadable individuals ahead of sleeping dogs.
Guided by principled uplift curve
Figure 4. The architecture of PTONet. Circles represent random variables; trapezoids denote functions; square rectangles depict function outputs; and rounded rectangles denote losses.
To address the second principle, we design the treatment and outcome loss function CTO, which simultaneously han- dles treatment and outcome prediction as well as treatment assignment bias. is formulated as follows:
persuadable individuals from sleeping dogs. 2) Within each of these two strata, individuals from the treatment and con- trol groups contribute equally to the curve value.
CTo (TO (x, (4) + L (xt 3k)
For proof of Proposition 4.1, please see Appendix E.
+ (xi, 4.) + aL (0,,4i,3))
(12)
We can better understand Proposition 4.1 by referring to the earlier examples. As demonstrated in Table 1, although the PUC does not fully distinguish among all four strata, effectively separates the persuadable and sleeping dog groups by incorporating the value function for individuals with negative outcomes. Furthermore, it ensures consistent value assignment within the treatment and control groups across these strata. As shown in Table 3, the PUC assigns a higher AUUQC score to the unbiased model f(!) than to the biased model ((2), demonstrating its ability to more accurately evaluate uplift models. Furthermore, a compar- ison between Figure 2 and Figure 3 reveals that the PUC consistently ranks persuadable individuals above sleeping dogs, even under the worst-case scenario.
The treatment loss function, L (I;; BCF((1, hr (1t f4)), aims to reconstruct the treatment assignment, preserv- ing treatment information even in high- sigh-dimensional covari- ate spaces. The outcome loss function, L (x 44,3)) U1)
BCE(V, by (zx, 44)), models the outcome given treatment and covariates. The balance loss function, C (x,, 44) = BCE(t,, e(zx)), estimates the propensity score. Finally, the targeted regularizer is defined as CTR(x,, 44 v:) BCE(y, by (z,, 4,) + + (((())] - which addresses treatment assignment bias and enhances model scalability. Here, is learnable parameter, and a is a hyperparameter.
Building upon the loss LTO we design a three-headed learner named Principled Treatment and Outcome Network (PTONet). The architecture of PTONet is presented in Fig- ure 4. The final loss of the PTONet is defined as:
Next, we propose helper function to guide the uplift mod- cling. Based on Proposition 4.1, the principled uplift curve suggests that an effective uplift model should satisfy the following principles: 1) rank persuadable individuals above sleeping dogs, and 2) minimize treatment assignment bias.
(D) C CTO = CTO (D) + PC (D),
(13)
where 33 is hyperparameter.
We emphasize that the core contribution of PTTONet lies in the introduction of the PUL function. PTONet is specifically designed as an illustrative uplift model tailored to the PUL function CPU with its backbone based on a variant of Drag- onNet (Shi et al., 2019). In particular, we adapt DragonNet by replacing its original two outcome heads with a single head for treatment reconstruction and another for outcome prediction. Detailed descriptions of each module in PTONet are provided in Table 6 and Appendices H and I.
To address the first principle, inspired by SMax in the PUC, we design a helper function as follows*:
g(4: v/) T(y; = f,),
(10)
which assigns a value of to samples in the TP and CN groups, and value of 0 to those in the TN and CP groups.
Using g(ts, y.) as a proxy label, we can train binary classi- fier to regularize the estimated causal effect f(x;) produced
'oor detailed discussion of treatment assignment bias and targeted regularizer, please refer Appendix H and Section 3 in Shi et al. (2019).
"To adapt the design of the BCE loss, we set g(t,, v)) instead of g(t,, yi) = -1 for TN and CP individuals.
6

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Table Performance comparison (meaniss) on synthetic data. SUC, SQC, JUC, JQC, and PUC denote their AUUQC values. The symbol (J) indicates lower is better; (1) indicates higher is better. The "TRUE" row represents the results of the true CATE. Compared to conventional curves, PUC values more closely approximate AUTGC. Moreover, models guided by PUC, including S-Learner (PU) and PTONet, consistenly demonstrate superior performance on both PUC and AUTGC.
Unbalanced Conventional Evaluation Metric
Balanced Evaluation Metric
Uplift Model
PEHE (L)
SUC (f)
SQC (t)
JUC (t)
JQC (t)
PUC (1)
AUTGC (1)
TRUE
0.000
0.835
0.586
0.779
0.581
1.000
1.000
T-Learner
1.
0.50220.31
0.35222.22
0.506
0.356
0.614
0. 0.704
TARNel
10
0.343
0.241
0.344_0.a0
0.243
40.31
0.565
CFRNet
1.246
0.320
00227
0.301 20.36
0.2222.022
0.384
0.528
DragonNet
1.055.40.85
0.570
0.519 =011
0.570
0.519 +0.9 .88
0.636
0.739 23
EUEN
1.117++0.04
0..07
0.285-0.24
0.381=0.30
0.269 +0.25
0.472+0.43
0.603
DESCN
1..23
0.444
0.404=0.33
0.49
0.4044...51
0.418
0.569
EFIN
1.868
0.4633:0.29
0.414=0.20
0.428
0.402
0.448
0.648
S S-Learner
1.209
0.4
0.347=0.25
0.476=00311
0.345
0.609
0.700
S S-Learner (U)
1.00940.25
0.667
0.468 10,22
0.631
0.467
0.800
0.847 10.28
S-Learner (PS)
1.01910.23
0.69010.29
0.484_0.20
0.654.0.27
0.48440.20
0.828
0.868 10.26
S-Learner (PU) PTONet
0.879 10.16
0.786
0.552 10.11
0.738 10.15
0.548 1..11
0.943 1..19
0.957 10.15
00
0.780
0.547=20.1
0.746
0.546
0.948
0.961 0.961
5. Experiments
SQC
S-Lenmer
T-Leamer
We conduct experiments to address the following three key research questions:
JUC
TARNet
CFRNet
DragonNet
RQQ)) Does the principled uplift curve yield results closer to the true CATE than other metrics?
BUEN
0.2
SUC
DESCN
EFIN
-RQ2) Does the principled uplift loss function effectively improve the CATE ranking ability of uplift models?
0.6
Kendall correlation
JQC
RQ33) How does the performance of PTONet compare to existing methods?
PUC
5.1. Experimental Setup
Figure 5. The Kendall (higher values are desirable) between AU- UQC and AUTGC sequences of uplift models synthetic data, evaluated for SUC, SQC, JUC, JQC, and PUC. PUC achieves the highest Kendall, followed by SUC and JUC.
We compare the performance of uplift models and their evaluation curves using a synthetic dataset and real-world Criteo dataset (Diemert Fustache et al., 2018; Diemert et al., 2021). In Appendix we present experimental results on the Lazada dataset (Zhong et al., 2022) to validate the scala- bility of the proposed method on higher-dimensional data. For all datasets, we conducted 50 performance comparison experiments, changing the random seed from 0 to 49.
training, validation, and test sets in an 8/1/1 ratio.
Evaluation Metrics. addition to the uplift and Qini curves discussed Table 7, we introduce three metries for the model evaluation and curve evaluation.
Dataset Description. The synthetic dataset contains n 10,000 units with q = 10 different covariates. The real- world Criteo dataset (Diemert Eustache al.,, 2018; Diemert et al., 2021),, open sourced by Critco AI Labs, is utilized for uplift modeling in a large-scale advertising scenario. It includes 25,309,483 instances with 1l continuous features, a binary treatment, and 2 candidate outcomes (visits and conversions). For our analysis, we specifically consider the visits variable as the outcome, We split the two datasets into
The first is the expected Precision in Estimation of Hetero- geneous Effect (PEHE, (Shalit et al., 2017)), defined as:
PEHE [E(((X - {)
(14)
This metric does not account for prioritization, making it an unbalanced conventional evaluation metric.
The second metric the Area Under True Gain Curve

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Table Performance comparison sde of AUUQC) on Critco data. PTÖNee achieves superior performance on PUC.
Unbalanced Conventional Evaluation Metric
Balanced Metric
Uplift Model
SUC (1)
SQC (1)
JUC (1)
JQC (1)
PUC (1)
S-Learner
0..08
0.105
0.03
0.085
0.141 10.02
T-Learner
0.356+0.32
0.073+0.07
0.022=0.02
0.057=0.06
0.085+0.11
TARNet
0.51220.04
0.103
0.036g0.01
0.091
0.104
CFRNet
0.276
0.057
0.034.4.002
0.044
0.105
DragonNet
0.518+0.02
0.107 #0.01
0.034=0.01
0.087
0.137
EUEN
0.509
0.105+0.01
0.033
0.085
0.186+0.01
DESCN
0.55610.06
10.04
0.02910.00
0.076..0.02
0.07210.05
EFIN
0.504
0.104
0.033
0.084
0.127
PTONet
0.50220.01
0.103
0.034
0.086
0.191
Table 6. Ablation studies (mean of PTONet modules on synthetic data. All modules enhance PTONet's performance.
Unbalanced Conventional Evaluation Metric
Balanced Evaluation Metric
Uplift Model
PEHE (L)
SUC (1)
SQC (1)
JUC (f)
JQC (1)
PUC (1)
AUTGC (f)
TRUE
0.000
0.835
0.586
0.779
0.581
1.000
1.000
PTONet w/o CPU
0.883 10.13
0.780
0.547
0.746 10.1
0.546610.09
10.15
0.961 10.11
1.138 36
0.52444.33
0.3684+.27
0.532+0.34
0.377 25
0. 0.659+0.45
0. 0.739
w/o CTR
0.84640.15
0.760+0.20
0.533+0.14
0.737+0.12
0.533+0.12
0,936+1.22
0,951+n.17
w/o L
1.059 0.33
0.533
0.374:0.26
0.545
0.380
0.678
0.753
w/o LT
1.012
0.60440.35
0.4244
0.601
0.429
0. 0.756+0.11 11
0.813
(AUTGC), which is formulated as follows:
by evaluating their correlation with AUTGC.
Area (SModa) ) Area (SRandem)
Since AUTGC is unobservable in real-world settings, use synthetic data. We compute the AUUQC and AUTGC metrics for uplift models trained on synthetic data over m = 50 epochs and quantify the rank correlation between these sequences using the Kendall coefficient.
AUTGC(S) =
(15)
Arca (STru) Arca (SRandom)
where STrue (x) = (xx) represents the true CATE ranking. To assess the alignment between curves and true CATE rankings, the Kendall correlation (Kendall, 1938) between the AUUQC values of m models and their AUTGC values calculated follows:
As shown in Figure 5, all AUUQC sequences positively correlate with AUTGC. However, except for the PUC, other metrics fail to fully align with AUTGC trends (Kendall < 1), indicating that high values of conventional curves do not reliably reflect strong CATE ranking performance. In contrast, the PUC achieves perfect alignment with AUTGC (Kendall = 1), making it a reliable metric for evaluating CATE ranking uplift models.
Kendall= m(m 1) Σ
I(AUTGC,
(16)
AUTGC,) I(AUUQC, AUUQC,)
A higher Kendall value indicates a closer alignment of the uplift and Qini curve with the true CATE ranking.
Baseline Implementations. We evaluate eight uplift models as baselines, including S-Learner (Künzel et al., 2019), T- Learner (Künzel et al., 2019), TARNet (Shalit et al., 2017), CFRNet (Shalit et al., 2017), DragonNet (Shi et al., 2019), EUEN (Ke et al., 2021), DESCN (Zhong et al., 2022) and EFIN (Liu et al., 2023). All basclines follow the same parameter tuning process, please see Appendix for details.
In addition to the results shown in Figure 5, all subsequent experiments based on synthetic data (including Tables to 6) further validate the alignment between PUC and AUTGC. Notably, some models that perform well on conventional metrics but fail to achieve comparable results on AUTGC.
In contrast, models that perform well on the PUC metric consistently exhibit strong performance on AUTGC as well.
Comparing Loss Functions through Simulation. To ad- dress RQ2, we first verify that uplift models guided by the PUC outperform conventional curves and other metrics. Fol- lowing Devriendt et al. (2020), we establish an uplift loss
5.2. Experimental Results
Comparing Evaluation Metrics through Simulation. To address RQI, we assess uplift and Qini curve effectiveness
8

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
function L prioritize individuals in the treatment group with positive outcomes over others, and propensity score loss function L to guide the CATE for treatment group individuals exceeds that of control group individuals. The loss functions are formulated as follows:
6. Conclusion
This paper reveals the limitations of previous uplift and Qini curves in evaluating uplift models, showing that they are vulnerable to manipulation by suboptimal ranking strategies, which can inflate the performance of biased models. We propose the Principled Uplift Curve (PUC), a new metric that balances the value of individuals with both positive and negative outcomes, providing a more accurate evaluation of uplift models. We also propose PTONet, a PUC-guided uplift model that directly optimizes the uplift model by max- imizing the PUC value. Extensive experiments demonstrate the effectiveness of the PUC for uplift model evaluation and of the PTONet for uplift model optimization.
(D,)) (CE T)T = (m))) (D,) = BCE(I((+ (x))(
(17)
For a fair performance comparison, augment the S- Learner with L CPS, and LP S-Learner (U), S- Learner (PS), and S-Learner (PU).
As shown in Table 4, although the S-Learner (PU) does not outperform the S-Learner (U) in terms of AUUQC for SUC, SQC, and JQC, it achieves the highest scores for both AU- UQC of PUC and AUTGC. Furthermore, when compared to other uplift models, introducing the PUL function solely into the S-Learner allows it outperform all models ex- cept PTONet on PUC and AUTGC. These findings suggest that models guided by the PUC metric produce results that are more robust and better aligned with the ground truth. The experimental results in Table 11 of Appendix I provide further empirical evidence supporting this conclusion.
A Acknowledgments
This work was supported in part by the National Key Research and Development Program of China (2024YFE0203700) National Natural Science Foundation of China (62376243, 62441605), and "Pioneer" and "Lead- ing Goose" R&D Program of Zhejiang (2025C02037). This work was also supported by Ant Group. All opinions in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
Additionally, we report the best performance of each model across multiple experiment rounds, shown in Table 12 in Appendix I. The uplift models optimized with L and CPS outperform the TRUE row in SUC, SQC, JUC, and JQC but underperform in PUC and AUTGC. This confirms that maximizing conventional curve values can introduce bias, leading to an inflated conventional curve value in a biased model. In contrast, our balanced evaluation metrie, PUC, remains unaffected by this issue, even when the model is directly trained with the PUL function LPU
Impact Statement
We trust that our work offers a meaningful contribution to the uplift modeling and causal ranking community. In this field, the pivotal yet under-addressed challenge is how to as- sess and guide the formulation of uplift models in real-world data scenarios. The contribution of our paper is that we fun- damentally uncover the issues associated with conventional uplift and Qini curves. In response, we propose novel and effective metric to address these problems, and furnish strategy for leveraging this metric to optimize uplift models. This work also has positive implications for the application of causal inference methods in real-world domains. Accord- ing to our findings, in areas such as healtheare and finance, existing evaluation metrics tend to favor individuals who have received treatment or have purchase history (Y = 1), which may inadvertently disadvantage those without such history (Y = 0) and encourage decisions that could harm the sleeping dog group. In contrast, our proposed PUC metric, along with the corresponding PTTONet model, better balances the interests of both groups. This promotes fairer decision -aaiing and helps protect the sleeping dog group, thereby supporting no-harm policy implementation.
Comparing the Performance of Uplift Models. To address RQ3, we report the performance comparison results on the synthetic dataset in Table 4, the Criteo dataset in Table 5 and the Lazada dataset in Table 13.
The results on synthetic data show that PTONet outper- forms other uplift models on PEHE, AUUQC of PUC, and AUTGC. For the real-world Criteo and Lazada dataset, de- spite the absence of the true CATE, the results indicate that PTONet outperforms others on the AUUQC of PUC.
To investigate the function of each module in PTONet, we conduct ablation studies as summarized in Table 6. The results show that LPU LT, and " significantly improve PTONet's performance on both PUC and AUTGC. Since CTR is specifically designed to address implicit treatment assignment bias, its limited effect on RCT data is expected. Among all components, the PUL objective function remains the most critical factor driving PTONet's effectivvness.
References
Belbahri, M., Murua, A., Gandouet, O., and Partovi Nia, V. Qini-based uplift regression. The Annals of Applied
9

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Statistics, 15(3):1247-1272, 2021.
Morgan, S. L. and Winship, C. Counterfactuulls and causal inference. Cambridge University Press, 2015.
Betlei, A., Diemert, E., and Amini, M.-R. Uplift modeling with generalization guarantees. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 55-65, 2021.
Radeliffe, N. Using control groups to target on predicted lift:
Building and assessing uplift model. Direct Markeling Analytics Journal, pp. 14-21, 2007.
Devriendt, F., Van Belle, J., Guns, T., and Verbeke, W.
Radcliffe, N. J. and Surry, P. D. Real-world uplift modelling with significance- uplift trees. White Paper TR- 2011-1, Stochastic Solutions, pp. 1-33, 2011.
Learning to rank for uplift modeling. IEEE Transactions on Knowledge and Data Engineering, 34(10):4888 4904, 2020.
Renaudin, C. and Martin, M. About evaluation met- rics for contextual uplift modeling. arXiv preprint arXiv:2107.00537, 2021.
Diemert, E., Betlei, A., Renaudin, C., Amini, M.-R., Gre- goir, T., and Rahier, T. A large scale benchmark for individual treatment effect prediction and uplift modeling. arXiv preprint ArXiv:21 10106, 2021.
Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
Diemert Eustache, B. A., Renaudin, C., and Massih-Rest Reza, A. A large scale benchmark for uplift modeling. Pro- ceedings of the AdKDD and TargeiAd Workshop, KDD, London, United Kingdom, August, 20, 2018. ACM, 2018.
Rubin, D. B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of edu- cational Psychology, 66(5)6688, 1974.
Gutierrez, P. and Gérardy, J.-Y. Causal inference and uplift modelling: A review of the literature. In International conference on predictive upplications and APIs, pp. 1-13. PMLR, 2017.
Rzepakowski, P. and Jaroszewicz, S. Decision trees for uplift modeling, In 2010 IEEE International Conference on Data Mining, pp. 441-450. IEEE, 2010.
Imbens, G. W. and Rubin, D. B. Causal inference statis- tics, social, and biomedical sciences. Cambridge univer- sity press, 2015.
Rzepakowski, P. and Jaroszewicz, S. Decision trees for up- lift modeling with single and multiple treatments. Knowl- edge and Information Systems, 32:303-322 327, 2012
Kallus, N. What's the harm? sharp bounds the fraction negatively affected by treatment. Advances in Neural Information Processing Systems, 35:15996-16009, 2022.
Shalit, U., Johansson, F. D., and Sontag, D. Estimating individual treatment effect: generalization bounds and algorithms. International conference on machine learn- ing. pp. 3076-3085. PMLR, 2017.
Kallus, N. Treatment effect risk: Bounds and inference. Management Science, (9(8):4579-4455 4590, 2023.
Shi, C., Blei, D., and Veitch, V. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
Ke, W., Liu, C., Shi, X., Dai, Y., Philip. Y., and Zhu, X. Addressing exposure bias in uplift modeling for large- scale online advertising. In 2021 IEEE International Conference Data Mining (ICDM), pp. 1156-1161. IEEE, 2021.
Sun, Z., Han, Q., Zhu, M., Gong, H., Liu, D., and Ma, Robust uplift modeling with large-scale contexts for real-time marketing. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Data Mining V. 1, pp. 1325-1336, 2025.
Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1/2):81 1938.
Künzel, S. R., Sekhon, J. S., Bickel, J., and Yu, B. Met- alearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):415-4111 2019.
Verbeken, B., Guerry, M.-A., and Verboven, S. Uplift model evaluation with ordinal dominance graphs. In European Conference on Machine Learning and Princi- ples and Practice of Knowledge Discovery in Databases- ECML/PKDD 22. ECML/PKDD'22 Uplift Modeling Tu- torial & Workshop.. 2022.
L Liu, D., Tang, X., Gao, H., Lyu, F., and He, X. Explicit fea- ture interaction-awant aware uplift network for online marketing. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4507-4515, 2023.
Wang, Y., Li, H., Zhu, M., Wu, A., Xiong, R., Wu, F,, and Kuang, K. Causal inference with complex treatments: survey. arXiv preprint arXiv:2407. /4000, 2024.
10

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Wu, A., Kuang, K., Zhu, M., Wang, Y., Zheng, Y., Han, K., Li, B., Chen, G., Wu, F,, and Zhang, K. Causality for large language models. arXiv preprint arX1v:2410.15319 2024.
Wu, A., Kuang, K., Xiong, R., and Wu, F. Instrumental variables in causal inference and machine learning: A survey, ACM Computing Surveys, 2025.
Xiong, R., Chin, A., and Taylor, S. J. Data-driven switch- back experiments: Theoretical tradcoff's and empirical bayes designs. arXiv preprint arXiv:2406.06700 2024.
Xu, J., Hong, N., Xu, Z., Zhao, Z., Wu, C., Kuang, K., Wang, J., Zhu, M., Zhou, J., Ren, K., et al. Data-driven learn- ing for data rights. data pricing, and privacy computing. Engineering. 25:66-76, 2023.
Yadlowsky, S., Fleming, S., Shah, N., Brunskill, E., and Wager, S. Evaluating treatment prioritization rules via reighweighted average treatment effects. Journal of the American Statistical Association, pp. 1-14, 2024.
Yamane, I., Yger, F,, Atif, J., and Sugiyama, M. Uplift modeling from separate labels. Advances in Neural Infor- mation Processing Systems, 31, 2018
Zhang, W., Li, J., and Liu, L. A unified survey of treat- ment effect heterogeneity modelling and uplift modelling. ACM Computing Surveys (CSUR), 54(8):1-36, 2021.
Zhao, Z., Bai, Y., Xiong, R., Cao,, Q. Ma, C., Jiang, N., Wu, F., and Kuang, K. Learning individual treatment ef- fects under heterogeneous interference in networks. ACM Transactions on Knowledge Discovery from Data, 18(8): 1-21, 2024a.
Zhao, Z.,, Wu, A., Kuang, K., Xiong, R., Li, B., Wang, Z. and Wu, F. Networked instrumental variable for treatment effect estimation with unobserved confounders. IEEE Transactions on Knowledge and Data Engineering, 2024b.
Zheng, Q. Xu, Y., Liu, H., Shi, B., Wang. J., and Dong, B. A survey of tax risk detection using data mining techniques. Engineering, 34:43-59, 2024.
Zhong, K. Xiao, F., Ren, Y., Liang, Y., Yao. W., Yang, X., and Cen, L. Descn: Deep entire space cross networks for individual treatment effect estimation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4612-4620, 2022.

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
A. Detailed Related Work Discussion
Uplift Modeling Methods. Uplift modeling is widely applied in fields such as marketing, advertising, and customer retention (Xu et al., 2023; Zheng et al., 2024; Zhao et al., 2024a;b; Wu et al., 2024; Wang et al., 2024: Sun et al., 2025; Wu et al., 2025). Uplift modeling methods target persuadable individuals while minimizing unnecessary interventions for others, mainly sleeping dog customers (Gutierrez & Gérardy, 2017; Zhang et al., 2021). Current research in uplift modeling primarily encompasses two distinct approaches: One branch focuses on minimizing the estimation error of causal effects. It includes meta-learners (Künzel et al., 2019), which employ existing machine learning techniques to model individuals' responses; tree-based methods (Radcliffe, 2007; Rzepakowski & Jaroszewicz, 2010; Radcliffe & Surry, 2011; I; Rzepakowski & Jaroszewicz, 2012), which deploy specialized tree or forest structures, employing a variety of metrics for splitting criteria; and deep methods (Shalit et al., 2017; Shi et al., 2019; Devriendt et al., 2020; Ke et al., 2021; Zhong et al., 2022; Liu et al., 2023), which leverage the strengths of neural networks incorporate more intricate and adaptable architectures in modeling the response treatment, aiming for improved accuracy in CATE estimation.
The second branch of approaches seeks to directly maximize the uplift and Qini curve while concurrently regressing on observed outcomes. It includes methods that set the objective function based on maximizing the Qini coefficient directly (Belbahri et al., 2021; Betlei et al., 2021), or employing learning- tiiring----akkk techniques with constraints aimed at maximizing the Qini measure (Devriendt et al., 2020). The most relevant to our methodology is Devriendt et al. (2020), which designs an uplift helper function to guide the uplift modeling. Inspired by this work, we design distinct helper functions to demonstrate that helper functions that maximize the uplift and Qini curve can lead to biased estimates. Contrarily, our method avoids such biases, offering more accurate ranking of CATE.
Evaluation Metrics for Uplift Model. In evaluating uplift models on real-world data, the uplift and Qini curves are commonly employed, categorized as separate uplift curve (Rzepakowski & Jaroszewicz, 2010; 2012). separate Qini curve (Radcliffe, 2007; Radcliffe & Surry, 2011) 1), joint uplift curve (Gutierrez & Gérardy, 2017), and joint Qini curve (Radcliffe, 2007; Devriendt et al., 2020). A notable study is Renaudin & Martin (2021), which discourages using uplift/Qini curves for observational data evaluation but suggests their applicability for randomized controlled trials (RCT) data. However, we highlight that conventional uplift/Qini curves may yield biased evaluations even when employing RCT data. The curve most similar to ours is the ROCini presented in Verbeken et al. (2022). However, it lacks theoretical foundation and performs poorly in experiments (please refer to Table 9).
In addition, the recently proposed TOC/AUTOC metric by Yadlowsky et al. (2024) has attracted significant attention in the community, as it may help alleviate some of the known limitations associated with conventional uplift modeling metrics. TOC/AUTOC can be interpreted as an extension of the conventional uplift and Qini curves by introducing a threshold u and applying a logarithmic transformation. Specifically, this metric places greater emphasis on the top-ranked individuals. As illustrated in Figure 2 of Yadlowsky al. (2024), when only the top 10% of the population is considered, the cumulative gain measured by AUTOC exceeds that of the Qini curve. In other words, this approach further amplifies the inherent imbalance issue in conventional uplift and Qini metrics.
Consequently, we propose a novel evaluation metric to mitigate above concerns.
B. Definitions of Uplift and Qini Curves
Table 7. Definitions of commonly applied uplift and Qini curves for uplift model evaluation and selection. The notations [T] and |C] represent the total number of individuals in the treatment and control groups, respectively.
Curve Type
Formula (Value Function)
Separate Uplift Curve (SUC)
Vsuc(k; S) [2'
(D.A)
(D.b)
101
Separate Qini Curve (SQC)
Vsqa (k:, 5) = RD, k)- RR(DD, k)
T
C
Joint Uplift Curve (JUC)
Vivo S) [D( NN(DDE
(D.A).
RE(DDA)
+ (D,
NO (D.A) Viqo(k, S) = (D) - RD(
Joint Qini Curve (JQC)
No (D.E)
The uplift and Qini curves are employed to evaluate the CATE rankings of uplift models (Gutierrez & Gérardy, 2017; Zhang et al., 2021). In addition to the metrics defined in the main text, we define the cumulative number of treatment and control
12

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
group individuals in the RCT data D as follows:
[(h)
N3 (D,k)= = Σ
T(4 = 1),
==((1]
(18)
I(K)
(D,
Σ
[[(t: 0),
(=1
where I[()) is an indicator function.
Based on the above definition, we present the value functions for four commonly used implementations of uplift and Qini curves: Separate Uplift Curve (SUC) (Rzepakowski & Jaroszewicz, 2010; 2012). Separate Qini Curve (SQC) (Radeliffe, 2007; Radcliffe Surry, 2011), Joint Uplift Curve (JUC) (Gutierrez Gérardy, 2017), and Joint Qini Curve (JQC) (Devriendt et al., 2020) in Table
These four curves share the same SMax ranking rule, and therefore, they exhibit the same limitations: they tend to underestimate the value of negative outcome individuals, potentially leading to incorrect evaluations. Although they attempt to address this issue using [C), or No and NT, as shown Table 8, these problems remain significant.
Table The contribution of different individuals to the value functions various uplift and Qini curves. Columns I and C represent the total contributions from the treatment and control groups, respectively.
Curve
PETP
PEON
STTP
STOP
LCTN
LCON
SDTN
SDCP
T
C
SUC
0
0
0
0
[
[T]
[7]
|C|
SQC
1
0
I
0
0
0
T
2
2 C
C
JUC
1/0
1
1/0
0/-1
-1
1
-1
0/-1
0/-1/-2
2/1/0
JQC
3
2
3
I
2
I
0
8
PUC
1
1
1
-1
-1
1
-1
-1
0
0
C. AUUQC Value Calculation Process
The specific code of these curves is provided in the utils file the source code. For instance, the calculation formula for the SUC as follows:
First, sort the eight samples in descending order based the value of ((1) resulting in the order {2, 1, 5, 4,3, 7). Us- ing the district value index formula in Equation (I), we obtain / = {0, 2, 6, 8). The corresponding RD k) and Rg (D, k) values are {0, 2, 2} and {0, O, -1, 22. respectively. Consequently, the (suc(k, S) values are {0, 00,0.25, 0.25, 0}. The Model value for this set of valid values, computed using the area function, is given by:
Arca(SModel) = 1.5.
(19)
Next, we calculate the Max curve value as {0.0.5, 0.5, 0.5, 0} and the corresponding district value indexes are (0, 6, 8), yielding an curve value of 3.
For the Random curve, the curve value is calculated as 0, with valid values ranging from [0,0] to [4, ATTE = 0J.
Finally, the SUC is computed as:
1.5 -
AUUQC
0.5.
(20)
3-0
Similarly, we sort the eight samples in descending order based on the value of x1), resulting in the order 1, 7, 6, 5, 2, 8, 4). The Vsuc(k, S) values obtained are {0, 0.5, 0.5, 0}, and the corresponding district value indexes are 2, 6, 8). The AUUQC value for this set
13

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Next, we calculate the Max curve values (0,0 0.5, 0.5, 0} and the corresponding district value indexes (0, 6,8). 8). yielding curve value of 3.
For the Random curve, the curve value is calculated as 0, with valid values ranging from [0,0] to [4, ATE = 0].
Finally, the AUUQC is computed as:
3 0 =1
AUUQC- = 3
(21)
D. Detailed Derivation of Individual Contributions
Let's compute the derivative of (suc(k; S) with respect to individual where T and Y 1:
(k, d Rg (D,))
(G( k)
(22)
di di
[
|C1
By applying the derivative both terms:
dVsuc 9) d
d (RT(D,A)
[ di (RS(W,) k))
(23)
di
T di
So, the derivative of (D, kt) is:
d
To (D) 1(4 = 1)I((= = 1).
(24)
Similarly, the derivative of (D,k)) is:
R (D.k) I(t, = 0)](( 1).
(25)
di
Substituting the derivatives into the expression for (V.) de
we get:
dVsuc(k, S)
1(t = 1)1((,+ 1) [
l(t: = 0)4(YH+ 1).
(26)
di
c
For the specific case where T 1 and Y (treatment group, positive outcome), the indicator functions will be: [[t,: = 1) = 1, I( = 1) 1, I[t, 0) = 0 and I(y = 1.
Thus, the derivative simplifies to:
d dVsc (k, S)
1 x x0 1 x 1
(27)
di
[
[T'
So, for T and Y 1, the derivative of the uplift function Vsvc(k, with respect to is:
dVsuc(k,
(28)
di
T
This derivative indicates that the inclusion of an individual from the treatment group with positive outcome contributes !TT to the rate of change of the uplift value, which is proportional to the number of treated individuals in the dataset.
A similar approach applies to the SQC. For JUC and JQC, where the value functions depend on NY (D, k) and Ng (D. k), we simplify the contribution calculation by considering their relative magnitudes, as outlined by Devriendt et al. (2020).
As shown Table 8, the TP individuals contribute more to the conventional curves than others, while CP individuals contribute least. This focus on individuals with positive outcomes undervalues the PFON individuals relative PFTP
14

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
individuals while overvaluing SD individuals relative to SDCP individuals, ultimately making the value of PECN individuals comparable to that of SDTN individuals. This imbalance even leads higher overall contribution from the individuals in the treatment group compared to those in the control group. Thus, an uplift model that maximizes conventional curves will rank T' individuals at the top, CP individuals at the bottom, and all other individuals in between." This ranking strategy can lead to incorrect rankings, such as placing PFCN below SDTN
E. Proof of Proposition 1
The theoretical guarantee of the PUC and PUL stems from Proposition 4.1 of the original paper, which states that the CATE for the persuadable group should be greater than the CATE for the sleeping dog group. Specifically, PUL influences the PTONet in two ways: it directly balances the value of the positive outcome (Y = 1) and negative outcome (Y = 0) groups, and it focuses not only on the accuracy of the CATE effect size estimation but also on the comparison of CATE magnitudes corresponding to different individuals.
Following the distributional treatment effect theorem in Kallus (2022; 2023), we define the persuadable stratum and sleeping dog stratum as P(Y 1, Y (0) 0| x) and (Y = 0, (0) 1|XX = z). According to the properties of joint probability, we can deduce the bounds that ((Y 1, - 0|X x) S (( X = x) PY = 0|X = x)
and (1) = 0, Y (0) 1|.X 2) s (LY 0, = = 2) + (0) =||== z)). z).
Applying the two equations in data D. we define the number of persuadable individuals in total k samples as N (1)) and the number of sleeping dog individuals in total l samples as (k)))), then we have the two bounds that (k)) S RT (D, k) + NR (D, k) and (k)) S R (D, k) + +NR (D,"(D,))
The difference between the two bounds is the value function of principled uplift curve. In other words, the principle uplift measures the difference between the total number of samples that may be in the persuadable stratum and the total number of samples that may be in the sleeping dog stratum. Thus, the proof of Proposition 1.1 is done that the principled uplift becomes larger as the persuadable individuals increase, and becomes smaller as the sleeping dog individuals increase.
As demonstrated by the analysis in Tables I to 3, the principled uplift curve value associated with the number of treatment group samples that may be in the persuadable stratum equals the number of control group samples within the same stratum. Similarly, the value corresponding to the number of treatment group samples that could belong to the sleeping dog stratum equates to the value for the number of control group samples in the sleeping dog stratum. Consequently, the ranking of treatment and control groups does not affect the curve value of the principled uplift. The proof of proposition 1.2 is done.
F. Principled Uplift Curve versus Conventional Curves
In this section, we transform the PUC into a form similar to the conventional curves to explore the differences between the two.
First, by substituting NY and Nº into the PUC, we obtain:
Vruc(k, S) (D, (g (D. = (D, (D. + (D. - R (D.
(29)
Based on this equation, the PUC can be interpreted as incorporating NT and No to balance the number of individuals across different strata, with the key distinction being that it assigns weights differently compared to conventional curves.
We can also compare conventional metrics with the PUC metric from the perspective of the ranking rule SMAX-
For conventional curves, including SUC, SQC, JUC, and JQC, SMax is defined as:
Smax (D,) 1(y, (1( 1) I(t, 0)),
(30)
whereas for PUC, the SMax is given by:
SMax (D.) = I(y = 1) (I(t, = 1) - I(tt, = 0)) + 1(y = -)((1tt = - -L(t, = 1)).
(31)
'The joint uplift curve is an exception, where dynamically changing numbers N3 (D,k) and No (D, k)) may lead catastrophic misrankings that intermix P.P and CP subgroups.
15

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
is evident that maximizing the PUC value requires ranking rule that builds upon the conventional curve prioritizing
individuals with 7 0, Y = and 7 1, the top, while placing individuals with T Y 0 and 7 0, Y at the bottom.
G. The Variants of Principled Uplift Curve The value function of the principled uplift eurve Equation (9) formulated as follows:
Vkk, 5) -D, NRC (D.
(32)
R (D, (D.
The principled uplift curve (v1) is defined as follows:
V(k, 5) min kD) (D,
(D, (D)))))
(33)
min( (D. k) RE(D,))
(D) ()(
The principled uplift curve (v2) is defined follows:
R (D,K), k) (D) k)
Vkk, 5) min(
[T|
[C
(34)
(D,R)
RG
min
T
IC|
where [2'l and [C| represent the total number individuals in the treatment group and the control group, respectively.
The principled uplift curve (v3) is defined follows:
Vkk, 5) = max RD, k) (RŞ(D,))
IT
CI
(35)
(R (D) (G(
minn
T
ICI
The principled uplift curve (y4) is defined follows:
RT k) (D,,),))
V(k, S) = max(
T
[C|
(36)
NRE k)
R (D,)))
max
[T]
C
The principled uplift curve (v5) is defined as follows:
(D, (R)(D,)),
Vkk, 8) min(
[
C
(37)
(D, K) (C(D,))
-max
C
The principled uplift curve (v6) is defined as follows:
V(k, 8) - RE(D)k) (D,
(
IC
(38)
(D,((D,)
(C(D)))
[T]
C
16

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Table Kendall between AUUQC and AUTGC sequences S-Leamer on synthetic data. The symbol (1) signifies higher values are desirable. The SUC, SQC, JUC, JQC, and PUC denote their AUUQC values.
Kendall (t)
S-Learner
SUC
0.776
SQC
0.776
JUC
0.812
JQC
0.765
PUC (vl)
0.158
PUC (v2)
0.552
PUC (v3)
0.560
PUC (v4)
0.601
PUC (v5)
0.688
PUC (v6)
0.616
PUC (v7)
0.514
PUC (v8)
0.545
PUC
1.000
This version is similar to the ROCini in Verbeken et al. (2022).
The principled uplift curve (v7) is defined follows:
V(k, () - (D, (D) (D),))) (D,
(39)
(D,) RE(D,K), (T( (g(
The principled uplift curve (v8) is defined as follows:
V(k - (RG N - -(NRG(D,K) + (D)
(40)
Following Figure 5, we apply the Kendall coefficient between AUUQC and AUTGC to quantify the rank correlation between the sequences of these two metrics. As Table 9 shown, according to their performance, we select the value function of the principled uplift as Vk, = RT DD k)) + N (D, RS (D. k) (D,,)))
H. Analysis of PTONet
Next, conduct comprehensive analysis of PTOOee from perspectives including computational complexity, scalability, potential overfitting issues, and detailed function of PTONet.
Computational Complexity. Based on the parameter tuning range we used, the number of parameters for PTONet ranges from 22,323 to 30,556, 160, with floating point operations ranging from 1,578 to 29,090.
Scalability, Our study uses both a synthetic dataset and the real-world Criteo dataset, Lazada dataset to represent small and large-scale scenarios. As shown in Tables 4, 5 and 13. PTONet consistently outperforms other models on the PUC metrie, As detailed in Table 10, the synthetic data, Criteo data, and Lazada data each present unique treatment and control ratios, positive outcome ratios, and varying numbers of covariates, allowing to evaluate scalability.
Potential Overfitting Issues. The propensity estimator, treatment reconstructor, and outcome estimator modules of PTONet each help prevent the loss of CATE information due to overfitting. Additionally, we employ early stopping on the validation set to further prevent overfitting. As shown Table 13, it can be observed that our method still outperforms other methods on the PUC metric even in high-dimensional real-world data.
17

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Detailed Function of PTONet. The role of PTONet is serve feasible model guided by the PUC metric. The novelty of its structure is not the focus of our paper, its backbone is merely variant of DragonNet (Shi et al., 2019). As shown in Table 4 of our paper, evaluates the S-Learner, S-Learner (PU), and PTONet, highlighting the performance impact of PUL and PTONet's backbone. Further detailed results are provided in Table 6. The PUL, treatment loss and balance loss notably boost the model's performance. As targeted regularizer loss is intended to improve model scalability on observational data, its lesser impact on RCT data is expected. Specifically, 'treatment assignment bias' refers to the potential bias that arises when the assignment of treatments is influenced by certain factors or variables, rather than being random. The most common methods mitigate this bias include the use of propensity score matching, weighted regression, or doubly robust learning, The principle of the 'targeted regularizer' lies in its ability to eliminate treatment assignment bias through the introduction of propensity scores, while also ensuring doubly robust learning based on semi semi-parametri estimation theory. This module, not an original contribution of this paper, was first proposed in Shi et al. (2019).
We have not extensively discussed the concepts of treatment assignment bias and the targeted regularizer because, while the former is a common issue in causal inference, it is not the primary focus of this paper. We introduced these concepts mainly to ensure the scalability of our model, which can adapt whether or not treatment assignment bias exists.
I. Experimental Details
Dataset Description. The synthetic dataset contains n = 10, 000 units with = 10 different covariates. For each unit i, the covariates are initially independently and identically sampled from a normal distribution Xij ~ (O 1) for j E (1.2..... .)). We generate the treatment assigmment T; using T; ~ Binomial(1, 0.1), denoting the binomial distribution in a single trial with success probability 0.1. is designed to simulate the real-world scenario in our business data, where the number of treated samples is significantly smaller than the number of control samples. This also ensures differentiation from the Criteo and Lazada datasets.
We simulate potential outcomes based on the covariates as follows:
Y(()) = 0.5 sin
x + +
(41)
((1) 0.1 E cos cos (X²)
2) + +
where the noise terms c ~ NO 0.1). Here, the sine and cosine functions are introduced to incorporate nonlinearity, while the different coefficients are used to adjust the proportion of samples with T(x) > 0. This adjustment helps simulate the roaldworld business scenario where positive outcomes are relatively rare.
The true ITE T. (111) Y,(0) and CATE ((x) = (1) Y-())X = z². The final observed outcome is generated as follows:
Y TT 00)
< <) + ++( = 0);
(42)
where of' ~ Binomial(1, 0.5). The first term represents the observed outcome for the persuadable group, the second term corresponds to the sleeping dogs, and the third term accounts for the observed outcome of sure things and lost causes. For details on the proportion of the treated group and the positive outcome rate, please refer to Table 10.
The real-world Criteo dataset (Diemert Eustache et al., 2018; Diemert et al., 2021), open sourced by Criteo AI Labs, is utilized for uplift modeling in a large-scale advertising scenario. It includes 25,309,483 instances with 11 continuous features, a binary treatment, and candidate outcomes (visits and conversions).
The real-world Lazada data is large-scale production dataset from the real voucher distribution business scenario in Lazada, leading South-Eass Asia (SEA) E-commerce platform of Alibaba Group. Our data processing follows Zhong et al. (2022).
The synthetic data, Criteo data, and Lazada data each present unique treatment and control ratios, positive outcome ratios, and varying numbers of covariates, allowing us to evaluate scalability.
Hy Hyperparameters Tuning, The range of values for hyperparameters shared by all methods is presented as follows: the representation dimension hdim (24, 25, 20,20, the batch size bs E (28, 29, 210 ,21) and the learning rate Ir
18

---

Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Table 10. Statistics of the synthetic dataset, Criteo datasct and Lazada dataset.
Dataset
Covariates
Training Data
Testing Data
Treated (percentage)
Positive Outcome
Treated (percentage)
Positive Outcome (percentage)
Total
Total
(percentage)
Synthetic
10
910 (10.1%)
3,913 (43.5%)
9,000
92 (9.2%)
447 (44.7%)
1,000
Criteo
12
10.7M (85.0%)
5.92K (4.70%)
12.6M
1.19M (85.0%)
65.4K (4.68%)
1,40M
Lazada
83
0.92M (22.1%)
83.0K (2.00%)
4.17M
0.47M (51.6%)
31.9K (3.54%)
0.91M
fle=4, 1e-3, 1,-2,, 21e_2,1e=_1 1e-1). Furthermore, for the hyperparameters a in CFRNet, DragonNet, DESCN, and PTONet, 70. Vi in DESCN, they are all confined to the range of {0.1, 0.5, 1, 5, 10). We utilize an Adam optimizer with maximum of 20 epochs and employ joint Qini the primary evaluation metric. We implement an early stopping mechanism with patience of 5 for all baselines, as suggested by (Liu et al., 2023). Based on the parameter tuning range we used, the number of parameters for PTONet ranges from 22,323 to 30,556,160, with floating point operations ranging from 1,578 to 29,090.
in DragonNet and PTONet, and So. 70.
Implementation Details. All experiments are conducted on an Intel(R) Xeon(R) Gold 6240 CPU 2.60GHz. We used the seikit-uplift package to conduct all AUUQC in this paper. The Kendall coefficient was computed using the 'kendalltau" function from the scipy.stats module.
More Ablation Studies. In addition to applying PUL directly to the S-Leamer in Table 4, we further incorporate PUL into T-Learner TARNet, and EUEN to more comprehensively validate its effectiveness. As Table 11 shown, The performance of these three models after incorporating the Principled Uplift loss function is comparable to that of S-Learner (PU) and PTONet, with significant improvement observed across all metrics. Furthermore, comparing the performance of the five models across various metries reveals that, except for the PUC metric, the results of the other metrics are inconsistent with AUTGC, highlighting the advantage of PUC in aligning with ground truth. Notably, PTONet achieves the best performance on both the PUC and AUTGC metries, further demonstrating its effectiveness.
Table 11 Ablation studies of different backhones on synthetic data. SUC, SQC, JUC, JQC, and PUC denote their AUUQC values. The symbol (L) indicates that lower values are preferable, while (f) signifies that higher values are desirable. "TRUE" represents the results of the ground truth CATE.
Unhalanced Conventional Evaluation Metrie
Balanced Evaluation Metric
Uplift Model
PEHE (L)
SUC (t)
SQC (t)
JUC (1)
JQC (1)
PUC (t)
AUTGC (1)
TRUE
0.000
0.835
0.586
0.779
0.581
1.000
1.000
S- S-Learner (PU)
0.879 + 0.16
0.786 + 0.16
0.552 + 0.11
0.738 = 0,15
0,548 + 0.11
0.943 = 0.19
0.957 = 0.15
T-Learner (PU)
0.867 +: 0.14
0.12 0.763 0.17
0.748 + 0.15
0.537 0.11
0.937 = 0.20
0.952 # 0.15
TARNet (PU)
0.893 + 0.08
0.759+0.11
0.533 0.09
0.754 0.11
0.534 + 0.08
0.944
0.957 0.11
EUEN (PU)
0.781 + 0.15
0.767 + 0.16
0.538 0.11
0.742 = 0.15
0.538 0.11
0.932 = 0.19
0.948 + 0.15
PTONet
0.883 + 0.13
0.780 0.14
0.547 + 0.10
0.746
0.546 0.10
0.948 = 0.15
0.961 + 0.11
More Case Studies. To further explore potential issues in uplift modeling guided by either conventional curves or propensity score, we conduct experiment comparing the performance of various models under same conditions, using fixed random seed of zero. We observe several notable phenomena from the results presented in Table 12.
From an evaluation standpoint, firstly, when comparing the S-Learner (U) and the S-Learner (PS), we find that a low PEHE does not necessarily imply an accurate CATE ranking, highlighting the disparity between estimation and ranking issues in causal inference. Secondly, apart from PUC, the magnitudes of AUUQC for other curves do not align with AUTGC. Notably, in the case of S-Learner (U) and S-Learner (PS), the AUUQC values of SUC, SQC, and JQC are even larger than the the value of TRUE.
From a modeling perspective, S-Learner (PS) achieves higher AUUQC values of conventional curves than S-Learner. These phenomena suggest that models with higher AUUQC values of conventional curves may have learned biased treatment assignments. Conversely, S-Learner (PU) performs better than S-Learner (U) and S-Learner (PS) on the AUUQC of PUC and AUTGC, showing the fcasibility of involving principled uplift to guide uplift modeling. Compared to Table 4, we find that although the best performance of the S-Learner (PU) is not better than the S-Learner (U) in the AUUQC of SUC, SQC,
19

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
Table 12. Performance comparison of different uplift models synthetic data, SUC, SQC, JUC, JQC, and PUC denote their AUUQC values. The symbol (1) indicates that lower values are preferable, while (1) signifies that higher values are desirable. "TRUE" represents the results the ground truth CATE. symbol indicates that uplift model's evaluation result exceeds ground truth result.
Unbalanced Conventional Evaluation Metric
Balanced Evaluation Metric
Uplift Model
PEHE (L)
SUC (1)
SQC (f)
JUC (T)
JQC (t)
PUC (f)
AUTGC (1)
TRUE
0.000
0.835
0.586
0.779
0.581
1.000
1.000
S-Learner
1.392
0.297
0.208
0.332
0.210
0.371
0.519
S-Learner (U)
1.133
0.847*
0.594"
0.677
0.580
0.892
0.918
Learneer (PS)
1.350
0.847*
0.594*
0.755
0.595
0.915
0.935
L-Larnner (PU)
1.158
0.783
0.549
0.752
0.547
0,946
0.959
T-Learner
1.254
0.596
0.418
0.631
0.426
0.680
0.755
TARNet
1.164
0.683
0.479
0.713
0.485
0.864
0.896
CFRNet
1.445
-0.034
-0.024
0.018
0.001
0.060
0.281
DragonNet
1.036
0.732
0.513
0.693
0.510
0.897
0.921
EUEN
1.177
0.602
0.422
0.526
0.410
0.688
0.761
DESCN
1.090
0.780
0.547
0.748
0.547
0.949
0.961
EFIN
1.751
0.619
0.434
0.599
0.431
0.744
0.804
PTONet
0.963
0.770
0.540
0.747
0.536
0.959
0.969
JUC, and JQC, the mean performance of S-Learner (PU) surpasses that of S-Learnr (U). This indicates that models guided by the principled uplift curve yield more robust results.
Results on Lazada Dataset. We include results from the tigh-dimensional Lazada dataset (Zhong et al., 2022) in Table 13. Due to the small curve values in the dataset, we retain an additional decimal place when presenting the results for improved precision. We observe that although DESCN performs best on SUC, SQC, JUC, and JQC, PTONet consistently outperforms other models the PUC metric in the Lazada dataset.
Table 13. Performance comparison (mcan on Lazada dataset. PTONet achieves superior performance on PUC.
Unbalanced Conventional Evaluation Metric
Balanced Metric
Uplift Model
SUC (t)
SQC (t)
JUC (t)
JQC (t)
PUC (t)
S-Learner
0.087540.005
0.0256+0.001
0.003540.001
0.0247+0.001
0.0052+4.000
T-Learnr
0.068440.017
0.0200+0.005
+0.001
0.0219+0.004
0.0066+0.001
TARNet
0.082240.015
0.0241 40.004
0.003310.001
0.023240.005
0.004440.001
CFRNet
0.0628+0.029
0.008144.009
0.0025+0.001
0.0176+0.0m
0.0038+0.002
DragonNet
0.0745+0.003
0.0218
0.0035+0.001
0.0249 +0.202 #0.002
0.0082+0.001
EUEN
0.0893
0.0262 00.003
0.0036 +0.001
0.0253 +0.003
0.0050.1..00
DESCN
0.1160
0.0340+o.002
0.004140.001
0.029510.002
0.0007
EFIN
0.064454.003
0.0189+0.004
0.002340.001
0.016440.002
0.0025+0.003
PTONet
0.042410.015
0.0124
0.002510.001
0.017410.005
0.010840.001
J. Limitations and Future Work
There are several directions for further exploration. Firstly, this paper only addresses the conventional setting of uplift modeling, precisely the scenario binary treatment and binary outcome in RCT data. The performance the principled uplift curve and PTONet in scenarios involving multi-valued or continuous treatment or outcome, and even in observational data settings, remains be explored. Furthermore, the performance of the principled uplift curve relies on the assumption of
20

---

Rethinking Causal Ranking: A Balanced Perspective Uplift Model Evaluation
unconfoundedness. The extent of the presence of unobserved confounders remains crucial question. Finally, the principled uplift curve in this paper only achieves the division of the persuadable and sleeping dog groups without the capability to identify sure things and lost causes. The presence of sure things and lost causes can also affect the performance of the principled uplift curve. Although one way to mitigate this effect is to filter out samples with estimated CATEs of 0 using an uplift model beforehand and then evaluate the remaining samples uniformly, we still anticipate the emergence of new metric the future that can address this issue more comprchensively.
21


Reviews:
[{"review_id": "iDLhOLOhUV", "reviewer": "Reviewer_AASw", "summary": "This paper proposes a new evaluation metric, the Principled Uplift Curve (PUC), which assigns equal importance to individuals with positive and negative outcomes and offers an unbiased evaluation of uplift models. The authors derive a new loss function with a new model architecture to reduce bias during\nuplift model training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We will address your concerns and questions one by one.\n\n**Cons:** The proposed evaluation metric could be discussed in more detail since it is an essential part of the proposed method.\n\n**Response 1:** Thank you for your concern. Based on your suggestion, we provide additional discussion on the evaluation metric as follows:\n\n- **Providing intuition behind** $S_{Max}(D_i)$:  \n  $S_{Max}(D_i)$ represents the maximum PUC value, which is achieved when all samples with $y = 1$ and $t = 1$, as well as those with $y = 0$ and $t = 0$, are ranked ahead of samples with $y = 1$ and $t = 0$, as well as those with $y = 0$ and $t = 1$.  \n\n- **Repositioning the intuition behind the PUC metric (Proposition 4.1):**  \n  We will move the first paragraph of explanation after Proposition 4.1 to the paragraph after formula (9) to improve readability.\n\n- **Clarifying the intuition behind** $g(t_i, y_i)$:  \nWe will clarify that $ g(t_i, y_i) $ assigns a value of 1 to samples with $ y = 1 $ and $ t = 1 $, as well as those with $ y = 0 $ and $t = 0$, while samples with $y = 1$ and $t = 0$, as well as those with $y = 0$ and $t = 1$, are assigned a value of 0.   \nUsing $g(t_i, y_i)$ as the label, we train a binary classifier to constrain $\\hat{\\tau}(x)$, ensuring that samples with $y = 1$ and $t = 1$, as well as those with $y = 0$ and $t = 0$, have a larger $\\hat{\\tau}(x)$, whereas samples with $y = 1$ and $t = 0$, as well as those with $y = 0$ and $t = 1$, have a smaller $\\hat{\\tau}(x)$.\n\n- **Providing intuition behind** $L^{PU}(D)$:  \n  This loss function encourages the CATE of samples with $y = 1$ and $t = 1$, as well as those with $y = 0$ and $t = 0$, to be greater than the CATE of samples with $y = 1$ and $t = 0$, as well as those with $y = 0$ and $t = 1$.  \n\nWe appreciate your feedback and will incorporate these intuitions in the final version of our paper.\n\n**Question 1:** Could you give some intuition about the discussion in section 3? What does the max curve indicate here in Fig 3.\n\n**Response 2:**  Thank you for your question. The **intuition** behind the discussion in Section 3 is that we aimed to verify **whether SUC and other regular metrics reach their maximum values only when the causal effect ranking is completely accurate**. If this were the case, then SUC would be a reliable metric. However, we found that **even when the causal effect ranking is entirely correct, SUC does not always attain its maximum value.** On the contrary, certain **incorrect causal effect rankings can lead to SUC achieving its highest score** (refer to Tables 2 and 3). This observation led us to further investigate SUC and related formulas, ultimately inspiring this paper.\n\nIn Figure 3, the max curve corresponds to $S_{Max}(D_i)$ in Equation (8). We have supplemented its underlying intuition in Response 1.\n\n**Question 2:** Is the experiment for the correlation between AUUQC and AUTGC as described in Appendix I? Do the results still hold for other data distribution?\n\n**Response 3:** Thank you for your question. Yes, the experimental setup in this paper is described in Appendix I. **The results still hold for other data distributions, as long as the data is RCT data.**\n\nThank you once again for your valuable feedback. If you have any further concerns or questions, we are always happy to address them. If you feel that our responses have addressed your concerns, we would appreciate it if you could consider raising your recommendation score.", "claims_and_evidence": "The paper claims that the traditional uplift and Qini curves might lead to biased evaluations and proposes the Principled Uplift Curve (PUC) and compares it with other evaluation metrics by their correlation with AUTGC.\n\nThe proposed method is demonstrated to outperform the existing method with both synthetic and real-world datasets.", "methods_and_evaluation_criteria": "The proposed architecture is evaluated in multiple real-world datasets benchmarks and evaluation metrics.", "theoretical_claims": "I checked the derivation of individual contributions in Appendix D and did not find any issue with it.", "experimental_designs_or_analyses": "I checked the experiment for the proposed model and evaluation metrics.\n\nThe model is shown to outperform the existing models in the proposed evaluation metric.\n\nThe proposed metric is shown to be more reliable with synthetic data. I think there could be more discussion and careful experiments on this.", "relation_to_broader_scientific_literature": "Evaluation is a challenging problem in this area. This paper proposes a new evaluation metric and shows it is more reliable than the existing ones, which could be a nice contribution to uplift modeling.", "essential_references_not_discussed": "I'm not aware of any essential references that are not discussed.", "other_strengths_and_weaknesses": "Pros:\n\n- The paper tackles a challenging problem in uplift modeling and proposes a method to improve the modeling and evaluation.\n- The proposed method is evaluated with many real-world benchmarks\n\nCons:\nThe proposed evaluation metric could be discussed in more detail since it is an essential part of the proposed method.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- Could you give some intuition about the discussion in section 3? What does the max curve indicate here in Fig 3. \n\n- Is the experiment for the correlation between AUUQC and AUTGC as described in Appendix I? Do the results still hold for other data distribution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4uHBHa4gx", "reviewer": "Reviewer_gjPZ", "summary": "This paper reveals the limitations of previous uplift and Qini curves in evaluating uplift models, demonstrating their susceptibility to manipulation by suboptimal ranking strategies that can artificially enhance the performance of biased models. To address this, the authors introduce the Principled Uplift Curve (PUC), a metric that accounts for both positive and negative outcomes, offering a new assessment of uplift models. Additionally, they propose PTONet, a PUC-guided uplift model that optimizes uplift predictions by directly maximizing the PUC value.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We will address each of your concerns one by one.\n\n**Weak 1:** A key concern is that the approach does not improve the worst-case scenario, ... conventional metrics at least ensure that PE^{TP} is ranked no lower than second place. ..., it also introduces a tradeoff in opportunity cost and does not guarantee improved decision-making performance in practical applications.\n\n**Response 1:** Thank you for your concern. As stated in the original manuscript, although PUC is not a perfect metric—it cannot fully identify the four groups—**it is more suitable than conventional metrics for evaluating uplift models.**  When the budget is limited, in the worst-case scenario, the PUC metric **at least ensures less harmful decisions** compared to conventional metrics. In the best-case scenario, PUC guarantees **both less harmful decisions and the highest possible gains**. Therefore, our metric outperforms conventional metrics.\n\nSpecifically, regarding 'conventional metrics ensuring that $PE^{TP}$ is ranked no lower than second place', it seems to overlook **the severe issue where $PE^{CN}$ is ranked behind $SD^{TN}$.** This means that using regular metrics for causal ranking in an uplift model forces decision-makers to target all potential customers ($PE^{TP}$ and $PE^{CN}$) at the expense of customers who would have originally clicked or purchased ($SD^{TN}$). Such a strategy should be avoided in practice as **it harms a large group of customers.**\n\nEven with a budget that covers only $PE^{TP}$ and not $SD^{TN}$, the regular metrics-guided model **loses a large number of potential customers, $PE^{CN}$**. This misstep can mislead decision-makers into believing there are no further growth opportunities when, in fact, potential customers remain untapped. (Please refer to Figure 2 in the original paper; in the worst-case scenario, samples from **3. $SD^{TN}$ to 6. $PE^{CN}$** are considered neither beneficial nor harmful under conventional metrics.)\n\nIn contrast, the PUC metric does not exhibit this issue **in the worst-ranking case**. If decision-makers realize that a small-budget promotion won't yield incremental returns, they can continue to expand the budget until the PUC slows down or the promotion is scaled back, **without harming customer interests or missing potential customers**. (Refer to Figure 3 in the original paper; decision-makers can clearly identify that only the groups from **1. $ST^{TP}$** to **4. $PE^{CN}$** are yielding benefits.)\n\n**Most importantly, in the best ranking case, PUC guided models can achieve the highest-gain decisions with the minimal budget, while conventional metrics cannot.** \n\nTherefore, the **PUC metric should be used** over regular metrics to select an uplift model that accurately targets potential customers without alienating existing customers who are willing to purchase. **Future work can improve upon the limitation of PUC's inability to identify the four groups, but regular curves should no longer be used for uplift model evaluation.**\n\nFinally, we appreciate you for suggesting the application scenarios in advertising and recommendation.\n\n**Weak 2:** It is not surprising that the proposed PTONet achieves the highest PUC and AUTGC, as it is specifically designed based on PUC. However, it does not outperform other methods on alternative evaluation metrics.\n\n**Reponse 2:**  Thank you for your concern. Our simulated data is not specifically designed for PUC; Our data generation process is simple and easy to follow:\n\n$T_i \\sim \\operatorname{Binomial}(1,0.1)$ is designed to simulate the real-world scenario in our business data, where **the number of treated samples is significantly smaller** than the number of control samples. This also ensures differentiation from the Criteo and Lazada datasets.\n\nThe outcome functions are defined as:\n\n$Y_i(0) = 0.5\\sin\\left(\\sum_j^q X_i^j + 1\\right) + \\epsilon^0_i $\n\n$Y_i(1) = 0.1\\left(\\sum_j^5 \\cos(X_i^j) + 2\\right) + \\epsilon^1_i $\n\nHere, the sine and cosine functions are introduced to incorporate nonlinearity, while the different coefficients are used to adjust the proportion of samples with $\\tau(x) > 0$. This adjustment helps simulate our real-world business scenario where **positive outcomes are relatively rare.**  For details on the proportion of the treated group and the positive outcome rate, please refer to Table 10 in the original paper.\n\nPTONet performs suboptimally with regular metrics but achieves the best performance with the PUC metric, which further **validates the issue of bias in regular metrics highlighted in this paper, and confirms that the PU loss function can directly improve the model's performance on the PUC metric.**\n\nThank you once again for your valuable feedback. If you have any further concerns or questions, we are always happy to address them. If you feel that our responses have addressed your concerns, we would appreciate it if you could consider raising your recommendation score.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "I did not thoroughly check each proof. However, they do not contradict my prior understanding.", "experimental_designs_or_analyses": "The paper evaluates the performance of uplift models and their corresponding evaluation metrics using both synthetic and real-world datasets. It conducts experiments on a synthetic dataset and the Criteo dataset (Diemert Eustache et al., 2018; Diemert et al., 2021) to assess model effectiveness in practical scenarios. Additionally, to examine the scalability of the proposed method in high-dimensional settings, the paper presents further experimental results on the Lazada dataset (Zhong et al., 2022).", "relation_to_broader_scientific_literature": "The paper contributes to the broader literature on uplift modeling methods and evaluation metrics for uplift models. It examines limitations in existing evaluation approaches, particularly the susceptibility of uplift and Qini curves to biased rankings. By introducing the Principled Uplift Curve (PUC) and the PTONet model, the paper offers a refined evaluation metric and an optimization-based modeling approach, adding to the ongoing research on uplift modeling and causal inference.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper should provide a stronger justification for the proposed methods and explain why they outperform existing metrics. A key concern is that the approach does not improve the worst-case scenario, where ST^TP and ST^CP are ranked first. In many real-world applications, such as advertising and recommendation systems with budget constraints, this ranking can lead to significant opportunity costs. In contrast, conventional metrics at least ensure that PE^{TP} is ranked no lower than second place. While the proposed method distinguishes persuadable individuals from sleeping dogs, it also introduces a tradeoff in opportunity cost and does not guarantee improved decision-making performance in practical applications.\n\nIt is not surprising that the proposed PTONet achieves the highest PUC and AUTGC, as it is specifically designed based on PUC. However, it does not outperform other methods on alternative evaluation metrics.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AWTX4KSYVV", "reviewer": "Reviewer_e4Zy", "summary": "In an RCT with two groups, treatment and control, an uplift model is supposed to rank four types of units - treatment positive, treatment negative, control positive and control negative in alignment with their CATE (which is unobservable). This paper claims that existing evaluation metrics such as uplift curves and Qini curves are biased towards treating all negatives the same way, i.e., they ignore the potential that a control negative could potentially be a treatment positive and must be ranked at par with treatment positives. It then proposes a simple fix to the evalution metrics that leads to the proposed 'Principled Uplift Curve'. This idea is then also used to add an additional loss function for uplift modeling. Experimental results show that the proposed fix correlates best with true CATE rankings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We will address each of your concerns one by one.\n\n**Claims&Methods:** ... explanation for the claim that equation 12 handles treatment assignment bias. ... clear citations should be added.\n\n**Response 1:** Thank you for your suggestion. We will revise the citation in line 315 of the original text to **\"(please refer to Section 3 in Shi et al. (2019))\"** for better readability.\n\nTreatment Assignment Bias occurs when treatment assignment is influenced by systematic factors rather than being entirely random, potentially leading to biased results. Since this paper focuses on RCT data, where treatment is assumed to be random, Treatment Assignment Bias is not a concern. The **Targeted Regularizer** in PTONet improves scalability to non-RCT data, enhancing its applicability in industry and future research.\n\n**Methods:**  a) Why does the architecture in Figure 4 have the input from h(X,T) being added to g(T,Y).\n\n**Response 2:** Thank you for your question. The function $\\sigma(x)$ can be derived as:\n\n$\\sigma(x) = \\frac{1}{1+\\exp(-\\hat\\tau(x))} = \\frac{1}{1+\\exp(-(h_Y(x,1)-h_Y(x,0)))},$\n\nThe function $h_Y$ corresponds to the arrow in Figure 4. To avoid this ambiguity, we will modify the arrow in Figure 4 from **$h(X,T)\\rightarrow g(T,X)$** to **$h_Y(X,T)\\rightarrow g(T,X)$**.\n\n**Analysis:** a)  why the outcome in the synthetic data is real-valued. b)  the form of the functions don't seem to have a rationale.\n\n**Response 3:** Apologies for any confusion in your reading. We forgot to emphasize in Appendix I that the final observed outcome is generated as follows:\n\n$Y_i = T_i \\mathbb{I}(\\tau_i > 0) + (1 - T_i) \\mathbb{I}(\\tau_i < 0) + \\epsilon_i^y \\mathbb{I}(\\tau_i = 0) $\n\nwhere $\\epsilon_i^y \\sim \\operatorname{Binomial}(1, 0.5)$. The first term represents the observed outcome for the **persuadable** group, the second term corresponds to the **sleeping dogs**, and the third term accounts for the observed outcome of **sure things** and **lost causes**.\n\nThe rationale behind this data generation process is as follows:\n\n$T_i$ is designed to simulate the real-world scenario in our business data, where **the number of treated samples is significantly smaller** than the number of control samples. \n\nIn outcome functions, the sine and cosine functions are introduced to incorporate nonlinearity, while the different coefficients are used to adjust the proportion of samples with $\\tau(x) > 0$. This adjustment helps simulate our real-world business scenario where **positive outcomes are relatively rare.** \n\nWe will include these details in Appendix I of the final version of the paper.\n\n **Comments 1:** what the order is when the index i ranges from 1 to I(k).\n\n**Response 4:** Thank you very much for your comments. **The ordered index used in $I(k)$ is based on the descending order of the score function $S$.** Due to the character limit of the response, please refer to Appendix C of the paper, where we provide a detailed explanation of the calculation process for $I(k)$.\n\n**Comments 1&2:** Is I_diff same as I? SUC in line 124 occurs before it is defined below.\n\n**Response 5:** Thank you for your correction. The term $I_{diff}$ is a typo and will be revised to $I$ in the final version. Similarly, \"SUC\" in line 124 will be corrected to \"uplift and Qini curve.\"\n\n**Questions:**  It would interesting to see if this holds for other uplift models too.\n\n**Response 7:** Thank you for your question. We have additionally included experiments with **T-Learner, TARNet, and EUEN models**. The results are as follows:\n| Synthetic      | PEHE (↓)     | SUC (↑) | SQC (↑) | JUC (↑) | JQC (↑) | PUC (↑) | AUTGC (↑) |\n| -------------- | ------------ | ------------------- | ----------------- | ---------------- | -------------- | --------------------- | ------------ |\n| T-Learner (PU) | 0.867 ± 0.14 | 0.763 ± 0.17        | 0.536 ± 0.12      | 0.748 ± 0.15     | 0.537 ± 0.11   | 0.937 ± 0.20          | 0.952 ± 0.15 |\n| TARNet (PU)    | 0.893 ± 0.08 | 0.759 ± 0.12        | 0.533 ± 0.09      | 0.754 ± 0.11     | 0.534 ± 0.08   | 0.944 ± 0.14          | 0.957 ± 0.11 |\n| EUEN (PU)      | 0.781 ± 0.15 | 0.767 ± 0.16        | 0.538 ± 0.11      | 0.742 ± 0.15     | 0.538 ± 0.11   | 0.932 ± 0.19          | 0.948 ± 0.15 |\n| PTONet         | 0.883 ± 0.13 | 0.780 ± 0.14        | 0.547 ± 0.10      | 0.746 ± 0.13     | 0.546 ± 0.10   | 0.948 ± 0.15          | 0.961 ± 0.11 |\n\nThe performance of these three **models after incorporating the Principled Uplift loss function is comparable to that of PTONet**, with significant improvement observed across all metrics. We will add these experiments in the final version of this paper.\n\nThank you once again for your valuable feedback. If you have any further concerns or questions, we are always happy to address them. If you feel that our responses have addressed your concerns, we would appreciate it if you could consider raising your recommendation score.", "claims_and_evidence": "I don't see enough clear explanation for the claim that equation 12 handles treatment assignment bias. While this is not the main point of the paper, given that it is included and important for the PTONet (the proposed uplift model), either it should be explained more or if it is from previous work, clear citations should be added.", "methods_and_evaluation_criteria": "The paper compares its fix against multiple uplift models and evaluation criteria present in literature. A few specific issues a) Why does the architecture in Figure 4 have the input from h(X,T) being added to g(T,Y). It seems like in equation 11, the BCE only takes in $\\sigma(x)$. b) the paper needs to address the text on treatment bias with some more explanation.", "theoretical_claims": "The main theoretical claim that the paper makes is that the proposed uplift curve is sound in its ranking. This follows immediately from the proposed fix.", "experimental_designs_or_analyses": "The experimental design is sound as per my understanding. A few issues a) I couldn't understand why the outcome in the synthetic data, in Appendix I is real-valued when the rest of the analysis is for a binary outcome. b) Also, the form of the functions don't seem to have a rationale mentioned in the text.", "relation_to_broader_scientific_literature": "This work challenges existing evaluation metrics in the uplift modeling literature. It draws specifically from Devriendt et. al. 2020 that uses helper functions to come up with loss functions for training uplift models.", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "Overall, the main contribution is to propose a fix to existing evaluation metrics in the uplift modeling literature. This is certainly important and impactful. The paper is written clearly overall. There are some parts which need more work that I specify later. My overall impression was that the proposed fix was 'obvious' and what anyone should do in the first place. On one hand, the presentation of the problem such that the solution is obvious, is a strength of the paper. However, in this case, I find that the paper lacks any further insight apart from this fix.", "other_comments_or_suggestions": "1) Section 2.2 has some notation that is not right. I(k) is assumed to be an ordered index but it is not made clear what the order is when the index i ranges from 1 to I(k). Is I_diff same as I?\n2) SUC in line 124 occurs before it is defined below. \n3) The plot in Figure 5 needs some more explanation about what the shaded region is and what the lines are.", "questions_for_authors": "I have mentioned most questions in the previous sections. Regarding the results, even the S-Learner with the loss function fix seems to be competitive with PTONet for the fixed evaluation metrics. It would interesting to see if this holds for other uplift models too.\n\n###update after rebuttal ###\n\nI am satisfied with the responses that the authors provided. On the one hand, the experiments that they performed underscore an important point that existing learners with the loss function fix seem to be competitive with their proposed learner. However, on the other hand I feel it takes away from the utility of PTONet. So, I'll keep my score unchanged.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTRjkk6k7G", "reviewer": "Reviewer_JDoP", "summary": "This paper proposes PTONet, a new uplift model that integrates the Principled Uplift Loss (PUL) to improve CATE ranking accuracy, outperforming existing models in experiments on simulated and real-world datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. If you have any additional concerns or questions, we would be happy to answer them. If you have no additional concerns, we would appreciate you considering increasing your recommendation score.", "claims_and_evidence": "The paper effectively presents its claims and supports them with clear evidence.", "methods_and_evaluation_criteria": "Yes, the paper proposes a method to improve CATE ranking accuracy.", "theoretical_claims": "The theoretical claims and their proof are correct.", "experimental_designs_or_analyses": "The paper's experimental design appears methodologically sound.", "relation_to_broader_scientific_literature": "The key contributions of this paper are positioned within the broader context of uplift modeling, which has been a subject of significant interest in domains like marketing, customer retention, and personalized treatment recommendations.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "KNV2AL7OUK", "reviewer": "Reviewer_MgqQ", "summary": "This work focuses on building and evaluating models for uplift modeling. The work finds a critical limitation in existing evaluation metrics, as many of these models do not weigh negative outcomes enough. The work finds that this lead to biased evaluations due to incorrect orderings between persuadable and sleeping dogs with negative outcomes, potentially resulting in biased models receiving higher curve values. The authors show this through both empirical results and theoretical results. Given the limitation of existing evaluation metrics, the work proposes the principled uplift curve (PUC), and show that it properly weighs different individuals in both the positive and negative outcome groups. The authors propose PTONet by integrating the PUC into the objective function to reduce bias during uplift model training. Through experimental results, the efficacy of PUC is shown by its alignment with ground-truth evaluation in synthetic settings, and the efficacy of PTONet is established in both synthetic and real data.\n\n## Update after Rebuttal \n\nI appreciate the authors work in answering the questions. Overall, I am satisfied with the response and have updated my score. I do hope the authors consider the limitations brought up by other reviewers and discuss them to ensure the paper does not over claim the contributions of the proposed metric (which I do not believe they currently do).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**References &Weaknesses 1:** I would appreciate more discussion on [1].\n\n**Response 1:** Thank you for your concern. As far as we know, TOC/AUTOC can be understood as introducing a threshold $u$ and a logarithmic function to the conventional uplift and Qini curves, as shown in the following formula (from Equation (2.5) in [1]):\n\n$\\begin{aligned} & \\operatorname{AUTOC}(S)=\\mathbb{E}\\left[\\left(-\\log \\left(1-F_S\\left(S\\left(X_i\\right)\\right)\\right)-1\\right)\\left(Y_i(1)-Y_i(0)\\right)\\right] \\\\ & \\operatorname{QINI}(S)=\\mathbb{E}\\left[\\left(F_S\\left(S\\left(X_i\\right)\\right)-\\frac{1}{2}\\right)\\left(Y_i(1)-Y_i(0)\\right)\\right]\\end{aligned}$\n\nThis indicates that the metric **places particular emphasis on the contribution of the top few individuals** (as shown in Figure 2 of [1], where, if only the top 10% of the population is considered, the overall gain from AUTOC is higher than that from QINI). In other words, this metric **amplifies the imbalance issue inherent in the uplift and Qini curves.** In contrast, the goal of our metric is the opposite—**we aim to address and mitigate this imbalance problem.**\n\n**Weaknesses 2&Questions 2:** What is the exact intuition for why the loss (11) is useful? What does identifying g(t_i, y_i) from the estimated treatment effect help guide models to assign higher CATEs to persuadable individuals?\n\n**Response 2:** Thank you for your question. The intuition behind the loss function in equation (11) is derived from Table 1. Specifically, **the TP ($T=1, Y=1$) and CN ($T=0, Y=0$) samples should be ranked ahead of TN ($T=1, Y=0$) and CP ($T=0, Y=1$).** \n\nTo incorporate this constraint during training, we introduce $g(t_i, y_i)$ as a binary classification task label, where the labels for TP and CN samples are 1, and the labels for TN and CP samples are 0. Then, we use the estimated causal effect $\\hat{\\tau}(x)$ as the input to train this binary classification task. This approach effectively constrains the model such that **the estimated causal effects $\\hat{\\tau}(x)$ for TP and CN are as large as possible, while for TN and CP, $\\hat{\\tau}(x)$ should be as small as possible.** In this way, when the trained model is tested or during model selection, the model will rank TP and CN ahead of TN and CP based on $\\hat{\\tau}(x)$ in descending order. The persuadable group corresponds to the TP and CN samples, while the sleeping dog group corresponds to the TN and CP samples.\n\n**Claims&Weaknesses 3&Questions 1:** The proof of Proposition 4.1 is not well formulated and very difficult to follow. When proving Proposition 4.1, how does the difference between the two bounds prove the first half of Proposition 4.1?\n\n**Response 3:** Thank you for your concern. Below, we will focus on explaining the meaning of these two bounds and why the difference between them is able to distinguish between the persuadable group and the sleeping dog group.\n\nIn appendix E, we define the number of persuadable individuals in total $k$ samples as $N^P(k)$ and the number of sleeping dog individuals in total $k$ samples as $N^S(k)$, then we have the two bounds that $N^P(k) \\le R^T(D,k) + NR^C(D,k)$ and $N^S(k) \\le R^C(D,k) + NR^T(D,k)$. \n\nThe first bound holds because the **persuadable group $(\\tau(x) > 0)$ only includes the TP ($T=1, Y=1$) and CN ($T=0, Y=0$) groups.** Therefore, the number of individuals in the persuadable group, $N^P(k)$, will always be less than or equal to the sum of the number of TP individuals, $R^T(D,k)$, and CN individuals, $NR^C(D,k)$. **Similarly, the sleeping dog group only includes the TN ($T=1, Y=0$) and CP ($T=0, Y=1$) groups.** Thus, $N^S(k) \\le R^C(D,k) + NR^T(D,k)$.\n\nWe aim for the evaluation metric PUC to correctly distinguish between the persuadable group and the sleeping dog group. This means we want **PUC to increase as the persuadable group grows, and decrease as the sleeping dog group increases.** Therefore, we define:\n\n$V_{\\operatorname{PUC}}(k,S) = R^T_{S}(D,k) + NR^C_{S}(D,k) - R^C_{S}(D,k) - NR^T_{S}(D,k).$\n\nAs we can see, the first two terms include all the individuals in the persuadable group, while the last two terms include all individuals in the sleeping dog group. **When the persuadable group is included in PUC, the value of PUC increases**; conversely, **when the sleeping dog group is included in PUC, the value of PUC decreases**. Thus, our metric can effectively distinguish between the persuadable group and the sleeping dog group.\n\nWe will include this explanation in Appendix E of the final version of the paper.\n\nThank you once again for your valuable feedback. If you have any further concerns or questions, we are always happy to address them. If you feel that our responses have addressed your concerns, we would appreciate it if you could consider raising your recommendation score.", "claims_and_evidence": "The primary claims of this work are: \n\n1) Existing uplift model evaluation curves can result in suboptimal ordering \n2) The proposed PUC provides a more balanced and unbiased evaluation compared to regular uplift and Qini curves \n3) The proposed PTONet enhances a model's ability to rank CATEs effectively. \n\nThe authors provide sufficient evidence for all of these claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand, and the inclusion of both synthetic data and real data is appreciated.", "theoretical_claims": "I examined the correctness of proposition 4.1. Truthfully, it is a bit difficult to follow. The rough steps make sense, though it is not clear why we define the value function has the difference between the two bounds -- this is stated without justification in the proof. The proof for the second half of 4.1 is more clear.", "experimental_designs_or_analyses": "The experimental design and analyses seem sound overall.", "relation_to_broader_scientific_literature": "The ability to accurately evaluate uplift models is critical across many fields, including marketing and advertising where personalized decision-making is key. The contributions in finding the weaknesses of existing evaluation metrics as well as the new proposed evaluation metric are hence quite important. The proposed PTONet also shows a new way to improve uplift modeling, which may be used in these downstream applications as well.", "essential_references_not_discussed": "I would appreciate more discussion on [1]. In the evaluation metrics considered in [1], every example will contribute to the TOC/AUTOC curve, which may mitigate some of found issues with most uplift modeling metrics. \n\n[1] Yadlowsky, Steve, et al. \"Evaluating treatment prioritization rules via rank-weighted average treatment effects.\" Journal of the American Statistical Association (2024): 1-14.", "other_strengths_and_weaknesses": "Strengths: \n\n1. The authors find a really subtle yet interesting flaw in how most uplift evaluation metrics are evaluated \n2. The clear examples of when an unbiased model is rated worse than a biased model in Table 2/3 is very compelling \n3. The correlation between the ground-truth AUTGC and PUC in the experimental results is very convincing, and seems to support the theoretical findings for PUC\n4. PTONet has strong performance in terms of PUC. The fact that the next performing model is PU S-Learner shows more credence to the authors findings regarding how to optimize the PUC. \n5. The ablations of the proposed method are convincing in terms of showing the importance of every part of the objective\n\nWeaknesses and Concerns:  \n\n1. The work does not mention the relationship to [1] (i.e., does TOC overcome many of the issues of past work?) \n2. The organization of the proposed method is quite difficult to follow. Specifically, it is not clear how the loss in (11) is formulated. A more clear description of this would be useful as this is the crux of the proposed method. \n3. The proof of Proposition 4.1 is not well formulated and very difficult to follow. Once again, as this is a major part of the proposed contribution, a more clear proof for this proposition would be appreciated.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. When proving Proposition 4.1, how does the difference between the two bounds prove the first half of Proposition 4.1? I would appreciate seeing the steps more clearly. \n2. What is the exact intuition for why the loss (11) is useful? What does identifying g(t_i, y_i) from the estimated treatment effect help guide models to assign higher CATEs to persuadable individuals?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper discusses the limitations of Qini curves/uplift models, focusing on bias in evaluation and offering a solution through PUC. Reviewers agreed that evaluation is challenging in this area. Authors adequately addressed concerns that were raised during the discussion including that the approach does not improve the worst-case scenario, and relation to Yadlowsky et al AUTOC evaluation measure.
Decision: Accept (poster)