# Multi-Treatment Uplift Evaluation on Non-Random Assignment Biased Data

Nathan Le Boudec, Nicolas Voisine, Bruno Crémilleux

► To cite this version:

Nathan Le Boudec, Nicolas Voisine, Bruno Crémilleux. Multi-Treatment Uplift Evaluation on Non-Random Assignment Biased Data. Data and Knowledge Engineering, 2026, 163, pp.102565. 10.1016/j.datak.2026.102565. hal-05482449

HAL Id: hal-05482449

https://hal.science/hal-05482449v1

Submitted on 28 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Multi-Treatment Uplift Evaluation on Non-Random Assignment Biased Data

Nathan Le Boudec<sup>a,b</sup>, Nicolas Voisine<sup>a</sup>, Bruno Crémilleux<sup>b</sup>

<sup>a</sup>Orange Research, Lannion, F-22300, France

<sup>b</sup>Université Caen Normandie, ENSICAEN, CNRS, Normandie Univ, GREYC UMR 6072, Caen, F-14000, France

## Abstract

Uplift quantifies the impact of an action (marketing, medical treatment) on an individual's behavior. Uplift prediction is based on the assumption that the target and control groups are equivalent. However, in real-world scenarios, customers are often selected for actions based on their prior behavior, introducing non-random assignment bias that distorts uplift estimation. This issue is even more present in the case of multi-treatment, as in the context of offer recommendation system, where multiple actions are possible for an individual. To the best of our knowledge, the effect of bias in multi-treatment uplift has not yet been studied. In this paper, we propose a novel protocol for evaluating multi-treatment uplift under non-random assignment bias. Using this protocol, we assess the performance of the main multi-treatment uplift methods from the literature. Our results show significant differences in their robustness to bias, providing valuable insights and guidelines for practical applications in biased settings.

## Keywords:

Uplift modeling, Bias, Benchmark, Treatment Effect Estimation, Machine Learning

## 1. Introduction

Uplift modeling is a predictive approach that aims to estimate the impact of several treatments on an individual's behavior [1, 2]. The term "treatment", as used in the literature on causal inference and uplift modeling, simply refers to any action whose impact is being measured, no matter the domain [2]. It can be defined as a machine learning task or a causal inference
problem [4]. In this paper, we adopt a machine learning approach. This type
of modeling is particularly useful in fields such as personalized medicine and
advertising. In the context of marketing recommendation systems, uplift
estimation is crucial to determine which treatment will generate the most
favorable impact in a customer. Beyond predicting whether a person will
respond to a treatment, uplift also estimates the effect of one treatment rela-
tive to others. In this way, uplift models help to optimally allocate marketing
resources by targeting those individuals for whom the effect of a treatment
will be most significant.

The literature initially focused on the case of single-treatment: only the impact of a treatment is measured, compared with the control group (i.e. non-treatment). Examples of uplift methods include meta-learners [5] or random forests [6]. This article deals with a more difficult situation: the multi-treatment case. The goal is to order and quantify the impact of several treatments relative to a control treatment.

Uplift methods are based on the assumption that the different treatment and control groups are homogeneous, i.e. that there is no bias between individuals in the different groups. In practice, the data are observational. In a recommendation system, the allocation of a treatment (e.g. a marketing offer) may depend on the characteristics of the individual. In other words, the probability of a customer being targeted by a treatment may depend on their past behavior or other characteristics. This can distort the uplift estimate and limit the effectiveness of the recommendation engine.

There are multiple biases. For instance, not answering a sales call intro-
duces a bias between individuals who answer a sales call and those who do
not. The aim of this article is to study the behavior of the main uplift meth-
ods in the case of “Non-Random Assignment” [7]. This bias, noted *NRA*, is
very common in recommendation systems. It occurs when the probability
of receiving a treatment depends on the individual’s characteristics. Little
work has been done on the study of biases in the context of uplift [10] and, to
the best of our knowledge, our work is the first to focus on multi-treatment
methods.

Our aim is to study how uplift methods behave when faced with bias
and the reality of the problems that are encountered by recommender sys-
tems. This context is characterized by unbalanced data, a large number of
treatments, and situations where uplift values are small. For that, we ad-
dress three research questions: how do the performances of the uplift meth-
ods change according to the rate of the bias, the number of treatments and
whether the uplift values are small or not (in the following, we refer to this characteristic as *contrast*). To answer these questions, we design an experimental protocol capable of quantifying the impact of bias in the case of multi-treatment. We explain the necessary properties of the protocol. For an accurate estimation of uplift, we show that it is necessary to use synthetic data, and we describe how to generate such data. Experimental results show that uplift methods behave differently in the presence of bias. We draw a number of lessons from these results that will be useful in designing future techniques adapted to the case of bias in multi-treatment. This will also help us design techniques to resist bias.

This article is organized as follows. Section 2 provides basic definitions and discusses how to evaluate uplift. We give an overview of the state-of-the-art of uplift methods in Section 3, as well as associated work, including multi-treatment uplift methods, benchmarks, and the impact of the number of treatments. We describe our protocol and the corresponding data generation in Section 4. Section 5 presents experimental results to better understand the impact of bias.

# 2. Background and Preliminaries

## 2.1. *Definition and Notation*

*Uplift*. Uplift [1, 2] is an estimate of the impact of a treatment on a given individual with respect to no treatment. Let $Y$ be a variable expressing the impact of a treatment (for example, in the case of marketing, “clicks” and “doesn’t click”) and $T$ a treatment variable. We denote $Y_i(T = 1)$ the result of an individual $\mathbf{x}_i$ when he receives treatment $T = 1$ and $Y_i(T = 0)$ for treatment $T = 0$. The CATE (Conditional Average Treatment Effect) is the causal effect of a treatment on an individual. It is defined as the difference in expected outcomes between the treated and control groups for a specific individual $\mathbf{x}_i : \mathbb{E}[Y_i(T = 1) - Y_i(T = 0)|X = \mathbf{x}_i]$. If this result is positive (resp. negative), then treatment $T = 1$ has a positive (resp. negative) impact for this individual. Under certain assumptions, uplift is equivalent to the CATE [3]. For a given individual $\mathbf{x}_i$, uplift is formally defined by :
$$ \tau(\mathbf{x}_i) = \mathbb{E}[Y|T = 1, X = \mathbf{x}_i] - \mathbb{E}[Y|T = 0, X = \mathbf{x}_i]. $$

Most of the literature focuses on the single-treatment uplift, where the treatment variable $T$ is binary. This situation corresponds to one group of *control* and a single group of *treatment*. In this case, the uplift is unique:
$$ \tau(\mathbf{x}_i) = \mathbb{E}[Y|T = 1, X = \mathbf{x}_i] - \mathbb{E}[Y|T = 0, X = \mathbf{x}_i]. $$
is no longer valid if there are several treatments ($T \in \{0, 1, ..., K\}$). This is because there is no longer a single uplift (between treatment and control), but K uplifts, i.e. an uplift between each of the treatments and the control. The control group is always denoted as $T = 0$, while the treatments can take values from 1 to K. The uplift of treatment $k$ is then $\tau_k(\mathbf{x}_i) = E[Y|T=k, X=\mathbf{x}_i] - E[Y|T=0, X=\mathbf{x}_i]$.

*Treatment policy.* The aim of multi-treatment uplift estimation is to determine the treatment with the best impact, i.e. the treatment with the highest uplift. A plan for assigning a specific treatment to each individual is a treatment policy. The optimal treatment, i.e. the optimal treatment policy, for the individual $\mathbf{x}_i$ is therefore: $\pi(\mathbf{x}_i) = \arg\max_{k=0,1,...,K} \tau_k(\mathbf{x}_i)$. For a given uplift model, the treatment policy will be the set of treatments that maximize the uplift estimation, denoted as $\hat{\tau}$, for each individual $\mathbf{x}_i$: $\hat{\pi}(\mathbf{x}_i) = \arg\max_{k=0,1,...,K} \tau_k(\mathbf{x}_i)$.

## 2.2. Uplift Evaluation

An intrinsic difficulty with uplift modeling is that it is impossible for an individual to know whether the chosen treatment is optimal, as his responses to alternative treatments cannot be observed. Also, classical machine learning evaluation methods cannot be used, as this would involve comparing counterfactual results, i.e. understanding what would have happened if a treated individual had not received a treatment, or *vice versa*. This is known as the fundamental problem of causal inference [2]. However, an individual's uplift can be estimated empirically by considering and comparing two groups of individuals: a set of individuals who have received treatment and a set of individuals who have not.

**Expected Outcome.** Although it is not possible with observational data to know the exact uplift, it is possible to compare policies with each other thanks to the *expected outcome* [15]. The *expected outcome* is a measure used to assess the average effect of a given $\pi$ treatment policy on a population. The principle is to estimate what the gain would have been (that is, the value of $Y$) if such a policy had been followed. Using a new random variable $Z$, respecting $E[Z] = E[Y|T=\pi(X)]$, the *expected outcome* estimates $E[Z]$ with test data, and thus $E[Y|T=\pi(X)]$ [15].

**Root Mean Square Error (RMSE).** In Section 4, we will use synthetic data to obtain exact uplift values for each individual. The RMSE can then be used. The RMSE is a classic measure quantifying the difference between
predicted and observed values. For a multi-treatment uplift prediction model, the RMSE for an individual $\mathbf{x}_i$ is defined by $\sqrt{\frac{1}{K} \sum_{k=1}^{K} (\hat{\tau}_k(\mathbf{x}_i) - \tau_k(\mathbf{x}_i))^2}$ where K is the total number of treatments, $\hat{\tau}_k(\mathbf{x}_i)$ is the estimate of uplift k for individual $\mathbf{x}_i$ and $\tau_k(\mathbf{x}_i)$ the true value of uplift k for this individual.

# 3. State-of-the-art

## 3.1. Overview of Multi-Treatment Uplift Methods

We present the main uplift methods in two broad categories: *meta-learners* and direct approach methods.

### 3.1.1. Meta-learners.

The *meta-learners* are very classical [5]. These approaches use standard machine learning models to estimate uplift, essentially by transforming the treatment effect estimation into a standard prediction problem.

**S-Learners.** *S-Learners* [18, 19] learn a unique model by including the treatment as an additional variable in the predictions. In other words, S-Learner uses the concatenation of treatment and covariates $(T, X)$ as the feature set. A response function $\hat{\mu}(x, t)$ is then defined as:

$$ \hat{\mu}(x, k) := \mathbb{E}[Y | (X = x, T = k)]. $$

This function can be learned using any machine learning algorithm. The uplift estimation $\hat{\tau}_k(x)$ is then computed as:

$$ \hat{\tau}_k(x) = \hat{\mu}(x, k) - \hat{\mu}(x, 0). $$

**T-Learners.** *T-Learners* [20] build a separate model for each treatment group $\hat{\mu}_k(x) = \mathbb{E}[Y | X, T = k]$. Uplift $\hat{\tau}_k(x)$ is estimated by comparing the results between different groups:

$$ \hat{\tau}_k(x) = \hat{\mu}_k(x) - \hat{\mu}_0(x). $$

**X-Learners.** *X-Learner* [19] is an extension of *T-Learner*. The difference is that *X-Learner* applies a change of variable to estimators and weights the variable using the propensity score [11].

**R-Learners.** *R-Learner* [22] is a two-step algorithm that estimates the outcomes $\hat{\mu}_k(x)$, as well as the propensity score, $\hat{e}(x)$. Uplift is then estimated by minimizing the following loss function:
$$L[\hat{\tau}(x)] = \frac{1}{n} \sum_{i=1}^{N} \left( (Y_i - \hat{\mu}^{(-i)}(X_i))^2 + (T_i - \hat{e}^{(-i)}(X_i)\hat{\tau}(X_i))^2 \right)$$

where $\hat{\mu}^{(-i)}$ and $\hat{e}^{(-i)}$ denote the out-of-fold held-out predictions made without using the i-th training sample.

**DR-Learners.** Finally, DR-Learner [21] is an approach combining T-Learner and the propensity score. The Doubly-Robust-Learner (DR-learner) combines the two-model approach with inverse propensity weighting. The outcome is transformed as follows:

$$Y_{i}^{DR-L} = \frac{T_{i}}{e(X_{i})} \left( \frac{1 - e(X_{i})}{e(X_{i})} \right) (Y_{i} - \hat{\mu}_{T_{i}}(X_{i})) + \hat{\mu}_{1}(X_{i}) - \hat{\mu}_{0}(X_{i})$$

### 3.1.2. Tree-Based Uplift Estimation Methods

In this article, we use methods based on decision trees to estimate uplift [12]. These methods include the ED, Chi, and CTS approaches. Each method leverages of the strengths of decision trees for modeling heterogeneous treatment effects.

**ED.** ED[6] is a decision tree-based approach that models the treatment effect by partitioning the feature space into regions where the treatment effect is homogeneous. The decision tree algorithm is used to learn the optimal splits based on the treatment effect in each region.

**CTS.** CTS [15] is a decision tree-based method for uplift modeling that focuses on maximizing the expected response at each split.

**Chi.** Chi [6] is another decision tree-based method for uplift modeling. It is based on the Chi-Square statistic, which is used to select the most significant splits for uplift estimation. The Chi method focuses on maximizing the statistical difference between treated and control groups at each decision node.

### 3.1.3. Uplift Estimation Methods Based on Deep Learning

TARNet [13] (Treatment-Agnostic Representation Network) is a state-of-the-art neural network-based approach for treatment effect estimation. It utilizes a shared feature representation with separate heads for the treatment and control groups, allowing a more flexible modeling of treatment effects.
This architecture helps to reduce estimation bias and has shown strong performance compared to traditional uplift estimation methods [14]. TARNet is the only state-of-the-art deep learning-based uplift method for which the code is available and usable for multi-treatment uplift modeling.

## 3.2. Benchmarks on Multi-Treatment Uplift Methods

Few studies directly compare uplift methods in a multi-treatment setting. One relevant study [8] benchmarks various uplift methods in multi-treatment scenarios and emphasizes the importance of using propensity scores to account for treatment assignment bias. This study demonstrates the utility of propensity score matching and inverse probability weighting, which are traditional methods for correcting assignment bias in causal inference. Additionally, [9] explores uplift methods in contexts where treatment impact is continuous, such as income-related interventions. The authors propose adaptations of traditional uplift modeling techniques to better capture variations in treatment effects. Their work emphasizes the importance of handling continuous outcomes with appropriate loss functions and model architectures. They also provide empirical evaluations demonstrating the advantages of these approaches in real-world economic and healthcare scenarios.

While these studies make significant strides in addressing multi-treatment scenarios, none of them specifically deal with biased data, an important consideration for real-world applications where data is often inherently biased. Our work seeks to fill this gap by focusing on how biased data impacts uplift estimation in multi-treatment settings.

## 3.3. The Impact of the Number of Treatments

Another crucial aspect that has not been sufficiently studied in the literature is the effect of the number of treatments in multi-treatment uplift modeling. The number of treatments is a key factor in real-world applications, especially in fields such as marketing, healthcare, and public policy, where multiple strategies can be tested simultaneously. To the best of our knowledge, no study has explored the impact of increasing the number of treatments on the quality of uplift estimation.

As the number of treatments grows, the complexity of modeling different treatment effects increases, which can significantly impact model performances. Indeed most uplift models use the treatment as a feature, so an increase in the number of treatment values increases the search space and therefore makes uplift estimation more difficult. Thus, our paper aims to
investigate how the number of treatments influences uplift estimation in the
presence of non-random assignment bias.

3.4. Contrast in Uplift Estimation

Contrast, referring to the difference in treatment responses between in-
dividuals i.e. the amplitude of uplift values, has not been studied in the
uplift estimation literature. This aspect is particularly important in multi-
treatment settings, where the impact of different treatments on an individ-
ual's outcome may vary significantly. Contrast can impact model perfor-
mance, especially when the model needs to differentiate between individuals
who respond differently to treatments. To our knowledge, no study has in-
vestigated the role of contrast in a multi-treatment scenario. Exploring this
aspect could provide crucial insight into the limitations of current methods
and inform the development of more robust uplift estimation techniques.

4. Protocol to Evaluate the Impact of the NRA Bias for the Multi-Treatment Uplift.

4.1. *Problematic*

NRA bias is a major issue in uplift estimation [7], particularly when it
comes to personalizing treatments in complex contexts such as recommender
systems. This bias arises when the probability of receiving a treatment de-
pends on the individual’s characteristics. Formally, this bias is defined when
$P(X|T = i) \neq P(X|T = j)$ with $i \neq j$. This problem has received limited
attention in the literature [10], and not yet in the case of multi-treatment.

There are few works on experimental comparisons of uplift methods. [8]
carried out a benchmark of uplift methods in the context of multi-treatment
and showed the interest of the propensity score [11]. Still in the case of multi-
treatment, [9] compares uplift methods whose treatment impact is continuous
(for example, an income). However, none of these studies uses biased data.

Moreover, contrast has not been studied either. However, this aspect is
crucial as it affects the ability of models to differentiate between individuals
with varying treatment responses. To the best of our knowledge, no study has
examined the impact of contrast in a multi-treatment setting, especially in
the presence of biased data. Investigating this aspect could provide valuable
insight into the limitations of existing methods and guide the development
of more robust approaches.
The impact of the number of treatments has also not been studied. This is a key factor because the number of treatments can be significant in real-world scenarios.

## 4.2. *Research Questions*

Our experimental study aims to answer the following research questions:

**RQ 1: how does the performance of uplift methods change with the bias?** For example, does the performance of uplift methods deteriorate as the bias rate increases? If so, does this degradation behave linearly, or do we see a collapse in performance above a certain bias rate?

**RQ 2: what is the impact of the number of treatments?** For example, do method performances deteriorate as the number of treatments increases?

**RQ 3: what is the impact of contrast?** The difference in the expectation of the variable Y between groups can be greater or lesser (this difference is the uplift between groups). The greater the difference, the easier the problem is and the smaller the difference, the harder the problem is. This parameter is called *contrast* (see Section 4.4). The aim is to study the performance of the methods as a function of uplift contrast.

The source code to execute the benchmark and generate datasets are available at: https://github.com/Nathpreums/Uplift_Multitreatment_Benchmark_NRA.

## 4.3. *Protocol*

An experimental evaluation requires a careful protocol design to ensure an objective comparison of the different methods. In our case, the protocol needs to assess the impact of bias. We will now explain how the following properties make it possible to achieve this task.

**Properties to evaluate the impact of bias.** To assess the impact of bias, the protocol must enable the addition of bias to the data while quantifying the rate of bias in order to control its introduction and study how the methods behave when the rate of bias increases. Data complexity (e.g. number of treatments, uplift values) is a parameter to evaluate the bias effect. We study data complexity by varying the number of treatment groups and the uplift values.

**Properties for objective comparison of methods.** Methods must be compared on an equitable basis. To achieve this, training and test data must be kept identical across methods. The same number of individuals must also
be maintained for each rate of bias introduced, so that differences in model performance can be attributed solely to the presence of bias, and not to a lack of data.

**Importance of synthetic data.** To evaluate uplift methods, we need precise control over population characteristics, treatments applied, rates of bias, modulation of data complexity and the exact uplift value for individuals. Generating synthetic data enables these expectations to be met. A synthetic dataset models a certain type of bias and lacks the complexity and the variance of real-world data, as data structure of each problem is unique. Due to selection bias, real-world data does not allow for an accurate estimation of either the outcome or the uplift. The true uplift value is crucial for evaluating our models and for the relevance of the benchmark, and is guaranteed with synthetic data. Section 4.4 details how we generate the data.

### 4.4. Data Generation

In order to create a structure adapted to our study, we generate data in both two-dimensional and three-dimensional data, according to the variables $X_1, X_2$ and $X_3$:

* each individual $\mathbf{x}_i$ is described according to these variables, $X_1, X_2$ and $X_3$. They are generated independently according to a uniform distribution in the interval $[0, 1]$.

* each variable $X_1, X_2$ and $X_3$ are then divided into equal intervals. In the case of two dimensional data, $X_1$ and $X_2$ are divided into 8 resulting $8 \times 8 = 64$ cells. To maintain the same number of cells in the three-dimensional case, we divide $X_1, X_2$, and $X_3$ into 4, which also results in $4 \times 4 \times 4 = 64$ cells. Each cell represents a subgroup of individuals sharing similar values for $X_1, X_2$ and $X_3$.

* for each cell, a uniform value of $\mathbb{E}[Y|X = \mathbf{x}_i, T = t_k]$ is assigned to each of the $K$ treatments. These values reflect the expected outcome for individuals in this cell when they receive the $t_k$ treatment.

* for each individual, a treatment $T$ is randomly selected from the available treatments $K$. Once the treatment has been assigned, the variable $Y$ is generated according to a Bernoulli distribution $Y \sim B(\mathbb{E}[Y|X = \mathbf{x}_i, T = t_k])$.
<table><caption>Table 1: Table containing the generated values of $E[Y|T, X]$ and $\tau_{T,t_0}$ for an individual $\mathbf{x}_i$ with two dimensions.</caption><thead><tr><th>X<sub>1</sub></th><th>X<sub>2</sub></th><th>E[Y|T = t<sub>1</sub>, X]</th><th>...</th><th>τ<sub>t<sub>1</sub>,t<sub>0</sub></sub>(X)</th><th>...</th><th>T</th><th>Y</th></tr></thead><tbody><tr><td><b>x<sub>i,1</sub></b></td><td><b>x<sub>i,2</sub></b></td><td>E[Y|T = t<sub>1</sub>, X = <b>x<sub>i</sub></b>]</td><td>...</td><td>τ<sub>t<sub>1</sub>,t<sub>0</sub></sub>(<b>x<sub>i</sub></b>)</td><td>...</td><td>t</td><td>0 or 1</td></tr></tbody></table>

Table 1 illustrates the data format. We model the contrast as follows. Within each cell, individuals have a constant expectation of $Y$ depending on treatment $k$ ($E[Y|T = k, X = \mathbf{x}_i]$ is constant, given $k$), $k = 0$ being the control group. This expectation can take two values: $e_{Y+}$ and $e_{Y-}$ ($e_{Y+} > e_{Y-}$). Cells with expectation $e_{Y+}$ (respectively $e_{Y-}$) are called “positive” (respectively “negative”) cells. The uplift $k$ ($\tau_k(\mathbf{x}_i) = E[Y|T = k, X = \mathbf{x}_i] - E[Y|T = 0, X = \mathbf{x}_i]$) in a positive cell will be: $e_{Y+} - e_{Y-}$ or $e_{Y+} - e_{Y-}$. The uplift will therefore be positive (because $e_{Y+} > e_{Y-}$) or zero. Similarly, in a negative cell for a treatment $t_k$, the uplift $k$ will be negative or zero. We model the contrast using three couples of values for $e_{Y-}$ and $e_{Y+}$: (0, 1), (0.2, 0.8), (0.4, 0.6). These pairs represent the different levels of contrast between cell expectations, from the highest (0, 1) to the lowest (0.4, 0.6). We have chosen these values to be centered around 0.5 because this ensures a balanced comparison of contrast levels. The greater the difference between $e_{Y-}$ and $e_{Y+}$, the more difficult the problem is considered.

We have chosen the values 2, 4, 8, 16 and 32 for the number of possible treatments, plus a control group (treatment $T = 0$).

We set the proportion of “positive” cells (cells for which $E[Y|T = t_k, X = \mathbf{x}_i] = e_{Y+}$) at 50% of all cells. This makes it possible to simulate scenarios where only certain areas of the variable space actually benefit from the treatments. We have chosen to use 6400 individuals per treatment in the training datasets, meaning each cell contains 100 individuals. To generate the *NRA* bias, we need at least twice the number of individuals per treatment group. We therefore generated 12,800 individuals per treatment in order to introduce imbalances between treatment groups. To guarantee robust results and obtain reliable estimates of model performance, each dataset is generated 10 times, for a total of 150 different datasets (3 pairs of $(e_{Y-}, e_{Y+})$ values, 5 values for the number of treatments, and 10 repetitions for each configuration). In parallel, we generated test datasets independent of the training datasets, with 15,000 individuals per treatment group. These data sets enable us to evaluate the models on samples not used during training.
### 4.5. NRA Generation

The *NRA* bias arises from an imbalance between the distributions of individuals among the different treatments. To introduce this bias in the data, we proceed as follows.

**Creation of bias pattern.** For each treatment, 50 % of the grid cells resulting from data generation are selected to form the bias pattern.

**Imbalance in the distribution of individuals.** To introduce an imbalance between bias and non-bias cells, for a given rate of bias we modify the number of individuals assigned to bias cells. Specifically, for a bias of $X\%$, the number of individuals present in the cells of the bias pattern is reduced by $X\%$, while proportionally increasing the number of individuals in the cells outside this pattern.

**Interpretation of bias rate.** A 50% bias rate means that there are 50% fewer individuals in the cells belonging to the bias pattern than in an unbiased situation (0% bias). These individuals are redistributed to cells that are not part of the bias pattern. A 0% bias corresponds to a balanced distribution, where the number of individuals is identical between the biased and unbiased zones. Thus, for cells with 100 individuals, a 20% bias corresponds to 80 individuals in the biased cells and 120 in the unbiased cells.

This generation mechanism enables different rates of *NRA* bias to be simulated in a controlled way.

## 5. Experimental Results

### 5.1. Experiments and Results

#### 5.1.1. Influence of bias (RQ 1)

Figures 1 and 2 respectively illustrate the *RMSE* and *expected outcome* of all models as a function of the *NRA* bias rate with two-dimensional (left parts of the figures) and three-dimensional data (right parts of the figures).

We can clearly see that this bias affects the overall performance of all models, and the performances of the models are similar between two- and three-dimensional data. For all methods, performance generally degrades as bias increases, with the exception of *S-Learner* and *T-Learner LR*.

A more detailed model-by-model analysis reveals distinct behaviors. In particular, *meta-learners* based on *XGBoost* (XGB) perform very well, as does *X-Learner* with *Random Forest* (RF). In contrast, forest-based models, such as the *ED*, *Chi* and *CTS* algorithms, show inferior performance, similar
<figure><img src="image_2.png" /><figcaption>Figure 1: Comparison of model *RMSE* as a function of the *NRA* bias rate. A point is obtained by averaging the RMSE over all datasets for each number of treatments and contrast level, at a fixed bias rate. The left figure corresponds to two-dimensional datasets, and the right figure to three-dimensional datasets.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: Comparison of models’ *expected outcome* as a function of the *NRA* bias rate. A point is obtained by averaging the expected outcome over all datasets for each number of treatments and contrast level, at a fixed bias rate. The left figure corresponds to two-dimensional datasets, and the right figure to three-dimensional datasets.</figcaption></figure>

to the deep learning method *TARNet*. Finally, other *meta-learners* using logistic regression perform significantly worse, as the *LR* model is probably too simplistic to capture the complexity of the data. Secondly, as regards *R-Learner* and *DR-Learner* models, although they perform poorly according to *RMSE*, they are on a par with random forests when we consider *expected outcome*. In fact, these two criteria do not emphasize the same properties. The *RMSE* measures the precision of the uplift estimate. The *expected outcome*, on the other hand, highlights whether or not a model is able to order treatments correctly. We therefore deduce that *R-Learner* and *DR-Learner* models predict uplift values poorly, but order their predictions better. In
terms of bias sensitivity, all methods see their performance fall with increas-
ing bias *NRA*, with the notable exception of *T-Learner* and *S-Learner* based
on logistic regression. These models, too simple to capture the complexity
of the data, are less affected by bias, because their performance already ap-
proaches that of random prediction. Among the other models, the majority
appear to have comparable resistance to bias, although *R-Learner* shows
slightly less sensitivity, as does *S-Learner XGB*.

5.1.2. Influence of the number of treatments (RQ 2)

The *RMSE* is more appropriate to study the influence of the number of
treatments. Indeed, the value of the *expected outcome* naturally increases
with the number of treatments in proportion to the number of so-called
“positive” cells. This is because the more treatments there are, the more
likely it is that there will be a treatment with a positive impact for each
individual.

<figure><img src="image_4.png" /><figcaption>Figure 3: Comparison of the models’ *RMSE* as a function of the number of treatments. A point is obtained by averaging the *RMSE* over all datasets with all bias values and all contrast levels, at a fixed number of treatments.</figcaption></figure>

Since the results are similar between two- and three-dimensional data, we
present them in a combined figure. The analysis of the results (Figure 3)
shows two types of behavior of the methods. A first group of methods shows
relatively constant performance regardless of the number of treatments. This
group includes *T-Learners* and *X-Learners*. For a second group of methods,
performance declines as the number of treatments increases. This group in-
cludes forest methods (ED, Chi, CTS), S-Learners, the deep learning method TARNet, as well as R-Learners and DR-Learners.

This deterioration in performance for the second group can be explained by the increasing complexity of the problem: the more treatments there are, the more possible values there are for the treatment variable, making the learning for these methods difficult. On the other hand, *T-Learners* and *X-Learners* are little affected by this increase in the number of treatments. Indeed, in these models, the treatment variable is managed independently: a separate model is built for each treatment, making these methods agnostic to variation in the number of treatments. We can enrich our analysis by considering the *expected outcome* for each level of treatment. The results show a similar trend for all models, with the exception of *DR-Learner*. This model shows an *RMSE* that increases with the number of treatments, indicating that the numerical uplift values deteriorate progressively. However, the *expected outcome* associated with this model remains constant. This means that the order of treatments suggested by the model maintains similar performance, regardless of the number of treatments considered. Thus, for *DR-Learner*, treatment recommendations do not appear to be sensitive to variation in the number of treatments.

We can therefore conclude that *T-Learners* and *X-Learners* are particularly well suited to problems where the number of treatments is high, unlike the other methods which see their performance decline in these situations. *DR-Learner* is not effective for predicting uplift when there are a large number of treatments, as the estimated uplift values may prove unreliable in this context. On the other hand, in situations where only the recommended treatment is relevant, such as an offer recommendation system, *DR-Learner* can be considered appropriate.

### 5.1.3. Contrast influence (RQ 3)

Contrast can significantly impact model performance. Low contrast corresponds to less pronounced uplifts, making it more difficult to distinguish optimal treatments.

However the results cannot be analyzed on the basis of the raw values of the *RMSE* and the *expected outcome*. In fact, *RMSE* tends to decrease as contrast decreases, even with similar model performances. A similar phenomenon is observed for *expected outcome*. As the contrast decreases, the amplitude of the expected outcome results also decreases, and good methods see their expected outcome decrease, while bad methods see their expected
<figure><img src="image_5.png" /><figcaption>Figure 4: Expected outcome normalized for the different methods. A point is obtained by averaging the *expected outcome* over all data sets for each bias value and number of treatments, at a fixed contrast level.</figcaption></figure>

outcome increase. This is because, as the contrast decreases, good decisions are less 'rewarded' and bad decisions are less 'penalized'.

In order to analyze the results, we propose a normalization of the *expected outcome*. To do this, we consider the *expected outcome* values of the optimal model (model predicting treatments with the best impact) and the random method (model predicting treatments randomly). We normalize the results so that the *expected outcome* of the optimal method is equal to 1, and the *expected outcome* of the random method is equal to 0. The formula used is: $EO \text{ normalized} = \frac{EO - EO \text{ (Random)}}{EO \text{ (Optimal)} - EO \text{ (Random)}}$. Essentially, this normalization transforms the raw expected outcome values into a relative performance score, where 1 represents a perfect prediction and 0 represents a random prediction. This enables a comparison of model performance even though the contrast varies. Figure 4 shows the results with the values normalized in this way. As before, the results between two- and three-dimensional data are similar, so we group them together in a figure.

Analysis of results based on the *expected outcome* shows a drop of performance related to contrast for the models *X-Learner XGB*, *X-Learner RF*, *T-Learner XGB*, and *S-Learner XGB*. Forests and *TARNet* are affected, but to a lesser extent. However, methods that were better with higher contrast are still better with lower contrast.
## 5.2. Analysis and Discussion

### 5.2.1. Overall summary of model performance

Figure 5 depicts the ranking of the methods in the form of a rank study according to *RMSE* and *expected outcome*. A point is obtained by averaging the *RMSE* or *expected outcome* over all datasets, with all bias values and contrast levels. The *meta-learners* based on *XGBoost* (*X*, *T* and *S-Learner*), as well as *X-Learner RF*, stand out overall as the best performers across all datasets. Logistic regression-based models, while resistant to bias, actually perform much more poorly. Random forest methods, such as *ED*, *Chi* and *CTS*, and the deep learning method *TARNet*, perform similarly. Although these methods are robust for uplift prediction with a low number of treatments, their performance deteriorates rapidly as the number of treatments increases. They are also sensitive to variations in bias, especially *TARNet*, which limits their application in complex scenarios, such as offer recommendation. It is also worth noting that for certain methods, the ranking differs depending on whether *RMSE* or *expected outcome* is used as the evaluation metric. This is because these metrics capture different aspects of model performance. Some models are better at ranking treatments than at accurately estimating uplift values. For instance, *R-Learner XGB* achieves a better ranking in terms of *expected outcome* than in terms of *RMSE*.

<table><caption>Table 2: Average *RMSE* and *Expected Outcome* for each uplift method across all datasets including various bias rate, number of treatments and contrast level.</caption><thead><tr><th>Method</th><th>RMSE</th><th>EO</th></tr></thead><tbody><tr><td>X-Learner RF</td><td><strong>0.2149</strong></td><td><strong>0.7452</strong></td></tr><tr><td>X-Learner XGB</td><td>0.2182</td><td>0.7447</td></tr><tr><td>S-Learner XGB</td><td>0.2254</td><td>0.7337</td></tr><tr><td>T-Learner XGB</td><td>0.2274</td><td>0.7348</td></tr><tr><td>TARNet</td><td>0.3062</td><td>0.6737</td></tr><tr><td>Chi</td><td>0.3112</td><td>0.6673</td></tr><tr><td>ED</td><td>0.3142</td><td>0.6638</td></tr><tr><td>CTS</td><td>0.3282</td><td>0.6585</td></tr><tr><td>T-Learner LR</td><td>0.4094</td><td>0.5740</td></tr><tr><td>DR-Learner</td><td>0.4118</td><td>0.6432</td></tr><tr><td>S-Learner LR</td><td>0.4165</td><td>0.5146</td></tr><tr><td>R-Learner</td><td>0.4937</td><td>0.6037</td></tr></tbody></table>
<figure><img src="image_6.png" /><figcaption>Figure 5: Overall ranking of different models according to performance measures. The shapes indicate the performance measure, while the lines show the standard deviation of each model’s ranking according to performance measures. *Average RMSE* corresponds to the average *RMSE* of the methods on the datasets with all the number of treatments. *RMSE T* = {3,5,9} (resp. *RMSE T* = {17,33}) corresponds to the average *RMSE* of the methods on datasets with 3, 5 or 9 (resp. 17 or 33) treatments. *Average Expected Outcome* corresponds to the average *expected outcome* of the methods on the datasets with all the number of treatments. *Expected Outcome T* = {3, 5, 9} (resp. *Expected Outcome T* = {17, 33}) corresponds to the average *expected outcome* of the methods on the data sets with 3, 5 or 9 (resp. 17 or 33) treatments.</figcaption></figure>

Table 2 summarizes the average performance of each method across all datasets. The methods are sorted according to their RMSE values. The overall ranking of the methods is similar to that shown in Figure 5. Additionally, Table 2 quantifies the exact performance values.

### 5.2.2. *Lessons learned from experiments*

The experimental results provide several insights into multi-treatment uplift models in the presence of bias, and they are the same whether the data is two-dimensional or three-dimensional. *Meta-learners* based on *XGBoost* are promising candidates for applications requiring the modeling of multiple treatments. Their ability to capture non-linearly separable data while maintaining robust performance makes them particularly suited to complex uplift problems. Based on *RMSE*, *T-Learner* outperforms *S-Learner* when the number of treatments is large, and conversely, *S-Learner* performs better when the number of treatments is small. Indeed, as shown in Section 5.1.2, *T-Learner* is not affected by a large number of treatments. In contrast, *S-Learner*'s performance degrades as the number of treatments increases,
causing it to fall behind *T-Learner* in such scenarios. Therefore, T-Learners
should be preferred for applications with a large number of treatments. Mod-
els based on logistic regression (such as *S* and *T-Learners LR*) show low sen-
sitivity to bias and contrast, but this is explained by their poor performance.
For applications with more complex data, methods using more sophisticated
models such as *XGBoost* should be preferred. Random forest methods are
not well suited to scenarios with a large number of treatments. The trend
observed is that these methods become inefficient as soon as the number of
treatments exceeds a certain threshold. The conclusion is similar for *TAR-
Net*. Finally, some models perform well according to *expected outcome*, but
not according to *RMSE*; the order of predicted treatments is good, but not
the uplift estimate. Calibration [23] is an avenue of research to improve this
estimate.

**6. Conclusion and Perspectives**

In this paper, we have investigated the impact of bias on the performance
of state-of-the-art multi-treatment uplift models. To our knowledge, this
is the first study to investigate the effects of bias in multi-treatment uplift
models. We have designed an experimental protocol for generating synthetic
uplift data and for incorporating and quantifying bias. Additionally, we have
proposed a new normalization method for interpreting the expected outcome,
allowing for fairer comparisons across different contrast levels.

The experimental results show that uplift models react differently to bias
and highlight some methods that work particularly well when dealing with
many treatments. In particular, meta-learners based on XGB perform well
and stay robust even as bias and the number of treatments increases. On the
other hand, simpler models, such as those using logistic regression, are less
accurate but seem less affected by bias because they are less flexible. We also
found that some models may not estimate uplift values very well, but still
manage to correctly rank the treatments. This suggests that it is important
to evaluate both the accuracy of uplift predictions and how well the model
ranks treatments.

This better understanding of the behavior of uplift methods will enable
us to define techniques to resist bias, which is an obvious further work. It
will also be interesting to evaluate these models in real-world contexts, such
as recommendation systems. Another future direction is the use of calibra-
tion strategies [23] to improve uplift estimation without sacrificing ranking
performance. On the other hand, it would be promising to investigate how
information from certain treatments can be used for other treatments: the
idea is to group individuals from two treatments that are similar with respect
to these treatments in order to limit the *NRA* bias.

References

[1] Radcliffe, N. J., Surry, P. D.: Real-world uplift modelling with significance-based uplift trees. In: White Paper TR-2011-1, Stochastic Solutions, 1–33, (2011).

[2] Rubin, D.: Estimating causal effects of treatments in randomized and nonrandomized studies. In: Journal of educational Psychology 66(5), 688, (1974).

[3] Zhang, W., Li, J., Liu, L.: A unified survey of treatment effect heterogeneity modelling and uplift modelling. In: ACM Computing Surveys (CSUR) 54, 8 (2021), 1–36.

[4] Gutierrez, P., Gérardy, J. Y.: Causal Inference and Uplift Modeling: A review of the literature. In: JMLR: Workshop and Conference Proceedings 67:1–13, 2016.

[5] Vanschoren, J.: Meta-learning. In: Automated machine learning : methods, systems, challenges, 35–61, (2019).

[6] Guelman, L., Guillén, M., Pérez-Marín, A. M.: Uplift random forests. In: Cybernetics and Systems 46(3-4), 230–248, (2015).

[7] Berk, R. A., Smyth, G. K., Sherman, L. W.: When random assignment fails : Some lessons from the minneapolis spouse abuse experiment. In: Journal of Quantitative Criminology 4(3), 209–223, (1988).

[8] Olaya, D., Coussement, K., Verbeke, W.: A survey and benchmarking study of multitreatment uplift modeling. In: Data Mining and Knowledge Discovery 34, 273–308, (2020).

[9] Gubela, R. M., Lessmann, S., Stöcker, B.: Multiple treatment modeling for target marketing campaigns: A large-scale benchmark study. In: Inf. Syst. Frontiers 26(3), 875–898, (2024).
[10] Rafla, M., Voisine, N., Crémilleux, B.: Evaluation of uplift models with non-random assignment bias. In: 20th Int. Symposium on Intelligent Data Analysis, IDA 2022, Rennes, France, April 20-22, pp. 251–263. Springer.

[11] Rubin, D.: Using propensity scores to help design observational studies : Application to the tobacco litigation. In: Health Services and Outcomes Research Methodology 2, 169–188, doi:10.1023/A :1020363010465, (2001).

[12] Rzepakowski, P., Jaroszewicz, S.: Decision trees for uplift modeling with single and multiple treatments. In: Knowl. Inf. Syst. 32(2), 303–327, (2012).

[13] Shalit, U., Johansson, F. D., Sontag, F.: Estimating individual treatment effect: generalization bounds and algorithms. In: Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 3076–3085). PMLR, (2017).

[14] Liu, D., Tang, X., Qiao, Y., Liu, M., Sun, Z., He, X., Ming, Z.: Benchmarking for Deep Uplift Modeling in Online Marketing. arXiv preprint arXiv:2406.00335 (2024). Available at: https://arxiv.org/abs/2406.00335

[15] Zhao, Y., Fang, X., Simchi-Levi, D.: Uplift modeling with multiple treatments and general response types. In: 2017 SIAM International Conference on Data Mining, (2017).

[16] Diemert, E., Betlei, A., Renaudin, C., Amini, M.R.: A Large Scale Benchmark for Uplift Modeling. In: KDD, 2018, London, United Kingdom. hal-02515860.

[17] Radcliffe, N.: Using control groups to target on predicted lift: Building and assessing uplift model. In: Direct Marketing Analytics Journal, 2007, pages 14–21.

[18] Athey S., Imbens G. W.: Machine learning methods for estimating heterogeneous causal effects. stat, 1050(5):1–26, 2015.
[19] Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10):4156-4165, Feb 2019.

[20] Behram Hansotia and Brad Rukstales. Incremental value modeling. Journal of Interactive Marketing, 16:35-46, 2002.

[21] Edward H. Kennedy. Optimal doubly robust estimation of heterogeneous causal effects, 2020.

[22] X Nie and S Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319, 09 2020.

[23] Y. T. Park, T. Xu, and M. Anany. Enhancing Uplift Modeling in Multi-Treatment Marketing Campaigns: Leveraging Score Ranking and Calibration Techniques. Best Buy Co., Inc, 2024.