# Evaluating the Efficacy of Instance Incremental vs. Batch Learning in Delayed Label Environments: An Empirical Study on Tabular Data Streaming for Fraud Detection

Kodjo Mawuena Amekoe, Mustapha Lebbah, Gregoire Jaffre, Hanene Azzag, Zaineb Chelly Dagdia

► To cite this version:

Kodjo Mawuena Amekoe, Mustapha Lebbah, Gregoire Jaffre, Hanene Azzag, Zaineb Chelly Dagdia. Evaluating the Efficacy of Instance Incremental vs. Batch Learning in Delayed Label Environments: An Empirical Study on Tabular Data Streaming for Fraud Detection. 2025. hal-05237366

HAL Id: hal-05237366

https://hal.science/hal-05237366v1

Preprint submitted on 24 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluating the Efficacy of Instance Incremental vs. Batch Learning in Delayed Label Environments: An Empirical Study on Tabular Data Streaming for Fraud Detection

Kodjo Mawuena Amekoe¹,³, Mustapha Lebbah¹,², Gregoire Jaffre³,
Hanene Azzag¹, and Zaineb Chelly Dagdia²

1- Sorbonne Paris Nord University - LIPN, UMR CNRS 7030, Villetaneuse, France

2- Paris-Saclay University - DAVID Lab, UVSQ, Versailles, France

3- Groupe BPCE, 7 promenade Germaine Sablon 75013 Paris, France

**Abstract.** Real-world tabular learning production scenarios typically involve evolving data streams, where data arrives continuously and its distribution may change over time. In such a setting, most studies in the literature regarding supervised learning favor the use of instance incremental algorithms due to their ability to adapt to changes in the data distribution. Another significant reason for choosing these algorithms is *avoid storing observations in memory* as commonly done in batch incremental settings. However, the design of instance incremental algorithms often assumes immediate availability of labels, which is an optimistic assumption. In many real-world scenarios, such as fraud detection or credit scoring, labels may be delayed. Consequently, batch incremental algorithms are widely used in many real-world tasks. This raises an important question: "In delayed settings, is instance incremental learning the best option regarding predictive performance and computational efficiency?" Unfortunately, this question has not been studied in depth, probably due to the scarcity of real datasets containing delayed information. In this study, we conduct a comprehensive empirical evaluation and analysis of this question using a real-world fraud detection problem and commonly used generated datasets. Our findings indicate that instance incremental learning is not the superior option, considering on one side state-of-the-art models such as Adaptive Random Forest (ARF) and other side batch learning models such as XGBoost. Additionally, when considering the interpretability of the learning systems, batch incremental solutions tend to be favored. Code: https://github.com/anselmeamekoe/DelayedLabelStream

**Keywords:** Stream · Label delay · Adaptive learning · Tabular Data · Interpretability

## 1 Introduction

Currently, monitoring and adapting/updating machine learning models is a clear requirement in many real-world problems. This is mainly due to the changes that may occur in the data distribution over time, violating the identical distribution hypothesis
(between the training and production/test dataset) usually made in a classical offline ma-
chine learning setting. Additionally, it is impractical to store all data in memory indef-
initely in tasks where data is generated at high speed. Over the last two decades, many
efforts have been made to develop learning systems that respect these requirements and
are able to provide predictions in real-time. Within these solutions, we can differenti-
ate instance incremental models, which have the capability to update their weights or
architecture using just one instance. Most state-of-the-art models in this category are
based on the Hoeffding Tree [12] and are often equipped with drift or change detection
mechanisms such as ADaptive WINdowing (ADWIN) [5]. We also have, batch incre-
mental learning systems in which the stream datasets are stored in chunks or batches
that are further used to learn a new static model or update the old one. An important
consideration in the evolving data stream is label delay [29,19,18,22], where labels can
be received with important delays depending on the task. For example, in card fraud
detection, the true nature of a card transaction (fraudulent or genuine) is only known
after some days or weeks, unless the card owner notices an anomaly on its card and
reports it immediately to the bank. In such a situation, it is unavoidable in a supervised
paradigm to store observations waiting for their labels; otherwise, semi-supervised or
unsupervised systems should be considered [18]. Besides labeling delay and for most
real-world problems, it is often a requirement to store observations for some fixed peri-
ods. For instance, banks are legally obliged to store transaction information for several
months and should make it available upon customer request. In this study, we inves-
tigate whether instance incremental solutions remain the best option in such delayed
situations by considering their predictive performance and computational efficiency.

Moreover, the interpretability of automatic decision-making systems is required in
many real work cases (e.g., GDPR, Article 22, IA Act in Europe). Regarding learning
with evolving data streams, interpretability involves understanding the decisions made
by the model/system, as well as changes within the data distribution and the model
itself [21]. We show that state-of-the-art batch incremental interpretable models outper-
form instance incremental models in terms of accuracy and explain why they might be
preferable for inherent interpretability for evolving data streams. Overall, we summa-
rize our contributions as follows:

- We provide background on delay and interpretability in learning with evolving data streams (Section 2).

- We designed a realistic supervised evaluation framework based on interleaved chunks, combined with label delays (Section 3).

- We empirically compare instance incremental, and batch incremental algorithms on the designed evaluation framework and revealed the superior performance of the latter in delayed settings using benchmark tabular stream datasets and real-world fraud modeling problem (Section 4).

- We analyze the performance of batch incremental models and demonstrate the im-
portance of storing past observations whenever possible, especially for tasks where
the target event is rare, such as fraud detection (Section 4.3).
## 2 Related work

### 2.1 Performance evaluation strategy.

The most common strategy used to assess performance in data streams is the **prequen-
tial** evaluation, also known as the test-then-train strategy. In this approach, each obser-
vation is first used for testing (updating the evaluation metric) and then for updating the
learning model. Typically, prequential evaluation is combined with a forgetting mech-
anism [16,15] (such as a sliding window or fading factor) to place more weight on
recent performance. Prequential evaluation is usually applied in settings where labels
are available immediately after the observation or after a fixed number of instances.
Observations may be stored in a buffer or chunk for supervised problems with varying
and significant label delays. Once the labels become available, they can be used to test
and/or update/retrain the learning model. This strategy is sometimes called **interleaved
chunks**. The interleaved chunks approach allows the use of common batch evaluation
metrics, such as AUCROC [8] and AUCPR [7].

### 2.2 Batch incremental versus Instance incremental learning.

The utility of rigorously comparing batch incremental and instance incremental ap-
proaches is underestimated in the learning literature on evolving data streams. This is
arguably due to the preference for automatically updating the model over time and/or
avoiding the storage of observations in memory whenever possible, which favors in-
stance incremental solutions. Consequently, attempts to use batch incremental solutions
are often biased towards using *small* predefined chunks/batches/windows of instances
to retrain or update an old model automatically. Among these approaches, we cite Ac-
curacy Weighted Ensemble (AWE) [31] in which base classifiers (typically C4.5, SVM,
Linear models) are weighted based on their expected test accuracy over time. Another
example is the Adaptive XGBoost (AXGB) solution [26], where new decision tree
models are created and appended to the boosted ensemble. More precisely, once the
ensemble is full, the oldest member is removed before appending a new one (the push
strategy), or older members are directly replaced with newer ones (the replacement
strategy).

Dynamic or continuous offline optimization of batch learners (using data collected
over time, e.g., days, months) alternate with online inference (real-time prediction)
is often neglected in the literature when comparing batch and instance incremental
learning; however, this strategy remains a widely used approach in real-world produc-
tion/deployment or *Human in Loop* Machine Learning Operations (MLOps) because
it makes human inspection and validation (bias correction, understanding of changes)
easier, thereby increasing trustworthiness. Among the studies that compare batch in-
cremental and instance incremental learning in a supervised setting, we note the work
by Read et al. [30], from which we summarize the main disadvantages of these two
approaches in Table 1.

Their experimental result reveals the superior predictive performance (but longer
running time) of the instance incremental ensemble Leveraging Bagging with Hoeffd-
ing Tree as a base learner (LB-HT) [6] over AWE using a window of 500 observations
<table><caption>Table 1: Comparative Disadvantages of Batch and Instance Incremental Learning</caption><thead><tr><td>Learning Approach</td><td>Main disadvantages</td></tr></thead><tbody><tr><td>Batch</td><td><ul><li>Require deciding the batch/windows size for retraining/updating the model;</li><li>Cannot learn the most recent examples until a new batch is complete.</li></ul></td></tr><tr><td>Instance</td><td><ul><li>Only learns a concept correctly from a large number of examples (e.g., the convergence of HT to a batch-trained Decision Tree is asymptotic and guaranteed only in a stationary environment) [12,21];</li><li>Has fewer established results than batch learning (e.g., evaluation, bias correction, interpretability).</li></ul></td></tr></tbody></table>

and an ensemble of 10 learners maximum. In [26], the window size was increased to 1,000 and the number of learners to 30 for batch incremental models; however, conclusions remain the same, i.e., the state-of-the-art instance incremental solution namely Adaptive Random Forest (ARF) [17] showed an overall best predictive accuracy over both the batch solution AXGB proposed by the authors and the AWE strategy. An important point missed in the comparison was hyperparameter optimization. Only a fixed reference configuration of hyperparameters was used when comparing models for all benchmark datasets. Can hyperparameter optimization change the final conclusion? The authors in [26] demonstrated the influence of optimization steps on the performance of the ensemble of XGBoost (called BXGBoost in the paper), which implies an increase of 14% in the average accuracy becoming the best batch incremental of their benchmarking.

In this study, we designed evaluation in accordance with real-world production scenarios and showed that batch incremental solutions such as XGBoost can have a superior predictive ability with a thorough optimization step.

## 2.3 Label delay in learning with evolving data stream

When learning from data streams, there are three main possibilities regarding label availability [17]: (i) immediate, (ii) delayed with a finite time interval, and (iii) delayed indefinitely. There are also settings where all these possibilities can occur simultaneously [18]. Most studies in the supervised paradigm assume immediate availability; however, in many real-world tasks, the label may arrive with an important delay. The (finite) delay mechanism can be *deterministic*, meaning fixed for every value of the feature vector, or (*semi*) *stochastic*, meaning it follows an unknown probability distribution [29]. An example of a semi-stochastic case is fraud detection, in which the labels
of fraudulent transactions are quickly revealed, whereas those of genuine transactions
are mostly revealed after a fixed prescription period. The authors of [29] proposed an
adaptive Linear Discriminant Analysis classifier combined with a weighing scheme to
handle the labeling delay using a real-world credit scoring dataset, but their analysis
was limited to only linear models, and no specific evaluation procedure was proposed.
In [17], the influence of label delay was investigated using a fixed delay mechanism
of 1,000 instances, and the authors concluded that this resulted in an important perfor-
mance drop for the ARF and Leveraging Bagging models. Unfortunately, the compar-
ison did not include batch incremental algorithms, and while stochastic delay appears
to be the most common delay mechanism in real-world problems, it was not investi-
gated. In [19], the authors designed a delayed evaluation framework, called continuous
re-evaluation, where the goal is to assess the system's or algorithm's capability to refine
its predictions over time before the delayed label arrives. Although this framework is
quite interesting for flight data, it is infeasible for many problems (e.g., fraud detection
and online credit scoring) to request the model to make predictions several times for
the same instance. Therefore, our evaluation in this study primarily concentrates on the
initial (first) prediction request. Additionally, the evaluation includes the comparison of
instance and batch incremental learning, which, to the best of our knowledge, is still
missing in the literature.

## 2.4 Interpretability in learning with evolving data stream

Much effort has been made in model interpretability in static settings, where the typ-
ical objective is to understand the decisions made by the underlying model. For this
purpose, two approaches exist: using post hoc tools such as SHAP [24] to explain the
decision of black box models or using a directly inherently interpretable model such
as an Explainable Boosting Machine [28] and TabSRA [3]. Although interpretability
in an evolving data stream is still in its early stages, the two approaches used in static
settings can also be adapted to stream or online settings. Regarding post hoc solutions,
we cite iSAGE [27] which offers an incremental version to the Shapley Additive Global
Importance (SAGE) [10] tool. Unlike static settings, interpretability also requires un-
derstanding changes that occur in the model behaviors or, at least, in its post hoc ex-
planations caused by incremental model updates [22]. Unfortunately, these approaches
have to contend with the reliability issue that post hoc methods suffer from in the static
settings [23]. Therefore, authors such as [22] favor the use of inherently interpretable
models for evolving data stream problems, where the more interpretable to humans is
likely to change less over time [21]. Based on this consideration, our experimental re-
sults reveal that batch incremental inherently interpretable models, besides being more
interpretable (as their architecture or weights change less frequently than their instance
incremental counterparts, and therefore easier for human tracking and understanding),
demonstrate superior predictive performance compared to their instance incremental
counterparts.
# 3 Problem formalization

## 3.1 Label delay

We consider a supervised learning setting for a continuous data stream $S_i = \{(\mathbf{x}_i, y_i)\}$ with $i = 1, ..., T$ where $T \to \infty$, $i$ is the unique identifier of each instance. For binary classification $y_i \in \{0, 1\}$ or $y_i \in \mathbb{R}$ for regression tasks. The input feature vector is $\mathbf{x} = (x_1, ..., x_p) \in \mathbb{R}^p$ (we do not consider settings where the input dimension changes over time).

In a delayed setting, labels are available with some delay. We denote by $dt(\mathbf{x}_i)$ the timestamp when observation $\mathbf{x}_i$ becomes available, $dt(y_i)$ is the timestamp of the corresponding label's arrival, and $\Delta t(\mathbf{x}_i) = dt(y_i) - dt(\mathbf{x}_i)$ is the label delay. Obviously, $\delta t(y_i) \ge dt(\mathbf{x}_i)$ i.e., $\Delta t(\mathbf{x}_i) \ge 0$ and $\Delta t(\mathbf{x}_i) = 0$ corresponds to the particular case where $y_i$ becomes immediately available (before the observation following $\mathbf{x}_i$). In general, the units of delay are seconds, minutes, days, and months, but there are also simplified settings where the delay is evaluated in terms of instances [17,19]. In such delayed settings, it is unavoidable to store observations until their labels become available. More precisely, once available, label $y_i$ can be used to update the metric and couple $\{(\mathbf{x}_i, y_i)\}$ to update the learning model. Following [19], we denote by $S_i = \{(\mathbf{x}_i, ?)\}$ at the timestamp $dt(\mathbf{x}_i)$, i.e., when the label is not yet available and $S_i = \{(., y_i)\}$ at the timestamp $dt(y_i)$, i.e., when the label becomes available.

## 3.2 Proposed predictive performance evaluation methodology

Algorithm 1 summarizes the learning process for batch incremental models. The inference is made in real-time, i.e., incrementally per instance, as in common stream learning, but the evaluation metric is applied to a chunk or batch of predictions (with potentially varying sizes). The batch is chosen such that there are sufficient labeled observations for consistent performance evaluation (we refer to this $B\_label[id_B].isFull()$ line 11 Algorithm 1). For real-world problems, this batch is usually defined in terms of time, for example, one day, one month of predictions, or one year, depending on the speed of the stream and the label delay. For the instance incremental and batch incremental methods, the observations are stored in a buffer $B_X$, waiting for their labels, which may be available after some delay. The main difference in data storage between the two learning strategies lies in the updating requirements. For instance incremental learning, every observation $\mathbf{x}_i$ is stored in the buffer, and once its label $y_i$ becomes available, the pair $(\mathbf{x}_i, y_i)$ is used immediately to update the model. For batch incremental learning, observations are stored until there are enough labels for retraining or updating the batch model (line 14 Algorithm 1). Therefore, batch incremental systems may require more storage resources than instance incremental ones. However, in situations where delay is significant and variable, the storage requirements can be quite similar, as highlighted in the *fraud dataset* (Section 4.1). Without loss of generality, we assume in this study that the number of labeled instances required for retraining or updating the supervised batch models is equal to or greater than the number required for periodic evaluation.

Finally, our evaluation framework includes the option of using a pre-trained model at the beginning of the stream for both instance incremental and batch incremental
learning. For many real-world problems, an initial model is typically optimized of-
line using data collected over time before deploying a learning system. Therefore, the
stream data represents the starting point of production or deployment. This practice is
becoming common in the incremental learning literature, where a fraction of data is
used either to train an initial model before the stream begins [21] or for hyperparameter
optimization [26].

**Algorithm 1 Evaluation for batch incremental models**

**Require:** S₁, S₂, S₃,... - data stream, f-initial pretrained model, evalckt - evaluation met-
ric e.g., Accuracy, AUCROC, AUCPR.

Ensure: B_X, B_label, B_pred, - buffer for storing observations, labels, predictions respectively

1: **for** S<sub>i</sub>, i = 1, ..., **do**

2:   i ← get_index(S<sub>i</sub>)

3:   id<sub>B</sub> ← get_batch(S<sub>i</sub>)

4:   **if** S<sub>i</sub> = {⟨x<sub>i</sub>, ?⟩} **then**

5:   y<sub>pred</sub> ← f(x<sub>i</sub>)

▷ Get the real-time prediction

6:   B_X[id<sub>B</sub>][i].add(x<sub>i</sub>)

▷ Add the observation to the buffer

7:   B_pred[id<sub>B</sub>][i].add(y<sub>pred</sub>)

8:   **else if** S<sub>i</sub> = {(. , y<sub>i</sub>)} **then**

9:   B_label[id<sub>B</sub>][i].add(y<sub>i</sub>)

10: **end if**

11: **if** B_label[id<sub>B</sub>].isFull() **then**

12:   result ← eval<sub>metric</sub>(B_label[id<sub>B</sub>], B_pred[id<sub>B</sub>])

13:   display(id<sub>B</sub>, result)

14: **if** B_label.isFullForTrain() **then**

15:   i<sub>train</sub>, labels ← Y_label.get_train_batch()

16:   X ← B_X.get_batch_data(i<sub>train</sub>)

17:   train(f, {X, labels})

▷ Update the model

18:   update(B_X)

▷ update the buffer by deleting unnecessary observations

19:   update(B_label)

20:   update(B_pred)

21: **end if**

22: **end if**

23: **end for**

4 Experiment analysis

4.1 Experiment setup

**Datasets** We consider two types of datasets: (1) benchmark datasets for generated bi-
nary classification, well-known in the literature on evolving data streams [17,26,20,21],
denoted as generated benchmark, and (2) a real-world bank transfer Fraud detection
dataset.

*Generated benchmark.* The datasets in this benchmark are publicly available and are summarized in Table 2:
<table><caption>Table 2: Datasets. N: number of instances, p: number of features, Type: Synthetic and Real. Drift A: abrupt, G: gradual, I<sub>f</sub>: incremental fast, and ?: unknown, MC: Minority Class</caption><thead><tr><th>Datasets</th><th>N</th><th>p</th><th># classes</th><th>Type</th><th>Drift</th><th>MC (%)</th></tr></thead><tbody><tr><td>AGR<sub>a</sub></td><td>1M</td><td>9</td><td>2</td><td>S</td><td>A</td><td>47.17</td></tr><tr><td>AGR<sub>g</sub></td><td>1M</td><td>9</td><td>2</td><td>S</td><td>G</td><td>47.17</td></tr><tr><td>HYPER<sub>f</sub></td><td>1M</td><td>10</td><td>2</td><td>S</td><td>I<sub>f</sub></td><td>50.00</td></tr><tr><td>SEA<sub>a</sub></td><td>1M</td><td>3</td><td>2</td><td>S</td><td>A</td><td>40.09</td></tr><tr><td>SEA<sub>g</sub></td><td>1M</td><td>3</td><td>2</td><td>S</td><td>G</td><td>40.09</td></tr><tr><td>Fraud</td><td>~ 6.5 M</td><td>18</td><td>2</td><td>R</td><td>?</td><td>&lt;0.10</td></tr></tbody></table>

- AGR<sub>a</sub> [17,26,20]: This dataset includes six discrete features and three continuous features. Instances are categorized into two classes using various functions, some of which adhere to decision rules, making it conducive to decision tree analysis. An abrupt drift occurs after every 250,000 instances (three in total).

- AGR<sub>g</sub> [17,26,20]: Similar to AGR<sub>a</sub>. Here, a gradual drift is used.

- HYPER<sub>f</sub> [17,26,21]: Simulates an incremental (fast) drift by changing the equation of hyperplane separating the two classes over time. This dataset is, therefore, linear model friendly. The number of features is set to 10.

- SEA<sub>a</sub> [17,26,21]: It comprises three continuous features (x<sub>1</sub>, x<sub>2</sub>, x<sub>3</sub>), with only the first two being pertinent to the target class. The first two dimensions are divided into four blocks. Within each block, an instance is assigned to class 1 if x<sub>1</sub> + x<sub>2</sub> ≤ θ, and to class 0 otherwise, with θ taking values from the set {8, 9, 7, 9.5}. Additionally, an abrupt drift occurs after every 250,000 instances (three in total).

- SEA<sub>g</sub> [17,26]: Similar to SEA<sub>a</sub>. Here, a gradual drift is employed rather than an abrupt one.

**Induce label delay in the generated benchmark.** First, 10% of each dataset (100,000 instances) is reserved as offline collected data, used for hyperparameter tuning [26] and pretraining an initial model [21] in both instance incremental and batch incremental learning. Therefore, 90% of each dataset is used for stream (online) evaluation. Each evaluation batch (Algorithm 1) contains approximately 1% of the dataset. Specifically, the evaluation batch size is generated following a Poisson distribution with a mean of 10,000. This varying evaluation batch size effectively reflects real-world settings, where monitoring logs of deployed learning systems must be displayed periodically (every day, week, or year, depending on the stream's speed). However, the number of instances may vary from one period to another.

In this study, we generated the label delay by assuming it is stochastic and follows a Poisson distribution. More precisely, we assume that the delay Δt(x<sub>i</sub>) for each instance observation x<sub>i</sub> follows a Poisson distribution with a mean of α × 10,000 instances, where α ∈ {0, 0.1, 1, 2, 3, 4, 5, 6, 7}.

**Fraud dataset.** The task consists in using known information about fraudulent bank transfer scenarios (via supervised machine learning) to assign a fraud score to each newly added IBAN (International Bank Account Number). Specifically, the goal is to
predict the probability that a new IBAN will be used for fraudulent transfers within the
next 30 days. The motivation for investigating instance incremental learning is the vari-
ation in label delay based on class labels. Most fraudulent IBANs are identified within
a few days, while the remaining IBANs are automatically labeled as genuine after 30
days. Hence, one might question whether quickly adapting the learning model with
these fraudulent instances could yield better performance compared to a batch learning
strategy, where the model is updated only after collecting labels for at least a month. For
example, in batch learning, labels from September are collected by the end of October,
and the model is updated in early November, whereas instance incremental learning
updates the model with September's fraudulent data as soon as it becomes available. A
key challenge is the class imbalance, where only about 0.10% of instances are fraudu-
lent on average. As a result, it is necessary to store at least 99.90% of instances for 30
days due to label delay. Additionally, unlike datasets in the *Generated benchmark*, this
dataset may experience various types of drift, such as gradual drift combining abrupt
changes, and some concepts may reoccur over time. The sample used in this study com-
prises approximately 6.5 million observations collected from September 2021 to August
2022. The initial three months (September, October, and November) are reserved as of-
fline collection for hyperparameter optimization and initial model training. The stream
evaluation spans eight months (from January 2022 to August 2022), with each month
comprising approximately 540,000 instances.

**Models benchmarked** For instance incremental learning, we consider:

- **Adaptive Random Forest (ARF)** [17]: This model is one of the best performing among instance incremental algorithms [17,26,20]. We use ARF with the ADWIN detector [5].

- **Leveraging Bagging with Hoeffding Tree as base learner (LB-HT)** [6]: We include this model in our comparison as it is the best performing model in [30].

- **Leveraging Bagging with Logistic Regression as base learner (LB_LR)** [6]: We explore in this study a non tree base learner for Leveraging Bagging. More specifically we use the Logistic Regression (LR) with a Stochastic Gradient Optimizer (SGD).

Alongside state-of-the-art instance incremental models, we include the following glass box algorithms for interpretability, as discussed in Section 2:

- **Logistic Regression (LR)**: This model can be used in both batch or instance in-
incremental models. In this study, we use it as an instance incremental model by
combining it with a Stochastic Gradient Optimizer (SGD)

- **Hoeffding Tree (HT)** [12]: This model offers interpretable and adaptive decision rules by following paths in the tree, particularly if it's shallow. Thus, we cap the maximum depth at 6. Additionally, we employ Naive Bayes Adaptive for leaf prediction, which enhances the performance of the traditional HT [21,14].

- **Hoeffding Adaptive Tree (HAT)**: The same model as the Hoeffding Tree, but with ADWIN [5] for drift handling.

For all instance incremental models, we used the Python-based package River [25]. As
batch incremental models, we consider:
- **XGBoost** [9]: It is a leading model in batch learning across various real-world applications and tabular learning competitions. We make XGBoost adaptive by re-training it from scratch after every new batch (Algorithm 1, line 17), referred to as **r_XGBoost**. Following [26,11], we use a stacking of the last $M = 3$ retrained XG-Boost models, denoted as **B_XGBoost**. Although stacking may increase prediction time, it helps preserve some previously learned concepts. We don't include Adaptive XGBoost (AXGB) as it was already outperformed by the optimized version in **B_XGBoost**.

- **Explainable Boosting Machine (EBM)** [28]: EBM is a Generalized Additive Model with pairwise interaction terms, featuring shape functions that are piece-wise constant and optimized using a boosting mechanism. We include this model in our comparison for its inherent interpretability—its shape function can be monitored to detect changes over time—unlike XGBoost. Similarly to XGBoost, we used re-training from scratch (**r_EBM**) and stacking (**B_EBM**) to make the EBM model adaptive over time. Rather than stacking naively ($M = 3$) EBM models, which may compromise interpretability, we merged the shape functions<fnref n="1" /> of the stacked models into a single final EBM model.

- **TabSRA** [3]: It is also an inherently interpretable model based on an attention mechanism called self-reinforcement. Monitoring TabSRA's attention weights and inherent feature attributions can help to understand changes over time. We also used the retraining (**riry_TabSRA**) to make the TabSRA model adaptive. In addition, we investigate the continual fine-tuning of the weights of an initially trained TabSRA model. The learning rate is reduced over time to avoid a catastrophic forgetting of previously learned concepts. This strategy is named **u_TabSRA**.

## Experiment details on the generated benchmark

**Tuning step.** The experiments on the *Generated benchmark* are done on 64-Core Processor CPU machine. For both instance and batch incremental models, hyperparameter optimization is conducted for each dataset using 10% of the data (considered as offline collected data). Specifically, 30 trials of Bayesian optimization (using Optuna [2]) are employed, with a maximum search time of 6 hours for each model and dataset. The best validation model (using prequential evaluation for instance incremental models and a 70/30 train/validation split for batch incremental models) is initialized at the stream's outset and updated over time for instance incremental models. To ensure a fair comparison, hyperparameter optimization is not conducted during the stream for batch incremental models; instead, the best hyperparameter configuration obtained from offline optimization is used for model retraining and updating, as outlined in [26]. Furthermore, only 70% of the batch is used for training during the data stream (with 30% reserved, as commonly practiced in batch learning, for validation, post hoc analysis, early stopping, bias correction, etc.). For simplicity, the evaluation batch in the *Generated benchmark* matches the training batch (Algorithm 1, lines 11 and 14)<fnref n="2" />.

<footnote n="1">¹ https://interpret.ml/docs/python/examples/merge-ebms.html</footnote>

<footnote n="2">² In this specific scenario, our proposed evaluation methodology aligns with interleaved chunks (Section 2) for batch incremental models.</footnote>
For the *Fraud* dataset, we examined a scenario where the evaluation batch spans one month while the training batch covers three months (Section 4.3). Finally, for all ensemble models (ARF, LB, XGBoost), the maximum number of learners is set to 100, and for tree-based models (ARF, LB\_HT, HAT, HT), the maximum depth is limited to 6 (please check GitHub<fnref n="3" /> for information regarding the hyperparameter search space).

**Evaluation metric and results aggregation.** For numerous binary classification tasks, such as fraud detection and online credit scoring, practitioners often seek the classifier’s confidence output rather than just the binary class obtained by applying a threshold. Therefore, we suggest using AUCROC [8] as the evaluation metric for each stream batch. To summarize the results across all batches (90 in total), we calculate the average and standard deviation of the AUCROC values. Additionally, we include the running time as a metric to assess the computational efficiency of each algorithm.

We point out that we do not use cross-validation in our evaluation process, as we cannot alter observations’ order without introducing artificial drift [21]. There are approaches of using different seeds in model training if applicable [20] or using distributed evaluation [4]. However, they are very computationally intensive, especially when adding hyperparameters optimization steps. For more details about the statistical significance in evaluating data streams, we refer the interested readers to [4,22,21]. We analyze in the next section the results on the *Generated benchmark*, which contains homogeneous datasets with well-known drift types and difficulties.

## 4.2 Results on the generated benchmark

**Do instance incremental algorithms really outperform batch incremental ones?**

We analyze the predictive performance (Table 3 and Fig. 1) by considering first a no delay setting [30,17,26,20], however using an initial tuning step for both instance and batch incremental as described in Section 4.1 in contrast to previous work. The key findings are:

- With an initial tuning step, the best batch incremental system with an *appropriate* learning batch often achieves a comparable or even superior performance to instance incremental systems for known unique type of drift (including abrupt, gradual, incremental).

- When considering interpretable solutions, batch incremental models (such as EBM and TabSRA) offer the advantage of less frequent changes in weights and architecture, making them easier for humans to follow. In addition, these models generally outperform instance incremental models in terms of predictive performance.

An important observation regarding the *Generated benchmark* is the impact of the inductive biases of the learning model. Specifically, linear-based models like LR and LB\_LR perform poorly on piece-wise constant-like datasets (AGR<sub>a</sub>, AGR<sub>g</sub>). Conversely, tree-based models (DT, HAT, HT) are less effective for modeling continuous (linear) data streams like HYPER<sub>f</sub>, although ensembling mitigates this issue (for XGBoost, EBM, LB\_HT, ARF). Hence, we emphasize the importance of alerting readers/practitioners that relying solely on average performance measures from *synthetic benchmark*, which

<footnote n="3">³ https://github.com/anselmeamekoe/DelayedLabelStream</footnote>
<table><caption>Table 3: Predictive performance on the *generated benchmark* in **no delay setting**. Mean and standard deviation AUCROC (%), reported for a data stream split in 90 batches. Italic highlights the best performance when comparing inherently interpretable (II) models, and bold is used for the overall best-performing model. We also report the average (resp. the average normalized by the best AUCROC of the given dataset) across the 5 stream datasets denoted as Avg (resp. N_Avg), as well as the average rank (Avg_Rank). Type: **II**=Inherently interpretable, **FC**=Full complexity model</caption><thead><tr><th>Model</th><th>Type</th><th>AGR<sub>a</sub></th><th>AGR<sub>g</sub></th><th>HYPER<sub>f</sub></th><th>SEA<sub>a</sub></th><th>SEA<sub>g</sub></th><th>Avg</th><th>N_Avg</th><th>Avg_Rank</th></tr></thead><tbody><tr><td>LB.ht</td><td>FC</td><td>94.54 ±4.43</td><td>90.96 ±6.74</td><td>93.41 ±0.39</td><td>88.90 ±0.89</td><td>88.87 ±0.83</td><td>91.34</td><td>98.01</td><td>6.20</td></tr><tr><td>LB.LR</td><td>FC</td><td>56.40 ±5.94</td><td>55.92 ±5.59</td><td>94.80 ±0.23</td><td>88.83 ±0.88</td><td>88.78 ±0.84</td><td>76.94</td><td>83.14</td><td>9.00</td></tr><tr><td>ARF</td><td>FC</td><td>94.82 ±4.12</td><td>92.12 ±6.12</td><td>90.36 ±1.29</td><td>88.92 ±0.89</td><td>88.89 ±0.84</td><td>91.02</td><td>97.67</td><td>6.00</td></tr><tr><td>r.XGBoost</td><td>FC</td><td><b>97.36</b> ±8.16</td><td><b>96.25</b> ±4.80</td><td>92.89 ±0.98</td><td>88.80 ±0.86</td><td>88.75 ±0.87</td><td><b>92.81</b></td><td><b>99.52</b></td><td>5.40</td></tr><tr><td>B.XGBoost</td><td>FC</td><td>96.59 ±9.13</td><td>95.91 ±5.72</td><td>92.27 ±1.40</td><td>88.89 ±0.88</td><td>88.84 ±0.86</td><td>92.50</td><td>99.21</td><td>5.80</td></tr><tr><td>HT</td><td>II</td><td>86.50 ±12.52</td><td>79.77 ±13.69</td><td>65.80 ±15.17</td><td>88.28 ±0.50</td><td>88.24 ±0.49</td><td>81.72</td><td>87.92</td><td>11.60</td></tr><tr><td>HAT</td><td>II</td><td>93.86 ±4.98</td><td>89.72 ±7.23</td><td>89.98 ±1.52</td><td>88.38 ±0.90</td><td>88.21 ±0.75</td><td>90.03</td><td>96.62</td><td>10.80</td></tr><tr><td>LR</td><td>II</td><td>58.34 ±7.49</td><td>58.12 ±6.98</td><td><b>94.85</b> ±0.23</td><td>88.90 ±0.88</td><td>88.86 ±0.84</td><td>77.82</td><td>84.05</td><td>7.00</td></tr><tr><td>DT</td><td>II</td><td>96.12 ±7.70</td><td>95.09 ±4.90</td><td>86.52 ±1.86</td><td>87.87 ±1.29</td><td>87.74 ±0.89</td><td>90.67</td><td>97.25</td><td>9.80</td></tr><tr><td>r.EBM</td><td>II</td><td>95.84 ±8.29</td><td>95.21 ±4.86</td><td>93.43 ±0.97</td><td>88.91 ±0.90</td><td>88.87 ±0.86</td><td>92.45</td><td>99.16</td><td>4.40</td></tr><tr><td>B.EBM</td><td>II</td><td>95.49 ±9.49</td><td>94.84 ±5.69</td><td>92.41 ±1.90</td><td><b>88.94</b> ±0.90</td><td><b>88.90</b> ±0.86</td><td>92.12</td><td>98.81</td><td>4.80</td></tr><tr><td>rTabSRA</td><td>II</td><td>97.10 ±8.23</td><td>95.73 ±5.23</td><td>93.07 ±1.20</td><td>88.89 ±0.91</td><td>88.88 ±0.86</td><td>92.73</td><td>99.45</td><td><b>4.20</b></td></tr><tr><td>uTabSRA</td><td>II</td><td><b>97.16</b> ±7.79</td><td>95.75 ±5.22</td><td>92.78 ±1.20</td><td>88.74 ±1.30</td><td>88.83 ±0.85</td><td>92.65</td><td>99.36</td><td>6.00</td></tr></tbody></table>

encompasses varying (unequal) types of bias/difficulty, to select stream learning approaches may be misleading for real-world problems.

Regarding the adaptation over time, the Hoeffding Tree (HT) appears to be more affected by changes (Fig. 1a and 1b) arguably due to its lack of explicit drift handling compared to the HAT (Hoeffding Adaptive Tree). We recall that the maximum depth of the tree is capped to 6 (for DT, HT, HAT) during the hyperparameters optimization; otherwise, it would be challenging to consider them as inherently interpretable. Overall, we observe that batch-based solutions (whether inherently interpretable or black box) can adapt to changes through incremental retraining, even without explicit drift management.

An important observation from this experimentation is the hyperparameter sensitivity of tree-based instance incremental algorithms. As long as the concept<fnref n="4" /> on which the parameters were tuned is still valid, instance incremental models with good inductive biases (tree-based ones such as ARF, LB.ht) manage to match the performance of batch incremental counterparts (highlighted by the Fig. 1a for AGR<sub>g</sub> dataset with Batch Id ∈ [0, 15]). However after the change or drift (Batch Id ∈ [15, 40]) these models struggle to converge, unlike batch learners. Yet, once the initial concept reoccurs (Batch Id ∈ [40, 65]) the models (instance incremental as well) converge to perfect AUCROC again.

Finally, we observe that for this *Generated benchmark*, employing stacking/updating

<footnote n="4">⁴ The function 5 of the Agrawal generator.</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1: AUCROC over time. The x-axis is the identifier (ID) of the batch/chunk, which is up to 90. The delay factor 0 corresponds to the no-delay setting, and 7 corresponds to the Poisson stochastic delay of average of 70,000 instances. The red vertical line indicates the change point for the SEA dataset Fig. 1a and 1c.</figcaption></figure>

of models learned over different periods (B_XGBoost, B_EBM, u_TabSRA) does not yield improved predictive performance; instead, it increases computational resources or running time (Table 4). We argue that this is due to the simplicity of the datasets, where typically small learning datasets (roughly 10,000 instances) suffice to learn the underlying concept. However, for more complex tasks, we anticipate that stacking may prove beneficial, as demonstrated in Section 4.3 for the *Fraud* dataset.

**How does the label delay impact incremental algorithms?** After comparing algorithms in no delay settings, it is crucial to consider the delay setting, which involves answering the following questions: (1) How does the label delay impact the performance of the stream learning system? (2) is instance incremental learning the best option regarding the predictive performance in such delayed settings? From Fig. 1, we can notice that the labeling delay impacts the recovery or adaptation of both instance and batch learners to changes, consequently affecting overall performance (Fig. 2), consistent with findings [17,18]. However, this impact tends to be limited for low-severity drift/changes. For the SEA<sub>α</sub> (Fig. 2c) dataset for instance (where the changes in the simulated decision functions are not significant), the average drop in AUCROC in delayed
<figure><img src="image_3.png" /><figcaption>Fig. 2: Impact of the label delay on the predictive performance. The x-axis indicates the delay factor ranging from 0 (no delay) to 7 (the Poisson stochastic delay of average 70,000 instances). The best model for each category is reported to make the figures more clearer (LB_LR, HT, u_TabSRA, B_EBM, B_EBM) as discarded. $I_f$ = incremental fast drift.</figcaption></figure>

context seems very low, even negative for some models (for both gradual and abrupt drift) in contrast to others generated datasets (AGR<sub>a</sub>, AGR<sub>g</sub>, HYPER<sub>f</sub>).

Besides the impact of the labeling delay on all stream learning models, we can notice (Fig. 2) that the best-performing batch learner usually provides similar or superior results compared to instance counterpart regardless of the length of the delay.

**Computational efficiency** In this part, runtimes of the compared models during the stream are analyzed, encompassing retraining and inference, as a measure of computational efficiency. We recognize that relying solely on runtime may not suffice, as it can vary with implementation. For tree-based models, parameter or node count is sometimes considered [26], yet comparing computational efficiency across different model classes (e.g., neural networks versus tree-based models) using this metric is challenging.

As shown in Table 4, state-of-the-art instance incremental learners like ARF are over twice as costly as batch counterparts such as r_XGBoost. We argue that this is due to the explicit drift monitoring process employed in instance-based learning models. Furthermore, employing batch model stacking (B_XGBoost, B_EBM) tends to notably increase inference time and, consequently, the runtime of the stream.
<table><caption>Table 4: Evaluation time (s). The average (Avg.) and standard deviation (Std.) time (the smaller, the better) is reported from 9 delayed configurations (factor).</caption><thead><tr><th>Model</th><th>Avg. runtime</th><th>Std. runtime</th></tr></thead><tbody><tr><td>LB_HT</td><td>5837.58</td><td>4428.63</td></tr><tr><td>LB_LR</td><td>7279.24</td><td>4273.45</td></tr><tr><td>ARF</td><td>3702.08</td><td>2622.41</td></tr><tr><td>r_XGBoost</td><td>1688.97</td><td>210.53</td></tr><tr><td>B_XGBoost</td><td>5101.62</td><td>1186.93</td></tr><tr><td>HT</td><td>60.07</td><td>11.90</td></tr><tr><td>HAT</td><td>248.20</td><td>63.00</td></tr><tr><td>LR</td><td>83.93</td><td>29.07</td></tr><tr><td>DT</td><td>976.94</td><td>47.11</td></tr><tr><td>r_EBM</td><td>1456.71</td><td>403.83</td></tr><tr><td>B_EBM</td><td>4167.64</td><td>3244.12</td></tr><tr><td>r_TabSRA</td><td>2117.04</td><td>298.78</td></tr><tr><td>u_TabSRA</td><td>2685.77</td><td>166.45</td></tr></tbody></table>

Concerning inherently interpretable solutions, batch incremental options like EBM and TabSRA exhibit longer runtimes compared to instance incremental models (HT, HAT, LR). It is worth noting that the significant runtime difference between Decision Tree (DT) and shallow instance incremental learners (HT, HAT, LR) is arguably attributed to the data filtering process required for selecting new batches for model re-training/updating (Ligne 15 Algorithm 1).

## 4.3 Results on the Fraud dataset

In this section, we compare the models using the *Fraud dataset* (described in Section 4.1). Unlike the *Generated benchmark*, this dataset displays a significant class imbalance (Table 2), a common situation in real-world applications such as fraud detection. For this dataset, we also consider both instance and batch incremental models presented in Section 4.1 except LB_LR which tends to be particularly ineffective at modeling nonlinear functions and has high runtime demands (Table 4). Furthermore, we incorporate two learning strategies for batch learners:

- **Static**: All batch learners are optimized using the initial offline collected datasets of three months but are not retrained/updated during the stream. This approach saves from retraining and validation costs, but may lead to significant performance drop after severe changes or drifts. Models trained with the static strategy will be prefixed with `static_`.

- **Propagate**: The instance from past fixed number of chunks (months in our case) are propagated and combined with the currently available one for retraining/updating the batch model. This approach may be very beneficial for tasks where certain groups or categories are very underrepresented or target event is very rare [11]. However, this approach may add an additional cost for storing past batches and may impair the adaptation to the newest concept, especially when the number of
past-considered chunks is important. In this study, we explore the trade-off by in-
vestigating the combination of the two previous months (chunks) with the current
one. This situation is equivalent to the case where the learning chunk is three times
the evaluation batch (Algorithm 1). Models trained with this strategy will be pre-
fixed with **propagate_**.

Similarly to the *Generated benchmark*, we use an initial optimization step for the *Fraud* dataset, as highlighted in Section 4.1. We optimize the batch incremental algorithm using 10 different parameter configurations and the instance incremental algorithms using 15 configurations<fnref n="5" /> including resampling methods [13,1]. We consider undersampling for ARF and LB-HT, which already have a long runtime due to online bagging, and investigate undersampling for HT and HAT. For LR, we examine cost sensitivity by modifying the misclassification cost of positive examples in the hyperparameter configuration<fnref n="6" />. The performance of instance incremental models was particularly poor without explicitly handling class imbalance.

As shown in Table 5, the best batch solution, XGBoost, consistently outperforms
(over the 8 months) the best instance incremental solution, namely Adaptive Random
Forest (ARF). Similarly, the Explainable Boosting Machine (EBM) provides the best
average AUCPR among the inherently interpretable solutions, surpassing all the in-
stance incremental solutions. Besides the overall superior predictive performance of
batch incremental solutions in this problem, the **propagate** strategy, which consists
of using the tree last months (chunks) of labeled data, tends to provide the best re-
sults. For XGBoost, the average AUCPR gain is up to 22% compared to the retrain
strategy (r_XGBoost) and 12% compared to the stacking strategy. We argue that this
because the target event (fraudulent examples) is rare. Consequently, using only one
chunk (one month) of observation should not be enough to retrain a consistent learn-
ing model, and finding the best weighting combination for the stacking approach is
not an easy task. This finding is following [31] where the author demonstrates that
varying the number of observations in the chunks (from 3,000 to 12,000 observations)
increases linearly the benefit (the number of frauds detected) by the learning model.
Particularly for the fraud dataset investigated in our study, the number of fraudulent
examples seen during the training or the updating of the model is very crucial for the
predictive performance, highlighted by the fact that a static approach with 3 months of
training data (static_XGBoost) outperforms in average the periodical retraining with 1
month (chunk) of observations (r_XGBoost). It also indicates that some concepts may
reappear over time (e.g., static_XGBoost outperforms propagate_XGBoost for March).
Overall, the results on this dataset suggest that storing some past observations may be
advantageous for the streaming learning model.

We also note an intriguing result (Table 5) for inherently interpretable instance in-
cremental models: Hoeffding Tree (HT) outperforms the HT equipped with the ADWIN
drift detector (HAT) in contrast to the results on the *Generated benchmark* (Table 3).

<footnote n="5">⁵ The experiment on this dataset is conducted on a private business Google Cloud Platform provided by our data provider, where running very long experiments is infeasible. Therefore, we used a maximum of 15 iterations (instead of 30 for the generated datasets) as a trade-off between tunability and predictive accuracy</footnote>

<footnote n="6">⁶ https://riverml.xyz/0.9.0/examples/imbalanced-learning/</footnote>
<table><caption>Table 5: Predictive performance on the fraud dataset. The evaluation is done in batches of one month, starting from January 2022 to August 2022, while predictions are made in real time (one instance at a time). AUCPR is used as a metric, and we also report the Average (Avg) and Standard deviation (Std) over the 8 months. Italic highlights the best performance when comparing inherently interpretable models (HT, HAT, LR, DT, EBM, TabSRA) models, and bold is used for the overall best-performing model.</caption><thead><tr><th>Model</th><th>Jan</th><th>Fev</th><th>Mar</th><th>Apr</th><th>May</th><th>Jun</th><th>Jul</th><th>Aug</th><th>Avg</th><th>Std</th></tr></thead><tbody><tr><td>LB_HT</td><td>5.26</td><td>7.32</td><td>10.89</td><td>6.76</td><td>10.18</td><td>8.27</td><td>10.65</td><td>19.27</td><td>9.83</td><td>4.03</td></tr><tr><td>ARF</td><td>9.38</td><td>14.08</td><td>22.73</td><td>13.12</td><td>12.50</td><td>12.27</td><td>9.90</td><td>19.87</td><td>14.23</td><td>4.39</td></tr><tr><td>static_XGBoost</td><td><strong>15.68</strong></td><td>23.12</td><td><strong>33.76</strong></td><td>14.69</td><td>19.48</td><td>17.31</td><td>12.75</td><td>22.55</td><td>19.92</td><td>6.25</td></tr><tr><td>r_XGBoost</td><td><strong>15.68</strong></td><td>23.00</td><td>24.96</td><td>17.53</td><td>18.23</td><td>20.54</td><td>14.65</td><td>18.60</td><td>19.15</td><td>3.29</td></tr><tr><td>B_XGBoost</td><td><strong>15.68</strong></td><td>23.00</td><td>24.49</td><td><strong>18.44</strong></td><td>21.02</td><td>23.05</td><td>16.36</td><td>23.78</td><td>20.73</td><td>3.24</td></tr><tr><td>propagate_XGBoost</td><td><strong>15.68</strong></td><td><strong>27.69</strong></td><td>32.04</td><td>18.24</td><td><strong>22.85</strong></td><td><strong>23.80</strong></td><td><strong>18.05</strong></td><td><strong>28.92</strong></td><td><strong>23.41</strong></td><td>5.46</td></tr><tr><td>HT</td><td>4.67</td><td>10.03</td><td>15.27</td><td>7.20</td><td>6.43</td><td>4.10</td><td>6.11</td><td>16.63</td><td>8.80</td><td>4.46</td></tr><tr><td>HAT</td><td>1.49</td><td>0.55</td><td>1.77</td><td>1.36</td><td>0.87</td><td>1.10</td><td>2.48</td><td>5.99</td><td>1.95</td><td>1.62</td></tr><tr><td>LR</td><td>6.91</td><td>9.11</td><td>12.25</td><td>6.74</td><td>7.32</td><td>5.77</td><td>7.52</td><td>15.79</td><td>8.93</td><td>3.19</td></tr><tr><td>static_DT</td><td>7.39</td><td>10.88</td><td>12.01</td><td>4.98</td><td>4.71</td><td>4.31</td><td>4.13</td><td>6.23</td><td>6.83</td><td>2.86</td></tr><tr><td>r_DT</td><td>7.39</td><td>11.31</td><td>8.07</td><td>7.05</td><td>7.52</td><td>8.13</td><td>7.62</td><td>6.59</td><td>7.96</td><td><strong>1.35</strong></td></tr><tr><td>propagate_DT</td><td>7.39</td><td>13.82</td><td>14.68</td><td>10.72</td><td>10.74</td><td>9.15</td><td>8.63</td><td>19.83</td><td>11.87</td><td>3.80</td></tr><tr><td>staticiry</td><td>12.60</td><td>17.91</td><td>23.69</td><td>11.06</td><td>13.62</td><td>8.91</td><td>9.81</td><td>20.94</td><td>14.82</td><td>5.08</td></tr><tr><td>static_ry</td><td>12.60</td><td>14.17</td><td>22.68</td><td>15.03</td><td>15.21</td><td>13.95</td><td>11.47</td><td>20.19</td><td>15.66</td><td>3.58</td></tr><tr><td>static_ry</td><td>12.60</td><td>19.10</td><td>26.38</td><td>16.65</td><td>18.73</td><td>19.01</td><td>13.99</td><td>19.56</td><td>18.25</td><td>3.91</td></tr><tr><td>static_ry</td><td>12.60</td><td>19.24</td><td>26.74</td><td>15.52</td><td>18.88</td><td>20.56</td><td>14.41</td><td>22.86</td><td>18.85</td><td>4.35</td></tr><tr><td>static_ry</td><td>13.36</td><td>20.80</td><td>25.94</td><td>12.26</td><td>14.85</td><td>16.73</td><td>9.27</td><td>14.24</td><td>15.93</td><td>4.91</td></tr><tr><td>static_ry</td><td>13.36</td><td>22.03</td><td>22.92</td><td>15.33</td><td>16.85</td><td>18.06</td><td>12.72</td><td>17.49</td><td>17.35</td><td>3.45</td></tr><tr><td>static_ry</td><td>13.36</td><td>20.80</td><td>25.94</td><td>17.21</td><td>17.55</td><td>21.55</td><td>15.06</td><td>22.06</td><td>19.04</td><td>3.89</td></tr><tr><td>static_ry</td><td>13.36</td><td>22.88</td><td>24.92</td><td>16.72</td><td>17.44</td><td>21.59</td><td>15.14</td><td>26.20</td><td>19.78</td><td>4.45</td></tr></tbody></table>

We argue that for such a dataset where the target event of interest is very rare (highly imbalanced dataset), explicit drift handling may be biased toward negative examples. This because many statistical drift detectors rely on accuracy metrics or error rates. Consequently, the HAT model may discard some parts (or branches) of the tree that are specialized in identifying concepts (fraudulent schemes) that may reoccur over time.

# 5 Conclusion and Discussions

We proposed a thorough comparison between batch and instance incremental models by examining the usefulness of the latter in situations where labels become available with varying delays, and interpretability is crucial. Our findings on commonly used synthetic data streams demonstrate that the best batch incremental approach (fully complex or inherently interpretable), combined with an effective updating strategy, provides similar or better performance than instance incremental counterparts. Additionally, the superior performance of batch learners is illustrated using a real-world fraud detection problem where the target class is scarce. We explain why, for such a dataset, it is necessary to store some observations while waiting for the labels due to the delay. Furthermore, we
demonstrate how storing some old but limited chunks of observations can benefit batch
incremental models, leading to up to a 22% relative improvement in AUCPR compared
to using only one chunk. Although we illustrate the usefulness of batch incremental so-
lutions, we do not cover the choice of the optimal batch (chunk) size, as it may depend
on several factors (e.g., the frequency of the target event, bias handling for underrepre-
sented groups) that we leave for future study. Nevertheless, we believe that choosing the
learning batch based on evaluation date (daily, monthly) can be a good starting point,
as illustrated with the fraud dataset.

We hope our findings will encourage and guide researchers in developing new learn-
ing strategies for evolving data streams, bridging the gap between academic research
and real-world business use cases.

References

1. Aguiar, G., Krawczyk, B., Cano, A.: A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework (2022). arXiv preprint arXiv:2204.03719 (2022)

2. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 2623–2631 (2019)

3. Amekoe, K.M., Dilmi, M.D., Azzag, H., Dagdia, Z.C., Lebbah, M., Jaffre, G.: Tabsra: An attention based self-explainable model for tabular learning. In: The 31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) (2023)

4. Bifet, A., de Francisci Morales, G., Read, J., Holmes, G., Pfahringer, B.: Efficient online evaluation of big data stream classifiers. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. pp. 59–68 (2015)

5. Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM international conference on data mining. pp. 443–448. SIAM (2007)

6. Bifet, A., Holmes, G., Pfahringer, B.: Leveraging bagging for evolving data streams. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part I 21. pp. 135–150. Springer (2010)

7. Boyd, K., Eng, K.H., Page, C.D.: Area under the precision-recall curve: point estimates and confidence intervals. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13. pp. 451–466. Springer (2013)

8. Bradley, A.P.: The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition 30(7), 1145–1159 (1997)

9. Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. pp. 785–794 (2016)

10. Covert, I., Lundberg, S.M., Lee, S.I.: Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems 33, 17212–17223 (2020)

11. Dal Pozzolo, A., Caelen, O., Le Borgne, Y.A., Waterschoot, S., Bontempi, G.: Learned lessons in credit card fraud detection from a practitioner perspective. Expert systems with applications 41(10), 4915–4928 (2014)
12. Domingos, P., Hulten, G.: Mining high-speed data streams. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 71–80 (2000)

13. Ferreira, L.E.B., Gomes, H.M., Bifet, A., Oliveira, L.S.: Adaptive random forests with re-sampling for imbalanced data streams. In: 2019 International Joint Conference on Neural Networks (IJCNN). pp. 1–6. IEEE (2019)

14. Gama, J., Rocha, R., Medas, P.: Accurate decision trees for mining high-speed data streams. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 523–528 (2003)

15. Gama, J., Sebastião, R., Rodrigues, P.P.: Issues in evaluation of stream learning algorithms. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 329–338 (2009)

16. Gama, J., Sebastião, R., Rodrigues, P.P.: On evaluating stream learning algorithms. Machine learning **90**, 317–346 (2013)

17. Gomes, H.M., Bifet, A., Read, J., Barddal, J.P., Enembreck, F., Pfharinger, B., Holmes, G., Abdessalem, T.: Adaptive random forests for evolving data stream classification. Machine Learning **106**, 1469–1495 (2017)

18. Gomes, H.M., Grzenda, M., Mello, R., Read, J., Le Nguyen, M.H., Bifet, A.: A survey on semi-supervised learning for delayed partially labelled data streams. ACM Computing Surveys **55**(4), 1–42 (2022)

19. Grzenda, M., Gomes, H.M., Bifet, A.: Delayed labelling evaluation for data streams. Data Mining and Knowledge Discovery **34**(5), 1237–1266 (2020)

20. Gunasekara, N., Pfahringer, B., Gomes, H., Bifet, A.: Gradient boosted trees for evolving data streams. Machine Learning pp. 1–28 (2024)

21. Haug, J., Broelemann, K., Kasneci, G.: Dynamic model tree for interpretable data stream learning. In: 2022 IEEE 38th International Conference on Data Engineering (ICDE). pp. 2562–2574. IEEE (2022)

22. Haug, J., Tramountani, E., Kasneci, G.: Standardized evaluation of machine learning methods for evolving data streams. arXiv preprint arXiv:2204.13625 (2022)

23. Kumar, I.E., Venkatasubramanian, S., Scheidegger, C., Friedler, S.: Problems with shapley-value-based explanations as feature importance measures. In: International Conference on Machine Learning. pp. 5491–5500. PMLR (2020)

24. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems **30** (2017)

25. Montiel, J., Halford, M., Mastelini, S.M., Bolmier, G., Sourty, R., Vaysse, R., Zouitine, A., Gomes, H.M., Read, J., Abdessalem, T., et al.: River: machine learning for streaming data in python (2021)

26. Montiel, J., Mitchell, R., Frank, E., Pfahringer, B., Abdessalem, T., Bifet, A.: Adaptive xg-boost for evolving data streams. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2020)

27. Muschalik, M., Fumagalli, F., Hammer, B., Hüllermeier, E.: isage: An incremental version of sage for online explanation on data streams. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 428–445. Springer (2023)

28. Nori, H., Jenkins, S., Koch, P., Caruana, R.: Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909.09223 (2019)

29. Plasse, J., Adams, N.: Handling delayed labels in temporally evolving data streams. In: 2016 IEEE International Conference on Big Data (Big Data). pp. 2416–2424. IEEE (2016)

30. Read, J., Bifet, A., Pfahringer, B., Holmes, G.: Batch-incremental versus instance-incremental learning in dynamic and evolving data. In: Advances in Intelligent Data Analysis XI: 11th International Symposium, IDA 2012, Helsinki, Finland, October 25-27, 2012. Proceedings 11. pp. 313–323. Springer (2012)
31. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 226–235 (2003)