# OSMAC: A Dynamic SMAC for Data Streams

Émile Royer, Maroua Bahri, Nikolaos Georgantas

► To cite this version:

Émile Royer, Maroua Bahri, Nikolaos Georgantas. OSMAC: A Dynamic SMAC for Data Streams. 37th International Conference on Tools with Artificial Intelligence (ICTAI 2025), Nov 2025, Athens, Greece. pp.73-80, 10.1109/ICTAI66417.2025.00018. http://hal-05383821

HAL Id: hal-05383821

https://inria.hal.science/hal-05383821v1

Submitted on 26 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# OSMAC: a Dynamic SMAC for Data Streams

Émile Royer

Inria

Paris, France

emile.royer@inria.fr

Maroua Bahri

Sorbonne Université

CNRS, LIP6

Paris, France

maroua.bahri@lip6.fr

Nikolaos Georgantas

Inria

Paris, France

nikolaos.georgantas@inria.fr

**Abstract**—Automated machine learning (autoML) methods often require multiple passes over data and are computationally intensive, rendering them unsuitable for streaming scenarios where data is continuously generated and distributions evolve over time. The few existing autoML solutions for stream learning mainly rely on random search or genetic algorithms, which struggle to maintain high performance in dynamic environments. By contrast, leading methods in batch learning such as the Sequential Model-based Algorithm Configuration (SMAC) leverage model-based approaches, suggesting opportunities for improvement in stream settings. To address these challenges and meet the requirements of stream scenarios, we introduce *OnlineSMAC*, a model-based optimizer for data streams. OnlineSMAC combines Bayesian optimization with an extension of the SMAC optimizer to dynamically select optimal processing pipelines and hyperparameters. Our results show that this approach is highly competitive, achieving performance on par with state-of-the-art stream autoML methods. This highlights the promising potential of using Bayesian optimization for data streams.

## I. INTRODUCTION

Machine learning has become a crucial tool for extracting valuable insights from data and modeling its underlying patterns. However, the performance of machine learning algorithms is highly dependent on the choice of hyperparameters and preprocessing pipelines. Manually tuning these components is not only time-consuming but also requires significant expertise, which has driven the development of automated Machine Learning (autoML) techniques [1].

AutoML [1]–[3] has demonstrated success in automating the selection of pipelines and hyperparameters for static datasets. Nevertheless, the dynamic nature of data streams presents unique challenges that are not adequately addressed by existing autoML methods. Current State-of-the-Art (SotA) methods typically require multiple passes over the data to converge on an optimal solution, resulting in high computational costs. This renders them unsuited for data stream settings, where data arrive continuously and the underlying distribution may evolve over time. Since the data distribution may shift, this leads to what is known as concept drift [4] and thus highlighting the need for adaptive optimization techniques that can adjust to these evolving changes. Additionally, configurations that were previously rejected may become relevant as data evolve, requiring continual reassessment of potential configurations.

A further constraint with data streams is resource usage [5]. Because of their real-time objective and deployment environment, models must process new data as fast as possible and

have minimal memory requirements. This implies a preference for efficient models, with heavier solutions like neural networks typically excluded due to their high training costs.

The existing autoML solutions for data streams rely on random search and genetic algorithms, e.g., ASML [6], Auto-Class [7], EvoAutoML [8]. While these methods offer some utility, they often fall short in terms of performance, particularly in dynamic environments. In contrast, model-based approaches have demonstrated good accuracy in offline settings, making them a highly promising avenue for advancing stream learning. To address this gap, we introduce *OnlineSMAC* (OSMAC), a novel optimizer that extends the Sequential Model-based Algorithm Configuration (SMAC) method [9]. OSMAC is designed to adaptively find the best processing pipeline and its hyperparameters in the context of streaming data.

Our main contributions include:

* We provide a formalization of the autoML problem for data streams.

* We present OnlineSMAC, that employs Bayesian optimization and responds dynamically to continuous data, making it well-suited for a wide range of online machine learning tasks.

* We provide an experimental study to evaluate OSMAC on a diverse set of data against automated methods and commonly used SotA ensemble methods.

* To enable reproducibility, we provide our source code.<fnref n="1" />

The structure of the paper is as follows. We define formally the problem and review the SotA in Section II. In Section III, we present our proposed method. Section IV outlines the results of our experiments, and we conclude with a discussion of the findings and potential future directions in Section V.

## II. BACKGROUND

### A. Problem formulation

Consider a data stream $S$ defined as a potentially infinite sequence of instances – or data points – $S = \{X_1, X_2, \dots, X_i, \dots\}$, where each instance $X_i \in \mathbb{R}^d$ arrives sequentially at time $i$, sampled IID from a probabilistic distribution [10].

AutoML automates a data pipeline composed of different components, such as data preprocessing, feature engineering, and model selection, by applying a hyperparameter optimization (HPO) technique to find the best-performing combination

<footnote n="1">¹Available at https://hal.science/hal-05290014 (and Software Heritage).</footnote>
and configuration of models. For this to happen, autoML addresses the Combined Algorithm Selection and Hyperparameter (CASH) optimization problem [8], [11].

**Definition 1** (The online CASH problem). Let $\mathcal{A}$ be a set of learning algorithms $A^j$, each with a set $\Lambda$ of hyperparameters $\lambda$. Given the stream $S$, $\mathcal{L}(A_{\lambda}^j(S_t), X_i)$ is the loss achieved by algorithm $A^j$ with parameters $\lambda$ trained on substream $S_t$ and evaluated on each instance $X_i$.

The goal of the online CASH problem is to find the algorithm $A^*$ and its hyperparameters that minimize the loss on a given stream:

$$A_{\lambda}^{*} \in \underset{A \in \mathcal{A}, \lambda \in \Lambda}{\operatorname{argmin}} \sum_{i} \mathcal{L}(A_{\lambda}(S_{t}), X_{i}).$$

## B. Related work

Multiple Sota batch solutions exist to tackle the CASH problem for static datasets. For instance, Auto-Weka [11] and Auto-sklearn [12] both use the SMAC optimizer to find the best model configuration. H2O [13], GAMA [14], and TPOT [15] employ a grid search approach. However, these solutions do not handle the unbounded and incremental nature of data streams, because they need multiple access to data which infringes the stream framework one-pass requirement.

One of the first automated solutions for streaming autoML is Self-Parameter Tuning (SPT) [16]. It uses the Nelder-Mead technique to optimize the hyperparameters. It is however limited to HPO on one model and can only process numerical parameters. EvoAutoML [8] mutates a random hyperparameter of the best pipeline with a uniform probability. It then considers all the different pipelines as an ensemble and trains them following the Leveraging Bagging [17] method. AutoClass [7] uses truncated normal distributions to derive a new configuration from the best-performing one; the new configuration is only added to the ensemble if it is predicted to be above a given threshold. These two methods rely on a genetic algorithm where they change only one parameter of a single model of the ensemble at a time, limiting their convergence speed and overall efficiency. Champions-Challengers (ChaCha) [18] uses offline search algorithms like beam search to build new challengers from a champion. Online AutoML (OAML) [19] is a hybrid method that works with any batch optimizer and is specifically focused on adapting to concept drifts. Neither OAML nor ChaCha introduce a new online optimizer, instead, they build upon existing optimizers, developing methods that leverage these established tools. More recently, Automated Streaming Machine Learning (ASML) [6] has been proposed as a method that replaces all the pipelines when a new window starts and retains only the best-performing one unchanged.

Overall, automated stream solutions have so far focused on variations of genetic search [6]–[8] or used offline search algorithms to build pipelines [18], [19]. To the best of our

knowledge, no existing method has yet introduced a model-based search strategy specifically designed to address the CASH problem for data streams. This represents a significant gap in the current research, as model-based approaches have good performance in offline settings and could offer substantial improvements in the online learning context.

## III. MODEL-BASED OPTIMIZATION ON STREAMING DATA

This paper presents OnlineSMAC, a novel automated method, that leverages Bayesian optimization for data stream learning. In this section, we provide an overview of batch SMAC and introduce our proposed method and how it integrates into the broader autoML pipeline.

### A. SMAC

SMAC [9], [20] is a well-known HPO technique that employs Bayesian optimization [21]. Many Bayesian optimizers use Gaussian processes as their surrogate model, however, they are poorly suited for handling categorical data. To overcome this, SMAC utilizes a regression random forest as its surrogate model, which is more flexible in managing both categorical and continuous hyperparameters. Unlike Gaussian processes, random forests do not provide a measure of uncertainty of their predictions. To approximate this uncertainty, SMAC uses the variance of the individual trees' predictions. The authors used their own derivation of the logarithmic Expected Improvement (logEI) as the acquisition function. It allows the computation of the regular Expected Improvement (EI) on the log-transformed data produced by SMAC.

Due to the discontinuous nature of the surface created by a random forest, SMAC combines neighborhood search with random search to optimize the acquisition function. Each cycle, it generates 20000 configurations to test by mixing chosen configurations with random ones. To find a new incumbent, SMAC compares each of the configurations against the current incumbent in a process called *intensification*, where the evaluated model is progressively trained on unseen subsets of the dataset. Between each training step, the performance of the model is compared to that of the incumbent; if the new model fares worse, it is abandoned, and the comparison restarts with another configuration; otherwise the training continues. If a model succeeds in matching or surpassing the incumbent at all steps of the training process and gets trained on the full dataset, it replaces the incumbent and stops the intensification.

### B. Online SMAC

While SMAC is a powerful optimizer, it is not adapted for evolving data streams because its intensification function requires multiple passes over the data to iteratively find the best configuration. We thus introduce OnlineSMAC, extending the SMAC method, specifically tailored for online hyperparameter optimization. By leveraging Bayesian optimization, OnlineSMAC dynamically adapts to incoming data, enabling the efficient discovery of optimal configurations in dynamic streaming environments. Algorithm 1 outlines the training phase of the algorithm.

<footnote>$^2S_t$ represents the set of instances observed from the stream so far, which are first used for prediction and then for evaluation, following the test-then-train evaluation protocol [10].</footnote>
The first distinction of our approach compared to SMAC lies in the choice of the surrogate model: while we retain the foundational use of random forests, we adopt a stream variant known as the Adaptive Random Forest Regressor (ARF<sub>R</sub>) [22]. Furthermore, we configure this model to eagerly split nodes, enabling faster adaptation to new data and efficiently incorporating relevant data points in subsequent iterations. In SMAC, the variance of individual trees is used to estimate the surrogate model's variance. In our case, we observed that this approach produced sub-par results when applied to data streams. To address this, we adopted the Jackknife method [23] to estimate the deviation, at the cost of local precision and a deviation from the spirit of the EI. Indeed, unlike the individual variance method, the Jackknife method provides a single variance value for all the predictions, updated only during learning phases. This compromise ultimately delivers robust performance. Our acquisition function remains the logEI used in SMAC, employing the revised and more precise formula from Amet et al. [24].

We initialize our system with random configurations, since OnlineSMAC performs algorithm selection and lacks a predefined default configuration to start from (see Algorithm 1, lines 2–3).

HPO models generate candidate configurations, aiming to find one that is better than the incumbent. To this end, in its intensification phase, SMAC iterates over thousands of candidates, stopping when it finds a suitable configuration. However, such a number of sequential comparisons would be too expensive to perform when working with data streams. Our method is instead constructed around non-overlapping sliding windows of length *p*.

For each window, candidates are trained in parallel (lines 9–10) on the incoming data instances, evaluated in test-then-train mode. At the end of period *p* (line 12), all candidate models are compared. However, a direct comparison between the candidates and the incumbent is not possible, as they were trained on different data. A fair comparison is only possible when both models have been trained on the same data, ensuring that performance differences reflect model quality rather than differences in training inputs. The candidate with the best performance over the training window becomes the new incumbent (line 14), see Figure 1. If the new incumbent corresponds to the same configuration as the previous one, we keep the previous incumbent's model, because it has been trained on more data. This indicates that the incumbent was not outperformed. Beyond serving as a reference for configuration selection, the incumbent is also the model that used to produce the final predictions in OnlineSMAC.

Before the next window starts, new candidates are generated using the function SELECTNEW. We follow the same generation process as SMAC (Algorithm 1, line 18), but only keep the top *B* (for budget) candidates (line 19), i.e. we select *B*/2 likely good candidates – those with high logEI – and *B*/2 random ones, to maintain exploration. We add the new incumbent's configuration in the candidate set (line 19), as comparisons between models and the incumbent are only meaningful when

**Algorithm 1 OnlineSMAC's training mechanism**

**Symbols:** Search space Σ, Budget *B*, Period or window length *p*, incumbent *i*, data stream *S*

**Require:** $B > 0, p > 0$

1: **procedure** TRAIN(Σ, *B*, *p*)

2: &nbsp;&nbsp;&nbsp;&nbsp;ms ← RANDOMCANDIDATES(Σ, *B*) ▸ First, initialise OSMAC

3: &nbsp;&nbsp;&nbsp;&nbsp;i ← RANDOMCONFIGURATION(Σ) ▸ Initialize the incumbent

4: &nbsp;&nbsp;&nbsp;&nbsp;M ← CREATESURROGATEMODEL()

5: &nbsp;&nbsp;&nbsp;&nbsp;t ← 0

6: &nbsp;&nbsp;&nbsp;&nbsp;**while** HASNEXT(S) **do** ▸ This is the training phase

7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(x<sub>t</sub>, y<sub>t</sub>) ← GETNEXTINSTANCE(S)

8: &nbsp;&nbsp;&nbsp;&nbsp;**Train**(i, x<sub>t</sub>, y<sub>t</sub>) ▸ Training the incumbent

9: &nbsp;&nbsp;&nbsp;&nbsp;**for all** m ∈ ms **do**

10: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Train**(m, x<sub>t</sub>, y<sub>t</sub>) ▸ Training the candidates

11: &nbsp;&nbsp;&nbsp;&nbsp;**t** ← **t** + 1

12: &nbsp;&nbsp;&nbsp;&nbsp;**if** t ≡ 0 (mod *p*) **then** ▸ A new cycle is started

13: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Train**(M, ms) ▸ Training the surrogate model

14: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;i ← argmin error ▸ Configuration with the best performance

15: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ms ← SELECTNEW(Σ, *B*, *M*, *i*)

16: **function** SELECTNEW(S, *B*, *M*, *i*)

17: &nbsp;&nbsp;&nbsp;&nbsp;C<sub>full</sub> ← SMACSELECTNEW(Σ, *M*, *i*) ▸ Use SMAC's candidate generator

18: &nbsp;&nbsp;&nbsp;&nbsp;C<sub>lim</sub> ← C<sub>full</sub>[: *B*] ▸ Keep the *B* first candidates

19: &nbsp;&nbsp;&nbsp;&nbsp;Append *i* to *C*<sub>lim</sub>

20: **return** *C*<sub>lim</sub>

<figure><img src="image_2.png" /><figcaption>Fig. 1: Selection process for the new incumbent: the former incumbent is kept if the configurations are equal</figcaption></figure>

they have been trained on the same data. As a result, although the budget specifies *B* candidates, a total of *B* + 1 models are trained during this phase. To avoid redundancy, the candidate set is deduplicated and topped-up with random configurations if necessary.

We also introduce a variant of the method based on an ensemble approach, called OnlineSMAC<sub>E</sub> (OSMAC<sub>E</sub>). Instead of relying on a single incumbent prediction, OSMAC<sub>E</sub> aggregates the *N* best configurations from the most recent cycle to form an ensemble. Predictions are then made via majority vote across these *N* incumbents. The model with the best performance among the incumbents serves as the reference for the logEI

<figure><img src="image_3.png" /><figcaption>Fig. 2: The autoML processing pipeline in OnlineSMAC at step *t*</figcaption></figure>
calculation.

### C. Processing pipeline

The autoML pipeline defines the data flow within the autoML system, detailing how data is preprocessed, how models are managed, and how their hyperparameters are optimized and selected. The pipeline comprises three key components: a data preprocessor, a feature preprocessor, and a learner, each with its own set of hyperparameters. An optimizer, OSMAC in our case, coordinates these components, searching for the best configuration to maximize performance.

OnlineSMAC covers a significant part of the autoML pipeline by itself, as shown in Figure 2. When building models, it selects at every step *t* both the algorithm and the hyperparameters for the data preprocessor and the learner.

## IV. EXPERIMENTS

This section provides an overview of the datasets, baseline, search space, and configurations used to evaluate the autoML systems for classification tasks.

### A. Setup

**Datasets.** We used real and synthetic datasets commonly used in the SotA to evaluate classification algorithms. The real datasets were sourced from the UCI Repository<fnref n="3" /> and the synthetic datasets were generated using MOA [10], where all settings were left to the default. The one exception is to require the classes to be balanced where possible. Besides, all datasets were preprocessed using Weka [25] with one-hot encoding for the nominal attributes. The characteristics of the datasets are summarized in Table I.

**Baselines.** All algorithms were implemented in Python 3 using River<fnref n="4" /> [29], a machine learning framework for data streams. We compare the regular and the ensemble version of OnlineSMAC against (i) streaming autoML solutions: ASML [6] and AutoClass (AC) [7], (ii) ensemble-based methods: Adaptive Random Forest (ARF) [30], Stream Random Patches (SRP) [31], and Leveraging Bagging (LB) [17]. Unless specified, all algorithms were evaluated using their default configuration as implemented in River. We include SMAC [20] in the accuracy comparison to provide a baseline for how the original model performs, even though it is not a streaming model (we use the SMAC3 implementation). For a fair comparison, all automated methods – OnlineSMAC, SMAC, ASML, and AutoClass – operate within the same search space.

**Search space.** The search space is a crucial element of the autoML pipeline, encompassing in our case preprocessors, classifiers, and their hyperparameters. The preprocessor component of the pipeline can be either MinMaxScaler, StandardScaler, or no preprocessing. The classifiers considered are ARF, Hoeffding Adaptive Tree (HAT), Naive Bayes (NB), Hoeffding Tree (HT), k-Nearest Neighbors (kNN), and Softmax [10]. The hyperparameter details are shown in Table II.

**Metrics.** The evaluation is made using three criteria: (i) the classification performance as the percentage of the overall accuracy (%) since all the datasets are balanced, (ii) the memory is reported as the memory consumption in megabytes, and (iii) the execution duration of a model is the time in seconds needed to process the entire stream, including all stages of the pipeline.

SMAC is a batch model, and partially implemented in C++ (whereas all other models are implemented entirely in Python). For this reason, we report only its accuracy for comparison, excluding runtime and memory usage.

### B. Protocol

The computations were carried out on a server running AlmaLinux 9, equipped with an Intel Xeon 5218 (Cascade Lake) processor. Each model was allocated two cores and 10 GB of RAM.

To mitigate random fluctuations, reduce bias, and ensure reproducibility, the algorithms are executed five times on each dataset, using a different seed for the pseudo-random number generator in each run. The specific seed values used were: 760, 10, 490, 158, 36. For each metric evaluated, we report the average performance over the five runs of each method on each dataset. We also provide the overall mean performance of the methods along with their average rank.

### C. Impact of hyperparameters' value

In order to select good defaults for OSMAC's hyperparameters, we study their impact on the model's behaviour. We change the update period (*p*) and the training budget (*B*) independently, and report on the different metrics after evaluation on the datasets listed in Table I.

For the update period *p*, we tested the values *p* = 500, 1000, 2000, and 5000 instances, with *B* set to 5. While there is some variation in the result, the difference for all metrics (Figure 3) was too small to be significant and no ranking can be given. We chose a value of *p* = 1000 as the default, to align with previous works [6], [7].

We evaluated budgets of 5, 10, 20, and 50 models. For these experiments, *p* had a value of 1000. For the budget, we find that an increase in the number of models trained degrades the memory footprint (Figure 4b) and the run duration (Figure 4c). However, we find no impact on the accuracy (Figure 4a). To reduce resource usage, we chose a budget of *B* = 5 for OnlineSMAC and, for the sake of fair comparison, we set the same budget for all the other composite models.

None of the two hyperparameter has a clear or significant impact on accuracy. However, the budget has a noticeable effect on running time.

### D. Results

Table III presents the predictive performance of the OSMAC versions compared to SotA automated and ensemble methods. The results demonstrate that our proposed method achieved competitive accuracy, closely matching that of existing automated methods. OnlineSMACEnns (OSMACEn) achieved the best average accuracy at 85.99 %, closely followed by ASML and

<footnote n="3">³https://archive.ics.uci.edu</footnote>

<footnote n="4">⁴https://riverml.xyz/</footnote>
<table><caption>TABLE I: Datasets configurations. A: abrupt drift, G: gradual drift, F: fast variant speed, S: slow variant speed. Maj and Min stands for the proportion of majority and minority class, respectively.</caption><thead><tr><th>Dataset</th><th>Type</th><th>#Instances</th><th>#Features</th><th>#Classes</th><th>Maj</th><th>Min</th></tr></thead><tbody><tr><td>Covtype [26]</td><td>real</td><td>581,012</td><td>54</td><td>7</td><td>36.5%</td><td>0.5%</td></tr><tr><td>Electricity [27]</td><td>real</td><td>45,312</td><td>8</td><td>2</td><td>57.5%</td><td>42.5%</td></tr><tr><td>Sensors [28]</td><td>real</td><td>2,219,803</td><td>5</td><td>55</td><td>3.0%</td><td>0.1%</td></tr><tr><td>Bananas</td><td>synthetic</td><td>5,300</td><td>2</td><td>2</td><td>55.2%</td><td>44.8%</td></tr><tr><td>Led</td><td>synthetic</td><td>1,000,000</td><td>24</td><td>10</td><td>10.0%</td><td>10.0%</td></tr><tr><td>Led<sub>A</sub></td><td>synthetic</td><td>1,000,000</td><td>24</td><td>10</td><td>10.1%</td><td>9.9%</td></tr><tr><td>Led<sub>G</sub></td><td>synthetic</td><td>1,000,000</td><td>24</td><td>10</td><td>10.1%</td><td>9.9%</td></tr><tr><td>Rbf</td><td>synthetic</td><td>1,000,000</td><td>10</td><td>5</td><td>30.0%</td><td>9.3%</td></tr><tr><td>Rbf<sub>F</sub></td><td>synthetic</td><td>1,000,000</td><td>10</td><td>5</td><td>30.0%</td><td>9.3%</td></tr><tr><td>Rbf<sub>S</sub></td><td>synthetic</td><td>1,000,000</td><td>10</td><td>5</td><td>30.0%</td><td>9.3%</td></tr><tr><td>Sea</td><td>synthetic</td><td>1,000,000</td><td>3</td><td>2</td><td>64.3%</td><td>35.7%</td></tr><tr><td>Sea<sub>A</sub></td><td>synthetic</td><td>1,000,000</td><td>3</td><td>2</td><td>59.9%</td><td>40.1%</td></tr><tr><td>Sea<sub>G</sub></td><td>synthetic</td><td>1,000,000</td><td>3</td><td>2</td><td>59.9%</td><td>40.1%</td></tr></tbody></table>

<table><caption>TABLE II: hyperparameters search space</caption><thead><tr><th>Model</th><th>Parameter</th><th>Type</th><th>Range</th></tr></thead><tbody><tr><td rowspan="5">kNN</td><td>neighbors</td><td>integer</td><td>[3, 8]</td></tr><tr><td>weighted</td><td>boolean</td><td>True, False</td></tr><tr><td>window size</td><td>integer</td><td>[100, 2000]</td></tr><tr><td>power</td><td>integer</td><td>[1, 5]</td></tr><tr><td>l2</td><td>real</td><td>[0, 10<sup>-1</sup>]</td></tr><tr><td>SoftMax</td><td>with std. dev.</td><td>boolean</td><td>True, False</td></tr><tr><td rowspan="3">HT, HAT</td><td>max. depth</td><td>integer</td><td>[10, 100]</td></tr><tr><td>grace period</td><td>integer</td><td>[50, 500]</td></tr><tr><td>delta</td><td>real</td><td>[10<sup>-9</sup>, 10<sup>-2</sup>]</td></tr><tr><td rowspan="7">ARF</td><td>tau</td><td>real</td><td>[2 · 10<sup>-2</sup>, 8 · 10<sup>-2</sup>]</td></tr><tr><td>threshold</td><td>integer</td><td>[0, 50]</td></tr><tr><td>split criterion</td><td>nominal</td><td>info. gain, Gini, Hellinger</td></tr><tr><td>prediction type</td><td>nominal</td><td>MC, NB, NBA</td></tr><tr><td>nb. models</td><td>integer</td><td>[3, 10]</td></tr><tr><td>max. depth</td><td>integer</td><td>[10, 100]</td></tr><tr><td>grace period</td><td>integer</td><td>[50, 500]</td></tr><tr><td rowspan="5">ARF</td><td>delta</td><td>real</td><td>[10<sup>-9</sup>, 10<sup>-2</sup>]</td></tr><tr><td>tau</td><td>real</td><td>[2 · 10<sup>-2</sup>, 8 · 10<sup>-2</sup>]</td></tr><tr><td>threshold</td><td>integer</td><td>[0, 50]</td></tr><tr><td>split criterion</td><td>nominal</td><td>info. gain, Gini, Hellinger</td></tr><tr><td>prediction type</td><td>nominal</td><td>MC, NB, NBA</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>(a) Accuracy</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Memory</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c) Duration</figcaption></figure>

<caption>Fig. 3: Impact of the period on the metrics: no significant difference exists</caption>

OnlineSMAC. However, because it had more often the best accuracy across datasets, ASML earned the best overall rank, OSMAC<sub>E</sub> ranked the second best overall. For each dataset, the accuracy difference between OSMAC<sub>E</sub> and ASML was statistically insignificant, often less than 1 pp. As shown by the statistical analysis in Figure 5a, OnlineSMAC and ASML perform equivalently in terms of accuracy. Both versions of OSMAC consistently outperformed SMAC across all datasets. This improvement is statistically significant, as confirmed by Tukey's HSD test at the 95% confidence level (Figure 5a).

Table IV shows the memory use of the methods, and Figure 5b the critical distance (CD) diagram of their average rank. All the methods used more memory for the synthetic and Sensors data because they have more instances. As expected, the ensemble version (OSMAC<sub>E</sub>) incurred slightly higher memory usage than the regular version, due to the need to aggregate predictions from the ensemble members. OnlineSMAC ranked as the second best method on average, just behind ASML. This result is not surprising, as our method keeps an internal surrogate model throughout execution, which contributes to its overall memory footprint. Regular ensemble-based methods (ARF, SRP, and LB) still consumed significantly more memory (see Figure 6a) because they rely on the same learner for all

models in the ensemble. In contrast, automated approaches use heterogeneous ensembles, which can be composed of more efficient models, potentially reducing memory consumption.

In terms of execution time, as seen in Table V, classical ensemble-based methods were faster than the automated approaches. This is expected, as automated methods must regularly refresh and adapt their ensemble, introducing a natural trade-off between execution time and predictive accuracy. Despite this, OSMAC stands out as the most efficient among automated methods, with an average execution time of 5971 s,
<table><caption>TABLE III: Accuracy (in %), with standard deviation between runs</caption><thead><tr><th></th><th>OSMAC</th><th>OSMAC<sub>E</sub></th><th>ASML</th><th>AC</th><th>ARF</th><th>SRP</th><th>LB</th><th>SMAC</th></tr></thead><tbody><tr><td>Covertype</td><td>95.57±0.15</td><td><b>95.76</b>±0.08</td><td>95.73±0.22</td><td>93.90±4.02</td><td>85.27±0.15</td><td>91.70±0.18</td><td>85.52±0.11</td><td>51.50±0.78</td></tr><tr><td>Electricity</td><td>87.39±1.02</td><td>88.53±0.57</td><td>88.45±0.36</td><td>88.00±3.43</td><td>87.43±0.09</td><td>87.35±0.63</td><td><b>88.98</b>±0.10</td><td>94.79±2.54</td></tr><tr><td>Sensors</td><td>95.12±0.16</td><td><b>95.74</b>±0.08</td><td>94.93±0.16</td><td>72.53±5.03</td><td>65.91±0.30</td><td>65.39±0.46</td><td>66.07±0.30</td><td>85.13±0.38</td></tr><tr><td>Bananas</td><td>83.48±1.61</td><td>85.97±2.29</td><td>82.69±2.66</td><td>84.14±2.78</td><td><b>87.04</b>±0.19</td><td>86.54±0.64</td><td>82.25±0.88</td><td>85.99±2.72</td></tr><tr><td>Led</td><td>73.56±0.13</td><td>73.19±0.10</td><td>73.87±0.11</td><td>73.50±1.02</td><td>70.26±0.69</td><td>69.87±1.89</td><td><b>73.89</b>±0.03</td><td>64.21±0.83</td></tr><tr><td>Led<sub>A</sub></td><td>73.60±0.11</td><td>73.36±0.07</td><td><b>73.98</b>±0.01</td><td>70.03±1.79</td><td>70.48±0.20</td><td>69.79±1.34</td><td>73.73±0.07</td><td>64.65±0.69</td></tr><tr><td>Led<sub>G</sub></td><td>72.61±0.14</td><td>72.39±0.02</td><td><b>73.12</b>±0.01</td><td>68.72±2.32</td><td>69.49±0.26</td><td>70.14±0.68</td><td>72.90±0.04</td><td>63.71±0.83</td></tr><tr><td>Rbf</td><td>91.81±0.07</td><td><b>91.96</b>±0.05</td><td>91.41±0.04</td><td>82.39±6.78</td><td>63.36±0.35</td><td>79.67±1.70</td><td>90.47±0.12</td><td>85.54±0.41</td></tr><tr><td>Rbf<sub>F</sub></td><td>86.42±0.43</td><td>87.80±0.48</td><td><b>88.20</b>±0.09</td><td>74.85±15.60</td><td>60.99±0.15</td><td>63.70±0.11</td><td>56.47±0.15</td><td>83.92±0.89</td></tr><tr><td>Rbf<sub>S</sub></td><td>92.01±0.06</td><td><b>92.16</b>±0.04</td><td>91.46±0.02</td><td>80.19±10.28</td><td>73.61±0.62</td><td>77.97±0.28</td><td>82.03±0.10</td><td>85.97±0.34</td></tr><tr><td>Sea</td><td>87.38±0.14</td><td>87.10±0.13</td><td>87.68±0.06</td><td>87.10±2.56</td><td>89.60±0.02</td><td>88.54±0.31</td><td><b>89.80</b>±0.01</td><td>85.18±0.16</td></tr><tr><td>Sea<sub>A</sub></td><td>87.21±0.17</td><td>87.09±0.16</td><td>87.79±0.04</td><td>86.81±2.48</td><td><b>89.27</b>±0.02</td><td>83.43±4.11</td><td>88.85±0.21</td><td>84.78±0.28</td></tr><tr><td>Sea<sub>G</sub></td><td>87.08±0.10</td><td>86.89±0.05</td><td>87.51±0.06</td><td>86.58±2.40</td><td>88.77±0.02</td><td>81.58±3.46</td><td><b>88.98</b>±0.06</td><td>84.62±0.23</td></tr><tr><td>Average</td><td>85.63</td><td><b>85.99</b></td><td>85.91</td><td>80.67</td><td>77.04</td><td>78.13</td><td>80.00</td><td>78.46</td></tr><tr><td>Avg. rank</td><td>3.46</td><td>3.15</td><td><b>2.69</b></td><td>5.00</td><td>4.69</td><td>5.69</td><td>3.31</td><td>6.38</td></tr></tbody></table>

<table><caption>TABLE IV: Memory consumption (in megabytes)</caption><thead><tr><th></th><th>OSMAC</th><th>OSMAC<sub>E</sub></th><th>ASML</th><th>AC</th><th>ARF</th><th>SRP</th><th>LB</th></tr></thead><tbody><tr><td>Covertype</td><td>4.92</td><td>6.28</td><td>6.45</td><td>28.09</td><td><b>1.01</b></td><td>16.05</td><td>9.71</td></tr><tr><td>Electricity</td><td>2.57</td><td>4.71</td><td>4.50</td><td>3.70</td><td>4.13</td><td>27.43</td><td><b>2.54</b></td></tr><tr><td>Sensors</td><td>4.76</td><td>5.11</td><td>4.25</td><td>4.47</td><td><b>0.19</b></td><td>0.35</td><td>0.88</td></tr><tr><td>Bananas</td><td><b>1.42</b></td><td>1.69</td><td>2.19</td><td>2.13</td><td>6.54</td><td>5.91</td><td>1.82</td></tr><tr><td>Led</td><td>7.86</td><td>11.60</td><td><b>0.51</b></td><td>53.78</td><td>42.67</td><td>565.79</td><td>347.74</td></tr><tr><td>Led<sub>A</sub></td><td>7.74</td><td>11.10</td><td><b>1.48</b></td><td>38.74</td><td>44.31</td><td>606.81</td><td>138.96</td></tr><tr><td>Led<sub>G</sub></td><td>7.99</td><td>12.20</td><td><b>0.68</b></td><td>40.10</td><td>28.02</td><td>555.18</td><td>201.52</td></tr><tr><td>Rbf</td><td>8.05</td><td>7.41</td><td>4.33</td><td>67.81</td><td><b>2.07</b></td><td>51.15</td><td>374.75</td></tr><tr><td>Rbf<sub>F</sub></td><td>7.48</td><td>7.13</td><td>1.38</td><td>15.36</td><td>13.57</td><td>13.64</td><td><b>1.13</b></td></tr><tr><td>Rbf<sub>S</sub></td><td>8.59</td><td>7.27</td><td><b>3.20</b></td><td>80.69</td><td>7.68</td><td>40.62</td><td>33.43</td></tr><tr><td>Sea</td><td>8.88</td><td>10.27</td><td><b>0.19</b></td><td>2.49</td><td>572.78</td><td>341.43</td><td>280.60</td></tr><tr><td>Sea<sub>A</sub></td><td>8.09</td><td>7.14</td><td><b>0.18</b></td><td>2.57</td><td>197.68</td><td>284.00</td><td>172.60</td></tr><tr><td>Sea<sub>G</sub></td><td>7.97</td><td>8.59</td><td><b>0.17</b></td><td>2.90</td><td>145.86</td><td>281.73</td><td>94.12</td></tr><tr><td>Average</td><td>6.64</td><td>7.73</td><td><b>2.27</b></td><td>26.37</td><td>82.04</td><td>214.62</td><td>127.68</td></tr><tr><td>Avg. rank</td><td>3.00</td><td>3.54</td><td><b>2.23</b></td><td>4.54</td><td>4.15</td><td>6.08</td><td>4.46</td></tr></tbody></table>

while its ensemble variant, OSMAC<sub>E</sub>, maintains a reasonable cost at 9914 s. Both significantly outperform AutoClass (19 225 s) and ASML, as also highlighted in Table V. This difference is well visible on datasets like Covertype or Sensors, where OSMAC ran four to ten times faster than AutoClass and ASML, while still achieving competitive performance. On simpler datasets such as Sea ones, where the feature space is low and the task is binary, AutoClass and ASML perform better. Overall, as illustrated in Figure 6b, OSMAC and OSMAC<sub>E</sub> maintain a strong balance between adaptability and efficiency, with executions times close to non-tuned ensemble baselines such as LB and SRP, while offering the added benefit of autonomous, dynamic model optimization.

## E. Discussion

These results lead to two key questions: what makes OnlineSMAC effective in optimizing model configurations for streaming data? And more importantly, what challenges still limit its performance – and how might they be addressed?

At the core of OSMAC lies the incumbent selection strategy, which evaluates models over sliding windows of 1000 instances from the data stream. This method assumes that instances of the data stream are independent and identically distributed (i.i.d.), meaning that the outcome of the evaluation for a

training window holds for future data. Under this assumption, a poorly performing configuration during such an evaluation is presumed to remain ineffective. However, real-world streams break this i.i.d. assumption due to concept drift, where data distributions evolve over time. While OSMAC addresses this with a surrogate model based on ARF<sub>R</sub> regressor, which adapts to drift, additional challenges persist.

Using a Random Forest – although efficient and robust – as a surrogate model has a well-known drawback when applied to regression in online settings: decision trees are suboptimal regressors. They operate in discrete steps rather than as smooth interpolators, resulting in a discontinuous surface. Moreover, unlike Gaussian processes, they lack mechanisms to estimate their uncertainty. In the batch setting, the vanilla SMAC [9] successfully used the variance of the individual trees’ predictions. Yet, when applied to streaming contexts, this error estimation method proved unreliable.

To address this, we used the Jackknife method, which produces a single, non-conditional value updated only during training. While effective, it cannot capture varying uncertainties across sampling points, such as higher uncertainty in less-explored regions. Although the Jackknife method delivered consistently strong results, validating its use within our solution.

That said, integrating more granular, dynamically adaptive
<table><caption>TABLE V: Execution time (in seconds)</caption><thead><tr><th></th><th>OSMAC</th><th>OSMAC<sub>E</sub></th><th>ASML</th><th>AC</th><th>ARF</th><th>SRP</th><th>LB</th></tr></thead><tbody><tr><td>Covertype</td><td>6 034.69</td><td>9 754.72</td><td>106 949.84</td><td>62 860.46</td><td><strong>442.57</strong></td><td>2 316.62</td><td>2 892.55</td></tr><tr><td>Electricity</td><td>153.66</td><td>278.37</td><td>921.32</td><td>815.76</td><td><strong>32.41</strong></td><td>92.88</td><td>70.46</td></tr><tr><td>Sensors</td><td>18 012.19</td><td>29 331.62</td><td>76 567.59</td><td>59 799.96</td><td><strong>4 012.28</strong></td><td>14 455.07</td><td>17 025.06</td></tr><tr><td>Bananas</td><td>16.53</td><td>29.90</td><td>48.81</td><td>55.74</td><td><strong>2.83</strong></td><td>6.43</td><td>4.86</td></tr><tr><td>Led</td><td>4 645.79</td><td>6 484.31</td><td>3 555.84</td><td>4 695.40</td><td><strong>1 224.81</strong></td><td>5 635.34</td><td>8 019.95</td></tr><tr><td>Led<sub>A</sub></td><td>4 686.50</td><td>6 124.15</td><td>3 645.06</td><td>4 858.68</td><td><strong>1 221.07</strong></td><td>5 856.37</td><td>8 021.14</td></tr><tr><td>Led<sub>G</sub></td><td>4 730.52</td><td>6 016.42</td><td>3 542.74</td><td>4 937.49</td><td><strong>1 192.62</strong></td><td>5 689.79</td><td>8 009.15</td></tr><tr><td>Rbf</td><td>10 945.98</td><td>20 847.35</td><td>80 615.83</td><td>39 980.41</td><td><strong>669.41</strong></td><td>2 290.70</td><td>2 738.24</td></tr><tr><td>Rbf<sub>F</sub></td><td>10 267.72</td><td>19 753.07</td><td>24 220.73</td><td>35 529.71</td><td><strong>749.26</strong></td><td>2 859.56</td><td>2 809.39</td></tr><tr><td>Rbf<sub>S</sub></td><td>11 222.03</td><td>22 261.80</td><td>53 777.36</td><td>33 244.51</td><td><strong>670.19</strong></td><td>2 314.16</td><td>2 489.82</td></tr><tr><td>Sea</td><td>2 367.71</td><td>2 699.12</td><td>1 087.32</td><td><strong>1 043.45</strong></td><td>1 520.87</td><td>2 034.50</td><td>1 073.17</td></tr><tr><td>Sea<sub>A</sub></td><td>2 296.17</td><td>2 649.77</td><td>1 069.19</td><td>1 061.55</td><td><strong>965.18</strong></td><td>1 468.88</td><td>1 016.40</td></tr><tr><td>Sea<sub>G</sub></td><td>2 246.45</td><td>2 663.07</td><td>1 089.53</td><td>1 045.62</td><td><strong>786.56</strong></td><td>1 676.51</td><td>954.82</td></tr><tr><td>Average</td><td>5 971.23</td><td>9 914.90</td><td>27 468.55</td><td>19 225.29</td><td><strong>1 037.70</strong></td><td>3 592.06</td><td>4 240.39</td></tr><tr><td>Avg. rank</td><td>4.23</td><td>5.69</td><td>4.92</td><td>4.85</td><td><strong>1.23</strong></td><td>3.62</td><td>3.46</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>(a) Accuracy</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Memory</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) Duration</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(a) Average accuracy with 95 % confidence interval</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Critical distance (CD) diagram for memory use</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(a) Memory use</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) Run duration</figcaption></figure>

<caption>Fig. 6: Cumulative results per method for all datasets</caption>

uncertainty estimation techniques could further enhance model selection and accelerate convergence in evolving environments.

## V. CONCLUSION

In this paper, we present OnlineSMAC and its ensemble variant OSMAC<sub>E</sub>, machine learning solutions inspired by the SMAC optimizer. While previous autoML approaches for data streams have relied on evolutionary algorithms, our work introduces a Bayesian optimization framework tailored for data streams. This is achieved by (i) adapting the surrogate
model for online learning and (ii) evaluating candidate models relative to one another rather than against a fixed incumbent.

Our main proposal opens a new avenue for online autoML, by leveraging insights from batch settings to create a model-based optimizer. Experimental results demonstrate that OnlineSMAC achieves competitive performance comparable to SotA methods in terms of classification accuracy, memory use, and run time. These findings highlight its potential as a promising direction for future research.

Future research directions include exploring the impact of initial configurations and developing reliable online error estimation for the surrogate model. Additionally, current evaluations of classification rely solely on raw accuracy. Incorporating multi-objective optimization could enable the selection of Pareto-optimal configurations, balancing accuracy, run time, and memory footprint.

## REFERENCES

[1] M. Baratchi et al., “Automated machine learning: Past, present and future,” *Artif Intell Rev*, vol. 57, no. 5, p. 122, 2024, ISSN: 1573-7462. DOI: 10.1007/s10462-024-10726-1.

[2] M. Bahri, F. Salutari, A. Putina, and M. Sozio, “AutoML: State of the art with a focus on anomaly detection, challenges, and research directions,” *Int J Data Sci Anal*, vol. 14, no. 2, pp. 113–126, 2022, ISSN: 2364-4168. DOI: 10.1007/s41060-022-00309-0.

[3] F. Hutter, L. Kotthoff, and J. Vanschoren, Eds., *Automated Machine Learning: Methods, Systems, Challenges* (The Springer Series on Challenges in Machine Learning). Cham: Springer International Publishing, 2019, ISBN: 978-3-030-05318-5. DOI: 10.1007/978-3-030-05318-5.

[4] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” *ACM Comput. Surv.*, vol. 46, no. 4, 44:1–44:37, 2014, ISSN: 0360-0300. DOI: 10.1145/2523813.

[5] A. Bifet, R. Gavaldà, G. Holmes, and B. Pfahringer, *Machine Learning for Data Streams: With Practical Examples in MOA*. The MIT Press, 2018, ISBN: 978-0-262-34604-7. DOI: 10.7551/mitpress/10654.001.0001.

[6] N. Verma, A. Bifet, B. Pfahringer, and M. Bahri, “ASML: A scalable and efficient AutoML solution for data streams,” in *Proceedings of the Third International Conference on Automated Machine Learning*, Paris, France: PMLR, 2024, pp. 11/1–26. [Online]. Available: https://proceedings.mlr.press/v256/verma24a.html.

[7] M. Bahri and N. Georgantas, “AutoClass: AutoML for data stream classification,” in *2023 IEEE International Conference on Big Data (BigData)*, Sorrento, Italy: IEEE, 2023, pp. 2658–2666. DOI: 10.1109/BigData59044.2023.10386362.

[8] C. Kulbach, J. Montiel, M. Bahri, M. Heyden, and A. Bifet, “Evolution-based online automated machine learning,” in *Advances in Knowledge Discovery and Data Mining*, J. Gama, T. Li, Y. Yu, E. Chen, Y. Zheng, and F. Teng, Eds., ser. Lecture Notes in Computer Science, vol. 13280, Cham: Springer International Publishing, 2022, pp. 472–484, ISBN: 978-3-031-05933-9. DOI: 10.1007/978-3-031-05933-9_37.

[9] F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in *Learning and Intelligent Optimization*, C. A. C. Coello, Ed., ser. Lecture Notes in Computer Science, vol. 6683, Berlin, Heidelberg: Springer, 2011, pp. 507–523, ISBN: 978-3-642-25566-3. DOI: 10.1007/978-3-642-25566-3_40.

[10] M. Bahri, A. Bifet, J. Gama, H. M. Gomes, and S. Maniu, “Data stream analysis: Foundations, major tasks and tools,” *WIREs DMKD*, vol. 11, no. 3, e1405, 2021. DOI: 10.1002/widm.1405.

[11] C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “AutoWEKA: Combined selection and hyperparameter optimization of classification algorithms,” in *Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, ser. KDD '13, New York, NY, USA: Association for Computing Machinery, 2013, pp. 847–855, ISBN: 978-1-4503-2174-7. DOI: 10.1145/2487575.2487629.

[12] M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter, “Efficient and robust automated machine learning,” in *Advances in Neural Information Processing Systems*, vol. 28, Montréal, Canada: Curran Associates, Inc., 2015, pp. 2962–2970, ISBN: 978-1-5108-2502-4.

[13] E. LeDell and S. Poirier, “H2O AutoML: Scalable automatic machine learning,” presented at the 7th ICML Workshop on Automated Machine Learning, 2020.

[14] P. Gijsbers and J. Vanschoren, “GAMA: Genetic automated machine learning assistant,” *JOSS*, vol. 4, no. 33, p. 1132, 2019, ISSN: 2475-9066. DOI: 10.21105/joss.01132.

[15] T. T. Le, W. Fu, and J. H. Moore, “Scaling tree-based automated machine learning to biomedical big data with a feature set selector,” *Bioinformatics*, vol. 36, no. 1, pp. 250–256, 2020, ISSN: 1367-4803. DOI: 10.1093/bioinformatics/btz470.

[16] B. Veloso, J. Gama, and B. Malheiro, “Self hyper-parameter tuning for data streams,” in *Discovery Science*, L. Soldatova, J. Vanschoren, G. Papadopoulos, and M. Ceci, Eds., Cham: Springer International Publishing, 2018, pp. 241–255, ISBN: 978-3-030-01771-2. DOI: 10.1007/978-3-030-01771-2_16.

[17] A. Bifet, G. Holmes, and B. Pfahringer, “Leveraging bagging for evolving data streams,” in *Machine Learning and Knowledge Discovery in Databases*, J. L. Balcázar, F. Bonchi, A. Gionis, and M. Sebag, Eds., ser. Lecture Notes in Computer Science, Barcelona, Spain: Springer, 2010, pp. 135–150, ISBN: 978-3-642-15880-3. DOI: 10.1007/978-3-642-15880-3_15.

[18] Q. Wu, C. Wang, J. Langford, P. Mineiro, and M. Rossi, “ChaCha for online AutoML,” in *Proceedings of the 38th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, vol. 139, Online: PMLR, 2021, pp. 11263–11273. [Online]. Available: https://proceedings.mlr.press/v139/wu21d.html.

[19] B. Celik, P. Singh, and J. Vanschoren, “Online AutoML: An adaptive AutoML framework for online learning,” *ML*, vol. 112, no. 6, pp. 1897–1921, 2023, ISSN: 1573-0565. DOI: 10.1007/s10994-022-06262-0.

[20] M. Lindauer et al., “SMAC3: A versatile bayesian optimization package for hyperparameter optimization,” *J. Mach. Learn. Res.*, vol. 23, no. 54, pp. 1–9, 2022, ISSN: 1533-7928. DOI: 10.48550/arXiv.2109.09831.

[21] R. Garnett, *Bayesian Optimization*. Cambridge: Cambridge University Press, 2023, ISBN: 978-1-108-42578-0. DOI: 10.1017/9781108348973.

[22] H. M. Gomes, J. P. Barddal, L. E. B. Ferreira, and A. Bifet, “Adaptive random forests for data stream regression,” in *Proceedings of the 26th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning*, Bruges, Belgium, 2018, ISBN: 978-2-87587-047-6.

[23] S. Wager, T. Hastie, and B. Efron, “Confidence intervals for random forests: The jackknife and the infinitesimal jackknife,” *JMLR*, vol. 15, no. 1, pp. 1625–1651, 2014, ISSN: 1532-4435.

[24] S. Ament, S. Daulton, D. Eriksson, M. Balandat, and E. Bakshy, “Unexpected improvements to expected improvement for Bayesian optimization,” in *Advances in Neural Information Processing Systems*, vol. 36, New Orleans, USA: Curran Associates, Inc., 2023, pp. 20577–20612, ISBN: 978-1-7138-9992-1.

[25] E. Frank, M. A. Hall, I. H. Witten, and C. J. Pal, “The WEKA workbench,” in *Data Mining: Practical Machine Learning Tools and Techniques*, ser. Morgan Kaufmann Series in Data Management Systems, 4th ed., Amsterdam: Elsevier, 2017, ISBN: 978-0-12-804291-5.

[26] J. Blackard and D. Dean, *Covertype*, UCI Machine Learning Repository, 1998. DOI: 10.24432/C50K5N.

[27] H. Harries and J. Gama, *Electricity*, 2004. [Online]. Available: https://www.inescporto.pt/~jgama/ales/ales_5.html.

[28] S. Madden, *Intel Lab Data*, 2004. [Online]. Available: https://db.csail.mit.edu/labdata/labdata.html.

[29] J. Montiel et al., “River: Machine learning for streaming data in Python,” *JMLR*, vol. 22, no. 110, pp. 1–8, 110 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1380.html.

[30] H. M. Gomes et al., “Adaptive random forests for evolving data stream classification,” *ML*, vol. 106, no. 9, pp. 1469–1495, 2017, ISSN: 1573-0565. DOI: 10.1007/s10994-017-5642-8.

[31] H. M. Gomes, J. Read, and A. Bifet, “Streaming random patches for evolving data stream classification,” in *2019 IEEE International Conference on Data Mining*, Beijing, China: IEEE, 2019, pp. 240–249. DOI: 10.1109/ICDM.2019.00034.