<figure><img src="image_1.png" /></figure>

HAL
open science

# Construction and Update of an Online Ensemble Score Involving Linear Discriminant Analysis and Logistic Regression

Benoît Lalloué, Jean-Marie Monnez, Eliane Albuisson

► To cite this version:

Benoît Lalloué, Jean-Marie Monnez, Eliane Albuisson. Construction and Update of an Online Ensemble Score Involving Linear Discriminant Analysis and Logistic Regression. Applied Mathematics, 2022, 13 (2), pp.228-242. 10.4236/am.2022.132018. hal-03134248v2

HAL Id: hal-03134248

https://hal.science/hal-03134248v2

Submitted on 4 Mar 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Construction and Update of an Online Ensemble Score Involving Linear Discriminant Analysis and Logistic Regression

Benoît Lalloué<fnref n="1" />,<fnref n="2" />, Jean-Marie Monnez<fnref n="1" />,<fnref n="2" />*, Eliane Albuisson<fnref n="3" />,<fnref n="4" />,<fnref n="5" />

**How to cite this paper:** Lalloué, B., Monnez, J.-M. and Albuisson, E. (2022) Construction and Update of an Online Ensemble Score Involving Linear Discriminant Analysis and Logistic Regression. *Applied Mathematics, 13*, 228-242.

https://doi.org/10.4236/am.2022.132018

Received: December 22, 2021

Accepted: February 25, 2022

Published: February 28, 2022

Copyright © 2022 by author(s) and
Scientific Research Publishing Inc.
This work is licensed under the Creative
Commons Attribution International
License (CC BY 4.0).

http://creativecommons.org/licenses/by/4.0/

<figure><img src="image_3.png" /></figure>

Open Access

## Abstract

The present aim is to update, upon arrival of new learning data, the parameters of a score constructed with an ensemble method involving linear discriminant analysis and logistic regression in an online setting, without the need to store all of the previously obtained data. Poisson bootstrap and stochastic approximation processes were used with online standardized data to avoid numerical explosions, the convergence of which has been established theoretically. This empirical convergence of online ensemble scores to a reference “batch” score was studied on five different datasets from which data streams were simulated, comparing six different processes to construct the online scores. For each score, 50 replications using a total of 10N observations (N being the size of the dataset) were performed to assess the convergence and the stability of the method, computing the mean and standard deviation of a convergence criterion. A complementary study using 100N observations was also performed. All tested processes on all datasets converged after N iterations, except for one process on one dataset. The best processes were averaged processes using online standardized data and a piecewise constant step-size.

## Keywords

Learning for Big Data, Stochastic Approximation, Medicine, Ensemble Method, Online Score

<footnote n="1">¹IECL (Institut Elie Cartan de Lorraine), Inria (Project-Team BIGS), Centre National de la Recherche Scientifique (CNRS), Université de Lorraine, Nancy, France</footnote>

<footnote n="2">²Inserm U1116, Centre d'Investigation Clinique Plurithématique 1433, Université de Lorraine, Nancy, France</footnote>

<footnote n="3">³IECL (Institut Elie Cartan de Lorraine), Centre National de la Recherche Scientifique (CNRS), Université de Lorraine, Nancy, France</footnote>

<footnote n="4">⁴Délégation à la Recherche Clinique et à l'Innovation(DRCI), CHRU de Nancy, Vandœuvre-lès-Nancy, France</footnote>

<footnote n="5">⁵Département Grand-Est de Recherche en Soins Primaires, Faculté de Médecine, Vandœuvre-lès-Nancy, France</footnote>

<footnote>Email: b.lalloue@gmail.com, *jean-marie.monnez@univ-lorraine.fr, eliane.albuisson@univ-lorraine.fr</footnote>
# 1. Introduction

When considering the problem of predicting the values of a dependent variable $y$, whether continuous (in the case of regression) or categorical (in the case of classification), from observed variables $x^1, \cdots, x^n$, which are themselves continuous or categorical, many different predictors can be constructed to address this problem. The principle of ensemble methods is to construct a set of “basic” individual predictors (using classical methods) whose predictions are then aggregated by average or by vote. Provided that the individual predictors are relatively good and sufficiently different from each other, ensemble methods generally yield more stable predictors than individual predictors [1].

This set of individual predictors can be constructed through different means, used separately or in combination, in order to obtain differences between them. Various types of regressions or rules of classification can be used as well as different samples (e.g. bootstrap), different variable selection methods (random, stepwise selection, shrinkage methods, etc.) or more generally by introducing a random element in the construction of predictors. Bagging [2], boosting [3], random forests [1] or Random Generalized Linear Models (RGLM) [4] are examples of ensemble methods. Another method for constructing an ensemble score in seven steps was recently proposed in Duarte et al. [5], used in Lalloué et al. [6] and will be used as a reference in this article:

1) Selection of $n_1$ classification rules.

2) Generation of $n_2$ bootstrap samples which are the same as for the $n_1$ rules.

3) Choice of $n_3$ modalities of a random selection of variables. For each bootstrap sample, selection of *m* variables according to these modalities.

4) Selection of $m^*$ variables among *m* by a classical method (stepwise, shrinkage, etc.).

5) For each classification rule, construction of the $n_2 n_3$ predictors corresponding to the bootstrap sample and the selected variables.

6) For each classification rule, aggregation of predictors into an intermediate score.

7) Aggregation of the $n_1$ intermediate scores from the previous step by averaging or voting.

Herein, we consider the case where $y$ is a binary variable and the classification rules are linear discriminant analysis (LDA) and logistic regression (LR).

In the context of online data, i.e. a flow of data arriving continuously, one wishes to be able to update such an ensemble score when new data becomes available, without having to store all of the previously obtained data and without performing the entire analysis. To achieve this goal, stochastic approximation processes [7] [8] [9] can be used. In particular, processes that we have previously studied theoretically [10] [11] will be detailed in Section 2.

However, the theoretical guarantees of convergence already demonstrated for this type of process provide little information on the practical choices to be made
in order to obtain the best performances: e.g. “classical” or averaged processes [8]
[10] [11], continuously decreasing step-size or decreasing piecewise constant step-
size [11] [12] or constant step-size [10], use at each step of a mini-batch of ob-
servations or all observations up to the current step in the case of LDA [10].
Therefore, Section 3 is dedicated to the empirical testing of several online en-
semble scores on several datasets, using several stochastic approximation pro-
cesses for each classifier and comparing the accuracy of the estimations. To avoid
a numerical explosion in the presence of heterogeneous data or outliers, an on-
line standardization of the data is used as tested in [10] [11]. Moreover, an in-
adequate choice of the step-size can also lead to a numerical explosion in the
non-asymptotic phase of the process or slow down its convergence. Thus, several
types of step-sizes are tested. A conclusion of this study is that processes which
have the best performance among those tested are not the classical processes
with a continuously decreasing step-size and a mini-batch of observations at
each step in the case of LDA.

## 2. Theoretical Construction and Update of an Online Ensemble Score

In order to be able to update online the ensemble score defined in [5] based on
linear discriminant analysis and logistic regression, each bootstrap sample and
each predictor must be updated when new data arrive [13]. Once the predictors
are updated, the intermediate scores and the resulting final ensemble score are
obtained using the same aggregation rules as for the offline ensemble method.

2.1. Updating the Bootstrap Samples

Starting from a sample size of *n*, the usual construction of a bootstrap sample
consists in drawing at random with replacement *n* elements of the sample. In the
case of a data stream, the Poisson bootstrap method proposed by Oza and Rus-
sell [14] can be used to update a bootstrap sample: for any new data, for each
bootstrap sample $b_i (i = 1, \dots, n_2)$, a realization $k_i$ of a random variable under
a Poisson law with parameter 1 is simulated, and the new data is added $k_i$
times to sample $b_i$. These new data can then be used to update the predictors
defined using sample $b_i$.

## 2.2. Updating the Predictors

Recursive stochastic approximation algorithms which take into account a
mini-batch of new data at each step can be used to update the predictors. Such
algorithms have been developed to estimate linear [10] or logistic [11] regression
parameters, or to estimate the class centers in unsupervised classification [15] or
the principal components of a factor analysis [16]. These algorithms do not re-
quire storing data and can, within a fixed timeframe, process more data than
offline methods. Stochastic approximation algorithms able to update predictors
obtained by linear discriminant analysis (LDA, equivalent to linear regression in
the case of a binary dependent variable) and logistic regression (LR) are described below.

### 2.2.1. Updating Logistic and Linear Regressions Using a Mini-Batch of Observations at Each Step

Note that all stochastic approximation algorithms described in this section use an online standardization of the data. Indeed, in practical applications, an inadequate choice of step-size of these processes or the presence of heterogeneous data or outliers can lead to numerical explosion issues in the non-asymptotic phase of the stochastic approximation process. To avoid numerical explosions in the presence of heterogeneous data, an online standardization of the data is proposed [10] [11]; in the case of a data stream, the moments of the regression variables are a priori not known, but can be estimated online in order to perform the standardization. However, in this instance, the convergence of the stochastic approximation process is not ensured by classical theorems and was therefore proven in [10] in the case of linear regression, and in [11] in the case of logistic regression. Moreover, a too rapid decrease in step-size may reduce the speed of convergence in the non-asymptotic phase of the process. For this reason, following [12], the use of a decreasing piecewise constant step-size has been tested in [11].

Consider first the case of logistic regression. Let $\mathcal{S}$ be a random variable taking its values in $\{0,1\}$ and $R = (R^1 \cdots R^p 1)'$ with $R^1, \cdots, R^p$ being random variables taking values in $\mathbb{R}$, $m = (E[R^1] \cdots E[R^p] 0)'$, $R^c = R - m$, $\sigma^k$ the standard deviation of $R^k$, $\Gamma$ the diagonal square matrix with diagonal elements $\frac{1}{\sigma^1}, \cdots, \frac{1}{\sigma^p}, 1$, $Z = \Gamma R^c$ the standardized $R$ vector, $\theta(p+1, 1)$ the vector of parameters and $h(u) = \frac{e^u}{1+e^u}$. The vector $\theta$ is the unique solution of the system of equations $\mathbb{E}[\nabla_x \ln(\frac{1+e^{Z'x}}{e^{Z'x}S})] = 0$, and thus of

$$ \mathbb{E}[Z(h(Z'x) - S)] = 0. \quad (1) $$

Let $((R_n, S_n), n \ge 1)$ denote an i.i.d. sample of $(R, S)$ and for $k \in \{1, \cdots, p\}$, $\bar{R}_n^k$ denote the average of the sample $(R_1^k, \cdots, R_n^k)$ of $R^k$ and $(V_n^k)^2 = \frac{1}{n} \sum_{i=1}^n (R_i^k - \bar{R}_n^k)^2$ its variance (both computed recursively), $\bar{R}_n$ the vector $(\bar{R}_n^1 \cdots \bar{R}_n^p 0)'$ and $\Gamma_n$ the diagonal matrix with diagonal elements $\frac{1}{\sqrt{\frac{n}{n-1}V_n^1}}$, $\cdots, \frac{1}{\sqrt{\frac{n}{n-1}V_n^p}}$, $1$.

Assume that a mini-batch of $m_n$ new observations $(R_i, S_i)$ constituting an i.i.d sample of $(R, S)$ is taken into account at step $n$. Denote $M_n = \sum_{i=1}^n m_i$
and $I_n = \{M_{n-1} + 1, \dots, M_n\}$. Define for $j \in I_n$, $\tilde{Z}_j = \Gamma_{M_{n-1}}(R_j - \bar{R}_{M_{n-1}})$ the vector $R_j$ standardized with respect to estimations of the means and variances of the components of $R$ at step $n-1$. Recursively define the stochastic approximation process $(X_n, n \ge 1)$ and the averaged process $(\bar{X}_n, n \ge 1)$:

$$X_{n+1} = X_n - a_n \frac{1}{m_n} \sum_{j \in I_n} \tilde{Z}_j (h(\tilde{Z}'_j X_n) - S_j) \quad (2)$$

$$\bar{X}_{n+1} = \frac{1}{n+1} \sum_{i=1}^{n+1} X_i = \bar{X}_n - \frac{1}{n+1} (\bar{X}_n - X_{n+1}) \quad (3)$$

In the case of linear regression, the same type of process is used in [10] taking $h(u) = u$.

The following theorem is established for linear regression in [10] and for logistic regression in [11]. Assume:

(H1a) There is no affine relation between the components of $R$. (H1b) The moments of order 4 of $R$ exist.

$$(H2a) \quad a_n > 0, \quad \sum_{n=1}^{\infty} a_n = \infty, \quad \sum_{n=1}^{\infty} \frac{a_n}{\sqrt{n}} < \infty, \quad \sum_{n=1}^{\infty} a_n^2 < \infty.$$

**Theorem.** Under H1a, H1b and H2a, $(X_n, n \ge 1)$ and $(\bar{X}_n, n \ge 1)$ converge almost surely to $\theta$.

In [10] and [11], these processes were compared to others (with or without online standardization, and with or without averaging) on real or simulated data. Empirical results showed the interest of using online standardization of the data to avoid numerical explosions as well as the better performance of averaged processes using a piecewise constant step-size (see Section 3).

### 2.2.2. Updating Linear Regression Using All Observations up to the Current Step

Recursively define the stochastic approximation processes $(X_n, n \ge 1)$ and $(\bar{X}_n, n \ge 1)$:

$$X_{n+1} = X_n - a_n \frac{1}{M_n} \sum_{i=1}^{n} \sum_{j \in I_i} \tilde{Z}_j (\tilde{Z}'_j X_n - S_j), \quad \tilde{Z}_j = \Gamma_{M_n} (R_j - \bar{R}_{M_n}) \quad (4)$$

$$X_{n+1} = \frac{1}{n+1} \sum_{i=1}^{n+1} X_i = \bar{X}_n - \frac{1}{n+1} (\bar{X}_n - X_{n+1}) \quad (5)$$

Note that $\frac{1}{M_n} \sum_{i=1}^{n} \sum_{j \in I_i} \tilde{Z}_j \tilde{Z}'_j = \Gamma_{M_n} \left( \frac{1}{M_n} \sum_{i=1}^{n} \sum_{j \in I_i} R_j R'_j - \bar{R}_{M_n} \bar{R}'_{M_n} \right) \Gamma_{M_n}$ and $\frac{1}{M_n} \sum_{i=1}^{n} \sum_{j \in I_i} \tilde{Z}_j S_j = \Gamma_{M_n} \left( \frac{1}{M_n} \sum_{i=1}^{n} \sum_{j \in I_i} R_j S_j - \bar{R}_{M_n} \bar{S}_{M_n} \right)$, $\bar{S}_{M_n} = \frac{1}{M_n} \sum_{i=1}^{M_n} S_i$. Thus, the updating does not necessitate storing previous data since all empirical means and variances can be recursively computed. The same type of process would not be possible without storing the data for logistic regression, since in this case, $\tilde{Z}_j$ in $\tilde{Z}_j h(\tilde{Z}'_j X_n)$ should be updated for all $j$.

Denote by $\lambda_{\max}$ the largest eigenvalue of the covariance matrix of $R$. Assume:
$$ (\text{H2b}) \quad \left( a_n = a < \frac{1}{\lambda_{\max}} \right) \quad \text{or} \quad \left( a_n \to 0, \sum_1^\infty a_n = \infty \right). $$

**Theorem.** Under H1a, H1b and H2b, $(X_n, n \ge 1)$ and $(\bar{X}_n, n \ge 1)$ converge almost surely to $\theta$.

This theorem was also proven in [10]. Empirical results again showed the interest of using online standardization of the data as well as all observations up to the current step to avoid numerical explosions and to increase the speed of convergence.

It is therefore possible to use the processes described in this section to update the predictors by linear discriminant analysis and logistic regression in the ensemble score, taking into account the sample of new data generated by the Poisson bootstrap at each step for each predictor.

# 3. Empirical Study of Convergence

## 3.1. Material and Methods

### 3.1.1. Datasets

Four datasets available on the Internet and one dataset derived from the EPHESUS study [17] were used, all of which have previously been utilized to test the performance of stochastic approximation processes with online standardized data in the case of online linear regression [10] and online logistic regression [11]. The Twonorm, Ringnorm, Quantum and Adult datasets are commonly used to test classification methods. Twonorm and Ringnorm, introduced by Breiman [18], contain simulated data with homogeneous variables. Quantum contains observed “clean” data, without outliers and with most of its variables on a similar scale. Adult and HOSPHF30D contain observed data with outliers, as well as heterogeneous variables of different types and scales. A summary of these datasets is provided in Table 1.

The following preprocessing was performed on the data:

* Twonorm and Ringnorm: no preprocessing.

* Quantum: a stepwise variable selection (using AIC) was performed on the 6197 observations without any missing value. The dataset with complete observations for the 12 selected variables was used.

* Adult2: from the Adult dataset, modalities of several categorical variables were merged (in order to obtain a larger number of observations for each modality) and all categorical variables were then replaced by sets of binary variables, leading to a dataset with 38 variables.

* HOSPHF30D: 13 variables were selected using a stepwise selection.

From each dataset, a data stream was simulated step by step by randomly drawing, with replacement, 100 new observations at each step. Online scores were then constructed and updated from these data streams.

### 3.1.2. Reference Batch Score

For each dataset, a batch ensemble score was constructed using an adapted method from Duarte et al. [5] with the following parameters:
1) Two classification rules were used: linear discriminant analysis (LDA) and logistic regression (LR).

2) A total of 100 bootstrap samples were drawn for both rules (i.e. the same samples were used by each rule).

3) All available variables were included.

4) For each classification rule, the 100 associated predictors were aggregated by arithmetic mean and the coefficients subsequently normalized such that the score varied between 0 and 100 (as described in [5], Subsection 4.4.2).

5) The aggregation between the two intermediate scores $S_{LDA}$ and $S_{LR}$ was achieved by arithmetic mean: $S = \lambda S_{LDA} + (1 - \lambda) S_{LR}$ with $\lambda = 0.5$.

The score obtained for each dataset was used as a “gold standard” to assess the convergence of the tested online processes (Figure 1).

<table><caption>Table 1. Description of the datasets.</caption><thead><tr><th>Dataset</th><th>N<sub>a</sub></th><th>N</th><th>p<sub>a</sub></th><th>p</th><th>Source</th></tr></thead><tbody><tr><td>Twonorm</td><td>7400</td><td>7400</td><td>20</td><td>20</td><td>http://www.cs.toronto.edu/~delve/data/datasets.html</td></tr><tr><td>Ringnorm</td><td>7400</td><td>7400</td><td>20</td><td>20</td><td>http://www.cs.toronto.edu/~delve/data/datasets.html</td></tr><tr><td>Quantum</td><td>50,000</td><td>15,798</td><td>78</td><td>12</td><td>derived from https://www.kdd.org/kdd-cup/view/kdd-cup-2004/Data</td></tr><tr><td>Adult2</td><td>45,222</td><td>45,222</td><td>14</td><td>38</td><td>derived from http://www.cs.toronto.edu/~delve/data/datasets.html</td></tr><tr><td>HOSPHF30D</td><td>21,382</td><td>21,382</td><td>29</td><td>13</td><td>derived from EPHESUS study</td></tr></tbody></table>

$N_a$: number of available observations; $N$: number of selected observations; $p_a$: number of available parameters; $p$: number of selected parameters.

<figure><img src="image_4.png" /><figcaption>Figure 1. Methodology of construction and update of the online ensemble score.</figcaption></figure>
### 3.1.3. Tested Processes

**Types of processes:** Three different types of stochastic processes ($X_n$) were used as defined below.

1) “Classical” stochastic gradient (notation $C_{-\_}$). At step $n$, card $I_n = m_n$ observations ($R_j, S_j$) were taken into account and the process was updated recursively: $X_{n+1} = X_n - a_n \frac{1}{m_n} \sum_{j \in I_n} \tilde{Z}_j (h(\tilde{Z}'_j X_n) - S_j)$, with $\tilde{Z}_j$ the vector of standardized explanatory variables, $S_j \in \{0,1\}$, $h(u)=u$ for the LDA, and $h(u) = \frac{e^u}{1+e^u}$ for the LR.

2) “Averaged” stochastic gradient (notation $A_{-\_}$): $\bar{X}_{n+1} = \frac{1}{n+1} \sum_{i=1}^{n+1} X_i$.

3) Only in the case of the LDA: a process taking into account all of the previous observations ($R_j, S_j$) at each step until the current step, $j \in I_1 \cup \cdots \cup I_n$ (final mention “all”) [10]: $X_{n+1} = X_n - a_n \frac{1}{M_n} \sum_{i=1}^n \sum_{j \in I_i} \tilde{Z}_j (\tilde{Z}'_j X_n - S_j)$, $\tilde{Z}_j = \Gamma_{M_n} (R_j - \bar{R}_{M_n})$

In all cases, the explanatory variables were standardized online (notation $S_{-\_}$): the principle and practicality of this method to avoid numerical explosions have already been shown [10] [11]. Indeed, for some datasets (Adult2, HOSPHF30D), processes with raw data led to a numerical explosion, contrary to those with online standardized data.

**Step-size choice:** Tested step-sizes $a_n$ were either:

1) Continuously decreasing: $a_n = c/(b+n)^\alpha$ (notation $\_\_$V);

2) Constant: $a_n = 1/p$ (with $p$ the number of explanatory variables) (notation $\_\_$C);

3) Piecewise constant [12]: $a_n = c/\left(b+\left\lfloor\frac{n}{\tau}\right\rfloor\right)^\alpha$ ($\lfloor\cdot\rfloor$ being the integer part, $\tau$ the size of the level) (notation $\_\_$P).

In all cases, $\alpha = 2/3$ was taken as suggested by Xu [8] in the case of linear regression, $b=1$ and $c=1$.

**Tested processes:** Six couples of processes were tested (Table 2). The latter were among those which performed best in the studies published for online LDA [10] and for online LR [11], or represented “usual” processes frequently used (apart from online data standardization). A total of 100 new observations were used per step. Each process was applied to each of the streams generated from the datasets.

In the notation describing a couple of processes, the first term is for the LDA and the second for the LR. For example, AS100Call-AS100P200 is the couple formed using for the LDA an averaged process (A) with online standardization of the data (S), 100 new observations per step (100), constant step-size (C), taking into account all the observations up to the current step (all), and for the LR an averaged process (A) with online standardization of the data (S), 100 new
<table><caption>**Table 2.** List of the couples of processes studied.</caption><thead><tr><th>Couple</th><th></th><th>Process type</th><th>Step-size</th><th>Level size</th><th>Use of all observations until the current step</th></tr></thead><tbody><tr><td rowspan="2">AS100C-AS100P200</td><td>LDA process</td><td>Averaged</td><td>Constant</td><td>-</td><td>No</td></tr><tr><td>LR process</td><td>Averaged</td><td>Piecewise constant</td><td>200</td><td>No</td></tr><tr><td rowspan="2">AS100Call-AS100P200</td><td>LDA process</td><td>Averaged</td><td>Constant</td><td>-</td><td>Yes</td></tr><tr><td>LR process</td><td>Averaged</td><td>Piecewise constant</td><td>200</td><td>No</td></tr><tr><td rowspan="2">AS100P50-AS100P50</td><td>LDA process</td><td>Averaged</td><td>Piecewise constant</td><td>50</td><td>No</td></tr><tr><td>LR process</td><td>Averaged</td><td>Piecewise constant</td><td>50</td><td>No</td></tr><tr><td rowspan="2">AS100P50all-AS100P50</td><td>LDA process</td><td>Averaged</td><td>Piecewise constant</td><td>50</td><td>Yes</td></tr><tr><td>LR process</td><td>Averaged</td><td>Piecewise constant</td><td>50</td><td>No</td></tr><tr><td rowspan="2">CS100V-CS100V</td><td>LDA process</td><td>Classical</td><td>Continuously decreasing</td><td>-</td><td>No</td></tr><tr><td>LR process</td><td>Classical</td><td>Continuously decreasing</td><td>-</td><td>No</td></tr><tr><td rowspan="2">CS100Vall-CS100V</td><td>LDA process</td><td>Classical</td><td>Continuously decreasing</td><td>-</td><td>Yes</td></tr><tr><td>LR process</td><td>Classical</td><td>Continuously decreasing</td><td>-</td><td>No</td></tr></tbody></table>

All processes used online standardized data and 100 new observations per step.

observations per step (100) and a piecewise constant step-size with levels of size 200 (P200).

Note that the six couples of processes can be grouped in three pairs. In each pair, for the LDA part, one couple of processes uses 100 observations at each step and the other all observations up to the current step, the processes for the LR part being the same.

**Convergence criterion:** The convergence criterion used was the relative difference of the norms $\frac{\|\theta^b - \hat{\theta}_N\|}{\|\theta^b\|}$ between the $\theta^b$ vector of coefficients obtained for the batch score and the $\hat{\theta}_N$ vector of coefficients estimated by a process after $N$ iterations, the variables being standardized and the score being normalized to vary between 0 and 100 [5]. Convergence was considered to have occurred when the value of this criterion was less than the arbitrary threshold of 0.05. Three indicators were compared for each couple of processes: the criterion value for the synthetic score $S_{LDA}$ obtained by aggregating the LDAs, the criterion value for the synthetic score $S_{LR}$ obtained by aggregating the LRs, and the criterion value for the final score $S$.

### 3.1.4. Convergence and Stability Analyses

In order to study the empirical convergence of the process, an analysis using a total of $10N$ observations was performed for each couple of processes. Since 100 observations are introduced at each step, the number of iterations of the process is $N/10$. Due to the stochastic nature of the processes studied, some variability is expected in the results. In order to evaluate this variability, the entire analysis using $10N$ observations was replicated 50 times for each couple of processes and
for each dataset. The mean, standard deviation (SD) and relative standard devia-
tion (RSD), *i.e.* the standard deviation divided by the mean, of the criterion values
were studied for the intermediary and final scores. For each dataset, the average
of the criterion values of all couples of processes was also studied.

For each replication and each dataset, the performance of the couples of
processes were ranked from the best (lowest relative difference of the norms for
the final score S) to the worst (highest relative difference of the norms for S).
Thereafter, the mean rank of each couple and its associated standard deviation
over the 50 replications were computed, first by dataset, and finally over all da-
tasets.

To study the long-term convergence of the process, a single analysis using
100N observations was performed for each couple of processes. Again, for each
dataset, the values of the criterion for the intermediary and final scores were
studied, and the couples of processes were ranked from the best to the worst.
The mean rank over all datasets was used to compare the global performance of
the couples. All analyses were performed with R 3.6.2.

3.2. Results

3.2.1. Convergence and Stability Analysis for 10N Observations

When replicating each couple of processes 50 times, the mean criterion values
were lower than 0.05 for all couples of processes applied on Twonorm, Ring-
norm and Quantum datasets (Table 3). However, only three out of six couples
of processes converged for Adult2 (AS100C-AS100P200, AS100P50all-AS100P50
and AS100Call-AS100P200) as well as for HOSPHF30D (AS100P50-AS100P50,
AS100P50all-AS100P50 and AS100Call-AS100P200). Note that for Twonorm,
Ringnorm and Quantum, the maximum criterion values (not shown) for all
couples of processes were always lower than 0.05 (i.e. even the worst performing
processes still converged), whereas it was not the case for certain couples applied
on Adult2 and HOSPHF30D.

Generally, intermediate LDA scores had smaller mean values, i.e. a faster con-
vergence, than intermediate LR scores. However, the worst performing interme-
diary process was the LDA process AS100P50 applied on Adult2. In most cases,
the mean criterion value for the final S score was between those of the two in-
termediate scores S<sub>LDA</sub> and S<sub>LR</sub>. For some couples of processes applied on some
datasets (for instance AS100Call-AS100C on Adult2 or AS100C-AS100P200 on
HOSPHF30D), this led to a convergence towards the reference of the final score
while one of the intermediate scores had not yet converged according to the cri-
terion.

When studying the rankings of the couples of processes over the 50 replica-
tions, the best couple of processes overall was AS100P50all-AS100P50. This couple
was consistently among the three best couples, and had the best performance for
three datasets. Note that the three best couples of processes across all datasets
were those using all observations until the current step for the LDA intermediary
scores.
<table><caption>Table 3. Mean, standard deviation and relative standard deviation of the criterion after 50 replications.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Twonorm</th><th colspan="3">Ringnorm</th><th colspan="3">Quantum</th><th colspan="3">Adult2</th><th colspan="3">HOSPHF30D</th></tr><tr><th>Mean</th><th>SD</th><th>RSD</th><th>Mean</th><th>SD</th><th>RSD</th><th>Mean</th><th>SD</th><th>RSD</th><th>Mean</th><th>SD</th><th>RSD</th><th>Mean</th><th>SD</th><th>RSD</th></tr></thead><tbody><tr><td rowspan="3">AS100C-AS100P200</td><td>S<sub>LDA</sub></td><td>0.0042*</td><td>0.0007</td><td>15.7%</td><td>0.0066*</td><td>0.0012</td><td>18.4%</td><td>0.0116*</td><td>0.0069</td><td>59.8%</td><td>0.0542</td><td>0.0151</td><td>27.9%</td><td>0.0669</td><td>0.0398</td><td>59.5%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0026*</td><td>0.0004</td><td>15.2%</td><td>0.0074*</td><td>0.0014</td><td>18.7%</td><td>0.0104*</td><td>0.0059</td><td>56.3%</td><td>0.0224*</td><td>0.0138</td><td>61.9%</td><td>0.0488*</td><td>0.0375</td><td>76.7%</td></tr><tr><td>S</td><td>0.0028*</td><td>0.0004</td><td>14.6%</td><td>0.0070*</td><td>0.0013</td><td>18.6%</td><td>0.0108*</td><td>0.0064</td><td>59.4%</td><td>0.0339*</td><td>0.0164</td><td>48.3%</td><td>0.0529</td><td>0.0382</td><td>72.3%</td></tr><tr><td rowspan="3">AS100Call-AS100P200</td><td>S<sub>LDA</sub></td><td>0.0011*</td><td>0.0001</td><td>12.3%</td><td>0.0016*</td><td>0.0003</td><td>17.4%</td><td>0.0073*</td><td>0.0039</td><td>53.5%</td><td>0.0564</td><td>0.0182</td><td>32.2%</td><td>0.0444*</td><td>0.0329</td><td>74.1%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0026*</td><td>0.0004</td><td>15.2%</td><td>0.0074*</td><td>0.0014</td><td>18.7%</td><td>0.0104*</td><td>0.0059</td><td>56.3%</td><td>0.0224*</td><td>0.0138</td><td>61.9%</td><td>0.0488*</td><td>0.0375</td><td>76.7%</td></tr><tr><td>S</td><td>0.0016*</td><td>0.0002</td><td>14.2%</td><td>0.0038*</td><td>0.0007</td><td>17.5%</td><td>0.0075*</td><td>0.0039</td><td>51.5%</td><td>0.0350*</td><td>0.0171</td><td>48.7%</td><td>0.0434*</td><td>0.0320</td><td>73.7%</td></tr><tr><td rowspan="3">AS100P50-AS100P50</td><td>S<sub>LDA</sub></td><td>0.0079*</td><td>0.0012</td><td>14.8%</td><td>0.0127*</td><td>0.0026</td><td>20.1%</td><td>0.0121*</td><td>0.0072</td><td>59.8%</td><td>1.8030</td><td>0.8576</td><td>47.6%</td><td>0.0670</td><td>0.0403</td><td>60.1%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0021*</td><td>0.0002</td><td>12.0%</td><td>0.0046*</td><td>0.0008</td><td>17.7%</td><td>0.0091*</td><td>0.0054</td><td>59.3%</td><td>0.0371*</td><td>0.0179</td><td>48.2%</td><td>0.0458*</td><td>0.0319</td><td>69.8%</td></tr><tr><td>S</td><td>0.0042*</td><td>0.0006</td><td>13.4%</td><td>0.0080*</td><td>0.0016</td><td>20.0%</td><td>0.0101*</td><td>0.0063</td><td>62.4%</td><td>1.1174</td><td>0.5210</td><td>46.6%</td><td>0.0493*</td><td>0.0360</td><td>72.9%</td></tr><tr><td rowspan="3">AS100P50all-AS100P50</td><td>S<sub>LDA</sub></td><td>0.0011*</td><td>0.0001</td><td>12.3%</td><td>0.0016*</td><td>0.0003</td><td>17.5%</td><td>0.0073*</td><td>0.0039</td><td>53.6%</td><td>0.0175*</td><td>0.0099</td><td>56.8%</td><td>0.0444*</td><td>0.0329</td><td>74.4%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0021*</td><td>0.0002</td><td>12.0%</td><td>0.0046*</td><td>0.0008</td><td>17.7%</td><td>0.0091*</td><td>0.0054</td><td>59.3%</td><td>0.0371*</td><td>0.0179</td><td>48.2%</td><td>0.0458*</td><td>0.0319</td><td>69.8%</td></tr><tr><td>S</td><td>0.0014*</td><td>0.0002</td><td>12.1%</td><td>0.0027*</td><td>0.0005</td><td>17.1%</td><td>0.0071*</td><td>0.0037</td><td>52.1%</td><td>0.0205*</td><td>0.0123</td><td>59.7%</td><td>0.0434*</td><td>0.0287</td><td>66.1%</td></tr><tr><td rowspan="3">CS100V-CS100V</td><td>S<sub>LDA</sub></td><td>0.0021*</td><td>0.0004</td><td>17.6%</td><td>0.0033*</td><td>0.0005</td><td>16.6%</td><td>0.0206*</td><td>0.0122</td><td>59.3%</td><td>0.0910</td><td>0.0305</td><td>33.6%</td><td>0.0912</td><td>0.0624</td><td>68.4%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0045*</td><td>0.0003</td><td>7.5%</td><td>0.0016*</td><td>0.0003</td><td>16.3%</td><td>0.0382*</td><td>0.0044</td><td>11.5%</td><td>0.2329</td><td>0.0233</td><td>10.0%</td><td>0.1499</td><td>0.0353</td><td>23.5%</td></tr><tr><td>S</td><td>0.0026*</td><td>0.0003</td><td>10.3%</td><td>0.0022*</td><td>0.0004</td><td>16.5%</td><td>0.0240*</td><td>0.0075</td><td>31.3%</td><td>0.0554</td><td>0.0114</td><td>20.6%</td><td>0.0918</td><td>0.0506</td><td>55.1%</td></tr><tr><td rowspan="3">CS100Vall-CS100V</td><td>S<sub>LDA</sub></td><td>0.0012*</td><td>0.0002</td><td>13.3%</td><td>0.0016*</td><td>0.0003</td><td>17.8%</td><td>0.0074*</td><td>0.0038</td><td>51.4%</td><td>0.0978</td><td>0.0204</td><td>20.8%</td><td>0.0438*</td><td>0.0322</td><td>73.7%</td></tr><tr><td>S<sub>LR</sub></td><td>0.0045*</td><td>0.0003</td><td>7.5%</td><td>0.0016*</td><td>0.0003</td><td>16.3%</td><td>0.0382*</td><td>0.0044</td><td>11.5%</td><td>0.2329</td><td>0.0233</td><td>10.0%</td><td>0.1499</td><td>0.0353</td><td>23.5%</td></tr><tr><td>S</td><td>0.0024*</td><td>0.0002</td><td>9.9%</td><td>0.0016*</td><td>0.0003</td><td>17.3%</td><td>0.0206*</td><td>0.0039</td><td>19.0%</td><td>0.0536</td><td>0.0087</td><td>16.2%</td><td>0.0855</td><td>0.0371</td><td>43.4%</td></tr></tbody><tfoot><tr><td colspan="2">Average (for S scores)</td><td>0.0025</td><td>0.0003</td><td>12.4%</td><td>0.0042</td><td>0.0008</td><td>17.8%</td><td>0.0133</td><td>0.0053</td><td>45.9%</td><td>0.2193</td><td>0.0978</td><td>40.0%</td><td>0.0610</td><td>0.0371</td><td>63.9%</td></tr></tfoot></table>

The observed differences in the average criterion were greater between datasets rather than between couples of processes (Table 4). Indeed, the means of each couple of processes were the lowest for Twonorm and Ringnorm compared to the other datasets. Conversely, all couples had their worst results for HOSPHF30D. Generally, all couples of processes performed better when applied on simulated data (Twonorm and Ringnorm) rather than on observed data (Quantum, Adult2, HOSPHF30D). This was also true when comparing the standard deviations and RSDs.

When comparing the overall variability of the rankings between the couples, AS100P50all-AS100P50 and AS100Call-AS100P200, the two best performing couples of processes on average, also had the lowest standard deviations for the mean overall rank (1.17 and 1.12 respectively), while the couple with the largest standard deviation was CS100Vall-CS100V (1.72). It appears that the two best couples of processes consisted of averaged processes with a piecewise constant or a constant step-size for one process in the case of LDA. It also appears that these two processes in the case of LDA used all observations up to the current step instead of a mini-batch of observations.

<footnote>*denotes criteria values < 0.05. SD: standard deviation; RSD: relative standard deviation.</footnote>
### 3.2.2. Convergence Analysis for 100N Observations

When studying the couples of processes after using 100N observations (i.e. N iterations) in order to assess the “long-term” convergence, the final online S score was very similar (criterion value < 0.05) to the reference “batch” score for all of the couples on four of the five datasets tested (Table 5).

<table><caption>Table 4. Mean (SD) rank of the processes across the 50 replications, by dataset and overall (ordered by overall rank).</caption><thead><tr><th>Dataset</th><th><em>Twonorm</em></th><th><em>Ringnorm</em></th><th><em>Quantum</em></th><th><em>Adult2</em></th><th><em>HOSPHF30D</em></th><th><em>Overall</em></th></tr></thead><tbody><tr><td>AS100P50all-AS100P50</td><td>1.04 (0.20)</td><td>3.00 (0.00)</td><td>1.68 (0.98)</td><td>1.40 (0.81)</td><td>2.54 (1.62)</td><td><em>1.93 (1.17)</em></td></tr><tr><td>AS100Call-AS100P200</td><td>1.96 (0.20)</td><td>4.00 (0.00)</td><td>2.10 (0.84)</td><td>2.90 (1.15)</td><td>2.52 (1.27)</td><td><em>2.70 (1.12)</em></td></tr><tr><td>CS100Vall-CS100V</td><td>3.40 (0.61)</td><td>1.06 (0.24)</td><td>5.24 (0.56)</td><td>4.08 (1.01)</td><td>4.60 (1.63)</td><td><em>3.68 (1.72)</em></td></tr><tr><td>AS100C-AS100P200</td><td>4.42 (0.81)</td><td>5.06 (0.24)</td><td>3.64 (1.03)</td><td>2.56 (1.05)</td><td>3.56 (1.43)</td><td><em>3.85 (1.30)</em></td></tr><tr><td>CS100V-CS100V</td><td>4.18 (0.66)</td><td>1.94 (0.24)</td><td>5.58 (0.70)</td><td>4.06 (0.98)</td><td>4.66 (1.62)</td><td><em>4.08 (1.53)</em></td></tr><tr><td>AS100P50-AS100P50</td><td>6.00 (0.00)</td><td>5.94 (0.24)</td><td>2.76 (0.94)</td><td>6.00 (0.00)</td><td>3.12 (1.27)</td><td><em>4.76 (1.66)</em></td></tr></tbody></table>

<table><caption>Table 5. Criterion value after 100N observation used for intermediary and final scores.</caption><thead><tr><th>Processes</th><th></th><th><em>Twonorm</em></th><th><em>Ringnorm</em></th><th><em>Quantum</em></th><th><em>Adult2</em></th><th><em>HOSPHF30D</em></th><th><em>Mean rank</em></th></tr></thead><tbody><tr><td rowspan="3">AS100C-AS100P200</td><td><em>S</em><sub>LDA</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0028*</td><td>0.0066*</td><td>0.0165*</td><td rowspan="3">2.8</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0007*</td><td>0.0007*</td><td>0.0033*</td><td>0.0069*</td><td>0.0206*</td></tr><tr><td><em>S</em></td><td><strong>0.0006*</strong></td><td><strong>0.0007*</strong></td><td><strong>0.0030*</strong></td><td><strong>0.0067*</strong></td><td><strong>0.0190*</strong></td></tr><tr><td rowspan="3">AS100Call-AS100P200</td><td><em>S</em><sub>LDA</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0046*</td><td>0.0153*</td><td>0.0060*</td><td rowspan="3">2.4</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0007*</td><td>0.0007*</td><td>0.0033*</td><td>0.0069*</td><td>0.0206*</td></tr><tr><td><em>S</em></td><td><strong>0.0005*</strong></td><td><strong>0.0007*</strong></td><td><strong>0.0039*</strong></td><td><strong>0.0120*</strong></td><td><strong>0.0149*</strong></td></tr><tr><td rowspan="3">AS100P50-AS100P50</td><td><em>S</em><sub>LDA</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0027*</td><td>2.756</td><td>0.0176*</td><td rowspan="3">3.4</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0032*</td><td>0.0346*</td><td>0.0203*</td></tr><tr><td><em>S</em></td><td><strong>0.0005*</strong></td><td><strong>0.0007*</strong></td><td><strong>0.0029*</strong></td><td><strong>1.6968</strong></td><td><strong>0.0192*</strong></td></tr><tr><td rowspan="3">AS100P50all-AS100P50</td><td><em>S</em><sub>LDA</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0046*</td><td>0.0100*</td><td>0.0060*</td><td rowspan="3">1.8</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0006*</td><td>0.0007*</td><td>0.0032*</td><td>0.0346*</td><td>0.0203*</td></tr><tr><td><em>S</em></td><td><strong>0.0005*</strong></td><td><strong>0.0007*</strong></td><td><strong>0.0039*</strong></td><td><strong>0.0193*</strong></td><td><strong>0.0147*</strong></td></tr><tr><td rowspan="3">CS100V-CS100V</td><td><em>S</em><sub>LDA</sub></td><td>0.0010*</td><td>0.0020*</td><td>0.0073*</td><td>0.0076*</td><td>0.0165*</td><td rowspan="3">5.2</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0033*</td><td>0.0009*</td><td>0.0168*</td><td>0.1002</td><td>0.0566</td></tr><tr><td><em>S</em></td><td><strong>0.0015*</strong></td><td><strong>0.0014*</strong></td><td><strong>0.0083*</strong></td><td><strong>0.0414*</strong></td><td><strong>0.0289*</strong></td></tr><tr><td rowspan="3">CS100Vall-CS100V</td><td><em>S</em><sub>LDA</sub></td><td>0.0005*</td><td>0.0006*</td><td>0.0033*</td><td>0.0287*</td><td>0.0153*</td><td rowspan="3">5.4</td></tr><tr><td><em>S</em><sub>LR</sub></td><td>0.0033*</td><td>0.0009*</td><td>0.0168*</td><td>0.1002</td><td>0.0566</td></tr><tr><td><em>S</em></td><td><strong>0.0017*</strong></td><td><strong>0.0007*</strong></td><td><strong>0.0090*</strong></td><td><strong>0.0281*</strong></td><td><strong>0.0290*</strong></td></tr><tr><td colspan="2"><em>Average (for S scores)</em></td><td><strong>0.0009</strong></td><td><strong>0.0008</strong></td><td><strong>0.0052</strong></td><td><strong>0.3007</strong></td><td><strong>0.0210</strong></td><td></td></tr></tbody></table>

<footnote>*denote criteria values < 0.05. *First abbreviation*: LDA process; *Second abbreviation*: LR process. *Type of processes*: C for classical SGD, A for ASGD. *Data*: R for raw data, S for online standardization of the data (1st number: number of new data per step). *Step-size*: V for continuously decreasing, C for constant, P for piecewise constant (2nd number: size of the steps of the piecewise constant step-size).</footnote>
Only the AS100P50-AS100P50 couple applied to the Adult2 dataset did not converge after 100N iterations (criterion = 1.697). More precisely, the result for the LDA part of this couple differed substantially from its batch counterpart (criterion = 2.756), whereas the LR part appeared to converge to the batch LR part (criterion = 0.035).

For each couple of processes, the best performances were achieved for the Twonorm and Ringnorm datasets, which consist of simulated data. The worst performances were obtained for Adult2 and HOSPHF30D datasets, which contain observed data.

Although these results are not directly comparable with the average results using 10N observations presented in the previous subsection (since there was only one replication using 100N observations), it should be noted that the criterion values of all couples of processes on all datasets were lower after 100N observations than the mean values after 10N observations, except for the LDA and global scores of AS100P50-AS100P50 applied on Adult2.

When the couples of processes were ranked from best to worst for each dataset and the average ranks were calculated across all datasets (Table 5), the two worst performing couples were CS100Vall-CS100V and CS100V-CS100V, i.e. the only two couples using classical processes and a continuously decreasing step-size. The best couple was again AS100P50all-AS100P50. It appears that the two best couples of processes were the same as for 10N observations and that the two worst couples were classical processes with a continuously decreasing step-size.

## 4. Conclusions

This study presented the construction of an online ensemble score obtained by aggregation of two rules of classification, LDA and LR, and bagging. The online ensemble score was constructed by using Poisson bootstrap and by associating stochastic approximation processes with online standardized data of different types, averaged or not, using either a mini-batch of data at each step or all observations up to the current step in the case of LDA, and different choices of step-sizes, whose convergence has already been theoretically established. The convergence of this overall online score towards the “batch” score was studied empirically. It appears that the two best processes were averaged processes with a piecewise constant step-size or a constant step-size for one process in the case of LDA and with the use of all observations up to the current step in the case of LDA instead of a mini-batch of observations. Thus, these were not the classical processes with a continuously decreasing step-size and a mini-batch of observations at each step in the case of LDA.

This study can be extended in several directions. More than two models of classification could be taken into account. Other classification models could be used such as the probit model. Other experiments could be carried out using randomly selected variables with different modalities of random selection [5] [6].
This study can also be extended to the regression framework when $y$ is a continuous variable.

## Acknowledgements

The authors thank Mr. Pierre Pothier for editing this manuscript.

## Funding

This work was supported by the investments for the Future Program under grant ANR-15-RHU-0004.

## Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

## References

[1] Genuer, R. and Poggi, J.M. (2017) Arbres CART et Forêts aléatoires, Importance et sélection de variables. https://hal.archives-ouvertes.fr/hal-01387654

[2] Breiman, L. (1996) Bagging Predictors. Machine Learning, 24, 123-140. https://doi.org/10.1007/BF00058655

[3] Freund, Y. and Schapire, R.E. (1996) Experiments with a New Boosting Algorithm. Proceedings of the Thirteenth International Conference on Machine Learning, 148-156.

[4] Song, L., Langfelder, P. and Horvath, S. (2013) Random Generalized Linear Model: A Highly Accurate and Interpretable Ensemble Predictor. BMC Bioinformatics, 14, Article No. 5. https://doi.org/10.1186/1471-2105-14-5

[5] Duarte, K., Monnez, J.M. and Albuisson, E. (2018) Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients. Applied Mathematics, 9, 954-974. https://doi.org/10.4236/am.2018.98065

[6] Lalloué, B., Monnez, J.M., Lucci, D. and Albuisson, E. (2021) Construction of Par- simonious Event Risk Scores by an Ensemble Method. An Illustration for Short- Term Predictions in Chronic Heart Failure Patients from the GISSI-HF Trial. Applied Mathematics, 12, 627-653. https://doi.org/10.4236/am.2021.127045

[7] Ljung, L., Pflug, G.C. and Walk, H. (1992) Stochastic Approximation and Optimiza- tion of Random Systems. Birkhäuser, Basel. https://doi.org/10.1007/978-3-0348-8609-3

[8] Xu, W. (2011) Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent. ArXiv11072490 Cs.

[9] Kek, S.L., Sim, S.Y, Leong, W.J. and Teo, K.L. (2018) Discrete-Time Nonlinear Sto- chastic Optimal Control Problem Based on Stochastic Approximation Approach. Ad- vances in Pure Mathematics, 8, 232-244. https://doi.org/10.4236/apm.2018.83012

[10] Duarte, K., Monnez, J.M. and Albuisson, E. (2018) Sequential Linear Regression with Online Standardized Data. PLoS ONE, 13, e0191186. https://doi.org/10.1371/journal.pone.0191186

[11] Lalloué, B., Monnez, J.M. and Albuisson, E. (2021) Streaming Constrained Binary Logistic Regression with Online Standardized Data. Journal of Applied Statistics.
https://doi.org/10.1080/02664763.2020.1870672

[12] Bach, F. (2014) Adaptivity of Averaged Stochastic Gradient Descent to Local Strong Convexity for Logistic Regression. *Journal of Machine Learning Research*, **15**, 595-627.

[13] Lalloué, B., Monnez, J.M. and Albuisson, E. (2019) Actualisation en ligne d'un score d'ensemble. *51 e Journées de Statistique*, Nancy, France, Jun 2019.
https://hal.archives-ouvertes.fr/hal-02152352

[14] Oza, N.C. and Russell, S.J. (2001) Online Bagging and Boosting. *Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics*, Key West, Florida, USA, 4-7 January 2001, 229-236.

[15] Cardot, H., Cénac, P. and Monnez, J.M. (2012) A Fast and Recursive Algorithm for Clustering Large Datasets with κ-Medians. *Computational Statistics & Data Analysis*, **56**, 1434-1449. https://doi.org/10.1016/j.csda.2011.11.019

[16] Monnez, J.M. and Skiredj, A. (2021) Widening the Scope of an Eigenvector Stochastic Approximation Process and Application to Streaming PCA and Related Methods. *Journal of Multivariate Analysis*, **182**, Article ID: 104694.
https://doi.org/10.1016/j.jmva.2020.104694

[17] Pitt, B., Remme, W., Zannad, F., Neaton, J., Martinez, F., Roniker, B., et al. (2003) Eplerenone, a Selective Aldosterone Blocker, in Patients with Left Ventricular Dysfunction after Myocardial Infarction. *The New England Journal of Medicine*, **348**, 1309-1321. https://doi.org/10.1056/NEJMoa030207

[18] Breiman, L. (1996) Bias, Variance, and Arcing Classifiers. Technical Report 460, University of California, Berkeley.