# Prediction model for rare events in longitudinal follow-up and resampling methods

Mathieu Berthe, Pierre Druilhet, Stéphanie Léger

► To cite this version:

Mathieu Berthe, Pierre Druilhet, Stéphanie Léger. Prediction model for rare events in longitudinal follow-up and resampling methods. 2025. hal-04919828

HAL Id: hal-04919828

https://hal.science/hal-04919828v1

Preprint submitted on 29 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Prediction model for rare events in longitudinal follow-up and
resampling methods

Mathieu Berthe ¹ & Pierre Druilhet ² & Stéphanie Léger ³

¹,²,³ Université Clermont Auvergne

Laboratoire de Mathématiques Blaise Pascal UMR 6620 - CNRS

Campus des Cézeaux

3, Place Vasarely

TSA 60026 CS 60026

63178 Aubière Cedex

¹ Mathieu.Berthe@math.univ-bpclermont.fr

² Pierre.Druilhet@math.univ-bpclermont.fr

³ Stephanie.Leger@math.univ-bpclermont.fr

Résumé

We consider the problem of model building for rare events prediction in longi-
tudinal follow-up studies. In this paper, we compare several resampling methods to
improve standard regression models on a real life example. We evaluate the effect
of the sampling rate on the predictive performances of the models. To evaluate the
predictive performance of a longitudinal model, we consider a validation technique
that takes into account time and corresponds to the actual use in real life.

**Keywords** : Rare events, longitudinal follow-up, oversampling, undersampling, SMOTE, ensemble-based methods, logistic regression.

# 1 Introduction

Prediction models for rare events appears in many research fields such as economic (Burez and Van den Poel, 2008), politics (King and Zeng, 2002), fraud detection (Bolton and Hand, 2002) or bank regulation (Calabrese and Osmetti, 2015). Modeling and predicting binary rare events present several difficulties. Strong imbalance between event and non-events induce biased estimations and poor predictive performances, usually underestimating the probability of event occurrences. In recent years, several strategies have been proposed to improve misclassification. For example, King and Zeng (2002) propose an explanatory logistic regression model with bias correction in a case-control study. Calabrese and Osmetti (2013) have developed a new regression model based on extreme value theory. More recently, Nuñez et al. (2017) improve the learning function in SVM by a low-cost post-processing strategy.
Another way to the improve predictive performance of a model with rare events is to rebalance artificially the dataset by resampling methods. For example, oversampling methods creates artificially new observations in the minority class, whereas undersampling methods delete observations in the majority class. Hybrid methods combine both oversampling and undersampling methods.

The choice of resampling rate, that is the final ratio between events and non-events, is a crucial point to improve predictive performance of the model. It is known that the optimal rate is highly dependent on the dataset (Batista et al., 2004; Zhu et al., 2017). Furthermore, resampling methods induce additional randomness in the dataset. The most common way to reduce this extra-variability is to use aggregation methods (Breiman, 1996). Other strategies to improve classifiers with rare events have been considered, such as weighting training instances (Pazzani et al., 1994) or using different misclassification costs for minority and majority events (Spears and Perlis, 1989).

The aim of the paper is to compare several resampling and aggregation methods on a real-life longitudinal follow-up study. We discuss the way to evaluate predictive performance in the case of longitudinal studies and then choose the optimal sampling rate adapted to our data set.

In Section 2, we review resampling and ensemble based methods. We also discuss the way to evaluate the predictive performance adapted to longitudinal follow-up. In Section 3, we compare several strategies applied to a real life example : we have followed a soccer teams during one year and we aimed to evaluate the risk of muscle injury before each match. We discuss the crucial choice of the sampling rate and the effect of aggregation methods. We also show that SMOTE methods (Chawla et al., 2002) applied to our dataset performs poorly.

## 2 Prediction models and sampling methods

In this section, we present several resampling methods combined with aggregation to improve the predictive performance of a logistic regression.

### 2.1 Standard logistic regression

Here, we recall the bases of logistic regression. For an individual $i$ ($i = 1, ..., n$), let $x_i$ be the $k+1$-vector of the $k$ explanatory variables plus the constant and let $y_i \in \{0, 1\}$ be the binary response which follows a Bernoulli distribution with parameter $\pi_i = P(y_i = 1 | x_i)$. In the standard logistic regression, it is assumed that

$$ \pi_i = \frac{1}{1 + e^{-x_i \beta}} \qquad (1) $$

where $x'$ is the transpose of $x$ and $\beta' = (\beta_0, \beta_1, ..., \beta_k)$ is the vector of unknown parameters, usually estimated by maximum likelihood (see e.g. McCullagh and Nelder (1989)). The
asymptotic variance of $\hat{\beta}$ is

$$V(\hat{\beta}) = \left[ \sum_{i=1}^{n} \pi_i (1 - \pi_i) \mathbf{x}_i \mathbf{x}_i' \right]^{-1} \quad (2)$$

For a new individual $x$, the probability of the event $y = 1$ is predicted by

$$\hat{\pi}(x) = \frac{1}{1 + e^{-x_i' \hat{\beta}}} = \hat{P}(y = 1 | x)$$

When the dataset contains few events, say less than 5%, it is known that logistic regression underestimate the probability of events and then poor predictive performances (see (King and Zeng, 2002))

## 2.2 Balancing unbalanced dataset

To overcome drawbacks induced by the unbalanced datasets, several sampling methods can be used to artificially rebalance the dataset. Several resampling methods on real data are compared in Batuwita and Palade (2010); Batista et al. (2004); Liu et al. (2009); Zhu et al. (2017). Drummond and Holte (2003) show that oversampling is better than undersampling and Japkowicz (2000) that random oversampling or undersampling methods improve substantially the predictive performance of the models so that more sophisticated oversampling or down-sizing methods approaches appear unnecessary. All these studies show that the best resampling method is highly dependent on the dataset.

In this section, we review the most common sampling methods, which can be used alone or combined.

### 2.2.1 Undersampling methods

The first way to rebalance an unbalanced dataset is to reduce the number of observations in the majority class (non-events). A random undersampling with rate $r$, $0 < r < 1$, creates a new dataset by removing at random from the initial dataset a proportion $r$ of observations from the majority class. If $r = 0$, then all the observations of the majority class are kept. If $r = 0.7$, then 70% of the observations of the majority class are removed.

In the case of very rare events, King and Zeng (2002) propose to use case-control designs (see also Breslow, 1996). This strategy is equivalent to selecting randomly one non-event for every event, resulting in a completely balanced dataset. In that case, if the proportion of events is $p$, then the rate of the undersampling is $r = (1-2p)/(1-p)$. Another more sophisticated strategy has been proposed in Tomek (1976) : for each event, the idea is to remove a non-event that forms a Tomek link. Kubat (2000) considers situations where Tomek link methods do not guarantee a performance gain.
The main drawback of undersampling methods is the loss of information when the number of removed observations is large. In Section 2.3, we consider aggregated methods that limit this loss of information.

## 2.2.2 Oversampling methods

At the opposite of undersampling methods, oversampling methods increase artificially the number of observations in the minority class (events). A random oversampling with rate (a:b) creates new observations by duplicating at random observations in the minority class until there are a non-events for b events in the new dataset. An oversampling (1:1) results in a completely balanced dataset. An oversampling (2:1) results in a dataset with 2 non-events for 1 event.

SMOTE (Chawla et al., 2002) is a more sophisticated method that creates synthetic observations in the minority class as follows : for each event observation, choose at random one of the *k* nearest neighbors that belongs to the minority class, with *k* fixed. The new synthetic observation is chosen at random between these two observations. It is also possible to reiterate the process to increase the oversampling rate. Figure 1 shows the effect of SMOTE with *k* = 2 and with one synthetic observation generated by events.

## 2.2.3 Hybrid sampling

It is known that these methods have some cons. Random undersampling can discard potentially useful data, whereas random oversampling creates exact copies of existing instances that may induce overfitting. To overcome these features, a solution is to mix undersampling and oversampling methods. For example, a random undersampling method with rate *c* combined with a (*a*:*b*)-oversampling method consists in removing at random a proportion *c* of non-event and then perform an oversampling to obtain *a* non-events for *b* events.

<figure><img src="image_2.png" /><figcaption>FIGURE 1 – SMOTE : for each event, one synthetic observation is created from one of the two nearest neighbors at random.</figcaption></figure>

As a remark, random over/under sampling methods can be seen as weighted logistic regressions (Manski and Lerman, 1977) where the weights are random. For the resampled dataset, the log-likelihood of the logistic regression can be written :
$$ \ln L_w(\beta|\mathbf{y}) = \sum_{i/y_i=1} w_i \ln(\pi_i) + \sum_{i/y_i=0} w_i \ln(1-\pi_i) $$

where the weight $w_i$ is the number of replication of $x_i$ for $y_i = 1$ in the random oversampling process and $w_i = 0/1$ for $y_i = 0$ in the random undersampling process.

## 2.3 Ensemble-based methods

Each sampling method described above induces a supplementary part of randomness in the dataset and therefore more variability in the predictions. Ensemble-based methods are the most common way to reduce this variability. The idea is to create K datasets from the same resampling scheme and to aggregate the predictors. Therefore, for a new individual i with covariate $x_i$, the predicted probability of event $\hat{\pi}_i$ is given by

$$ \hat{\pi}_i = \frac{1}{K} \sum_{k=1}^{K} \hat{\pi}_i^{[k]}, $$

where $\hat{\pi}_i^{[k]}$ is the predictor obtained from the k$^{th}$ dataset. The choice of K will be discussed in Section 3.2.3.

As a variant, when using a pure oversampling methods, the non-events may be replaced by K bootstrap samples, similarly to Bagging (Breiman, 1996). In the same way, when using a pure oversampling method, the events may be replaced by a bootstrap sample of them. The effects of this bootstrap variant on the aggregated predictors are displayed in Table 2.

## 2.4 Predictive performance evaluation in longitudinal follow-up

To evaluate the predictive performance of a model, training and test datasets should be chosen carefully. In longitudinal follow-up studies, events are highly dependent on the past and change the future. In this context, it is impossible to use standard validation strategies like cross validation or random split of the dataset into learning and test datasets. Indeed, with this strategy, the risk is to confuse causes and consequences and to overestimate predictive performances. Therefore, it is more natural to use a longitudinal strategy (see Fig. 2) that corresponds to the way the models are used in real life : at time t, we only use previous information to predict the risk $\hat{\pi}_{ti}$ to have an event on the individual i, then we compare our prediction with the real observation $y_{ti}$. At the end, we have a collection of $(\hat{\pi}_{tj}, y_{tj})$, $t = 1, ..., T$ and $i = 1, ..., I$.

The usual way to compare the ability of several models to predict a binary response is to compare their ROC curves, AUCs or Peirce indices. We recall that a ROC curve is a parametric curve defined as follows : for a given threshold $0 \le \gamma \le 1$, we predict $y_{ti}$ by
<figure><img src="image_3.png" /><figcaption>FIGURE 2 – Longitudinal validation.</figcaption></figure>

$\hat{y}_{ti} = 0$ if $\hat{\pi}_{ti} < \gamma$ and $\hat{y}_{ti} = 1$ if $\hat{\pi}_{ti} \ge \gamma$. Then, we compare the predicted response $\hat{y}_{ti}$ with the real outcome $y_{ti}$. The sensitivity and the specificity, which depend on $\gamma$ are defined by

$$ \text{sensitivity}(\gamma) = \frac{TP}{TP + FN} \qquad \text{specificity}(\gamma) = \frac{TN}{TN + FN} $$

with $TN$, $TP$, $FN$, $FP$ are the number of true negative, true positive, false negative, false positive. For example $TP = \#\{y_{ti} = 1, \hat{y}_{ti} = 1\}$. The ROC curve is therefore the parametric curve $\{(\text{sensitivity}(\gamma), 1 - \text{specificity}(\gamma)) ; 0 \le \gamma \le 1\}$. As shown in Raeder et al. (2012), the choice of an evaluation metric plays an important role in learning on unbalanced data. From the ROC curve, we can derived two global metrics : the area under ROC curve (AUC) and the Pierce index (PI) defined by

$$ \text{PI} = \max_{\gamma \in [0,1]} \{\text{sensitivity}(\gamma) + \text{specificity}(\gamma) - 1\}. $$

which is particularly adapted to rare events.

The Pearce index represents a good compromise between sensitivity and specificity. It can be shown that $PI = 1 - d^*$, where $d^*$ is the Manhattan distance between the point (0,1) and its closest point on the ROC curve. It is also the euclidean distance between the further point on the ROC curve from the diagonal, up to a factor $\sqrt{2}$. The model with the highest AUC or PI will be considered as the best predictive model.

## 3 Comparison of resampling methods in a real life longitudinal follow-up.

In this section, we apply and compare the methods described in Section 2 in a real life situation. We have followed a soccer teams of the french Ligue 1 Championship during the season 2018-2019. We aim to build a model that evaluate the individual risk of non-contact muscle injury for each player before each match. To build the model, we use
the data collected during the seasons 2015-2018 and the season 2018 until the match.
A review of football player injury prediction methods can be found in Eetvelde et al.
(2021). Several predictive methods are compared in Carey DL et al. (2018). From Daniel
and Javier (2017), the average incidence of muscle injuries for a player during a match is
about 4%. In our dataset we observe a similar rate, so that non-contact muscle injuries
are considered as rare events.

To evaluate the predictive performances of the model, we use the longitudinal valida-
tion described in Section 2.4. Before each match, we predict the risk of muscle injury for
each player *i* based on all preceding observations. Then, we and compare the prediction
with the real outcomes, that is, muscle injury or not of player *i* during the match. Of
course, players that do not play the match are not considered.

During the season, 50 matches had been played and 16 non-contact muscle injuries
have been observed. To train the model before the first match, we use the data collected
during the seasons 2015-2018. Then, iteratively, we use the data collected until the day
before each match of the season 2018-2019 to predict the probability of injury for the next
match.

**3.1 The dataset**

The dataset include 42 soccer players on which data are collected daily and during
matches. After each match, the response variable is observed : *y* = 1 if an injury is
observed and *y* = 0 otherwise. For each player, we have the following covariates that are
considered in the literature as risk factors.

- Cumulative workload during training and matches over 21 days.

- Cumulative playing time over 21 days.

- Recovery time : number of days since the last match.

- Risk of relapse : ratio between the number of days disability due to injury and the average number of days of disability in the team. It aims to quantify the risk of relapse after an injury.

- Acceleration ratio : ratio between the number of accelerations performed over the 7 days preceding the match and the number of accelerations performed on the 21 days preceding the match

- Deceleration ratio : ratio between the number of deceleration performed over the 7 days preceding the match and the number of deceleration performed on the 21 days preceding the match

- Speed ratio : ratio between the average speed over the 7 days preceding the match and the average speed over the 21 days preceding the match.

- Player ID : player identifier.

Workload, Cumulative playing time and Recovery time allow to quantify player activity.
Acceleration, deceleration and speed ratio are used to assess the player sport performance
before the match. Another important covariate is the player ID. In usual longitudinal
studies, the aim is to extrapolate the model on other individuals. Therefore individuals
(here, players) are considered as random effects. In our case, we want to predict future
observations on the individuals that are included in the studies. Therefore, players are
considered as fixed effect, allowing to personalize the risk of injury. We will not consi-
der interaction between factors, since they have not shown, in preliminary studies, any
improvement of the predictive ability of the models, mainly due to overfitting.

## 3.2 Comparison of resampling methods

In this section, we compare the predictive performance of several resampling stra-
tegies applied to logistic regression. The performances metrics are evaluated on the 50
matches played during the season 2018-2019, by using the longitudinal validation des-
cribed in Section 2.4. Several resampling methods are evaluated : undersampling alone,
undersampling + bootstrap on events, oversampling alone, oversampling + bootstrap on
the events, both oversampling and undersampling. When several sampling strategies are
combined, we first use undersampling, then oversampling or SMOTE.

3.2.1 Effect of sampling rates on predictive performances

Here, we evaluate the effect of the balancing rate on AUC in random oversampling, SMOTE and random undersampling methods applied to logistic regression. The results are displayed in Fig. 3. Each method is run 15 times. Then, we compute the average AUC over the runs. We also compute the standard deviation for both metric. Note that the initial dataset imbalance is (25:1), that is 25 non-events for 1 event.

For random oversampling (Fig. 3.a, red line), the average AUC increases from 0.72 to 0.77 when the sampling rate goes from (25 : 1) to or (5 : 3). Then, the AUC decreases, probably due to an overfit on the events. For SMOTE (Fig. 3.a, blue line) the effect on AUC is always negative. This is mainly due to events that are isolated in the covariate space and therefore create synthetic events in the middle of non-events : for example, in Fig. 1 two isolated events on the left induce two synthetic events in the middle of a cluster of non-events.

In Fig. 3.b, we can see that undersampling methods slightly improve the average AUC for an undersampling rate between 0.2 and 0.3 with a AUC gain about 0.03. When the sampling rate is too large, say greater than 0.7 for our dataset, the predictive performance of the model worsen since too many non-event individuals are removed.

3.2.2 Comparison of several pure and hybrid resampling methods

Here, we compare random oversampling and undersampling methods studied in Section
3.2.1 with hybrid methods, SMOTE or plain logistic regression. Again, for each strategy,
<figure><img src="image_4.png" /><figcaption>FIGURE 3 – Average AUC over 15 runs with twice the standard deviation against sampling rates.</figcaption></figure>

<table><caption>TABLE 1 – Comparison of average AUC and Peirce index with standard deviation (std) for several resampling methods.</caption><thead><tr><th>Undersampling<sup>(1)</sup></th><th>Oversampling<sup>(2)</sup></th><th>SMOTE<sup>(3)</sup></th><th>AUC (std)</th><th>Peirce<sup>(4)</sup> (std)</th><th>Sensitivity</th><th>Specificity</th></tr></thead><tbody><tr><td>None</td><td>None</td><td>None</td><td>0.72</td><td>0.51</td><td>0.75</td><td>0.76</td></tr><tr><td>None</td><td>5:3</td><td>None</td><td>0.78 (0.007)</td><td>0.560 (0.056)</td><td>0.81</td><td>0.75</td></tr><tr><td>None</td><td>1:1</td><td>None</td><td>0.75 (0.005)</td><td>0.551 (0.026)</td><td>0.81</td><td>0.73</td></tr><tr><td>0.3</td><td>None</td><td>None</td><td>0.75 (0.063)</td><td>0.518 (0.051)</td><td>0.75</td><td>0.76</td></tr><tr><td>0.5</td><td>None</td><td>None</td><td>0.74 (0.046)</td><td>0.536 (0.036)</td><td>0.81</td><td>0.72</td></tr><tr><td>0.3</td><td>5:3</td><td>None</td><td>0.78 (0.009)</td><td>0.547 (0.028)</td><td>0.81</td><td>0.74</td></tr><tr><td>0.5</td><td>1:1</td><td>None</td><td>0.77 (0.013)</td><td>0.536 (0.037)</td><td>0.81</td><td>0.73</td></tr><tr><td>None</td><td>None</td><td>10:3</td><td>0.62 (0.038)</td><td>0.32 (0.082)</td><td>0.81</td><td>0.70</td></tr><tr><td>0.4</td><td>None</td><td>10:3</td><td>0.64 (0.047)</td><td>0.34 (0.064)</td><td>0.81</td><td>0.53</td></tr></tbody><tfoot><tr><td colspan="7">(1) : Oversampling rate<br>(2) : Undersampling rate<br>(3) : SMOTE oversampling rate<br>(4) : Peirce index.</td></tr></tfoot></table>

we run 15 times the model. So, we obtain an average AUC and Peirce index with related standard deviations. Sensitivities and specificities are calculated for the run whose Peirce index is the closest to the average. The results are displayed in Table 1 and for some models, ROC curves are displayed in Fig. 4.

The plain logistic regression, i.e. without additional resampling method, has an AUC equal to 0.72 and a Peirce index equal to 0.510 with a sensitivity 0.75 and a specificity 0.76. Random oversampling improves the prediction performance for a large range of sampling rates. For example, an oversampling rate of (5:3) gives average AUC and Peirce equal to 0.78 and 0.56, the sensibility increases to 0.75 whereas the specificity slightly decreases from to 0.75. As already seen in Section 3.2.1, SMOTE methods give poor results and undersampling should be used with caution, only with a small removal rate.

In conclusion, for our dataset, the resampling methods with highest AUC and Peirce
index are pure random oversampling (5:3) followed by hybrid undersampling 0.3 / oversampling (5:3). Note that the second method has a slightly lower average Peirce index (0.547) for the same average AUC.

<figure><img src="image_5.png" /><figcaption>FIGURE 4 – Comparison of ROC curves for several resampling methods with aggregation.</figcaption></figure>

### 3.2.3 Ensemble-based methods

Resampling methods add randomness in the output. Ensemble based methods, described in section 2.3, aim to stabilize the model and in some situations to improve the predictive performance, similarly to Bagging methods. There is no consensus about the right number of aggregations, which is usually between 20 and 100 for Bagging methods (Breiman, 1996; Bühlmann and Yu, 2002), depending on the dataset.

To evaluate the effect of the number of aggregations needed to stabilize the prediction for our dataset, we display, in Fig. 5, AUC and Peirce index against the number of aggregations for two of the best models obtained in Section 3.2.2: the first one is an hybrid undersampling with $r = 0.3$ and overampling (5:3) and the second one is an undersampling with $r = 0.5$ combined with a bootstrap sampling on the events. For the two models, AUC is stabilized after 20 iterations (Fig. 5.a and 5.b) whereas Peirce index needs more iterations to be stabilized (Fig. 5.c and 5.d).

To save computer time, we now compare ensemble-based methods with 20 iterations for the models used in Section 3.2.2. The results are displayed in Table 2, line 1-6, whose means and standard deviations of AUC and Peirce index, sensitivity and specificity are obtained in the same way as in Table 1. We omit SMOTE methods that have shown poor results.

It can be observed that the main effect of aggregation methods is to reduce the variability of AUC and Peirce index. For example, for random undersampling with rate 0.3, the standard deviation of AUC decreases from 0.063 to 0.005. For random oversampling
(5:3) it decreases from 0.007 to 0.002. The effects of aggregation methods on the mean AUC and Peirce index depend on the resampling methods. For undersampling, aggregation methods improve slightly the average AUC and Peirce index, whereas there is no significant effect for oversampling.

In table 2, line 8-11, we have considered a bootstrap of the events when an undersampling method is used and a bootstrap sample of the non-events when an oversampling method is used. It is seen that Bootstrap has no significant effect on the mean AUC and Peirce index, but increases their variability. In line 7 of the same table, we have performed a stratified bootstrap on the events and non-events. The predictive performance is better than that of the plain logistic regression but lower than over/under sampling or hybrid methods with optimized rate.

Among all the models considered here, the best predictive models are the hybrid models undersampling 0.5 and oversampling (1:1) or undersampling 0.3 and oversampling (5:3).

<figure><img src="image_6.png" /><figcaption>FIGURE 5 – AUC and Peirce index variation against the number of aggregation in ensemble-based methods for two resampling methods.</figcaption></figure>
<table><caption>TABLE 2 – Effect of the ensemble-based methods on the mean and standard deviation of AUC and Peirce index for several resampling methods.</caption><thead><tr><th>Undersampling<sup>(1)</sup></th><th>Oversampling<sup>(2)</sup></th><th>AUC (std)</th><th>Peirce<sup>(3)</sup> (std)</th><th>Sensitivity</th><th>Specificity</th></tr></thead><tbody><tr><td>None</td><td>5:3</td><td>0.78 (0.002)</td><td>0.562 (0.005)</td><td>0.75</td><td>0.81</td></tr><tr><td>None</td><td>1:1</td><td>0.77 (0.001)</td><td>0.548 (0.002)</td><td>0.75</td><td>0.80</td></tr><tr><td>0.3</td><td>None</td><td>0.77 (0.005)</td><td>0.554 (0.035)</td><td>0.81</td><td>0.75</td></tr><tr><td>0.5</td><td>None</td><td>0.76 (0.003)</td><td>0.561 (0.017)</td><td>0.81</td><td>0.75</td></tr><tr><td>0.3</td><td>5:3</td><td>0.78 (0.009)</td><td>0.566 (0.010)</td><td>0.81</td><td>0.74</td></tr><tr><td>0.5</td><td>1:1</td><td>0.79 (0.003)</td><td>0.564 (0.021)</td><td>0.81</td><td>0.75</td></tr><tr><td>Boots</td><td>Boots</td><td>0.76 (0.004)</td><td>0.526 (0.025)</td><td>0.81</td><td>0.66</td></tr><tr><td>Boots</td><td>5:3</td><td>0.78 (0.002)</td><td>0.555 (0.008)</td><td>0.81</td><td>0.74</td></tr><tr><td>Boots</td><td>1:1</td><td>0.78 (0.006)</td><td>0.541 (0.029)</td><td>0.81</td><td>0.72</td></tr><tr><td>0.3</td><td>Boots</td><td>0.74 (0.007)</td><td>0.549 (0.030)</td><td>0.81</td><td>0.75</td></tr><tr><td>0.5</td><td>Boots</td><td>0.76 (0.002)</td><td>0.567 (0.034)</td><td>0.81</td><td>0.76</td></tr></tbody><tfoot><tr><td colspan="6">(1) : Oversampling rate or bootstrap if used.</td></tr><tr><td colspan="6">(2) : Undersampling rate or bootstrap if used.</td></tr><tr><td colspan="6">(3) : Peirce index.</td></tr></tfoot></table>

### 3.2.4 Longitudinal validation vs cross-validation

Longitudinal validation described in Section 2.4 corresponds to the way the model is used in practice. It is therefore the most relevant method to evaluate the predictive performance.

Usual cross-validation methods such as leave-one-out cross-validation (LOOCV) use future information to predict the outcome. For example, for our data set analyzed with plain logistic regression, i.e. without resampling methods, AUC and Peirce indices obtained by LOOCV are equal to 0.84 and 0.665 whereas they are equal to 0.72 and 0.51 for longitudinal validation. For the best strategy found in 3.3.2, i.e. 0.5 undersampling followed by oversampling (5 : 3) with aggregation, AUC and Peirce index are equal to 0.85 and 0.672 for LOOCV and to 0.78 and 0.566 for longitudinal validation. So, we can see that LOOCV overestimate the true predictive performance of the models.

Another validation strategy consists in using the dataset based on the seasons 2015-2018 to train the model and the dataset of the season 2018-2019 to test the model (see, e.g., Carey DL et al., 2018). This approach is relevant if it is not possible to update the model with fresh data or if we want to use the model for other individuals or players. However, in the case of individual follow-up, the model loose information from the near past. For example, with this validation approach, AUC and Peirce index are equal to 0.650 and 0.25 for the plain logistic regression 0.681 and 0.31 for the hybrid undersampling 0.5, oversampling (5 : 3). We can see that this validation strategy tends to underestimate the predictive performance of the model as it is used in practice.
# 4 Conclusion

We have shown how resampling methods can improve substantially predictive models for rare events. The best resampling method and the optimal sampling rate are specific to each dataset. Most often they are calibrated by cross validation. However, in the case of longitudinal follow-up, usual cross-validation methods tend to overestimate the predictive quality of the model. Therefore, it is important to use a validation method adapted to longitudinal follow-up.

Pure random oversampling or hybrid under/oversampling with optimized sampling rate appear to be the most effective method to improve a logistic regression for rare events. SMOTE was ineffective for our dataset structure, mainly due to isolated events in the space of explanatory variables. Moreover, ensemble-based methods and predictor aggregation reduce the effects of the variability of resampling methods onto the predictors.

# Acknowledgment

The authors are grateful to Olivier Brachet (Innovation Performance Analytics) for having provided the dataset.

# Funding

This research has been supported by the European Regional Development Fund and the Region Auvergne-Rhone-Alpes

# Références

Batista, G., Prati, R., and Monard, M.-C. (2004). A study of the behavior of several methods for balancing machine learning training data. *SIGKDD Explorations*, 6:20–29.

Batuwita, R. and Palade, V. (2010). Efficient resampling methods for training support vector machines with imbalanced datasets. *The 2010 International Joint Conference on Neural Networks (IJCNN)*, pages pp. 1–8.

Bühlmann, P. and Yu, B. (2002). Analyzing bagging. *Annals of Statistics*, 30.

Bolton, R. J. and Hand, D. J. (2002). Statistical fraud detection : a review. *Statistical Science*, 17(3):235–255.

Breiman, L. (1996). Bagging predictors. *Machine Learning*, 24:123–140.

Breslow, N. E. (1996). Statistics in epidemiology : The case-control study. *Journal of the American Statistical Association*, 91(433):14–28.
Burez, J. and Van den Poel, D. (2008). Handling class imbalance in customer churn prediction. *Expert Systems with Applications*, 36:4626–4636.

Calabrese, R. and Osmetti, S. (2013). Modelling small and medium enterprise loan defaults as rare events: The generalized extreme value regression model. *Journal of Applied Statistics*, 40:1172–1188.

Calabrese, R. and Osmetti, S. (2015). Improving forecast of binary rare events data: A gam-based approach. *Journal of Forecasting*, 34.

Carey DL, O. K., R. W., KM, C., J. C., and ME, M. (2018). Predictive modelling of training loads and injury in australian football. *Int. J. Comput. Sci. Sport*, 17:19–66.

Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). Smote: Synthetic minority over-sampling technique. *Artificial Intelligence Research*, 16(1):321–357.

Daniel, C. and Javier, R.-G. (2017). The prevalence of injuries in professional soccer players. *Journal of Orthopedic Research and Therapy*.

Drummond, C. and Holte, R. (2003). C4.5, class imbalance, and cost sensitivity: Why under-sampling beats oversampling. *Proceedings of the ICML'03 Workshop on Learning from Imbalanced Datasets*.

Eetvelde, H. V., Mendonça, L. D., Ley, C., Romain, S., and Tischer, T. (2021). Machine learning methods in sport injury prediction and prevention: a systematic review. *Journal of Experimental Orthopaedics*, pages 1–15.

Japkowicz, N. (2000). The class imbalance problem: Significance and strategies. *Proceedings of the 2000 International Conference on Artificial Intelligence ICAI*.

King, G. and Zeng, L. (2002). Logistic regression in rare events data. *Political Analysis*, 9.

Kubat, M. (2000). Addressing the curse of imbalanced training sets: One-sided selection. *Fourteenth International Conference on Machine Learning*.

Liu, X.-Y., Wu, J., and Zhou, Z.-H. (2009). Exploratory undersampling for class-imbalance learning. *Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on*, 39:539–550.

Manski, C. F. and Lerman, S. R. (1977). The estimation of choice probabilities from choice based samples. *Econometrica*, 45(8):1977–1988.

McCullagh, P. and Nelder, J. (1989). *Generalized Linear Model*. Chapman Hall.
Nuñez, H., Gonzalez-Abril, L., and Angulo, C. (2017). Improving svm classification on imbalanced datasets by introducing a new bias. *Journal of Classification*.

Pazzani, M. J., Merz, C. J., Murphy, P. M., Ali, K. M., Hume, T., and Brunk, C. (1994). Reducing misclassification costs. In *ICML*.

Raeder, T., Forman, G., and Chawla, N. (2012). Learning from imbalanced data : Evaluation matters. *Data Mining : Foundations and Intelligent Paradigms. Intelligent Systems Reference Library, vol 23*. Springer, 23.

Spears, D. F. and Perlis, D. (1989). Explicitly biased generalization. *Comput. Intell.,* 5 :67–81.

Tomek, I. (1976). Two modifications of cnn. *EEE Transactions on Systems Man and Communications*, pages 769–772.

Zhu, B., Baesens, B., Backiel, A., and vanden Broucke, S. (2017). Benchmarking sampling techniques for imbalance learning in churn prediction. *Journal of the Operational Research Society*, 69 :1–17.