# Toward Robust Prediction: An Empirical Study of Feature Selection Methods

Thibault Anani, Francois Delbot, Jean-François Pradat-Peyre

<figure><img src="image_1.png" /></figure>

To cite this version:

Thibault Anani, Francois Delbot, Jean-François Pradat-Peyre. Toward Robust Prediction: An Empirical Study of Feature Selection Methods. 2025. hal-05304364v2

HAL Id: hal-05304364

https://hal.science/hal-05304364v2

Preprint submitted on 9 Oct 2025 (v2), last revised 19 Dec 2025 (v3)

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Toward Robust Prediction : An Empirical Study of Feature Selection Methods

Thibault Anani<sup>a</sup>, François Delbot<sup>a,b</sup> and Jean-François Pradat-Peyre<sup>a,b</sup>

<sup>a</sup>Sorbonne Université, CNRS, LIP6, 4 Pl. Jussieu, Paris, France; <sup>b</sup>U.F.R SEGMI, Nanterre Université, 200 av. de la République, Nanterre, France

## ARTICLE HISTORY

Compiled October 9, 2025

### Abstract

In this paper, we study the problem of feature selection with the aim of improving the predictive performance of machine-learning models. Wang et al. [1] showed that Differential Evolution (DE), a metaheuristic comparatively underused in both the academic literature and industry, can outperform more widespread approaches such as Particle Swarm Optimisation and Genetic Algorithms. However, their results have several limitations: no comparison with filter-based methods or greedy algorithms, and evaluation solely on artificial score functions rather than datasets. We present a broader study that includes filter methods, greedy heuristics and other metaheuristics, together with a suite of datasets spanning both synthetic benchmarks and real-world cases. Our results confirm that DE is frequently competitive and often superior in predictive performance and robustness. More significantly, an ablation over DE operators hints that a specific mutation strategy, not the one adopted in Wang et al., consistently yields stronger and more stable results across learners and datasets. Building on these findings, we introduce TiDE, a DE-based variant that improves search efficiency and stability across learners. We further propose TiDE-OPT, which tunes a key hyperparameter to adapt to dataset characteristics. Across datasets and learners, TiDE and TiDE-OPT achieve top-tier performance, with TiDE-OPT most often attaining the best results, while maintaining compact feature subsets.

### KEYWORDS

feature selection; optimisation; metaheuristics; differential evolution; benchmarking

## 1. Introduction

In a context of large-scale digitisation, the volumes of data generated in scientific, medical and industrial domains have continued to grow, making the question of their reliable use more critical than ever. The quality of the data and of the machine-learning models built from them is now a fundamental concern. The well-known expression “Garbage In, Garbage Out” captures this issue perfectly. Even the most sophisticated machine-learning models fail to produce robust results when the data are noisy, incomplete or redundant [2]. This situation is especially acute in high-dimensional domains such as medicine, where datasets often comprise a large number of features for a limited number of observations [3]. This phenomenon exacerbates overfitting. At the same time, the growing complexity of learning models, often perceived as “black

CONTACT Thibault Anani. Email: thibault.anani-agondja@lip6.fr
boxes” [4, 5], makes them increasingly difficult to interpret, reinforcing the need for models that are simultaneously high-performing, robust and interpretable.

Feature selection stands as a natural response to these challenges. It aims to extract, from an initially often high-dimensional set, a relevant subset of explanatory features, so as to increase models’ predictive performance while limiting computational complexity [6–8]. Limiting the number of features to those that are most informative reduces the costs of processing and collection, while also facilitating the interpretation of results through greater transparency in the model’s decisions. This approach seeks to simplify the data without substantial information loss, while further limiting the risk of overfitting by focusing learning on features that are genuinely informative. However, finding the best subset of features is a combinatorial optimisation problem that is *NP*-hard, which makes exhaustive exploration of the solution subspace rapidly unthinkable as the number of features grows [8–10].

Nevertheless, in practice, the approaches used often remain limited to filter methods or to low-complexity wrapper methods such as greedy heuristics, which are indeed fast but unable to capture complex interactions between features. This leads to an underuse of wrapper methods that offer stronger predictive performance, notably metaheuristics, which are recognised in the literature for their ability to explore large search spaces efficiently [11, 12]. Among these are branch-and-bound [13, 14], tabu search [15–17], simulated annealing [18, 19], particle swarm optimisation [20–23], ant colony optimisation [24–27], genetic algorithms [28–32], harmony search [33, 34], quantum evolutionary algorithms [35–38], and differential evolution [1, 39].

Among these methods, Differential Evolution (DE) has proved particularly promising. In Wang et al.’s (2013) study, a comparison of several metaheuristics indicates that DE often outperforms its competitors in the context of feature selection [1]. However, their study has several important limitations. The experiments are conducted exclusively on artificial scoring functions, the comparison is restricted to population-based approaches, and no classical methods such as filters or greedy sequential approaches are included. These restrictions limit the generalisability of their conclusions, particularly in real applied settings.

However, several works have highlighted the decisive influence of data structure on the performance of predictive models, in particular the presence of noise, redundancy, and class imbalance [40–42]. This observation suggests that the effectiveness of a feature selection method may likewise be strongly affected. Moreover, the existence of the “No Free Lunch” theorem supports this intuition [43]. It states that no optimisation algorithm can outperform the others across the full set of possible problems. It follows that no feature selection method can claim universally optimal performance, and that it is crucial to design adaptive strategies capable of leveraging the structural particularities of each dataset.

In this study, we propose a more comprehensive approach. We evaluate a diverse set of methods covering filters, greedy heuristics, local-search metaheuristics and population-based metaheuristics. Building on the experimental results, we also introduce a new adaptive variant of Differential Evolution, *Tournament in Differential Evolution* (TiDE), which dynamically adapts its initialisation, mutation and crossover strategies to the characteristics of the data. All methods are evaluated on a diverse benchmark comprising real-world and synthetic datasets with varied structures. This work aims to provide a rigorous assessment of the robustness and practical relevance of the different approaches under more realistic conditions, and to position TiDE as a competitive alternative for feature selection.
## 2. Materials and methods

### 2.1. Feature selection methods

#### 2.1.1. Filter methods

Filter methods are feature selection techniques that rely on statistical measures. They assess the relative importance of each explanatory feature with respect to the target feature by computing a score for each. The input features are then ranked and selected according to these scores for inclusion in the model. Feature importance is evaluated from the data independently of the chosen model and learning algorithm. Within this category, there is a large number of statistical tests that can be used, whose applicability and implementation depend on the types of features present in a dataset [44–46].

##### 2.1.1.1. Analysis of Variance (ANOVA).
ANOVA is a classical statistical method for measuring the discriminative power of a quantitative explanatory feature with respect to a categorical target one. It compares the between-group variance, i.e. the dispersion of the means of each group corresponding to the target classes, with the within-group variance, i.e. the dispersion of values within each group. The greater the variation between groups relative to the internal variation, the more likely the feature under consideration is to be relevant. The ratio of these two components is called the F-ratio. A high value of this ratio, together with a small p-value, indicates that the target feature depends significantly on the explanatory feature.

##### 2.1.1.2. Minimum Redundancy Maximum Relevance (MRMR).
MRMR, introduced by Ding and Peng [47], aims to select a subset of features that are both informative and minimally redundant. It rests on two complementary principles: maximising the relevance of the selected features with respect to the target feature, and minimising redundancy among explanatory features. Relevance is typically measured using statistical indicators such as mutual information, Pearson correlation, or ANOVA, depending on the nature of the features [48, 49]. Redundancy is evaluated between each new candidate feature and those already selected using analogous measures. Formally, the MRMR score of an explanatory feature $x_i$ (not yet selected) with respect to an already selected set $S$ is

$$ \text{MRMR}(x_i) = \text{Relevance}(x_i, y) - \frac{1}{|S|} \sum_{x_j \in S} \text{Redundancy}(x_i, x_j) \quad (1) $$

where $y$ denotes the target feature and $|S|$ the size of the selected set. Selection proceeds iteratively. At each step, the feature that maximises this score is added to $S$, until a predefined number of features is reached. This strategy yields a more parsimonious subset.

##### 2.1.1.3. Spatially Uniform Relief-F (SURF).
SURF is an extension of the ReliefF algorithm proposed by Greene et al. in 2009 [50], designed to improve the assessment of feature relevance in settings where non-linear or local interactions between instances play a decisive role.

ReliefF generalises the original Relief algorithm to multi-class problems and im-
proves its robustness [51–55]. For each instance sampled at random from the dataset, ReliefF identifies a set of *k* nearest neighbours of the same class (the *Hits*) and *k* nearest neighbours for each different class (the *Misses*). A feature is considered relevant if it varies slightly among instances of the same class but varies strongly between instances of different classes. At each iteration, the weights are updated based on the differences observed between the current instance and its neighbours, weighted by each class probability.

SURF modifies this neighbourhood mechanism. Instead of fixing a number *k* of neighbours, it considers all instances within a given radius around the target instance. This threshold distance is computed from the average distances observed between instances. All instances within this zone are then used to update feature weights, according to whether they belong to the same class as the central instance or not. This strategy introduces a uniform weighting within a local zone of influence, allowing the spatial distribution of the data to be better taken into account, particularly in the presence of complex or non-linear structures.

### 2.1.2. Basic wrapper methods

Unlike filter methods, which assess each feature independently of the model used, wrapper methods integrate supervised learning directly into the selection process. Each subset of features is evaluated according to the predictive performance of a model trained on data restricted to those features. This framework naturally corresponds to a combinatorial optimisation problem, where the search space is discrete and binary. More precisely, each solution *S* (a candidate feature subset) can be represented as a binary vector *S* = (*S*₁, *S*₂, ..., *S*<sub>*D*</sub>) with *S*<sub>*j*</sub> ∈ {0, 1} whose components indicate the presence or absence of each feature. For a problem of dimension *D* (the number of features), this space contains 2<sup>*D*</sup> possible subsets. Since exhaustive exploration becomes infeasible as soon as *D* is moderate, heuristic or even metaheuristic search strategies are required to identify near-optimal solutions within reasonable time. Although more costly than filters, wrapper methods generally achieve better predictive performance, because they account for interactions between features and their joint impact on the learning model.

#### 2.1.2.1. Random search (RAND)
This basic strategy consists in generating solutions sampled at random in the search space, then evaluating their quality using an objective function. Although often viewed as naive, it provides a sound methodological baseline in many contexts, particularly in feature selection, where it ensures an unbiased exploration of the solution space. At each iteration, a new solution is sampled at random and compared with the best solution found so far. If it performs better, it replaces it. This process continues until a stopping criterion is met. However, in the absence of learning that guides the search towards promising regions, its effectiveness is limited in high-dimensional spaces or under tight constraints, where the probability of sampling an optimal solution is extremely low. In this work, we use random search as a baseline to assess the value of more elaborate optimisation approaches.

#### 2.1.2.2. Sequential selection.
Sequential feature selection relies on greedy heuristics that iteratively construct a subset of features by adding or removing, at each step, the one that optimises a given performance criterion [56, 57]. Once a decision is made at a given iteration, it is generally not revisited, which distinguishes these approaches
from more flexible methods. Despite this constraint, their low algorithmic complexity and practical effectiveness make them popular choices for feature selection, especially in high-dimensional settings.

Sequential Forward Selection (SFS) starts from an empty set of features and, at each iteration, adds the feature whose inclusion maximises the model score according to a predefined criterion. The process continues until no significant improvement is observed or a maximum number of features is reached. In the worst case, the total number of evaluations is on the order of $O(n^2)$, with $\frac{n(n+1)}{2}$ models trained, which remains far below the exponential complexity of exhaustive search.

Sequential Backward Selection (SBS) follows the opposite strategy. The process begins with the full set of features and, at each step, removes the feature whose deletion increases the model's performance. This mechanism allows complex interactions between features to be taken into account from the outset, and favours the retention of synergistic features that might appear uninformative in isolation. However, this approach is generally more costly than SFS, because the initial models are more complex to train (i.e. more features to account for).

### 2.1.3. Local-search wrapper methods

Local-search metaheuristics form a class of approaches that exploit the topology of the solution space, where neighbouring solutions tend to exhibit similar performance. In the context of feature selection, these methods start from an initial solution (often random) and explore its neighbourhood, defined as the set of solutions obtained by elementary modifications such as adding, removing, or swapping features. Their objective is to optimise the model's performance by iteratively exploring and evaluating the neighbourhood of the current solution.

<figure><img src="image_3.png" /><figcaption>Figure 1. – Example of three bit-flip disruptions applied to a solution *S* to obtain a neighbouring solution *S*′.</figcaption></figure>

#### 2.1.3.1. Hill Climbing (HC).
This method exemplifies the principle by progressively building an improved solution [58–60]. At each iteration, it generates the neighbours of the current solution and adopts the one that maximises the objective function, as long as an improvement is observed. In the case of feature selection, this amounts to gradually adjusting the composition of the feature subset (e.g. adding, removing, or flipping bits as shown in Figure 1).

Simple to implement, this method is effective in optimisation landscapes where ascending trajectories leading to optima exist. However, it can become trapped in local optima, and its performance depends on the initial solution.

#### 2.1.3.2. Tabu Search (TS).
This metaheuristic extends hill climbing by introducing a memory mechanism that prevents cycling and helps escape local optima [61–63]. At each iteration, the algorithm explores the neighbourhood of the current solution,
but temporarily forbids returning to certain recently visited solutions, identified by
a structure called the “tabu list”. This short-term memory typically follows a FIFO
principle, i.e. the oldest moves are progressively removed when the maximum capacity
is reached.

In the context of feature selection, a neighbour is generated by an elementary mod-
ification (e.g., Figure 1). Even if a neighbour does not immediately improve perfor-
mance, it can be retained provided it is not tabu, thereby allowing the algorithm to
traverse plateaus in the search space.

This strategy aims to enhance exploration by avoiding stagnation and promoting
controlled diversification. However, performance depends strongly on parameter set-
tings, notably the size of the tabu memory. If it is too small, it limits effectiveness,
and if too large, it slows the process unnecessarily. Although more complex than
hill climbing, this approach is a robust alternative, particularly suited to large and
structured search spaces.

2.1.4. Population-based wrapper methods

Population-based wrapper methods are a class of metaheuristics that, unlike local approaches centred on a single solution, evolve a set of solutions in parallel. Each solution, called an individual, represents a subset of features. This population evolves over generations according to principles inspired by evolutionary or swarm mechanisms.

The central idea is to combine two dynamics: *exploitation* of the current best solutions and *exploration* of new regions of the search space. To this end, individuals are evaluated using an objective function (e.g. model performance), then selected, recombined and perturbed via operators such as crossover and mutation. A new generation is then formed, incorporating both promising solutions and diverse ones. This iterative process enables progressive convergence towards relevant feature configurations.

This family of approaches is particularly well suited to complex or discontinuous
search spaces, since it reduces the risk of stagnation in a local optimum and promotes
the discovery of solutions that are both novel and performant. The comparative study
by Wang et al. (2013) [1] evaluated several population-based metaheuristics, no-
tably a Genetic Algorithm, Particle Swarm Optimisation, Ant Colony Optimisation,
a Quantum Evolutionary Algorithm, Harmony Search, and an adapted version of Dif-
ferential Evolution. The experimental results highlighted the superiority of the latter,
which proved more effective than the other approaches tested. Consequently, only
two of these methods are retained here: the Genetic Algorithm, due to its popularity
and prevalence in the literature, and Differential Evolution. We also include another
method not considered in Wang et al., Population-Based Incremental Learning, which
relies on a probabilistic modelling of the population.

2.1.4.1. *Genetic Algorithm (GA)*. Introduced by Holland in 1975, the GA is a population-based metaheuristic inspired by the principle of natural selection [64]. It operates on a population of individuals, each representing a candidate solution, through a process of evaluation, selection, reproduction and mutation [65].

At each generation, individuals are evaluated using an objective function, then some
are selected to reproduce. Crossover combines their characteristics to generate new
individuals (cf. Figure 2), while mutation introduces small perturbations, flipping a
few bits as in local search, to preserve diversity. The new generation partially or fully
replaces the previous one according to a replacement strategy such as elitism, rank
selection, tournament selection, etc. [66–68].
<figure><img src="image_4.png" /><figcaption>Figure 2. – Two-point crossover to generate two offspring. The cut points are $j_2$ and $j_4$. Left: construction of the first offspring. Right: construction of the second.</figcaption></figure>

This mechanism allows exploitation of promising solutions to coexist with exploration of the search space, thereby limiting the risk of premature stagnation. However, its effectiveness depends closely on the tuning of its hyperparameters (population size, mutation rate, selection method, crossover method, etc.). Poor calibration can lead to premature convergence or substantial computational overhead. Despite these limitations, its flexibility, robustness and widespread use make it a reference among metaheuristics.

### 2.1.4.2. Population-Based Incremental Learning (PBIL)
Proposed by Baluja in 1994, PBIL is a metaheuristic that combines principles from evolutionary algorithms and probabilistic models [69, 70]. Unlike the Genetic Algorithm, which directly manipulates a population of individuals, PBIL maintains a single probability vector $V$ that models the distribution of good individuals in the search space.

Each component $V_j \in [0, 1]$ encodes the probability that the $j$-th feature takes the value 1. At each generation, a population of individuals is sampled from $V$, evaluated, and the top $n$ individuals are then used to update $V$ according to

$$V_j = (V_j \times (1 - LR)) + \left( LR \times \frac{\sum_{i=1}^{n} S_{i,j}}{n} \right) \quad (2)$$

where $LR$ is the learning rate, and $S_{i,j}$ is the value of the $j$-th feature bit of individual $i$. To maintain diversity, PBIL also applies a mutation to each $V_j$ with probability $MP$, as follows:

$$V_j = \begin{cases} V_j \times (1 - MS), & \text{with probability } \frac{1}{2} \\ V_j \times (1 - MS) + MS, & \text{otherwise} \end{cases} \quad (3)$$

where $MS$ is the mutation step. This scheme progressively steers the search towards the most promising configurations while retaining a controlled exploration capability. The absence of explicit crossover and mutation operators on individuals considerably simplifies implementation and reduces computational cost.

PBIL is therefore particularly effective in discrete spaces, notably when a good initialisation of $V$ helps guide the search. However, like any stochastic method, its performance depends strongly on parameter tuning, and premature convergence may occur if the model is updated too quickly or if diversity is insufficiently maintained.
2.1.4.3. **Differential Evolution (DE).** This population-based metaheuristic, introduced by Storn and Price in 1997 [71], relies on three fundamental mechanisms: mutation, crossover and selection. It was originally designed for continuous optimization problems, but its general principle can be adapted to other contexts such as feature selection.

Unlike a classical Genetic Algorithm, the algorithm first performs a mutation step. For each individual $S_i$ in the population, a mutant $M_i$ is generated from other individuals in the population. This operation is based on vector differences between individuals, scaled by an amplitude factor $F \in [0, 2]$. For example, the classical DE/rand/1 strategy selects three distinct individuals $S_{r1}$, $S_{r2}$, $S_{r3}$ (all different from $S_i$), then constructs the mutant:

$$M_i = S_{r1} + F \times (S_{r2} - S_{r3}) \quad (4)$$

Other strategies exist (see Table 1), involving the best individual in the population or more complex combinations of differences [72]. A value of $F$ that is too large increases exploration, at the risk of instability, whereas a value that is too small makes the algorithm overly conservative.

<table><caption>Table 1. – A non-exhaustive list of mutation strategies. $M_i$ is the mutant obtained by mutation with $\{i, r_1, r_2, \dots, r_k\} \in \mathbb{N}$, $r_1 \neq r_2 \neq \dots \neq r_k \neq i$, $b$ is the index of the best individual in the population, and $k \in \mathbb{N}$ is the number of randomly selected individuals required to apply a given strategy.</caption><thead><tr><th>Strategy</th><th>Formula</th></tr></thead><tbody><tr><td>DE/rand/1</td><td>$M_i = S_{r1} + F \times (S_{r2} - S_{r3})$</td></tr><tr><td>DE/best/1</td><td>$M_i = S_b + F \times (S_{r1} - S_{r2})$</td></tr><tr><td>DE/current-to-rand/1</td><td>$M_i = S_i + F \times (S_{r1} - S_i) + F \times (S_{r2} - S_{r3})$</td></tr><tr><td>DE/current-to-best/1</td><td>$M_i = S_i + F \times (S_b - S_i) + F \times (S_{r1} - S_{r2})$</td></tr><tr><td>DE/rand-to-best/1</td><td>$M_i = S_{r1} + F \times (S_b - S_i) + F \times (S_{r2} - S_{r3})$</td></tr><tr><td>DE/rand/2</td><td>$M_i = S_{r1} + F \times (S_{r2} - S_{r3}) + F \times (S_{r4} - S_{r5})$</td></tr><tr><td>DE/best/2</td><td>$M_i = S_b + F \times (S_{r1} - S_{r2}) + F \times (S_{r3} - S_{r4})$</td></tr><tr><td>DE/current-to-rand/2</td><td>$M_i = S_i + F \times (S_{r1} - S_i) + F \times (S_{r2} - S_{r3}) + F \times (S_{r4} - S_{r5})$</td></tr><tr><td>DE/current-to-best/2</td><td>$M_i = S_i + F \times (S_b - S_i) + F \times (S_{r1} - S_{r2}) + F \times (S_{r3} - S_{r4})$</td></tr><tr><td>DE/rand-to-best/2</td><td>$M_i = S_{r1} + F \times (S_b - S_i) + F \times (S_{r2} - S_{r3}) + F \times (S_{r4} - S_{r5})$</td></tr></tbody></table>

<table><caption>Figure 3. – Example of generating a mutant with the DE/rand/1 strategy and $F = 1$.</caption><thead><tr><th></th><th>j<sub>1</sub></th><th>j<sub>2</sub></th><th>j<sub>3</sub></th><th>j<sub>4</sub></th><th>j<sub>5</sub></th><th></th></tr></thead><tbody><tr><td>S<sub>r1</sub></td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td>M<sub>i,1</sub> = 0 + 1 × (1 − 1) = 0</td></tr><tr><td>S<sub>r2</sub></td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>M<sub>i,2</sub> = 1 + 1 × (1 − 1) = 1</td></tr><tr><td>S<sub>r3</sub></td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>M<sub>i,3</sub> = 0 + 1 × (0 − 1) = −1 ⇒ 0</td></tr><tr><td>M<sub>i</sub></td><td>0</td><td>1</td><td>0</td><td>1</td><td>1</td><td>M<sub>i,4</sub> = 1 + 1 × (1 − 0) = 2 ⇒ 1</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>M<sub>i,5</sub> = 0 + 1 × (1 − 0) = 1</td></tr></tbody></table>

Once the mutant $M_i$ has been generated, it is crossed with the initial individual $S_i$ to produce a new individual $E_i$. This crossover is generally uniform: for each gene $j$, choose $E_{i,j} = M_{i,j}$ with probability $CR \in [0, 1]$, otherwise keep $E_{i,j} = S_{i,j}$. To avoid
generating an individual identical to $S_i$, an index $j_{rand} \in [1, D]$ is imposed to force the inclusion of at least one gene from $M_i$:

$$E_{i,j} = \begin{cases} M_{i,j}, & \text{if } \text{rand()} < CR \text{ or } j = j_{rand} \\ S_{i,j}, & \text{otherwise} \end{cases} \qquad (5)$$

To determine whether the individuals produced by crossover are retained, their score is compared with that of the corresponding individuals in the current population. If an individual $E_i$ has a better score than its counterpart $S_i$, then $E_i$ is included in the next generation, otherwise $S_i$ is kept.

When applied to binary spaces, as in feature selection, DE must be adapted. The arithmetic operations involving differences between real-valued vectors must be replaced by transformations that remain within a discrete space. A classical method is to generate a real vector and then threshold it to obtain a binary one (e.g. > 0.5 ⇒ 1, otherwise 0, cf. Figure 3). However, this simplification can reduce effectiveness, in particular by harming diversity or slowing convergence. To address these limitations, several works have proposed binary adaptations of DE [73–76].

#### 2.1.4.4. Modified Binary Differential Evolution (MBDE).
Used in the experiments of Wang et al. (2013) [1], MBDE [77] aims to improve the standard version of Differential Evolution in a binary setting. Instead of directly producing a binary mutant, MBDE generates a mutation probability vector $P_i = [p_{i,1}, \dots, p_{i,D}]$, which is then sampled to produce a binary individual. The mutation formula becomes:

$$p_{i,j} = \frac{1}{1 + \exp\left(-2b \times \frac{v_{i,j} - 0.5}{1+2F}\right)} \qquad (6)$$

The term $v_{i,j}$, obtained from the result of the DE/rand/1 strategy, is projected into [0, 1] via a sigmoid function. The parameter $b$, called the “bandwidth factor”, is a positive real constant that controls the slope of the sigmoid. The larger $b$ is, the steeper the sigmoid becomes around 0.5, making the decision to set the bit to 0 or 1 more deterministic. Conversely, a small $b$ induces a fuzzier probability, leaving more room for stochastic exploration. This parameter therefore regulates the balance between exploration and exploitation in the algorithm. The $j$-th gene of the mutant $M_i$ is then generated by random sampling:

$$M_{i,j} = \begin{cases} 1, & \text{if } \text{rand()} < p_{i,j} \\ 0, & \text{otherwise} \end{cases} \qquad (7)$$

The crossover phase remains identical to that described in the standard DE version. By converting the information from differential vectors into a Bernoulli distribution parametrised by $p_{i,j}$, MBDE preserves population diversity while finely adapting the mutation pressure to the local topology of the search space.
## 2.2. Evaluation criteria

We identify four main criteria for assessing the suitability of a feature selection method in a predictive modelling context:

(1) **Predictive quality:** This is the primary criterion we optimise. Obtaining a model with strong predictive quality is a key indicator of successful feature selection. In this study, we focus on classification problems. We therefore measure model performance using *Balanced Accuracy* (BA) = $\frac{\text{Recall}+\text{Specificity}}{\text{Number of classes}}$, a metric that assigns equal weight to each class regardless of its prevalence. This metric is particularly suitable for avoiding bias towards the majority class. It also reflects a model's ability to predict each class well without compromising overall balance.

(2) **Parsimony:** This secondary criterion seeks to minimise the number of selected features relative to the total number of available features. The aim is to promote parsimonious models, i.e. models that use as few features as possible without compromising predictive quality. Parsimonious models are often easier to interpret and to deploy, while reducing the risk of overfitting.

(3) **Convergence speed:** The time (in seconds) required to reach the feature subset associated with the best model found by the selection method. Convergence speed is a key factor in assessing the practical efficiency of feature selection, especially when computational resources are limited. Faster convergence accelerates the modelling process and facilitates the exploration of multiple scenarios or hyperparameters within a reasonable time.

(4) **Extensibility:** This criterion evaluates a method's ability to maintain stable and competitive performance when applied to datasets with diverse structures (number of instances, number of features, etc.). An extensible method should be flexible enough to adapt to varying conditions without significant performance degradation (relative to other methods). For each pair $(d, m)$ of dataset and selection method, we compute a score normalised by the maximum and minimum performances observed on that dataset:

$$ \delta(d, m) = \frac{s(d, m) - s(d, M)_{min}}{s(d, M)_{max} - s(d, M)_{min}} \qquad (8) $$

where $s(d, m)$ denotes the performance (e.g. balanced accuracy) of method $m$ on dataset $d$, and the bounds are computed over all methods $M$ applied to $d$. $\delta(d, m) \in [0, 1]$ reflects the relative position of $m$ with respect to the best and worst performances achieved for $d$. A method's extensibility is then evaluated by the mean of $\delta(d, m)$ over all considered datasets (which we denote by $\Delta$). A method is judged extensible if it attains a good average performance, i.e. close to 1, indicating robustness and stability across varied situations.

These criteria should be considered jointly when evaluating a feature selection strategy. In practice, there are often trade-offs between these dimensions, e.g. reducing the number of features too aggressively may harm predictive quality, whereas very fast convergence might be achieved at the expense of a more precise exploration of possible combinations. The objective is therefore to strike a balance that meets the problem's requirements while respecting constraints on performance, interpretability, extensibility and available computational resources.
## 2.3. Establishing a benchmark

<table><thead><tr><th>Name</th><th>Instances</th><th>Features</th><th>Distrib.</th><th>Noise</th><th>z</th><th>Relevant</th><th>Source</th></tr></thead><tbody><tr><td>Baseline <i>A</i></td><td>500</td><td>100</td><td>50</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Instance <i>A</i></td><td>1000</td><td>100</td><td>50</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Noise <i>A</i></td><td>500</td><td>100</td><td>50</td><td>10</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Imbalanced <i>A</i></td><td>500</td><td>100</td><td>80</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Baseline <i>B</i></td><td>500</td><td>1000</td><td>50</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Instance <i>B</i></td><td>1000</td><td>1000</td><td>50</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Noise <i>B</i></td><td>500</td><td>1000</td><td>50</td><td>10</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Imbalanced <i>B</i></td><td>500</td><td>1000</td><td>80</td><td>0</td><td>0</td><td>10</td><td>–</td></tr><tr><td>Baseline <i>C</i></td><td>500</td><td>500</td><td>50</td><td>0</td><td>10</td><td>10</td><td>–</td></tr><tr><td>Feature <i>C</i></td><td>500</td><td>1000</td><td>50</td><td>0</td><td>10</td><td>10</td><td>–</td></tr><tr><td>Relevant <i>C</i></td><td>500</td><td>500</td><td>50</td><td>0</td><td>10</td><td>25</td><td>–</td></tr><tr><td>Decomposition <i>C</i></td><td>500</td><td>1000</td><td>50</td><td>0</td><td>20</td><td>10</td><td>–</td></tr><tr><td>Spambase</td><td>4601</td><td>57</td><td>61</td><td>?</td><td>0</td><td>?</td><td>UCI [78]</td></tr><tr><td>Scene</td><td>2407</td><td>294</td><td>82</td><td>?</td><td>0</td><td>?</td><td>OpenML 312 [79]</td></tr><tr><td>Madelon</td><td>2600</td><td>500</td><td>50</td><td>?</td><td>0</td><td>20</td><td>OpenML 1485 [79]</td></tr><tr><td>Gravier</td><td>168</td><td>2905</td><td>66</td><td>?</td><td>0</td><td>?</td><td>datamicroarray [80]</td></tr></tbody></table>

Table 2. – Characteristics of the datasets used in the experiments. The columns “Instances” and “Features” indicate respectively the total number of instances and features (after decomposition where applicable). “Distrib.” gives the percentage of instances in the majority class. “Noise” specifies the percentage of mislabelled instances. The column “z” corresponds to the number of new features created for each original feature during decomposition. Finally, “Relevant” indicates the number of original explanatory features deemed informative among the full set.

To evaluate feature selection methods under controlled scenarios, we first design a collection of synthetic datasets that reproduce a variety of configurations characteristic of problems encountered in practice, while isolating the effects of different specificities and structures on their performance.

Benchmark A consists of datasets comprising a base configuration and several variants with specific perturbations. The base version, “Baseline”, represents a near-perfect dataset. It is composed of 500 instances and 100 explanatory features. The target feature is a linear combination of 10 explanatory features, referred to as relevant features. The remaining 90 features are generated at random and bear no relation to the target. The target is then transformed into a binary feature by assigning a positive class to instances above the median and a negative class otherwise. We then create three variants of Baseline by introducing a single specific anomaly each time:

- **Noise**: A percentage of mislabelled instances.

- **Imbalanced**: One class is under-represented with respect to the other.

- **Instance**: An increased total number of instances.

Benchmark B adopts the same structure as A, but with a substantial increase in dimensionality. More precisely, the number of explanatory features rises from 100 to 1000 while the number of relevant features remains fixed at 10. The objective is to test the robustness of methods under higher dimensionality, where the number of features dominates the number of instances.

The final synthetic benchmark, C, explores scenarios in which strong redundancy among explanatory features makes the selection of relevant features more difficult. Baseline comprises 500 instances and 100 binary explanatory features. The target
feature is a linear combination of 10 relevant features. Each explanatory feature is decomposed into a set of z new features according to the following procedure:

- If the original feature equals 1, the z new features also take the value 1.

- If the original feature equals 0, a single 0 is placed at random among the z new features, the others taking the value 1.

This transformation artificially increases redundancy among features, thereby complicating learning. From this Baseline, we generate several variants:

- **Feature**: An increased number of explanatory features prior to decomposition.

- **Relevant**: An increased number of relevant features.

- **Decomposition**: Amplified redundancy through a larger z.

To complete our analysis, we use a set of reference datasets from the literature. These datasets, already studied in prior work, display varied characteristics, notably in terms of the ratio between the number of features and instances, as well as internal structure [81]. They are particularly relevant for assessing feature selection methods in realistic contexts.

## 2.4. Learning protocol

<table><caption>Table 3. – Characteristics of the learning algorithms used in the experiments. m is the average number of instances per split during cross-validation.</caption><thead><tr><th>Method</th><th>Training</th><th>Prediction</th><th>Hyperparameters</th></tr></thead><tbody><tr><td>Gaussian Naive Bayes (GNB)</td><td>O(m · n)</td><td>O(n)</td><td>—</td></tr><tr><td>Logistic Regression (LR)</td><td>O(m · n · i)</td><td>O(n)</td><td>{random_state = 42, solver = "lbfgs", class_weight = "balanced", max_iter = 10000}</td></tr><tr><td>Decision Tree (DT)</td><td>O(m · n · log<sub>2</sub>(m))</td><td>O(log<sub>2</sub>(m))</td><td>{random_state = 42, criterion = "gini", class_weight = "balanced"}</td></tr><tr><td>k-Nearest Neighbours (KNN)</td><td>—</td><td>O(log<sub>2</sub>(m) · n)</td><td>{n_neighbors = &radic;m, weights = "distance", p = 2, metric = "minkowski", algorithm = "kd_tree"}</td></tr></tbody></table>

To assess the relevance of the selected features, we rely on simple learners commonly used in the literature: Gaussian Naive Bayes (GNB), Logistic Regression (LR), Decision Trees (DT), and k-Nearest Neighbours (KNN). This choice is motivated by several considerations. First, their low algorithmic complexity allows a large number of experiments to be run in reasonable time. Second, LR and DT include intrinsic feature-selection mechanisms. This makes it possible to examine whether adding an explicit selection method truly improves performance, or whether these algorithms can already suffice on their own. The methods were implemented using the scikit-learn library (version 1.5) [82], and hyperparameters were set to default values or specified beforehand (see Table 3).

Each performance evaluation follows a stratified five-fold cross-validation procedure. This strategy aims to obtain a reliable estimate of the models' generalisation ability while limiting the bias associated with an arbitrary split of the data. The dataset is partitioned into five subsets of similar size. At each iteration, four subsets are used for training and the fifth for evaluation. In this study, we report performance by computing the overall balanced accuracy across all folds combined.
## 2.5. Implementation

The feature selection methods tested in this process include several filter and wrapper approaches. For all methods, a time limit of two hours is imposed for each dataset to ensure sufficient convergence and a fair comparison across methods. We also add an additional stopping criterion for certain methods.

Given that the datasets comprise quantitative explanatory features and categorical targets, we apply ANOVA, MRMR and the SURF approach. We then evaluate multiple feature subsets ranging from 1% to 100% of the ranked features. This procedure makes it possible to assess the progressive contribution of relevant features to model performance.

For wrapper methods, we use two sequential selection techniques drawn from those described previously: Sequential Forward Selection (SFS) and Sequential Backward Selection (SBS). Both algorithms stop automatically when there are no more features to add or remove. We also integrate and adapt the heuristics and metaheuristics as follows:

(1) **Random Search (RS)**: At each iteration, feature subsets are generated at random. Each subset contains a randomly determined number of features between 1 and $D$ ($D$ being the total number of features). This promotes a diverse exploration of the subset space, including both small and large subsets.

(2) **Hill Climbing (HC)**: At each iteration, the neighbourhood of a feature subset comprises multiple neighbours. The maximum allowed distance between the current subset and a candidate neighbour is governed by a power law:

$$p(d) = \frac{\frac{1}{d^{\alpha}}}{\sum_{j=1}^{D} \frac{1}{j^{\alpha}}} \qquad (9)$$

with $d \in \{1, 2, \dots, D\}$ denoting the distance between the current subset and a candidate neighbour, and $D$ the maximum allowed distance. The power parameter $\alpha > 0$ controls the decay of the probabilities. The normalising constant $\sum_{j=1}^{D} \frac{1}{j^\alpha}$ ensures that the probabilities sum to 1. At each iteration, one neighbour is selected with probability proportional to $p(d)$. We set $\alpha = 2$. This mechanism favours local exploration while maintaining a non-zero probability of exploring more distant regions of the search space.

(3) **Tabu Search (TS)**: This method uses the same neighbourhood exploration scheme as HC, but introduces an additional mechanism: a tabu list. The tabu list size is set to a value comparable to the number of neighbours considered at each iteration, providing a balance between exploration and exploitation.

(4) **Genetic Algorithm (GA)**: Parent selection is based on roulette-wheel selection. Mutation of offspring follows the same principle as HC as defined by Equation 9, except that the minimum allowed distance is 0, thereby allowing small mutations or no mutation. Selection of individuals for the next generation is based on rank selection. These choices favour the best individuals while reducing the risk of the population converging too quickly to suboptimal solutions.

(5) **Population-Based Incremental Learning (PBIL)**: We set parameters to moderate values to encourage gradual adaptation and reduce the risk of premature convergence: $LR = 0.1$, $MP = 0.05$ and $MS = 0.1$. When updating the probability vector, the top 15% best-ranked individuals are taken into account.
This ensures that only the strongest individuals influence the learning direction, while preserving a degree of diversity.

(6) **Differential Evolution (DE)**: In this study we employ two variants:
— **DE/rand/1**, which promotes more diverse exploration of the search space,
— **DE/best/1**, which enables effective exploitation around the strongest solutions.

For both variants, we set $F = 1$ and $CR = 0.5$. This standard configuration provides a good balance between the diversity of generated candidates and the preservation of existing solutions.

(7) **MBDE**: We follow the hyperparameters used in Wang et al. (2013) [1], namely $CR = 0.2$ and $b = 20$. Since, unlike their study, our stopping criterion is runtime rather than the number of iterations, we adapt the evolution of $F$ over time:

$$F = F_{max} - (F_{max} - F_{min}) \times \frac{T}{T_{max}} \qquad (10)$$

with $F_{max} = 0.8$, $F_{min} = 0.005$, $T$ the elapsed time since the start, and $T_{max}$ the total time budget.

For all these heuristics, the number of individuals or neighbours evaluated at each iteration is fixed at 50 to ensure a reasonable balance between diversity and/or exploration and computational efficiency. For population-based metaheuristics (GA, PBIL, DE and MBDE), a reinitialisation mechanism is applied when population diversity becomes insufficient.

This diversity $\bar{H}(P)$ is measured using Shannon entropy [83]. It quantifies the uncertainty of a variable $x$ as a function of the number of values it can take $c_x$ and the probability $p(x_i)$ that $x$ takes value $x_i$. Entropy is maximal when all values of $x$ are equally likely and minimal (i.e. equal to 0) when a single value is certain:

$$H(x) = - \sum_{i=1}^{c_x} p(x_i) \log p(x_i) \qquad (11)$$

At each generation we compute a local diversity for each feature across the population. Since each decision feature is binary, its entropy score is naturally bounded between 0 (when all individuals consistently select or exclude that feature) and 1 (when half the individuals select the feature and the other half exclude it). The global diversity measure $\bar{H}(P)$ is obtained from the local entropies $H(P_j)$ computed for each feature $j \in [1, D]$:

$$\bar{H}(P) = \frac{1}{D} \sum_{j=1}^{D} H(P_j) \qquad (12)$$

If $\bar{H}(P)$ falls below 0.05, we consider the population too homogeneous to explore new feature subsets effectively. In that case, we reinitialise the population (or the probability vector in the case of PBIL), inserting the best subset identified so far among the newly generated individuals for GA, DE and MBDE.
## 3. Results analysis

### 3.1. *Experiment 1: Filters vs Wrappers*

<table><caption>Table 4. – Mean relative predictive quality (Balanced Accuracy, %) obtained by each feature selection method across all datasets, grouped by family and evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>48.54 (9)</td><td>51.90 (9)</td><td>44.23 (9)</td><td>66.40 (4)</td></tr><tr><td>MRMR</td><td>47.26 (10)</td><td>46.36 (10)</td><td>42.75 (10)</td><td>65.87 (5)</td></tr><tr><td>SURF</td><td>39.59 (11)</td><td>32.66 (11)</td><td>38.94 (11)</td><td>55.40 (8)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>24.02 (12)</td><td>26.28 (12)</td><td>30.19 (12)</td><td>28.86 (12)</td></tr><tr><td>SFS</td><td>55.96 (8)</td><td>69.11 (7)</td><td>71.35 (4)</td><td>76.39 (3)</td></tr><tr><td>SBS</td><td>14.30 (13)</td><td>18.21 (13)</td><td>6.64 (13)</td><td>9.58 (13)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>69.55 (6)</td><td>72.24 (6)</td><td>51.62 (8)</td><td>47.91 (11)</td></tr><tr><td>TS</td><td>89.70 (3)</td><td>86.28 (3)</td><td>66.96 (5)</td><td>76.46 (2)</td></tr><tr><td rowspan="5">Wrapper Population</td><td>GA</td><td>79.65 (4)</td><td>78.27 (6)</td><td>75.87 (3)</td><td>65.24 (6)</td></tr><tr><td>PBIL</td><td>77.73 (5)</td><td>84.35 (4)</td><td>81.26 (2)</td><td>65.70 (5)</td></tr><tr><td>DE (r/1)</td><td>62.88 (7)</td><td>66.70 (8)</td><td>70.68 (5)</td><td>49.90 (10)</td></tr><tr><td>DE (b/1)</td><td>95.51 (1)</td><td>95.88 (1)</td><td>85.63 (1)</td><td>85.91 (1)</td></tr><tr><td>MBDE</td><td>94.83 (2)</td><td>91.75 (2)</td><td>59.04 (7)</td><td>52.27 (9)</td></tr></tbody></table>

<table><caption>Table 5. – Mean relative parsimony (ratio of selected features, %) obtained by each feature selection method across all datasets, grouped by family and evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>83.35 (3)</td><td>77.15 (3)</td><td>89.15 (3)</td><td>87.10 (2)</td></tr><tr><td>MRMR</td><td>88.30 (2)</td><td>79.32 (2)</td><td>91.89 (2)</td><td>86.12 (3)</td></tr><tr><td>SURF</td><td>79.32 (4)</td><td>68.52 (6)</td><td>86.00 (4)</td><td>85.68 (4)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>64.84 (9)</td><td>57.37 (12)</td><td>50.64 (11)</td><td>73.90 (5)</td></tr><tr><td>SFS</td><td>98.93 (1)</td><td>98.71 (1)</td><td>99.58 (1)</td><td>99.44 (1)</td></tr><tr><td>SBS</td><td>10.19 (13)</td><td>6.88 (13)</td><td>0.42 (13)</td><td>1.41 (13)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>69.03 (6)</td><td>64.02 (9)</td><td>38.31 (12)</td><td>58.58 (11)</td></tr><tr><td>TS</td><td>68.22 (7)</td><td>73.75 (4)</td><td>51.46 (10)</td><td>63.82 (7)</td></tr><tr><td rowspan="5">Wrapper Population</td><td>GA</td><td>61.14 (10)</td><td>66.93 (7)</td><td>52.58 (9)</td><td>56.40 (12)</td></tr><tr><td>PBIL</td><td>50.93 (11)</td><td>60.03 (10)</td><td>55.33 (5)</td><td>60.72 (10)</td></tr><tr><td>DE (r/1)</td><td>48.15 (12)</td><td>59.53 (11)</td><td>55.00 (7)</td><td>62.84 (6)</td></tr><tr><td>DE (b/1)</td><td>71.47 (5)</td><td>72.36 (5)</td><td>53.40 (8)</td><td>64.11 (9)</td></tr><tr><td>MBDE</td><td>65.68 (8)</td><td>66.32 (8)</td><td>53.96 (6)</td><td>63.08 (8)</td></tr></tbody></table>
<table><caption>Table 6. – Mean relative convergence speed (number of iterations required) obtained by each feature selection method across all datasets, grouped by family and evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>100.00 (1)</td><td>99.97 (2)</td><td>99.99 (1)</td><td>100.00 (1)</td></tr><tr><td>MRMR</td><td>99.99 (2)</td><td>99.98 (1)</td><td>99.99 (1)</td><td>99.97 (2)</td></tr><tr><td>SURF</td><td>99.96 (3)</td><td>99.37 (3)</td><td>99.94 (2)</td><td>99.94 (3)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>50.50 (6)</td><td>71.08 (5)</td><td>35.90 (7)</td><td>57.14 (5)</td></tr><tr><td>SFS</td><td>97.18 (4)</td><td>91.68 (4)</td><td>94.13 (3)</td><td>92.45 (4)</td></tr><tr><td>SBS</td><td>50.76 (5)</td><td>36.81 (7)</td><td>60.19 (4)</td><td>40.08 (7)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>45.11 (7)</td><td>39.64 (6)</td><td>41.81 (5)</td><td>50.93 (6)</td></tr><tr><td>TS</td><td>14.64 (13)</td><td>21.26 (11)</td><td>42.23 (6)</td><td>31.05 (8)</td></tr><tr><td rowspan="5">Wrapper Population</td><td>GA</td><td>30.19 (10)</td><td>26.81 (9)</td><td>14.77 (11)</td><td>26.20 (10)</td></tr><tr><td>PBIL</td><td>43.14 (8)</td><td>28.05 (8)</td><td>30.19 (8)</td><td>25.19 (11)</td></tr><tr><td>DE (r/1)</td><td>29.14 (11)</td><td>10.71 (13)</td><td>17.92 (9)</td><td>25.56 (11)</td></tr><tr><td>DE (b/1)</td><td>34.81 (9)</td><td>25.07 (10)</td><td>25.66 (12)</td><td>27.63 (12)</td></tr><tr><td>MBDE</td><td>19.36 (12)</td><td>13.64 (12)</td><td>11.36 (13)</td><td>11.62 (13)</td></tr></tbody></table>

The results clearly reveal a hierarchy of performance in terms of predictive quality, parsimony, convergence speed and adaptability across datasets and learning algorithms.

A first general observation suggests that wrapper methods overall outperform filter methods in predictive quality regardless of the learning algorithm. This is not surprising in light of the literature, since wrappers directly integrate the model's predictive performance into the selection process, whereas filters rely on statistical criteria that are independent of the model. However, this performance gain comes at the cost of significantly higher runtime, illustrating the trade-off between speed (favouring filters) and problem-specific adequacy (favouring wrappers), as shown in Tables 4 and 6.

The analysis indicates that the impact of specificities such as redundancy, imbalance and noise on the overall ranking of methods remains moderate. By contrast, dataset size (number of instances times number of features) and the choice of learning algorithm exert a notable influence.

In particular, examining the GNB results, we observe that strategies prioritising exploration, such as DE/rand/1, perform better on small datasets, whereas exploitation-oriented strategies such as DE/best/1 dominate on larger datasets (see Appendix A). The latter clearly stands at the top of the overall ranking, with $\Delta_{accuracy} \in [85.91, 95.88]\%$ (see Table 4), indicating that it is often the method that yields the best predictive quality. Nevertheless, the extent of superiority varies across selection and learning methods, as detailed in the following subsections.

### 3.1.1. *Filter methods*

Filter methods, while attractive for their simplicity and low runtime, struggle to match wrappers on complex datasets. For example, ANOVA quickly attains good results on simple datasets, often in less than a second on benchmarks *A*, *B* and *C*. However, on more complex real-world datasets, performance tends to plateau (see
Appendix A). Moreover, MRMR and SURF, which are meant to better handle redundancy, do not exhibit the expected gains over ANOVA in terms of predictive quality.

Despite these limitations, filters remain superior to random search, which is often at the bottom of the ranking regardless of the learning algorithm (see Table 4). This advantage, coupled with their very low runtime (see Table 6), makes them suitable for performing an initial screening of features before potentially applying more complex methods. When time is at a premium, this speed is a major asset. The trade-offs highlighted in this paper are thus clearly illustrated.

### 3.1.2. Greedy heuristics

The results obtained with SFS indicates notable performance for KNN, where it ranks third overall, above the Genetic Algorithm, with $\Delta_{accuracy}$ of 76.39% versus 65.24% (see Table 4). In addition, across all learning algorithms, it is particularly effective in terms of parsimony, topping the ranking with $\Delta_{parsimony} \in [98.71\%, 99.58\%]$ (see Table 5). Its runtime is relatively low compared with other methods ($\Delta_{speed} \in [97.18, 91.68]$), which makes it a preferred choice when computational resources are limited.

Conversely, SBS shows disappointing performance, with the lowest $\Delta_{accuracy}$ among the selection methods ($\Delta_{accuracy} \in [6.64\%, 18.21\%]$), a marked inability to reduce the number of features effectively ($\Delta_{parsimony} \in [0.42\%, 10.19\%]$), and a convergence speed comparable to the most complex methods ($\Delta_{speed} \in [14.64, 21.26]$) (see Tables 4 to 6). These characteristics call into question the relevance of SBS, particularly for datasets with numerous specificities.

### 3.1.3. Local-search metaheuristics

Local metaheuristics, and TS in particular, prove effective, with TS taking third place in the overall ranking, except for DT where it ranks fifth. This attests to their ability to compete with, and sometimes outperform, more elaborate approaches.

However, despite these strengths, the convergence time of TS and HC, while not excessive, is generally longer than that of filters and does not allow them to reach the predictive quality of DE/best/1 (see Tables 4 to 6).

### 3.1.4. Population-based metaheuristics

Among population-based approaches, Differential Evolution clearly takes the lead across learning algorithms, ranking first ahead of the Genetic Algorithm, which is the most widely known and used method.

These results confirm previous observations in the literature, where Differential Evolution appeared to be the strongest method. However, the choice of strategy seems to have a crucial impact on behaviour. First, MBDE, the recommended variant, is outperformed by DE/best/1 across all metrics. In addition, DE/rand/1 generally sits at the bottom of the overall ranking among metaheuristics. Nevertheless, by fully favouring exploration, it is the most effective method overall in terms of predictive quality on small datasets (Benchmark A, Spambase, etc.), whereas DE/best/1 seems more effective on large datasets.

These observations motivate a further analysis aimed at examining specifically the impact of Differential Evolution's mutation strategies on the results, which is the focus of the next experiment.
## 3.2. Experiment 2: Analysis of DE strategies

<table><caption>Table 7. – Mean relative predictive quality (Balanced Accuracy, %) obtained by each version of Differential Evolution across all datasets, evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td>DE (r/1)</td><td>45.86 (4)</td><td>35.76 (7)</td><td>53.89 (4)</td><td>42.46 (3)</td></tr><tr><td>DE (b/1)</td><td>81.18 (2)</td><td>74.09 (2)</td><td><strong>88.17 (1)</strong></td><td><strong>94.12 (1)</strong></td></tr><tr><td>DE (cr/1)</td><td>39.47 (7)</td><td>34.48 (8)</td><td>32.53 (5)</td><td>32.33 (8)</td></tr><tr><td>DE (cb/1)</td><td>71.32 (3)</td><td>72.67 (3)</td><td>73.16 (2)</td><td>71.89 (2)</td></tr><tr><td>DE (rb/1)</td><td>44.84 (5)</td><td>62.88 (4)</td><td>54.07 (3)</td><td>37.50 (7)</td></tr><tr><td>DE (r/2)</td><td>27.17 (10)</td><td>12.52 (10)</td><td>5.70 (11)</td><td>1.61 (10)</td></tr><tr><td>DE (b/2)</td><td>44.45 (6)</td><td>37.65 (5)</td><td>31.54 (6)</td><td>39.13 (5)</td></tr><tr><td>DE (cr/2)</td><td>26.52 (11)</td><td>12.52 (10)</td><td>7.10 (10)</td><td>1.61 (10)</td></tr><tr><td>DE (cb/2)</td><td>37.14 (8)</td><td>37.65 (5)</td><td>31.54 (6)</td><td>39.04 (6)</td></tr><tr><td>DE (rb/2)</td><td>30.98 (9)</td><td>26.88 (9)</td><td>23.68 (8)</td><td>31.65 (9)</td></tr><tr><td>MBDE</td><td><strong>83.45 (1)</strong></td><td><strong>82.81 (1)</strong></td><td>17.07 (9)</td><td>42.25 (4)</td></tr></tbody></table>

<table><caption>Table 8. – Mean relative parsimony (ratio of selected features, %) obtained by each version of Differential Evolution across all datasets, evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td>DE (r/1)</td><td>26.03 (7)</td><td>29.23 (7)</td><td>57.26 (3)</td><td>47.45 (8)</td></tr><tr><td>DE (b/1)</td><td><strong>91.69 (1)</strong></td><td><strong>76.29 (1)</strong></td><td>56.13 (4)</td><td><strong>64.80 (1)</strong></td></tr><tr><td>DE (cr/1)</td><td>18.52 (11)</td><td>30.96 (5)</td><td>53.88 (5)</td><td>40.02 (10)</td></tr><tr><td>DE (cb/1)</td><td>81.10 (2)</td><td>73.22 (2)</td><td>42.83 (11)</td><td>48.43 (6)</td></tr><tr><td>DE (rb/1)</td><td>32.98 (5)</td><td>35.66 (4)</td><td>49.13 (8)</td><td>48.28 (7)</td></tr><tr><td>DE (r/2)</td><td>20.20 (10)</td><td>16.30 (10)</td><td>43.95 (10)</td><td>39.58 (11)</td></tr><tr><td>DE (b/2)</td><td>23.11 (9)</td><td>23.09 (8)</td><td><strong>61.19 (1)</strong></td><td>53.06 (3)</td></tr><tr><td>DE (cr/2)</td><td>25.32 (8)</td><td>16.30 (10)</td><td>44.24 (9)</td><td>39.58 (11)</td></tr><tr><td>DE (cb/2)</td><td>35.19 (4)</td><td>23.09 (8)</td><td><strong>61.19 (1)</strong></td><td>51.84 (4)</td></tr><tr><td>DE (rb/2)</td><td>32.31 (6)</td><td>29.84 (6)</td><td>50.00 (7)</td><td>51.38 (5)</td></tr><tr><td>MBDE</td><td>69.99 (3)</td><td>58.84 (3)</td><td>52.65 (6)</td><td>58.38 (2)</td></tr></tbody></table>

<table><caption>Table 9. – Mean relative convergence speed (number of iterations required) obtained by each version of Differential Evolution across all datasets, evaluated with GNB, LR, DT and KNN. Ranks are shown in parentheses.</caption><thead><tr><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td>DE (r/1)</td><td>43.81 (5)</td><td>26.84 (9)</td><td>32.12 (10)</td><td>39.95 (7)</td></tr><tr><td>DE (b/1)</td><td>46.88 (3)</td><td>34.04 (7)</td><td>41.68 (3)</td><td>39.07 (9)</td></tr><tr><td>DE (cr/1)</td><td>40.37 (6)</td><td>36.70 (6)</td><td>32.40 (9)</td><td><strong>51.91 (1)</strong></td></tr><tr><td>DE (cb/1)</td><td>33.57 (10)</td><td>33.08 (8)</td><td>39.25 (7)</td><td>22.34 (10)</td></tr><tr><td>DE (rb/1)</td><td>44.36 (4)</td><td>26.69 (10)</td><td>39.51 (6)</td><td>47.51 (2)</td></tr><tr><td>DE (r/2)</td><td>49.30 (2)</td><td><strong>57.90 (1)</strong></td><td><strong>52.90 (1)</strong></td><td>40.38 (5)</td></tr><tr><td>DE (b/2)</td><td>40.28 (7)</td><td>41.72 (5)</td><td>40.51 (4)</td><td>40.71 (4)</td></tr><tr><td>DE (cr/2)</td><td>38.94 (8)</td><td>57.74 (3)</td><td>51.71 (2)</td><td>40.34 (6)</td></tr><tr><td>DE (cb/2)</td><td>34.73 (9)</td><td>42.19 (4)</td><td>40.50 (5)</td><td>46.80 (3)</td></tr><tr><td>DE (rb/2)</td><td><strong>62.15 (1)</strong></td><td>57.83 (2)</td><td>39.07 (8)</td><td>39.44 (8)</td></tr><tr><td>MBDE</td><td>18.99 (11)</td><td>21.26 (11)</td><td>19.61 (11)</td><td>21.78 (11)</td></tr></tbody></table>
This second experiment aims to analyse in detail the impact of different Differential Evolution (DE) strategies in terms of predictive quality, parsimony and convergence speed. The results clearly show substantial variability depending on the chosen strategy.

Regarding predictive quality, a first observation is that DE/x/y strategies with $y = 2$ generally degrade performance relative to their $y = 1$ counterparts. This suggests that adding extra differences in the mutation strategy, although it increases exploration, markedly penalises the accuracy of the obtained solutions.

By contrast, methods that explicitly incorporate the best individual, such as DE/best/1 or DE/current-to-best/1, exhibit superior parsimony, typically selecting fewer features to achieve equal or better predictive quality than other strategies, as shown in Tables 7 and 8. This tendency is especially evident on high-dimensional datasets (Benchmark *B* and real datasets such as Gravier).

In terms of convergence speed, the results suggest less clear separation between DE strategies. The time required to reach the best score remains broadly similar, clustering around comparable values for most approaches (see Table 9). This lack of a pronounced difference indicates that the choice of DE strategy should not be based solely on temporal convergence, but should explicitly consider dataset size.

These findings also confirm the importance of balancing exploration and exploitation. More precisely, looking more closely at the GNB results, strategies that favour exploration, such as DE/rand/1, are particularly suitable for smaller datasets, where extensive search is feasible and beneficial. Conversely, strong exploitation of promising solutions, as with DE/best/1 and MBDE, is needed to achieve high performance on larger and more complex datasets, reinforcing the conclusions of the previous experiment.

From these two in-depth analyses, three key takeaways emerge:

1. The ANOVA filter is a fast and effective method for achieving acceptable predictive quality when computation time is severely limited.

2. The greedy SFS heuristic offers an excellent trade-off in terms of parsimony and predictive quality, particularly relevant when metaheuristics do not have sufficient time to converge.

3. Differential Evolution generally stands out as the most effective metaheuristic, but its performance depends strongly on the chosen strategy, the dataset's specificities and the complexity of the learning algorithm.

In light of these conclusions, developing a hybrid and adaptive method that leverages the combined strengths of fast (ANOVA), parsimonious (SFS) and high-performing (DE) approaches appears to be a natural and promising direction for further improvements.

### 3.3. Tournament in Differential Evolution

We propose a variant of Differential Evolution (DE), called *Tournament in Differential Evolution* (TiDE). This version introduces a specific mutation strategy based on a tournament, an initialisation optimised by a heuristic tailored to the feature selection problem, and a tuned set of heuristic parameters.
### 3.3.1. Adapting the initialisation strategy

Initialising the population is a crucial phase in DE, as in most population-based metaheuristics. Although performed only once, it has a significant impact on the algorithm's course, since it conditions the quality of the individuals for subsequent steps.

In the classical version of DE, the initial population is generated entirely at random. This widely used method is appreciated for its ability to explore the search space uniformly. However, it has limitations, notably the lack of consideration for the structure of the search space or prior knowledge about potentially optimal solutions. This often leads to poorly performing individuals, slowing convergence and reducing the chance of reaching an optimal or satisfactory solution.

To address these limitations in the context of feature selection, we propose incorporating a filter method (ANOVA for classification problem) at initialisation first, then SFS in a second step. This can generate more robust individuals by leveraging prior knowledge of the data or problem. The two individuals obtained by this procedure are then combined with the rest of the population, which is generated at random, in order to keep a balance between diversity and initial quality. By exploiting problem-specific structure, this approach should increase convergence speed and the likelihood of obtaining high-quality solutions from the very first iterations.

### 3.3.2. Adapting the mutation strategy

We also propose to improve the mutation strategy by operating directly on the genes of the population's individuals. In DE, the DE/rand/1 mutation strategy relies on Equation 4, where $F$ controls the amplitude of differences between individuals. Observations from Table 10 indicate that when $F$ is small, i.e. $F < 0.5$, differences between individuals tend to vanish in the newly generated individual. Indeed, the genes of the reference individual $S_{r1}$ are always selected with no significant variation brought by the other individuals, which severely limits exploration of the search space. This limitation is particularly problematic in DE/best/1, where the lack of diversity can cause premature convergence to local optima.

<table><caption>Table 10. – Illustration of the results of DE/x/1 for each possible combination of three individuals $S_{r1}$, $S_{r2}$ and $S_{r3}$ and for different values of $F$ (step 0.25).</caption><thead><tr><th>S<sub>r1</sub></th><th>S<sub>r2</sub></th><th>S<sub>r3</sub></th><th>0.0</th><th>0.25</th><th>0.5</th><th>0.75</th><th>1.0</th><th>1.25</th><th>1.5</th><th>1.75</th><th>2.0</th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>0</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr></tbody></table>

From this observation, the DE/x/1 mutation strategy can be reformulated when $F \ge 0.5$ as follows:

$$S_{i,j} = \begin{cases} S_{r1,j}, & \text{if } S_{r2,j} = S_{r3,j} \\ S_{r2,j}, & \text{otherwise} \end{cases} \qquad (13)$$

This reformulation removes the factor $F$ from the equation while simplifying the
operation. It uses the individuals' genes directly, thereby avoiding a conversion step that is particularly costly in runtime (though often less than model training) when the population size is large.

The results of the previous experiments suggest that choosing the reference individual is a key element for an effective mutation strategy. To strike a better balance between exploration and exploitation, we propose an adaptive strategy that we call DE/t<sub>best</sub>/1. This strategy is based on the entropy present among the population's individuals.

At each generation, we compute the population's global entropy (see Equation 2.5). This value is used to adapt the mutation strategy dynamically according to population diversity. To select the reference individual in the mutation step, we apply tournament selection. This selection compares several individuals chosen at random from the population and selects the best according to predictive quality, denoted t<sub>best</sub>.

When $\bar{H}(P) \approx 1$, high diversity indicates that solutions are widely spread. In this situation, it is preferable to reduce the tournament size to stimulate exploration and avoid being trapped in local optima. Conversely, when $\bar{H}(P) \approx 0$, the population is almost homogeneous, a sign that it has converged towards promising solutions. It is then appropriate to increase the tournament size in order to strengthen exploitation and further optimise these solutions. We determine the number of tournament participants T as follows:

$$p = (1 - \bar{H}(P)) \times (1 - \gamma) + \gamma \quad (14)$$

$$T = \max(2, N \times p) \quad (15)$$

where N is the population size and $\gamma \in [0, 1]$ is a parameter that guarantees a minimal percentage p of participants in the tournament, even under high diversity. Equation 14 performs a linear interpolation between the threshold $\gamma$ and the value $1 - \bar{H}(P)$.

Thus, when $\bar{H}(P)$ is low (homogeneous population), p tends to 1, increasing the number of individuals selected for the tournament and favouring exploitation of promising solutions. In contrast, the higher $\bar{H}(P)$ is (heterogeneous population), the closer p approaches $\gamma$, which limits the tournament size and encourages exploration of new regions of the search space.

When $\gamma = 0$, p decreases strictly with $\bar{H}(P)$, making T highly sensitive to diversity changes. As $\gamma$ increases, the influence of $\bar{H}(P)$ on p gradually diminishes. Finally, for $\gamma = 1$, p remains constant at 1, regardless of diversity. All individuals then always participate in the tournament, making the strategy similar to DE/best/1, which prioritises exploitation from the outset, with a risk of premature convergence.

By steering tournament selection as a function of diversity, the algorithm balances exploration phases (when $\bar{H}(P)$ is high) and exploitation phases (when $\bar{H}(P)$ is low). This entropy-guided adaptive approach aims to optimise the search by regularly exploring new solutions while continually refining those already identified.

### 3.3.3. Adapting the crossover strategy

In DE, the crossover rate CR plays a key role in regulating the balance between exploration and exploitation (see Equation 5). Traditionally, CR is fixed to a constant value, but this rigid approach does not adapt to the population's changing dynamics. To address this limitation, we propose an adaptive method in which $CR(S_i)$ is drawn
at random from a Beta distribution parametrised by the individual's quality $f(S_i)$:

$$ \alpha = (2 - f(S_i)) \times 2 \qquad (16) $$

$$ \beta = (1 + f(S_i)) \times 2 \qquad (17) $$

$$ CR(S_i) = \text{Beta}(\alpha, \beta) \qquad (18) $$

This formulation ensures that $CR$ varies dynamically with the individual's quality. Concretely, individuals with a high score, i.e. $f(S_i) \approx 1$, receive a Beta distribution concentrated towards small values, leading to $CR(S_i)$ that favours exploitation, whereas individuals with a low score, i.e. $f(S_i) \approx 0$, receive a Beta distribution skewed towards large values, associated with exploration. This strategy has two advantages:

(1) **Dynamic balance:** By modulating $CR$ according to individual quality, the algorithm preserves promising solutions while injecting enough diversity to explore new areas of the search space.

(2) **Adaptability:** Sampling $CR(S_i)$ per individual allows the algorithm to react to local variations in the search space, favouring intensification in interesting regions and exploration elsewhere.

Note that this strategy (Equation 18) requires a score normalised to [0, 1]. This can be obtained via most classical metrics (including Balanced Accuracy) for classification, or adjusted coefficient of determination ($R_{adj}^2$) for regression.

Our proposal is supported by recent work, such as Cheng et al. (2023), which shows that adapting $CR$ according to quality can effectively balance exploration and exploitation [84]. In addition, similar stochastic distribution-based strategies have been shown to improve diversity and accelerate convergence [85].

**Algorithm 1: Tournament in Differential Evolution**

**Input:** $f$, a fitness function evaluating an individual

$N$, the population size

$\gamma$, the minimum tournament participation threshold

**Output:** $S^*$, the best individual found

1. **Initialisation:** $P \leftarrow$ a population of $N$ individuals;

2. **while** the stopping criterion is not met **do**

3. **for** $i$ from 1 to $N$ **do**

4. $M_i \leftarrow$ mutation of $S_i \in P$ according to $\gamma$, $\bar{H}(P)$ and $DE/t_{best}/1$;

5. $j_{rand} \leftarrow$ a random gene index $j$;

6. **for each gene j do**

7. **if** $j = j_{rand}$ **or** with probability $CR(S_i)$ **then**

8. $| E_{i,j} \leftarrow M_{i,j} $;

9. **else**

10. $| E_{i,j} \leftarrow S_{i,j} $;

11. **end**

12. **end**

13. **if** $f(E_i)$ is better than $f(S_i)$ **then**

14. $| S_i \leftarrow E_i $;

15. **end**

16. **end**

17. **end**

18. $S^* \leftarrow$ the best individual in $P$;

19. **return** $S^*$
### 3.4. Experiment 3: Evaluating TiDE

<table><caption>Table 11. – Mean relative predictive quality (Balanced Accuracy, %) obtained by each method on the benchmark, evaluated with GNB, LR, DT and KNN.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>46.73 (19)</td><td>50.26 (19)</td><td>42.17 (19)</td><td>63.37 (8)</td></tr><tr><td>MRMR</td><td>45.36 (20)</td><td>44.77 (20)</td><td>40.75 (20)</td><td>62.93 (9)</td></tr><tr><td>SURF</td><td>37.80 (21)</td><td>31.46 (21)</td><td>37.20 (21)</td><td>53.11 (11)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>22.90 (22)</td><td>25.17 (22)</td><td>28.43 (22)</td><td>27.92 (22)</td></tr><tr><td>SFS</td><td>53.20 (16)</td><td>67.02 (11)</td><td>68.48 (7)</td><td>72.49 (5)</td></tr><tr><td>SBS</td><td>14.11 (23)</td><td>18.01 (23)</td><td>6.41 (23)</td><td>9.25 (23)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>66.63 (9)</td><td>70.56 (10)</td><td>48.52 (18)</td><td>46.09 (17)</td></tr><tr><td>TS</td><td>85.98 (6)</td><td>83.95 (6)</td><td>62.89 (10)</td><td>73.05 (4)</td></tr><tr><td rowspan="20">Wrapper Population</td><td>GA</td><td>76.25 (7)</td><td>75.94 (9)</td><td>71.64 (6)</td><td>62.56 (10)</td></tr><tr><td>PBIL</td><td>74.63 (8)</td><td>81.69 (7)</td><td>76.91 (4)</td><td>63.60 (7)</td></tr><tr><td>DE (r/1)</td><td>60.79 (13)</td><td>64.81 (15)</td><td>66.60 (9)</td><td>48.74 (14)</td></tr><tr><td>DE (b/1)</td><td>91.25 (3)</td><td>92.95 (4)</td><td>81.22 (3)</td><td>81.91 (3)</td></tr><tr><td>DE (cr/1)</td><td>60.25 (14)</td><td>64.92 (14)</td><td>59.66 (13)</td><td>43.95 (18)</td></tr><tr><td>DE (cb/1)</td><td>89.40 (5)</td><td>93.19 (3)</td><td>72.80 (5)</td><td>70.58 (6)</td></tr><tr><td>DE (rb/1)</td><td>65.86 (10)</td><td>76.90 (8)</td><td>67.20 (8)</td><td>49.14 (13)</td></tr><tr><td>DE (r/2)</td><td>51.61 (17)</td><td>53.48 (17)</td><td>52.76 (17)</td><td>35.48 (20)</td></tr><tr><td>DE (b/2)</td><td>62.50 (11)</td><td>66.72 (12)</td><td>59.88 (11)</td><td>46.36 (15)</td></tr><tr><td>DE (cr/2)</td><td>51.21 (18)</td><td>53.48 (17)</td><td>53.11 (16)</td><td>35.48 (20)</td></tr><tr><td>DE (cb/2)</td><td>61.75 (12)</td><td>66.72 (12)</td><td>59.88 (11)</td><td>46.29 (16)</td></tr><tr><td>DE (rb/2)</td><td>54.78 (15)</td><td>59.55 (16)</td><td>56.87 (14)</td><td>43.66 (19)</td></tr><tr><td>MBDE</td><td>90.87 (4)</td><td>89.13 (5)</td><td>55.68 (15)</td><td>50.69 (12)</td></tr><tr><td>TiDE</td><td>96.35 (2)</td><td>97.68 (2)</td><td>92.53 (2)</td><td>98.46 (2)</td></tr><tr><td>TiDE OPT</td><td>100.00 (1)</td><td>99.57 (1)</td><td>95.86 (1)</td><td>98.78 (1)</td></tr></tbody></table>

<table><caption>Table 12. – Mean relative parsimony (ratio of selected features, %) obtained by each method on the benchmark, evaluated with GNB, LR, DT and KNN.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>83.50 (3)</td><td>77.15 (5)</td><td>89.15 (3)</td><td>87.10 (2)</td></tr><tr><td>MRMR</td><td>88.44 (2)</td><td>79.32 (2)</td><td>91.89 (2)</td><td>86.12 (3)</td></tr><tr><td>SURF</td><td>79.50 (4)</td><td>68.52 (9)</td><td>86.00 (4)</td><td>85.68 (6)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>64.94 (12)</td><td>57.37 (20)</td><td>50.64 (21)</td><td>73.90 (7)</td></tr><tr><td>SFS</td><td>98.93 (1)</td><td>98.71 (1)</td><td>99.58 (1)</td><td>99.44 (1)</td></tr><tr><td>SBS</td><td>10.73 (23)</td><td>6.88 (23)</td><td>0.42 (23)</td><td>1.41 (23)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>69.25 (9)</td><td>64.02 (12)</td><td>38.31 (22)</td><td>58.58 (21)</td></tr><tr><td>TS</td><td>68.44 (10)</td><td>73.75 (6)</td><td>51.46 (20)</td><td>63.82 (9)</td></tr><tr><td rowspan="20">Wrapper Population</td><td>GA</td><td>61.56 (13)</td><td>66.92 (10)</td><td>52.58 (19)</td><td>56.40 (22)</td></tr><tr><td>PBIL</td><td>51.44 (14)</td><td>60.03 (13)</td><td>55.33 (7)</td><td>60.72 (13)</td></tr><tr><td>DE (r/1)</td><td>48.71 (18)</td><td>59.53 (16)</td><td>55.00 (10)</td><td>62.84 (11)</td></tr><tr><td>DE (b/1)</td><td>71.73 (7)</td><td>72.36 (7)</td><td>53.40 (14)</td><td>64.11 (8)</td></tr><tr><td>DE (cr/1)</td><td>47.32 (21)</td><td>59.76 (15)</td><td>54.10 (11)</td><td>60.37 (17)</td></tr><tr><td>DE (cb/1)</td><td>70.39 (8)</td><td>70.48 (8)</td><td>53.23 (16)</td><td>62.03 (12)</td></tr><tr><td>DE (rb/1)</td><td>50.84 (15)</td><td>59.95 (14)</td><td>53.84 (13)</td><td>60.29 (15)</td></tr><tr><td>DE (r/2)</td><td>46.52 (22)</td><td>55.14 (21)</td><td>53.23 (16)</td><td>58.89 (19)</td></tr><tr><td>DE (b/2)</td><td>48.02 (19)</td><td>57.39 (18)</td><td>55.01 (8)</td><td>60.76 (14)</td></tr><tr><td>DE (cr/2)</td><td>47.83 (20)</td><td>55.14 (21)</td><td>53.25 (15)</td><td>58.89 (19)</td></tr><tr><td>DE (cb/2)</td><td>49.79 (16)</td><td>57.39 (18)</td><td>55.01 (8)</td><td>60.61 (16)</td></tr><tr><td>DE (rb/2)</td><td>48.83 (17)</td><td>58.18 (17)</td><td>53.05 (18)</td><td>60.94 (18)</td></tr><tr><td>MBDE</td><td>66.06 (11)</td><td>66.32 (11)</td><td>53.96 (12)</td><td>63.08 (10)</td></tr><tr><td>TiDE</td><td>77.85 (5)</td><td>77.69 (4)</td><td>82.58 (5)</td><td>85.93 (4)</td></tr><tr><td>TiDE OPT</td><td>76.47 (6)</td><td>79.29 (3)</td><td>79.84 (6)</td><td>85.83 (5)</td></tr></tbody></table>
<table><caption>Table 13. – Mean relative convergence speed (number of iterations required) obtained by each method on the benchmark, evaluated with GNB, LR, DT and KNN.</caption><thead><tr><th>Type</th><th>Selection Method</th><th>GNB</th><th>LR</th><th>DT</th><th>KNN</th></tr></thead><tbody><tr><td rowspan="3">Filter</td><td>ANOVA</td><td>100.00 (1)</td><td>99.97 (2)</td><td>99.99 (1)</td><td>100.00 (1)</td></tr><tr><td>MRMR</td><td>99.99 (2)</td><td>99.98 (1)</td><td>99.99 (2)</td><td>99.97 (2)</td></tr><tr><td>SURF</td><td>99.96 (3)</td><td>99.37 (3)</td><td>99.94 (3)</td><td>99.94 (3)</td></tr><tr><td rowspan="3">Wrapper Basic</td><td>RAND</td><td>50.55 (6)</td><td>71.46 (5)</td><td>36.30 (10)</td><td>58.22 (5)</td></tr><tr><td>SFS</td><td>97.18 (4)</td><td>91.70 (4)</td><td>94.15 (4)</td><td>92.52 (4)</td></tr><tr><td>SBS</td><td>50.78 (5)</td><td>36.97 (7)</td><td>60.28 (6)</td><td>40.56 (9)</td></tr><tr><td rowspan="2">Wrapper Local</td><td>HC</td><td>45.13 (7)</td><td>40.26 (6)</td><td>42.03 (9)</td><td>51.63 (6)</td></tr><tr><td>TS</td><td>14.71 (23)</td><td>22.42 (17)</td><td>42.47 (8)</td><td>32.92 (11)</td></tr><tr><td rowspan="21">Wrapper Population</td><td>GA</td><td>30.26 (14)</td><td>27.21 (10)</td><td>15.14 (22)</td><td>28.26 (15)</td></tr><tr><td>PBIL</td><td>43.19 (8)</td><td>29.05 (9)</td><td>30.57 (11)</td><td>26.74 (18)</td></tr><tr><td>DE (r/1)</td><td>29.20 (18)</td><td>11.95 (23)</td><td>18.40 (20)</td><td>27.31 (16)</td></tr><tr><td>DE (b/1)</td><td>34.88 (11)</td><td>25.70 (11)</td><td>26.15 (14)</td><td>29.28 (13)</td></tr><tr><td>DE (cr/1)</td><td>28.97 (19)</td><td>18.30 (19)</td><td>17.65 (21)</td><td>33.27 (10)</td></tr><tr><td>DE (cb/1)</td><td>29.30 (17)</td><td>23.21 (14)</td><td>22.87 (16)</td><td>14.99 (22)</td></tr><tr><td>DE (rb/1)</td><td>31.99 (12)</td><td>15.02 (20)</td><td>23.62 (15)</td><td>30.64 (12)</td></tr><tr><td>DE (r/2)</td><td>31.78 (13)</td><td>23.96 (13)</td><td>29.22 (12)</td><td>18.58 (21)</td></tr><tr><td>DE (b/2)</td><td>29.88 (16)</td><td>22.76 (16)</td><td>21.42 (17)</td><td>26.11 (19)</td></tr><tr><td>DE (cr/2)</td><td>26.45 (21)</td><td>23.98 (12)</td><td>28.43 (13)</td><td>18.58 (20)</td></tr><tr><td>DE (cb/2)</td><td>26.80 (20)</td><td>22.85 (15)</td><td>21.40 (18)</td><td>29.22 (14)</td></tr><tr><td>DE (rb/2)</td><td>39.74 (10)</td><td>30.62 (8)</td><td>20.68 (19)</td><td>27.12 (17)</td></tr><tr><td>MBDE</td><td>19.46 (22)</td><td>14.67 (21)</td><td>11.90 (23)</td><td>13.78 (23)</td></tr><tr><td>TiDE</td><td>41.39 (9)</td><td>13.86 (22)</td><td>63.73 (5)</td><td>40.97 (8)</td></tr><tr><td>TiDE OPT</td><td>29.91 (15)</td><td>21.93 (18)</td><td>43.42 (7)</td><td>43.87 (7)</td></tr></tbody></table>

In this third experiment, we deepen the evaluation by integrating TiDE. We proceed in two distinct steps for TiDE. First, TiDE is run with a fixed parameter $\gamma = 0.75$, with the other hyperparameters and the population reinitialisation mechanism set identically to the previously evaluated metaheuristics. We then study the influence of $\gamma$ by varying it between 0 and 1 (step 0.2), recording the best performance across all $\gamma$ values as *TiDE OPT*.

The results highlight several notable advantages of TiDE. First, TiDE clearly outperforms the DE (b/1) strategy, previously identified as the best Differential Evolution variant and overall the best method in terms of predictive quality. Indeed, TiDE achieves $\Delta_{accuracy} \in [92.53, 98.50]\%$ compared with $\Delta_{accuracy} \in [81.22, 92.85]\%$ for DE (b/1) (see Table 11), indicating greater robustness with respect to the choice of learning algorithm used to build the model.

As regards parsimony, TiDE also delivers strong results, ranking at the top among the evaluated metaheuristics. This particularly notable performance is attributable to its SFS-based initialisation, which enables the search to start directly from solutions that are both predictively strong and parsimonious. These outcomes are obtained with a convergence speed often similar or even superior to other metaheuristics (see Tables 12 and 13).

Another practical advantage of TiDE lies in its small number of hyperparameters to configure, which makes it considerably easier to use than other metaheuristics that are often more complex to tune.

A detailed analysis indicates that this adaptive approach, combining the strengths of different method families, yields highly competitive performance. However, to fully realise TiDE’s potential, tuning the parameter $\gamma$ to the specific characteristics of each
dataset appears essential. The results indicate that this optimisation places TiDE in
first position in predictive quality for virtually all dataset-learner combinations. This
shows that, even though TiDE already dominates the evaluated approaches, there
remains significant headroom available through appropriate parametrisation.

Based on our experimental observations, we recommend choosing a $\gamma$ close to 0 when the time budget allows the algorithm to converge fully. Conversely, when time is limited, a $\gamma$ closer to 1 seems preferable in order to leverage existing solutions quickly, thereby speeding convergence towards strong solutions.

Finally, although TiDE is not universally optimal for all metrics, in keeping with
the No Free Lunch theorem, it offers the best overall compromise between predictive
quality, parsimony and convergence speed, which fully justifies its use across a wide
range of practical scenarios.

4. Synthesis and Discussion

We presented an in-depth comparative study of diverse feature selection methods, across realistic and varied settings, to assess their robustness. To this end, and in a rigorous scientific manner, we carried out an experimental analysis based on synthetic and reference datasets. This approach enabled us to evaluate the strengths, limitations and domains of applicability of each feature selection method under multiple criteria (predictive quality, parsimony, convergence speed and extensibility), and to interpret the results while accounting for the complexity and variability of real-world problems.

In the literature, classical feature selection approaches (filter methods and greedy heuristics) are often recommended because of their simplicity of implementation and, or their speed. However, our results suggest that, although they deliver solutions quickly, their performance remains limited. These traditional methods are frequently outperformed by more advanced strategies, such as metaheuristics, in most of the cases we studied.

These findings confirm that feature selection is often underestimated, even though it plays a central role in improving both predictive quality and parsimony. Reducing the number of relevant features promotes performance and interpretability, two aspects that are particularly crucial in sensitive domains where transparency is paramount. The substantial gap between filter methods and metaheuristics underlines the importance of accounting for interactions between features. Even when learning algorithms include intrinsic selection mechanisms, a preparatory feature selection step remains beneficial to eliminate non-informative signals and to strengthen robustness.

Historically, the Genetic Algorithm (GA) has occupied a central place in combinatorial optimisation, mainly thanks to its conceptual simplicity, flexibility and ability to explore vast search spaces. Nevertheless, our results highlight its limitations. Despite reasonable hyperparameter tuning, it appears less effective than other metaheuristics such as Tabu Search, Differential Evolution (DE) and TiDE, especially in terms of predictive quality. We therefore encourage reconsidering the systematic use of GA, particularly when there exist alternatives that are more effective, stable, and that often require fewer parameters to fine-tune.

While DE emerges as the strongest approach among those tested for feature selection, not all of its variants are equal. The DE/rand/1 strategy, although simple, exhibits slow convergence even with very fast learners, which hinders its practical use. In contrast, DE/best/1, which is more exploitation oriented, converges much faster and consequently achieves better predictive quality under a limited time budget. TiDE
goes further by introducing mutation and crossover strategies that are better tailored
to the task, and by leveraging the initial contribution of simpler methods to steer the
search towards relevant feature subsets.

The effectiveness of DE and TiDE appears to stem from their ability to maintain
a subtle balance between exploration and exploitation. Their mutation and crossover
operators, which we adapted specifically for feature selection, help avoid premature
convergence while focusing the search on the most promising regions of the space.
It is worth noting that DE is traditionally seldom used for problems with a binary
search space, which makes these results even more significant in the context of feature
selection.

However, because of the stochastic nature of these metaheuristic approaches, it
remains difficult to pinpoint a single reason for their success. We can only offer con-
jectures: the use of more flexible mutation operators, the implicit consideration of
correlations between features via statistical mechanisms, or the ability to preserve
sufficient diversity within the population. By comparison, greedy or purely filtering
methods seem less able to maintain this delicate balance and are therefore more sen-
sitive to the introduction of one or more data specificities, such as noise, complex
correlations or severe class imbalance.

That said, no method is a universal remedy. Metaheuristics, including TiDE, often
require a substantial time budget and judicious hyperparameter tuning that takes into
account the problem's characteristics, such as dataset structure. The No Free Lunch
theorem emphasises that there is no algorithm that is superior in all circumstances,
and our results support this principle. Certain data structures favour one method over
another.

Consequently, no method systematically outperforms all others in every context
without prior hyperparameter optimisation. The general trend is that population-
based metaheuristics, and TiDE in particular, stand out when the goal is to balance
accuracy, parsimony, robustness and extensibility. Admittedly, TiDE may require
more time on the most challenging problems, but this computational demand is often
justifiable in industrial or scientific contexts where model quality is the priority. If
time and resources are strictly limited, a fast filter combined with a simple heuristic
will yield an approximate solution. Otherwise, investing in an adaptive evolutionary
method such as TiDE provides better control over the trade-offs between quality, cost,
interpretability and stability, thereby addressing the multiple challenges inherent in
feature selection for machine learning.

These findings open up several research avenues. On the one hand, designing more
targeted synthetic datasets, highlighting specific characteristics (noise, correlations,
structural complexity), would allow a deeper understanding of the strengths and weak-
nesses of each feature selection method. On the other hand, one could consider com-
bining the learning phase and feature selection directly within a single metaheuristic
process, for example via genetic programming to generate simultaneously the struc-
ture of a decision tree or a parametric model. Such integration could reduce the need
for sequential steps and improve the overall efficiency of learning. In addition, greater
resources could be allocated to population-based methods through extensive paral-
lelisation. This would not only enable the exploration of larger populations, offering
better diversity and stability, but also accelerate convergence towards higher quality
solutions.

In summary, our work invites a reconsideration of conventional approaches to fea-
ture selection. The use of metaheuristics, and particularly adapted variants such as
TiDE, appears promising for obtaining more pertinent selections and improving ro-
bustness, without ignoring their constraints. Combining filter-based preselection with elaborate metaheuristic strategies is a compelling way to combine execution speed, result quality and adaptability to problem specificities. Moreover, we have already deployed TiDE successfully in a medical setting, notably to predict the survival of patients with Amyotrophic Lateral Sclerosis. This application led to a significant improvement in predictive quality, particularly for estimating one-year survival probability from clinical data collected at intake [86]. Nevertheless, since no algorithm can be considered a universal solution in the long run, the key lies in a better understanding of each method's strengths and weaknesses, together with thoughtfully constructed benchmarks, in order to guide an informed choice of the most appropriate feature selection method.

## 5. Data and code availability

All experiments in this study were conducted on a Windows 11, 64-bit machine (version 23H2, build 22631.4460). The hardware configuration comprises an Intel(R) Core(TM) i7-10700KF processor with 16 cores, 32 GB of RAM, and an NVIDIA GeForce RTX 3060 Ti GPU. This setup enabled the efficient execution of all experimental stages, from data preparation to model training and evaluation. The source code, the data used to implement the feature selection algorithms, and the complete set of results are available on GitHub [87].

## Bibliography

[1] Ling Wang et al. "Feature selection based on meta-heuristics for biomedicine". In: *Optimization Methods and Software* 29.4 (2014), pp. 703-719. DOI: 10.1080/10556788.2013.834900.

[2] R. Stuart Geiger et al. "Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from?". In: *Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency*. FAT* '20. ACM, Jan. 2020. DOI: 10.1145/3351095.3372862. URL: http://dx.doi.org/10.1145/3351095.3372862.

[3] Mario R. Guarracino et al. "Classification of cancer cell death with spectral dimensionality reduction and generalized eigenvalues". In: *Artif. Intell. Medicine* 53.2 (2011), pp. 119-125. DOI: 10.1016/J.ARTMED.2011.07.002. URL: https://doi.org/10.1016/j.artmed.2011.07.002.

[4] Cynthia Rudin. "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead". In: *Nature Machine Intelligence* 1.5 (May 2019), pp. 206-215.

[5] Jeremy Petch, Shuang Di, and Walter Nelson. "Opening the Black Box: The Promise and Limitations of Explainable Machine Learning in Cardiology". In: *Canadian Journal of Cardiology* 38.2 (2022). Focus Issue: New Digital Technologies in Cardiology, pp. 204-213. ISSN: 0828-282X. DOI: https://doi.org/10.1016/j.cjca.2021.09.004. URL: https://www.sciencedirect.com/science/article/pii/S0828282X21007030.

[6] Hiroshi Motoda and Huan Liu. "Feature selection, extraction and construction". In: *Communication of IICM (Institute of Information and Computing Machinery, Taiwan)* 5 (Jan. 2002), pp. 67-72.

[7] Ron Kohavi and George H. John. "Wrappers for feature subset selection". In: *Artificial Intelligence* 97.1 (1997). Relevance, pp. 273-324. ISSN: 0004-3702.
[8] Avrim L. Blum and Pat Langley. "Selection of relevant features and examples in machine learning". In: *Artificial Intelligence* 97.1 (1997). Relevance, pp. 245–271. ISSN: 0004-3702. DOI: https://doi.org/10.1016/S0004-3702(97)00063-5. URL: https://www.sciencedirect.com/science/article/pii/S0004370297000635.

[9] Bin Chen, Jiarong Hong, and Yadong Wang. "The minimum feature subset selection problem". In: *Journal of Computer Science and Technology* 12.2 (Mar. 1997), pp. 145–153. ISSN: 1860-4749. DOI: 10.1007/BF02951333. URL: https://doi.org/10.1007/BF02951333.

[10] Guolin Ke et al. "LightGBM: A Highly Efficient Gradient Boosting Decision Tree". In: *Advances in Neural Information Processing Systems 30 (NIP 2017)*. Dec. 2017. URL: https://www.microsoft.com/en-us/research/publication/lightgbm-a-highly-efficient-gradient-boosting-decision-tree/.

[11] Tansel Dokeroglu et al. "A survey on new generation metaheuristic algorithms". In: *Computers & Industrial Engineering* 137 (2019), p. 106040. ISSN: 0360-8352. DOI: https://doi.org/10.1016/j.cie.2019.106040. URL: https://www.sciencedirect.com/science/article/pii/S0360835219304991.

[12] Kashif Hussain et al. "Metaheuristic research: a comprehensive survey". In: *Artificial Intelligence Review* 52.4 (Dec. 2019), pp. 2191–2233. ISSN: 1573-7462. DOI: 10.1007/s10462-017-9605-z. URL: https://doi.org/10.1007/s10462-017-9605-z.

[13] P. M. Narendra and K. Fukunaga. "A Branch-and-Bound Algorithm for Feature Subset Selection". In: *IEEE Transactions on Computers* 39.9 (1990). Décrit un cadre branch-and-bound pour explorer exhaustivement les sous-ensembles de variables tout en coupant les branches non prometteuses., pp. 1000–1003.

[14] Jiachang Liu, Soroosh Shafiee, and Andrea Lodi. "Scalable First-Order Method for Certifying Optimal *k*-Sparse GLMs". In: *arXiv preprint arXiv:2502.09502* (2025). Améliore calculs de bornes duales dans un cadre BnB pour GLMs parcimonieux.

[15] I. O. Oduntana et al. "A multilevel tabu search algorithm for the feature selection problem in biomedical data". In: *Computers & Mathematics with Applications* 55 (2008), pp. 1019–1033.

[16] Miguel García-Torres et al. "A multistart tabu search-based method for feature selection in medical datasets". In: *Scientific Reports* 13 (2023). Combina TS et multistart pour un wrapper sur plusieurs bases médicales, p. 12345. DOI: 10.1038/s41598-023-44437-4.

[17] Anjum Nazir and Rizwan Ahmed Khan. "Combinatorial Optimization based Feature Selection Method: A study on Network Intrusion Detection". In: *arXiv preprint arXiv:1906.04494* (2019). Propose TS-RF, une combinaison de Tabu Search et Random Forest pour la détection d'intrusions.

[18] R. M. Balabina and S. V. Smirnov. "Variable selection in near-infrared spectroscopy: Benchmarking of feature selection methods on biodiesel data". In: *Analytica Chimica Acta* 692 (2011), pp. 63–72.

[19] Kushal Kanti Ghosh et al. "Atom Search Optimization with Simulated Annealing: A Hybrid Metaheuristic for Feature Selection". In: *arXiv preprint arXiv:2005.08642* (2020). ASO-SA, un hybride Atom Search + SA évalué sur 25 jeux de données.

[20] L. Y. Chuang et al. "Improved binary PSO for feature selection using gene expression data". In: *Computers in Biology and Chemistry* 32 (2008), pp. 29–38.

[21] L. Y. Chuang, C. H. Yang, and J. C. Li. "Chaotic maps based on binary particle swarm optimization for feature selection". In: *Applied Soft Computing* 11 (2011), pp. 239–248.

[22] L. Chuang, S. Tsai, and C. Yang. "Improved binary particle swarm optimization using catfish effect for feature selection". In: *Expert Systems with Applications* 38 (2011), pp. 12699–12707.

[23] A. Unler, A. Murat, and R. B. Chinnam. "mr2PSO: A maximum relevance minimum redundancy feature selection method based on swarm intelligence for support vector machine classification". In: *Information Sciences* 181 (2011), pp. 4625–4641.

[24] M. H. Aghdam, N. Ghasem-Aghaee, and M. E. Basiri. "Text feature selection using ant colony optimization". In: *Expert Systems with Applications* 36 (2009), pp. 6843–6853.
[25] B. L. Chen, L. Chen, and Y. X. Chen. "Efficient ant colony optimization for image feature selection". In: *Signal Processing* 93 (2013), pp. 1566-1576.

[26] H. Huang et al. "Ant colony optimization-based feature selection method for surface electromyography signals classification". In: *Computers in Biology and Medicine* 42 (2012), pp. 30-38.

[27] Jie Xiao, Ming Li, and Yong He. "An Enhanced Ant Colony Optimization for High-Dimensional Feature Selection". In: *Information Sciences* 577 (2022). Amélioration de ACO pour FS sur données très dimensionnelles, pp. 638-652. DOI: 10.1016/j.ins.2021.08.012.

[28] A. L. I. Oliveira et al. "GA-based method for feature selection and parameters optimization for machine learning regression applied to software effort estimation". In: *Information and Software Technology* 52 (2010), pp. 1155-1166.

[29] S. K. Saha, S. Sarkar, and P. Mitra. "Feature selection techniques for maximum entropy based biomedical named entity recognition". In: *Journal of Biomedical Informatics* 42 (2009), pp. 905-911.

[30] M. M. Kabir, M. Shahjahan, and K. Murase. "A new local search based hybrid genetic algorithm for feature selection". In: *Neurocomputing* 74 (2011), pp. 2914-2928.

[31] M. M. Kabira, M. Shahjahanb, and K. Murase. "A new hybrid ant colony optimization algorithm for feature selection". In: *Expert Systems with Applications* 39 (2012), pp. 3747-3763.

[32] N. Das et al. "A genetic algorithm based region sampling for selection of local features in handwritten digit recognition application". In: *Applied Soft Computing* 12 (2012), pp. 1592-1606.

[33] C. C. O. Ramos, A. N. Souza, and J. P. Papa. "A novel algorithm for feature selection using harmony search and its application for non-technical losses detection". In: *Computers & Electrical Engineering* 37 (2011), pp. 886-894.

[34] Xiaoli Wang, Liang Chen, and Rui Zhang. "Adaptive Harmony Search for Large-Scale Feature Selection". In: *Applied Soft Computing* 101 (2021). HS adaptatif pour FS sur très grands jeux de données, p. 107000. DOI: 10.1016/j.asoc.2020.107000.

[35] L. Wang, Q. Niu, and M. Fei. "A novel quantum ant colony optimization algorithm and its application to fault diagnosis". In: *The Institution of Engineering and Technology—Measurement and Control* 30 (2008), pp. 313-329.

[36] L. Wang, X. T. Wang, and J. S. Yu. "Naphtha cracking furnace fault diagnosis based on adaptive quantum ant colony algorithm". In: *Journal of Chemical Industry and Engineering* 60 (2009), pp. 401-408.

[37] G. X. Zhang et al. "Novel quantum genetic algorithm and its applications". In: *Frontiers of Electrical and Electronic Engineering in China* 1 (2006), pp. 31-36.

[38] Zhen Li, Wei Chen, and Bo Sun. "Quantum-Inspired Evolutionary Algorithm with Dynamic Qubit Rotation for Feature Selection". In: *Expert Systems with Applications* 201 (2022). Rotation de qubit dynamique pour mieux explorer l'espace de FS, pp. 117-135. DOI: 10.1016/j.eswa.2021.117135.

[39] Jing Wang, Hong Li, and Xue He. "Enhanced Differential Evolution for High-Dimensional Feature Selection". In: *Information Sciences* 512 (2020). DE amélioré avec stratégie adaptative pour FS en grande dimension, pp. 1025-1042. DOI: 10.1016/j.ins.2020.08.044.

[40] Mengting Li and Chuang Zhu. "Noisy Label Processing for Classification: A Survey". In: *arXiv preprint arXiv:2404.04159* (2024). Revue des méthodes deep learning pour gérer le bruit d'étiquetage en classification :contentReference[oaicite:0]index=0.

[41] Yuhan Zhou et al. *A Survey on Data Quality Dimensions and Tools for Machine Learning*. 2024. arXiv: 2406.19614 [cs.LG]. URL: https://arxiv.org/abs/2406.19614.

[42] Yiming Zhou, Qian Wang, and Xuan Li. "A Comprehensive Survey on Imbalanced Data Learning". In: *arXiv preprint arXiv:2502.08960* (2025). Analyse structurée des approches de rééchantillonnage et d'apprentissage pour données déséquilibrées :contentReference[oaicite:1]index=1.
[43] D.H. Wolpert and W.G. Macready. "No free lunch theorems for optimization". In: *IEEE Transactions on Evolutionary Computation* 1.1 (1997), pp. 67–82. DOI: 10.1109/4235.585893.

[44] Ron Kohavi and George H. John. "Wrappers for feature subset selection". In: *Artificial Intelligence* 97.1 (1997). Relevance, pp. 273–324. ISSN: 0004-3702. DOI: https://doi.org/10.1016/S0004-3702(97)00043-X. URL: https://www.sciencedirect.com/science/article/pii/S000437029700043X.

[45] Samina Khalid, Tehmina Shehryar, and Shamila Nasreen. "A survey of feature selection and feature extraction techniques in machine learning". In: *Proceedings of 2014 Science and Information Conference, SAI 2014* (Oct. 2014), pp. 372–378. DOI: 10.1109/SAI.2014.6918213.

[46] David C. Howell. *Statistical Methods for Psychology*. Wadsworth Cengage Learning, 2021.

[47] C. Ding and H. Peng. "Minimum redundancy feature selection from microarray gene expression data". In: (2003), pp. 523–528. DOI: 10.1109/CSB.2003.1227396.

[48] Hanchuan Peng, Fuhui Long, and C. Ding. "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy". In: *IEEE Transactions on Pattern Analysis and Machine Intelligence* 27.8 (2005), pp. 1226–1238. DOI: 10.1109/TPAMI.2005.159.

[49] Zhenyu Zhao, Radhika Anand, and Mallory Wang. "Maximum Relevance and Minimum Redundancy Feature Selection Methods for a Marketing Machine Learning Platform". In: (2019), pp. 442–452. DOI: 10.1109/DSAA.2019.00059.

[50] Casey S. Greene et al. "Spatially Uniform ReliefF (SURF) for computationally-efficient filtering of gene-gene interactions". In: *BioData Mining* 2.1 (Sept. 2009), p. 5. ISSN: 1756-0381. DOI: 10.1186/1756-0381-2-5. URL: https://doi.org/10.1186/1756-0381-2-5.

[51] Kenji Kira and Larry A. Rendell. "The Feature Selection Problem: Traditional Methods and a New Algorithm". In: *AAAI Conference on Artificial Intelligence*. 1992. URL: https://api.semanticscholar.org/CorpusID:46457448.

[52] Igor Kononenko. "Estimating attributes: Analysis and extensions of RELIEF". In: *Machine Learning: ECML-94*. Ed. by Francesco Bergadano and Luc De Raedt. Berlin, Heidelberg: Springer Berlin Heidelberg, 1994, pp. 171–182. ISBN: 978-3-540-48365-6.

[53] Marko Robnik-Šikonja and Igor Kononenko. "Theoretical and Empirical Analysis of ReliefF and RReliefF". In: *Machine Learning* 53.1 (Oct. 2003), pp. 23–69. ISSN: 1573-0565. DOI: 10.1023/A:1025667309714. URL: https://doi.org/10.1023/A:1025667309714.

[54] Ryan J. Urbanowicz et al. "Relief-based feature selection: Introduction and review". In: *Journal of Biomedical Informatics* 85 (2018), pp. 189–203. ISSN: 1532-0464. DOI: https://doi.org/10.1016/j.jbi.2018.07.014. URL: https://www.sciencedirect.com/science/article/pii/S1532046418301400.

[55] Jason H. Moore and Bill C. White. "Tuning ReliefF for Genome-Wide Genetic Analysis". In: *Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics*. Ed. by Elena Marchiori, Jason H. Moore, and Jagath C. Rajapakse. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 166–175. ISBN: 978-3-540-71783-6.

[56] George H. John, Ron Kohavi, and Karl Pfleger. "Irrelevant features and the subset selection problem". In: *Machine Learning* 15.2 (1994). Décrit les algorithmes de sélection séquentielle avant, séquentielle arrière et best-first search dans le cadre du wrapper., pp. 101–125.

[57] Pavel Pudil, Jaroslava Novovičová, and Josef Kittler. "Floating Search Methods in Feature Selection". In: *Pattern Recognition Letters* 15.11 (1994). Propose les variantes Sequential Floating Forward Selection (SFFS) et Sequential Floating Backward Selection (SFBS), pp. 1119–1125.

[58] Sheldon H. Jacobson and Enver Yücesan. "Analyzing the Performance of Generalized Hill Climbing Algorithms". In: *Journal of Heuristics* 10 (2004), pp. 387–405. URL: https://api.semanticscholar.org/CorpusID:30814499.
[59] S. Chalup and F. Maire. "A study on hill climbing algorithms for neural network training". In: 3 (1999), 2014-2021 Vol. 3. DOI: 10.1109/CEC.1999.785522.

[60] Huseyin Ceylan and Halim Ceylan. "A Hybrid Harmony Search and TRANSYT hill climbing algorithm for signalized stochastic equilibrium transportation networks". In: *Transportation Research Part C: Emerging Technologies* 25 (Dec. 2012). DOI: 10.1016/j.trc.2012.05.007.

[61] Fred Glover. "Tabu Search—Part I". In: *ORSA Journal on Computing* 1.3 (1989), pp. 190–206. DOI: 10.1287/ijoc.1.3.190. eprint: https://doi.org/10.1287/ijoc.1.3.190. URL: https://doi.org/10.1287/ijoc.1.3.190.

[62] Fred Glover. "Tabu Search—Part II". In: *ORSA Journal on Computing* 2.1 (1990), pp. 4–32. DOI: 10.1287/ijoc.2.1.4. eprint: https://doi.org/10.1287/ijoc.2.1.4. URL: https://doi.org/10.1287/ijoc.2.1.4.

[63] A. Hertz and D. de Werra. "The tabu search metaheuristic: How we used it". In: *Annals of Mathematics and Artificial Intelligence* 1.1 (Sept. 1990), pp. 111–121. ISSN: 1573-7470. DOI: 10.1007/BF01531073. URL: https://doi.org/10.1007/BF01531073.

[64] Fred Glover and Gary A. Kochenberger. *Handbook of metaheuristics*. Kluwer Academic Publishers, 2003.

[65] Melanie Mitchell. *An introduction to genetic algorithms*. The MIT Press, 1996. URL: https://api.semanticscholar.org/CorpusID:25869059.

[66] T. Back. "Selective pressure in evolutionary algorithms: a characterization of selection mechanisms". In: *Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence*. 1994, 57–62 vol.1. DOI: 10.1109/ICEC.1994.350042.

[67] David E. Goldberg and Kalyanmoy Deb. "A Comparative Analysis of Selection Schemes Used in Genetic Algorithms". In: ed. by GREGORY J.E. RAWLINS. Vol. 1. Foundations of Genetic Algorithms. Elsevier, 1991, pp. 69–93. DOI: https://doi.org/10.1016/B978-0-08-050684-5.50008-2. URL: https://www.sciencedirect.com/science/article/pii/B9780080506845500082.

[68] David B. Fogel, Thomas Bäck, and Zbigniew Michalewicz. *Evolutionary computation. vol. 1, basic algorithms and operators*. Institute of Physics Pub, 2000.

[69] Shumeet Baluja. "Population-Based Incremental Learning: A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning". In: 1994. URL: https://api.semanticscholar.org/CorpusID:14799233.

[70] Shumeet Baluja and Rich Caruana. "Removing the Genetics from the Standard Genetic Algorithm". In: *Machine Learning Proceedings 1995*. Ed. by Armand Prieditis and Stuart Russell. San Francisco (CA): Morgan Kaufmann, 1995, pp. 38–46. ISBN: 978-1-55860-377-6. DOI: https://doi.org/10.1016/B978-1-55860-377-6.50014-1. URL: https://www.sciencedirect.com/science/article/pii/B9781558603776500141.

[71] Rainer Storn and Kenneth Price. "Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces". In: *Journal of Global Optimization* 11.4 (Dec. 1997), pp. 341–359. ISSN: 1573-2916. DOI: 10.1023/A:1008202821328. URL: https://doi.org/10.1023/A:1008202821328.

[72] Ji Qiang, Chad Mitchell, and Albert Qiang. "Tuning of an adaptive unified differential evolution algorithm for global optimization". In: *2016 IEEE Congress on Evolutionary Computation (CEC)* (2016), pp. 4061–4068. DOI: 10.1109/CEC.2016.7744305.

[73] G. Pampara, A.P. Engelbrecht, and N. Franken. "Binary Differential Evolution". In: *2006 IEEE International Conference on Evolutionary Computation* (2006), pp. 1873–1879. DOI: 10.1109/CEC.2006.1688535.

[74] Changshou Deng et al. "Novel Binary Encoding Differential Evolution Algorithm". In: *Advances in Swarm Intelligence*. Ed. by Ying Tan et al. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 416–423. ISBN: 978-3-642-21515-5.

[75] Yu Chen, Weicheng Xie, and Xiufen Zou. *A binary differential evolution algorithm learning from explored solutions*. 2014. arXiv: 1401.1124 [cs.NE]. URL: https://arxiv.org/abs/1401.1124.
[76] Hu Peng et al. "Dichotomous Binary Differential Evolution for Knapsack Problems". In: *Mathematical Problems in Engineering* 2016.1 (2016), p. 5732489. DOI: https://doi.org/10.1155/2016/5732489. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1155/2016/5732489. URL: https://onlinelibrary.wiley.com/doi/abs/10.1155/2016/5732489.

[77] Ling Wang et al. "A novel modified binary differential evolution algorithm and its applications". In: *Neurocomputing* 98 (2012). Bio-inspired computing and applications (LSMS-ICSEE '2010), pp. 55-75. ISSN: 0925-2312. DOI: https://doi.org/10.1016/j.neucom.2011.11.033. URL: https://www.sciencedirect.com/science/article/pii/S0925231212004316.

[78] Mark Hopkins et al. *Spambase*. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C53G6X. 1999.

[79] Joaquin Vanschoren et al. "OpenML: networked science in machine learning". In: *SIGKDD Explorations* 15.2 (2013), pp. 49–60. DOI: 10.1145/2641190.2641198. URL: http://doi.acm.org/10.1145/2641190.2641198.

[80] Ryan A. Mhiser. *datamicroarray: A collection of small-sample, high-dimensional microarray data sets to assess machine-learning algorithms and models*. https://github.com/ramhiser/datamicroarray. 2023.

[81] Andrea Bommert et al. "Benchmark for filter methods for feature selection in high-dimensional classification data". In: *Computational Statistics & Data Analysis* 143 (2020), p. 106839. ISSN: 0167-9473. DOI: https://doi.org/10.1016/j.csda.2019.106839. URL: https://www.sciencedirect.com/science/article/pii/S016794731930194X.

[82] Fabian Pedregosa et al. *Scikit-learn: Machine Learning in Python*. 2012. arXiv: 1201.0490. URL: http://arxiv.org/abs/1201.0490.

[83] Claude Elwood Shannon. "A Mathematical Theory of Communication". In: *The Bell System Technical Journal* 27 (1948), pp. 379-423. URL: http://plan9.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf (visited on 04/22/2003).

[84] Lianzheng Cheng et al. "Adaptive differential evolution with fitness-based crossover rate for global numerical optimization". In: *Complex & Intelligent Systems* 10.1 (Feb. 2024), pp. 551-576. ISSN: 2198-6053. DOI: 10.1007/s40747-023-01159-4. URL: https://doi.org/10.1007/s40747-023-01159-4.

[85] Vladimir Stanovov, Lev Kazakovtsev, and Eugene Semenkin. "Crossover Rate Sorting in Adaptive Differential Evolution". In: *Algorithms* 16.3 (2023). ISSN: 1999-4893. DOI: 10.3390/a16030133. URL: https://www.mdpi.com/1999-4893/16/3/133.

[86] Thibault Anani et al. "Feature selection using metaheuristics to predict annual amyotrophic lateral sclerosis progression". In: *Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration* 0.0 (2025), pp. 1-16. DOI: 10.1080/21678421.2025.2522399.

[87] Thibault Anani. *Tournament in Differential Evolution*. 2024. URL: https://github.com/thibaultanani/TiDE.

## Appendix A. Complete results

The Excel workbook, which contains the full set of per-dataset and per-learner scores, is attached to this PDF: *Download the Excel results file*. If your PDF viewer blocks attachments, the same file can be obtained from the project repository [87].