# Ensemble Feature Selection in Binary Machine Learning Classification: A Novel Application of the Evaluation Based on Distance from Average Solution (EDAS) Method

Authors: Dharyll Prince M. Abellana; Robert R. Roxas; Demelo M. Lao; Paula E. Mayol; Sanghyuk Lee
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Combining filters in an ensemble to improve feature selection performance is a growing field in the literature. Current techniques, however, are focused on approaches that suffer from drawbacks such as sensitivity to skewed distribution, among others. To address this gap, this paper investigates the applicability of multiple criteria decision-making in ensemble feature selection. This paper adopts the Evaluation based on Distance from Average Solution (EDAS) method due to its many familiar elements to the feature selection community. An experiment was performed on six datasets and a control group. The paper uses the six datasets as levels of the blocking factor. A negative control group (i.e., no feature selection) was adopted to compare with the proposed algorithm. Results show that the proposed ensemble FS algorithm was able to reduce the dataset without compromising the performance of the classifier. The findings in this study would contribute to the literature in several ways. First, the paper is one of the few works to demonstrate how MCDM can be used in feature selection with promising results. Second, this paper is one of the few works to demonstrate the significance of including datasets as levels of a blocking factor when performing significance testing. Finally, this paper is the first to demonstrate the applicability of EDAS as an ensemble FS algorithm. As such, the findings in this paper could spark the cross-fertilization of feature selection and MCDM.


---

## Body

## 1. Introduction

The curse of dimensionality is a phenomenon that arises when analyzing data in high-dimensional spaces [1]. In classification, its major consequences are poor classification performance, noisy data, and hard-to-generalize results, to name a few [1]. Many scholars consider dimensionality reduction as the most straightforward approach to addressing the curse of dimensionality [2]. In the current literature, there are two major ways for performing dimensionality reduction: (i) feature selection (FS) and (ii) feature projection (FP) [2]. FP transforms data from high-dimensional space to a lower-dimensional space while retaining the relationships between the original features [3]. For example, principal component analysis generates new features using the linear combinations of the original features [3]. While this technique can be useful for summarizing the original features using fewer variables, the generated features are usually not straightforward to interpret. Thus, it is usually not preferred when interpretation is crucial in the modeling process. By contrast, FS selects a subset of the original features that best represents the original features [2]. For example, the features can be ranked according to their association with the class and retained only the top 10% in the reduced dataset. Because this approach only selects a subset of the original features, interpretation of the model would be straightforward as no new features are created [2]. Thus, this approach is preferable in many modeling scenarios. Filter-based FS is a type of FS that relies on the characteristics of the training data to select features. Unlike other FS algorithms, it does not require the training of a classifier to evaluate a feature. This makes filter-based FS algorithms computationally tractable.The most commonly used filter methods include (i) Chi-squareχ2-based ranking, (ii) information gain (IG)-based ranking, (iii) Relief-F algorithm, and (iv) symmetric uncertainty-based ranking [4]. These methods have been successfully applied in many important applications. For example, Rachburee and Punlumjeak [5] used Chi-square and IG in reducing the dataset in training a classifier for predicting student performance. Lin et al. [6] used SU for evaluating the relevance of features with the class labels in biological data analysis. Ullah and Mahmoud [7] used IG in developing a filter-based FS model for anomaly-based intrusion detection systems. Finally, Mursalin et al. [8] used a correlation-based filter in developing an automatic epileptic seizure detection system. The choice of filters allows filter-based FS to be useful for a wide array of applications. While this makes filter-based FS flexible, its evaluation is largely limited to the extent of how a chosen filter captures the characteristics of the data. For example, IG might be better than the correlation for capturing characteristics of some features, while correlation might be better for the other features. Hence, the choice of filter is a crucial aspect of filter-based FS. It is then imperative to include as many filters as possible to better capture the characteristics of the data. In the current literature, scholars tackled this problem using ensemble feature selection. This type of FS adopts an ensemble (or a group) of filters to evaluate the features.While efforts have been made in this area, a significant gap is the lack of a comprehensive framework for combining different filters. Most of the works on this area use simple arithmetic mean for aggregation [9] despite exhibiting several issues, such as sensitivity to skewed distributions. While some works used a weighted average, the weighted average does not have a mechanism for handling contradicting filters (e.g., redundancy vs. relevance). Clearly, there is a compelling need for more systematic aggregation methods that would enable users to understand the feature selection process results better. An approach called multiple criteria decision-making (MCDM) would naturally fill this gap. MCDM methods have been used extensively in relevant fields for choosing the best option among a pool of alternatives, especially when numerous criteria or evaluation metrics contradict each other [10]. For example, when selecting a site for an energy plant, a decision might select among different locations. Choosing the best location depends on several criteria (or metrics) such as site features, rent costs, and population growth [10]. MCDM provides a systematic approach for the decision-maker to select the best option among the alternatives while considering the evaluation metrics. Aside from selecting the best option, MCDM also offers approaches for ranking the alternatives subject to numerous conflicting criteria. For example, Bączkiewicz et al. [11] developed an e-commerce recommender system based on MCDM that ranks products based on their comparisons.Numerous real-world applications (e.g., Naeemah and Wong [12], Nenadić [13], and Karande et al. [14]) demonstrate the capability of MCDM in solving selection and ranking problems. Since feature selection is essentially about selecting the best features that describe the dataset, it can be viewed under the lenses of MCDM. In this case, the features are the decision alternatives, and the filters are the criteria (or evaluation metrics). Since MCDM has extensive literature for solving selection and ranking problems, this would be beneficial for feature selection. This would offer a unified framework for structuring ensemble feature selection problems. Moreover, well-established principles have already been laid out in MCDM to ensure that the solutions are consistent with the problem formulation. FS results would offer a more insightful interpretation than current ensemble FS approaches. While MCDM could be a useful framework for FS, few works explored how it could be used in FS. As such, it is crucial to introduce MCDM into the field using a method that uses familiar elements used in ensemble FS. Among the MCDM approaches, it can be argued that the Evaluation based on Distance from Average Solution (EDAS) method would be familiar in the ensemble FS community. For one, EDAS uses “averages” to arrive at an aggregate score [15, 16]. For another, EDAS ranks a set of alternatives, which happens to be one of the primary goals of filter-based feature selection. Thus, EDAS would provide an easy entry point for introducing MCDM to the ensemble FS literature. Since there is no clear procedure for choosing one MCDM method over another [17], the previously stated advantages would offer a good starting point for integrating MCDM in ensemble FS.This paper demonstrates the applicability of EDAS in performing ensemble FS. The goal of the proposed approach is to reduce binary classification datasets by retaining only a proportion of the features (without significantly compromising classification performance) based on the ranking generated by EDAS. The paper aims to answer the following research questions: (i) How is the performance of the binary classifier affected when reducing the number of features? (ii) How does EDAS compare to the conventional aggregation method (i.e., linear aggregation method [18]) for creating ensembles? and (iii) What is the stability of EDAS as an ensemble FS algorithm? To answer these research questions, a randomized complete block design (RCBD) is used to design the experiment. Due to potential interactions between datasets and classifiers, this paper uses only one classifier (i.e., decision tree classification) to isolate effects that may be attributed to such an interaction. This is admissible since filter-based FS is agnostic to classifiers. The area under the curve (AUC) of the receiver operating characteristic (ROC) curve is used as the response variable, being a suitable metric for binary classification [19, 20]. To properly evaluate the performance of EDAS, it is compared with two control groups – (a) No Feature Selection (negative control group) and (b) using the Linear Aggregation method (positive control group).Recognizing that some datasets might be easier to learn than other datasets, the Training Dataset is used as a blocking factor. As such, five datasets – (i) QSAR Biodegradation (ii), LSVT Voice Rehabilitation, (iii) Musk Version 2, (iv) Ionosphere, and (v) Sonar, Mines vs. Rocks, comprise the levels of the blocking factor. These datasets can be accessed publicly in Dua and Graff [21]. The findings in this study offer several contributions to the current literature. First, the paper is one of the few works to demonstrate how MCDM can be used in FS. Specifically, this paper is the first to demonstrate the applicability of EDAS as an ensemble FS algorithm. Moreover, the results of the study offer a more robust inference due to the adopted experimental design compared to the majority of the existing works, which only used a descriptive approach in reporting the results. The findings in this study offer an avenue for the cross-fertilization of ideas between feature selection and MCDM.


## 2. Related Works

### 2.1. Feature Selection

FS has gained much consideration from scholars due to its promising results [2, 22, 23]. FS refers to the method of finding subsets of the original feature set that efficiently describe the input data while reducing the effects of noise and irrelevant features and still provide good results for the task [2, 22, 23]. Findings in the literature reveal that FS provides several benefits in modeling and analysis. With fewer features than the original configuration, models are much easier to interpret [24]. Similarly, with fewer features, the models become less expensive to train due to less memory and computation cost [25]. Moreover, in some cases, the removal of redundant features increases the predictive accuracy model when applied to untrained data [26]. Hence, FS enables the models to become easier to generalize for real-world applications. FS comprises three types of approaches: (i) filter methods, (ii) wrapper methods, and (iii) embedded methods. Filter methods rely on the general characteristics of the training data to select features [26]. It does not require any induction algorithm. Thus, it is used as a preprocessing step for classification.Since filter methods are agnostic to the classifier, they may not always yield direct improvements to the classification. Wrapper methods involve optimizing a predictor as part of the selection process [26]. In order to find the best features, wrapper methods would repeatedly train a classifier [26]. As such, while wrapper methods are not agnostic to the classifier’s performance, they are more computationally expensive than filter methods [26]. Thus, when significantly many features are involved, filter methods are usually preferred over wrapper methods. Some works have been done to combine wrapper and filter methods in the current literature. For a detailed review, see Venkatesh and Anuradha [27], Chandrashekar and Sahin [28], and Khoshgoftaar et al. [29], among others. Embedded methods can be considered as a combination of filter and wrapper methods. Such methods are “embedded” in that it allows some classification algorithms to use inherent characteristics of the data to select good features during the training process [30]. Among these three FS methods, the filter methods are the most widely studied due to their computational efficiency and ease of implementation, among others.


### 2.2. Filter-Based Feature Selection

Filter-based FS employs the use of filters, which utilize the general characteristics of the data itself to judge the quality of a feature [31]. For example, IG measures the amount of information gained from a feature with respect to a class label [31]. This would be useful when comparing two probability distributions. Similarly, chi-square χ2 measures the relationship between two features (or attributes, in general) [32]. As such, when two features have high similarity, it can be used to eliminate the redundant feature. When a feature is highly similar to the class, it can be used to reason about its relevance. This measure is helpful in cases when the attributes are nominal. When the features are numeric, correlation coefficients such as Pearson r [33], Spearman ρ [33], Kendall τ [34], or point-biserial correlation [35], could be used. In general, this type of FS is termed correlation-based FS as “associations” are used in the decision to retain or discard features [36]. Another type of filter-based FS is “instance-based FS.” One of the most notable examples of this filter-based FS is the Relief-F (RF) algorithm [37]. The RF algorithm uses a probability value that the two instances are from the same or different classes. RF belongs to a larger family of FS algorithms called the “Relief” algorithms. This type of FS is based on the premise that high-quality features should have different values for instances from different classes and similar values for instances from the same class [37].There are other filter-based FS approaches developed in the literature, such as hybrid methods. Nonetheless, the commonality of these algorithms is their aim to select features based on their statistical properties. The selection is usually done by ranking the features based on the scores calculated by the FS algorithm. With this, filter-based FS is essentially independent of the classifier. As such, it is also termed “agnostic to the classifier.” This property makes the filter-based FS approach efficient when dealing with large datasets since knowledge about the classifier performance is not needed to judge the quality of the features. Filter-based FS has been successfully applied in many real-world problems. For instance, Koprinska et al. [37] employed correlation- and instance-based FS for electricity load forecasting. Likewise, Ambusaidi et al. [38] built an intrusion detection system using a filter-based FS approach. Moreover, Guru et al. [2] used filter-based FS for text categorization. While filter-based FS has the advantage over other FS approaches in terms of efficiency as it is independent of the classifier, it can only judge a feature’s quality using a single performance measure. Several filters would be needed to capture more characteristics of a feature. In the current literature, scholars introduce ensemble FS to provide a framework for combining several performance measures and using this combined measure to judge the quality of a feature.


### 2.3. Ensemble Feature Selection

Ensemble learning is a prolific field in ML [39]. It is based on combining multiple models instead of a single model to solve a particular problem [39]. Traditionally, ML approaches use a single learning model to solve a problem. Using ensembles (or combinations of different models), however, has demonstrated its usefulness in improving the performance of several ML algorithms [39, 40]. Such an improvement has been observed in several application areas. For instance, Umer et al. [41] adopted an ensemble learning approach in predicting numeric ratings for Google apps. Xu et al. [42] used an ensemble learning approach for developing a forest fire detection system. Wang et al. [43] used ensemble learning for classifying single-cell data and projection across reference atlases. Ensemble learning is not limited to classification problems only. It has also been adopted in other areas of ML and data mining, such as clustering and FS, among others. The general idea of ensemble learning in FS is to repeat the FS process several times to generate diverse feature selectors and combine their outputs afterward [9]. Various works have shown that integrating ensemble learning is superior to conventional FS in many aspects. The two major improvements offered by ensemble FS are: (i) overcoming the local optimum problem and (ii) stability [9].The key factor for achieving these improvements is the diversity of the individual FS selectors used [9, 39]. Subsequently, diversity can be achieved in three ways: (i) varying the individual filters, (ii) varying the input data, or (iii) both [9]. Another factor that should be considered when developing ensemble FS algorithms is the choice of aggregation function, i.e., how individual FS selectors will be combined. Majority of the works in current literature apply simple operations such as minimum, median, arithmetic mean, and geometric mean [39]. More sophisticated approaches have also been used by some works, such as robust rank aggregation [44] and SVM-Rank [45]. While current approaches are straightforward, several drawbacks can be drawn from their results. For example, the arithmetic mean makes an assumption that all feature selectors will be given equal weights. This is usually not applicable to many data mining problems. While a weighted average can be used instead of the simple arithmetic mean, this still has the drawback of not being able to decide the weights in a more methodically grounded manner. Moreover, there is the possibility that filter selections might have conflicting objectives. For example, a higher correlation between the feature and the class is desired. But between features, the objective reverses, and a lower correlation is desired. These types of problems cannot be handled in a straightforward manner by the existing approaches. While this problem is not well-studied in the context of FS, it has been paid a significant amount of attention in a field known as multiple criteria decision-making.


### 2.4. Ensemble Feature Selection as a Multiple Criteria Decision-Making Problem

MCDM is an important subset of modern decision science (more generally, soft computing) aimed at supporting decision-makers faced with multiple decision criteria and alternatives [46, 47]. The primary use of MCDM is to support a decision-maker in choosing the preferable variant from many possible options, considering a multitude of criteria characterizing the acceptability of individual decision variants [48]. Moreover, they have been developed to designate a preferred alternative, classify alternatives in a small number of alternatives, or rank alternatives in a subjective preference order [49]. MCDM has been used in several decision-making environments such as sustainable renewable energy development [50], forestry [51], tourism and hospitality management [52], assessment of green technology [53], environmental science [54], facility decisions [55], business processes information management [56], product lifecycle analysis [57], systematic failure mode and effects analyses [58], sustainability engineering [47], supplier selection [59], air traffic control [60], urban transport planning [61], flood susceptibility assessment [62], modeling trust propagation and opinion dynamics [63], and information security risk assessment [64].MCDM is useful when decision-makers want to select an alternative out of many possible options with several criteria that might be conflicting. There are several types of problems in MCDM (e.g., prioritization, ranking, and structural modeling) [17]. Among such types, ensemble FS could be most naturally stated as a ranking problem, wherein the features (alternatives) are ranked according to the scores calculated by the filters (criteria). The most widely used ranking algorithms in MCDM include - Preference Ranking Organization Method for Enrichment Evaluations (PROMETHEE), ELimination Et Choix Traduisant la REalite (ELECTRE), Vlsekriterijumska Optimizacija l KOm-promisno Resenje method (VIKOR), and Evaluation based on Distance from Average Solution (EDAS) [17, 65]. While there are various approaches for solving the ranking problem in MCDM, there is no clear procedure yet for choosing one technique over another [17]. Such a decision is made by users commonly through familiarity with the technique in the field, popularity of the technique, and desirability to derive a certain measure available only on a particular method [17, 65]. In ensemble FS, EDAS would offer an easier point of entry for scholars due to the familiarity of the approach. In particular, common ensemble FS approaches make use of “averages” to aggregate individual filter results. The EDAS would be able to address issues with the current approaches while providing an easier point of entry to the field.


## 2.1. Feature Selection

FS has gained much consideration from scholars due to its promising results [2, 22, 23]. FS refers to the method of finding subsets of the original feature set that efficiently describe the input data while reducing the effects of noise and irrelevant features and still provide good results for the task [2, 22, 23]. Findings in the literature reveal that FS provides several benefits in modeling and analysis. With fewer features than the original configuration, models are much easier to interpret [24]. Similarly, with fewer features, the models become less expensive to train due to less memory and computation cost [25]. Moreover, in some cases, the removal of redundant features increases the predictive accuracy model when applied to untrained data [26]. Hence, FS enables the models to become easier to generalize for real-world applications. FS comprises three types of approaches: (i) filter methods, (ii) wrapper methods, and (iii) embedded methods. Filter methods rely on the general characteristics of the training data to select features [26]. It does not require any induction algorithm. Thus, it is used as a preprocessing step for classification.Since filter methods are agnostic to the classifier, they may not always yield direct improvements to the classification. Wrapper methods involve optimizing a predictor as part of the selection process [26]. In order to find the best features, wrapper methods would repeatedly train a classifier [26]. As such, while wrapper methods are not agnostic to the classifier’s performance, they are more computationally expensive than filter methods [26]. Thus, when significantly many features are involved, filter methods are usually preferred over wrapper methods. Some works have been done to combine wrapper and filter methods in the current literature. For a detailed review, see Venkatesh and Anuradha [27], Chandrashekar and Sahin [28], and Khoshgoftaar et al. [29], among others. Embedded methods can be considered as a combination of filter and wrapper methods. Such methods are “embedded” in that it allows some classification algorithms to use inherent characteristics of the data to select good features during the training process [30]. Among these three FS methods, the filter methods are the most widely studied due to their computational efficiency and ease of implementation, among others.


## 2.2. Filter-Based Feature Selection

Filter-based FS employs the use of filters, which utilize the general characteristics of the data itself to judge the quality of a feature [31]. For example, IG measures the amount of information gained from a feature with respect to a class label [31]. This would be useful when comparing two probability distributions. Similarly, chi-square χ2 measures the relationship between two features (or attributes, in general) [32]. As such, when two features have high similarity, it can be used to eliminate the redundant feature. When a feature is highly similar to the class, it can be used to reason about its relevance. This measure is helpful in cases when the attributes are nominal. When the features are numeric, correlation coefficients such as Pearson r [33], Spearman ρ [33], Kendall τ [34], or point-biserial correlation [35], could be used. In general, this type of FS is termed correlation-based FS as “associations” are used in the decision to retain or discard features [36]. Another type of filter-based FS is “instance-based FS.” One of the most notable examples of this filter-based FS is the Relief-F (RF) algorithm [37]. The RF algorithm uses a probability value that the two instances are from the same or different classes. RF belongs to a larger family of FS algorithms called the “Relief” algorithms. This type of FS is based on the premise that high-quality features should have different values for instances from different classes and similar values for instances from the same class [37].There are other filter-based FS approaches developed in the literature, such as hybrid methods. Nonetheless, the commonality of these algorithms is their aim to select features based on their statistical properties. The selection is usually done by ranking the features based on the scores calculated by the FS algorithm. With this, filter-based FS is essentially independent of the classifier. As such, it is also termed “agnostic to the classifier.” This property makes the filter-based FS approach efficient when dealing with large datasets since knowledge about the classifier performance is not needed to judge the quality of the features. Filter-based FS has been successfully applied in many real-world problems. For instance, Koprinska et al. [37] employed correlation- and instance-based FS for electricity load forecasting. Likewise, Ambusaidi et al. [38] built an intrusion detection system using a filter-based FS approach. Moreover, Guru et al. [2] used filter-based FS for text categorization. While filter-based FS has the advantage over other FS approaches in terms of efficiency as it is independent of the classifier, it can only judge a feature’s quality using a single performance measure. Several filters would be needed to capture more characteristics of a feature. In the current literature, scholars introduce ensemble FS to provide a framework for combining several performance measures and using this combined measure to judge the quality of a feature.


## 2.3. Ensemble Feature Selection

Ensemble learning is a prolific field in ML [39]. It is based on combining multiple models instead of a single model to solve a particular problem [39]. Traditionally, ML approaches use a single learning model to solve a problem. Using ensembles (or combinations of different models), however, has demonstrated its usefulness in improving the performance of several ML algorithms [39, 40]. Such an improvement has been observed in several application areas. For instance, Umer et al. [41] adopted an ensemble learning approach in predicting numeric ratings for Google apps. Xu et al. [42] used an ensemble learning approach for developing a forest fire detection system. Wang et al. [43] used ensemble learning for classifying single-cell data and projection across reference atlases. Ensemble learning is not limited to classification problems only. It has also been adopted in other areas of ML and data mining, such as clustering and FS, among others. The general idea of ensemble learning in FS is to repeat the FS process several times to generate diverse feature selectors and combine their outputs afterward [9]. Various works have shown that integrating ensemble learning is superior to conventional FS in many aspects. The two major improvements offered by ensemble FS are: (i) overcoming the local optimum problem and (ii) stability [9].The key factor for achieving these improvements is the diversity of the individual FS selectors used [9, 39]. Subsequently, diversity can be achieved in three ways: (i) varying the individual filters, (ii) varying the input data, or (iii) both [9]. Another factor that should be considered when developing ensemble FS algorithms is the choice of aggregation function, i.e., how individual FS selectors will be combined. Majority of the works in current literature apply simple operations such as minimum, median, arithmetic mean, and geometric mean [39]. More sophisticated approaches have also been used by some works, such as robust rank aggregation [44] and SVM-Rank [45]. While current approaches are straightforward, several drawbacks can be drawn from their results. For example, the arithmetic mean makes an assumption that all feature selectors will be given equal weights. This is usually not applicable to many data mining problems. While a weighted average can be used instead of the simple arithmetic mean, this still has the drawback of not being able to decide the weights in a more methodically grounded manner. Moreover, there is the possibility that filter selections might have conflicting objectives. For example, a higher correlation between the feature and the class is desired. But between features, the objective reverses, and a lower correlation is desired. These types of problems cannot be handled in a straightforward manner by the existing approaches. While this problem is not well-studied in the context of FS, it has been paid a significant amount of attention in a field known as multiple criteria decision-making.


## 2.4. Ensemble Feature Selection as a Multiple Criteria Decision-Making Problem

MCDM is an important subset of modern decision science (more generally, soft computing) aimed at supporting decision-makers faced with multiple decision criteria and alternatives [46, 47]. The primary use of MCDM is to support a decision-maker in choosing the preferable variant from many possible options, considering a multitude of criteria characterizing the acceptability of individual decision variants [48]. Moreover, they have been developed to designate a preferred alternative, classify alternatives in a small number of alternatives, or rank alternatives in a subjective preference order [49]. MCDM has been used in several decision-making environments such as sustainable renewable energy development [50], forestry [51], tourism and hospitality management [52], assessment of green technology [53], environmental science [54], facility decisions [55], business processes information management [56], product lifecycle analysis [57], systematic failure mode and effects analyses [58], sustainability engineering [47], supplier selection [59], air traffic control [60], urban transport planning [61], flood susceptibility assessment [62], modeling trust propagation and opinion dynamics [63], and information security risk assessment [64].MCDM is useful when decision-makers want to select an alternative out of many possible options with several criteria that might be conflicting. There are several types of problems in MCDM (e.g., prioritization, ranking, and structural modeling) [17]. Among such types, ensemble FS could be most naturally stated as a ranking problem, wherein the features (alternatives) are ranked according to the scores calculated by the filters (criteria). The most widely used ranking algorithms in MCDM include - Preference Ranking Organization Method for Enrichment Evaluations (PROMETHEE), ELimination Et Choix Traduisant la REalite (ELECTRE), Vlsekriterijumska Optimizacija l KOm-promisno Resenje method (VIKOR), and Evaluation based on Distance from Average Solution (EDAS) [17, 65]. While there are various approaches for solving the ranking problem in MCDM, there is no clear procedure yet for choosing one technique over another [17]. Such a decision is made by users commonly through familiarity with the technique in the field, popularity of the technique, and desirability to derive a certain measure available only on a particular method [17, 65]. In ensemble FS, EDAS would offer an easier point of entry for scholars due to the familiarity of the approach. In particular, common ensemble FS approaches make use of “averages” to aggregate individual filter results. The EDAS would be able to address issues with the current approaches while providing an easier point of entry to the field.


## 3. Proposed Ensemble Feature Selection Algorithm

This section presents the methodology employed in the study. First, the description of the dataset and the corresponding statistical treatments are provided. Afterward, the procedure for performing the EDAS method is laid down.


### 3.1. Dataset Description and Statistical Treatment

This study uses six datasets obtained from public repositories as presented in Table1. A variety of datasets are employed following the consensus in the literature that some datasets may be more difficult to learn by a classifier. Thus, variations due to this factor should be blocked. Data preprocessing techniques, such as discretization, were performed to reduce noise in the data. Four filters: (i) feature relevance (using Kendall τ or Chi-square χ2), (ii) feature redundancy (using Kendall τ or Chi-square χ2), (iii) symmetric uncertainty, and (iv) Relief-F are used to compose the ensemble. A decision tree (DT), with the classification and regression tree (CART) algorithm, is employed as the classifier to determine how the performance changes when following the filtering produced by the proposed algorithm. Only a single classifier is used in this study to avoid a potential interaction between the classifiers and the dataset, which has been alluded to in previous works. To properly detect significant differences, the experiment adopts one control group and one experimental group. The control group uses the original dataset (i.e., no FS performed on the dataset, or all features retained), while the experimental group uses the proposed ensemble FS algorithm (and is fixed to retain only 20% of the features). Since the dataset only has two classes, then the area under the curve (AUC) is used as the performance measure, following results in the current literature (e.g., Bradley [19], Kohl [20]) regarding binary classification problems in machine learning. The study uses a randomized complete block design (RCBD) as the experimental design. The six datasets comprise the blocking factor. Thus, the blocking factor has six levels. The factor of interest consists of two groups – the control group and the experimental group. The response variable is the “AUC” of the CART classifier. The use of the “AUC” is important since it saves the study from comparison with other ensemble FS algorithms. For example, there would be no need to compare the proposed algorithm to a randomized algorithm since an AUC of 0.5 means that the classifier is essentially spitting random guesses [19]. Ideally, a two-way analysis of variance (ANOVA) would be performed to test for significant differences between the levels in the factor of interest; however, in cases where violations to the ANOVA assumptions are made, the aligned rank transformation (ART) would be used. The ART transforms the original distribution of AUCs into aligned ranks, these ranks can then be used directly by the two-way ANOVA to make inferences about the main effects and interaction effects [66].Table 1
Dataset description. Each dataset was obtained from public databases such as Kaggle or the university of California, Irvine (UCI) machine learning repository.
No.NameNo. Of featuresNo. Of instancesSource1QSAR Biodegradation Dataset411055[21, 67]2Musk (version 2) Dataset1666598[21]3Ionosphere Dataset33351[21]4LSVT Voice Rehabilitation Dataset310126[21, 68]5Connectionist Bench (Sonar, Mines vs. Rocks) Dataset60208[21]6Gender Recognition by Voice203168[69]


### 3.2. Evaluation Based on Distance from Average Solution (EDAS) Method

EDAS is an MCDM method that evaluates the alternatives (i.e., features) using two measures: (i) positive distance from average (PDA) and (ii) negative distance from average (NDA) [70–72]. The best alternative has the highest PDA value or lowest NDA value. Let there be n features and m filters, then the decision matrix is X=xijn×m, where xij is the score of feature i under filter j. The average solution under each filter is stored in vector AV=AVj1×m, where AVj=∑i=1mxij/n. The PDA and NDA are then constructed from the beneficial and nonbeneficial criteria. The beneficial criteria are the filters that are to be maximized: (i) feature relevance, (ii) symmetric uncertainty, and (iii) Relief-F. The nonbeneficial criteria are the filters that are to be minimized. Feature redundancy is the only nonbeneficial criterion employed in this paper. The FS problem could be viewed as an MCDM problem using the framework shown in Figure 1. Using the procedure developed by Keshavarz Ghorabaee et al. [70], the steps in performing EDAS is as follows (i.e., alternatives pertain to features and criteria pertain to filters in this paper): Figure 1.Step 1.
Select the most important criteria that describe the alternatives.Step 2.
Construct the decision matrixX as follows:(1)X=xijn×m=x11⋯x1m⋮⋱⋮xn1⋯xnm.Step 3.
Determine the average solutionAV=AVj1×m vector with respect to each criterion j, where AVj=∑i=1nxij/n.Step 4.
Calculate the positive distance from average matrixPDA=PDAijn×m and the negative distance from average NDA=NDAijn×m matrix accordingly with the type of criterion (i.e., beneficial or nonbeneficial) as follows:If criterionj is beneficial, then(2)PDAij=max0,xij−AVjAVj,NDAij=max0,AVj−xijAVj.If criterionj is nonbeneficial, then(3)PDAij=max0,AVj−xijAVj,NDAij=max0,xij−AVjAVj.Step 5.
Determine the weighted sum ofPDA and NDA for each of the n alternatives over all m criteria as follows:(4)SPi=∑j=1mwjPDAij,SNi=∑j=1mwjNDAij.Step 6.
Normalize the values of eachSPi and SNi as follows:(5)NSPi=SPimaxi∈1,…,nSPi,NSNi=1−SNimaxi∈1,…,nSNi.Step 7.
Calculate the appraisal scoreASi for each alternative i as follows:(6)ASi=12NSPi+NSNi.Step 8.
Rank the alternatives from largest to smallestASi. The best alternative has the largest ASi.Figure 1
Ensemble filter-based feature selection as a multiple criteria decision-making (MCDM) problem. At Level 1, the overall objective of the MCDM problem is to perform ensemble feature selection by combining the beneficial and nonbeneficial criteria at Level 2. The adopted filters at Level 3 will then serve as the criteria to judge the quality of the features (i.e., the alternatives) at Level 4.


## 3.1. Dataset Description and Statistical Treatment

This study uses six datasets obtained from public repositories as presented in Table1. A variety of datasets are employed following the consensus in the literature that some datasets may be more difficult to learn by a classifier. Thus, variations due to this factor should be blocked. Data preprocessing techniques, such as discretization, were performed to reduce noise in the data. Four filters: (i) feature relevance (using Kendall τ or Chi-square χ2), (ii) feature redundancy (using Kendall τ or Chi-square χ2), (iii) symmetric uncertainty, and (iv) Relief-F are used to compose the ensemble. A decision tree (DT), with the classification and regression tree (CART) algorithm, is employed as the classifier to determine how the performance changes when following the filtering produced by the proposed algorithm. Only a single classifier is used in this study to avoid a potential interaction between the classifiers and the dataset, which has been alluded to in previous works. To properly detect significant differences, the experiment adopts one control group and one experimental group. The control group uses the original dataset (i.e., no FS performed on the dataset, or all features retained), while the experimental group uses the proposed ensemble FS algorithm (and is fixed to retain only 20% of the features). Since the dataset only has two classes, then the area under the curve (AUC) is used as the performance measure, following results in the current literature (e.g., Bradley [19], Kohl [20]) regarding binary classification problems in machine learning. The study uses a randomized complete block design (RCBD) as the experimental design. The six datasets comprise the blocking factor. Thus, the blocking factor has six levels. The factor of interest consists of two groups – the control group and the experimental group. The response variable is the “AUC” of the CART classifier. The use of the “AUC” is important since it saves the study from comparison with other ensemble FS algorithms. For example, there would be no need to compare the proposed algorithm to a randomized algorithm since an AUC of 0.5 means that the classifier is essentially spitting random guesses [19]. Ideally, a two-way analysis of variance (ANOVA) would be performed to test for significant differences between the levels in the factor of interest; however, in cases where violations to the ANOVA assumptions are made, the aligned rank transformation (ART) would be used. The ART transforms the original distribution of AUCs into aligned ranks, these ranks can then be used directly by the two-way ANOVA to make inferences about the main effects and interaction effects [66].Table 1
Dataset description. Each dataset was obtained from public databases such as Kaggle or the university of California, Irvine (UCI) machine learning repository.
No.NameNo. Of featuresNo. Of instancesSource1QSAR Biodegradation Dataset411055[21, 67]2Musk (version 2) Dataset1666598[21]3Ionosphere Dataset33351[21]4LSVT Voice Rehabilitation Dataset310126[21, 68]5Connectionist Bench (Sonar, Mines vs. Rocks) Dataset60208[21]6Gender Recognition by Voice203168[69]


## 3.2. Evaluation Based on Distance from Average Solution (EDAS) Method

EDAS is an MCDM method that evaluates the alternatives (i.e., features) using two measures: (i) positive distance from average (PDA) and (ii) negative distance from average (NDA) [70–72]. The best alternative has the highest PDA value or lowest NDA value. Let there be n features and m filters, then the decision matrix is X=xijn×m, where xij is the score of feature i under filter j. The average solution under each filter is stored in vector AV=AVj1×m, where AVj=∑i=1mxij/n. The PDA and NDA are then constructed from the beneficial and nonbeneficial criteria. The beneficial criteria are the filters that are to be maximized: (i) feature relevance, (ii) symmetric uncertainty, and (iii) Relief-F. The nonbeneficial criteria are the filters that are to be minimized. Feature redundancy is the only nonbeneficial criterion employed in this paper. The FS problem could be viewed as an MCDM problem using the framework shown in Figure 1. Using the procedure developed by Keshavarz Ghorabaee et al. [70], the steps in performing EDAS is as follows (i.e., alternatives pertain to features and criteria pertain to filters in this paper): Figure 1.Step 1.
Select the most important criteria that describe the alternatives.Step 2.
Construct the decision matrixX as follows:(1)X=xijn×m=x11⋯x1m⋮⋱⋮xn1⋯xnm.Step 3.
Determine the average solutionAV=AVj1×m vector with respect to each criterion j, where AVj=∑i=1nxij/n.Step 4.
Calculate the positive distance from average matrixPDA=PDAijn×m and the negative distance from average NDA=NDAijn×m matrix accordingly with the type of criterion (i.e., beneficial or nonbeneficial) as follows:If criterionj is beneficial, then(2)PDAij=max0,xij−AVjAVj,NDAij=max0,AVj−xijAVj.If criterionj is nonbeneficial, then(3)PDAij=max0,AVj−xijAVj,NDAij=max0,xij−AVjAVj.Step 5.
Determine the weighted sum ofPDA and NDA for each of the n alternatives over all m criteria as follows:(4)SPi=∑j=1mwjPDAij,SNi=∑j=1mwjNDAij.Step 6.
Normalize the values of eachSPi and SNi as follows:(5)NSPi=SPimaxi∈1,…,nSPi,NSNi=1−SNimaxi∈1,…,nSNi.Step 7.
Calculate the appraisal scoreASi for each alternative i as follows:(6)ASi=12NSPi+NSNi.Step 8.
Rank the alternatives from largest to smallestASi. The best alternative has the largest ASi.Figure 1
Ensemble filter-based feature selection as a multiple criteria decision-making (MCDM) problem. At Level 1, the overall objective of the MCDM problem is to perform ensemble feature selection by combining the beneficial and nonbeneficial criteria at Level 2. The adopted filters at Level 3 will then serve as the criteria to judge the quality of the features (i.e., the alternatives) at Level 4.


## 4. Results and Discussion

In this section, the results of the experiment are presented. A decision tree classifier (i.e., CART) is used to measure and generate the response variable of the experiment. The default configurations of CART in sci-kit learn are used to control variations that might be caused by micro-optimization and hyperparameter optimization. This configuration would place the control group and the experimental group on the same level of intervention. Furthermore, this allows the control of such variations without including blocking factors or covariates due to changes in parameter configurations. In effect, the analysis of the experimental results is made more straightforward. The AUC of the classifier for each dataset in both the control group (i.e., all features included), and the experimental group (i.e., features reduced using the proposed ensemble FS algorithm) is calculated using stratified k-fold cross-validation with 10 folds. The receiver operating characteristic (ROC) curve for the control group and the experimental group in each dataset are presented in Figures2 and 3, respectively. In general, the ROC curves show that the CART classifier performs well in each dataset for both the control group and the experimental group. In the literature, it is conventional to summarize the results of the different folds using the arithmetic mean as the measure for central tendency and the standard deviation as the measure for dispersion. As such, Figures 2 and 3 show the average AUC and the corresponding standard deviation in each dataset.Figure 2
Receiver operating characteristics (ROC) curve of the CART classifier over the six datasets used in the study. No feature selection is applied to the datasets used here. (a) Biodegradation dataset. (b) Ionosphere dataset. (c) LSVT dataset. (d) Musk dataset. (e) Sonar dataset. (f) Voice dataset.
(a)(b)(c)(d)(e)(f)Figure 3
Receiver operating characteristics (ROC) curve of the CART classifier over the six datasets used in the study. The proposed ensemble FS algorithm, which is configured to retain only20% of the features, is applied to each of the datasets here. (a) Biodegradation dataset. (b) Ionosphere dataset. (c) LSVT dataset. (d) Musk dataset. (e) Sonar dataset. (f) Voice dataset.
(a)(b)(c)(d)(e)(f)At this point, these measures simply summarize the results of the ten folds, following the convention, but they must not be used for inference due to the skewness of the AUC distributions as can be verified in the violin plots in Figure4. It can be seen in Figure 4 that several issues, such as multimodal distributions, high skewness, and the presence of outliers, hinder the use of parametric tests on the AUC distributions. As such, Table 2 presents the descriptive statistics for the control group and the experimental group in each dataset. A quick inspection of Table 2 shows that the control group and the experimental group may not be that different. However, there appears to be a difference in the performance between the datasets. For example, it would appear that the CART classifier performed well on the “Voice” dataset as compared to “LSVT” regardless of whether or not feature selection is performed. To further substantiate these observations, significance testing is performed. First, the distribution of AUCs is transformed using ART. The generated aligned ranks were then directly used as the response variable for the two-way ANOVA. The ANOVA results are presented in Table 3. These results can be summarized as follows: first, there is no significant interaction effect, so the main effects of the factor of interest (i.e., type of feature selection) and the blocking factor (i.e., dataset) can be interpreted in a straightforward manner. Second, the test failed to reject the null hypothesis of no significant difference between the control group and the experimental group. As such, it can be inferred that the CART classifier performed just as well on the reduced dataset (i.e., filtered by the proposed ensemble FS algorithm) as with the complete dataset (i.e., no FS performed). Finally, the blocking factor (i.e., dataset) is significant. As such, including this variable as a blocking factor is important in comparing the performance of the proposed ensemble FS algorithm to the control group.Figure 4
Violin plot of the CART classifier’s area under the curve (AUC) of the receiver operating characteristic (ROC) curve over the six datasets used in the study. The dots outside the boxplots represent the outliers. In the legends box, “EDAS” pertains to the proposed ensemble feature selection algorithm, while “No FS” pertains to no feature selection performed on the dataset. It can be easily inferred from the figure that the distributions are skewed. As such, the mean and the variance (presented in Figure3) would not be appropriate when making inferences about the data. (a) Biodegradation dataset. (b) Ionosphere dataset. (c) LSVT dataset. (d) Musk dataset. (e) Sonar dataset. (f) Voice dataset.
(a)(b)(c)(d)(e)(f)Table 2
Descriptive Statistics. Due to the skewness of the distributions, the presence of outliers, and the heterogeneity of variance, the median is used as the center and the interquartile range is used as the measure of dispersion in describing the distribution of the receiver operating characteristic (ROC) curves’ area under the curve (AUC) across the type of feature selection and the datasets used. The number of folds correspond to the k-fold cross-validation performed on the datasets when evaluating the AUC of the CART classifier.
EDASNo FSDatasetMedian (AUC)Interquartile range (AUC)Number of foldsMedian (AUC)Interquartile range (AUC)Number of foldsBiodegradation0.7950.078100.7700.02510Ionosphere0.8850.078100.8750.04810LSVT0.7250.115100.7000.12810Musk0.9100.028100.9450.01010Sonar0.7300.078100.6850.13310Voice0.9800.007100.9700.00810Table 3
Two-way analysis of variance (ANOVA) on the aligned ranks. Due to violations of the ANOVA, the original distribution of the area under the curve (AUC) is transformed using an aligned rank transformation (ART). The ART allows the use of the ANOVA on the aligned ranks to compare groups with multiple factors. In effect, the approach becomes a nonparametric ANOVA. In this study, the “Dataset” is a blocking factor while the “Type of Feature Selection” is the factor of interest.
FactorDegrees of freedomDegrees of freedom of residualsF-ratiop-valueType of Feature Selection11080.9630.329Dataset510876.570<0.001∗∗∗FS× Dataset51081.7940.120Several reasons could be linked to how EDAS was able to effectively reduce the dataset without significantly hampering the performance of the classifier. First, due to EDAS′ capability to distinguish between positive and negative distances from the average, it may be able to assign more adaptive scores to the features. In turn, the ranking of the features is well informed of the problem’s structure. Second, due to the incorporation of beneficial and cost criteria, the algorithm has better information on whether or not a feature should be ranked higher or lower depending on its score on each filter. Third, the choice of filters (i.e., criteria) making up the ensemble could extract characteristics in the data that would be helpful for training the classifier. Finally, the process used by EDAS to incorporate information about the filters and the features could be naturally suitable for creating an ensemble for FS. These findings reveal important implications for the larger body of knowledge. For one, MCDM demonstrates its capability as a framework for doing ensemble filter-based FS. It can be inferred that the problem structure of MCDM could be naturally extended to FS. This result is compelling as it could spark a more intensified interaction between MCDM and FS, which would benefit both fields.Another important finding in this study is the significance of the blocking factor. The study finds that the classifier’s performance significantly varies between the datasets considered. As such, this result would compel scholars to include datasets as a blocking factor when comparing the performance of different FS algorithms. Otherwise, the inference would be misleading, and it would be difficult to determine whether any significant difference was due to the FS algorithm or the datasets. Drawing an inference from these results, it can be said that the proposed ensemble algorithm could reduce the dataset without significantly compromising the classifier’s performance in terms of AUC. While this finding is compelling, some limitations still exist. First, since only the ANOVA on the aligned ranks is used to infer about the groups, the interpretation was limited to the mean ranks instead of the mean AUC. If a parametric test was performed, the result would have obtained higher statistical power. Second, only one classifier was considered in the analysis. As such, the interpretation is limited only to the considered conditions. Finally, only fixed factors were considered in the analysis. Hence, it is not possible to infer beyond the levels adopted. For example, it is impossible to infer the performance of the proposed ensemble algorithm when retaining only 12% of the features or any other proportion that is not considered in the analysis since the experiment retained only 20% of the features. Likewise, it would not be possible to infer the performance of the algorithm when considering other datasets.The proposed ensemble FS algorithm was able to aggregate the different filters in a comprehensive manner. For example, it overcomes the problem of simple averages or weighted averages by considering the beneficial and nonbeneficial types of filters. This allowed the algorithm to rank the features based on their distance to the average solution. While these improvements are compelling, from a methodological standpoint, some limitations still exist in the proposed ensemble algorithm. For one, EDAS depended on the initial weights given to each filter. If different weights were assigned to the filters, the final ranking of the features would have been different. As such, the proposed ensemble FS algorithm is unable to automatically determine the best appraisal score for each feature as it is fixated on the choice of weights given to each filter. Thus, it might be possible to alter the final rankings by adopting a different set of weights for the filters, which would affect the performance of the proposed ensemble FS algorithm. These limitations must be considered by users when using EDAS for ensemble FS.


## 5. Conclusion and Future Works

In this study, an ensemble FS algorithm was explored using the EDAS method of MCDM. Results showed that the proposed ensemble FS algorithm could select smaller subsets of the original features without significantly compromising classification performance. The findings in this paper would significantly contribute to the literature. Firstly, it is one of the few papers investigating the applicability of MCDM to ensemble FS. Second, it is the first to demonstrate the applicability of EDAS as an ensemble FS algorithm. Third, it provides the literature with a new framework for doing ensemble FS through the lenses of MCDM. Thus, the paper could pave the way for the cross-fertilization of MCDM and feature selection. While the results in this study are compelling, some limitations are present. For one, the design of the experiment uses only one dataset and one classifier for evaluating the performance of the proposed ensemble FS algorithm. Future works could expound on this and adopt additional classifiers and datasets. As such, additional factors may be incorporated into the analysis. For another, other MCDM models could be explored to address current limitations, such as the fixation on the choice of weights. Finally, future works could evaluate the proposed ensemble FS algorithm using additional performance measures such as stability.



---

*Source: 4126536-2022-09-05.xml*