Improving Mutual Information based Feature Selection by Boosting Unique Relevance

1 INTRODUCTION . High-dimensional datasets tend to contain irrelevant or redundant features , leading to extra computation , larger storage , and decreased performance ( Bengio et al. , 2013 ; Gao et al. , 2016 ; Bermingham et al. , 2015 ; Hoque et al. , 2016 ) . Mutual Information ( MI ) ( Cover & Thomas , 2006 ) based feature selection , which is a classifier independent filter method , addresses those issues by selecting a relevant feature subset . We start this paper by discussing the value of MI based feature selection ( MIBFS ) . Interpretability : Dimensionality reduction methods consist of two classes : feature extraction and feature selection . Feature extraction transforms original features into new features with lower dimensionality ( e.g. , PCA ) . This method may perform well in dimensionality reduction , but the extraction process ( e.g. , projection ) loses the physical meaning of features ( Chandrashekar & Sahin , 2014 ; Sun & Xu , 2014 ; Nguyen et al. , 2014 ; Gao et al. , 2016 ) . In contrast , feature selection preserves the interpretability by selecting a relevant feature subset . This helps to understand the hidden relationship between variables and makes techniques such as MIBFS preferred in various domains ( e.g. , healthcare ) ( Kim et al. , 2015 ; Liu et al. , 2018 ; Chandrashekar & Sahin , 2014 ) . Generalization : Feature selection methods are either classifier dependent or classifier independent ( Guyon & Elisseeff , 2003 ; Chandrashekar & Sahin , 2014 ) . Examples of the former type include the wrapper method and the embedded method ( e.g. , LASSO ( Hastie et al. , 2015 ) ) which performs feature selection during the training of a pre-defined classifier . The classifier dependent method tends to provide good performance as it directly makes use of the interaction between features and accuracy . However , the selected features are optimized for the pre-defined classifier and may not perform well for other classifiers . The filter method , which is classifier independent , scores each feature according to its relevance with the label . As a filter method , MIBFS quantifies relevance using MI as MI can capture the dependencies between random variables ( e.g. , feature and label ) . Consequently , the feature subset selected by MIBFS is not tied to the bias of the classifier and is relatively easier to generalize ( Bengio et al. , 2013 ; L. et al. , 2011 ; Meyer et al. , 2008 ) . Performance : Although MIBFS is an old idea dating back to 1992 ( Lewis , 1992 ) , it still can provide competitive performance in dimensionality reduction ( see several recent survey works ( Zebari & et al , 2020 ; Venkatesh & Anuradha , 2019 ) ) . We now provide a new perspective using the Information Bottleneck ( Tishby et al. , 2000 ) ( IB ) to explain the superior performance of MIBFS and suggest why MI is the right metric for feature selection . IB was proposed to search for the solution that achieves the largest possible compression , while retaining the essential information about the target and in ( Shwartz-Ziv & Tishby , 2017 ) , IB is used to explain the behavior of neural networks . Specifically , let X be the input data to the neural network , Y be the corresponding label and X̃ be the hidden representation of neural networks . Shwartz-Ziv & Tishby ( 2017 ) demonstrate that the learning process in neural networks consists of two phases : ( i ) empirical error minimization ( ERM ) , where I ( X̃ ; Y ) gradually increases to capture relevant information about the label Y . ( ii ) representation compression , where I ( X̃ , X ) decreases and I ( X̃ ; Y ) remains almost unchanged , which may be responsible for the absence of overfitting in neural networks . We note that the goal of MIBFS is to find the minimal feature subset with maximum MI with respect to the label ( Brown et al. , 2012 ) . Mathematically , the goal can be written as follows . S∗ = arg min f ( arg max S⊆Ω I ( S ; Y ) ) , ( 1 ) where f ( A , B , · · · ) = ( |A| , |B| , · · · ) , |A| represents the number of features in A and Ω is the set of all features , S ⊆ Ω is the selected feature subset and S∗ is the optimal feature subset . In such a manner , MIBFS naturally converts the representation learning process of neural networks to the process of feature selection ( if we consider S as a type of hidden representation X̃ ) and attempts to obtain an equivalent learning outcome . Specifically , maximizing the I ( S ; Y ) corresponds to the ERM phase and minimizing the size of S corresponds to the representation compression phase . We believe this new perspective sheds light on the superior performance of MIBFS in dimensionality reduction and rationalizes the use of MI for feature selection . We note that finding the optimal feature subset S∗ in ( 1 ) through exhaustive search is computationally intractable . Therefore , numerous MIBFS algorithms ( Meyer et al. , 2008 ; Yang & Moody , 2000 ; Nguyen et al. , 2014 ; Bennasar et al. , 2015 ; Peng et al. , 2005 ) are proposed and attempt to select the optimal feature subset following the criterion of Maximize Relevance with Minimum Redundancy ( MRwMR ) ( Peng et al. , 2005 ) . In this paper , we explore a promising feature property , called Unique Relevance ( UR ) , which is the key to select the optimal feature subset in ( 1 ) . We note that UR has been defined for a long time and it is also known as strong relevance ( Kohavi & John , 1997 ) . However , only very few works ( Liu et al. , 2018 ; Liu & Motani , 2020 ) look into it and the use of UR for feature selection remains largely uninvestigated . We fill in this gap and improve the performance of MIBFS by exploring the utility of UR . We describe the flow of the remaining paper together with several contributions as follows . 1 . We shortlist seven state-of-the-art ( SOTA ) and classical MIBFS algorithms and uncover the fact that all of them ignore UR and end up underperforming , namely they select a non-negligible number of redundant features , contradicting the objective of minimal feature subset in ( 1 ) . In fact , it turns out that the minimal feature subset in ( 1 ) must contain all features with UR . 2 . We point out that , the heart of the problem is that existing MIBFS algorithms following the criterion of MRwMR ( Peng et al. , 2005 ) , which lacks a mechanism to explicitly identify the UR of features . This motivates us to augment MRwMR and include the objective of boosting UR , leading to a new criterion for MIBFS , called MRwMR-BUR . 3 . We estimate UR using the KSG estimator ( Kraskov et al. , 2004 ) and conduct experiments with five representative MIBFS algorithms on six datasets . The results indicate that the algorithms that boost UR consistently outperform their unboosted counterparts when tested with three classifiers . 4 . We improve MRwMR-BUR by proposing a classifier based approach to estimate UR and our experimental results indicate that this approach further improves the classification performance of MRwMR-BUR based algorithms . 2 BACKGROUND AND DEFINITIONS . We now formally define the notation used in this paper . We denote the set of all features by Ω = { Xk , k = 1 , · · · , M } , where M is the number of features . The feature Xk ∈ Ω and the label Y are both vectors of length N , where N is the number of samples . Let S ⊆ Ω be the set of selected features and S̃ ⊆ Ω be the set of unselected features , i.e. , Ω = { S , S̃ } . 2.1 INFORMATION CONTENT : OR , UR , II . The information content presents in a feature can be divided into three parts : Overlapped Relevance , Unique Relevance and Irrelevant Information ( see Feature A in Fig . 1 ) . Relevant information is quantified using MI with the label . The Unique Relevance ( UR ) of a feature Xk is defined as the unique relevant information which is not shared by any other features in Ω . Mathematically , UR can be calculated as the MI loss when removing that feature from Ω . By the chain rule for MI ( Cover & Thomas , 2006 ) , UR can be written as UR = I ( Ω ; Y ) − I ( Ω\Xk ; Y ) = I ( Xk ; Y |Ω\Xk ) . ( 2 ) We note that UR is equivalent to strong relevance defined in ( Kohavi & John , 1997 ; Brown et al. , 2012 ) . The Overlapped Relevance ( OR ) of a feature Xk is the relevant information content of a feature Xk which is shared ( or overlapped ) with other features in Ω . By the chain rule for MI , OR can be written as OR = I ( Xk ; Y ) − ( I ( Ω ; Y ) − I ( Ω\ ( Xk ) ; Y ) ) . ( 3 ) The definition of OR is equivalent to weak relevance defined in ( Kohavi & John , 1997 ; Brown et al. , 2012 ) . Consider an example of Ω containing two features Xj and Xk , then the OR of feature Xk is I ( Xj ; Xk ; Y ) , which is known as multivariate mutual information . A positive value of I ( Xj ; Xk ; Y ) is a sign of redundancy , while a negative value expresses synergy ( McGill , 1954 ) . Irrelevant information ( II ) can be understood as the noise in the signal . Overfitting to the irrelevant aspects of the data will confuse the classifier , leading to decreased accuracy ( John et al. , 1994 ; Song et al. , 2011 ) . Mathematically , we define II of feature Xk as II = H ( Xk ) − I ( Xk ; Y ) = H ( Xk|Y ) . ( 4 ) We note that a feature Xk can be completely irrelevant with respect to the label Y if I ( Xk ; Y ) = 0 . There is another popular type of decomposition called partial information decomposition ( PID ) ( Williams & Beer , 2010 ) which decomposes the total mutual information of a system into three parts : unique information , redundant information , synergistic information and a follow-up work ( Bertschinger & et al , 2014 ) attempts to quantify each term based on ideas from decision theory . The definition of UR is the same as the unique information in ( Williams & Beer , 2010 ) , but calculated differently from ( Bertschinger & et al , 2014 ) . Furthermore , OR is equal to the difference of shared information and synergistic information in ( Bertschinger & et al , 2014 ) . 2.2 ESTIMATION OF MUTUAL INFORMATION . In this paper , we estimate MI using the KSG estimator ( Kraskov et al. , 2004 ) which uses the K nearest neighbors of points in the dataset to detect structure in the underlying probability distribution . In a recent work ( Gao et al. , 2018 ) , the KSG estimator is proven to be consistent under some mild assumptions . We note that the KSG estimator is not applicable when the random variable being studied is a mixture of continuous and discrete values . For the case with mixed random variables , we can apply the mixed KSG estimator ( Gao & et al , 2017 ) , which demonstrates good performance at handling mixed variables . We note that the features of all datasets studied in this paper are either purely discrete ( real-valued ) or continuous while all labels are purely discrete ( real-valued ) ( see Table 1 ) . Hence , we use the KSG estimator ( Kraskov et al. , 2004 ) to compute MI quantities .

Summary: In this paper, the authors recognized the function of unique relevance (UR) of features for optimal feature selection and augmented the existing mutual information based feature selection (MIBFS) methods by boosting unique relevance (BUR). As a result, they proposed a new criterion called MRwMR-BUR. Experimental results are provided to show that MIBFS with UR consistently outperform their unboosted conterparts in terms of peak accuracy and number of features required.