Improving Mutual Information based Feature Selection by Boosting Unique Relevance

1 INTRODUCTION . High-dimensional datasets tend to contain irrelevant or redundant features , leading to extra computation , larger storage , and decreased performance ( Bengio et al. , 2013 ; Gao et al. , 2016 ; Bermingham et al. , 2015 ; Hoque et al. , 2016 ) . Mutual Information ( MI ) ( Cover & Thomas , 2006 ) based feature selection , which is a classifier independent filter method , addresses those issues by selecting a relevant feature subset . We start this paper by discussing the value of MI based feature selection ( MIBFS ) . Interpretability : Dimensionality reduction methods consist of two classes : feature extraction and feature selection . Feature extraction transforms original features into new features with lower dimensionality ( e.g. , PCA ) . This method may perform well in dimensionality reduction , but the extraction process ( e.g. , projection ) loses the physical meaning of features ( Chandrashekar & Sahin , 2014 ; Sun & Xu , 2014 ; Nguyen et al. , 2014 ; Gao et al. , 2016 ) . In contrast , feature selection preserves the interpretability by selecting a relevant feature subset . This helps to understand the hidden relationship between variables and makes techniques such as MIBFS preferred in various domains ( e.g. , healthcare ) ( Kim et al. , 2015 ; Liu et al. , 2018 ; Chandrashekar & Sahin , 2014 ) . Generalization : Feature selection methods are either classifier dependent or classifier independent ( Guyon & Elisseeff , 2003 ; Chandrashekar & Sahin , 2014 ) . Examples of the former type include the wrapper method and the embedded method ( e.g. , LASSO ( Hastie et al. , 2015 ) ) which performs feature selection during the training of a pre-defined classifier . The classifier dependent method tends to provide good performance as it directly makes use of the interaction between features and accuracy . However , the selected features are optimized for the pre-defined classifier and may not perform well for other classifiers . The filter method , which is classifier independent , scores each feature according to its relevance with the label . As a filter method , MIBFS quantifies relevance using MI as MI can capture the dependencies between random variables ( e.g. , feature and label ) . Consequently , the feature subset selected by MIBFS is not tied to the bias of the classifier and is relatively easier to generalize ( Bengio et al. , 2013 ; L. et al. , 2011 ; Meyer et al. , 2008 ) . Performance : Although MIBFS is an old idea dating back to 1992 ( Lewis , 1992 ) , it still can provide competitive performance in dimensionality reduction ( see several recent survey works ( Zebari & et al , 2020 ; Venkatesh & Anuradha , 2019 ) ) . We now provide a new perspective using the Information Bottleneck ( Tishby et al. , 2000 ) ( IB ) to explain the superior performance of MIBFS and suggest why MI is the right metric for feature selection . IB was proposed to search for the solution that achieves the largest possible compression , while retaining the essential information about the target and in ( Shwartz-Ziv & Tishby , 2017 ) , IB is used to explain the behavior of neural networks . Specifically , let X be the input data to the neural network , Y be the corresponding label and X̃ be the hidden representation of neural networks . Shwartz-Ziv & Tishby ( 2017 ) demonstrate that the learning process in neural networks consists of two phases : ( i ) empirical error minimization ( ERM ) , where I ( X̃ ; Y ) gradually increases to capture relevant information about the label Y . ( ii ) representation compression , where I ( X̃ , X ) decreases and I ( X̃ ; Y ) remains almost unchanged , which may be responsible for the absence of overfitting in neural networks . We note that the goal of MIBFS is to find the minimal feature subset with maximum MI with respect to the label ( Brown et al. , 2012 ) . Mathematically , the goal can be written as follows . S∗ = arg min f ( arg max S⊆Ω I ( S ; Y ) ) , ( 1 ) where f ( A , B , · · · ) = ( |A| , |B| , · · · ) , |A| represents the number of features in A and Ω is the set of all features , S ⊆ Ω is the selected feature subset and S∗ is the optimal feature subset . In such a manner , MIBFS naturally converts the representation learning process of neural networks to the process of feature selection ( if we consider S as a type of hidden representation X̃ ) and attempts to obtain an equivalent learning outcome . Specifically , maximizing the I ( S ; Y ) corresponds to the ERM phase and minimizing the size of S corresponds to the representation compression phase . We believe this new perspective sheds light on the superior performance of MIBFS in dimensionality reduction and rationalizes the use of MI for feature selection . We note that finding the optimal feature subset S∗ in ( 1 ) through exhaustive search is computationally intractable . Therefore , numerous MIBFS algorithms ( Meyer et al. , 2008 ; Yang & Moody , 2000 ; Nguyen et al. , 2014 ; Bennasar et al. , 2015 ; Peng et al. , 2005 ) are proposed and attempt to select the optimal feature subset following the criterion of Maximize Relevance with Minimum Redundancy ( MRwMR ) ( Peng et al. , 2005 ) . In this paper , we explore a promising feature property , called Unique Relevance ( UR ) , which is the key to select the optimal feature subset in ( 1 ) . We note that UR has been defined for a long time and it is also known as strong relevance ( Kohavi & John , 1997 ) . However , only very few works ( Liu et al. , 2018 ; Liu & Motani , 2020 ) look into it and the use of UR for feature selection remains largely uninvestigated . We fill in this gap and improve the performance of MIBFS by exploring the utility of UR . We describe the flow of the remaining paper together with several contributions as follows . 1 . We shortlist seven state-of-the-art ( SOTA ) and classical MIBFS algorithms and uncover the fact that all of them ignore UR and end up underperforming , namely they select a non-negligible number of redundant features , contradicting the objective of minimal feature subset in ( 1 ) . In fact , it turns out that the minimal feature subset in ( 1 ) must contain all features with UR . 2 . We point out that , the heart of the problem is that existing MIBFS algorithms following the criterion of MRwMR ( Peng et al. , 2005 ) , which lacks a mechanism to explicitly identify the UR of features . This motivates us to augment MRwMR and include the objective of boosting UR , leading to a new criterion for MIBFS , called MRwMR-BUR . 3 . We estimate UR using the KSG estimator ( Kraskov et al. , 2004 ) and conduct experiments with five representative MIBFS algorithms on six datasets . The results indicate that the algorithms that boost UR consistently outperform their unboosted counterparts when tested with three classifiers . 4 . We improve MRwMR-BUR by proposing a classifier based approach to estimate UR and our experimental results indicate that this approach further improves the classification performance of MRwMR-BUR based algorithms . 2 BACKGROUND AND DEFINITIONS . We now formally define the notation used in this paper . We denote the set of all features by Ω = { Xk , k = 1 , · · · , M } , where M is the number of features . The feature Xk ∈ Ω and the label Y are both vectors of length N , where N is the number of samples . Let S ⊆ Ω be the set of selected features and S̃ ⊆ Ω be the set of unselected features , i.e. , Ω = { S , S̃ } . 2.1 INFORMATION CONTENT : OR , UR , II . The information content presents in a feature can be divided into three parts : Overlapped Relevance , Unique Relevance and Irrelevant Information ( see Feature A in Fig . 1 ) . Relevant information is quantified using MI with the label . The Unique Relevance ( UR ) of a feature Xk is defined as the unique relevant information which is not shared by any other features in Ω . Mathematically , UR can be calculated as the MI loss when removing that feature from Ω . By the chain rule for MI ( Cover & Thomas , 2006 ) , UR can be written as UR = I ( Ω ; Y ) − I ( Ω\Xk ; Y ) = I ( Xk ; Y |Ω\Xk ) . ( 2 ) We note that UR is equivalent to strong relevance defined in ( Kohavi & John , 1997 ; Brown et al. , 2012 ) . The Overlapped Relevance ( OR ) of a feature Xk is the relevant information content of a feature Xk which is shared ( or overlapped ) with other features in Ω . By the chain rule for MI , OR can be written as OR = I ( Xk ; Y ) − ( I ( Ω ; Y ) − I ( Ω\ ( Xk ) ; Y ) ) . ( 3 ) The definition of OR is equivalent to weak relevance defined in ( Kohavi & John , 1997 ; Brown et al. , 2012 ) . Consider an example of Ω containing two features Xj and Xk , then the OR of feature Xk is I ( Xj ; Xk ; Y ) , which is known as multivariate mutual information . A positive value of I ( Xj ; Xk ; Y ) is a sign of redundancy , while a negative value expresses synergy ( McGill , 1954 ) . Irrelevant information ( II ) can be understood as the noise in the signal . Overfitting to the irrelevant aspects of the data will confuse the classifier , leading to decreased accuracy ( John et al. , 1994 ; Song et al. , 2011 ) . Mathematically , we define II of feature Xk as II = H ( Xk ) − I ( Xk ; Y ) = H ( Xk|Y ) . ( 4 ) We note that a feature Xk can be completely irrelevant with respect to the label Y if I ( Xk ; Y ) = 0 . There is another popular type of decomposition called partial information decomposition ( PID ) ( Williams & Beer , 2010 ) which decomposes the total mutual information of a system into three parts : unique information , redundant information , synergistic information and a follow-up work ( Bertschinger & et al , 2014 ) attempts to quantify each term based on ideas from decision theory . The definition of UR is the same as the unique information in ( Williams & Beer , 2010 ) , but calculated differently from ( Bertschinger & et al , 2014 ) . Furthermore , OR is equal to the difference of shared information and synergistic information in ( Bertschinger & et al , 2014 ) . 2.2 ESTIMATION OF MUTUAL INFORMATION . In this paper , we estimate MI using the KSG estimator ( Kraskov et al. , 2004 ) which uses the K nearest neighbors of points in the dataset to detect structure in the underlying probability distribution . In a recent work ( Gao et al. , 2018 ) , the KSG estimator is proven to be consistent under some mild assumptions . We note that the KSG estimator is not applicable when the random variable being studied is a mixture of continuous and discrete values . For the case with mixed random variables , we can apply the mixed KSG estimator ( Gao & et al , 2017 ) , which demonstrates good performance at handling mixed variables . We note that the features of all datasets studied in this paper are either purely discrete ( real-valued ) or continuous while all labels are purely discrete ( real-valued ) ( see Table 1 ) . Hence , we use the KSG estimator ( Kraskov et al. , 2004 ) to compute MI quantities .

Summary: This work suggests improving mutual informaton based feature selection methods with an extra term (i.e., the unique relevance (UR)), and introduces a hyper-parameter $\beta$ to weight the UR. The work is easy to follow. However, the perspectives and methods are not novel. And there is a technical flaw in the analysis.