Improving Mutual Information based Feature Selection by Boosting Unique Relevance
Authors: Shiyu Liu
Mehul Motani
Venue:  (2021)
Keywords: Feature Selection
Mutual Information
Unique Relevance

Abstract: Mutual Information (MI) based feature selection makes use of MI to evaluate each feature and eventually shortlist a relevant feature subset, in order to address issues associated with high-dimensional datasets.
Despite the effectiveness of MI in feature selection, we have noticed that many state-of-the-art algorithms disregard the so-called unique relevance (UR) of features, which is a necessary condition for the optimal feature subset. In fact, in our study of seven state-of-the-art and classical MIBFS algorithms, we find that all of them underperform as they ignore UR of features and arrive at a suboptimal selected feature subset which contains a non-negligible number of redundant features. We point out that the heart of the problem is that all these MIBFS algorithms follow the criterion of Maximize Relevance with Minimum Redundancy (MRwMR), which does not explicitly target UR. This motivates us to augment the existing criterion with the objective of boosting unique relevance (BUR), leading to a new criterion called MRwMR-BUR. We conduct extensive experiments with several MIBFS algorithms with and without incorporating UR. The results indicate that the algorithms that boost UR consistently outperform their unboosted counterparts in terms of peak accuracy and number of features required. Furthermore, we propose a classifier based approach to estimate UR that further improves the performance of MRwMR-BUR based algorithms.

Under review as a conference paper at ICLR 2021
IMPROVING MUTUAL INFORMATION BASED FEATURE SELECTION BY BOOSTING UNIQUE RELEVANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Mutual Information (MI) based feature selection makes use of MI to evaluate each feature and eventually shortlist a relevant feature subset, in order to address issues associated with high-dimensional datasets. Despite the effectiveness of MI in feature selection, we have noticed that many state-of-the-as algorithms disregard the so-called unique relevance (UR) of features, which is a necessary condition for the optimal feature subset. In fact, in our study of seven state-of-the-ar and classical MIBFS algorithms, UR of features and arrive at a suboptimal selected feature subset which contains non-negligible number of redundant features. We point out that the heart of the problem is that all these MIBFS algorithms follow the criterion of Maximize Relevance with Minimum Redundancy (MRwMR), which does not explicitly target UR. This motivates us to augment the existing criterion with the objective boosting unique relevance (BUR), leading new criterion called MRwMR- BUR. We conduct extensive experiments with several MIBFS algorithms with and without incorporating UR. The results indicate that the algorithms that boost UR consistently outperform their unboosted counterparts in terms of peak accuracy and number of features required. Furthermore, we propose a classifier based approach estimate UR that further improves the performance of MRwMR -UR based algorithms.
find that all of them underperform as they ignore
INTRODUCTION
High-dimensional datasets tend to contain irrelevant or redundant features, leading to extra computa- tion, larger storage, and decreased performance (Bengio et al. 2013 Gao et al. 2016 Bermingham et al. 2015 Hoque et al.) 2016). Mutual Information (MI) (Cover & Thomas 2006 based feature selection, which a classifier independent filter method, addresses those issues by selecting a relevant feature subset. We start this paper by discussing the value of MI based feature selection (MIBFS).
Interpretability: Dimensionality reduction methods consist of two classes: feature extraction and feature selection. Feature extraction transforms original features into new features with lower dimensionality (c.g., PCA). This method may perform well in dimensionality reduction, but the extraction process (e.g.. projection) loses the physical meaning of features (Chandrashekar & Sahin 2014 (Sun & Xu the interpretability by selecting relevant feature subset. This helps to understand the hidden relationship between variables and makes techniques such as MIBFS preferred in various domains
et al. 2014 Gao et al.) 2016). In contrast, feature selection preserves
(e.g., healthcare) (Kim al. 2015 Liu et al. 2018 Chandrashekar & Sahin 2014).
Generalization: Feature selection methods are either classifier dependent or classifier independent Guyon Elisseeff 2003; Chandrashekar Sahin 2014). Examples of the former type include the wrapper method and the embedded method (c.g.. LASSO (Hastic et al. 2015)) which performs feature selection during the training of pre-defined classifier. The classifier dependent method tends to provide good performance as it directly makes use of the interaction between features and accuracy. However, the selected features are optimized for the pre-defined classifier and may not perform well for other classifiers. The filter method, which is classifier independent, scores each feature according to its relevance with the label. As a filter method, MIBFS quantifies relevance using MI as MI can capture the dependencies between random variables (c.g.. feature and label). Consequently, the feature subset selected by MIBFS is not tied to the bias of the classifier and is relatively easier to generalize (Bengio et al. 2013 L. et al. 2011 Meyer et al. 2008

---

Under review as a conference paper at ICLR 2021
1992). it still can provide competitive performance in dimensionality reduction (see several recent survey works (Zebari & et al 2020; Venkatesh & Anuradha 2019).. We now provide a new perspective using the Information Bottleneck (Tishby et al. (000) (IB) to explain the superior performance of MIBFS and suggest why MI is the right metric for feature selection. IB was proposed to search for the solution that achieves the largest possible compression, while retaining the essential information about the target and in (Shwartz-Ziv & Tishby 2017), IB is used to explain the behavior of neural networks. Specifically, let X be the input data the neural network, Y be the corresponding label and X be the hidden representation of neural networks. SSwwrtz-ZZiv & Tishby (017)) demonstrate that the learning process in neural networks consists of two phases: (i) empirical error minimization (ERM), where ((; gradually increases to capture relevant information about the label Y. (ii) representation compression, where / T(X; X) decreases and I [(x; Y) remains almost unchanged, which may be responsible for the absence of overfitting in neural networks.
Performance: Although MIBFS is old idea dating back to 1992 (Lwiss
We note that the goal of MIBFS is to find the minimal feature subset with maximum MI with respect to the label (Brown et al. 2012 Mathematically, the goal can be written as follows.
S* arg min (arg max ((S))),
SCD
where S(A, B, B.....) (IAI, (......, |A| represents the number of features in A and Ω is the set of all features, S C is the selected feature subset and S* is the optimal feature subset. In such manner, MIBFS naturally converts the representation learning process of neural networks to the process of feature selection (if we consider S type of hidden representation X) and attempts to obtain an equivalent learning outcome. Specifically, maximizing the I(S; Y) corresponds to the ERM phase and minimizing the size of S corresponds to the representation compression phase. We believe this new perspective sheds light on the superior performance of MIBFS in dimensionality
reduction and rationalizes the use of MI for feature selection.
We note that finding the optimal feature subset S* in (1) through exhaustive search is computationally intractable. Therefore, numerous MIBFS algorithms (Meyer et al.) 2008 Yang & Moody 2000) Nguyen al. 20144] Bennasar et al.) 2015 Peng et al. 2005) are proposed and attempt to select the optimal feature subset following the criterion of Maximize Relevance with Minimum Redundancy (MRwMR) (Peng et al. 2005).
In this paper, we explore a promising feature property, called Unique Relevance (UR), which is the key to select the optimal feature subset in (I). We note that UR has been defined for long time and it is also known as strong relevance (Kohavi & John 1997). However, only very few works (Liu let al., 2018 Liu & Motani 2020) look into it and the use of UR for feature selection remains largely uninvestigated. We fill in this gap and improve the performance of MIBFS by exploring the utility of UR. We describe the flow of the remaining paper together with several contributions as follows.
1. We shortlist seven state-of-the-art (SOTA) and classical MIBFS algorithms and uncover the fact that all of them ignore UR and end up underperforming. namely they select non-negligible number of redundant features, contradicting the objective of minimal feature subset in ( In fact, it turns out that the minimal feature subset in (L) must contain all features with UR.
2. We point out that, the heart of the problem is that existing MIBFS algorithms following the criterion of MRwMR (Peng et al. 2005). which lacks mechanism to explicitly identify the UR of features. This motivates us to augment MRwMR and include the objective of boosting UR, leading new criterion for MIBFS, called MRRMR-BUR.
3. We estimate UR using the KSG estimator (Kraskov et al. (004)) and conduct experiments with five representative MIBFS algorithms on six datasets. The results indicate that the algorithms that boost UR consistently outperform their unboosted counterparts when tested with three classifiers.
4. We improve MR --R by proposing a classifier based approach to estimate UR and our experimental results indicate that this approach further improves the classification performance of MRRMRR-UR based algorithms.
2 BACKGROUND AND DEFINITIONS
We now formally define the notation used in this paper. We denote the set of all features by Ω = (Xx, ks I,... M), where M is the number of features. The feature X5 E Ω and the label Y
2

---

Under review as a conference paper at ICLR 2021
Feature A
Feature B
Overlapped Relevance (OR)
Unique Relevance (UR)
Irelevant Information (II)
Figure I:
Entropy feature and B from the perspective of OR, UR and Feature A represents a feature with UR and OR with respect to the label, Feature B represents feature with no UR and some OR with respect to the label.
both vectors length where N is the number of samples. Let S C Q be the set of selected features and 5 C be the set of unselected features, i.e., Q {S S}.
2.1 INFORMATION CONTENT: OR, UR,
The information content presents in feature can divided into three parts: Overlapped Relevance, Unique Relevance and Irrelevant Information (see Feature A in Fig. 1.
Relevant information quantified using MI with the label. The Unique Relevance (UR) of a feature X₁ is defined as the unique relevant information which is not shared by any other features in Q. Mathematically, UR can be calculated as the MI loss when removing that feature from Q. By the chain rule for MI (Cover & Thomas 2006). UR can be written as
= 7(2; Y) xx: Y) = T(XN: X)))
We note that UR is equivalent to strong relevance defined in (Kohavi & John 1997 Brown et al. 2012). The Overlapped Relevance (OR) feature Xş is the relevant information content feature Xg which is shared (or overlapped) with other features in Ω. By the chain rule for MI, OR can be written as
OR = I(X;;)) (I(); I())
The definition of OR is equivalent to weak relevance defined in (Kohavi John 1997: Brown et al. 2012). Consider example of S2 containing two features X, and Xx. then the OR of feature X is T[X;; ji Xg; Y), which is known as multivariate mutual information. A positive value of / (X5; Xx; Y) a sign of redundancy, while negative value expresses synergy (MeGill 1954).
Irrelevant information (II) can be understood as the noise the signal. Overfitting to the irrelevant aspects of the data will confuse the classifier, leading decreased accuracy (John et al. 1994
Song
et al. 2011) Mathematically. we define of feature Xx as
II H (Xx) (Xx;; Y) (Xx))Y)
We note that a feature Xς. can be completely irrelevant with respect to the label Y if Y
There is another popular type of decomposition called partial information decomposition (PID) (Williams & Beer 2010) which decomposes the total mutual information of a system into three parts: unique information, redundant information, synergistic information and a follow-up work (Bertschinger & et all 2014) attempts quantify each term based on ideas from decision theory. The definition of UR is the same the unique information in (Williams & Beer 2010). but calculated differently from (Bertschinger & et all 2014). Furthermore, OR is equal to the difference of shared information and synergistic information in (Bertschinger et all 2014).
2.2 ESTIMATION OF MUTUAL INFORMATION
In this paper, we estimate MI using the KSG estimator (Kraskov et al. 2004) which uses the K nearest neighbors of points in the dataset to detect structure in the underlying probability distribution.

---

Under review as a conference paper at ICLR 2021
Gas sensor
Colon
Sonar
Madelon
Leukemia
Isolet
Features
128
2000
60
500
7070
617
Instances
13874
62
208
2600
72
1560
Classes
2
2
26
Data Type
Cont.
Disc.
Cont.
Cont.
Disc.
Cont.
UR (%)
2.34%
4.3%
28.3%
29.6%
34.9%
37.1%
Table 1: Information of experimental datasets showing the fraction of features with UR.
In recent work (Gao et al., 2018), the KSG estimator is proven to be consistent under some mild assumptions. We note that the KSG estimator is not applicable when the random variable being studied is a mixture of continuous and discrete values. For the case with mixed random variables, can apply the mixed KSG estimator (Gao & et all (017). which demonstrates good performance at handling mixed variables. We note that the features of all datasets studied in this paper are either purely discrete (real-valued) or continuous while all labels are purely discrete (real-valued) (see Table D Hence, use the KSG estimator (Kraskov et al. 2004) to compute MI quantities.
A NEW CRITERION FOR MI BASED FEATURE SELECTION
In this section, first show the crucial role of UR in selecting the optimal feature subset in 1 Next, we conduct experiments to uncover the fact that all studied MIBFS algorithms are underperforming and hence, motivating MRwMR-BU
new criterion for MIBES.
3.1 A NECESSARY CONDITION FOR OPTIMALITY
Recall the goal of MIBFS in (I) is to find the minimal feature subset with maximum MI with respect to the label (Brown et al. 2012). Several works YYu & Liu 2004 John et al. 1994) have noticed that UR is necessary condition for the optimal solution in (1) This can be simply proved follows. Proposition 1.
The optimal feature subset S in ( which has maximum T(S; Y) with minimum [S], must contain all features with UR.
Proof. Assume there exists feature X3. E Q with non-zero UR. Suppose we have a feature subset S Ω Xxx which has maximum I(S;)) with minimum SS.. Since X₁. has non-zero UR, we have [(2; Y) T(N) Y) by definition. Therefore, I(22 Y) T(S; Y) as (R; Y) > I(S; Y)
given C Xx- But this contradicts the initial assumption that [(; Y) is maximum.
We note that the optimal feature subset S* may also contain features with OR and no UR certain situations. For example, consider feature subset T which contains all features with UR, it is possible have another feature Xm & T which contributes higher joint MI (i.e., I(X my T; Y) I(T;Y)).
3.2 MOTIVATING THE MRwMR- -UUR CRITERION
In this subsection, we shortlist seven SOTA and classical MIBES algorithms: Mutual Information Maximization (MIM) (Lewis] 1992). Joint Mutual Information (JMI) (Yang & Moody 2000). Joint Mutual Information Maximization (JMI)) (Bennasar et 2015). minimal Redundancy Maximum
Relevance (mRMR) (Peng et al.) 2005), greedy search algorithm (GSA) (Brown et al.) 2012). Mutual Information Convex Optimization (MICO) (Sun et all 2019). Conditional Mutual Information based Feature Selection considering Interaction (CMIFSI) (Tang et al.) (019) We simulate their feature selection process using Sonar dataset (Dua & Graff 2017 7 and evaluate the variation joint MI T(S; Y) more features are selected.
In Fig. 2 we classify features as three types: (i) features with nonzero UR, (ii) features with zero UR, and (iii) redundant features, namely that all redundant features (c.g..Xpedun)) can be removed after saturation of joint MI without decreasing the joint MI (i.e., [S ; Y) = [ ([Seatrr Xredun Y)). In Fig. (left), we provide an example of the GSA feature selection process and summarize the performance of all algorithms in Fig. 2 (right), where redundancy rate represents the percentage of redundant features in the selected feature subset when the joint MI saturation is firstly reached.
In Fig. (left,, we observe that GSA is able to perform well at the beginning and select features with UR. However, more features selected, the joint distribution of the selected feature subset
4

---

Under review as a conference paper at ICLR 2021
algorithm
redundancy rate
0.254
Joint MI I(;)) Y)
MIM
36.9%
0.252
JMI
21.6%
JMIM
25.6%
0.25
Feature with UR
mRMR
27.5%
Feature without UR
0.248
GSA
14.7%
Redundant Feature
CMIFSI
19.4%
10
30
40 S
MICO
24.3%
Number of Features in Selected Feature Subset
Figure 2: (left) Feature selection with GSA. (right) Redundancy rates for various algorithms.
becomes more complex, and then GSA tends to select some redundant features. Similar performance can be observed for other algorithms (see Fig. (right)). We note that, redundant features contribute nothing, but increase the size of the selected feature subset, undermining the objective of minimal feature subset requirement in (... Surprisingly. all of the SOTA and classical algorithms studied are selecting non-negligible number of redundant features, which uncovers the fact that all of them underperforming. Lastly, we note that similar performance can be observed on another dataset (Gas Sensor dataset (Alexander et al. 2012) well (see A.I in Appendix).
The heart of the problem is that features with UR are not prioritized during the selection process as all these algorithms follow the MRwMR criterion which lacks mechanism to identify UR. Furthermore, based on our analysis of various datasets (see Table T.. features with UR usually make up a very small fraction of the total number of features and hence, is difficult select them without explicitly targeting them. This motivates us to augment MRwMR and include the objective of boosting unique relevance (BUR), leading to a new criterion, called MRwMR-BLL -UR. Section 6 we discuss results showing that the new criterion can significantly reduce the redundancy rate.
4 PERFORMANCE EVALUATION
In this section, we shortlist five representative MIBFS algorithms and modify them to include the objective of BUR. We conduct experiments compare the performance of each algorithm before and after boosting UR, so as to evaluate the proposed MRwMR-BUR criterion.
4.1 EXPERIMENT SETUP
To support the proposed MR BUR criterion, we conduct experiments using six public datasets
(Peng et al., 2005 Dua & Graff, 2017 Little et al. 2007: Alexander et al.) 2012) described in Table T
and compare the performance of MRwMR- BUR to MRwMR via Support Vector Machine (SVM)
(Cortes & Vapnik 1995), K-Nearest Neighbors (KNN) (Larose & Larose 2014) and Random Forest
(RF) (Breiman 2001). We shortlist five representative MRwMR based algorithms: MIM (Lewis)
1992), JMI (Yang & Moody 22000), JMIM (Bennasar et al., (015), mRMR (Peng al., 2005) and
(Bown et al. 2012). Those algorithms are augmented to include the objective of BUR as
(5) where Joia is the MRwMR based algorithm (c.g., MIM) and Jnew corresponding MR form and Ju RuR(X (X;) returns the UR of feature X,, We note that this modification is slightly different for JMI JMI is not bounded and the score increases more features are selected. Therefore, we divide the original form of JMI by the size of the selected feature subset and include BUR as shown in (5. Moreover, the details for each algorithm with and without BUR are provided in Appendix A.2 We will denote the algorithm that extends XYZ XYZ BUR (e.g., MIM and MIM BUR).
(X.) = (1 B) x Jota(X,) + B x (U.(
For each run, the dataset is randomly split into two subsets: training dataset (75%) and testing dataset (25%). We apply all MRwMR and MRwMR-BLL BUR (with 3 = 0.1) based algorithms on the same training dataset to select features and evaluate them using the same testing dataset. To ensure fair comparison, the parameters of all classifiers are tuned via grid search and note that all methods share the same grid search range and step size. Some key parameters are tuned as follows. (i) the number of neighbors K for KNN is tuned from 3 to 30 with step size of 2. (ii) the regularization coefficient for SVM is chosen from (0.01, 0.1, 1, 10). (ii) the number of trees in the RF is chosen from (10,15, .... 50). Lastly, the source code will be released at the camera ready stage.
5

---

Under review as a conference paper at ICLR 2021
4.2 PERFORMANCE COMPARISON
The results averaged over 20 runs via RF are shown in Fig. 3 For each algorithm, we report two results: (i) the peak averaged accuracy and its spread over 20 runs using boxplot. (ii) the number of features required to obtain the peak accuracy. Moreover, to better summarize the performance difference between the MRwMR based algorithm and its MRRMR-BUR form, we report two sets of results for each dataset: the peak averaged accuracy improvement and the feature improvement. Specifically, as shown below the subplot of Fig. 3 from left to right, it is the minimum, maximum and averaged improvement over five comparisons between the MRwMR based algorithm and its MRwMR-BUU form (c.g., compare MIM to MIM BUR, JMI to JMI BUR and so on). A negative value of the improvement indicates that the peak averaged accuracy of the MRwMR -UR based algorithm is lower than its original form or the number of features required for the MRwMR-BUL based algorithm is more than its original form.
In terms of MRwMR based algorithms, we observe that MIM tends to provide the worst performance in terms of peak averaged accuracy and number of features required (see Figure (b), (c), (f)). We suspect it is because that MIM assumes features are independent from each other, leading to degraded performance. For other MRwMR based algorithms, JMI and GSA generally perform better than other MRwMR based algorithms (see Figure 3 (a), (d), (d), (f)). This finding agrees with (Brown al., 2012] Liu et al. 2018) as GSA greedy nature and JMI can increase the complementary information between features.
Comparing the performance of MRwMR to MR we observe that the performance of most MR MR based algorithms is improved after including the BUR objective (e.g., compare MIM to MIM _UR). For example, the maximum and average accuracy improvement for the Leukemia dataset is approximately 0.014 and 0.005, respectively. Moreover, fewer number of features (i.c., 9.8 features on average) are required. Similar improvements can be observed for the other datasets as well. We note that similar trends can be found using SVM and KNN well (see A.3 in Appendix).
ESTIMATING UR VIA A CLASSIFIER BASED APPROACH
In this section, we first introduce an alternative classifier based approach to estimate UR in Section 5.1 Next, we compare to performance of estimating UR using the KSG estimator in Section 5.2
5.1 THE CLASSIFIER BASED APPROACH TO ESTIMATE UR
The UR of feature X in (2 can be equivalently expressed as ([XxYY[
(6) We note that the term (Y in (6) is constant for every candidate feature during the feature selection process. Therefore, boosting UR is equivalent to boosting HY (x),,, which is defined as
(Y
((Y = -E (log P(Y))))))))
The RHS of 7 is a function of the likelihood P(Y| Xx). which we can estimate using a classifier Q with parameter& 0, Hence, the estimated UR of feature X2 can be written as
N
UR = H( ~
1 N
(0)
(8)
where N is the number of training samples. The reason for estimating UR via a classifier is two-fold: (i) We understand that the estimation of high-dimensional MI is still in the development stage. This approach provides an alternate way to estimate UR and the estimated UR will approach the true value as the number of samples N grows given the classifier Q is a consistent estimator (Lehmann) & Casella 2006). (ii) We note that different classifiers may favour the UR of features differently due to their working mechanisms and assumptions. For example, the improvement of MR BUR over MRwMR is marginal using SVM with linear kernel (see (see in Appendix). We suspect that this is because MI quantifies a non-linear relationship between random variables and this non- linear relationship may not be of much help to the linear classifier. Estimating UR via a classifier attempts to adapt the value of UR to different classifiers. Furthermore, we note that estimating UR in such a manner changes MRwMR-BUL highlight that the selected feature subset still preserves the interpretability and is preferred in some performance-oriented tasks if the classifier based approach can further improve the performance.
based algorithms to a classifier dependent method, but

---

Under review as a conference paper at ICLR 2021
Gas sensor
Colon
Sonar
Madelon
Leukemia
Isolet
MIM_BUR
99.45 (88)
72.75%(77) (0.23%(41) (23)
86.34%((33) (6.)4%(42)
MIM_BUR BUR.CLF
(6)3
86.41 (19)
80.57% (47)
74.32%(57)
98.64%(88
87.69 87.69%(131)
mRMR_BUR
87.09%(7) 9714
72.57%(77) 81.2
(6.13%(131) (6.63%(63
mRMR BUR CLF
99.50%
81.53%(37) (31)
(5719%(57)
88.50%(135) (53)
JMI BUR
(75)
(39)
71393%(63) (4.25%(48)
(7.)
87.94%(143)
JMI_BR_CILP
(3.76%(3 (72)
81.79% (..9%)(4) (40)
(5.47%(57)
(9.33%(64)
87.58%((127)
JMIM_BUR
80.72%(34) (1992%(19) (8))
(6.88%(61) (3.32%(67)
85,44%((12)
JMIM BUR.CLF
(799
76.31
81.13% (16)
73. (37)
85.72% (724)%(72)
GSA BUR
(874
81.94 (13)
(26)
(5.74%((5) (59
(1.25%(121)
CLF
99.50%(33
38.25%(2)
81973%(49)
74.58 (57)
99.21 % (65)
(106)
Table 2: Peak averaged accuracy and corresponding number of features required for MRwMR-BUR based algorithms (with 3 0.1) using Random Forest. The algorithm with an extension of CLF indicates that the algorithm uses the classifier based approach to estimate UR.
5.2 PERFORMANCE COMPARISON
We apply this new approach estimate UR and re-implement the MRwMR-BUR based algorithms in Section 4 We highlight that all estimated URs are first normalized from 0 to I using min-max normalization. The peak averaged accuracy and corresponding number of features required are shown in Table with 3 = 0.1. We note that the UR is estimated using the classifier being tested. In Table 2. the KSG estimator is used to estimate UR except for the CLF named algorithms, which use the classifier based approach. We highlight the best accuracy using shading. In general, we find that the classifier based approach outperforms the KSG estimator in terms of peak averaged accuracy and number of features required in most datasets and algorithms studied. For example, the accuracy of the classifier based approach for mRMR- -UR on the Leukemia dataset is increased by 0.025 while 10 fewer features are required. Furthermore, similar performance trends can be observed using KNN and SVM as well (see A.3 in Appendix).
6 REFLECTIONS
In this section, we present some reflections and suggestions for future work.
Potentially Better MIBFS Algorithms: We highlight that, in this paper, we are not proposing new MIBFS algorithm. Instead, we explore a new criterion for the design of MIBFS algorithms that incorporate UR into the objective. Our experimental results demonstrate that the proposed MRwMR-BUL BUR criterion has superior performance over the existing MRwMR criterion. We believe this new insight can inspire better MIBFS algorithms without increasing algorithm complexity. In fact, the UR only needs to be calculated once and can be reused in subsequent calculations. Hence, the complexity of the UR augmented algorithms is comparable to the original algorithms.
Redundancy Rates: By not prioritizing UR, existing MIBFS algorithms choose high fraction of redundant features and end up underperforming. Our experiments in Section 3.2 demonstrate that the redundancy rate of GSA is 14.7%. By incorporating UR, the redundancy rate of GSA_BUR is significantly reduced to 9.09% with 3 = 0.05 and 6.25% with B = 0.1.
Always Select Features with UR First?: The results in Section 3.1 show that the minimal feature subset S* in ( must include all features with UR. However, it may not be a good idea to include only features with UR at first when the selected feature size is constrained. This is because features with UR (e.g., X2)) may not contribute to higher relevance in the short term given S C X2 (e.g.. I(Xx; Y|S) = as removing conditional features may decrease the MI.
Optimal Value of 3: The performance of MRRMR-BUR based algorithms relies on the value of B, which balances the original objective of maximizing relevance with prioritizing UR. From extensive experiments, we have found that using 3 = 0.1 to balance UR and relevance. Alternatively, 3 can be thought of as a ttyper-aramettr and tuned via a validation dataset. The theoretical determination of the optimal value/ 3 is clearly worth deep thought. This could help to further minimize redundancy by prioritizing more important features.
7

---

Under review as a conference paper at ICLR 2021
0.39
0.89
0.97
0.88
0.96
0.95
0.91
0.83
0.81
0.59
127
150
(a) Isolet Dutaset
(b) Leukemia Dataset
Accuracy/Feature Inprovement: (0.15, 0.9%, 0,38% 0.38%))(-77 15, 3.6)
Accumecy/Fottuur Improvement: (-0.05%, 1.45, 0.5% 0.5% 44, 9.8)
0.106
0.75
0.095
0.994
0.005
0.72
0.002
01911
01900
969697
(c) Madelon
(d) Gas Sensor Dutuset
Accuracy/Feattne Inpeovement: (-0.6% 1.1, 0.22%///2-- 11)38)
Accuracy/Featurr Improvement: (-0.02%, 0.15%, 21-.,.
0.50
0.00
0.83
0.55
0.70
0.75
0.00
(c) Colon Dataset
(1) Sonar Dataset
Accuracy/Feature Improvement (1.3%, 2.5%, 1.9%) 7.6)
Accuracy/eentue Improement; (0.5%, 1.7%, (888) 15,
Figure 3: Performance Comparison between MRwMR based algorithms and corresponding MRwMR-BUP forms on six public datasets. The vertical and horizontal axes are classification accuracy and the corresponding number of features required, respectively. The results shown are averages over 20 trials via Random Forest with 8 = 0.1. The box corresponds to the lower and upper quartile. The horizontal line inside the box depicts the peak averaged accuracy and the whiskers depict the minimum and maximum over the trials. In each subplot, we report two sets of results: peak averaged accuracy improvement and feature improvement. Specifically, from left to right, is the minimum, maximum and averaged improvement over five comparisons between the MRwMR based algorithm and its MRwMR BUR form (e.g., compare MIM to MIM.B BUR. JMI to JMI.BU and so on). A negative value of improvement indicates that the peak averaged accuracy of the MRwMR - BUR algorithm lower than its original form or the number of features required for the BURRR-BUR algorithm is more than its original form. Similar Performance can be observed for SVM and KNN as well (see A.3 in Appendix).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Vergara Alexander et al. Chemical gas sensor drift compensation using classifier ensembles, Sensors and Actuators B. Chemical, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3598):1798-1828, 2013.
Mohamed Bennasar, Yulia Hicks, and Rossitza Setchi. Feature selection using joint mutual informa- tion maximisation. Expert Systems with Applications, 42(22):8520-85)) 8532, 2015.
Mairead I.. Bermingham al. Application of high-dimensional feature selection: evaluation for genomic prediction in man. Scientific reports, 5: 5:103-112 (12-112, 2015.
Nils Bertschinger and et al. Quantifying unique information. Entropy, 16(4):1661-2183 2014.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Gavin Brown et al. Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. Journal of machine learning research, 13(1):27-66, 2012.
Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Electrical Engineering, 40(1):16-28) 2014.
Computers &
Corinna Cortes and Vladimir Vapnik. Support- networks. Machine learning, 20(3):273-297, 1995.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory, 2nd edition. John Wiley & Sons, 2006.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt:tttp//cchhiee ics. www.ccc..o mml
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Variational information maximization for feature selection. In Proceedings of the 30th Advances Neural Information Processing Systems, pp. 487-495, 2016.
Weihao Gao and al. Estimating mutual information for discrete-continuuut mixtures. In Proceed- ings of Advances in neural information processing systems, pp. 5986-5997, 2017.
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor informa- tion estimators. IEEE Transactions on Information Theory, 64(8):5629-55)) 5661, 2018.
Isabelle Guyon and André Elisseeff. introduction to variable and feature selection. Journal of machine learning research, 3(Mar): 111-111571111 2003.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman & Hal//CRC, 2015.
N. Hoque et al. A fuzzy mutual information based feature selection method for classification. Fuzzy Information and Engineering, 8(3):355-384, 2016.
George H. John, Ron Kohavi, and Karl Pfleger. Irrelevant features and the subset selection problem.
In Proceedings of the 8th International Conference on Machine Learning, pp. 121-129, 1994.
Been Kim, Julie A. Shah, and Finale Doshi-Velez. Mind the gap: A generative approach to in- terpretable feature selection and extraction. In Proceedings of Advances Neural Information Processing Systems, pp. 2260-2268, 2015.
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial intelligence, 97 (1-2):273-324, 1997.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, (96))066-113)) 2004.

---

Under review as a conference paper at ICLR 2021
Ladha L. et al. Feature selection methods and algorithms. International journal on computer science and engineering, 1(3):1787 1797, 2011.
Daniel T. Larose and Chantal D. Larose. Discovering knowledge in data: introduction data mining. John Wiley & Sons, 2014.
Brich L.. Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
David D. Lewis. Feature selection and feature extraction for text categorization. In Proceedings of the workshop on Speech and Natural Language, pp. 212-217, 1992.
Max Little et al. Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. Biomedical engineering online, 6(1):23, 2007.
Shiyu Liu and Mehul Motani. Exploring unique relevance for mutual information based feature selection. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2747-2752, 2020.
Shiyu Liu, Jia Yao, Chongyu Zhou, and Mehul Motani. SURI: Feature selection based on unique relevant information for health data. Proceedings of IEEE International Conference on Bioinfor- matics and Biomedicine, pp. 687-692, 2018
W. McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
Patrick Emmanuel Meyer, Colas Schretter, and Gianluca Bontempi. Information-theorehe feature selection in microarray data using variable complementarity, IEEE Journal of Selected Topics in Signal Processing, 2(3):261-274, 2008.
Xuan Vinh Nguyen et Effective global approaches for mutual information based feature selection. In Proceedings of the 20th international conference on Knowledge discovery and data mining, pp. 512-521, 2014.
Hanchuan Peng. Fuhui Long, and Chris Ding. Feature selection based mutual information: criteria of max-dependency, max-relevance, and mii-redundanny. IEEE Transactions on Pattern Analysis 2005.
& Machine Intelligence,
Ravid Shwartz- Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 00007000300000 2017.
Qinbao Song. Jingjie Ni, and Guangtao Wang. fast clustering- blased feature subset selection algorithm for nigh-dimensional data. IEEE transactions on knowledge and data engineering, 25 (1):1-14, 2011.
Lin Sun and Jiucheng Xu. Feature selection using mutual information based uncertainty measures for tumor classification. Bio- sio-medical materials and engineering. 24(1):763-770, 2014.
Zhenqiang Sun and al. Mutual information based multi-label feature selection via constrained convex optimization. Neurocomputing, 329:447-456, 2019.
Xiaochuan Tang, Yuanshun Dai, and Yanping Xiang. Feature selection based on feature interactions with application to text categorization. Expert Systems with Applicutions, 120:207-216, 2019.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
B. Venkatesh and J. Anuradha. review of feature selection and its methods. Cybernetics and Information Technologies, 19():33-26, 2019.
Paul L. Williams and Randall D. Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv: 1004.20005555 2010.
10

---

Under review as a conference paper at ICLR 2021
Howard Hua Yang and John Moody. Data visualization and feature selection: New algorithms for nongaussian data. In Proceedings of Advances in Neural Information Processing Systems, pp. 687-693, 2000.
Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundaney. Journal of machine learning research, 5(Oct); 1205-12:105-2224 2004.
Rizgar Zebari and et al. A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. Journal of Applied Science and Technology Trends, 1(2):66-70, 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
In A.I, we provide more results of the redundancy experiment conducted in Section 3.2 Next, we show the scoring function of studied MIBFS algorithms before and after including BUR in A.2 Lastly, we show the performance comparison between MRwMR and MRwMR-BUL -URR as well as the effect of the classifier based approach using SVM and KNN in A.3
A.I REDUNDANCY EXPERIMENT ON THE GAS SENSOR DATASET
In Section 3.2 we conduct experiments on the Sonar dataset to demonstrate that all studied MIBFS algorithms are selecting redundant features, undermining the objective minimal feature subset in In this subsection, conduct the same experiment using the Gas Sensor dataset and summarize the performance in Table 3
algorithm
redundancy rate
MIM
23.6%
JMI
22.5%
JMIM
24.44%
mRMR
18.6%
GSA
15,4%
CMIFSI 27.7%
MICO
20.5%
Table 3: Redundancy Experiment Results on the Gas Sensor Dataset
A.2 SCORING FUNCTIONS THE FIVE MRWMR ALGORITHMS WITH/WITHOUT BUR
In this subsection, we follow the same notation as in Section 2 Namely, X,, Y, S, and are the feature, the label, the set of selected features, and the set of unselected features. respectively. (X,, denotes the unique relevance of feature X; (i.e., Jon (x,
[S2; Y) ( (X,): ): Y)). The five
representative MRwMR algorithms are: MIM Lewis (1992), JMI Yang & Moody (2000), JMIM Bennasar et (2015). mRMR Peng et al. (2005) and GSABrown et al. (0112). The scoring functions for these algorithms with and without BUR are shown below.
(X.) arg max I(X;Y)
(9)
X,E3
JumARR (X.) = arg max (I 3) x 1(X;Y)+++++))))))) + 8x Jxx(X,)
(10)
X
(X,) arg max
Σ I(x
(11)
X.EŠ
X,E8
JIMEBR (X.) = arg max
(1- x Σ 1(X,, x
1
+Bx x Jun(X.)
(12)
S
X:eš
X,ES
1
Juneee (X;) = arg max I(X;Y)-
T(X,, X,)
(13)
[
X,E3
X;ES
J JuRos SUR (Xi) arg max (1- x (I(X;:Y)
I((,,,,))) + (X,,
(14)
S
X,ES
X,ES
Jnamal Xx) = arg max
min I (X;,, X,:Y)
(15)
x.cš
X,69
JANNA BUR
(X.((.)) (Xi) = arg max (1 (1- B) x (min I [X,, Xy:)) + B x J(n(X,)
(16)
XyES
X.E
Josn (X,) arg max S(X,) X:ES
(17)
JOsA BUR (Xi) = arg max (1 (1-3) 3) x I(X,, + +x xx((x
(18)
X,6
12

---

Under review as a conference paper at ICLR 2021
A.3 MORE EXPERIMENTAL RESULTS USING SVM AND KNN
In the main paper, we only provide experimental results using random forest. We note that similar performance trends can be observed using SVM and KNN as well (see SVM in Table 4 and KNN in 5).
Gas sensor
Colon
Sonar
Madelon
Leukemia
Isolet
MIM
96.76%(933
81.13%(67)
96.65%(99) (1.52%(36) (3702%(37)
88.22%(1448
MIM BUR
82.23%((63) 96.76%((93)
61.83%(35) (3.75%(39)
97.72%(97)
88.73%((21)
MIMBBUR.CL
97.23%
82.58 (65)
74.28 (48)
65.21 % (42)
89.48% 98123%(81)
MRMR
96.76%(999
62.07%(39) 75.27%(49) 83.11%(40)
97.72%(777
(1333%((135)
mRMR BUR
96.84% 994%%)555 (95)
75.49 82.97%(42)
(43)
(2.07%(39)
97.72%(777
89.62 (131)
mRMR BUR_CLF
(89)
84,31 % (37)
(4.23%(48)
98.63% (477
(61)
(9.45%((127)
JMI
73.32%((17) 98.84%(89)
61.25%(31) (4113%(41)
89.91%((33) (6.78%(86)
JMI BUR
96.84%(889
73.98% (15)
73.25%(37)
89.47%((444) (86) (355)
JMLBB BUR_CLF
97.23%
73.62%((1)
75.32%(49)
(1.88%(37)
(8.15%(75)
90.08 (118)(11))
JMIM
96.70%((98)
(6.85%((19
(3.35%(49)
(4125%((1)
88.58%((38)) (63)
IMIM_BUR
(9.7
(23)
73.37%(38)
62.31%(44)
88.62%((44)) (9.28%(94)
IMIM_BRR.CO
79.62% 97.35%(95)
(27)
(2.47%(43) (3762%(37
98.42%(71)
89.21 % (133)
GSA
96.48%:(95)
(357
(3.51%(39) (36)
89.37%((38) (6.82%(67)
GSA_BUR
96.48%(95
(2728%(27)
(311
63.27%(33
97.14%(54)
(1.75%((31)
GSA BUR CLF
96.86%
(323
(3.95%(46)
98.62% 63.72%(37)
(68) 90.02 (137)
Table 4: Peak averaged accuracy and corresponding number of features required for each MRwMR and MRwMR-BLL BUR based algorithm 0.10 using SVM. For MRwMR-BUL based algorithms, the algorithm with an extension of CLF indicates that the algorithm uses the classifier based method to estimate UR. Otherwise, the algorithm estimates UR using the KSG estimator.
Gas sensor
Colon
Sonar
Madelon
Leukemia
Isolet
MIM
(9.10)(87))
(4.12%(63)
95.51%(46) 73.38%(57) (2495%(24)
(1349%(139)
MIM_BUR
85.32%(411 (85)
83.44%((23
75.76%((11
96.13%
(1385%(133))
MIM BUR CLF
(80)8%(80)
86.42%((45
84.82 % (45)
(7.21%(67)
80.02%(131) (29)
mRMR
98.)6%((83)
82.57%(45)
(30)
81.02%(129) 97.72%(554 (61)
RRM BUR
99.08%(811
85.02 (32))%
85.81 (31)
76.21%((7)
82.13%((117) 97.70%(533
MARRR_BU_CC
99.21 (81)
86.52 (35))
83.9
76.58%(69)
98.61 (48))
(1215%(121)
JMI
99.16%(722
(0772%(47)
8384
79. 150(()))
95.58%((6)
(1375%((335
JMI BUR
99.14%%(65
(43)
85.02% (45)
(533)
96.52%(45)
(1348%((337
JMLBUR_CLF
99.32%
71.35% (41)
(391 79.72%(59) (483
81.05%
JMIM
(85)
72.
(4.17%(49) (85)
95.81%((4) (6.)
19.16%%(337
JMIM BUR
76322%(63) (8))
(3782%(47)
(6.36%((3)
79349%(139) 934)
IMIM_BUR.CLI
98197%(81)
(7.58%(33)
84.23% (41)
79.51 (53)
929)
80.27 (131)
GSA
95.98%(58)
85. (72))
83.46%(48)
78. (5.1))(33)
95.61%(59)
(1397%
GSA_BUR
99.019((56)
(61)
(406
78, (854
80.25%(121) (5.84%((2)
GSA_BUR_CLF
99.27 (67)
88.53%
85.51 (45)
78.57 (49) 98.63%
81.23% (117)
Table Peak averaged accuracy and corresponding number of features required for each MRwMR and MRWMR-BUL BUR based algorithm (3 = 0.1) using KNN. For MRwMR-BU based algorithms, the algorithm with an extension of CLF indicates that the algorithm uses the classifier based method to estimate UR. Otherwise, the algorithm estimates UR using the KSG estimator.
13


Reviews:
[{"review_id": "j4ERfUOeT7I", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors recognized the function of unique relevance (UR) of features for optimal feature selection and augmented the existing mutual information based feature selection (MIBFS) methods by boosting unique relevance (BUR). As a result, they proposed a new criterion called MRwMR-BUR. Experimental results are provided to show that MIBFS with UR consistently outperform their unboosted conterparts in terms of peak accuracy and number of features required. \n\nOverall,  this paper has thrown new light upon MI based feature selection and the results are valuable. However, I think the paper could be improved from the following two aspects.\n\n1. To their credit, the authors have introduced the background of MI, OR, UR, and II.  However, some of the points are not made clear. For example, at the end of Sec. 3.1, they stated that \"We note that the optimal feature subset S* may also contain features with OR and no UR at certain situations. For example.....\",   which seems contradictory to the Proposition 1.\n\n2. The authors are suggested to  improve the organization and the presentation of the paper. The current version is not easy to follow. For example, there appears the term J_{UR}(X_i) in Eq. (5), but I do not see any explicit definition of it until I reading Appendix A.2.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IG_7fQst1PA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: The author response has not changed my opinion that there is insufficient new material in this paper vs the ISIT paper, and the presentation of the material from the ISIT paper does not note that this material was previously presented there. Without clarity in what is the novel material claimed in this paper it should not be accepted.\n\nThis paper presents a modification to existing information theoretic feature selection algorithms which adds a strong relevance term estimated using a k-nn MI estimator. It's a slightly expanded copy of a paper published at IEEE International Symposium on Information Theory 2020, referenced as \"Exploring unique relevance for mutual information based feature selection\" Liu & Motani 2020. This paper contains the same experimental results, same plots, same theoretical description and is in most ways a direct copy of the ISIT paper, violating ICLR's dual submission policy. I think the only new material is the experimental results on the CLF algorithms.\n\nThe authors should note that GSA-BUR has already been published as the SURI technique (referenced as Liu et al 2018), even considering this paper is a version of the ISIT 2020 paper.\n\nThe notion of \"redundancy rate\" is ill-defined, and the experiments which measure it are not discussed. If it's measuring the joint mutual information then it's a measure of the approximation used, and also a factor of the greedy search algorithm (which is used by all the criteria considered in the paper).\n\nThe proof of proposition 1 follows from the definition and has been known since the 1990s when strong relevance was introduced.\n\nGiven the CLF variants use a classifier to estimate the probabilities then the authors should validate that the features are still widely useful (by transfering the features found using the SVM to the RF) or compare performance against a wrapper like RFE as it's similarly expensive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5rhO7x26fg0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work suggests improving mutual informaton based feature selection methods with an extra term (i.e., the unique relevance (UR)), and introduces a hyper-parameter $\\beta$ to weight the UR. The work is easy to follow. However, the perspectives and methods are not novel. And there is a technical flaw in the analysis. \n\n1. It seems to me, in terms of methodology, that the difference of this work to [Liu, 2018] and [Liu, 2020] is that this work has two ways to estimate UR, one is based on the KSG estimator, another is based on a classifier. However, the objective (i.e., introducing a weighted term on UR) is not new and is shown in both [Liu, 2018] and [Liu, 2020].\n\n2. Author wants to justify the theoretical guarantee of mutual information based feature selection. However, the perspectives are not new. For example, it is widely acknowledged that feature selection can be interpreted with information bottleneck. There is also a very early work that explicitly implement this idea [1]. On the other hand, author wants to link the feature selection with the state-of-the-art on the learning dynamics of deep neural networks (i.e., the fitting phase and the compression phase). However, the connection seems strange. Note that, one is on the objective itself, another is on the training (or optimization) of the objective.  \n[1] Hecht, Ron M., and Naftali Tishby. \"Extraction of relevant speech features using the information bottleneck method.\" In Ninth European Conference on Speech Communication and Technology. 2005.\n\n3. The UR terminology is not new, and has been mentioned in very early works in mutual information based feature selection.\n\n4. A technical flaw: I disagree that author mentions that \"the UR is the same to the unique information in the partial information decomposition (PID) frameowork\". Note that, in PID, there are only three equations but with four unknowns, which makes the estimation of unique information an underdetermined problem (unless we made extra assumptions). However, UR can be simply estimated by its analytical expression.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "1RkC4yl2rdw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an investigation of Mutual information based feature selection methods and the use of unique relevance (UR) with mutual information. \n\n**Cons:**\n- This paper investigates the feature selection methods based on mutual information and integrates the UR term to the methods. \n- The authors compared their method with different Mutual information features selection based methods. The results indicate that the method improves the baseline. \n- As mentioned by the authors, using the UR measure can be beneficial in many domains, such expandability of a neural network model.\nThe experimental section is acceptable.\n\n**Weakness:** \n- This paper has substantial overlap with the literature [1-3], especially [1].\n- It would be beneficial to mention how this work is different from these papers.\n- A Comparison with these methods is missing.\n\n**Minor comments:**\n1) In the abstract, mention that MIBFS stands for mutual information based feature selection.\n2) Equation (1) is not consistent ( $arg \\  min\\  f(...)$ )\n3) In equation (4), what is function $H()$. A description right after using the function for the first time is beneficial.\n4) Selected feature subset -> set of selected features / selected features\n\n**Score:**\n- I vote to reject this paper. My main concern is the substantial overlap with the published literature. In my opinion, there is not enough novel material in this paper for this conference.\n\n**Questions:**\n1) What is the difference between the method presented in this paper and [1]\n2) Provide a comparison with the literature [1-3].\n\n**Additional Feedback:**\n- An optimal set of features in the classification does not necessarily contain features with unique relevance to the class labels. For instance, assume two features that do not have any unique information, but their combination provides unique information. The investigation of this issue by using a synthetic data set would be interesting. \n\n**References:**\n- [1] Liu, Shiyu, and Mehul Motani. \"Exploring Unique Relevance for Mutual Information based Feature Selection.\" 2020 IEEE International Symposium on Information Theory (ISIT). IEEE, 2020.\n- [2] Liu, Shiyu, and Mehul Motani. \"Feature selection based on unique relevant information for health data.\" arXiv preprint arXiv:1812.00415 (2018).\n- [3] Liu, Shiyu, et al. \"Suri: Feature selection based on unique relevant information for health data.\" 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2018.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: During the discussion among reviewers, we have shared the concern that this work has a significant overlap with [Liu et al. 2018] and [Liu & Motani 2020]. Although the authors tried to address this concern by the author response, I also think that the difference is not enough. In particular, the reviewers pointed out that Figure 1, Table 1, and Figure 3 are exactly the same with those in [Liu, 2020], and Proposition 2 in [Liu & Motani 2020] is Proposition 1 in this paper. Since these overlaps are not acceptable, I will reject the paper.
Decision: Reject