Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: Update: The author response has not changed my opinion that there is insufficient new material in this paper vs the ISIT paper, and the presentation of the material from the ISIT paper does not note that this material was previously presented there. Without clarity in what is the novel material claimed in this paper it should not be accepted.

This paper presents a modification to existing information theoretic feature selection algorithms which adds a strong relevance term estimated using a k-nn MI estimator. It's a slightly expanded copy of a paper published at IEEE International Symposium on Information Theory 2020, referenced as "Exploring unique relevance for mutual information based feature selection" Liu & Motani 2020. This paper contains the same experimental results, same plots, same theoretical description and is in most ways a direct copy of the ISIT paper, violating ICLR's dual submission policy. I think the only new material is the experimental results on the CLF algorithms.

The authors should note that GSA-BUR has already been published as the SURI technique (referenced as Liu et al 2018), even considering this paper is a version of the ISIT 2020 paper.

The notion of "redundancy rate" is ill-defined, and the experiments which measure it are not discussed. If it's measuring the joint mutual information then it's a measure of the approximation used, and also a factor of the greedy search algorithm (which is used by all the criteria considered in the paper).

The proof of proposition 1 follows from the definition and has been known since the 1990s when strong relevance was introduced.

Given the CLF variants use a classifier to estimate the probabilities then the authors should validate that the features are still widely useful (by transfering the features found using the SVM to the RF) or compare performance against a wrapper like RFE as it's similarly expensive.
review_title: This paper is a slightly expanded version of work published elsewhere.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors recognized the function of unique relevance (UR) of features for optimal feature selection and augmented the existing mutual information based feature selection (MIBFS) methods by boosting unique relevance (BUR). As a result, they proposed a new criterion called MRwMR-BUR. Experimental results are provided to show that MIBFS with UR consistently outperform their unboosted conterparts in terms of peak accuracy and number of features required. 

Overall,  this paper has thrown new light upon MI based feature selection and the results are valuable. However, I think the paper could be improved from the following two aspects.

1. To their credit, the authors have introduced the background of MI, OR, UR, and II.  However, some of the points are not made clear. For example, at the end of Sec. 3.1, they stated that "We note that the optimal feature subset S* may also contain features with OR and no UR at certain situations. For example.....",   which seems contradictory to the Proposition 1.

2. The authors are suggested to  improve the organization and the presentation of the paper. The current version is not easy to follow. For example, there appears the term J_{UR}(X_i) in Eq. (5), but I do not see any explicit definition of it until I reading Appendix A.2.

review_title: A meaningful job for MI based feature selection

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work suggests improving mutual informaton based feature selection methods with an extra term (i.e., the unique relevance (UR)), and introduces a hyper-parameter $\beta$ to weight the UR. The work is easy to follow. However, the perspectives and methods are not novel. And there is a technical flaw in the analysis. 

1. It seems to me, in terms of methodology, that the difference of this work to [Liu, 2018] and [Liu, 2020] is that this work has two ways to estimate UR, one is based on the KSG estimator, another is based on a classifier. However, the objective (i.e., introducing a weighted term on UR) is not new and is shown in both [Liu, 2018] and [Liu, 2020].

2. Author wants to justify the theoretical guarantee of mutual information based feature selection. However, the perspectives are not new. For example, it is widely acknowledged that feature selection can be interpreted with information bottleneck. There is also a very early work that explicitly implement this idea [1]. On the other hand, author wants to link the feature selection with the state-of-the-art on the learning dynamics of deep neural networks (i.e., the fitting phase and the compression phase). However, the connection seems strange. Note that, one is on the objective itself, another is on the training (or optimization) of the objective.  
[1] Hecht, Ron M., and Naftali Tishby. "Extraction of relevant speech features using the information bottleneck method." In Ninth European Conference on Speech Communication and Technology. 2005.

3. The UR terminology is not new, and has been mentioned in very early works in mutual information based feature selection.

4. A technical flaw: I disagree that author mentions that "the UR is the same to the unique information in the partial information decomposition (PID) frameowork". Note that, in PID, there are only three equations but with four unknowns, which makes the estimation of unique information an underdetermined problem (unless we made extra assumptions). However, UR can be simply estimated by its analytical expression.  
review_title: Perspectives and methods are not novel, there is a technical flaw.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an investigation of Mutual information based feature selection methods and the use of unique relevance (UR) with mutual information. 

**Cons:**
- This paper investigates the feature selection methods based on mutual information and integrates the UR term to the methods. 
- The authors compared their method with different Mutual information features selection based methods. The results indicate that the method improves the baseline. 
- As mentioned by the authors, using the UR measure can be beneficial in many domains, such expandability of a neural network model.
The experimental section is acceptable.

**Weakness:** 
- This paper has substantial overlap with the literature [1-3], especially [1].
- It would be beneficial to mention how this work is different from these papers.
- A Comparison with these methods is missing.

**Minor comments:**
1) In the abstract, mention that MIBFS stands for mutual information based feature selection.
2) Equation (1) is not consistent ( $arg \  min\  f(...)$ )
3) In equation (4), what is function $H()$. A description right after using the function for the first time is beneficial.
4) Selected feature subset -> set of selected features / selected features

**Score:**
- I vote to reject this paper. My main concern is the substantial overlap with the published literature. In my opinion, there is not enough novel material in this paper for this conference.

**Questions:**
1) What is the difference between the method presented in this paper and [1]
2) Provide a comparison with the literature [1-3].

**Additional Feedback:**
- An optimal set of features in the classification does not necessarily contain features with unique relevance to the class labels. For instance, assume two features that do not have any unique information, but their combination provides unique information. The investigation of this issue by using a synthetic data set would be interesting. 

**References:**
- [1] Liu, Shiyu, and Mehul Motani. "Exploring Unique Relevance for Mutual Information based Feature Selection." 2020 IEEE International Symposium on Information Theory (ISIT). IEEE, 2020.
- [2] Liu, Shiyu, and Mehul Motani. "Feature selection based on unique relevant information for health data." arXiv preprint arXiv:1812.00415 (2018).
- [3] Liu, Shiyu, et al. "Suri: Feature selection based on unique relevant information for health data." 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2018.

review_title: The paper presents an investigation of Mutual information based feature selection methods and the use of unique relevance (UR) with mutual information. 

Meta-review: During the discussion among reviewers, we have shared the concern that this work has a significant overlap with [Liu et al. 2018] and [Liu & Motani 2020]. Although the authors tried to address this concern by the author response, I also think that the difference is not enough. In particular, the reviewers pointed out that Figure 1, Table 1, and Figure 3 are exactly the same with those in [Liu, 2020], and Proposition 2 in [Liu & Motani 2020] is Proposition 1 in this paper. Since these overlaps are not acceptable, I will reject the paper.
Decision: Reject