Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the well-defined feature selection problem. The authors pointed out that the redundancy issue of the weight-based feature selection methods. To tackle this, the authors explored the second-order feature covariance matrix and proposed a two-stage framework including feature correlation matrix via knowledge contrastive distillation and feature selection on the masked correlation matrix via graph segmentation. Extensive experiments demonstrated the effectiveness of the proposed methods. 
main_review: Pros:
1. The motivation is well introduced with illustrative examples in Figure 1.
2. The graph segmentation-based framework is novel and interesting to me, which is different from the mainstream weight-based feature selection.
3. The philosophy of knowledge contrastive distillation is logically sound. I have a minor question that will be posted in the Cons section.
4. The experimental results are extensive. The authors compared with 10 methods including several recent deep methods on 12 datasets and demonstrated the significant improvements. 
5. The in-depth exploration is a plus, which helps understand the proposed algorithm.
Cons:
1. The authors employed the knowledge contrastive distillation to learn the feature correlation matrix. The traditional unsupervised feature selection methods based on the first-order feature matrix usually employs a clustering method to purse pseudo labels for feature selection. It is suggested that the authors would like to do another ablation study without knowledge contrastive distillation.
2. The motivation of this paper is to address the redundancy issue of selected features. Although the authors provided the illustrative examples in Figure 1, it would be better to demonstrate the proposed method does not suffer from this issue on the same datasets in the experimental part. 

summary_of_the_review: This paper considers the well-defined feature selection problem. The authors pointed out that the redundancy issue of the weight-based feature selection methods. To tackle this, the authors explored the second-order feature covariance matrix and proposed a two-stage framework including feature correlation matrix via knowledge contrastive distillation and feature selection on the masked correlation matrix via graph segmentation. Extensive experiments demonstrated the effectiveness of the proposed methods.  The motivation is well introduced with illustrative examples. The graph segmentation-based framework is novel and interesting to me, which is different from the mainstream weight-based feature selection. The philosophy of knowledge contrastive distillation is logically sound.
 It is suggested that the authors would like to do another ablation study without knowledge contrastive distillation. Although the authors provided the illustrative examples in Figure 1, it would be better to demonstrate the proposed method does not suffer from this issue on the same datasets in the experimental part.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a two-stage second-order unsupervised feature selection via knowledge contrastive distillation model that incorporates the second-order covariance matrix with the first-order data matrix for unsupervised feature selection. A sparse attention matrix that
can represent second-order relations between features is learned in the first stage, a relational graph based on the learned attention matrix is learned to perform graph segmentation for feature selection.
main_review: strengths
1. The proposed method is interesting;
2. Experimental results are good;
weaknesses
1. Some presentations are not clear enough.
1.1 The first line of page 4, why $\Theta_M$ is forced to be symmetric?
1.2 Why the masked matrix can be defined by Eq.(2)?
2. Since the GCN training process relies on pseudo labels, how to ensure the reliability of the clutering results obtained by PCA and K-means?
summary_of_the_review: Interesting idea, but some unclear presentations and motivations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Feature selection reduces high-dimensional data by identifying comprehensible informative features. This paper proposed an unsupervised feature selection method, SOFT, by combining the information from the second-order covariance matrix with the first-order data matrix. By empirical experiments, the author(s) demonstrated the effectiveness of the proposed method.
main_review: # Strengths

This paper proposed an interesting feature selection algorithm, which leveraged the information from the second-order covariance matrix with the first-order data matrix. On empirical experiments, the proposed method showed superior performance over several baselines, to some extend.

# Weaknesses

Here are some of my comments/questions. I would appreciate it if the author(s) could give a response. If I am wrong, please correct me. Thanks.

- The author(s) stated that "Most of these methods apply the linear feature selection matrices and select the representative features by ranking their feature weight vector. Such operations treat the feature set independently and fail to tackle the complex high-order relationship...". While, as far as I know, the model in [3] actually can globally exploit the complex feature relationships, and the models in [1] and [3] are both flexible to nonlinear feature selection. So what are the limitations of existing methods that the paper was trying to address?

- The authors demonstrated the effectiveness of the proposed method through empirical experiments. Currently, there are many new developments in feature selection, for example, [1] and [2]. I noticed the author(s) cited [1], but why not compare SOFT to more non-linear feature selection models such as AEFS? If so, the effectiveness of the proposed method will be relatively convincing.

- Comparing Table 2 with Figure 3, it is observed that the performance of SOFT is not so stable as other methods, not only on L.Cancer as the author(s) pointed out. So I encourage the author(s) might compare the stability of selected features from SOFT with baseline comparisons (for example, do stability analysis for selected features from different methods) before this paper is published in influential conferences or journals.

- From Table 4 in supplementary material,  it seems that SOFT has no obvious advantage over other baselines in terms of efficiency, doesn't it? If so, efficiency has no advantage and performance is not too stable, then what is the purpose of SOFT? could the author(s) clarify or explain the motivation more? I look forward to hearing that. Thanks.

- Additionally, this paper is generally well written, but some places (some issues) in this paper should be further clarified/fixed.

   - The font for the legend in Figure 3 is too small;

   - The same full name and abbreviation appeared several times in the paper;

   - About the architecture of used baselines, the author(s) stated that "For TSFS, CAE, and InfFS, we use default settings provided in their open-source codes."; however, it is not so clear. For example, for CAE, both linear and nonlinear structures are discussed in [3]. Which one did the author(s) of this paper use?







   - [1] Han et al., Autoencoder inspired unsupervised feature selection. In International Conference on Acoustics, Speech and Signal Processing, 2018.

   - [2] Doquet et al., Agnostic feature selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019. 


   - [3] Muhammed Fatih Balın, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning, 2019.
summary_of_the_review: In general, I think this paper is relatively interesting, and I was initially interested by the title of this paper. I would like to increase my score if the author(s) could give convincing responses to the previous comments/questions in the Weaknesses part. Thanks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: For unsupervised feature selection, this paper proposes a two-stage second-order method. Knowledge contrastive distillation is also incorporated for feature learning. Experiments on various datasets validate the effectiveness.


main_review: For unsupervised feature selection, this paper proposes a two-stage second-order method. Knowledge contrastive distillation is also incorporated for feature learning. Experiments on various datasets validate the effectiveness.

Pros:

(1) The overall presentation and organization are good.

(2) The presented results are good on these related datasets.

Cons:

(1) My main concern lies in the experiments. NDFS achieves the best results among the compared baselines. However, NDFS is published in AAAI 2012, which is 9 years ago. The improvement of the proposed method over NDFS is also marginal on several datasets. Besides, the difference among these baselines is also marginal. Though the authors also compare with two recent methods, including CAE and InfFS, their performance is even lower than NDFS since they adopted different datasets in their original papers.  Therefore, the experimental results are not convincing. 

(2) The novelty is not satisfying. All these modules, including the second-order information, have been well studied in the area of unsupervised learning. The proposed method is a combination of these methods. The perspective for unsupervised feature selection by feature relationship learning and graph segmentation is not new from my point of view.

(3) For the clustering task, many methods have already achieved much better results on COIL20, ORL, and even larger datasets, including CIFAR10 and CIFAR100.

(4) The computational complexity is very expensive, which is much higher than many compared methods.

(5) This paper is rejected by NeurIPS 2021, and I happened to review this paper several months ago. The authors do not incorporate any suggestions in this resubmitted manuscript. The main body is totally the same, and only one dataset for evaluation is changed.


summary_of_the_review: This paper proposes a rather complex and expensive new method for unsupervised feature selection that performs marginally better than the baselines (many quite old) on the set of datasets they have chosen to present in the paper. The exposition could also be clearer, but the idea seems more complex than it is interesting. In all, the paper does not rise to the standards of significance and novelty of ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a two-step unsupervised feature selection algorithm. Contraru to classic FS approaches, the authors try to infer important relations between pairs of features. In the final step, a graph segmentation approach is used to select the most important features. Experimental results show promising results.
main_review: Overall, I think it is an interesting approach to unsupervised feature selection. Although the computational cost, both spatial and temporal, is very high (a correlation matrix is used), the idea of searching for strong feature correlations is interesting. However, I have some concerns regarding the contribution:

1.- Some of the decisions are not clearly stated. For instance: In the graph construction, why the authors delete the las 10% features? Why they set to zero all values smaller than the median? In my opinion, the authors should include an ablation study that would shed some light over all those decisions.

2.- The same idea should be applied to the pseudo label generation and the evaluation metrics. I expect to see how this algorithm behaves when using a more complex approach than kmeans, like DEC [1] or IMSAT [2], for instance.

3.- Regarding to the problem of the accuracy drop whenever they increase the selected features (20% or higher), I wonder if there could be a different graph segmentation algorithm that could prevent it.


[1] Xie, J., Girshick, R., & Farhadi, A. (2016, June). Unsupervised deep embedding for clustering analysis. In International conference on machine learning (pp. 478-487). PMLR.

[2] Hu, W., Miyato, T., Tokui, S., Matsumoto, E., & Sugiyama, M. (2017, July). Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning (pp. 1558-1567). PMLR.
summary_of_the_review: Overall, I think it is an interesting idea with promising results, but more experiments have to be done to clearly state the performance of the algorithm, as well as a clear reasoning behing all the decisions the authors made in it.
review_title: null

Meta-review: This paper proposes a new two stage second-order unsupervised feature selection method via knowledge contrastive distillation. In the first stage, a sparse attention matrix that represents second order statistics is learned. In the second stage, a relational graph based on the learned attention matrix is constructed to perform graph segmentation for feature selection. 

This proposed method contains some new and interesting ideas and is novel in the unsupervised feature selection setting, though some components such as the second order affinity matrix are not totally new. The proposed method is technically sound. The authors compared their method with 10 methods including several recent deep methods on 12 datasets and demonstrated consistent improvements.
 
However, there are some concerns from the reviewers, even after the discussion phase. 1) The computational efficiency of the proposed method seems to be low. Since one goal of feature selection is to speed up downstream tasks, the efficiency of feature selection itself should also be considered. I suggest the authors analyze the computational bottleneck of the proposed method and improve the efficiency. 2) More ablation studies can be added to illustrate how the proposed method removes the redundancy issues of the selected features. 3) Some metrics like supervised classification accuracy can be potentially used as a metric. Though supervised classification is impossible in the unsupervised learning setting, running the experiments on some datasets that have labels by pretending having on label is one way to evaluate the method.

Overall, the paper provides some new and interesting ideas. However, given the above concerns, the novelty and significance of the paper will degenerate. Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a strong one if the concerns can be well addressed.
Decision: Reject