Robust Feature Selection using Sparse Centroid-Encoder

Abstract: We develop a sparse optimization problem for the determination of the total set of features that discriminate two or more classes. This is a sparse implementa- tion of the centroid-encoder for nonlinear data reduction and visualization called Sparse Centroid-Encoder (SCE). We also provide an iterative feature selection al- gorithm that first ranks each feature by its occurrence, and the optimal number of features is chosen using a validation set. The algorithm is applied to a wide vari- ety of data sets including, single-cell biological data, high dimensional infectious disease data, hyperspectral data, image data, and GIS data. We compared our method to various state-of-the-art feature selection techniques, including three neural network-based models (DFS, SG-L1-NN, G-L1-NN), Sparse SVM, and Random Forest. We empirically showed that SCE features produced better classi- fication accuracy on the unseen test data, often with fewer features.

Reviews and author responses:

This paper presents a neural network based approach for feature selection. The presented approach uses the existing work from the literature (based on centroid-encoder neural networks). As the primary novelty, the authors introduce the sparsity term (which is the L_1 norm of the parameters as a regularization term) to the original centroid-encoder loss (compare Eq. 1 and Eq. 2 in the paper). Authors then introduce k-fold training to choose the dominant features and demonstrate their results on four different datasets. Both of those additions (use of L1 norm and k-fold training) are known in the literature and therefore, this paper reads as an application paper.  Strengths:
1) Introduces a sparsity term to the centroid encoder networks. 
2) Provides results on four different datasets.

Weaknesses: 
1) The novelty is limited in this paper and it reads as an application paper. 
2) It is also not clear whether a real deep network can be replaced with this feature-selection algorithm while a deep network does this job in an end to end manner. The paper needs to revise its motivation to highlight the benefits of using a feature selection algorithm over using an end to end deep architecture with actual comparisons with state of the art networks.
3) Section 2.2. which describes the actual technique is too short (about half a page).
4) The experiments section focusing on datasets and experimental details take more space in this paper than the section providing the details of the proposed technique formally. 
5) There are some statements made without providing solid evidence to clarify what is actually meant (for example the statement about lamda. See below: Summary of the review).
6) Figure 1 does not explain clearly how the introduced network differs from the Centroid-encoders. That needs to be redrawn with more details.
 This paper presents a neural network based approach for feature selection. The presented approach uses the existing work from the literature (based on centroid-encoder neural networks). As the primary novelty, the authors introduce the sparsity term (which is the L_1 norm of the parameters as a regularization term) to the original centroid-encoder loss (compare Eq. 1 and Eq. 2 in the paper). Authors then introduce k-fold training to choose the dominant features and demonstrate their results on four different datasets. Both of those additions (use of L1 norm and k-fold training) are known in the literature and therefore, this paper reads as an application paper. 

It is also not clear whether a real deep network can be replaced with this feature-selection algorithm while a deep network does this job in an end to end manner. The paper needs to revise its motivation to highlight the benefits of using a feature selection algorithm over using an end to end deep architecture with actual comparisons with state of the art networks. 

This version of the submitted paper, focuses more on the datasets and the experiments than focusing on the analysis of the newly introduced term and providing details of the introduced technique in a more formal way. For example, it is not clear why the authors picked L1 norm instead of L2? 

The statement made by the authors: “ A larger value of lamda will promote higher sparsity resulting more near-zero weights in SPL. In other words, lamda is a knob that controls the number of features selected from the input data.” is also vague. Does that mean if I use 1Million as the value of lamda, I would get a sparser value? Can the authors support that statement with some results? That is also missing in this paper.

As a result, this paper reads as an application paper and this version also reads as an incomplete work. Consequently, I recommend reject.


This paper proposes a sparse centroid-encoder network by inserting a sparsity promoting layer between the input layer and the first hidden layer of the centroid-encoder network (Ghosh et al. (2018), Aminian et al. (2021)). In such an architecture, the authors define the loss function as the centroid-encoder loss (Ghosh et al. (2018), Aminian et al. (2021)) with a l_1 penalty on the sparsity promoting layer. Feature selection experiments are conducted on five datasets. +The relation between the proposed method and centroid-encoder is clearly explained.

+The authors provide source code for the proposed method.

-The proposed method is a simple extension of the centroid-encoder network by adding a l_1 sparsity regularization, which is incremental.

-The performance improvement of the proposed method over the baselines is small, especially on the GM12878 and MNIST datasets. Experimental results are not convincing.
 Imposing sparsity on the centroid-encoder network is incremental, and the novelty is very limited. The performance improvement is also not significant.


The paper extends the space centroid-encoder NN model with an additional layer for performing feature selection. Sparse features are induced with L1 regularisation. strengths:

- Good performance on most of the studied data sets with less features and at least comparable performance than the other comparison model.
- simple idea

weaknesses:

- not a significant innovation, the idea of L1 selection has been well established and is in fact known not to satisify oracle properties (Zou 2006)
- Since L1 is known to be inconsistent, presumably it is here too, but there is no theoretical study nor empirical study on simulated data where the truth is known.
- The statement "We think the multivariate approach of SCE makes it a better features detector than RF"
seems to be unsupported. RF also is multivariate and features are evaluated in the context of others, allbeit a little more opaquely. Since there's no theory supporting this over RF, no synthetic experiments where truth is known, and no investigation into the features selected by RF/SCE on the datasets (the single cell data and the microarrays), it seems possible only to make the statement that the sparse model has better performance on some data.
- It's unclear what the errors are as it is not stated.

References:

1. Zou, H. (2006). The Adaptive Lasso and Its Oracle Properties. In Journal of the American Statistical Association (Vol. 101, Issue 476, pp. 1418–1429). Informa UK Limited. https://doi.org/10.1198/016214506000000735 The improvement is a logical application of L1 sparsity inducing regularisation to insert a feature selection layer before an existing model. The performance on some dataset seems competitive, on others (the single cell data) it is at least comparable. Unfortunately the lack of investigation, either empirically or theoretically, into the actual selections reduces the significance of this work.

The authors propose a novel feature selection algorithm by introducing a L1-penalty over the Centroid-Encoder loss function. The experimental results shows the algorithm can outperform DFS algorithms in certain datasets. 

The main contribution of this paper is the addition of the l1-regularization to the Centroid-Encoder loss, changing it to a feature selection algorithm. Overall, I think the idea is interesting, but the contribution is minimal. The authors only introduce a l1-regularization loss over a state-of-the-art algorithm. It could be viewed as a special case of the DFS algorithm applied to the CE loss instead of the classic cross-entropy loss. In fact, the DFS authors mentioned their l1-regularization mask can be applied to any gradient descent minimization loss.

Regarding to the experimental section, it requires to be tested over more datasets like the ones in the 2005 NIPS Feature Selection Challege. Importante recent supervised FS algorithms are also missing, namely: ILFS [1], SFS [2], CAE [3] or L2X [4].

There is also no information about the NN architecture: number of layers, activation functions, optimizers or number of training epochs. Furthermore, some works mentioned above are able to achieve similar results in the MNIST dataset by only using over 100 features. 


[1] Roffo, G., Melzi, S., & Cristani, M. (2015). Infinite feature selection. In Proceedings of the IEEE International Conference on Computer Vision (pp. 4202-4210).

[2] Cancela, B., Bolón-Canedo, V., Alonso-Betanzos, A., & Gama, J. (2020). A scalable saliency-based feature selection method with instance-level information. Knowledge-Based Systems, 192, 105326.

[3] Balın, M. F., Abid, A., & Zou, J. (2019, May). Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning (pp. 444-453). PMLR.

[4] Chen, J., Song, L., Wainwright, M., & Jordan, M. (2018, July). Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning (pp. 883-892). PMLR. Overall, it is an interesting approach, but the contribution is minimal for a high venue like ICLR. Also the experimental section should be expanded to include more datasets and state-of-the-art FS algorithms.

Meta-review: All reviewers point out the lack of significant novelty in the proposed method. In addition, there is a lack of theoretical justification, and relatively weak empirical results, to support the claim that the proposed method is competitive or outperforms the myriad of methods existing for feature selection.

Decision: Reject