Algorithmic stability and generalization of an unsupervised feature selection algorithm

1 Introduction . High-dimensional data is challenging due to the curse of dimensionality [ 7 ] . Dimensionality reduction is an important technique for dealing with such data , comprising two typical approaches : feature extraction and feature selection . The former , including principal component analysis ( PCA ) [ 35 ] and autoencoder ( AE ) [ 41 , 5 ] , is widely used in various fields such as biology [ 3 ] and computer vision [ 49 , 18 , 38 ] . Nonetheless , new features produced by feature extraction form a new space and , in general , do not have a direct correspondence to original features , leading to difficulty in deriving interpretable insights for the domain problems , such as biomarker identification and drug discovery . Alternatively , feature selection identifies essential features from the original feature space , providing critical interpretations and insights in many tasks [ 10 , 45 ] , e.g. , gene functional enrichment analysis , biomarker detection , and high-throughput screening for drug discovery [ 22 , 47 , 27 ] . Stability is an important characteristic of an algorithm , which quantifies the sensitivity of the output to the perturbation of its training samples [ 12 , 19 ] . The current stability analyses of feature selection algorithms mainly focus on similarity-based and frequency-based stability measures for practical assessment , e.g. , [ 16 , 21 , 34 ] , rather than the algorithms themselves , leaving feature selection with proper algorithmic stability [ 8 ] an unmet need . Especially , existing unsupervised feature selection algorithms use the empirical error , or its proxy , to represent generalization error but provide no theoretical guarantee of such use , leaving their ability to generalize to new data unclear . To address these issues , in this paper we propose a novel unsupervised feature selection algorithm with a proven algorithmic stability guarantee . Our approach trains a neural network ( NN ) to globally score all features and a dependent sub-NN to select the features with the highest scores to reconstruct the ∗Correspondence should be addressed to : qiang.cheng @ uky.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . original data . Mathematically , we analyze our new algorithm and provide a generalization error upper bound , ensuring its algorithmic stability for guaranteed learning performance . Our contributions are summarized in the following : • We propose an innovative approach for feature selection . It constructs a NN with a feature scorer to globally score all the features and a dependent sub-NN with a feature selector to locally evaluate the representation ability of highly scored features . Thus , our algorithm is capable of both globally exploring and locally excavating essential features . • We establish performance guarantees for our algorithm , including a proven convergence rate O ( 1/ ( nmin { √ λ1 , λ1 } ) + 1/n ) to ensure uniform stability and a rate O ( 1/ ( √ nmin { √ λ1 , λ1 } ) + 1/ √ n ) for generalization error . Here , n is the number of training samples , and λ1 is a regularization parameter . • We confirm the effectiveness of our proposed algorithm with extensive experiments on 10 real datasets . It achieves more competitive performance for data reconstruction and downstream classification tasks than state-of-the-art methods . Notably , the features selected by our algorithm have performance comparable to the original features . Further , the properties revealed by our theoretical analysis and the stability of our algorithm-selected features are empirically verified . The remainder of the paper is organized as follows . We first discuss the related work , then present our proposed algorithm . Next , we show that our proposed algorithm is uniformly stable and has a guaranteed generalization bound . Finally , we conduct extensive experiments to validate our proposed new algorithm and the properties related to our proved generalization bound . 2 Related work . In the literature , the approaches for bounding generalization error can be generally grouped into two categories : One by controlling the complexity of hypothesis spaces and the other by focusing on the property of learning algorithms . The former usually need to define an additional complexity measure on the hypothesis space , such as the Vapnik-Chervonenkis dimension [ 44 ] and the Rademacher complexity [ 6 ] ; the latter mainly include algorithmic stability [ 8 ] and robustness-based analysis [ 50 ] . In this paper , we focus on algorithmic stability to analyze the stability and generalization of our proposed new algorithm . Algorithmic stability in learning theory . Algorithmic stability is an important tool for analyzing the generalization of algorithms in learning theory . It characterizes the sensitivity of the loss when the inputs to an algorithm are changed . Simply speaking , if an algorithm is stable , then its loss does not change significantly when the training samples are modified slightly , such as deleting and replacing a sample . Since the inception of the notion [ 42 , 43 ] , the concepts and properties of stability have become a significant learning theory topic . Bousquet and Elisseeff [ 8 ] have shown that stability has a direct connection with generalization in that the uniform stability of a learning algorithm implies a tight generalization error bound . Thanks to this connection , algorithmic stability has been widely used for generalization analysis of learning algorithms , including regularized least squares regression [ 8 ] , multi-class support-vector-machine classification [ 39 ] , multi-task learning [ 28 ] , supervised autoencoders [ 23 ] , and distributed learning [ 46 ] . Especially in [ 28 ] , the auxiliary tasks are used for regularization , and so is the reconstruction [ 23 ] . It is noted that all these studies are on supervised learning . For unsupervised learning , stability analysis of algorithms is yet to be developed ; particularly , the role of regularization in the stability bound remains elusive . Stability studies of feature selection algorithms . A few existing studies about the stability of feature selection algorithms are mainly on stability measures for quantifying the sensitivity to input changes . In [ 21 ] a Shannon entropy-based stability measure is proposed for evaluating the stability of selected features . In [ 16 ] the stability of a feature selection algorithm is assessed by comparing the average pairwise similarity of all feature subsets obtained from different subsamples by the algorithm . The recent work [ 34 ] generalizes the requirements of stability measures into five properties and proposes a statistical estimator for stability to satisfy these properties . Although able to provide relevant assessment information about the selected features to some extent , most stability measures have no essential connection with the corresponding feature selection algorithms . In addition , they are demonstrated empirically in general , without theoretical insights and generalization guarantees . The only exception appears to be the theoretical proof for uniform weighting stability [ 25 ] . This work builds an algorithm called feature weighting as regularized energy-based learning ( FREL ) , proves stability , and extends to a further ensemble version . FREL is for supervised feature selection , and the uniform weighting stability does not appear to be linked with generalization . In brief , despite the wide use of algorithmic stability in analyzing many learning algorithms , its use and analysis for unsupervised feature selection algorithms are still unmet needs . Unsupervised feature selection algorithms . Many unsupervised feature selection algorithms have been proposed in recent years to meet the demand for effectively handling large-scale and highdimensional data . They can be generally classified into four categories : filter , wrapper , embedder , and hybrid approaches [ 2 , 24 ] . In this paper , we will mainly compare our algorithm with ten typical unsupervised feature selection algorithms , including Laplacian score ( LS ) [ 17 ] , principal feature analysis ( PFA ) [ 30 ] , SPEC [ 52 ] , multi-cluster feature selection ( MCFS ) [ 9 ] , unsupervised discriminative feature selection ( UDFS ) [ 51 ] , nonnegative discriminative feature selection ( NDFS ) [ 26 ] , Autoencoder feature selector ( AEFS ) [ 15 ] , agnostic feature selection with slack variables ( AgnoS-S ) [ 11 ] ,2 graph-based infinite feature selection ( Inf-FS ) [ 40 ] , and concrete autoencoders ( CAE ) [ 1 ] . LS , SPEC , and Inf-FS are filter approaches ; PFA can be classified as a wrapper approach ; the other six are embedded algorithms . Among them , AEFS , AgnoS-S , and CAE are AE-based feature selection methods : AEFS combines AE regression and ` 2,1 regularization on the weights of the encoder to obtain a subset of useful features ; AgnoS-S adopts AE with the ` 1 norm on slack variables in the first layer of AE to implement feature selection ; CAE replaces the first hidden layer of AE with a concrete selector layer [ 31 ] , and then it selects the features with a high probability of connection to the nodes of the concrete selection layer . While frequently used , these methods have no theoretical analysis of their algorithmic stability and leave their generalization abilities to new data unclear . Indeed , they generally lack such needed abilities , as demonstrated in our experiments ( see Table 3 ) . 3 Notations and preliminary . Let n , m , k , and d be the numbers of training samples , features , selected features , and reduced dimensions . Let X ∈ Rn×m be a sample matrix , and XT be its transposition . A lowercase capital letter like wm denotes a vector ; Diag ( wm ) represents a diagonal matrix with the diagonal wm . wmaxkm stands for the operation to keep the k largest entries of wm while making the other entries 0 . Let φ : Rm → Rm be an element-wise operation . ‖ · ‖F , ‖ · ‖2 , ‖ · ‖1 , and ‖ · ‖∞ respectively denote the Frobenius , ` 2 , ` 1 , and infinity norms . The sample space of data is denoted as X . We assume all samples are bounded , i.e. , ∀x ∈ X , ∃κ1 > 0 , such that ‖x‖2 6 κ1 . Besides , ∀x , x′ ∈ [ 0 , ∞ ) , for a function f , ∆t ( f ( x ) , x′ ) , f ( x ) − f ( x+ t∆x ) , where ∆x = x′ − x and t ∈ [ 0 , 1 ] . Let S , { xi ∈ X , i = 1 , 2 , . . . , n } be a finite set of training samples which are independently and identically distributed according to an unknown distribution P. We denote the sets after removing and replacing the i-th element from S by S\i , { x1 , . . . , xi−1 , xi+1 , . . . , xn } and Si , { x1 , . . . , xi−1 , x′i , xi+1 , . . . , xn } , respectively . Let A be an algorithm , and AS be a function picked by A from the hypothesis space H based on S. A is assumed to be deterministic and symmetric with respect to S. Let ` : X × X → [ 0 , +∞ ) be a loss function . The generalization error is L ( A , S ) , Ex [ ` ( AS , x ) ] = ∫ X ` ( AS , x ) dP , and the empirical error is Lemp ( A , S ) , 1 n n∑ i=1 ` ( AS , xi ) , where xi ∈ S , i = 1 , 2 , . . . , n. To study the stability of A , the leave-one-out error is used , Lloo ( A , S ) , 1 n n∑ i=1 ` ( AS\i , xi ) . 2In [ 11 ] , three variants of AE-based agnostic feature selection algorithms are proposed . AgnoS-S is generally the best among the three ; thus , in this paper we will compare our algorithm with AgnoS-S. By the empirical and leave-one-out errors , uniform stability is defined as follows : Definition 1 ( Uniform Stability [ 8 ] ) . An algorithm A has uniform stability β with respect to ` if ∀S ∈ Xn , i = 1 , 2 , . . . , n , ‖ ` ( AS , · ) − ` ( AS\i , · ) ‖∞ 6 β , where β is a function of n. Generally , for a uniformly stable algorithm , β decreases as O ( 1/n ) .

Summary: The authors propose a novel methodology for unsupervised feature selection. Their procedure is based on simultaneously training a feature importance scorer neural network and a feature selector neural network; both are embedded inside an encoder in an autoencoder. The overall network is trained by optimizing a regularized estimator, where the regularization parameter induces some algorithmic stability. The authors provide theoretical guarantees of the estimator in the form of upper-bounds for the algorithmic stability and generalization. Further, they demonstrate the effectiveness of their approach on multiple real datasets and empirically understand various features of their algorithm (e.g. sensitivity to choices of parameters) via demonstrations on synthetic data.