Title: Joint Feature and Differentiable $ k $-NN Graph Learning using Dirichlet Energy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Feature Selection

Differential k-NN Graph

Dirichlet Energy
Abstract: Feature selection (FS) plays an important role in machine learning, which extracts important features and accelerates the learning process. In this paper, we propose a deep FS method that simultaneously conducts feature selection and differentiable $ k $-NN graph learning  based on the Dirichlet Energy. The Dirichlet Energy identifies important features by measuring their smoothness on the graph structure, and facilitates the learning of a new graph that reflects the inherent structure in new feature subspace. We employ Optimal Transport theory to address the non-differentiability issue of learning $ k $-NN graphs in neural networks, which theoretically makes our method applicable to other graph neural networks for dynamic graph learning. Furthermore, the proposed framework is interpretable, since all modules are designed algorithmically. We validate the effectiveness of our model with extensive experiments on both synthetic and real-world datasets.

Joint Feature and Differentiable k -NN Graph Learning using Dirichlet Energy

Abstract

Feature selection (FS) plays an important role in machine learning, which extracts important features and accelerates the learning process. In this paper, we propose a deep FS method that simultaneously conducts feature selection and differentiable k -NN graph learning based on the Dirichlet Energy. The Dirichlet Energy identifies important features by measuring their smoothness on the graph structure, and facilitates the learning of a new graph that reflects the inherent structure in new feature subspace. We employ Optimal Transport theory to address the non-differentiability issue of learning k -NN graphs in neural networks, which theoretically makes our method applicable to other graph neural networks for dynamic graph learning. Furthermore, the proposed framework is interpretable, since all modules are designed algorithmically. We validate the effectiveness of our model with extensive experiments on both synthetic and real-world datasets.

1 Introduction

Feature selection (FS) is a critical technique in machine learning that identifies informative features within the original high-dimensional data. By removing irrelevant features, FS speeds up the learning process and enhances computational efficiency. In many real-world applications such as

image processing, bioinformatics, and text mining [1-3], FS techniques are widely used to identify important features, thereby providing some explanations about the results and boosting the learning performance [4-7].

While numerous FS methods have been proposed in both supervised and unsupervised settings, as several studies highlighted [8, 9], the nature of FS is more unsupervised due to the unavailability of task-specific labels in advance. The selected features should be versatile to arbitrary downstream tasks, which motivates us to focus on the unsupervised FS in this study. Related works have recently resorted to neural networks to exploit the nonlinear information within feature space. For example, AEFS [9] uses the group Lasso to regularize the parameters in the first layer of the autoencoder, so as to reconstruct original features based on the restricted use of original features. Another well-known method is CAE [8], which selects features by learning a concrete distribution over the input features. However, most unsupervised deep methods rely on the reconstruction performance to select useful features. On the one hand, if there exists noise in the dataset, the reconstruction performance will be terrible even if useful features are selected, since the noise cannot be reconstructed with these informative features (see our reconstruction experiments on Madelon in Section 4.3). On the other hand, it is difficult to explain why selected features reconstruct the original data well. These issues prompt us to seek a new target for unsupervised deep FS.

As the saying goes, ' birds of a feather flock together ', the homophily principle [10] suggests that similar samples tend to be connected in a natural graph structure within real-world data. This graph structure is useful for describing the intrinsic structure of the feature space, and is commonly used in machine learning studies [11, 12]. Building upon this graph structure, He et al. [13] introduce the Dirichlet Energy, which they call 'locality preserving power', as a powerful tool for unsupervised FS that is able to identify informative features reflecting the intrinsic structure of the feature space.

In many practical applications, the graph structure is not naturally defined and needs to be constructed manually based on the input features using some similarity measurements. The quality of features affects the quality of the constructed graph. As highlighted in [14], the useless features increase the amount of unstructured information, which hinders the exploration of inherent manifold structure within data points and deteriorates the quality of constructed graph. Therefore, the reference [14] proposes the UDFS method to discard such nuisance features and constructs a k -nearest-neighbor (NN) graph on the selected features using the heat kernel. Despite the good performance of UDFS, constructing graphs using the heat kernel may not reflect the intrinsic structure of the feature space. Besides, the sorting algorithms in learning the k -NN graph in UDFS is non-differentiable in neural networks, which restricts its application in downstream networks.

In this paper: (1) We propose a deep unsupervised FS network that performs simultaneous feature selection and graph learning by minimizing the Dirichlet Energy, thereby revealing and harnessing the intrinsic structure in the dataset. (2) Within the network, a Unique Feature Selector (UFS) is devised to approximate discrete and distinct feature selection using the Gumbel Softmax technique combined with decomposition algorithms. (3) Moreover, a Differentiable Graph Learner (DGL) is devised based on the Optimal Transport theory, which is capable of obtaining a differentiable k -NN graph that more accurately reflects the intrinsic structure of the data than traditional graph constructing methods. Due to the differentiability, DGL is also theoretically capable of serving as a learnable graph module for other graph-based networks. (4) The entire framework is developed algorithmically. Unlike most deep learning networks with complex components that are tough to decipher, each core module in our framework has an algorithmic and physically interpretable design, which greatly facilitates observing and understanding the network's internal operations during the learning process. (5) Experimental results on both synthetic datasets and real-world datasets demonstrate the effectiveness of our method.

Notations. For an arbitrary matrix M ∈R a × b , m i , m i , and m i,j denote the i -th row, the i -th column, and the ( i, j ) -th entry of M , respectively. Given a vector m ∈ R b , its ℓ 2 -norm is defined as ∥ m ∥ 2 = √ ∑ b i =1 m 2 i . Based on this, the Frobenius norm of M is defined as ∥ M ∥ F = √∑ a i =1 ∥ m i ∥ 2 2 . When a = b , the trace of M is defined as tr( M ) = ∑ a i =1 m i,i . Given two matrices M , N ∈ R a × b , we define their inner product as ⟨ M , N ⟩ = ∑ a i =1 ∑ b j =1 m i,j n i,j . 1 b denotes a b -dimensional column vector with all entries being 1 , and I b denotes a b -order identity matrix. Bool( cond ) is a boolean operator that equals 1 if cond is true, otherwise it equals 0 . Moreover, given a vector m ∈ R b , we define its sorting permutation in ascending order as σ ∈ R b , namely, m σ 1 ≤ m σ 2 ≤ · · · ≤ m σ b .

(a)

(b)

(c)

2 Dirichlet Energy

Let X ∈ R n × d be the data matrix with the n samples and d -dimensional features. In this paper, we assume that the features have zero means and normalized variances, 2 namely, 1 ⊤ n x i = 0 and x ⊤ i x i = 1 for i ∈ { 1 , . . . , d } . According to the homophily assumption [10], we assume that data X forms an inherent graph structure G with nodes standing for samples and edges standing for their correlations. In G , similar samples are more likely to connect to each other than dissimilar ones. Graph G can be represented with a similarity matrix S ∈ R n × n + , where s i,j denotes the similarity between x i and x j . If s i,j = 0 , it means that there is no connection between x i and x j in G , which is common in k -NN graphs since we only consider the local structure of the data space. Given an adjacency matrix S , 3 we define the Laplacian matrix [15] as L S = D -S , where D is a diagonal matrix whose diagonal entries represent the degrees of data points, namely, d i,i = ∑ n j =1 s i,j .

Based on the Laplacian matrix L S , we introduce the Dirichlet Energy [16] as a powerful tool to identify important features. Specifically, given the Laplacian matrix L S , the Dirichlet Energy of a graph signal v is defined as

In graph theory, each dimensional feature x i ∈ R n can be seen as a graph signal on G . The Dirichlet Energy in Eq. (1) provides a measure of the local smoothness [11] of each feature on graph G , which is small when the nodes that are close to each other on G have similar feature values. Hence, the Dirichlet Energy can be used to identify informative features by evaluating the consistency of the distribution of feature values with the inherent data structure. To demonstrate this, we provide an example in Fig. 1, where we generate a 2 -NN graph G including two bubbles, and compose the data X using the two-dimensional coordinates of the graph nodes. Then we set the graph signal v as the first coordinate x 1 and visualize it on G in Fig. 1(a). In Fig. 1(b) we change v to a random noise vector. While in Fig. 1(c), we change the graph structure to a random 2 -NN graph. We compute the Laplacian matrix L S of each figure and present the corresponding Dirichlet Energy in the figures. We can see that Fig. 1(a) achieves the best smoothness, whereas both Fig. 1(b) and Fig. 1(c) have poor smoothness due to a mismatch between the graph signal and the graph structure.

Based on the Dirichlet Energy, a well-known FS method called Laplacian Score (LS) 4 is proposed in [13]. However, the Laplacian matrix in LS is precomputed and fixed. If X contains too many irrelevant features, the quality of the graph G will be poor and not reflect the underlying structure. As illustrated in Fig. 1(c), a poor-quality graph will lead to the poor smoothness even if the right feature is selected, this insight motivates us to learn graph and features jointly during the learning process.

3 Proposed Method

In this paper, we devise a collaborative neural network driven by the Dirichlet Energy for joint feature and graph learning, as illustrated in Fig. 2. Generally, the proposed framework consists of two

𝝃𝝃

𝑖𝑖

(

1

)

𝝃𝝃

𝑖𝑖

(

2

)

𝝃𝝃

𝑖𝑖

(

𝑘𝑘+1

)

…

𝝃𝝃

𝑖𝑖

(

𝑘𝑘

)

1

𝑠𝑠𝑠𝑠

2

𝑛𝑛𝑑𝑑

3

𝑟𝑟𝑑𝑑

4

𝑠𝑠𝑡

𝒆𝒆

𝒊𝒊

𝒔𝒔

𝑖𝑖

Differentiable

k

-NN Selector

Dirichlet

Energy

𝜹𝜹

𝑖𝑖

(

𝑘𝑘

)

-𝟏𝟏

…

…

…

…

𝜖𝜖𝑰𝑰

𝒙𝒙1

𝒙𝒙

𝑑𝑑

𝒙𝒙4

𝒙𝒙3

𝒙𝒙2

�

𝒙𝒙2

�

𝒙𝒙1

�

𝒙𝒙

𝑚𝑚

�

𝑿𝑿

𝑬𝑬

Feature Selection

Distance

Measuring

DGL

𝒔𝒔

1

𝒔𝒔

2

𝒔𝒔

3

𝒔𝒔

𝑛𝑛

…

…

…

…

…

…

…

…

…

…

…

𝑤𝑤

11𝑤𝑤

21 𝑤𝑤31𝑤𝑤41

𝑤𝑤

𝑑𝑑1

𝑤𝑤

12𝑤𝑤

22 𝑤𝑤32𝑤𝑤42

𝑤𝑤

𝑑𝑑2

𝑤𝑤

1𝑚𝑚

𝑤𝑤

2𝑚𝑚

𝑤𝑤

3𝑚𝑚

𝑤𝑤

4𝑚𝑚

𝑤𝑤

𝑑𝑑𝑚𝑚

…

Gumbel

softmax

Gumbel

softmax

Gumbel

softmax

̂

𝑓𝑓

11

̂

𝑓𝑓 21

̂

𝑓𝑓 31

̂

𝑓𝑓

41

̂

𝑓𝑓

𝑑𝑑1

̂

𝑓𝑓

12

̂

𝑓𝑓 22

̂

𝑓𝑓 32

̂

𝑓𝑓

42

̂

𝑓𝑓

𝑑𝑑2

̂

𝑓𝑓

1𝑚𝑚

̂

𝑓𝑓

2𝑚𝑚

̂

𝑓𝑓

3𝑚𝑚

̂

𝑓𝑓

4𝑚𝑚

̂

𝑓𝑓

𝑑𝑑𝑚𝑚

�

𝑭𝑭

𝑓𝑓

11

𝑓𝑓 21

𝑓𝑓 31

𝑓𝑓

41

𝑓𝑓

𝑑𝑑1

𝑓𝑓

12

𝑓𝑓 22

𝑓𝑓 32

𝑓𝑓

42

𝑓𝑓

𝑑𝑑2

𝑓𝑓

1𝑚𝑚

𝑓𝑓

2𝑚𝑚

𝑓𝑓

3𝑚𝑚

𝑓𝑓

4𝑚𝑚

𝑓𝑓

𝑑𝑑𝑚𝑚

…

𝑭𝑭

UFS

…

…

…

…

…

…

…

…

…

…

⊤

: Matrix Multiplication

: Addition

: Matrix Transposition

: Cholesky Decomposition

𝑺𝑺

𝑳𝑳

𝑆𝑆

𝒆𝒆

1

𝒆𝒆

2

𝒆𝒆

3

𝒆𝒆

𝑛𝑛

Selecting

�

𝒙𝒙1

Selecting

�

𝒙𝒙2

…

Selecting

�

𝒙𝒙

𝑚𝑚

Dirichlet Energy

: Eigendecomposition

⊤

⊤

-𝟏𝟏

: Inverse

X

→

**

modules: the Unique Feature Selector (UFS) and the Differentiable k -NN Graph Learner (DGL). At the beginning, the input features X are selected with the learnable feature mask F generated by UFS, which is carefully designed to avoid the duplicate feature selection. Based on the selected data ˆ X , we measure the distances between different samples, and feed the resulting distance vectors of each sample into DGL to learn their k nearest neighbors. The adaptive graph structure and informative features are learned jointly under the Dirichlet Energy, so as to identify the optimal feature subset that effectively captures the underlying data structure.

3.1 Unique Feature Selector

Based on the original data X , the goal of FS is to identify a feature subset ˆ X ∈ R n × m from the original features by minimizing a prespecified target L obj ( ˆ X ) :

where m ≤ d denotes the number of selected features, and F ∈ R d × m denotes the selection matrix selecting m features from X . Different from existing methods that use the reconstruction error as L obj ( ˆ X ) , in this paper, we utilize the Dirichlet Energy in Eq. (1) for FS as follows:

Given the selection number m , L obj updates the network parameters by minimizing the Dirichlet Energy, thereby selecting m features that best reflect the intrinsic structure.

The constraints in problem (2) indicate that an ideal result F should be exact and unique . Exact means the result should exactly be the original features, instead of their linear combinations. Unique means each feature should be selected only once under a given number m . These two properties require F to be a binary and column-full-rank matrix including m orthogonal one-hot column vectors.

3.1.1 Approximating Discrete Feature Selection

It is difficult to learn a discrete F in neural networks due to its non-differentiable property. Inspired by [8], we propose to learn the discrete distribution using the Gumbel Softmax [17, 18] technique:

where W = [ w 1 , w 2 , . . . , w m ] denotes a learnable parameter. The random vector g i consists of d Gumbel-distributed variables g i,j , which is generated with u i,j sampled from Uniform distribution. Based on w i and g i , we obtain the approximated FS vector ˆ f i that represents the i -th selected feature. The distribution of ˆ f i is controlled by a non-negative temperature parameter T . A smaller value of parameter T will generate a better approximation of the one-hot vector, but will be more likely to be stuck in a poor local minimum. As suggested in [8], we employ the annealing schedule on T by initializing it with a high value and then gradually decreasing it during the learning process.

3.1.2 Selecting Unique Features

Despite having obtained the approximated FS vectors in neural networks, Eq. (4) does not consider the uniqueness requirement of FS. This is because Eq. (4) learns each selected feature separately, and does not consider the orthogonal constraint between columns in ˆ F , which is prone to result in the repeated selection of the same features. To address this issue, we develop a unique feature selector (UFS) in Algorithm 1, where ˆ 0 ∈ R ( d -m ) × m denotes the zero matrix. First, we add a small ⊤

1:

procedure

UFS(

ˆ

F

,

ϵ

)

2:

P

Λ

P

⊤

= ˆ

F

⊤

ˆ

F

+

ϵ

I

m

3:

LL

⊤

= ˆ

F

⊤

ˆ

F

+

ϵ

I

m

4:

F

=

[

Λ

1

/

2

P

⊤

ˆ

0

]

(

L

-

1

)

⊤

5:

return

F

Algorithm 1 UFS

6: end procedure enough perturbation ϵ I m ( ϵ > 0) on ˆ F ˆ F . Next, we perform the eigendecomposition (line 2) and the Cholesky decomposition (line 3) on the perturbed result respectively, and correspondingly obtain the diagonal matrix Λ ∈ R m × m , the orthogonal matrix P ∈ R m × m , and the lower triangle matrix L ∈ R m × m . Based on Λ , P , and L , we obtain the selection matrix F in line 4 and have the following conclusion:

Proposition 3.1. Given any real matrix ˆ F ∈ R d × m , one can always generate a column-orthogonal matrix F through Algorithm 1.

The proof of Proposition 3.1 is provided in Appendix S1. On the one hand, the small perturbation ϵ I m guarantees the column-full-rank property of F , thereby avoiding the duplicate selection results. On the other hand, the orthogonality property in Proposition 3.1 facilitates the approximation of discrete FS based on the matrix ˆ F . 5 We verify the efficacy of UFS in Section 4.2.

3.2 Differentiable k -NN Graph Learner

The existence of noise and irrelevant features may negatively affect the quality of the constructed graph. As depicted in Fig. 1, a low-quality graph structure can significantly perturb the smoothness of features and undermine the performance of feature selection. Hence, we propose to learn an adaptive graph during the learning process using the selected features.

3.2.1 Learning an Adaptive k -NN Graph Using Dirichlet Energy

Considering the objective function in Eq. (3), a natural way is to learn the similarity matrix S based on the Dirichlet Energy in L obj . However, this may yield a trivial solution where, for sample x i , only the nearest data point can serve as its neighbour with probability 1 , while all the other data points will not be its neighbours. To avoid this trivial solution, we propose to learn an adaptive graph by incorporating the Tikhonov regularization [19] of S into the Dirichlet Energy:

where α denotes the trade-off parameter between the Dirichlet Energy and the Tikhonov regularization. Note that each row s i in S can be solved separately, instead of tuning α manually, we model α as

a sample-specific parameter α i and determine it algorithmically, which plays an important role in learning k nearest neighbors for each sample. Based on problem (5), we define the distance matrix E with its entries being e i,j = ∥ (ˆ x i -ˆ x j ) ∥ 2 2 , then we solve each row s i in problem (5) separately as

Problem (6) can be solved easily by constructing the Lagrangian function and then using the KarushKuhn-Tucker(KKT) conditions [20]. By doing so, we obtain the solution of s i,j as

where σ = [ σ 1 , . . . , σ n ] denotes the sorting permutation over e i , i.e. e i,σ 1 ≤ · · · ≤ e i,σ n and δ ( k ) i denotes the selection vector identifying the k minimal values in e i .

Recall that we aim to learn k nearest neighbors for each sample, which implies that there are only k nonzero elements in s i corresponding to the nearest neighbors. To this end, we determine the trade-off parameters α i such that s i,σ k > 0 and s i,σ k +1 ≤ 0 . Then we have:

where ξ ( k ) i denotes an indicator vector identifying the k -th minimal value in e i . Setting α i as the maximum and substituting it into Eq. (7), we obtain the final solution as:

The detailed derivation of solution (9) can be found in Appendix S3. We note that the formulation in problem (6) bears similarity to CLR proposed in [21]. In Appendix S4, we discuss the connection between our method and CLR, and highlight the differences between the two w.r.t. the feature utilization and the sorting operation. Remarkably, the k -NN can be obtained easily in CLR using offthe-shelf sorting algorithms, which is not the case for neural networks due to the non-differentiability of sorting algorithms. To address this issue, we propose to transform the k -NN selection into a differentiable operator utilizing the Optimal Transport (OT) [22] technique as follows.

3.2.2 Differentiable k -NN Selector

Let µ = [ µ 1 , µ 2 , · · · , µ n 1 ] ⊤ and ν = [ ν 1 , ν 2 , · · · , ν n 2 ] ⊤ be two discrete probability distributions defined on the supports A = { a i } n 1 i =1 and B = { b j } n 2 j =1 respectively . The goal of OT is to find an optimal transport plan Γ ∈ R n 1 × n 2 between A and B by minimizing the following transport cost:

where C ∈ R n 1 × n 2 denotes the cost matrix with c i,j = h ( a i -b j ) > 0 being the transport cost from a i to b j . It is widely known that the solution of the OT problem between two discrete univariate measures boils down to the sorting permutation [23-25]. As stated in [25], if h is convex, the optimal assignment can be achieved by assigning the smallest element in A to b 1 , the second smallest to b 2 , and so forth, which eventually yields the sorting permutation of A .

Given a distance vector e , to learn selection vectors δ ( k ) and ξ ( k +1) , we set A = e , and B = [0 , 1 , . . . , k +1] , and define µ , ν , and c ij as

The optimal transport plan of problem (10) assigns the i -th smallest value e σ i to b i if 1 ≤ i ≤ k +1 , and assigns the remaining n -k -1 values in e to b k +2 . Namely,

̸

̸

Given a sample p , once we obtain the optimal transport assignment Γ based on e p , we calculate the variables δ ( k ) p and ξ ( k +1) p as follows:

where Γ i and Γ k +1 denote the i -th and the ( k +1) -th column of Γ , respectively. However, problem (10) is still non-differentiable. To address this issue, we consider the following entropy regularized OT problem:

where γ is a hyperparameter. The differentiability of problem (14) has been proven using the implicit function theorem (see [26, Theorem 1]). Note that a smaller γ yields a better approximation to the original solution in problem (10), but may compromise the differentiability of problem (14) [25]. Problem (14) can be solved efficiently using the iterative Bregman projections algorithm [27], the details of which are provided in Appendix S5.

4 Experiments

Our experiments fall into three parts: (1) Toy Experiments : First, we verify the FS ability and the graph learning ability of the proposed method on synthetic datasets. (2) Quantitative Analysis : Next, we compare the performance of selected features in various downstream tasks on real-world datasets and compare our method with other unsupervised FS methods. (3) Ablation Study : Finally, we verify the effect of UFS and DGL by testing the performance of the corresponding ablated variants. We also provide the sensitivity analysis in Appendix S6.6. The implementation details of all experiments can be found in Appendix S6.1.

4.1 Datasets

For the toy experiments, we generate three 20 -dimensional datasets named Blobs, Moons, and Circles (see Appendix S6.1.1 for generation details). On top of that, we evaluate the proposed method on twelve real-world datasets that include text, biological, image, and artificial data. Table 1 exhibits the details of these datasets, which include many high-dimensional datasets to test the performance of our method. We standardize all features to zero means and normalize them with the standard deviation.

4.2 Toy Experiments

In this section, we consider three synthetic binary datasets with increasing difficulty in separating different classes. The first two dimensions in each dataset contain useful features that indicate the underlying structure, while the remaining 18 dimensions are random noise sampled from N (0 , 1) . The presence of noise obscures the inherent structure of the data, which makes the graph learning process highly challenging. To see this, we generate 3-D plots of each dataset using the useful features and one noise feature, along with their 2-D projections on each plane, which are shown in Fig. 3(a). We can see that the noise blurs the boundary of different classes, especially in Moons and Circles. In addition, we used a heat kernel (abbreviated as Heat) with σ = 1 to learn the 5 -NN graph on 20 -dimensional features, as shown in Fig. 3(b). We can see that the heavy noise obscures the underlying structure of data points, resulting in a chaotic graph outcome.

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

X

1

2

1

0

1

2

X

2

1

0

1

2

4

3

2

1

0

1

2

3

2

1

0

1

2

1

0

1

2

2

1

0

1

2

1

0

1

2

0

5

10

15

20

0.00

0.25

0.50

0.75

1.00

2

1

0

1

2

1

0

1

2

0

5

10

15

20

0.0

0.5

1.0

1.5

2.0

X

1

2

1

0

1

X

2

1

0

1

2

4

3

2

1

0

1

2

1

0

1

1

0

1

1

0

1

1

0

1

0

5

10

15

20

0.00

0.25

0.50

0.75

1.00

1

0

1

1

0

1

0

5

10

15

20

0.0

0.5

1.0

1.5

2.0

X

1

2

1

0

1

2

X

2

2

1

0

1

2

4

3

2

1

0

1

2

3

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

0

5

10

15

20

0.00

0.25

0.50

0.75

1.00

2

1

0

1

2

2

1

0

1

2

0

5

10

15

20

0.0

0.5

1.0

1.5

2.0

Circles

Moons

Blobs

(a)

3

D Plot

(b)

k

-NN by Heat

(d) FS by Ours

(c)

k

-NN by Ours

(e)

k

-NN w/o UFS

(f) FS w/o UFS

Results. We test our method on toy datasets for selecting m = 2 target features. The results are presented in Fig. 3(c) and Fig. 3(d), which demonstrate the success of our method in learning target features and intrinsic structures simultaneously. Moreover, it can be seen from Fig. 3(d) that the proposed network obtains the approximately discrete FS vectors.

Learning Without Unique Feature Selector. In addition, we conduct an ablation study by removing the UFS module from the network and updating F using Eq. (4) only. The results are shown in Fig. 3(e) and Fig. 3(f), where we can see that, without UFS, the ablated model repeatedly selects the same feature on all datasets. It is also noteworthy that the nodes in the graph are mostly connected either horizontally or vertically, indicating the effectiveness of DGL in learning the local structure solely based on the single selected feature.

4.3 Quantitive Analysis

Circles. In addition, we used a heat kernel (abbreviated as Heat) with σ = 1 to learn the 5 -NN graph on 20-dimensional features, as shown in Fig. 3(b). We can see that the heavy noise obscures the underlying structure of data points, resulting in a chaotic graph outcome. Results : We test our method on toy datasets for selecting m = 2 target features. The results are presented in Fig. 3(c) and Fig. 3(d), which demonstrate the success of our method in learning target features and intrinsic structures simultaneously. Moreover, it can be seen from Fig. 3(d) that the proposed network obtains the approximately discrete FS vectors. Experimental Settings. In this section, we evaluate our method on real-world data. We partition each dataset into training data and testing data using an 8:2 ratio and identify useful features using training data. We then evaluate the performance of selected features on three downstream tasks: (1) Classification Accuracy : We train a random forest (RF) [39] classifier with 1000 trees using selected features and evaluate the prediction accuracy on the testing data. (2) Clustering Accuracy : We cluster the testing set with selected features using k -means [40], where the cluster number is set to #Classes. Then we align the results with true labels and calculate the accuracy. (3) Reconstruction RMSE : We build a 1-hidden-layer network with ReLU activation to reconstruct the original data using selected features. The hidden dimension is set to 3 m/ 2 except for AllFea, where the hidden size is set to d . The network is learned on the training set with selected features, and evaluated on the testing set using root mean square error (RMSE) normalized by d .

Learning Without Unique Feature Selector : In addition, we conduct an ablation study by removing the UFS module from the network and only using Eq. (4) for FS. The results are shown in Fig. 3(e) and Fig. 3(f), where we can see that the ablated model repeatedly selects the same feature on all datasets, which verifies the efficacy of UFS. It is also noteworthy that the nodes in the graph are mostly connected either horizontally or vertically, indicating that DGL is able to learn the local structure relying only on the single selected feature. Competing methods. We compare our methods with four deep methods ( CAE [8], DUFS [14], WAST [41], AEFS [9]) and three classical methods ( LS [13], RSR [42], UDFS [43]). Besides, we use all features ( AllFea ) as the baseline. To evaluate the performance of each FS method on downstream tasks, we average the results over 10 random runs with the feature number m varied in { 25 , 50 , 75 , 100 , 150 , 200 , 300 } except for Madelon, where m is varied in { 5 , 10 , 15 , 20 } since Madelon consists of only 20 useful features [29]. Appendix S6.1 provides details of the overall evaluation workflow, including the implementation and the parameter selection of each method.

4.3 Quantitive Analysis Experimental Settings : In this section, we evaluate our method on real-world data. We partition each dataset into training data and testing data using an 8:2 ratio and identify useful features using training data. We then evaluate the performance of selected features on three downstream tasks: (1) Classification Accuracy : We train a random forest (RF) [29] classifier with 1000 trees using selected features and evaluate the prediction accuracy on the testing data. (2) Clustering Accuracy : Results. Similar to [14], we present the best result w.r.t. m in Table 2, the standard deviations is provided in Appendix S6.2. We also present some reconstruction results on PIX10 by our method in Appendix S6.3. From Table 2, we find that: (1) Our method generally achieves the best performance in all three tasks, indicating that our method selects more useful features. (2) In particular, we beat DUFS in all Classification and Clustering tasks, as well as most cases of the Reconstruction tasks. Recall that DUFS also selects features based on the Dirichlet Energy, this result shows that the model (5) in our method explores a superior graph structure compared to the traditional Heat method. (3)

7

In classification and clustering, the best performance is mostly achieved by FS methods with fewer features, which verifies the necessity of FS. (4) AllFea achieves five optimums in Reconstruction, which is not surprising since, theoretically, AllFea can be projected to original features with an identity matrix. However, in biological data, the best reconstruction results are achieved by FS methods, probably because the high-dimensional data leads to overfitting in networks. (5) It is noteworthy that the reconstruction of Madelon poses a significant challenge for FS methods, indicating the difficulty of reconstructing noise even using useful features. This observation supports our claim regarding the lack of reasonability in selecting features based on the reconstruction performance in Section 1.

4.4 Ablation Study

In this experiment, we demonstrate the efficacy of the UFS and the DGL modules through ablation studies on six datasets: Madelon, PCMAC, Jaffe, PIX10, GLIOMA, and PROSTATE.

Effect of FS. Recall that we have demonstrated the efficacy of UFS in Fig. 3. To further verify the efficacy of FS in graph learning, we remove the entire FS module from the framework and learn the graph using all features based on DGL. We also compare the graph learning result using Heat. We

cluster the obtained graphs with the spectral clustering (SC) method to verify their qualities. We tune the parameter σ of Heat in { 1 , 2 , . . . , 5 } , and fix k = 5 for our method and the variant. The results are shown in Table 3, which shows that FS has a positive effect on graph learning compared with 'DGL only'. Besides, in Appendix S6.4, we visualize the learned graph on COIL-20 and Jaffe using t-SNE, which shows that using fewer features, we achieve separable graphs that contain fewer inter-class connections than other methods.

Effect of DGL. To verify the efficacy of DGL, we remove it from the model and learn the ablated variant with a fixed graph learned by Heat. Similar to Section 4.3, we first learn selected features using competing method, then evaluate the features in downstream tasks. We present the classification result in Table 3, and leave the other results in Appendix S6.5 due to limited space. We can see that our method significant outperforms the ablated variant, especially in Madelon. This is probably because the noise undermine the graph structure and disrupt the learning of informative features.

5 Discussion

Conclusion. This paper proposes a deep unsupervised FS method that learns informative features and k -NN graph jointly using the Dirichlet Energy. The network is fully differentiable and all modules are developed algorithmically to present versatility and interpretability. We demonstrate the performance of our method with extensive experiments on both synthetic and real-world datasets.

Broader Impact. This paper presents not only an effective deep FS method, but also a differentiable k -NN graph learning strategy in the context of deep learning. This technique is particularly useful for end-to-end learning scenarios that require graph learning during the training process. And we do notice this practical need in existing literature, see [44] for example. We believe our study will inspire researchers who work on the dimensionality reduction and graph-related researches.

Limitations. The major limitation of the proposed method is the lack of scalability, for which we do not evaluate our method on large datasets. This is because problem (14) requires an iterative solution, requiring storage of all intermediate results for back-propagation. While literature [26] proposes a memory-saving approach by deriving the expression of the derivative of Γ mathematically (see [26, Section 3]), it still requires at least O ( nk ) space to update all intermediate variables to learn k nearest neighbors for a singe sample, which results in a O ( n 2 k ) space complexity to learn for all n samples. This is a huge memory cost on large datasets. Although learning in batch seems to be the most straightforward solution, in our method, the neighbours of each sample are determined based on the global information of L S , which has an n × n size. This requires to load the entire batch's information during each iteration, for which we cannot employ subgraph sampling as other graph learning methods did to mitigate memory overhead. Another limitation of the proposed method is the low computational speed, as it is reported that the OT-based sorting can be slow [45].

The future developments of the proposed method are twofold. First, we will try more differentiable sorting algorithms to enhance computational speed. For example, reference [45] proposes to construct differentiable sorting operators as projections onto the permutahedron, which achieves a O ( n log n ) forward complexity and a O ( n ) backward complexity. Second, due to the large cost of the global relationship in L S , we are considering adopting a bipartite graph [16, 46] to make batch learning feasible. This graph introduces a small number of anchor points, which are representative of the entire feature space. By doing this, smoothness can be measured based on the distance between samples to anchors, for which sample-to-sample relationships are no longer needed and the batch learning is enabled. It is worth noting that this idea is still in its conceptual stage, and we will explore its feasibility in upcoming research.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62276212 and Grant 61872190, in part by the National Key Research and Development Program of China under Grant 2022YFB3303800, and in part by the Key Research and Development Program of Jiangsu Province under Grant BE2021093.

References

Y. Li, C. Luo, and S. M. Chung, 'Text clustering with feature selection by using statistical data,' IEEE Trans. Knowl. Data Eng. , vol. 20, no. 5, pp. 641-652, 2008.

C. He, K. Li, Y. Zhang, Y. Zhang, Z. Guo, X. Li, M. Danelljan, and F. Yu, 'Strategic preys make acute predators: Enhancing camouflaged object detectors by generating camouflaged objects,' arXiv preprint arXiv:2308.03166 , 2023.

H.-J. Yu and D.-S. Huang, 'Normalized feature vectors: A novel alignment-free sequence comparison method based on the numbers of adjacent amino acids,' IEEE/ACM Trans. Comput. Biol. Bioinformatics , vol. 10, no. 2, pp. 457-467, 2013.

C. He, K. Li, Y. Zhang, L. Tang, Y. Zhang, Z. Guo, and X. Li, 'Camouflaged object detection with feature decomposition and edge reconstruction,' in Proc. CVPR , 2023, pp. 22 046-22 055.

Z. Sun, G. Bebis, and R. Miller, 'Object detection using feature subset selection,' Pattern Recognit. , vol. 37, no. 11, pp. 2165-2176, 2004.

L. Xu, R. Wang, F. Nie, and X. Li, 'Efficient top-k feature selection using coordinate descent method,' in Proc. AAAI , vol. 37, no. 9, 2023, pp. 10 594-10 601.

C. He, K. Li, Y. Zhang, G. Xu, L. Tang, Y. Zhang, Z. Guo, and X. Li, 'Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping,' arXiv preprint arXiv:2305.11003 , 2023.

M. F. Balın, A. Abid, and J. Zou, 'Concrete autoencoders: Differentiable feature selection and reconstruction,' in Proc. ICML , vol. 97, 2019, pp. 444-453.

K. Han, Y. Wang, C. Zhang, C. Li, and C. Xu, 'Autoencoder inspired unsupervised feature selection,' in Proc. ICASSP , 2018, pp. 2941-2945.

M. McPherson, L. Smith-Lovin, and J. M. Cook, 'Birds of a feather: Homophily in social networks,' Annu. Rev. Sociol. , vol. 27, no. 1, pp. 415-444, 2001.

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, 'The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,' IEEE Signal Process. Mag. , vol. 30, no. 3, pp. 83-98, 2013.

R. Wang, P. Wang, D. Wu, Z. Sun, F. Nie, and X. Li, 'Multi-view and multi-order structured graph learning,' IEEE Trans. Neural Netw. Learn. Syst. , pp. 1-12, 2023.

X. He, D. Cai, and P. Niyogi, 'Laplacian score for feature selection,' in Proc. NIPS , 2005, pp. 507-514.

O. Lindenbaum, U. Shaham, E. Peterfreund, J. Svirsky, N. Casey, and Y. Kluger, 'Differentiable unsupervised feature selection based on a gated laplacian,' in Proc. NIPS , vol. 34, 2021, pp. 1530-1542.

U. Von Luxburg, 'A tutorial on spectral clustering,' Stat. Comput. , vol. 17, pp. 395-416, 2007.

F. R. Chung, Spectral graph theory . American Mathematical Soc., 1997, vol. 92.

E. Jang, S. Gu, and B. Poole, 'Categorical reparameterization with gumbel-softmax,' arXiv preprint arXiv:1611.01144 , 2017.

C. J. Maddison, A. Mnih, and Y. W. Teh, 'The concrete distribution: A continuous relaxation of discrete random variables,' in Proc. ICLR , 2017.

A. Tikhonov and V. Arsenin, Solutions of Ill-posed Problems . Winston, 1977.

S. P. Boyd and L. Vandenberghe, Convex Optimization . Cambridge University Press, 2014.

F. Nie, X. Wang, M. Jordan, and H. Huang, 'The constrained laplacian rank algorithm for graph-based clustering,' in Proc. AAAI , vol. 30, no. 1, 2016.

L. V. Kantorovich, 'Mathematical methods of organizing and planning production,' Manage. Sci. , vol. 6, no. 4, pp. 366-422, 1960.

F. Santambrogio, Optimal Transport for Applied Mathematicians . Birkhäuser, 2015.

G. Peyré and M. Cuturi, 'Computational optimal transport,' arXiv preprint arXiv:1803.00567 , 2020.

M. Cuturi, O. Teboul, and J.-P. Vert, 'Differentiable ranking and sorting using optimal transport,' in Proc. NIPS , vol. 32, 2019.

Y. Xie, H. Dai, M. Chen, B. Dai, T. Zhao, H. Zha, W. Wei, and T. Pfister, 'Differentiable top-k with optimal transport,' in Proc. NIPS , vol. 33, 2020, pp. 20 520-20 531.

J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyré, 'Iterative bregman projections for regularized transportation problems,' SIAM J. Sci. Comput. , vol. 37, no. 2, pp. A1111-A1138, 2015.

K. Lang, 'Newsweeder: Learning to filter netnews,' in Machine Learning Proceedings 1995 . Morgan Kaufmann, 1995, pp. 331-339.

I. Guyon, J. Li, T. Mader, P. A. Pletscher, G. Schneider, and M. Uhr, 'Competitive baseline methods set new standards for the nips 2003 feature selection benchmark,' Pattern Recognit. Lett. , vol. 28, no. 12, pp. 1438-1444, 2007.

C. L. Nutt, D. R. Mani, R. A. Betensky, P. Tamayo, J. G. Cairncross, C. Ladd, U. Pohl, C. Hartmann, M. E. McLaughlin, T. T. Batchelor, P. M. Black, A. von Deimling, S. L. Pomeroy, T. R. Golub, and D. N. Louis, 'Gene Expression-based Classification of Malignant Gliomas Correlates Better with Survival than Histological Classification1,' Cancer Res. , vol. 63, no. 7, pp. 1602-1607, 2003.

S. A. Nene, S. K. Nayar, H. Murase et al. , 'Columbia object image library (coil-100),' Tech. Rep. CUCS-005-96 , 1996.

H. Peng, F. Long, and C. H. Q. Ding, 'Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 27, no. 8, pp. 1226-1238, 2005.

D. Cai, C. Zhang, and X. He, 'Unsupervised feature selection for multi-cluster data,' in Proc. SIGKDD , 2010, pp. 333-342.

I. Petricoin, Emanuel F., D. K. Ornstein, C. P. Paweletz, A. Ardekani, P. S. Hackett, B. A. Hitt, A. Velassco, C. Trucco, L. Wiegand, K. Wood, C. B. Simone, P. J. Levine, W. M. Linehan, M. R. Emmert-Buck, S. M. Steinberg, E. C. Kohn, and L. A. Liotta, 'Serum Proteomic Patterns for Detection of Prostate Cancer,' J. Natl. Cancer Inst. , vol. 94, no. 20, pp. 1576-1578, 2002.

M. J. Lyons, J. Budynek, and S. Akamatsu, 'Automatic classification of single facial images,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 21, no. 12, pp. 1357-1362, 1999.

J. Khan, J. S. Wei, M. Ringner, L. H. Saal, M. Ladanyi, F. Westermann, F. Berthold, M. Schwab, C. R. Antonescu, C. Peterson et al. , 'Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks,' Nat. Med. , vol. 7, no. 6, pp. 673-679, 2001.

A. Spira, J. E. Beane, V. Shah, K. Steiling, G. Liu, F. Schembri, S. Gilman, Y.-M. Dumas, P. Calner, P. Sebastiani et al. , 'Airway epithelial gene expression in the diagnostic evaluation of smokers with suspect lung cancer,' Nat. Med. , vol. 13, no. 3, pp. 361-366, 2007.

T. Sim, S. Baker, and M. Bsat, 'The cmu pose, illumination, and expression database,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 25, no. 12, pp. 1615-1618, 2003.

L. Breiman, 'Random forests,' Mach. Learn. , vol. 45, pp. 5-32, 2001.

J. MacQueen, 'Classification and analysis of multivariate observations,' in 5th Berkeley Symp. Math. Statist. Probability , 1967, pp. 281-297.

G. Sokar, Z. Atashgahi, M. Pechenizkiy, and D. C. Mocanu, 'Where to pay attention in sparse training for feature selection?' in Proc. NIPS , 2022.

P. Zhu, W. Zuo, L. Zhang, Q. Hu, and S. C. Shiu, 'Unsupervised feature selection by regularized self-representation,' Pattern Recognit. , vol. 48, no. 2, pp. 438-446, 2015.

Y. Yang, H. T. Shen, Z. Ma, Z. Huang, and X. Zhou, 'L2,1-norm regularized discriminative feature selection for unsupervised learning,' in Proc. IJCAI , 2011, pp. 1589-1594.

S. Miao, Y. Luo, M. Liu, and P. Li, 'Interpretable geometric deep learning via learnable randomness injection,' in Proc. ICLR , 2023.

M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga, 'Fast differentiable sorting and ranking,' in Proc. ICML , vol. 119, 2020, pp. 950-959.

W. Liu, J. He, and S.-F. Chang, 'Large graph construction for scalable semi-supervised learning,' in Proc. ICML , 2010, pp. 679-686.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, 'Scikit-learn: Machine learning in Python,' J. Mach. Learn. Res. , vol. 12, pp. 2825-2830, 2011.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'Pytorch: An imperative style, high-performance deep learning library,' Proc. NIPS , vol. 32, 2019.

S1 Proof of Proposition 3.1

Proposition 3.1. Given any real matrix ˆ F ∈ R d × m , one can always generate a column-orthogonal matrix F through Algorithm 1.

Proof. We begin our proof by showing the feasibility of Algorithm 1 for any real matrix ˆ F , as the eigendecomposition, Cholesky decomposition, and inverse mentioned in the algorithm are subject to specific conditions. For simplicity, we represent A = ˆ F ⊤ ˆ F + ϵ I m , with ϵ > 0 . Note that for any nonzero real column vector z ∈ R m , we have

Hence, the matrix A is positive-definite and can be eigendecomposed as A = P Λ P -1 , where P ∈ R m × m is the square matrix whose i -th column p i is the eigenvector of A and Λ ∈ R m × m is the diagonal matrix whose diagonal entries are the corresponding eigenvalues. Moreover, it is easy to show that A is symmetric, for which we have P ⊤ = P -1 . Therefore, we prove that A can be decomposed as A = P Λ P ⊤ ( line 2 in Algorithm 1).

Since A is symmetric and positive-definite, we will be able to perform Cholesky decomposition on A as LL ⊤ = A ( line 3 in Algorithm 1), which yields a lower triangular matrix L ∈ R m × m whose diagonal entries are all real and positive. This means that the determinant of L is larger than zero and L is invertible, which provides the feasibility of L -1 ( line 4 in Algorithm 1). Consequently, the feasibility of Algorithm 1 for any real matrix ˆ F is proved.

Next, we show that F is a column-orthogonal matrix. We denote Q = [ Λ 1 / 2 P ⊤ ˆ 0 ] and have:

Then we prove the orthogonality of the matrix F as follows:

The proof is completed.

S2 Discussion on Algorithm 1

Although Algorithm 1 theoretically guarantees the orthogonality of the selection matrix F , utilizing this algorithm directly would bring us back to the problem of how to choose features and how to obtain discrete results. On one hand, the non-uniqueness of eigendecomposition in line 2 prevents us from ensuring the discrete properties of matrix F . On the other hand, it is important to note that in line 4 of Algorithm 1, we aim to construct a column full-rank matrix Q = [ Λ 1 / 2 P ⊤ ˆ 0 ] , whereas

the construction of Q is also not unique since we can insert d -m zero rows at any position within the original matrix Λ 1 / 2 P ⊤ to achieve the column full-rankness. The placement of these zero rows directly affects the result of feature selection.

Guided by Algorithm 1, we devise a more empirical approach by calculating F with F = ˆ F ( L -1 ) ⊤ , which effectively tackles the above two concerns. By doing so, we avoid the non-uniqueness of eigendecomposition, thereby obtaining a solution that is as discrete as ˆ F . Additionally, this approach ensures that the information of feature selection in ˆ F is retained within the column full-rank matrix.

Actually, F is an ϵ -approximation of column-orthogonal matrix, since we have:

The experimental results in Section 4.2 verify the effectiveness of this approach in successfully avoiding duplicate feature selection.

S3 Derivation of the Solution to Problem 5

Recall that we aim to solve the following problem to learn an adaptive k -NN graph:

Based on ˆ X , we define the quantity e i,j = ∥ (ˆ x i -ˆ x j ) ∥ 2 2 , then we solve each row in problem (S5) separately as:

We first omit the constraint s i,i = 0 and consider it later, and solve problem (S6) with the first two constraints, the Lagrangian function of which is as follows:

where λ i and β i,j are Lagrange multipliers. The derivative of L ( s i , λ i , β i ) w.r.t. s i,j is:

Then we have the Karush-Kuhn-Tucker(KKT) conditions [20] of problem (S7) as follows:

Then we have:

Recall that there are only k nonzero elements in s i corresponding to the nearest neighbors of sample i , according to the constraint ∑ n j =1 s i,j = 1 on k nonzero entries in s i , we have:

where δ ( k ) i denotes the selection vector identifying the k minimal values in e i , and σ = [ σ 1 , . . . , σ n ] denotes the sorting permutation over e i , i.e. e i,σ 1 ≤ · · · ≤ e i,σ n . Without loss of generality, we assume e i has no duplicates, namely e i,σ 1 < · · · < e i,σ n . Considering the constraint s i,i = 0 , since e i,i = 0 being the minimal value in e i holds for all samples, we replace e i,i with a sufficiently large value to skip over this trivial solution.

Substituting (S11) into (S10), we have:

Recall that there are only k nonzero entries in s i , we have

Note that we assume α i > 0 , then we have

where ξ ( k ) i is an indicator vector identifying the k -th minimal value in e i . According to (S14), we set α i as its maximal value as follows:

Substituting S15 into Eq. S12, we have:

Eq. (S16) is used for implementation in our code. Note that since e i,σ 1 < · · · < e i,σ k < e i,σ k +1 < . . . e i,σ n , we have

Then we obtain the solution of s i,j as

which is exactly the solution of Eq. (9) in our main paper:

S4 Connection to CLR

We note that a similar formulation to problem 6 has been proposed in [21] (coined CLR), which expects closer samples to have higher similarity. It aligns with the notion of 'smoothness' as we mentioned in Section 2. However, our method differs from CLR in at least two crucial aspects: Firstly, CLR measures the distance quantity e i,j across all original features, making it more sensitive to the noise and irrelevant features in the original data. In contrast, our approach learns the graph structure using only informative features, resulting in enhanced robustness against noisy features. Secondly, it is important to note that CLR is proposed in the context of traditional machine learning, where optimization is straightforward, as δ ( k ) i and ξ ( k +1) i can be updated using off-the-shelf sorting algorithms. Different from CLR, problem 6 is introduced in the realm of deep learning, where conventional sorting algorithms are non-differentiable and not applicable. This poses a huge challenge in learning an adaptive k -NN graph in neural networks. To overcome this challenge, we proposed to transform the topk selection into a differentiable operator using the Optimal Transport technique.

S5 Iterative Bregman Projections

In this paper, we employ the iterative Bregman projections [27] algorithm to solve the following problem:

Wefirst initialize two variables u ∈ R k +2 and K ∈ R n × ( k +2) as u i = 1 / ( k +2) and k i,j = e -c i,j /γ , respectively. Then based on the following formulations, we repeatedly updating u and v for ζ iterations:

where the division in Eq. (S21) is element-wise. In this paper, we set ζ as 200 . After updating ζ iteration, we obtain the optimal transport plan Γ as

S6 Supplementary Experimental Details

S6.1 Implementation Details

All experiments are conducted on a server equipped with an RTX 3090 GPU and an Intel Xeon Gold 6240 (18C36T) @ 2.6GHz x 2 (36 cores in total) CPU.

S6.1.1 Synthetic Datasets

We generate three datasets for toy experiments: (1) Blobs, (2) Moons, and (3) Circles. For each dataset, we generate the first two features using the scikit-learn library [47] by adding noise sampled from N (0 , 0 . 1) . Additionally, we generate 18 -dimensional noise features sampled from N (0 , 1) .

S6.1.2 Competing Methods

The implementation details of different methods, as well as their corresponding parameter selections are provided below:

Our Method : Our method is implemented using the PyTorch framework [48]. We train our method using the Adam optimizer for 1000 epochs on all datasets, with the learning rate searched from { 10 -4 , 10 -3 , 10 -2 , 10 -1 , 10 0 , 10 1 } . We search the parameter γ in { 10 -3 , 10 -2 , 10 -1 } and the parameter k in { 1 , 2 , 3 , 4 , 5 } . Note that the implementation of differentiable topk selector is based on the code provided by [26] in https://papers.nips.cc/paper_files/paper/2020/hash/ ec24a54d62ce57ba93a531b460fa8d18-Abstract.html , which provides a more memory-saving backward implementation compared to directly using the autograd method in PyTorch.

Where to Pay Attention in Sparse Training ( WAST ) [41]: We use the official code released in https://github.com/GhadaSokar/WAST . The parameter settings were adopted in accordance with Appendix A.1 of the original paper. Specifically, we train each dataset for 10 epochs using stochastic gradient descent with a learning rate of 0.1 for all datasets except for SMK, where the learning rate is set to 0.01. For the parameter λ , we set λ = 0 . 9 on Madelon and PCMAC, λ = 0 . 4 on all image datasets, λ = 0 . 1 on all biological datasets except SMK, and λ = 0 . 01 on SMK. The remaining parameters are kept as they were in the original paper.

Differentiable Unsupervised Feature Selection ( DUFS ) [14]: We use the official code released in https://github.com/Ofirlin/DUFS and use the parameter-free loss version of DUFS. For all datasets, we set k = 2 , and train the method with SGD with a learning rate of 1 for 10000 epochs according to Appendix S7 in the original paper. We set the parameter C = 5 on all datasets except for SRBCT, COIL, and PIX10, where C is set to 2 .

Concrete AutoEncoder ( CAE ) [8]: We use the official code released in https://github. com/mfbalin/Concrete-Autoencoders . Since we could not find too much description about the parameter settings on different datasets in the original paper, we run CAE with default settings in the code.

AutoEncoder Feature Selector ( AEFS ) [9]: The original code provided by the authors is implemented in MATLAB, and it requires a prohibitively long time to run this method on MATLAB. Therefore, following the treatment in [41], we use the code provided by the authors of CAE in https://github.com/Ofirlin/DUFS (see experiments/generate_comparison_figures.py in their repository). We search the parameter α in { 10 -9 , 10 -6 , 10 -3 , 10 0 , 10 3 , 10 6 , 10 9 } , and the size of the hidden layer in { 128 , 256 , 512 , 1024 } .

Laplacian Score ( LS ) [13]: We use the official code released in http://www.cad.zju.edu. cn/home/dengcai/Data/ReproduceExp.html#LaplacianScore . We use the heat kernel for graph construction, and fix the size of neighbors k as 5 for all datasets.

Regularized Self-Representation ( RSR ) [42]: We use the official code released in https://github.com/AISKYEYE-TJU/RSR-PR2015 . We search the parameter λ in { 10 -9 , 10 -6 , 10 -3 , 10 0 , 10 3 , 10 6 , 10 9 } .

Unsupervised Discriminative Feature Selection ( UDFS ) [43]: We use the code provided in https://guijiejie.github.io/code.html . We use the heat kernel for graph construction, and fix the size of neighbors k as 5 for all datasets. We search the parameter γ in { 10 -9 , 10 -6 , 10 -3 , 10 0 , 10 3 , 10 6 , 10 9 } .

S6.1.3 Evaluation Workflow

The overall evaluation workflow in Section 4.3 is shown in Algorithm S2, which includes two steps:

Given the dataset X and a prespecified feature number m , we first randomly split the dataset using an 8 : 2 ratio and select features based on the training set X tr by FS method F using different parameters Θ , as shown in Algorithm S1. This allows us to obtain FS results under different parameter candidates θ i , along with the corresponding reduced training data X ′ tr and testing data X ′ te . Based on the reduced data, we perform classification tasks on these datasets with the random forest, thereby obtaining the classification performance for each parameter combination. We select the parameter combination with the best classification performance as the optimal parameter θ ∗ for F .

Based on the optimal parameter θ ∗ , we construct the FS model F θ ∗ and evaluate its performance in different downstream tasks. To avoid randomness, we randomly split the dataset 10 times. With each random split, we use the training set X tr to select features, and obtain reduced training set X ′ tr and testing set X ′ te . We use these sets for downstream tasks including classification, clustering, and reconstruction, and obtain corresponding performance metrics. For each downstream task, we calculate the average metric over 10 runs as the performance of F for the given number m .

For each dataset, we vary the value of m and follow the aforementioned procedure to obtain the corresponding performance. For each downstream task, we report the best metric and the corresponding feature number m as the performance of the FS method in this downstream task.

Algorithm S1 Param_tuning

Input: Training data ( X tr , y tr ) , testing data ( X te , y te ) , selected number m , FS method F , and parameter set Θ = { θ i } .

Output: Optimal parameter θ ∗

.

1: for θ i in Θ do

2:

ξ = F θ i ( X tr , m ) ; ▷ Determining selected features ξ by F under the parameter θ i .

3: X ′ tr = X tr (: , ξ ) , X ′ te = X te (: , ξ ) ; ▷ Generating reduced datasets using selected features.

4: ACC = RF( X ′ tr , y tr , X ′ te , y te ); ▷ Evaluating selected features with random forest classifier.

5: if ACC > ACC ∗ then

6: ACC ∗ = ACC;

7: θ ∗ = θ i ;

8: end if

9: end for

Algorithm S2 Overall evaluation workflow

Input: Original dataset ( X , y ) , selected feature number m , FS method F , parameter set Θ = { θ i } , and downstream tasks T = {T i } .

Output: . Performance M = { M i } in downstream tasks.

1: Partitioning the dataset into training data ( X tr , y tr ) and testing data ( X te , y te ) .

3: for j = 1 : 10 do

2: Determining θ ∗ by Param_tuning( ( X tr , y tr ) , ( X te , y te ) , m , F , Θ ) in Algorithm S1;

4: Partitioning the dataset into training data ( X tr , y tr ) and testing data ( X te , y te ) .

6: X ′ tr = X tr (: , ξ ∗ ) , X ′ te = X te (: , ξ ∗ ) ;

5: ξ ∗ = F θ ∗ ( X tr , m ) ;

7: for T i in T do

9: end for

8: m { i, j } = T i ( X ′ tr , y tr , X ′ te , y te ) ; ▷ Evaluating the performance in downstream tasks.

10: end for

T

11:

for

i

in

T

do

13: end for

12: M i = Average ( m { i, : } ) ;

S6.1.4 Evaluation Metrics

We employ two metrics in our experiments: the accuracy (ACC) and the root mean square error (RMSE) normalized by d .

The formulation of ACC is

where y ∈ R n denotes the groundtruth labels and ˆ y ∈ R n denotes the prediction label. The formulation of RMSE normalized by d is

where X = [ x 1 ; x 2 ; . . . ; x n ] and ˆ X = [ˆ x 1 ; ˆ x 2 ; . . . ; ˆ x n ] denote the original feature matrix and the reconstructed feature matrix, respectively.

S6.1.5 Formulation of Heat Kernel Method

Here we describe the heat kernel (Heat) method compared in this paper. To implement Heat, we first compute the similarity matrix ˆ S as follows:

Based on ˆ S , we keep the k -nearest neighbors for each sample. Namely, for each sample i , we obtain its similarity vector s i as

where K ( x i ) denotes the k nearest neighbors of x i . When we need to calculate the Laplacian matrix using S (for example, when we analyze the effect of DGL in Section 4.4), we use the symmetrized version of S :

S6.2 Standard Deviations of Quantitative Analysis

Table S1, Table S2, and Table S3 exhibit the mean and the standard deviations of the results in Table 2 in Section 4.3.

S6.3 Reconstruction Results on PIX10

Fig. S1 presents the reconstruction results on PIX10 achieved by our method. We can see that our method is able to reconstruct the original images of 10000 dimensions reasonably well with only 300 features. Notably, the reconstructions capture important appearance details, including reflections, hair tips, and facial features, demonstrating the effectiveness of our method.

Reconstructed Images

Images

S6.4 Graph Visualization

We visualize COIL-20 and Jaffe with t-SNE, and plot the graph structures obtained by different methods. The results are shown in Fig. S2, where red lines represent the intra-class connections and blue lines represent inter-class connections. Unlike 'Heat' and 'DGL only' that use the original features for visualization, we visualize the data points using only selected features. Remarkably, our method successfully achieves separable structures using t-SNE, demonstrating its ability to capture features that reflect the intrinsic data structure.

(a) Heat

(b) DGL only

(c) Our

S6.5 Extended Results for Ablation Study

Table S4 and Table S5 present the clustering result and the reconstruction result of the ablation study on DGL in Section 4.4:

S6.6 Additional Experiment: Parameter Sensitivity Analysis

In this section, we analyze the effect of the parameters of our method, including the learning rate, the number of nearest neighbors k , the hyperparameter γ in the entropy regularized OT problem 14, and the selected feature number m . We use four real-world datasets, including a text dataset PCMAC, an artificial dataset Madelon, an image dataset Jaffe, and a biological dataset PROSTATE.

The analysis is based on the optimal parameter obtained in Section 4.3. For each dataset, we fix the values of the remaining parameters and vary the value of one parameter at a time. We retrain our method using the updated parameter combination and evaluate the corresponding FS result with the random forest. This allows us to observe the impact of different parameters on the performance of our method. For example, to analyze the effect of the learning rate on Madelon, we keep k , γ , and m at their optimal values, then we vary the learning rate in { 10 -4 , 10 -3 , 10 -2 , 10 -1 , 10 0 , 10 1 } (the range as we described in Appendix S6.1.2), and evaluate their corresponding performance in the classification task with the random forest. The overall results are shown in Fig. S3, where the stars represent the results using optimal parameters.

One the one hand, we observe that the variations in the learning rate, k , γ have little impact on the performance of our method across different datasets. This suggests that we can set a value within a proper range for these parameters, without the need to determine their values on different datasets. On the other hand, the most sensitive parameter is m , where a higher number of features contributes to better results, aligning with intuition and observations from existing literature. However, it is important to emphasize that more features are not always better. As demonstrated in Section 4.3, the FS methods consistently outperform AllFea in most classification and clustering tasks. Fewer features not only result in lower computational costs but also contribute to faster learning speeds. This suggests the need to adjust the value of m , for example, starting with a relatively small value and gradually increasing it until the model performance begins to decline.

10

4

10

3

10

2

10

1

10 0

10 1

learning rate

40

60

80

100

ACC (%)

1

2

3

4

5

k

40

60

80

100

ACC (%)

10

3

10

2

10

1

γ

40

60

80

100

ACC (%)

5

10

15

20

m

40

60

80

100

ACC (%)

(a) Madelon

10

4

10

3

10

2

10

1

10 0

10 1

learning rate

40

60

80

100

ACC (%)

1

2

3

4

5

k

40

60

80

100

ACC (%)

10

3

10

2

10

1

γ

40

60

80

100

ACC (%)

25

50

75

100

150

200

300

m

40

60

80

100

ACC (%)

(b) PCMAC

10

4

10

3

10

2

10

1

10 0

10 1

learning rate

40

60

80

100

ACC (%)

1

2

3

4

5

k

40

60

80

100

ACC (%)

10

3

10

2

10

1

γ

40

60

80

100

ACC (%)

25

50

75

100

150

200

300

m

40

60

80

100

ACC (%)

(c) Jaffe

10

4

10

3

10

2

10

1

10 0

10 1

learning rate

40

60

80

100

ACC (%)

1

2

3

4

5

k

40

60

80

100

ACC (%)

10

3

10

2

10

1

γ

40

60

80

100

ACC (%)

25

50

75

100

150

200

300

m

40

60

80

100

ACC (%)

(d) PROSTATE

Peer reviews:
comments: SUMMARY:
The authors present a method to jointly learn important features as well as a $k$-nearest neighbour graph for data. They extensively motivate and evaluate their method.

STRENGTHS:
- Great method, very novel and sensible.

- Excellent and thorough evaluation.

- Method is well-justified by theoretical explanations.

- Good ablation studies allow disentangling of the different components of the method.

- It is very admirable that the authors include the reconstruction metric in their main results, where their method does not perform as well. I agree with the authors that this is not a good metric for noisy data, but it is great that the authors do not try to hide their performance nonetheless.

WEAKNESSES:
- Very densely written, the paper should be more accessible than this. If the authors are able to improve the clarity of the writing, the impact of this work will greatly increase.

- To the point above, I would recommend a few sentences about the intuition behind the method.

**Overall**: Excellent work, some intuitive explanations about the method would greatly improve the reader's understanding of how the method works.

QUESTIONS:
- In equation 5, if we require $s_{i, i} = 0$, then why is $\mathbf{S} = I$ a solution?

LIMITATIONS:
The authors do a great job describing the limitations of their method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a deep FS method that simultaneously conducts feature selection and differentiable k-NN graph learning based on the Dirichlet Energy. The Dirichlet Energy identifies important features by measuring their smoothness on the graph structure, and facilitates the learning of a new graph that reflects the inherent structure in the new feature subspace during the training process using selected features. The authors employ the Gumbel Softmax technique and the Optimal Transport theory to address the non-differentiability issues of learning discrete FS results and learning k-NN graphs in neural networks, which theoretically makes our model applicable to other graph neural networks. Furthermore, the proposed framework is interpretable, since all modules are designed algorithmically.\n\nSTRENGTHS:\nThe originality of this paper is satisfying, which proposes a deep FS method that simultaneously conducts feature selection and differentiable k-NN graph learning based on the Dirichlet Energy and the significance is mainly built on this.\n\nThe quality and clarity of this paper is good based on the clear presentation of the proposed method shown in Figure 2.\n\nWEAKNESSES:\n1. The biggest problem of this paper is the limited novelty in formulation of the proposed method, which is mainly obtained by combination of the existing works, i.e., dirichlet energy or the differentiable learner. The authors should better highlight their novelty in Section 3 of this paper, which make it significantly different from the existing works.\n\n2. The compared methods in the experiment are not enough for comparison and the authors can add one or more methods for comparison to validate the effectiveness of the proposed method.\n\n3. The adopted datasets in the experiment are almost with small scales. The largest number of the adopted datasets is 2600, i.e., madelon. The authors can add one or more datasets with large scales for comparison in the experiment.\n\n4. The experimental improvements of the proposed method compared with the existing methods are not obvious, i.e, the proposed method on SRBCT is 0.98 while UDFS is 1.00.\n\n5. The authors can further explain the interpretability of the proposed method in the corresponding section , which is also an advantage of the given network.\n\nQUESTIONS:\n1. The biggest problem of this paper is the limited novelty in formulation of the proposed method, which is mainly obtained by combination of the existing works, i.e., dirichlet energy or the differentiable learner. The authors should better highlight their novelty in Section 3 of this paper, which make it significantly different from the existing works.\n\n2. The compared methods in the experiment are not enough for comparison and the authors can add one or more methods for comparison to validate the effectiveness of the proposed method.\n\n3. The adopted datasets in the experiment are almost with small scales. The largest number of the adopted datasets is 2600, i.e., madelon. The authors can add one or more datasets with large scales for comparison in the experiment.\n\n4. The experimental improvements of the proposed method compared with the existing methods are not obvious, i.e, the proposed method on SRBCT is 0.98 while UDFS is 1.00.\n\n5. The authors can further explain the interpretability of the proposed method in the corresponding section , which is also an advantage of the given network.\n\nLIMITATIONS:\n1. The biggest problem of this paper is the limited novelty in formulation of the proposed method, which is mainly obtained by combination of the existing works, i.e., dirichlet energy or the differentiable learner. The authors should better highlight their novelty in Section 3 of this paper, which make it significantly different from the existing works.\n\n2. The compared methods in the experiment are not enough for comparison and the authors can add one or more methods for comparison to validate the effectiveness of the proposed method.\n\n3. The adopted datasets in the experiment are almost with small scales. The largest number of the adopted datasets is 2600, i.e., madelon. The authors can add one or more datasets with large scales for comparison in the experiment.\n\n4. The experimental improvements of the proposed method compared with the existing methods are not obvious, i.e, the proposed method on SRBCT is 0.98 while UDFS is 1.00.\n\n5. The authors can further explain the interpretability of the proposed method in the corresponding section , which is also an advantage of the given network.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes an unsupervised feature selection method by minimizing the Dirichlet energy, and the energy function is on the other hand based on the k-NN graph computed from the selected features. In this sense, the features and the k-NN graph are jointly learned. To avoid discrete operations, the author(s) use Gumbel softmax to approximate the selection operation, and use OT-based sorting to construct k-NN graph. The proposed method is validated using a series of numerical experiments.\n\nSTRENGTHS:\nOverall the proposed method is an interesting joint feature selection and graph learning method, and the whole procedure is differentiable, thus useful for downstream analyses.\n\nWEAKNESSES:\n1. One major weakness of the proposed algorithm is its large computational cost, which the author(s) have mentioned in the article. The bottleneck seems to be in the sorting part. In fact, there are different variants of differentiable sorting, and it has been shown in some reports that OT-based sorting can be slow [1]. The author(s) may try other differentiable sorting algorithms such as [1], which has $O(n\\log n)$ forward complexity and $O(n)$ backward complexity.\n2. There exist several inconsistencies between the proposed method and the actual implementation, such as the symmetrized k-NN graph (page 2) and the actual computation of the $F$ matrix in page 5. Although the author(s) have explained their motivation, it downgrades the rigor of the method. Also Proposition 3.1 looks irrelevant if the actual $F$ is not computed in this way.\n3. Following point 2, I think a deeper reason for such inconsistencies is that the optimization problem (2) is difficult by nature, and Section 3.1.1 and Section 3.1.2 are tackling the constraints in (2) separately, not jointly. For example, Section 3.1.1 attempts to make $F$ approximately discrete, and Section 3.1.2 is concerned with $F^{T}F=I$. It is very likely that they are not met simultaneously using the current algorithm.\n\n[1] Blondel, M., Teboul, O., Berthet, Q., & Djolonga, J. (2020). Fast differentiable sorting and ranking. In International Conference on Machine Learning (pp. 950-959).\n\nQUESTIONS:\n1. I think both the title of Section 3.1.1 and the statement \"having selected *exact* original features\" in Section 3.1.2 need to be modified, as the Gumbel softmax technique is an approximation to the discrete optimization problem, not necessarily an exact one.\n2. The author(s) may consider a direct convex relaxation of the constraints in (2), which at least jointly handles the two different constraints.\n3. Some comments on the points in the weaknesses section would be helpful.\n\nLIMITATIONS:\nThe author(s) have remarked that the proposed method may be difficult to scale up.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an unsupervised feature selection method using Dirichlet energy. The proposed method learns the KNN graph and the feature selection jointly to reduce the influence on the feature selection quality by the noisy and irrelevant features. \n\nThe feature selection component minimizes the Dirichlet energy with Gumbel Softmax smoothed one-hot feature selection matrix. To ensure each resulting feature dimension is unique, the column orthogonality is enforced by linear decompositions.\n\nThe graph learning component also minimizes the Dirichlet energy. To avoid a trivial solution (graph with only self-connecting edges), Tikhonov regularization is applied to ensure the probabilistic behavior of similarity metrics and no self-loops.\n\nEmpirical results on both synthetic datasets and real-world applications show that the proposed method can effectively select useful features and has better performance than other baseline methods.\n\nSTRENGTHS:\n* The paper is clearly motivated and well-structured. The derivations are clear and in detail. Overall the paper presentation is good.\n\n* The proposed method is interesting and novel, and the effectiveness of the proposed method is supported by both synthetic and real-world datasets.\n\n* The authors also discussed the limitations of the method and lies out the future directions to improve.\n\nWEAKNESSES:\n* As the authors mentioned in the discussion section, the method does not scale to large-scale problems.\n\n* The method is under the assumption that the features are continuous and of low dimension, whereas many real-world problems may face noncontinuous (e.x. categorical) or high dimensional (e.x. sparse BOW) features.\n\nQUESTIONS:\n* How can the method be generalized to handle noncontinuous categorical features?\n\n* Is a constant feature a trivial minimizer for the Dirichlet energy? How does the proposed method handle this?\n\nLIMITATIONS:\nSee the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}