SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\mathbf {r}} \langle \boldsymbol {C}, \boldsymbol {\Gamma} \rangle , \quad \text {s . t .} \boldsymbol {\Gamma} \mathbf {1} _ {n _ {2}} = \boldsymbol {\mu}, \boldsymbol {\Gamma} ^ {\top} \mathbf {1} _ {n _ {1}} = \boldsymbol {\nu}, \Gamma_ {i, j} \geq 0, \tag {10}
$$
where  $C \in \mathcal{R}^{n_1 \times n_2}$  denotes the cost matrix with  $c_{i,j} = h(\mathfrak{a}_i - \mathfrak{b}_j) > 0$  being the transport cost from  $\mathfrak{a}_i$  to  $\mathfrak{b}_j$ . It is widely known that the solution of the OT problem between two discrete univariate measures boils down to the sorting permutation [23-25]. As stated in [25], if  $h$  is convex, the optimal assignment can be achieved by assigning the smallest element in  $\mathcal{A}$  to  $\mathfrak{b}_1$ , the second smallest to  $\mathfrak{b}_2$ , and so forth, which eventually yields the sorting permutation of  $\mathcal{A}$ .
Given a distance vector  $e$ , to learn selection vectors  $\delta^{(k)}$  and  $\xi^{(k+1)}$ , we set  $\mathcal{A} = e$ , and  $\mathcal{B} = [0, 1, \ldots, k+1]$ , and define  $\mu, \nu$ , and  $c_{ij}$  as
$$
\mu_ {i} = \frac {1}{n}, \quad \nu_ {j} = \left\{ \begin{array}{l l} 1 / n, & 1 \leq j \leq k + 1 \\ (n - k - 1) / n, & j = k + 2 \end{array} , \quad c _ {i j} = \left(\mathfrak {a} _ {i} - \mathfrak {b} _ {j}\right) ^ {2} = \left(e _ {i} - j + 1\right) ^ {2}. \right. \tag {11}
$$
The optimal transport plan of problem (10) assigns the  $i$ -th smallest value  $e_{\sigma_i}$  to  $\mathfrak{b}_i$  if  $1 \leq i \leq k + 1$ , and assigns the remaining  $n - k - 1$  values in  $\pmb{e}$  to  $\mathfrak{b}_{k + 2}$ . Namely,
$$
\Gamma_ {\sigma_ {i}, j} = \left\{ \begin{array}{l l} 1 / n, & i f (1 \leq i \leq k + 1 a n d j = i) o r (k + 1 <   i \leq n a n d j = k + 2) \\ 0, & i f (1 \leq i \leq k + 1 a n d j \neq i) o r (k + 1 <   i \leq n a n d j \neq k + 2) \end{array} . \right. \tag {12}
$$
Table 1: Details of real-world data.
Type	Dataset	#Samples	#Features	#Classes	Type	Dataset	#Samples	#Features	#Classes
Text	PCMAC [28]	1943	3289	2	Artificial	Madelon [29]	2600	500	2
Biological	GLIOMA [30]	50	4434	4	Image	COIL-20 [31]	1440	1024	20
LUNG [32]	203	3312	5	Yale [33]	165	1024	15
PROSTATE [34]	102	5966	2	Jaffe [35]	213	676	10
SRBCT [36]	83	2308	4	PIX10 [14]	100	10000	10
SMK [37]	187	19993	2	warpPIE10P [38]	210	2420	10
Given a sample  $p$ , once we obtain the optimal transport assignment  $\Gamma$  based on  $e^p$ , we calculate the variables  $\delta_p^{(k)}$  and  $\xi_p^{(k + 1)}$  as follows:
$$
\boldsymbol {\delta} _ {p} ^ {(k)} = n \sum_ {i = 1} ^ {k} \boldsymbol {\Gamma} _ {i}, \quad \boldsymbol {\xi} _ {p} ^ {(k + 1)} = n \boldsymbol {\Gamma} _ {k + 1}, \tag {13}
$$
where  $\Gamma_{i}$  and  $\Gamma_{k + 1}$  denote the  $i$ -th and the  $(k + 1)$ -th column of  $\Gamma$ , respectively. However, problem (10) is still non-differentiable. To address this issue, we consider the following entropy regularized OT problem:
$$
\min  _ {\boldsymbol {\Gamma}} \left\langle \boldsymbol {C}, \boldsymbol {\Gamma} \right\rangle + \gamma \sum_ {i, j} \Gamma_ {i, j} \log \Gamma_ {i, j} \quad \text {s . t .} \boldsymbol {\Gamma} \mathbf {1} _ {k + 2} = \boldsymbol {\mu}, \boldsymbol {\Gamma} ^ {\top} \mathbf {1} _ {n} = \boldsymbol {\nu}, \Gamma_ {i, j} \geq 0, \tag {14}
$$
where  $\gamma$  is a hyperparameter. The differentiability of problem (14) has been proven using the implicit function theorem (see [26, Theorem 1]). Note that a smaller  $\gamma$  yields a better approximation to the original solution in problem (10), but may compromise the differentiability of problem (14) [25]. Problem (14) can be solved efficiently using the iterative Bregman projections algorithm [27], the details of which are provided in Appendix S5.
Our experiments fall into three parts: (1) Toy Experiments: First, we verify the FS ability and the graph learning ability of the proposed method on synthetic datasets. (2) Quantitative Analysis: Next, we compare the performance of selected features in various downstream tasks on real-world datasets and compare our method with other unsupervised FS methods. (3) Ablation Study: Finally, we verify the effect of UFS and DGL by testing the performance of the corresponding ablated variants. We also provide the sensitivity analysis in Appendix S6.6. The implementation details of all experiments can be found in Appendix S6.1.
For the toy experiments, we generate three 20-dimensional datasets named Blobs, Moons, and Circles (see Appendix S6.1.1 for generation details). On top of that, we evaluate the proposed method on twelve real-world datasets that include text, biological, image, and artificial data. Table 1 exhibits the details of these datasets, which include many high-dimensional datasets to test the performance of our method. We standardize all features to zero means and normalize them with the standard deviation.
4.2 Toy Experiments
In this section, we consider three synthetic binary datasets with increasing difficulty in separating different classes. The first two dimensions in each dataset contain useful features that indicate the underlying structure, while the remaining 18 dimensions are random noise sampled from  $\mathcal{N}(0,1)$ . The presence of noise obscures the inherent structure of the data, which makes the graph learning process highly challenging. To see this, we generate 3-D plots of each dataset using the useful features and one noise feature, along with their 2-D projections on each plane, which are shown in Fig. 3(a). We can see that the noise blurs the boundary of different classes, especially in Moons and Circles. In addition, we used a heat kernel (abbreviated as Heat) with  $\sigma = 1$  to learn the 5-NN graph on 20-dimensional features, as shown in Fig. 3(b). We can see that the heavy noise obscures the underlying structure of data points, resulting in a chaotic graph outcome.
Figure 3: Toy results on synthetic datasets, where higher similarities are presented with thicker connections in  $k$ -NN graphs, and we only present the connections to 5-NN for each sample. Blue bars and orange bars represent the distribution of  $f_{1}$  and  $f_{2}$  in the FS matrix  $F$ , respectively.
Results. We test our method on toy datasets for selecting  $m = 2$  target features. The results are presented in Fig. 3(c) and Fig. 3(d), which demonstrate the success of our method in learning target features and intrinsic structures simultaneously. Moreover, it can be seen from Fig. 3(d) that the proposed network obtains the approximately discrete FS vectors.
Learning Without Unique Feature Selector. In addition, we conduct an ablation study by removing the UFS module from the network and updating  $F$  using Eq. (4) only. The results are shown in Fig. 3(e) and Fig. 3(f), where we can see that, without UFS, the ablated model repeatedly selects the same feature on all datasets. It is also noteworthy that the nodes in the graph are mostly connected either horizontally or vertically, indicating the effectiveness of DGL in learning the local structure solely based on the single selected feature.
Experimental Settings. In this section, we evaluate our method on real-world data. We partition each dataset into training data and testing data using an 8:2 ratio and identify useful features using training data. We then evaluate the performance of selected features on three downstream tasks: (1) Classification Accuracy: We train a random forest (RF) [39] classifier with 1000 trees using selected features and evaluate the prediction accuracy on the testing data. (2) Clustering Accuracy: We cluster the testing set with selected features using  $k$ -means [40], where the cluster number is set to #Classes. Then we align the results with true labels and calculate the accuracy. (3) Reconstruction RMSE: We build a 1-hidden-layer network with ReLU activation to reconstruct the original data using selected features. The hidden dimension is set to  $3m / 2$  except for AllFea, where the hidden size is set to  $d$ . The network is learned on the training set with selected features, and evaluated on the testing set using root mean square error (RMSE) normalized by  $d$ .
Competing methods. We compare our methods with four deep methods (CAE [8], DUFS [14], WAST [41], AEFS [9]) and three classical methods (LS [13], RSR [42], UDFS [43]). Besides, we use all features (AllFea) as the baseline. To evaluate the performance of each FS method on downstream tasks, we average the results over 10 random runs with the feature number  $m$  varied in  $\{25, 50, 75, 100, 150, 200, 300\}$  except for Madelon, where  $m$  is varied in  $\{5, 10, 15, 20\}$  since Madelon consists of only 20 useful features [29]. Appendix S6.1 provides details of the overall evaluation workflow, including the implementation and the parameter selection of each method.
Results. Similar to [14], we present the best result w.r.t.  $m$  in Table 2, the standard deviations is provided in Appendix S6.2. We also present some reconstruction results on PIN10 by our method in Appendix S6.3. From Table 2, we find that: (1) Our method generally achieves the best performance in all three tasks, indicating that our method selects more useful features. (2) In particular, we beat DUFS in all Classification and Clustering tasks, as well as most cases of the Reconstruction tasks. Recall that DUFS also selects features based on the Dirichlet Energy, this result shows that the model (5) in our method explores a superior graph structure compared to the traditional Heat method. (3)
Table 3: Results in ablation studies, where "w/o" is short for "without".
Task	Method	Madelon	PCMAC	Jaffe	PIX10	GLIOMA	PROSTATE
Effect of FS (Clu. with SC)	Heat	0.54±0.01	0.51±0.00	0.60±0.13	0.41±0.06	0.48±0.05	0.55±0.02
DGL only	0.50±0.00	0.50±0.00	0.57±0.05	0.76±0.09	0.55±0.04	0.56±0.07
Our	0.58±0.01	0.51±0.00	0.80±0.07	0.77±0.04	0.59±0.04	0.65±0.02
Effect of DGL (Cla. with RF)	w/o DGL	0.51±0.02	0.72±0.02	0.97±0.01	0.98±0.03	0.66±0.10	0.81±0.04
Our	0.90±0.01	0.79±0.01	1.00±0.01	1.00±0.00	0.81±0.09	0.89±0.05
In classification and clustering, the best performance is mostly achieved by FS methods with fewer features, which verifies the necessity of FS. (4) AllFea achieves five optimums in Reconstruction, which is not surprising since, theoretically, AllFea can be projected to original features with an identity matrix. However, in biological data, the best reconstruction results are achieved by FS methods, probably because the high-dimensional data leads to overfitting in networks. (5) It is noteworthy that the reconstruction of Madelon poses a significant challenge for FS methods, indicating the difficulty of reconstructing noise even using useful features. This observation supports our claim regarding the lack of reasonability in selecting features based on the reconstruction performance in Section 1.
In this experiment, we demonstrate the efficacy of the UFS and the DGL modules through ablation studies on six datasets: Madelon, PCMAC, Jaffe, FIX10, GLIOMA, and PROSTATE.
Effect of FS. Recall that we have demonstrated the efficacy of UFS in Fig. 3. To further verify the efficacy of FS in graph learning, we remove the entire FS module from the framework and learn the graph using all features based on DGL. We also compare the graph learning result using Heat. We
cluster the obtained graphs with the spectral clustering (SC) method to verify their qualities. We tune the parameter  $\sigma$  of Heat in  $\{1,2,\dots ,5\}$ , and fix  $k = 5$  for our method and the variant. The results are shown in Table 3, which shows that FS has a positive effect on graph learning compared with "DGL only". Besides, in Appendix S6.4, we visualize the learned graph on COIL-20 and Jaffe using t-SNE, which shows that using fewer features, we achieve separable graphs that contain fewer inter-class connections than other methods.
Effect of DGL. To verify the efficacy of DGL, we remove it from the model and learn the ablated variant with a fixed graph learned by Heat. Similar to Section 4.3, we first learn selected features using competing method, then evaluate the features in downstream tasks. We present the classification result in Table 3, and leave the other results in Appendix S6.5 due to limited space. We can see that our method significant outperforms the ablated variant, especially in Madelon. This is probably because the noise undermine the graph structure and disrupt the learning of informative features.
Conclusion. This paper proposes a deep unsupervised FS method that learns informative features and  $k$ -NN graph jointly using the Dirichlet Energy. The network is fully differentiable and all modules are developed algorithmically to present versatility and interpretability. We demonstrate the performance of our method with extensive experiments on both synthetic and real-world datasets.
Broader Impact. This paper presents not only an effective deep FS method, but also a differentiable  $k$ -NN graph learning strategy in the context of deep learning. This technique is particularly useful for end-to-end learning scenarios that require graph learning during the training process. And we do notice this practical need in existing literature, see [44] for example. We believe our study will inspire researchers who work on the dimensionality reduction and graph-related researches.
Limitations. The major limitation of the proposed method is the lack of scalability, for which we do not evaluate our method on large datasets. This is because problem (14) requires an iterative solution, requiring storage of all intermediate results for back-propagation. While literature [26] proposes a memory-saving approach by deriving the expression of the derivative of  $\Gamma$  mathematically (see [26, Section 3]), it still requires at least  $\mathcal{O}(nk)$  space to update all intermediate variables to learn  $k$  nearest neighbors for a single sample, which results in a  $\mathcal{O}(n^2 k)$  space complexity to learn for all  $n$  samples. This is a huge memory cost on large datasets. Although learning in batch seems to be the most straightforward solution, in our method, the neighbours of each sample are determined based on the global information of  $L_{S}$ , which has an  $n \times n$  size. This requires to load the entire batch's information during each iteration, for which we cannot employ subgraph sampling as other graph learning methods did to mitigate memory overhead. Another limitation of the proposed method is the low computational speed, as it is reported that the OT-based sorting can be slow [45].
The future developments of the proposed method are twofold. First, we will try more differentiable sorting algorithms to enhance computational speed. For example, reference [45] proposes to construct differentiable sorting operators as projections onto the permutahedron, which achieves a  $\mathcal{O}(n\log n)$  forward complexity and a  $\mathcal{O}(n)$  backward complexity. Second, due to the large cost of the global relationship in  $L_{S}$ , we are considering adopting a bipartite graph [16, 46] to make batch learning feasible. This graph introduces a small number of anchor points, which are representative of the entire feature space. By doing this, smoothness can be measured based on the distance between samples to anchors, for which sample-to-sample relationships are no longer needed and the batch learning is enabled. It is worth noting that this idea is still in its conceptual stage, and we will explore its feasibility in upcoming research.
This work was supported in part by the National Natural Science Foundation of China under Grant 62276212 and Grant 61872190, in part by the National Key Research and Development Program of China under Grant 2022YFB3303800, and in part by the Key Research and Development Program of Jiangsu Province under Grant BE2021093.
[41] G. Sokar, Z. Atashgahi, M. Pechenizkiy, and D. C. Mocanu, “Where to pay attention in sparse training for feature selection?” in Proc. NIPS, 2022.
[42] P. Zhu, W. Zuo, L. Zhang, Q. Hu, and S. C. Shiu, "Unsupervised feature selection by regularized self-representation," Pattern Recognit., vol. 48, no. 2, pp. 438-446, 2015.
[43] Y. Yang, H. T. Shen, Z. Ma, Z. Huang, and X. Zhou, “L2,1-norm regularized discriminative feature selection for unsupervised learning,” in Proc. IJCAI, 2011, pp. 1589–1594.
[44] S. Miao, Y. Luo, M. Liu, and P. Li, "Interpretable geometric deep learning via learnable randomness injection," in Proc. ICLR, 2023.
[45] M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga, "Fast differentiable sorting and ranking," in Proc. ICML, vol. 119, 2020, pp. 950-959.
[46] W. Liu, J. He, and S.-F. Chang, "Large graph construction for scalable semi-supervised learning," in Proc. ICML, 2010, pp. 679-686.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
[48] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Proc. NIPS, vol. 32, 2019.
S1 Proof of Proposition 3.1
Proposition 3.1. Given any real matrix  $\tilde{\pmb{F}}\in \mathcal{R}^{d\times m}$ , one can always generate a column-orthogonal matrix  $\pmb{F}$  through Algorithm 1.
Proof. We begin our proof by showing the feasibility of Algorithm 1 for any real matrix  $\hat{\pmb{F}}$ , as the eigendecomposition, Cholesky decomposition, and inverse mentioned in the algorithm are subject to specific conditions. For simplicity, we represent  $\pmb{A} = \hat{\pmb{F}}^{\top}\hat{\pmb{F}} + \epsilon \pmb{I}_m$ , with  $\epsilon > 0$ . Note that for any nonzero real column vector  $\pmb{z} \in \mathcal{R}^m$ , we have
$$
\boldsymbol {z} ^ {\top} \boldsymbol {A} \boldsymbol {z} = \boldsymbol {z} ^ {\top} \left(\hat {\boldsymbol {F}} ^ {\top} \hat {\boldsymbol {F}} + \epsilon \boldsymbol {I} _ {m}\right) \boldsymbol {z} = \left(\hat {\boldsymbol {F}} \boldsymbol {z}\right) ^ {\top} \left(\hat {\boldsymbol {F}} \boldsymbol {z}\right) + \epsilon \boldsymbol {z} ^ {\top} \boldsymbol {z} = \sum_ {i = 1} ^ {d} \left(\hat {\boldsymbol {f}} ^ {i} \boldsymbol {z}\right) ^ {2} + \epsilon \sum_ {i = 1} ^ {m} z _ {i} ^ {2} > 0. \tag {S1}
$$
Hence, the matrix  $\mathbf{A}$  is positive-definite and can be eigendecomposed as  $\mathbf{A} = \mathbf{P}\Lambda \mathbf{P}^{-1}$ , where  $\mathbf{P} \in \mathcal{R}^{m \times m}$  is the square matrix whose  $i$ -th column  $\mathbf{p}_i$  is the eigenvector of  $\mathbf{A}$  and  $\Lambda \in \mathcal{R}^{m \times m}$  is the diagonal matrix whose diagonal entries are the corresponding eigenvalues. Moreover, it is easy to show that  $\mathbf{A}$  is symmetric, for which we have  $\mathbf{P}^{\top} = \mathbf{P}^{-1}$ . Therefore, we prove that  $\mathbf{A}$  can be decomposed as  $\mathbf{A} = \mathbf{P}\Lambda \mathbf{P}^{\top}$  (line 2 in Algorithm 1).
Since  $\mathbf{A}$  is symmetric and positive-definite, we will be able to perform Cholesky decomposition on  $\mathbf{A}$  as  $\boldsymbol{L}\boldsymbol{L}^{\top} = \boldsymbol{A}$  (line 3 in Algorithm 1), which yields a lower triangular matrix  $\pmb{L} \in \mathcal{R}^{m \times m}$  whose diagonal entries are all real and positive. This means that the determinant of  $\pmb{L}$  is larger than zero and  $\pmb{L}$  is invertible, which provides the feasibility of  $L^{-1}$  (line 4 in Algorithm 1). Consequently, the feasibility of Algorithm 1 for any real matrix  $\hat{\pmb{F}}$  is proved.
Next, we show that  $\pmb{F}$  is a column-orthogonal matrix. We denote  $\pmb{Q} = \left[ \begin{array}{c}\Lambda^{1 / 2}\pmb{P}^{\top}\\ \hat{\mathbf{0}} \end{array} \right]$  and have:
$$
\boldsymbol {Q} ^ {\top} \boldsymbol {Q} = \left[ \boldsymbol {P} \boldsymbol {\Lambda} ^ {1 / 2}, \hat {\boldsymbol {0}} ^ {\top} \right] \left[ \begin{array}{c} \boldsymbol {\Lambda} ^ {1 / 2} \boldsymbol {P} ^ {\top} \\ \hat {\boldsymbol {0}} \end{array} \right] = \boldsymbol {P} \boldsymbol {\Lambda} \boldsymbol {P} ^ {\top} = \boldsymbol {A} = \boldsymbol {L L} ^ {\top}. \tag {S2}
$$
Then we prove the orthogonality of the matrix  $\pmb{F}$  as follows:
$$
\begin{array}{l} \boldsymbol {F} ^ {\top} \boldsymbol {F} = \boldsymbol {L} ^ {- 1} \boldsymbol {Q} ^ {\top} \boldsymbol {Q} (\boldsymbol {L} ^ {- 1}) ^ {\top} \\ = \boldsymbol {L} ^ {- 1} \boldsymbol {L} \boldsymbol {L} ^ {\top} (\boldsymbol {L} ^ {- 1}) ^ {\top} \\ = \boldsymbol {L} ^ {- 1} \boldsymbol {L} \boldsymbol {L} ^ {\top} (\boldsymbol {L} ^ {\top}) ^ {- 1} \\ = \boldsymbol {I} _ {m} \\ \end{array}
$$
S2 Discussion on Algorithm 1
Although Algorithm 1 theoretically guarantees the orthogonality of the selection matrix  $\mathbf{F}$ , utilizing this algorithm directly would bring us back to the problem of how to choose features and how to obtain discrete results. On one hand, the non-uniqueness of eigendecomposition in line 2 prevents us from ensuring the discrete properties of matrix  $\mathbf{F}$ . On the other hand, it is important to note that in line 4 of Algorithm 1, we aim to construct a column full-rank matrix  $Q = \begin{bmatrix} \Lambda^{1/2} P^{\top} \\ \hat{0} \end{bmatrix}$ , whereas the construction of  $Q$  is also not unique since we can insert  $d - m$  zero rows at any position within the original matrix  $\Lambda^{1/2} P^{\top}$  to achieve the column full-rankness. The placement of these zero rows directly affects the result of feature selection.
Guided by Algorithm 1, we devise a more empirical approach by calculating  $\pmb{F}$  with  $\pmb {F} = \hat{\pmb{F}} (\pmb{L}^{-1})^{\top}$ , which effectively tackles the above two concerns. By doing so, we avoid the non-uniqueness of eigendecomposition, thereby obtaining a solution that is as discrete as  $\hat{\pmb{F}}$ . Additionally, this approach ensures that the information of feature selection in  $\hat{\pmb{F}}$  is retained within the column full-rank matrix.
Actually,  $\pmb{F}$  is an  $\epsilon$ -approximation of column-orthogonal matrix, since we have:
$$
\begin{array}{l} \boldsymbol {F} ^ {\top} \boldsymbol {F} = \boldsymbol {L} ^ {- 1} \hat {\boldsymbol {F}} ^ {\top} \hat {\boldsymbol {F}} (\boldsymbol {L} ^ {- 1}) ^ {\top} \\ = \boldsymbol {L} ^ {- 1} (\boldsymbol {A} - \epsilon \boldsymbol {I} _ {m}) (\boldsymbol {L} ^ {- 1}) ^ {\top} \\ = \boldsymbol {L} ^ {- 1} \boldsymbol {A} \left(\boldsymbol {L} ^ {- 1}\right) ^ {\top} - \epsilon \boldsymbol {L} ^ {- 1} \left(\boldsymbol {L} ^ {- 1}\right) ^ {\top} \tag {S4} \\ = \boldsymbol {L} ^ {- 1} \boldsymbol {L} \boldsymbol {L} ^ {\top} (\boldsymbol {L} ^ {- 1}) ^ {\top} - \epsilon \boldsymbol {L} ^ {- 1} (\boldsymbol {L} ^ {- 1}) ^ {\top} \\ = \boldsymbol {I} _ {m} - \epsilon \boldsymbol {L} ^ {- 1} (\boldsymbol {L} ^ {- 1}) ^ {\top}. \\ \end{array}
$$
The experimental results in Section 4.2 verify the effectiveness of this approach in successfully avoiding duplicate feature selection.
S3 Derivation of the Solution to Problem 5
Recall that we aim to solve the following problem to learn an adaptive  $k$ -NN graph:
$$
\begin{array}{l} \min _ {\boldsymbol {S}} \operatorname {t r} (\hat {\boldsymbol {X}} ^ {\top} \boldsymbol {L} _ {S} \hat {\boldsymbol {X}}) + \frac {\alpha}{2} \| \boldsymbol {S} \| _ {F} ^ {2}, \quad \mathrm {s . t .} \boldsymbol {S} \mathbf {1} _ {n} = \mathbf {1} _ {n}, s _ {i, j} \geq 0, s _ {i, i} = 0, \\ = \min  _ {s _ {i, j}} \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \| \left(\hat {\boldsymbol {x}} ^ {i} - \hat {\boldsymbol {x}} ^ {j}\right) \| _ {2} ^ {2} s _ {i, j} + \alpha_ {i} s _ {i, j} ^ {2}, \quad \text {s . t .} \sum_ {j = 1} ^ {n} s _ {i, j} = 1, s _ {i, j} \geq 0, s _ {i, i} = 0. \tag {S5} \\ \end{array}
$$
Based on  $\hat{\pmb{X}}$ , we define the quantity  $e_{i,j} = \| (\hat{\pmb{x}}^i - \hat{\pmb{x}}^j) \|_2^2$ , then we solve each row in problem (S5) separately as: