SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \min  _ {s _ {i, j}} \frac {1}{2} \sum_ {j = 1} ^ {n} e _ {i, j} s _ {i, j} + \alpha_ {i} s _ {i, j} ^ {2} \quad \text {s . t .} \sum_ {j = 1} ^ {n} s _ {i, j} = 1, s _ {i, j} \geq 0, s _ {i, i} = 0, \\ = \min  _ {s _ {i, j}} \frac {1}{2} \sum_ {j = 1} ^ {n} \left(s _ {i, j} + \frac {e _ {i , j}}{2 \alpha_ {i}}\right) ^ {2} \quad \text {s . t .} \sum_ {j = 1} ^ {n} s _ {i, j} = 1, s _ {i, j} \geq 0, s _ {i, i} = 0, \tag {S6} \\ = \min _ {\pmb {s} ^ {i}} \frac {1}{2} \| \pmb {s} ^ {i} + \frac {\pmb {e} ^ {i}}{2 \alpha_ {i}} \| _ {2} ^ {2} \quad \mathrm {s . t .} \pmb {s} ^ {i} \mathbf {1} _ {n} = 1, s _ {i, j} \geq 0, s _ {i, i} = 0. \\ \end{array}
$$
We first omit the constraint  $s_{i,i} = 0$  and consider it later, and solve problem (S6) with the first two constraints, the Lagrangian function of which is as follows:
$$
\mathcal {L} \left(\boldsymbol {s} ^ {i}, \lambda_ {i}, \boldsymbol {\beta} ^ {i}\right) = \frac {1}{2} \left\| \boldsymbol {s} ^ {i} + \frac {\boldsymbol {e} ^ {i}}{2 \alpha_ {i}} \right\| ^ {2} - \lambda_ {i} \left(\boldsymbol {s} ^ {i} \mathbf {1} _ {n} - 1\right) - \sum_ {j = 1} ^ {n} s _ {i, j} \beta_ {i, j}, \tag {S7}
$$
where  $\lambda_{i}$  and  $\beta_{i,j}$  are Lagrange multipliers. The derivative of  $\mathcal{L}(s^i,\lambda_i,\beta^i)$  w.r.t.  $s_{i,j}$  is:
$$
\frac {\partial \mathcal {L}}{\partial s _ {i , j}} = s _ {i, j} + \frac {e _ {i , j}}{2 \alpha_ {i}} - \lambda_ {i} - \beta_ {i, j} \tag {S8}
$$
Then we have the Karush-Kuhn-Tucker(KKT) conditions [20] of problem (S7) as follows:
$$
\left\{ \begin{array}{l} s _ {i, j} + \frac {e _ {i , j}}{2 \alpha_ {i}} - \lambda_ {i} - \beta_ {i, j} = 0 \\ \sum_ {j = 1} ^ {n} s _ {i, j} = 1 \\ s _ {i, j} \geq 0 \\ \beta_ {i, j} \geq 0 \\ \beta_ {i, j} s _ {i, j} = 0. \end{array} \right. \tag {S9}
$$
$$
s _ {i, j} = \left(\lambda_ {i} - \frac {e _ {i , j}}{2 \alpha_ {i}}\right) + \tag {S10}
$$
Recall that there are only  $k$  nonzero elements in  $s^i$  corresponding to the nearest neighbors of sample  $i$ , according to the constraint  $\sum_{j=1}^{n} s_{i,j} = 1$  on  $k$  nonzero entries in  $s^i$ , we have:
$$
\sum_ {j = 1} ^ {k} \left(\lambda_ {i} - \frac {e _ {i , \sigma_ {j}}}{2 \alpha_ {i}}\right) = 1 \Rightarrow \lambda_ {i} = \frac {1}{k} + \frac {1}{k} \frac {\mathbf {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}{2 \alpha_ {i}} \quad \text {w i t h} \quad \delta_ {i, j} ^ {(k)} = \operatorname {B o o l} \left(e _ {i, j} \leq e _ {i, \sigma_ {k}}\right), \tag {S11}
$$
where  $\delta_{i}^{(k)}$  denotes the selection vector identifying the  $k$  minimal values in  $\boldsymbol{e}^i$ , and  $\sigma = [\sigma_1, \dots, \sigma_n]$  denotes the sorting permutation over  $\boldsymbol{e}^i$ , i.e.  $e_{i,\sigma_1} \leq \dots \leq e_{i,\sigma_n}$ . Without loss of generality, we assume  $\boldsymbol{e}^i$  has no duplicates, namely  $e_{i,\sigma_1} < \dots < e_{i,\sigma_n}$ . Considering the constraint  $s_{i,i} = 0$ , since  $e_{i,i} = 0$  being the minimal value in  $\boldsymbol{e}^i$  holds for all samples, we replace  $e_{i,i}$  with a sufficiently large value to skip over this trivial solution.
Substituting (S11) into (S10), we have:
$$
s _ {i, j} = \left(\frac {1}{k} + \frac {1}{k} \frac {\mathbf {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}{2 \alpha_ {i}} - \frac {e _ {i , j}}{2 \alpha_ {i}}\right) + \tag {S12}
$$
Recall that there are only  $k$  nonzero entries in  $\boldsymbol{s}^i$ , we have
$$
\frac {1}{k} + \frac {1}{k} \frac {\mathbf {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}{2 \alpha_ {i}} - \frac {e _ {i , \sigma_ {k}}}{2 \alpha_ {i}} > 0, \quad \frac {1}{k} + \frac {1}{k} \frac {\mathbf {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}{2 \alpha_ {i}} - \frac {e _ {i , \sigma_ {k + 1}}}{2 \alpha_ {i}} \leq 0. \tag {S13}
$$
Note that we assume  $\alpha_{i} > 0$ , then we have
$$
\frac {1}{2} \left(k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k)} - e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}\right) <   \alpha_ {i} \leq \frac {1}{2} \left(k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}\right) \quad \text {w i t h} \quad \xi_ {i, j} ^ {(k)} = \operatorname {B o o l} \left(e _ {i, j} = e _ {i, \sigma_ {k}}\right), \tag {S14}
$$
where  $\pmb{\xi}_i^{(k)}$  is an indicator vector identifying the  $k$ -th minimal value in  $\pmb{e}^i$ . According to (S14), we set  $\alpha_{i}$  as its maximal value as follows:
$$
\alpha_ {i} = \frac {1}{2} \left(k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}\right). \tag {S15}
$$
Substituting S15 into Eq. S12, we have:
$$
\begin{array}{l} s _ {i, j} = (\frac {1}{k} + \frac {1}{k} \frac {\pmb {e} ^ {i} \pmb {\delta} _ {i} ^ {(k)}}{2 \alpha_ {i}} - \frac {e _ {i , j}}{2 \alpha_ {i}}) _ {+} \\ = (\frac {2 \alpha_ {i} + e ^ {i} \delta_ {i} ^ {(k)} - k e _ {i , j}}{2 k \alpha_ {i}}) + \\ = \left(\frac {k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)} + e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)} - k e _ {i , j}}{k \left(k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}\right)}\right) + \tag {S16} \\ = \left(\frac {\boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e _ {i , j}}{k \boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - \boldsymbol {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}\right) _ {+} \\ \end{array}
$$
Eq. (S16) is used for implementation in our code. Note that since  $e_{i,\sigma_1} < \dots < e_{i,\sigma_k} < e_{i,\sigma_{k+1}} < \dots < e_{i,\sigma_n}$ , we have
$$
k e ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - \boldsymbol {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)} = k e _ {i, \sigma_ {k + 1}} - \sum_ {p = 1} ^ {k} e _ {i, \sigma_ {p}} = \sum_ {p = 1} ^ {k} \left(e _ {i, \sigma_ {k + 1}} - e _ {i, \sigma_ {p}}\right) > 0. \tag {S17}
$$
Then we obtain the solution of  $s_{i,j}$  as
$$
s _ {i, \sigma_ {j}} = \left\{ \begin{array}{l l} \frac {\boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e _ {i , \sigma_ {j}}}{k \boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - \boldsymbol {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}}, & 1 \leq j \leq k \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {S18}
$$
which is exactly the solution of Eq. (9) in our main paper:
$$
s _ {i, \sigma_ {j}} = \frac {\boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - e _ {i , \sigma_ {j}}}{k \boldsymbol {e} ^ {i} \boldsymbol {\xi} _ {i} ^ {(k + 1)} - \boldsymbol {e} ^ {i} \boldsymbol {\delta} _ {i} ^ {(k)}} \cdot \operatorname {B o o l} (1 \leq j \leq k). \tag {S19}
$$
S4 Connection to CLR
We note that a similar formulation to problem 6 has been proposed in [21] (coined CLR), which expects closer samples to have higher similarity. It aligns with the notion of "smoothness" as we mentioned in Section 2. However, our method differs from CLR in at least two crucial aspects: Firstly, CLR measures the distance quantity  $e_{i,j}$  across all original features, making it more sensitive to the noise and irrelevant features in the original data. In contrast, our approach learns the graph structure using only informative features, resulting in enhanced robustness against noisy features. Secondly, it is important to note that CLR is proposed in the context of traditional machine learning, where optimization is straightforward, as  $\delta_i^{(k)}$  and  $\xi_i^{(k + 1)}$  can be updated using off-the-shelf sorting algorithms. Different from CLR, problem 6 is introduced in the realm of deep learning, where conventional sorting algorithms are non-differentiable and not applicable. This poses a huge challenge in learning an adaptive  $k$ -NN graph in neural networks. To overcome this challenge, we proposed to transform the top-  $k$  selection into a differentiable operator using the Optimal Transport technique.
S5 Iterative Bregman Projections
In this paper, we employ the iterative Bregman projections [27] algorithm to solve the following problem:
$$
\min  _ {\mathbf {r}} \left\langle \boldsymbol {C}, \boldsymbol {\Gamma} \right\rangle + \gamma \sum_ {i, j} \Gamma_ {i, j} \log \Gamma_ {i, j}, \quad \text {s . t .} \boldsymbol {\Gamma} \mathbf {1} _ {k + 2} = \boldsymbol {\mu}, \boldsymbol {\Gamma} ^ {\top} \mathbf {1} _ {n} = \boldsymbol {\nu}, \Gamma_ {i, j} \geq 0. \tag {S20}
$$
We first initialize two variables  $\pmb{u} \in \mathcal{R}^{k+2}$  and  $\pmb{K} \in \mathcal{R}^{n \times (k+2)}$  as  $u_i = 1/(k+2)$  and  $k_{i,j} = e^{-c_{i,j}/\gamma}$ , respectively. Then based on the following formulations, we repeatedly updating  $\pmb{u}$  and  $\pmb{v}$  for  $\zeta$  iterations:
$$
\boldsymbol {v} = \frac {\mu}{K u}, \quad \boldsymbol {u} = \frac {\nu}{K ^ {\top} v}, \tag {S21}
$$
where the division in Eq. (S21) is element-wise. In this paper, we set  $\zeta$  as 200. After updating  $\zeta$  iteration, we obtain the optimal transport plan  $\Gamma$  as
$$
\boldsymbol {\Gamma} = \operatorname {d i a g} (\boldsymbol {v}) \boldsymbol {K} \operatorname {d i a g} (\boldsymbol {u}). \tag {S22}
$$
S6 Supplementary Experimental Details
S6.1 Implementation Details
All experiments are conducted on a server equipped with an RTX 3090 GPU and an Intel Xeon Gold 6240 (18C36T) @ 2.6GHz x 2 (36 cores in total) CPU.
S6.1.1 Synthetic Datasets
We generate three datasets for toy experiments: (1) Blobs, (2) Moons, and (3) Circles. For each dataset, we generate the first two features using the scikit-learn library [47] by adding noise sampled from  $\mathcal{N}(0, 0.1)$ . Additionally, we generate 18-dimensional noise features sampled from  $\mathcal{N}(0, 1)$ .
S6.1.2 Competing Methods
The implementation details of different methods, as well as their corresponding parameter selections are provided below:
- Our Method: Our method is implemented using the PyTorch framework [48]. We train our method using the Adam optimizer for 1000 epochs on all datasets, with the learning rate searched from  $\{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}\}$ . We search the parameter  $\gamma$  in  $\{10^{-3}, 10^{-2}, 10^{-1}\}$  and the parameter  $k$  in  $\{1, 2, 3, 4, 5\}$ . Note that the implementation of differentiable top- $k$  selector is based on the code provided by [26] in https://papers.nips.cc/paper_files/paper/2020/bitstream/ec24a54d62ce57ba93a531b460fa8d18-AbsAbstract.html, which provides a more memory-saving backward implementation compared to directly using the autograd method in PyTorch.
- Where to Pay Attention in Sparse Training (WAST) [41]: We use the official code released in https://github.com/GhadaSokar/WAST. The parameter settings were adopted in accordance with Appendix A.1 of the original paper. Specifically, we train each dataset for 10 epochs using stochastic gradient descent with a learning rate of 0.1 for all datasets except for SMK, where the learning rate is set to 0.01. For the parameter  $\lambda$ , we set  $\lambda = 0.9$  on Madelon and PCMAC,  $\lambda = 0.4$  on all image datasets,  $\lambda = 0.1$  on all biological datasets except SMK, and  $\lambda = 0.01$  on SMK. The remaining parameters are kept as they were in the original paper.
- Differentiable Unsupervised Feature Selection (DUFS) [14]: We use the official code released in https://github.com/0firlin/DUFS and use the parameter-free loss version of DUFS. For all datasets, we set  $k = 2$ , and train the method with SGD with a learning rate of 1 for 10000 epochs according to Appendix S7 in the original paper. We set the parameter  $C = 5$  on all datasets except for SRBCT, COIL, and FIX10, where  $C$  is set to 2.
- Concrete AutoEncoder (CAE) [8]: We use the official code released in https://github.com/mfbalin/Concrete-Autoencoders. Since we could not find too much description about the parameter settings on different datasets in the original paper, we run CAE with default settings in the code.
- AutoEncoder Feature Selector (AEFS) [9]: The original code provided by the authors is implemented in MATLAB, and it requires a prohibitively long time to run this method on MATLAB. Therefore, following the treatment in [41], we use the code provided by the authors of CAE in https://github.com/Ofirlin/DUFS (see experiments/generate comparision_figures.py in their repository). We search the parameter  $\alpha$  in  $\{10^{-9}, 10^{-6}, 10^{-3}, 10^{0}, 10^{3}, 10^{6}, 10^{9}\}$ , and the size of the hidden layer in  $\{128, 256, 512, 1024\}$ .
- Laplacian Score (LS) [13]: We use the official code released in http://www.cad.zju.edu.cn/home/dengcai/Data/ReproduceExp.html#LaplacianScore. We use the heat kernel for graph construction, and fix the size of neighbors  $k$  as 5 for all datasets.
- Regularized Self-Representation (RSR) [42]: We use the official code released in https://github.com/AISKYEYE-TJU/RSR-PR2015. We search the parameter  $\lambda$  in  $\{10^{-9}, 10^{-6}, 10^{-3}, 10^{0}, 10^{3}, 10^{6}, 10^{9}\}$ .
- Unsupervised Discriminative Feature Selection (UDFS) [43]: We use the code provided in https://guijiejie.github.io/code.htm1. We use the heat kernel for graph construction, and fix the size of neighbors  $k$  as 5 for all datasets. We search the parameter  $\gamma$  in  $\{10^{-9}, 10^{-6}, 10^{-3}, 10^{0}, 10^{3}, 10^{6}, 10^{9}\}$ .
S6.1.3 Evaluation Workflow
The overall evaluation workflow in Section 4.3 is shown in Algorithm S2, which includes two steps:
1. Given the dataset  $X$  and a prespecified feature number  $m$ , we first randomly split the dataset using an  $8:2$  ratio and select features based on the training set  $X_{tr}$  by FS method  $\mathfrak{F}$  using different parameters  $\Theta$ , as shown in Algorithm S1. This allows us to obtain FS results under different parameter candidates  $\theta_i$ , along with the corresponding reduced training data  $X_{tr}'$  and testing data  $X_{te}'$ . Based on the reduced data, we perform classification tasks on these datasets with the random forest, thereby obtaining the classification performance for each parameter combination. We select the parameter combination with the best classification performance as the optimal parameter  $\theta^*$  for  $\mathfrak{F}$ .
2. Based on the optimal parameter  $\theta^{*}$ , we construct the FS model  $\mathfrak{F}_{\theta^{*}}$  and evaluate its performance in different downstream tasks. To avoid randomness, we randomly split the dataset 10 times. With each random split, we use the training set  $X_{tr}$  to select features, and obtain reduced training set  $X_{tr}^{\prime}$  and testing set  $X_{te}^{\prime}$ . We use these sets for downstream tasks including classification, clustering, and reconstruction, and obtain corresponding performance metrics. For each downstream task, we calculate the average metric over 10 runs as the performance of  $\mathfrak{F}$  for the given number  $m$ .
For each dataset, we vary the value of  $m$  and follow the aforementioned procedure to obtain the corresponding performance. For each downstream task, we report the best metric and the corresponding feature number  $m$  as the performance of the FS method in this downstream task.
Algorithm S1 Param_tuning
Input: Training data  $(X_{tr},y_{tr})$ , testing data  $(X_{te},y_{te})$ , selected number  $m$ , FS method  $\mathfrak{F}$ , and parameter set  $\Theta = \{\theta_i\}$ .
Output: Optimal parameter  $\theta^{*}$
1: for  $\theta_{i}$  in  $\Theta$  do
2:  $\pmb {\xi} = \mathfrak{F}_{\theta_i}(\pmb {X}_{tr},m);$ $\triangleright$  Determining selected features  $\pmb{\xi}$  by  $\mathfrak{F}$  under the parameter  $\theta_{i}$
3:  $X_{tr}^{\prime} = X_{tr}(:,\pmb {\xi}),X_{te}^{\prime} = X_{te}(:,\pmb {\xi});$ $\triangleright$  Generating reduced datasets using selected features.
4: ACC = RF(\(X_{tr}', \)y_{tr}', \(X_{te}', \)y_{te})\); Evaluating selected features with random forest classifier.
5: if ACC > ACC* then
6:  $\mathrm{ACC}^{*} = \mathrm{ACC}$
7:  $\theta^{*} = \theta_{i}$
8: end if
9: end for
Algorithm S2 Overall evaluation workflow
Input: Original dataset  $(X, y)$ , selected feature number  $m$ , FS method  $\mathfrak{F}$ , parameter set  $\Theta = \{\theta_i\}$ , and downstream tasks  $\mathfrak{T} = \{\mathcal{T}_i\}$ .
Output: Performance  $M = \{M_{i}\}$  in downstream tasks.
1: Partitioning the dataset into training data  $(\pmb{X}_{tr},\pmb{y}_{tr})$  and testing data  $(\pmb{X}_{te},\pmb{y}_{te})$ .
2: Determining  $\theta^{*}$  by Param_tuning(( $\mathbf{X}_{tr}, \mathbf{y}_{tr}$ ,  $(\mathbf{X}_{te}, \mathbf{y}_{te})$ ,  $m, \Im, \Theta$ ) in Algorithm S1;
3: for  $j = 1:10$  do
4: Partitioning the dataset into training data  $(X_{tr}, y_{tr})$  and testing data  $(X_{te}, y_{te})$ .
5:  $\pmb{\xi}^{*} = \mathfrak{F}_{\theta^{*}}(\pmb{X}_{tr},m)$
6:  $\bar{\pmb{X}}_{tr}^{\prime} = \bar{\pmb{X}}_{tr}(:,\pmb{\xi}^{*}),\bar{\pmb{X}}_{te}^{\prime} = \bar{\pmb{X}}_{te}(:,\pmb{\xi}^{*})$
7: for  $\mathcal{T}_i$  in  $\mathfrak{T}$  do
8:  $m\{i,j\} = \mathcal{T}_i(\pmb{X}_{tr}',\pmb{y}_{tr},\pmb{X}_{te}',\pmb{y}_{te})$ ;  $\triangleright$  Evaluating the performance in downstream tasks.
9: end for
0: end for
1: for  $\mathcal{T}_i$  in  $\mathfrak{T}$  do
2:  $M_{i} = \operatorname{Average}(m\{i, : \})$ ;
3: end for
S6.1.4 Evaluation Metrics
We employ two metrics in our experiments: the accuracy (ACC) and the root mean square error (RMSE) normalized by  $d$ .
The formulation of ACC is
$$
\operatorname {A C C} (\boldsymbol {y}, \hat {\boldsymbol {y}}) = \frac {\sum_ {i = 1} ^ {n} \operatorname {B o o l} \left(y _ {i} = \hat {y} _ {i}\right)}{n}, \tag {S23}
$$
where  $\pmb{y} \in \mathcal{R}^n$  denotes the groundtruth labels and  $\hat{\pmb{y}} \in \mathcal{R}^n$  denotes the prediction label.
The formulation of RMSE normalized by  $d$  is
$$
\operatorname {R M S E} (\boldsymbol {X}, \hat {\boldsymbol {X}}) = \sqrt {\frac {\sum_ {i = 1} ^ {n} \| \boldsymbol {x} ^ {i} - \hat {\boldsymbol {x}} ^ {i} \| _ {2} ^ {2}}{n \times d}}, \tag {S24}
$$
where  $\mathbf{X} = [x_1; x_2; \ldots; x_n]$  and  $\hat{\mathbf{X}} = [\hat{x}_1; \hat{x}_2; \ldots; \hat{x}_n]$  denote the original feature matrix and the reconstructed feature matrix, respectively.
S6.1.5 Formulation of Heat Kernel Method
Here we describe the heat kernel (Heat) method compared in this paper. To implement Heat, we first compute the similarity matrix  $\hat{S}$  as follows:
$$
\hat {s} _ {i, j} = \exp \left(- \frac {\| \boldsymbol {x} ^ {i} - \boldsymbol {x} ^ {j} \| _ {2} ^ {2}}{2 \sigma^ {2}}\right), \tag {S25}
$$
Based on  $\hat{S}$ , we keep the  $k$ -nearest neighbors for each sample. Namely, for each sample  $i$ , we obtain its similarity vector  $s^i$  as
$$
s _ {i, j} = \left\{ \begin{array}{l l} \hat {s} _ {i, j}, & \boldsymbol {x} ^ {j} \in \Re (\boldsymbol {x} ^ {i}) \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {S26}
$$
where  $\Re (\pmb{x}^i)$  denotes the  $k$  nearest neighbors of  $\pmb{x}^i$ . When we need to calculate the Laplacian matrix using  $\pmb{S}$  (for example, when we analyze the effect of DGL in Section 4.4), we use the symmetrized version of  $\pmb{S}$ :
$$
\tilde {\boldsymbol {S}} = \frac {\boldsymbol {S} ^ {\top} + \boldsymbol {S}}{2} \tag {S27}
$$
S6.2 Standard Deviations of Quantitative Analysis
Table S1, Table S2, and Table S3 exhibit the mean and the standard deviations of the results in Table 2 in Section 4.3.
Table S1: Classification results with standard deviations.
Dataset	CAE	DUFS	WAST	AEFS	LS	RSR	UDFS	Our	AllFea
Madelon	0.65±0.04	0.52±0.04	0.87±0.02	0.87±0.01	0.51±0.01	0.83±0.02	0.74±0.06	0.90±0.01	0.73±0.02
PCMAC	0.79±0.06	0.77±0.02	0.83±0.01	0.71±0.03	0.68±0.03	0.91±0.02	0.87±0.02	0.79±0.01	0.93±0.01
COIL-20	1.00±0.01	1.00±0.00	1.00±0.00	1.00±0.00	0.98±0.01	1.00±0.00	1.00±0.00	1.00±0.00	1.00±0.00
Yale	0.72±0.05	0.72±0.05	0.70±0.05	0.70±0.06	0.67±0.07	0.72±0.08	0.71±0.06	0.81±0.05	0.75±0.06
Jaffe	0.97±0.02	0.97±0.01	0.97±0.01	0.97±0.01	0.97±0.03	0.98±0.02	0.98±0.02	1.00±0.01	0.98±0.02
PIX10	0.99±0.02	0.98±0.03	0.99±0.02	0.98±0.03	0.97±0.06	0.97±0.04	0.97±0.04	1.00±0.00	0.97±0.03
warpPIE10P	0.96±0.04	0.96±0.03	0.95±0.03	0.98±0.04	0.96±0.05	0.98±0.03	0.97±0.05	0.99±0.02	0.98±0.04
GLIOMA	0.68±0.10	0.66±0.12	0.72±0.09	0.63±0.12	0.67±0.13	0.67±0.11	0.68±0.10	0.81±0.09	0.72±0.14
LUNG	0.86±0.05	0.87±0.05	0.87±0.04	0.87±0.05	0.92±0.04	0.93±0.05	0.91±0.06	0.94±0.03	0.90±0.05
PROSTATE	0.88±0.05	0.81±0.07	0.81±0.06	0.81±0.07	0.88±0.07	0.90±0.07	0.90±0.05	0.89±0.05	0.89±0.06
SRBCT	0.99±0.02	0.94±0.05	0.98±0.03	0.95±0.05	0.98±0.04	1.00±0.00	1.00±0.00	0.98±0.03	0.99±0.02
SMK	0.67±0.06	0.66±0.07	0.65±0.07	0.66±0.08	0.72±0.10	0.72±0.11	0.73±0.09	0.75±0.06	0.68±0.10
Table S2: Clustering results with standard deviations.
Dataset	CAE	DUFS	WAST	AEFS	LS	RSR	UDFS	Our	AllFea
Madelon	0.60±0.01	0.52±0.01	0.52±0.01	0.55±0.04	0.52±0.01	0.61±0.02	0.57±0.05	0.60±0.01	0.58±0.04
PCMAC	0.52±0.01	0.51±0.01	0.51±0.01	0.51±0.01	0.52±0.01	0.51±0.01	0.51±0.01	0.53±0.02	0.51±0.01
COIL-20	0.69±0.03	0.59±0.04	0.65±0.03	0.60±0.04	0.53±0.04	0.59±0.03	0.60±0.04	0.66±0.03	0.63±0.05
Yale	0.55±0.06	0.55±0.07	0.56±0.06	0.54±0.06	0.58±0.06	0.60±0.08	0.58±0.05	0.62±0.04	0.61±0.08
Jaffe	0.85±0.07	0.80±0.06	0.83±0.05	0.80±0.07	0.76±0.09	0.83±0.08	0.83±0.09	0.87±0.06	0.82±0.06
PIX10	0.86±0.05	0.79±0.04	0.85±0.06	0.79±0.04	0.85±0.10	0.72±0.08	0.81±0.09	0.87±0.07	0.78±0.05
warpPIE10P	0.55±0.05	0.42±0.03	0.44±0.04	0.53±0.06	0.55±0.05	0.57±0.06	0.49±0.08	0.51±0.05	0.45±0.05
GLIOMA	0.69±0.07	0.65±0.11	0.65±0.10	0.62±0.10	0.63±0.13	0.65±0.16	0.68±0.11	0.75±0.08	0.62±0.08
LUNG	0.64±0.07	0.64±0.11	0.62±0.10	0.66±0.06	0.65±0.11	0.69±0.10	0.61±0.05	0.72±0.12	0.69±0.11
PROSTATE	0.64±0.11	0.59±0.05	0.58±0.05	0.59±0.05	0.71±0.13	0.63±0.10	0.69±0.11	0.68±0.09	0.64±0.10
SRBCT	0.76±0.09	0.54±0.05	0.56±0.08	0.57±0.08	0.56±0.07	0.60±0.11	0.59±0.11	0.63±0.10	0.52±0.06
SMK	0.60±0.06	0.58±0.05	0.58±0.06	0.58±0.05	0.59±0.05	0.59±0.05	0.61±0.07	0.64±0.05	0.60±0.07
Table S3: Reconstruction results with standard deviations.
Dataset	CAE	DUFS	WAST	AEFS	LS	RSR	UDFS	Our	AllFea
Madelon	0.99±0.01	0.99±0.00	0.98±0.00	0.98±0.00	0.99±0.00	0.98±0.00	0.98±0.00	0.98±0.00	0.28±0.00
PCMAC	1.14±0.17	1.03±0.06	1.05±0.06	1.05±0.07	1.04±0.06	1.17±0.19	1.07±0.12	1.30±0.22	0.78±0.03
COIL-20	0.48±0.03	0.41±0.02	0.43±0.01	0.41±0.01	0.48±0.02	0.45±0.02	0.41±0.02	0.38±0.02	0.27±0.01
Yale	0.63±0.03	0.56±0.02	0.56±0.02	0.56±0.02	0.78±0.04	0.57±0.03	0.60±0.03	0.54±0.02	0.50±0.01
Jaffe	0.31±0.05	0.25±0.05	0.26±0.05	0.25±0.05	0.33±0.08	0.26±0.05	0.25±0.05	0.22±0.01	0.23±0.04
PIX10	0.49±0.03	0.43±0.04	0.46±0.05	0.43±0.02	0.63±0.05	0.46±0.04	0.47±0.05	0.39±0.03	1.20±0.23
warpPIE10P	0.30±0.03	0.26±0.02	0.27±0.02	0.26±0.02	0.45±0.06	0.27±0.03	0.26±0.02	0.25±0.01	0.25±0.04
GLIOMA	0.74±0.05	0.71±0.04	0.71±0.04	0.72±0.05	0.71±0.04	0.72±0.04	0.72±0.04	0.69±0.04	1.86±0.59
LUNG	0.94±0.06	0.77±0.02	0.77±0.02	0.78±0.02	0.82±0.04	0.81±0.02	0.79±0.02	0.80±0.03	0.82±0.28
PROSTATE	1.00±0.14	0.78±0.08	0.77±0.08	0.77±0.09	0.72±0.10	0.74±0.10	0.71±0.10	0.73±0.13	1.48±0.26
SRBCT	0.84±0.03	0.77±0.02	0.77±0.02	0.78±0.02	0.80±0.02	0.79±0.02	0.78±0.02	0.75±0.02	0.83±0.03
SMK	0.98±0.07	0.68±0.02	0.68±0.03	0.68±0.02	0.78±0.04	0.78±0.04	0.73±0.03	0.69±0.03	4.32±0.45
S6.3 Reconstruction Results on pix10
Fig. S1 presents the reconstruction results on pix10 achieved by our method. We can see that our method is able to reconstruct the original images of 10000 dimensions reasonably well with only 300 features. Notably, the reconstructions capture important appearance details, including reflections, hair tips, and facial features, demonstrating the effectiveness of our method.
Figure S1: Reconstruction results on  $\mathrm{PIX10}$  with 300 features, where the training data are indicated in blue and testing data are indicated in light orange.
S6.4 Graph Visualization
We visualize COIL-20 and Jaffe with t-SNE, and plot the graph structures obtained by different methods. The results are shown in Fig. S2, where red lines represent the intra-class connections and blue lines represent inter-class connections. Unlike "Heat" and "DGL only" that use the original features for visualization, we visualize the data points using only selected features. Remarkably, our method successfully achieves separable structures using t-SNE, demonstrating its ability to capture features that reflect the intrinsic data structure.
(a) Heat
(b) DGL only