SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Denote the feasible region as  $\mathcal{F}$ , which is a closed convex set. Let  $K^0$  be the initial kernel matrix. Let  $\hat{K}$  be the projection of  $K^0$  onto  $\mathcal{F}$  ( $\hat{K} = \min_{K\in \mathcal{F}}\| K - K^0\| _F^2$ ). Let  $K^{*}$  be the unknown ground-truth with positive semi-definiteness (PSD) in the feasible region  $\mathcal{F}$ . By definition, we have:
$$
\left\| \hat {K} - K ^ {0} \right\| _ {F} ^ {2} \leq \left\| K ^ {*} - K ^ {0} \right\| _ {F} ^ {2}.
$$
Adding and subtracting  $\hat{K}$  in the right-hand side, we get:
$$
\left\| K ^ {*} - K ^ {0} \right\| _ {F} ^ {2} = \left\| K ^ {*} - \hat {K} + \hat {K} - K ^ {0} \right\| _ {F} ^ {2}.
$$
By the Pythagorean theorem [71], we have:
$$
\left\| K ^ {*} - \hat {K} + \hat {K} - K ^ {0} \right\| _ {F} ^ {2} = \left\| K ^ {*} - \hat {K} \right\| _ {F} ^ {2} + \left\| \hat {K} - K ^ {0} \right\| _ {F} ^ {2} - 2 \langle K ^ {*} - \hat {K}, K ^ {0} - \hat {K} \rangle_ {F},
$$
where  $\langle \cdot, \cdot \rangle_F$  denotes the Frobenius inner product. Since  $\hat{K}$  is the projection of  $K^0$  onto the feasible region  $\mathcal{F}$ , we have  $\langle K - \hat{K}, K^0 - \hat{K} \rangle_F \leq 0$  for all PSD matrices  $K$  in  $\mathcal{F}$  [72]. Therefore, we have:
$$
\left\| K ^ {*} - K ^ {0} \right\| _ {F} ^ {2} = \left\| K ^ {*} - \hat {K} + \hat {K} - K ^ {0} \right\| _ {F} ^ {2} \geq \left\| K ^ {*} - \hat {K} \right\| _ {F} ^ {2} + \left\| \hat {K} - K ^ {0} \right\| _ {F} ^ {2}.
$$
If  $K^0$  is already a PSD matrix in  $\mathcal{F}$ , we have:
$$
\hat {K} = K ^ {0} \text {a n d} \| K ^ {*} - K ^ {0} \| _ {F} ^ {2} = \| K ^ {*} - \hat {K} \| _ {F} ^ {2}.
$$
Otherwise, we obtain:
$$
\| \hat {K} - K ^ {0} \| _ {F} ^ {2} \geq 0,
$$
$$
\left\| K ^ {*} - K ^ {0} \right\| _ {F} ^ {2} \geq \left\| K ^ {*} - \hat {K} \right\| _ {F} ^ {2}. \tag {14}
$$
Thus, we have  $\| K^{*} - \hat{K}\|_{F}\leq \| K^{*} - K^{0}\|_{F}$ , which provides an improved estimate  $\hat{K}$  to  $K^{*}$ .
B Formula Derivations of Kernel Self-expressive Learning
B.1 Formula Derivations of Kernel Self-expressive Learning with Proximal p-norm
Due to the non-convex nature of the  $\| Z\| _p^p$  term and the penalty term, finding a closed-form solution for the  $Z$ -update is not straightforward. However, we can relax the matrix optimization to an elementwise optimization without loss of generality, thanks to the format of our chosen objective function. Below, we present the deriving process:
$$
\begin{array}{l} \mathcal {L} = \lambda \| Z \| _ {p} ^ {p} + \gamma \sum_ {i, j} \max  (z _ {i j} - 1, 0) + \operatorname {T r} (U ^ {\top} (Z - C)) + \frac {1}{2 \rho} \| Z - C \| _ {F} ^ {2} \\ = \lambda \sum_ {i, j} (z _ {i j}) ^ {p} + \gamma \sum_ {i, j} \max  (z _ {i j} - 1, 0) + \sum_ {i, j} \frac {1}{2 \rho} (z _ {i j} - c _ {i j}) ^ {2} + u _ {i j} (z _ {i j} - c _ {i j}) \\ = \sum_ {i, j} \left(\lambda z _ {i j} ^ {p} + \gamma \mathbb {I} _ {\left(z _ {i j} > 1\right)} \cdot \left(z _ {i j} - 1\right) + \frac {1}{2 \rho} \left(z _ {i j} - c _ {i j}\right) ^ {2} + u _ {i j} \left(z _ {i j} - c _ {i j}\right)\right), \\ \end{array}
$$
where  $\mathbb{I}_{(z_{ij} > 1)} = \left\{ \begin{array}{ll}1, & \text{if } z_{ij} > 1, \\ 0, & \text{otherwise.} \end{array} \right.$  is an indicator function and  $\mathcal{L}_p = \mathcal{L} + \| \phi(X) - \phi(X)C\|_F^2$ .
We can use a solver to find out the  $Z$ -update via
$$
\frac {\partial \mathcal {L} _ {p}}{\partial z _ {i j}} = \lambda p z _ {i j} ^ {p - 1} + \gamma \mathbb {I} _ {(z _ {i j} > 1)} + \frac {1}{\rho} (z _ {i j} - c _ {i j}) + u _ {i j} = 0.
$$
Using the Alternating Direction Method of Multipliers (ADMM) approach [61], we can update the variables as follows.
-  $C$ -update: Update  $C$  by minimizing the augmented Lagrangian with respect to  $C$ :
$$
\begin{array}{l} C_{t + 1} = \operatorname *{arg  min}_{C}\mathcal{L}_{p}(C,Z_{t},U_{t}) \\ = (2 K + \frac {1}{\rho} I _ {n}) ^ {- 1} (2 K + U _ {t} + \frac {1}{\rho} Z _ {t}), \\ \end{array}
$$
where  $K\coloneqq \phi (X)^\top \phi (X)$  and  $I_{n}$  is the identity matrix of size  $n\times n$
-  $Z$ -update: Update  $Z$  via solving
-  $U$ -update: Update  $U$  via
$$
U _ {t + 1} = U _ {t} + \rho \left(Z _ {t + 1} - C _ {t + 1}\right).
$$
We employ the ADMM process to solve the problem using the augmented Lagrangian function. However, despite the existence of a derivation method for this process, the computation proves to be excessively slow due to the following factors:
- Matrix inversion: Inverting  $(2K + \frac{1}{\rho} I_n)^{-1}$  has a computational complexity of  $O(n^3)$ .
- Solver for  $Z$ -update: The golden-section solver is chosen to update  $z_{ij}$ , resulting in an approximate computational complexity of  $\log \left( \frac{1}{\epsilon} \right)$  for each entry [73], which is relatively high.
Additionally, some potential complex solutions may arise when performing the element-wise update. As a result, we provide the algorithm for future researchers to further investigate and optimize.
B.2 Formula Derivations of Kernel Self-expressive Learning with Schatten p-norm
For  $\frac{1}{2} < p < 1$ , we can define a relaxation of the optimization problem as:
$$
\mathcal {L} _ {S _ {p}} = \| \phi (X) - \phi (X) U V ^ {\top} \| _ {F} ^ {2} + \frac {\lambda}{2} \| U \| _ {F} ^ {2} + \frac {\lambda}{2} \| V \| _ {F} ^ {2}.
$$
With this definition, we can employ gradient descent to update the variables iteratively. The main steps of the algorithm are as follows:
-  $U$ -update: Compute the gradient of  $\mathcal{L}_{S_p}$  with respect to  $U$ :
$$
\frac {\partial \mathcal {L} _ {S _ {p}}}{\partial U} = 2 K (U V ^ {\top} - I) V + \lambda U,
$$
where  $K\coloneqq \phi (X)^\top \phi (X)$ . Update  $U$  according to the gradient:
$$
U _ {\text {n e w}} = U _ {\text {o l d}} - \alpha \nabla_ {U} \mathcal {L} _ {S _ {p}},
$$
where  $\alpha$  is the learning rate.
-  $V$ -update: Use the closed-form solution
$$
V = 2 K U (2 U ^ {\top} K U + \lambda I) ^ {- 1}.
$$
- Convergence condition:
If the difference between the updated  $U$  and the old  $U$  is less than a pre-defined tolerance  $(\| U - U_{\mathrm{old}} \|_F^2 < tol)$ , the algorithm is considered as converged, and we can terminate the loop. Otherwise, continue iterating with the new  $U$  and  $V$  values.
Upon convergence, the affinity matrix can be computed as  $A = (|C| + |C|^{\top}) / 2$ , where  $C = UV^{\top}$ . This algorithm provides an efficient way to update the Schatten  $p$ -norm in the proposed KSL-Sp model, enhancing its robustness and clustering accuracy.
B.3 Formula Derivations of Adaptive Kernel Least-Squares Representation
The problem that we are doing with is:
$$
\min  _ {K \succeq 0, C} \| K - K ^ {0} \| _ {F} ^ {2} + \operatorname {T r} (K - 2 K C + C ^ {\top} K C) + \lambda \| C \| _ {F} ^ {2}.
$$
Then we derive the augmented Lagrange equation in preparation for ADMM.
- By introducing an auxiliary variable A, we have:
$$
\begin{array}{l} \min  _ {K \succeq 0, A, C} \| K - K ^ {0} \| _ {F} ^ {2} + \operatorname {T r} (A - 2 A C + C ^ {\top} A C) + \lambda \| C \| _ {F} ^ {2} \\ \begin{array}{l} \text {s . t .} K = A. \end{array} \\ \end{array}
$$
- Augmented Lagrange equation is calculated by
$$
\begin{array}{l} \mathcal {L} (K, A, C, U) = \left\| K - K ^ {0} \right\| _ {F} ^ {2} + \operatorname {T r} (A - 2 A C + C ^ {\top} A C) + \lambda \left\| C \right\| _ {F} ^ {2} \\ + \operatorname {T r} (U ^ {\top} (K - A)) + \frac {\rho}{2} \| K - A \| _ {F} ^ {2}, \\ \end{array}
$$
where  $U$  is Lagrange multiplier,  $\rho$  is the updating step size.
ADMM consists of iteratively updating  $K$ ,  $A$ ,  $C$ , and  $U$  as follows:
1.  $K$ -update:  $K = \arg \min_{K} \mathcal{L}(K, A_t, C_t, U_t)$ . By First-Order Necessary Condition (FONC), we have
$$
\frac {\partial \mathcal {L}}{\partial K} = 0 \Rightarrow K = \frac {2 K ^ {0} - U + \rho A}{\rho + 2}.
$$
2.  $K$ -PSD:  $K_{t+1} \gets \mathcal{P}_1(K)$  by projecting  $K$  onto the PSD set via Eq. (6).
3.  $A$ -update:  $A_{t + 1} = \arg \min_A\mathcal{L}(K_{t + 1},A,C_t,U_t)$ . By FONC, we have
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial A} = \rho A + (I - 2 C ^ {\top} + C C ^ {\top} - U - \rho K) = 0 \\ \Rightarrow A = \frac {\rho K + U + 2 C ^ {\top} - C C ^ {\top} - I}{\rho}. \\ \end{array}
$$
4.  $C$ -update:  $C_{t + 1} = \arg \min_C\mathcal{L}(K_{t + 1},A_{t + 1},C,U_t)$ . By FONC, we have
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial C} = (2 \lambda I + A + A ^ {\top}) C - 2 A ^ {\top} = 0 \\ \Rightarrow C = 2 (2 \lambda I + A + A ^ {\top}) ^ {- 1} A ^ {\top}. \\ \end{array}
$$
5.  $U$ -update:
$$
U _ {t + 1} = U _ {t} + \rho \left(K _ {t + 1} - A _ {t + 1}\right).
$$
C.1 Datasets
To evaluate the performance of our proposed approaches, we select four benchmark datasets commonly used in machine learning and computer vision, including:
- Yale64 [64]  $^{3}$ : The Yale64 dataset consists of face images of 15 different individuals, each with different facial expressions, lighting conditions, and facial details. It contains a total of 165 images of size  $64 \times 64$  pixels in 4,096-dimensional vectors.
- Umist [65]  $^{4}$ : The Umist dataset consists of face images of 20 different individuals, each with different facial expressions, lighting conditions, and facial details. It contains a total of 575 images of size  $32 \times 32$  pixels in 1,024-dimensional vectors.
- USPS [66]  $^{5}$ : The USPS dataset consists of handwritten digit images of size  $16 \times 16$  pixels in 10 classes. For the five repeated experiments in this paper, we use five different random seeds to randomly select 1,000 images with 256-dimensional vectors as a subset.
- Isolet [67]  $^{6}$ : The Isolet dataset is a speech dataset that contains recordings of 26 different speakers pronouncing the names of the English alphabet. It contains 1,560 recordings and each recording is of 617-dimensional vectors.
C.2 Baselines
The proposed approaches are evaluated against a range of baseline methods, including
- ZERO: This statistical imputation method replaces missing values with zeros.
- MEAN: This statistical imputation method replaces missing values with the mean value of all observed values in the corresponding features.
- kNN [26]: This statistical imputation method predicts missing values based on the values of the  $k$ -nearest neighbors in the available data. (default value  $k = 10$ )
- EM [68]: This statistical imputation method estimates the missing values based on the Expectation Maximization algorithm.
- SVT [27]: This matrix completion method uses a singular value thresholding algorithm to find the low-rank matrix that best fits the available data.
- GR [28]: This matrix completion method uses an online algorithm to estimate the low-rank subspace of the data and then updates the subspace with new data points.
<sup>3</sup>https://www.kaggle.com/datasets/olgabelitskaya/yale-face-database
<sup>4</sup>https://cs.nyu.edu/ roweis/data/umist_cropped.mat
<sup>5</sup>https://cs.nyu.edu/ roweis/data/usps_all.mat
<sup>6</sup>http://archive.ics.uci.edu/ml/datasets/ISOLET
- KFMC [29]: This matrix completion method uses a kernelized factorization approach to estimate the high-rank matrix that best fits the available data. (default: polynomial kernel)
- DC [38]: This distance calibration method calibrates an initial distance matrix through double centering algorithm and transformation between similarity and distance.
- TRF [35]: This distance calibration method calibrates an initial distance matrix to satisfy all the triangle inequalities by solving a convex optimization problem.
- EE [39]: This distance calibration method calibrates an initial distance matrix to a Euclidean embeddable matrix by solving a convex optimization problem. (default value  $\gamma = \frac{0.02}{\max\{d_{ij}^0\}}$ ).
C.3 Implementation Details
Data Preprocessing. To obtain better clustering performance, we conduct different preprocessing strategies. In the Yale64 and USPS datasets, we use the original data without preprocessing; in the Umist and Isolet datasets, we perform the normalization to normalize all data samples into  $[-1, 1]$ .
Generation of Incomplete Data. For experiments in Sections 5.1, 5.2 and 5.3, we generate an incomplete data matrix  $X^0$  by randomly replacing each value in the data matrix  $X \in \mathbb{R}^{d \times n}$  with an NA value with a given probability  $r$  (missing completely at random). We repeat this process five times and report the average performance of the baseline methods using  $X^0$  as input.
Data Imputation Methods. We first impute the missing values in  $X^0$  and then calculate the kernel matrix based on the imputed matrix  $\hat{X}$ . We then perform spectral clustering on the kernel matrix.
Distance Calibration Methods. We first estimate an initial distance matrix  $D^0$  on the incomplete data matrix  $X^0$  via Eq. (1). We then calibrate  $D^0$  to  $\hat{D}$  and calculate the kernel matrix based on  $\hat{D}$ . We finally perform spectral clustering on the resulting kernel matrix.
D Validation of Kernel Correction on Kernel and Distance Estimation
Two errors, relative-mean-square error  $(\mathbf{RMSE} = \frac{\|\hat{A} - A^{*}\|_{F}^{2}}{\|A^{0} - A^{*}\|_{F}^{2}})$  and relative error  $(\mathbf{RE} = \frac{\|\hat{A} - A^{*}\|_{F}}{\|A^{*}\|_{F}})$ , are used to evaluate the estimation accuracy, where  $A^0$  is the naive kernel/distance matrix obtained by Eq. (1),  $\hat{A}$  is the estimated one using different methods, and  $A^{*}$  as the ground-truth. Besides, Recall measures the search accuracy of local relationships, which is defined as the overlapping ratio of two sets of top-10 similar items obtained by  $\hat{A}$  and  $A^{*}$ , respectively.
Table 5: Quality comparisons of Gaussian kernel and Euclidean distance under a missing ratio  ${80}\%$  .
Dataset	Metric	Naive	ZERO	MEAN	kNN	EM	SVT	GR	KFMC	DC	TRF	EE	KC
Yale64	RMSE-K↓	1.000	11.52	3.009	11.46	3.009	11.39	11.13	8.838	2.564	0.994	0.741	0.624
RMSE-D↓	1.000	12.45	76.82	14.26	76.82	12.52	12.27	11.62	37.97	0.995	0.755	0.674
RE-K↓	0.113	0.382	0.195	0.381	0.195	0.380	0.376	0.335	0.180	0.112	0.097	0.089
RE-D↓	0.064	0.227	0.564	0.243	0.564	0.228	0.225	0.219	0.396	0.064	0.056	0.053
Recall↑	0.721	0.063	0.275	0.063	0.275	0.066	0.070	0.183	0.571	0.722	0.751	0.767
Umist	RMSE-K↓	1.000	2.849	1.535	2.451	1.527	2.849	1.797	3.818	1.748	0.923	0.438	0.382
RMSE-D↓	1.000	20.80	27.64	21.20	27.62	20.80	8.397	14.05	110.4	0.916	0.440	0.431
RE-K↓	0.189	0.319	0.234	0.295	0.233	0.319	0.253	0.369	0.250	0.181	0.125	0.117
RE-D↓	0.107	0.487	0.561	0.492	0.561	0.487	0.309	0.400	1.122	0.102	0.071	0.070
Recall↑	0.726	0.092	0.171	0.119	0.172	0.092	0.596	0.248	0.226	0.740	0.771	0.785
USPS	RMSE-K↓	1.000	0.471	0.292	0.411	0.393	0.469	0.389	0.475	0.386	0.547	0.262	0.223
RMSE-D↓	1.000	2.820	4.378	2.936	2.850	2.825	2.003	2.100	67.82	0.501	0.252	0.240
RE-K↓	0.460	0.316	0.248	0.295	0.287	0.315	0.287	0.317	0.286	0.340	0.235	0.217
RE-D↓	0.268	0.451	0.562	0.460	0.445	0.451	0.380	0.389	2.210	0.190	0.135	0.132
Recall↑	0.071	0.054	0.065	0.069	0.036	0.054	0.122	0.196	0.056	0.161	0.186	0.197
Isolet	RMSE-K↓	1.000	1.188	0.796	0.966	0.993	1.188	1.001	1.390	0.956	0.817	0.339	0.291
RMSE-D↓	1.000	7.210	12.35	7.620	9.538	7.210	4.595	4.365	153.5	0.809	0.332	0.309
RE-K↓	0.282	0.307	0.251	0.277	0.280	0.307	0.282	0.332	0.275	0.255	0.164	0.152
RE-D↓	0.160	0.429	0.561	0.441	0.490	0.429	0.342	0.334	1.978	0.144	0.092	0.089
Recall↑	0.254	0.045	0.050	0.059	0.034	0.045	0.187	0.225	0.041	0.299	0.270	0.300
E Comprehensive Experiments on Standard Spectral Clustering
The robustness analysis is depicted in Fig. 1 in the main text, with accompanying numerical results in Table 6. Further, Fig. 2 displays clustering performance using ACC, NMI, and PUR metrics, underscoring the KC method's superiority over imputation baselines, particularly at large missing ratios. Notably, as the missing ratio escalates, imputation methods' accuracy and clustering efficacy decline sharply. Specifically, at missing ratios above  $50\%$ , their reliability wanes due to the paucity of data for imputation. In contrast, our KC method maintains consistent, and often superior, performance even with large missingness, exemplified by the results on the Yale64 and Umist datasets.
Table 6: Robustness analysis of kernel and distance estimation, neighborhood relationship, and standard spectral clustering on Umist. Bold font indicates best, and underline indicates second-best.
Metric	Missing	Naive	ZERO	MEAN	kNN	EM	SVT	GR	KFMC	DC	TRF	EE	KC
RE-K↓	20%	0.028	0.116	0.072	0.023	0.108	0.106	0.043	0.025	0.072	0.028	0.029	0.028
50%	0.066	0.232	0.157	0.115	0.180	0.232	0.111	0.076	0.162	0.065	0.062	0.057
80%	0.189	0.319	0.234	0.295	0.233	0.319	0.253	0.369	0.250	0.181	0.125	0.117
RE-D↓	20%	0.016	0.087	0.112	0.055	0.138	0.088	0.055	0.029	0.110	0.016	0.017	0.016
50%	0.037	0.237	0.302	0.222	0.281	0.237	0.144	0.090	0.372	0.037	0.035	0.033
80%	0.107	0.487	0.561	0.492	0.561	0.487	0.309	0.400	1.122	0.102	0.071	0.070
Recall↑	20%	0.954	0.887	0.912	0.952	0.870	0.887	0.944	0.964	0.932	0.954	0.950	0.956
50%	0.899	0.558	0.672	0.778	0.630	0.558	0.856	0.910	0.783	0.899	0.907	0.914
80%	0.726	0.092	0.171	0.119	0.172	0.092	0.596	0.248	0.226	0.740	0.771	0.785
ARI↑	20%	0.455	0.370	0.405	0.451	0.388	0.380	0.436	0.439	0.427	0.456	0.466	0.460
50%	0.443	0.258	0.332	0.326	0.325	0.256	0.398	0.448	0.361	0.434	0.450	0.451
80%	0.373	0.070	0.206	0.082	0.207	0.067	0.304	0.140	0.216	0.370	0.371	0.377
Figure 2: Robustness analysis of standard spectral clustering using the Gaussian kernel on the Yale64 and Umist datasets under a wide range of missing ratios, i.e.,  $r \in [20\%, 80\%]$ .
F Comprehensive Experiments on Self-expressive Affinity Learning
Hyperparameters of Affinity Learning Algorithms. All kernel self-expressive affinity learning algorithms in this study adhere to a unified framework, i.e.,  $\min_{C} \frac{1}{2} \| \phi(X) - \phi(X)C \|_F^2 + \lambda \mathcal{R}(C)$ , where  $\lambda$  is a regularization hyperparameter. Empirically, we choose  $\lambda$  from the interval [0, 30] to roughly achieve better clustering performance, as detailed in Table 7. Note that the optimal  $\lambda$  may vary across algorithm implementations under different settings.
Table 7: Values of the hyperparameter  $\lambda$  in all experiments in Section 5.3.
Method	Yale64	Umist	USPS	Isolet
KSSC	0.08	0.1	0.07	0.06
KLSR	8	25	8	25
KSL-Sp	0.6	1	10	10
AKLSR	0.5	0.4	0.3	0.5
Clustering Performance on the Exponential Kernel. We investigate the benefits of the corrected Exponential kernel for kernel self-expressive affinity learning, including KSSC [19] and KLSR [3]. Tables 8 shows that our KC algorithm outperforms baselines regarding the NMI metric on the Exponential kernel based clustering, offering a systematic solution to clustering incomplete data.
Table 8: NMI Performance of self-expressive affinity on Exponential kernel with  ${80}\%$  missing.
Method	Dataset (λ)	Naive	ZERO	MEAN	kNN	EM	SVT	GR	KFMC	DC	TRF	EE	KC
KSSC	Yale64 (0.03)	0.209	0.366	0.632	0.371	0.635	0.354	0.362	0.283	0.596	0.203	0.624	0.638
Umist (0.03)	0.115	0.614	0.665	0.583	0.662	0.613	0.669	0.326	0.657	0.110	0.676	0.684
USPS (0.05)	0.020	0.039	0.368	0.250	0.227	0.039	0.356	0.348	0.022	0.017	0.479	0.472
Isolet (0.06)	0.064	0.069	0.123	0.085	0.094	0.069	0.120	0.132	0.071	0.060	0.634	0.661
KLSR	Yale64 (20)	0.614	0.304	0.573	0.309	0.565	0.295	0.312	0.328	0.586	0.618	0.625	0.634
Umist (0.7)	0.127	0.570	0.655	0.557	0.653	0.566	0.682	0.487	0.642	0.130	0.688	0.693
USPS (25)	0.083	0.309	0.433	0.354	0.217	0.310	0.341	0.381	0.440	0.217	0.513	0.515
Isolet (20)	0.453	0.413	0.571	0.468	0.417	0.417	0.543	0.510	0.530	0.517	0.659	0.676
Trade-off between Efficiency and Performance of the KC Algorithm. To enhance the efficiency of the KC algorithm, we explore the use of randomized singular value decomposition (rSVD) [53] - a method that focuses on identifying the top- $k$  singular values (refer to Section 3.3). We compare the performance of the KC method with SD or rSVD on the USPS dataset. The results, presented in Table 9, reveal that rSVD greatly enhances the operational efficiency of the KC algorithm. However, it is crucial to choose an appropriate value for  $k$  (number of top singular values), as it heavily impacts the quality of clustering. Smaller  $k$  values (e.g., 10 or 20) result in poorer clustering due to the loss of important singular value information. On the other hand, selecting a suitable  $k$  value (e.g., 50 or 100) significantly reduces running time while maintaining clustering performance comparable to SD, particularly for the standard spectral clustering algorithm (SC). Striking a trade-off between efficiency and performance remains an intriguing avenue for future research.
Table 9: Comparisons of rSVD and SD in the KC algorithm on the USPS with  $80\%$  missing.
Method	rSVD	SD
Metric	k=10	k=20	k=50	k=100
RE-K	0.695	0.419	0.262	0.291	0.217
NMI-SC	0.026	0.164	0.431	0.439	0.472
NMI-KSSC	0.021	0.019	0.018	0.019	0.360
NMI-KLSR	0.021	0.122	0.210	0.018	0.485
Time (sec)	1.96	2.12	3.10	5.76	72.97
Comparisons of KSL-Sp and AKLSR. AKLSR outpaces KSL-Sp in execution speed. Algorithm 4 delineates each AKLSR update step explicitly, ensuring quicker convergence, whereas KSL-Sp in Algorithm 3 relies on gradient descent for  $U$  updates, leading to extended convergence times. Moreover, AKLSR's performance can vary with the presence or absence of a PSD constraint. Table 10 showcases the efficiency of AKLSR without PSD and its effectiveness with PSD on the Umist dataset.
Table 10: Comparisons of KSL-Sp and AKLSR on Umist using the Gaussian kernel obtained by KC.
Metric	SC	KSL-Sp	AKLSR w/o PSD	AKLSR w/ PSD
ARI	0.377	0.395	0.398	0.403
Time (sec)	-	12.57	0.04	0.23