SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fangchen Yu $^{1}$ , Runze Zhao $^{1}$ , Zhan Shi $^{1}$ , Yiwen Lu $^{1}$   
Jicong Fan $^{1,2}$ , Yicheng Zeng $^{2}$ , Jianfeng Mao $^{1,2}$ , Wenye Li $^{1,2*}$
<sup>1</sup> The Chinese University of Hong Kong, Shenzhen, China
2 Shenzhen Research Institute of Big Data, Shenzhen, China
{fangchenyu, runzezhao, zhanshi1, yiwenlu1}@link.cuhk.edu.cn
fanjicong@cuhk.edu.cn, statzyc@sribd.cn, jfmao@cuhk.edu.cn, wyli@cuhk.edu.cn
Spectral clustering has gained popularity for clustering non-convex data due to its simplicity and effectiveness. It is essential to construct a similarity graph using a high-quality affinity measure that models the local neighborhood relations among the data samples. However, incomplete data can lead to inaccurate affinity measures, resulting in degraded clustering performance. To address these issues, we propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. Firstly, we introduce a new kernel correction method that enhances the quality of the kernel matrix estimated on incomplete data with a theoretical guarantee, benefiting classical spectral clustering on pre-defined kernels. Secondly, we develop a series of affinity learning methods that equip the self-expressive framework with  $\ell_p$ -norm to construct an intrinsic affinity matrix with an adaptive extension. Our methods outperform existing data imputation and distance calibration techniques on benchmark datasets, offering a promising solution to spectral clustering on incomplete data in various real-world applications.
Spectral clustering [1, 2, 3] has become a widely used and effective method for clustering non-convex data and finds diverse applications in computer vision [4, 5], natural language processing [6, 7], and bioinformatics [8, 9]. Generally, the first step in spectral clustering involves constructing an affinity matrix that captures the similarity between data points, followed by performing normalized cut [4] on the corresponding graph to partition the data into clusters. The quality of the affinity matrix is a critical factor that determines the effectiveness of the clustering performance [2, 3]. However, incomplete data is commonly seen in practice, leading to inaccurate affinities and degraded clustering performance [10, 11, 12]. As such, obtaining a high-quality affinity matrix with missing data is a challenging task that requires specialized techniques. In this paper, we aim to improve the quality of affinity matrices, which naturally enhance the performance of spectral clustering on incomplete data.
In recent years, the development of methods to construct affinity matrices with full information for spectral clustering has garnered significant attention. Two types of affinity matrices are typically utilized, namely pre-defined similarity matrices [1, 2, 13, 14] and self-expressive affinity matrices [3, 15, 16, 17, 18, 19, 20]. Pre-defined similarity matrices are easy to compute and encompass well-known kernels, including the Gaussian kernel, Exponential kernel, and Polynomial kernel. The Gaussian kernel is the most widely used due to its simplicity and effectiveness [2]. Differently, self-expressive affinity matrices learn the affinity matrix  $C$  by representing each data point as a
linear combination of other data points, i.e.,  $\min_C\frac{1}{2}\| X - XC\| _F^2 +\lambda \mathcal{R}(C)$ , where the columns of  $X\in \mathbb{R}^{d\times n}$  are data points,  $\mathcal{R}(C)$  denotes a regularization term, and  $\lambda$  is a hyperparameter, as exemplified by Sparse Subspace Clustering (SSC) [15], Low-Rank Representation (LRR) [16], and Least-Squares Representation (LSR) [17, 18]. Moreover, self-expressive methods have been expanded by integrating kernel functions into the framework, such as kernel SSC (KSSC) [19], kernel LRR (KLRR) [20], and kernel LSR (KLSR) [3], through  $\min_C\frac{1}{2}\| \phi (X) - \phi (X)C\| _F^2 +\lambda \mathcal{R}(C)$ , where  $\phi (X)$  is a mapping function and  $K\coloneqq \phi (X)^{\top}\phi (X)$  denotes a kernel matrix. Despite the varying techniques on affinity matrices, the kernel matrix remains a cornerstone of spectral clustering for pre-defined similarities and self-expressive affinities. Nevertheless, constructing a high-quality kernel/affinity matrix is an ongoing challenge when facing incomplete data. The primary obstacle is the inability to directly calculate a kernel matrix or learn an affinity matrix due to missing values.
To address the challenge of incomplete data, data imputation techniques [21, 22, 23] such as statistical imputation and matrix completion, are conventional approaches to fill in the missing entries of the data matrix before clustering. Statistical imputation methods [24, 25], such as zero, mean, and  $k$ -nearest neighbors ( $k\mathrm{NN}$ ) imputation [26], are fast and easy to implement and can flexibly handle missing data. However, these methods may introduce bias and fail to capture the true underlying data structure, hindering the accurate affinity measures. Matrix completion methods [27, 28, 29, 30], on the other hand, aim to accurately recover the underlying structure of the data by finding a low-rank or high-rank approximation of the data matrix. Although they can handle missing data in a principled way, their performance heavily depends on the consistency of data structure and the assumptions being made, easily affected by the data distribution. Most importantly, almost all imputation methods do not offer a guarantee for the quality of the affinity matrix calculated on the imputed data [31].
Distance calibration techniques [32, 33] have emerged as an alternative solution by obtaining a high-quality distance matrix for incomplete data. These techniques calibrate an initial non-metric distance matrix estimated on incomplete data to a distance metric. Several methods have been proposed, such as the metric nearness model [34, 35, 36], which finds the nearest approximation that satisfies all triangle inequalities [37]. The double-centering algorithm [38] converts the initial non-metric distance matrix into a Euclidean distance matrix, while the Euclidean embedding method [39, 40, 41] ensures the Euclidean embeddable property is satisfied through convex optimization. Although some of these methods guarantee the quality of the calibrated distance [39], they only apply to distance-based affinities like the Gaussian kernel. Moreover, these methods may not generate high-quality affinity matrices based on calibrated distance, making it necessary to develop dedicated methods that can apply to a family of kernels and affinities with theoretical guarantees.
To address the above issues, we propose an imputation-free framework to directly learn high-quality affinity matrices via two main techniques, i.e., kernel correction and affinity learning, aiming to improve the spectral clustering performance on incomplete data, with contributions as follows.
- We propose an imputation-free framework based on kernel correction and affinity learning with improved clustering performance, providing convenient tools to deal with incomplete data. To our best knowledge, this is the first systematical work to discuss spectral clustering on incomplete data, which is a commonly seen problem in practice.
- We introduce a novel kernel correction method that directly focuses on the kernel matrix estimated from the incomplete data, and corrects it to satisfy specific mathematical properties such as positive semi-definiteness (PSD). We show that the corrected kernel matrix becomes closer to the unknown ground-truth with a theoretical guarantee, which is beneficial to spectral clustering algorithms and cannot be assured by imputation and calibration methods.
- We develop a series of new affinity learning methods to equip the self-expressive framework with the  $\ell_p$ -norm to capture the underlying structure of data samples better. Additionally, we combine kernel correction and affinity learning to arrive at an adaptive learning method that simultaneously learns the high-quality kernel and the self-expressive affinity matrix.
- We conduct extensive experiments that demonstrate the effectiveness of proposed methods on various benchmark datasets, showing superior results in terms of kernel estimation, spectral clustering, and affinity learning on incomplete data, compared to existing data imputation and distance calibration approaches.
2.1 Spectral Clustering and Affinity Learning
Standard Spectral Clustering involves three steps [2]: first, a Gaussian kernel matrix  $K \in \mathbb{R}^{n \times n}$  is calculated to measure the similarity between  $n$  data points with  $K_{ij} = \exp(-\| x_i - x_j\|^2 / \sigma^2)$  where  $\sigma$  is a hyperparameter. Then, an affinity graph  $A \in \mathbb{R}^{n \times n}$  is constructed using the kernel matrix, which can take the form of an  $\epsilon$ -neighborhood graph, a fully connected graph, or a  $k$ -nearest neighbors ( $kNN$ ) graph. An  $\epsilon$ -neighborhood graph connects pairwise points with a threshold value  $\epsilon$ , while a fully connected graph connects all points. Empirically, a  $kNN$  graph is the most popular one that connects each point to its  $k$ -nearest neighbors, resulting in sparse local relationships and relatively high clustering accuracy [3]. Finally, the normalized cut algorithm [1] is applied to the affinity graph  $A$  to partition the data into clusters based on the normalized Laplacian matrix.
Self-expressive Affinity Learning is a framework to learn affinity matrices by modeling the relationships between data points, i.e.,  $\min_{C} \frac{1}{2} \|X - XC\|_{F}^{2} + \lambda \mathcal{R}(C)$ , with different types of regularization terms  $\mathcal{R}(C)$ . Sparse Subspace Clustering (SSC) [15] assumes that the data points lie in a low-dimensional subspace and seeks to find a sparse affinity matrix with  $\mathcal{R}(C) = \|C\|_{1} = \sum_{i,j=1}^{n} |c_{ij}|$  under a constraint  $\mathrm{diag}(C) = 0$ . Low-Rank Representation (LRR) [16] is another approach that seeks a low-rank affinity matrix with  $\mathcal{R}(C) = \|C\|_{*}$  (nuclear norm of  $C$ ). Least-Squares Representation (LSR) [17, 18] involves solving a least-squares problem to find the representation matrix with  $\mathcal{R}(C) = \frac{1}{2} \|C\|_{F}^{2}$ . Their kernel variants, i.e., KSSC [19], KLRR [20], and KLSR [3] are used to extend their applicability to non-linearly separable data by applying a kernel function to data points with  $\min_{C} \frac{1}{2} \| \phi(X) - \phi(X)C \|_{F}^{2} + \lambda \mathcal{R}(C)$ , showing promising performance in spectral clustering.
2.2 Missing Data Processing Techniques
Data Imputation is a popular technique [21, 22, 23] for dealing with incomplete data by filling in missing values. Statistical imputation methods, such as zero, mean imputation, and  $k$ -nearest neighbors ( $kNN$ ) approach [26], have been widely used in practice. These methods replace the missing value with a zero, mean, or  $k$ -weighted value of non-missing elements in the corresponding feature. Additionally, matrix completion [27, 28, 29, 30] is a machine learning-based technique that fills missing values by solving a matrix factorization problem under assumptions on data structures such as low-rank or high-rank. However, correctly estimating missing values based on observed data is difficult, especially for a large missing ratio, and there is no guarantee on the quality of the affinity matrix on imputed data. This motivates us to design imputation-free approaches in Sections 3 and 4.
Distance Calibration is a specialized approach to obtaining a valid distance metric from an initial non-metric distance matrix, which can be applied to incomplete data. For any two incomplete data samples  $x_{i}, x_{j} \in \mathbb{R}^{d}$ , a new vector  $x_{i}(I) \in \mathbb{R}^{|I|}$  is formed by selecting the observed values of  $x_{i}$  on the set  $I$ , where  $I$  is an index set of all features observed in both samples. The pairwise Euclidean distance between  $x_{i}$  and  $x_{j}$  can then be heuristically estimated by [39, 41]
$$
d _ {i j} ^ {0} = \left\| x _ {i} (I) - x _ {j} (I) \right\| _ {2} \cdot \sqrt {\frac {d}{| I |}}. \tag {1}
$$
However, the initial Euclidean distance matrix  $D^0 = [d_{ij}^0] \in \mathbb{R}^{n \times n}$  estimated on incomplete data is usually not a distance metric due to missing values. Distance calibration methods [34, 35, 38, 39] can correct  $D^0$  to a distance metric by making different assumptions and leveraging various properties. More details of these methods are discussed in Section 3.1.
3 Methodology-I. Kernel Correction
3.1 Revisiting Distance Calibration Methods
We begin with the definition of a distance metric [42], then delve into distance calibration methods, discussing the assumptions and properties underlying each method, as well as their limitations.
Definition 1. A distance metric is defined as a  $n \times n$  real symmetric matrix  $D$  that satisfies
$$
d _ {i j} = d _ {j i} \geq 0, d _ {i i} = 0, d _ {i k} \leq d _ {i j} + d _ {j k}, f o r a l l 1 \leq i, j, k \leq n.
$$
The properties of non-negativity, reflexivity, symmetry, and triangle inequality ensure that the distance metric produces meaningful and valuable distance measurements.
- The Triangle Fixing (TRF) Algorithm [35] to obtain a distance metric is based on the metric nearness model [34, 36], which finds the nearest approximation of a non-metric  $D^0$  by solving:
$$
\underset {D \in \mathbb {R} ^ {n \times n}} {\text {m i n i m i z e}} \| D - D ^ {0} \| _ {F} ^ {2}, \text {s u b j e c t t o} d _ {i j} = d _ {j i} \geq 0, d _ {i i} = 0, d _ {i k} \leq d _ {i j} + d _ {j k}, \forall 1 \leq i, j, k \leq n. \tag {2}
$$
However, there are two significant limitations. First, the scalability of the algorithm is severely limited by the matrix size  $n$ . Since the number of  $O(n^3)$  constraints grows rapidly as  $n$  increases, the optimization time can become lengthy, taking several hours for a few thousand samples. Second, the performance of the algorithm depends on the extent of violation of the triangle inequalities in the initial distance matrix  $D^0$ . With a small missing ratio of features,  $D^0$  already satisfies most triangle inequalities, i.e.,  $d_{ik}^0 \leq d_{ij}^0 + d_{jk}^0$ , then the algorithm typically yields only marginal improvement.
- The Double-Centering (DC) Algorithm [38] starts with the Euclidean Distance Matrix (EDM) [43] and utilizes Property 1 [44], a well-known result in classical Multi-Dimensional Scaling (cMDS) [45], to build the connection between the similarity matrix and the EDM.
Definition 2. A  $n \times n$  real symmetric matrix  $D$  is called an EDM if there exists  $x_1, x_2, \ldots, x_n \in \mathbb{R}^d$  such that  $d_{ij} = \| x_i - x_j\|^2$  for  $i, j = 1, 2, \ldots, n$ . (Note that it uses squared Euclidean distance.)
Property 1. A  $n \times n$  real symmetric matrix  $D$  is EDM if and only if  $\operatorname{diag}(D) = 0$ ,  $S \coloneqq -\frac{1}{2} JDJ \succeq 0$ ,  $J \coloneqq I - ee^{\top} / n$ , where  $I$  is the identity matrix and  $e$  is the  $n$ -dimension vector of all ones.
The DC algorithm first finds the nearest positive semi-definite  $\hat{S}$  by solving
$$
\underset {S \in \mathbb {R} ^ {n \times n}} {\text {m i n i m i z e}} \| S - (- \frac {1}{2} J (D ^ {0} \circ D ^ {0}) J) \| _ {F} ^ {2}, \text {s u b j e c t t o} S \succeq 0, \tag {3}
$$
where  $D^0$  is the initial Euclidean distance matrix and  $\circ$  denotes the Hadamard product. The algorithm then transforms  $\hat{S}$  into an EDM  $\hat{D}$  using  $\hat{d}_{ij} = \hat{s}_{ii} + \hat{s}_{jj} - 2\hat{s}_{ij}$ , and obtain the calibrated (non-squared) distance matrix by  $\hat{D} \gets [\hat{d}_{ij}^{1/2}]$ , but unfortunately, the quality of  $\hat{D}$  cannot be guaranteed, and important information may be lost during the transformation [38].
- The Euclidean Embedding (EE) Algorithm [39, 40, 41] leverages Euclidean embeddable property [39, 46] to obtain a calibrated embeddable distance matrix by solving
$$
\underset {D \in \mathbb {R} ^ {n \times n}} {\text {m i n i m i z e}} \| D - D ^ {0} \| _ {F} ^ {2}, \text {s u b j e c t t o} \exp (- \gamma D) \succeq 0, d _ {i i} = 0, d _ {i j} = d _ {j i} \geq 0, \forall 1 \leq i, j \leq n. \tag {4}
$$
Definition 3. A  $n \times n$  real symmetric matrix  $D$  is said Euclidean embeddable if there exists  $x_1, \ldots, x_n$  in Euclidean space and a distance function  $\rho$  such that  $d_{ij} = \rho(x_i, x_j) \geq 0$  for  $i, j = 1, \ldots, n$ .
Property 2. [Theorem 2, [39]] If a  $n \times n$  real symmetric matrix  $D$  is Euclidean embeddable, then the kernel  $K := \exp(-\gamma D)$  is positive semi-definite for any  $\gamma > 0$ .
It is important to note that these methods can only provide a calibrated distance matrix with benefits for distance-based kernels and are not a universal solution for dealing with incomplete data in spectral clustering tasks, which motivates us to further design a kernel-specialized method in Section 3.2.
3.2 Kernel Correction Algorithm
To overcome the limitations of distance calibration, we propose a new kernel correction algorithm that directly focuses on the construction of a high-quality kernel with a theoretical guarantee. We consider an incomplete data matrix  $X \in \mathbb{R}^{d \times n}$  with missing values and an initial kernel matrix  $K^0 \in \mathbb{R}^{n \times n}$  estimated from  $X$ . Inspired by our previous work [31, 47, 48], our goal is to correct the initial kernel  $K^0$  to an improved estimate  $\hat{K}$  that satisfies the PSD property based on the Lemma 1 [49].
Lemma 1. A valid kernel matrix is a  $n \times n$  real symmetric matrix that satisfies the PSD property.
Naturally, we recover the PSD property with the minimum cost by solving the following model:
$$
\underset {K \in \mathbb {R} ^ {n \times n}} {\text {m i n i m i z e}} \| K - K ^ {0} \| _ {F} ^ {2}, \text {s u b j e c t t o} K \succeq 0, k _ {i j} = k _ {j i} \in [ l, u ], \forall 1 \leq i, j \leq n, \tag {5}
$$
where  $l, u$  denote the lower bound and upper bound, respectively.
It is worth noting that the solution  $\hat{K}$  to Eq. (5) provides an improved estimate of the unknown ground-truth  $K^{*}$  compared to  $K^0$ , as illustrated in Theorem 1. The proof comes from Kolmogorov's criterion [50, 31, 47, 48], which characterizes the best estimation in an inner product space, provided in Appendix A.
Theorem 1.  $\| K^{*} - \hat{K}\|_{F}\leq \| K^{*} - K^{0}\|_{F}$  . The equality holds if and only if  $K^0\succeq 0$  , i.e.,  $K^0 = \hat{K}$
Regarding the kernel type, Gaussian kernel is a widely-used non-linear kernel that has elements in the range  $[0,1]$ . In this case, the feasible region in Eq. (5) is defined as  $\mathcal{F} = \{K\in \mathbb{R}^{n\times n}\mid K\succeq 0,k_{ii} = 1,k_{ij} = k_{ji}\in [0,1],\forall 1\leq i,j\leq n\}$ , which is a closed convex set. The solution to Eq. (5) is the projection of  $K^0$  onto  $\mathcal{F}$ , denoted by  $\hat{K}$ . However, finding the direct projection is complex, and no closed form of  $\hat{K}$  exists. Thus, we break down  $\mathcal{F}$  into two simpler, closed convex subsets  $\mathcal{F}_1$  and  $\mathcal{F}_2$ :  $\mathcal{F}_1 = \{K\in \mathbb{R}^{n\times n}\mid K\succeq 0\}$ ,  $\mathcal{F}_2 = \{K\in \mathbb{R}^{n\times n}\mid k_{ii} = 1,k_{ij} = k_{ji}\in [0,1],\forall 1\leq i,j\leq n\}$ , with  $\mathcal{F} = \mathcal{F}_1\cap \mathcal{F}_2$ . Then  $\hat{K}$  can be efficiently solved by iteratively projecting  $K^0$  onto  $\mathcal{F}_1$  and  $\mathcal{F}_2$  [31, 40, 47]. Denote  $\mathcal{P}_1,\mathcal{P}_2$  as the projection onto  $\mathcal{F}_1,\mathcal{F}_2$ , respectively, in the form of
$$
\left\{ \begin{array}{l} \mathcal {P} _ {1} (K) = U \hat {\Sigma} U ^ {\top} \text {w h e r e} K = U \Sigma U ^ {\top}, \hat {\Sigma} _ {i j} = \max  \left\{\Sigma_ {i j}, 0 \right\}, \\ \mathcal {P} _ {2} (K) = \left[ \mathcal {P} _ {2} \left(k _ {i j}\right) \right] \text {w h e r e} \mathcal {P} _ {2} \left(k _ {i j}\right) = \operatorname {m e d i a n} \left\{0, k _ {i j}, 1 \right\}, \mathcal {P} _ {2} \left(k _ {i i}\right) = 1, \end{array} \right. \tag {6}
$$
where  $U\Sigma U^{\top}$  gives the spectral decomposition (SD) of  $K$ .
We use Dykstra's projection algorithm [51] to find the optimal projection, summarized in Algorithm 1:
Algorithm 1 Kernel Correction (KC)
Input:  $K^0\in \mathbb{R}^{n\times n}$  : an initial (non-PSD) kernel matrix;  $\mathcal{P}_1,\mathcal{P}_2$  : the projection onto  $\mathcal{F}_1$  and  $\mathcal{F}_2$  maxiter: maximum iterations (default 100); tol: tolerance (default  $10^{-5}$    
Output:  $\hat{K}\in \mathbb{R}^{n\times n}$  : the optimal corrected kernel matrix.   
1: Dykstra's Projection: Y, P, Q are auxiliary variables.   
2: Initialize  $X_0 = K^0$  and  $P_{0} = Q_{0} = \mathbf{0}_{n\times n}$    
3: for  $t = 0,1,\ldots ,maxiter$  do   
4:  $Y_{t} = \mathcal{P}_{2}(X_{t} + P_{t})$    
5:  $P_{t + 1} = X_t + P_t - Y_t$    
6:  $X_{t + 1} = \mathcal{P}_1(Y_t + Q_t)$    
7:  $Q_{t + 1} = Y_t + Q_t - X_{t + 1}$    
8: if  $\| X_{t + 1} - X_t\| _F <   tol$  then   
9: break   
10: end if   
11: end for   
12: Set  $\hat{K} = X_{t}$
The convergence guarantee of Algorithm 1 relies on the Boyle-Dykstra's result [52]:
Lemma 2. Given a real symmetric matrix  $K^0 \in \mathbb{R}^{n \times n}$ , the sequence  $\{X_t\}$  generated in Algorithm 1 converges to  $\hat{K} = \min_{K \in \mathcal{F} = \mathcal{F}_1 \cap \mathcal{F}_2} \| K - K^0 \|_F^2$  as  $t \to \infty$ .
3.3 Limitation and Complexity Analysis
The potential limitation primarily stems from the time complexity of Algorithm 1. The pre-iteration time complexity of the KC algorithm is currently at  $O(n^{3})$ , which mainly arises from the spectral decomposition (SD) in the projection operation  $\mathcal{P}_1$  and poses challenges when dealing with large-scale datasets. To address this issue, a possible solution is to replace the spectral decomposition with a randomized singular value decomposition (rSVD) [53]. The rSVD approach seeks top- $k$  singular values and effectively reduces the time complexity to  $O\left(n^{2} \cdot \log(k) + 2n \cdot k^{2}\right)$ . However, the trade-off between efficiency and efficacy necessitates further investigation. Besides, we can transform Dykstra's projection into a parallel version with cyclic projection [54] to achieve better scalability. The storage complexity is  $O(n^{2})$  to store the dense kernel matrix in memory.
3.4 Discussion on Kernel Correction
The proposed kernel correction approach provides an imputation-free method with several benefits to spectral clustering, compared to data imputation and distance calibration methods.
Differences with Previous Kernel Methods Some literature in kernel learning [55, 56] has studied missing data in a supervised manner. However, our work focuses on unsupervised learning and addresses complete but inaccurate (noisy) kernels due to the presence of incomplete observations, which is fundamentally different from previous work [57] primarily dealing with incomplete kernels.
Advantages over Data Imputation Our approach eliminates the need for domain knowledge in handling incomplete data by bypassing the imputation step. This enables us to generate a kernel matrix that is theoretically guaranteed, offering potential advantages for various kernel-based applications. Notably, our approach demonstrates significant improvements over imputation methods for spectral clustering when dealing with a high proportion of missing data. Additionally, the high-quality kernel produced by our approach can serve as a valuable reference for improving the accuracy of missing value estimation in kernel-based imputation methods [58, 59], which is worth further investigation.