Title: Boosting Spectral Clustering on Incomplete Data via Kernel Correction and Affinity Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spectral Clustering

Incomplete Data

Kernel Correction

Self-expressive Affinity Learning
Abstract: Spectral clustering has gained popularity for clustering non-convex data due to its simplicity and effectiveness. It is essential to construct a similarity graph using a high-quality affinity measure that models the local neighborhood relations among the data samples. However, incomplete data can lead to inaccurate affinity measures, resulting in degraded clustering performance. To address these issues, we propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. Firstly, we introduce a new kernel correction method that enhances the quality of the kernel matrix estimated on incomplete data with a theoretical guarantee, benefiting classical spectral clustering on pre-defined kernels. Secondly, we develop a series of affinity learning methods that equip the self-expressive framework with $\ell_p$-norm to construct an intrinsic affinity matrix with an adaptive extension. Our methods outperform existing data imputation and distance calibration techniques on benchmark datasets, offering a promising solution to spectral clustering on incomplete data in various real-world applications.

Boosting Spectral Clustering on Incomplete Data via Kernel Correction and Affinity Learning

Abstract

Spectral clustering has gained popularity for clustering non-convex data due to its simplicity and effectiveness. It is essential to construct a similarity graph using a high-quality affinity measure that models the local neighborhood relations among the data samples. However, incomplete data can lead to inaccurate affinity measures, resulting in degraded clustering performance. To address these issues, we propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. Firstly, we introduce a new kernel correction method that enhances the quality of the kernel matrix estimated on incomplete data with a theoretical guarantee, benefiting classical spectral clustering on pre-defined kernels. Secondly, we develop a series of affinity learning methods that equip the selfexpressive framework with ℓ p -norm to construct an intrinsic affinity matrix with an adaptive extension. Our methods outperform existing data imputation and distance calibration techniques on benchmark datasets, offering a promising solution to spectral clustering on incomplete data in various real-world applications.

1 Introduction

Spectral clustering [1, 2, 3] has become a widely used and effective method for clustering non-convex data and finds diverse applications in computer vision [4, 5], natural language processing [6, 7], and bioinformatics [8, 9]. Generally, the first step in spectral clustering involves constructing an affinity matrix that captures the similarity between data points, followed by performing normalized cut [4] on the corresponding graph to partition the data into clusters. The quality of the affinity matrix is a critical factor that determines the effectiveness of the clustering performance [2, 3]. However, incomplete data is commonly seen in practice, leading to inaccurate affinities and degraded clustering performance [10, 11, 12]. As such, obtaining a high-quality affinity matrix with missing data is a challenging task that requires specialized techniques. In this paper, we aim to improve the quality of affinity matrices, which naturally enhance the performance of spectral clustering on incomplete data.

In recent years, the development of methods to construct affinity matrices with full information for spectral clustering has garnered significant attention. Two types of affinity matrices are typically utilized, namely pre-defined similarity matrices [1, 2, 13, 14] and self-expressive affinity matrices [3, 15, 16, 17, 18, 19, 20]. Pre-defined similarity matrices are easy to compute and encompass well-known kernels, including the Gaussian kernel, Exponential kernel, and Polynomial kernel. The Gaussian kernel is the most widely used due to its simplicity and effectiveness [2]. Differently, self-expressive affinity matrices learn the affinity matrix C by representing each data point as a

linear combination of other data points, i.e., min C 1 2 ∥ X -XC ∥ 2 F + λ R ( C ) , where the columns of X ∈ R d × n are data points, R ( C ) denotes a regularization term, and λ is a hyperparameter, as exemplified by Sparse Subspace Clustering (SSC) [15], Low-Rank Representation (LRR) [16], and Least-Squares Representation (LSR) [17, 18]. Moreover, self-expressive methods have been expanded by integrating kernel functions into the framework, such as kernel SSC (KSSC) [19], kernel LRR (KLRR) [20], and kernel LSR (KLSR) [3], through min C 1 2 ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F + λ R ( C ) , where ϕ ( X ) is a mapping function and K := ϕ ( X ) ⊤ ϕ ( X ) denotes a kernel matrix. Despite the varying techniques on affinity matrices, the kernel matrix remains a cornerstone of spectral clustering for pre-defined similarities and self-expressive affinities. Nevertheless, constructing a high-quality kernel/affinity matrix is an ongoing challenge when facing incomplete data. The primary obstacle is the inability to directly calculate a kernel matrix or learn an affinity matrix due to missing values.

To address the challenge of incomplete data, data imputation techniques [21, 22, 23] such as statistical imputation and matrix completion, are conventional approaches to fill in the missing entries of the data matrix before clustering. Statistical imputation methods [24, 25], such as zero, mean, and k -nearest neighbors ( k NN) imputation [26], are fast and easy to implement and can flexibly handle missing data. However, these methods may introduce bias and fail to capture the true underlying data structure, hindering the accurate affinity measures. Matrix completion methods [27, 28, 29, 30], on the other hand, aim to accurately recover the underlying structure of the data by finding a low-rank or high-rank approximation of the data matrix. Although they can handle missing data in a principled way, their performance heavily depends on the consistency of data structure and the assumptions being made, easily affected by the data distribution. Most importantly, almost all imputation methods do not offer a guarantee for the quality of the affinity matrix calculated on the imputed data [31].

Distance calibration techniques [32, 33] have emerged as an alternative solution by obtaining a high-quality distance matrix for incomplete data. These techniques calibrate an initial non-metric distance matrix estimated on incomplete data to a distance metric. Several methods have been proposed, such as the metric nearness model [34, 35, 36], which finds the nearest approximation that satisfies all triangle inequalities [37]. The double-centering algorithm [38] converts the initial non-metric distance matrix into a Euclidean distance matrix, while the Euclidean embedding method [39, 40, 41] ensures the Euclidean embeddable property is satisfied through convex optimization. Although some of these methods guarantee the quality of the calibrated distance [39], they only apply to distance-based affinities like the Gaussian kernel. Moreover, these methods may not generate high-quality affinity matrices based on calibrated distance, making it necessary to develop dedicated methods that can apply to a family of kernels and affinities with theoretical guarantees.

To address the above issues, we propose an imputation-free framework to directly learn high-quality affinity matrices via two main techniques, i.e., kernel correction and affinity learning, aiming to improve the spectral clustering performance on incomplete data, with contributions as follows.

We propose an imputation-free framework based on kernel correction and affinity learning with improved clustering performance, providing convenient tools to deal with incomplete data. To our best knowledge, this is the first systematical work to discuss spectral clustering on incomplete data, which is a commonly seen problem in practice.

We introduce a novel kernel correction method that directly focuses on the kernel matrix estimated from the incomplete data, and corrects it to satisfy specific mathematical properties such as positive semi-definiteness (PSD). We show that the corrected kernel matrix becomes closer to the unknown ground-truth with a theoretical guarantee, which is beneficial to spectral clustering algorithms and cannot be assured by imputation and calibration methods.

We develop a series of new affinity learning methods to equip the self-expressive framework with the ℓ p -norm to capture the underlying structure of data samples better. Additionally, we combine kernel correction and affinity learning to arrive at an adaptive learning method that simultaneously learns the high-quality kernel and the self-expressive affinity matrix.

We conduct extensive experiments that demonstrate the effectiveness of proposed methods on various benchmark datasets, showing superior results in terms of kernel estimation, spectral clustering, and affinity learning on incomplete data, compared to existing data imputation and distance calibration approaches.

2 Related Work

2.1 Spectral Clustering and Affinity Learning

Standard Spectral Clustering involves three steps [2]: first, a Gaussian kernel matrix K ∈ R n × n is calculated to measure the similarity between n data points with K ij = exp( -∥ x i -x j ∥ 2 /σ 2 ) where σ is a hyperparameter. Then, an affinity graph A ∈ R n × n is constructed using the kernel matrix, which can take the form of an ϵ -neighborhood graph, a fully connected graph, or a k -nearest neighbors ( k NN) graph. An ϵ -neighborhood graph connects pairwise points with a threshold value ϵ , while a fully connected graph connects all points. Empirically, a k NN graph is the most popular one that connects each point to its k -nearest neighbors, resulting in sparse local relationships and relatively high clustering accuracy [3]. Finally, the normalized cut algorithm [1] is applied to the affinity graph A to partition the data into clusters based on the normalized Laplacian matrix.

Self-expressive Affinity Learning is a framework to learn affinity matrices by modeling the relationships between data points, i.e., min C 1 2 ∥ X -XC ∥ 2 F + λ R ( C ) , with different types of regularization terms R ( C ) . Sparse Subspace Clustering (SSC) [15] assumes that the data points lie in a lowdimensional subspace and seeks to find a sparse affinity matrix with R ( C ) = ∥ C ∥ 1 = ∑ n i,j =1 | c ij | under a constraint diag(C) = 0 . Low-Rank Representation (LRR) [16] is another approach that seeks a low-rank affinity matrix with R ( C ) = ∥ C ∥ ∗ (nuclear norm of C ). Least-Squares Representation (LSR) [17, 18] involves solving a least-squares problem to find the representation matrix with R ( C ) = 1 2 ∥ C ∥ 2 F . Their kernel variants, i.e., KSSC [19], KLRR [20], and KLSR [3] are used to extend their applicability to non-linearly separable data by applying a kernel function to data points with min C 1 2 ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F + λ R ( C ) , showing promising performance in spectral clustering.

2.2 Missing Data Processing Techniques

Data Imputation is a popular technique [21, 22, 23] for dealing with incomplete data by filling in missing values. Statistical imputation methods, such as zero, mean imputation, and k -nearest neighbors ( k NN) approach [26], have been widely used in practice. These methods replace the missing value with a zero, mean, or k -weighted value of non-missing elements in the corresponding feature. Additionally, matrix completion [27, 28, 29, 30] is a machine learning-based technique that fills missing values by solving a matrix factorization problem under assumptions on data structures such as low-rank or high-rank. However, correctly estimating missing values based on observed data is difficult, especially for a large missing ratio, and there is no guarantee on the quality of the affinity matrix on imputed data. This motivates us to design imputation-free approaches in Sections 3 and 4.

Distance Calibration is a specialized approach to obtaining a valid distance metric from an initial non-metric distance matrix, which can be applied to incomplete data. For any two incomplete data samples x i , x j ∈ R d , a new vector x i ( I ) ∈ R | I | is formed by selecting the observed values of x i on the set I , where I is an index set of all features observed in both samples. The pairwise Euclidean distance between x i and x j can then be heuristically estimated by [39, 41]

However, the initial Euclidean distance matrix D 0 = [ d 0 ij ] ∈ R n × n estimated on incomplete data is usually not a distance metric due to missing values. Distance calibration methods [34, 35, 38, 39] can correct D 0 to a distance metric by making different assumptions and leveraging various properties. More details of these methods are discussed in Section 3.1.

3 Methodology-I. Kernel Correction

3.1 Revisiting Distance Calibration Methods

We begin with the definition of a distance metric [42], then delve into distance calibration methods, discussing the assumptions and properties underlying each method, as well as their limitations.

Definition 1. A distance metric is defined as a n × n real symmetric matrix D that satisfies

The properties of non-negativity, reflexivity, symmetry, and triangle inequality ensure that the distance metric produces meaningful and valuable distance measurements.

The Triangle Fixing (TRF) Algorithm [35] to obtain a distance metric is based on the metric nearness model [34, 36], which finds the nearest approximation of a non-metric D 0 by solving:

However, there are two significant limitations. First, the scalability of the algorithm is severely limited by the matrix size n . Since the number of O ( n 3 ) constraints grows rapidly as n increases, the optimization time can become lengthy, taking several hours for a few thousand samples. Second, the performance of the algorithm depends on the extent of violation of the triangle inequalities in the initial distance matrix D 0 . With a small missing ratio of features, D 0 already satisfies most triangle inequalities, i.e., d 0 ik ≤ d 0 ij + d 0 jk , then the algorithm typically yields only marginal improvement.

The Double-Centering (DC) Algorithm [38] starts with the Euclidean Distance Matrix (EDM) [43] and utilizes Property 1 [44], a well-known result in classical Multi-Dimensional Scaling (cMDS) [45], to build the connection between the similarity matrix and the EDM.

Definition 2. A n × n real symmetric matrix D is called an EDM if there exists x 1 , x 2 , . . . , x n ∈ R d such that d ij = ∥ x i -x j ∥ 2 for i, j = 1 , 2 , . . . , n . (Note that it uses squared Euclidean distance.)

Property 1. A n × n real symmetric matrix D is EDM if and only if diag( D ) = 0 , S := -1 2 JDJ ⪰ 0 , J := I -ee ⊤ /n, where I is the identity matrix and e is the n -dimension vector of all ones.

The DC algorithm first finds the nearest positive semi-definite ˆ S by solving

where D 0 is the initial Euclidean distance matrix and ◦ denotes the Hadamard product. The algorithm then transforms ˆ S into an EDM ˆ D using ˆ d ij = ˆ s ii + ˆ s jj -2ˆ s ij , and obtain the calibrated (nonsquared) distance matrix by ˆ D ← [ ˆ d 1 / 2 ij ] , but unfortunately, the quality of ˆ D cannot be guaranteed, and important information may be lost during the transformation [38].

The Euclidean Embedding (EE) Algorithm [39, 40, 41] leverages Euclidean embeddable property [39, 46] to obtain a calibrated embeddable distance matrix by solving

Definition 3. A n × n real symmetric matrix D is said Euclidean embeddable if there exists x 1 , . . . , x n in Euclidean space and a distance function ρ such that d ij = ρ ( x i , x j ) ≥ 0 for i, j = 1 , . . . , n .

Property 2. [Theorem 2, [39]] If a n × n real symmetric matrix D is Euclidean embeddable, then the kernel K := exp( -γD ) is positive semi-definite for any γ > 0 .

It is important to note that these methods can only provide a calibrated distance matrix with benefits for distance-based kernels and are not a universal solution for dealing with incomplete data in spectral clustering tasks, which motivates us to further design a kernel-specialized method in Section 3.2.

3.2 Kernel Correction Algorithm

To overcome the limitations of distance calibration, we propose a new kernel correction algorithm that directly focuses on the construction of a high-quality kernel with a theoretical guarantee. We consider an incomplete data matrix X ∈ R d × n with missing values and an initial kernel matrix K 0 ∈ R n × n estimated from X . Inspired by our previous work [31, 47, 48], our goal is to correct the initial kernel K 0 to an improved estimate ˆ K that satisfies the PSD property based on the Lemma 1 [49].

Lemma 1. A valid kernel matrix is a n × n real symmetric matrix that satisfies the PSD property.

Naturally, we recover the PSD property with the minimum cost by solving the following model:

where l, u denote the lower bound and upper bound, respectively.

It is worth noting that the solution ˆ K to Eq. (5) provides an improved estimate of the unknown ground-truth K ∗ compared to K 0 , as illustrated in Theorem 1. The proof comes from Kolmogorov's criterion [50, 31, 47, 48], which characterizes the best estimation in an inner product space, provided in Appendix A.

Regarding the kernel type, Gaussian kernel is a widely-used non-linear kernel that has elements in the range [0 , 1] . In this case, the feasible region in Eq. (5) is defined as F = { K ∈ R n × n | K ⪰ 0 , k ii = 1 , k ij = k ji ∈ [0 , 1] , ∀ 1 ≤ i, j ≤ n } , which is a closed convex set. The solution to Eq. (5) is the projection of K 0 onto F , denoted by ˆ K . However, finding the direct projection is complex, and no closed form of ˆ K exists. Thus, we break down F into two simpler, closed convex subsets F 1 and F 2 : F 1 = { K ∈ R n × n | K ⪰ 0 } , F 2 = { K ∈ R n × n | k ii = 1 , k ij = k ji ∈ [0 , 1] , ∀ 1 ≤ i, j ≤ n } , with F = F 1 ∩ F 2 . Then ˆ K can be efficiently solved by iteratively projecting K 0 onto F 1 and F 2 [31, 40, 47]. Denote P 1 , P 2 as the projection onto F 1 , F 2 , respectively, in the form of

where U Σ U ⊤ gives the spectral decomposition (SD) of K .

We use Dykstra's projection algorithm [51] to find the optimal projection, summarized in Algorithm 1:

Algorithm 1 K ernel C orrection ( KC )

Input: K ∈ R : an initial (non-PSD) kernel matrix; P 1 , P 2 : the projection onto F 1 and F 2 ; maxiter : maximum iterations (default 100 ); tol : tolerence (default 10 -5 ).

0 n × n Output: ˆ K ∈ R n × n : the optimal corrected kernel matrix. 1: ▷ Dykstra's Projection: Y, P, Q are auxiliary variables. 2: Initialize X 0 = K 0 and P 0 = Q 0 = 0 n × n . 3: for t = 0 , 1 , . . . , maxiter do 4: Y t = P 2 ( X t + P t ) , 5: P t +1 = X t + P t -Y t , 6: X t +1 = P 1 ( Y t + Q t ) , 7: Q t +1 = Y t + Q t -X t +1 . 8: if ∥ X t +1 -X t ∥ F < tol then 9: break 10: end if 11: end for 12: Set ˆ K = X t .

The convergence guarantee of Algorithm 1 relies on the Boyle-Dykstra's result [52]:

Lemma 2. Given a real symmetric matrix K 0 ∈ R n × n , the sequence { X t } generated in Algorithm 1 converges to ˆ K = min K ∈F = F 1 ∩F 2 ∥ K -K 0 ∥ 2 F as t →∞ .

3.3 Limitation and Complexity Analysis

The potential limitation primarily stems from the time complexity of Algorithm 1. The pre-iteration time complexity of the KC algorithm is currently at O ( n 3 ) , which mainly arises from the spectral decomposition (SD) in the projection operation P 1 and poses challenges when dealing with largescale datasets. To address this issue, a possible solution is to replace the spectral decomposition with a randomized singular value decomposition (rSVD) [53]. The rSVD approach seeks topk singular values and effectively reduces the time complexity to O ( n 2 · log( k ) + 2 n · k 2 ) . However, the trade-off between efficiency and efficacy necessitates further investigation. Besides, we can transform Dykstra's projection into a parallel version with cyclic projection [54] to achieve better scalability. The storage complexity is O ( n 2 ) to store the dense kernel matrix in memory.

3.4 Discussion on Kernel Correction

The proposed kernel correction approach provides an imputation-free method with several benefits to spectral clustering, compared to data imputation and distance calibration methods.

Differences with Previous Kernel Methods Some literature in kernel learning [55, 56] has studied missing data in a supervised manner. However, our work focuses on unsupervised learning and addresses complete but inaccurate (noisy) kernels due to the presence of incomplete observations, which is fundamentally different from previous work [57] primarily dealing with incomplete kernels.

Advantages over Data Imputation Our approach eliminates the need for domain knowledge in handling incomplete data by bypassing the imputation step. This enables us to generate a kernel matrix that is theoretically guaranteed, offering potential advantages for various kernel-based applications. Notably, our approach demonstrates significant improvements over imputation methods for spectral clustering when dealing with a high proportion of missing data. Additionally, the high-quality kernel produced by our approach can serve as a valuable reference for improving the accuracy of missing value estimation in kernel-based imputation methods [58, 59], which is worth further investigation.

Advantages over Distance Calibration Firstly, our approach can be applied to a wide range of kernels and yields an improved kernel matrix. In contrast, distance calibration methods only benefit distance-based kernels and lack quality guarantees. Moreover, our algorithm, which corrects the Gaussian kernel, can generate a high-quality distance matrix via d ij = √ -σ 2 log( k ij ) . By incorporating the second-order term of d 0 ij in k 0 ij = exp( -( d 0 ij ) 2 /σ 2 ) , our algorithm becomes more sensitive to changes in distance values, while the Euclidean embedding algorithm relies on K 0 = exp( -γD 0 ) using the first-order term of d 0 ij . Empirical evidence suggests that the corrected distance obtained from our corrected Gaussian kernel is more accurate than the calibrated distance derived from the Euclidean embedding method, benefiting to the distance-based spectral clustering.

Benefits to Spectral Clustering Our approach is tailored to improve the kernel quality, making it highly advantageous for spectral clustering. This improvement benefits both standard spectral clustering using the Gaussian kernel and affinity learning based on the self-expressive framework. In the case of X -based self-expressive affinity, we can apply the corrected linear kernel K := X ⊤ X to the optimization, i.e., min C 1 2 ∥ X -XC ∥ 2 F + λ R ( C ) = min C 1 2 Tr( K -2 KC + C ⊤ KC )+ λ R ( C ) . Similarly, for K -based self-expressive affinity, which involves using the Gaussian kernel or other kernels, our approach can be applied to K := ϕ ( X ) ⊤ ϕ ( X ) on min C 1 2 ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F + λ R ( C ) .

4 Methodology-II. Affinity Learning

Taking advantage of the corrected kernel matrix, we have further designed a series of kernel-based affinity learning algorithms to acquire high-quality affinity matrices with an adaptive extension.

4.1 Kernel Self-expressive Learning with ℓ p -norm

We utilize the kernel self-expressive framework, i.e., min C 1 2 ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F + λ R ( C ) , and propose new affinity learning methods by enhancing the sparsity of the affinity matrix C using the ℓ p norm [60]. Specifically, our approach diverges from previous work [19, 20, 3] by incorporating two forms of ℓ p -norm, i.e., proximal p -norm and Schatten p -norm ( 0 < p < 1 ), as regularization terms.

where the proximal p -norm is expressed as ∥ C ∥ p p := ∑ n i,j =1 | c ij | p .

We construct an augmented Lagrangian function L p ( C, Z, U ) in Eq. (8), and solve it by using the Alternating Direction Method of Multipliers (ADMM) approach [61] summarized in Algorithm 2.

However, the KSL-Pp algorithm involves numerous hyper-parameters and entails a non-convex optimization process during the Z -update step (Line 5 in Algorithm 2), making it difficult to effectively utilize. Computation speedup needs further investigations. Thus, we focus on the Schatten p -norm.

Algorithm 2 K ernel S elf-expressive L earning with P roximal p -norm ( KSL-Pp )

Input: K : a kernel; λ, p, ρ, γ, α : hyperparameters; maxiter : maximum iterations; tol : tolerance. Output: ˆ C ∈ R n × n : the optimal affinity matrix of Eq. (7).

1: Initialize C , Z , U

0 0 0 . ▷ Refer to Appendix B.1 for formula derivations.

2: for t = 0 , 1 , . . . , maxiter do

3: C -update: C t +1 ← (2 K + 1 ρ I ) -1 (2 K + U t + 1 ρ Z t ) ;

4: Z -update: Z t +1 ← ∂ L p ∂z ij = λpz p -1 ij + γ I ( z ij > 1) + 1 ρ ( z ij -c ij ) + u ij = 0 ;

5: U -update: U t +1 ← U t + ρ ( Z t +1 -C t +1 ) .

6: if (primal) ∥ C t +1 -z t +1 ∥ F ≤ tol and (dual) ∥ Z t +1 -Z t ∥ F ≤ tol then ; break ; end if .

7: end for

8: Set ˆ C = ( | C t | + | C ⊤ t | ) / 2 .

Definition 5 ( Kernel Self-expressive Learning with Schatten p -norm ) .

where Schatten p -norm ∥ C ∥ S p := ( ∑ n i =1 σ p i ( C )) 1 /p and σ i ( C ) denotes the i -th singular value of C .

Drawing from prior research [62], for 1 2 < p < 1 and k ≥ rank ( C ) , the following always holds true:

Thus, for 1 2 < p < 1 , we define L S p ( U, V ) = ∥ ϕ ( X ) -ϕ ( X ) UV ⊤ ∥ 2 F + λ 2 ∥ U ∥ 2 F + λ 2 ∥ V ∥ 2 F as a relaxation of the optimization problem in Eq. (9) without loss of generality. By employing the ADMM approach [61] and the gradient descent method [63] in the U -update step (i.e., Line 4 in Algorithm 3), we design the KSL-Sp algorithm as follows with details in the Appendix B.2.

Algorithm 3 K ernel S elf-expressive L earning with S chatten p -norm ( KSL-Sp )

Input: K : a kernel matrix; λ : a hyperparameter; maxiter : maximum iterations; tol : tolerance. Output: ˆ C ∈ R n × n : the optimal affinity matrix of Eq. (9).

1:

Initialize U 0 , V 0 . ▷ Refer to Appendix B.2 for formula derivations.

2: for t = 0 , 1 , . . . , maxiter do

3: U -update: U t +1 ← ∂ U L S p = 2 K ( UV ⊤ t -I ) V t + λU = 0 ;

4: V -update: V t +1 ← 2 KU t +1 (2 U ⊤ t +1 KU t +1 + λI ) -1 by ∂ V L S p = 0 .

5: if ∥ U t +1 -U t ∥ F < tol then ; break ; end if .

6: end for

7: Set C t = U t V ⊤ t and ˆ C = ( | C t | + | C ⊤ t | ) / 2 .

4.2 Adaptive Kernel Self-expressive Learning

To achieve adaptive affinity learning on incomplete data, we formulate a new joint optimization problem by incorporating KLSR [3] and kernel correction to learn kernel and affinity matrices iteratively with the PSD constraint of K :

Firstly, we introduce an auxiliary variable A ∈ R n × n into

and then we derive the augmented Lagrange function L ( K,A,C,U ) in preparation for ADMM:

where U is a Lagrange multiplier and ρ is the updating step size, finally arriving at the Algorithm 4.

Our adaptive learning framework could be combined with other kernel self-expressive learning algorithms; however, due to space constraints, further research is required to explore these possibilities:

Algorithm 4 A daptive K ernel L eastS quares R epresentation ( AKLSR )

Input: K 0 : an initial kernel; λ, ρ : hyperparameters; maxiter : maximum iterations; tol : tolerance. Output: ˆ C ∈ R n × n : the optimal affinity matrix of Eq. (10).

1: Initialize K , A , C , U

0 0 0 0 . ▷ Refer to Appendix B.3 for formula derivations.

2: for t = 0 , 1 , . . . , maxiter do

3: K -update: K ← 1 ρ +2 (2 K 0 -U t + ρA t ) by ∂ L ∂K = 0 ;

4: K -PSD: K t +1 ←P 1 ( K ) by projecting K onto the PSD set via Eq. (6);

5: A -update: A t +1 ← 1 ρ ( ρK t +1 + U t +2 C ⊤ t -C t C ⊤ t -I ) by ∂ L ∂A = 0 ;

6: C -update: C t +1 ← 2(2 λI + A t +1 + A ⊤ t +1 ) -1 A ⊤ t +1 by ∂ L ∂C = 0 ;

7: U -update: U t +1 ← U t + ρ ( K t +1 -A t +1 ) .

8: if ∥ C t +1 -C t ∥ F < tol then ; break ; end if .

9: end for

10: Set ˆ C = ( | C t | + | C ⊤ t | ) / 2 .

5 Experiments

We evaluate the performance on four benchmark datasets, including two face image datasets Yale64 [64] and Umist [65], a handwritten digit image dataset USPS [66], and a speech dataset Isolet [67]. We use a subset of USPS with 1,000 randomly selected samples. All experiments 2 are conducted five times in MATLAB on a ThinkStation with a 2.1 GHz Intel i7-12700 Core and 32GB RAM.

Various methods dealing with incomplete data are considered for comparison: 1) statistical imputation: ZERO , MEAN , k -nearest neighbors ( k NN ) [26], Expectation Maximization ( EM ) [68]; 2) matrix completion: Singular Value Thresholding ( SVT ) [27], Grassmanian Rank-one Update Subspace Estimation ( GR ) [28], Kernelized Factorization Matrix Completion ( KFMC ) [29]; 3) distance calibration: Double-Centering ( DC ) [38], Triangle Fixing ( TRF ) [35], Euclidean Embedding ( EE ) [39]. Our KC method uses the spectral decomposition (SD). Details are provided in Appendix C.

5.1 Validation of Kernel Correction on Gaussian Kernel and Euclidean Distance

When handling incomplete data that is missing completely at random, the quality of the estimated Gaussian kernel matrix and Euclidean distance matrix is measured using the relative-mean-square error ( RMSE = ∥ ˆ A -A ∗ ∥ 2 F ∥ A 0 -A ∗ ∥ 2 F ) and the relative error ( RE = ∥ ˆ A -A ∗ ∥ F ∥ A ∗ ∥ F ), where A 0 is the naive kernel/distance matrix obtained by Eq. (1), ˆ A is the estimated one using different methods, and A ∗ as the ground-truth. All Gaussian kernels are calculated by k ij = exp( -d 2 ij /σ 2 ) , where σ = median { d ij } . Our KC algorithm yields Gaussian kernel and Euclidean distance matrices with the lowest RMSEs and REs, validating the theoretical guarantee of kernel matrices in Theorem 1, as demonstrated in Table 1. Furthermore, the accuracy of the top-10 nearest neighbors is evaluated using Recall values [69]. The KC algorithm achieves the highest Recalls with improved local relationships, which in turn benefits spectral clustering, particularly for k NN graphs. More numerical results are in Appendix D.

5.2 Comparative Studies on Standard Spectral Clustering

We adopt the Gaussian kernel for standard spectral clustering and obtain its k NN graph ( k = 10 ), as the input for clustering via the normalized cut algorithm [1]. We evaluate the clustering performance using Accuracy ( ACC ), Normalized Mutual Information ( NMI ), Purity ( PUR ), and Adjusted Rand Index ( ARI ). Table 2 shows that our KC algorithm, which corrects the Gaussian kernel, achieves the best clustering performance under a large missing ratio 80% (i.e., missing completely at random).

We also examine robustness across varying missing ratios. Fig. 1 shows the KC method's advantage over imputation, especially at large missingness. Numerical results are in Appendix E.

The KC method consistently surpasses others in kernel estimation, evidenced by reduced relative errors (RE-K) and enhanced neighborhood ties in the k NN graph, seen through higher Recall values.

At small missing ratios, the KC method offers incremental improvements over imputation. However, at large missing ratios, imputation methods falter due to scant observed data, leading to increased errors and compromised clustering. Conversely, the KC method maintains stable clustering outcomes.

ZERO

MEAN

EM

SVT

GR

KFMC

KC

20%

40%

60%

80%

Missing Ratio r

0.0

0.1

0.2

0.3

0.4

RE-K

Yale64

20%

40%

60%

80%

Missing Ratio r

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Yale64

20%

40%

60%

80%

Missing Ratio r

0.0

0.1

0.2

0.3

0.4

ARI

Yale64

20%

40%

60%

80%

Missing Ratio r

0.0

0.1

0.2

0.3

0.4

RE-K

Umist

20%

40%

60%

80%

Missing Ratio r

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Umist

20%

40%

60%

80%

Missing Ratio r

0.0

0.1

0.2

0.3

0.4

0.5

ARI

Umist

5.3 Comparative Studies on Self-expressive Affinity Learning

We delve deeper into the advantages of the corrected Gaussian kernel in kernel self-expressive affinity learning, encompassing methods like KSSC [19], KLSR [3], KSL-Sp, and AKLSR, with KSL-Pp excluded due to its impracticality. As Table 3 illustrates, our KC method excels over baselines in the NMI. Notably, both KSL-Sp and AKLSR algorithms employ corrected kernels to yield dependable affinity matrices, thereby elevating performance. Furthermore, our imputation-free framework could augment spectral clustering using other kernels, such as the Exponential kernel. A comprehensive comparison and in-depth analysis are available in Appendix F. Collectively, our study presents a holistic approach to clustering incomplete data, paving the way for subsequent research.

5.4 Spectral Clustering on Block-missing Data: A Case Study

Beyond the experiments addressing data missing completely at random, we generate a block of appropriate sizes located randomly in images and values in the block are missing where the missingness is systematically related to locations. As illustrated in Table 4, under this block-missing paradigm [70], our KC method persistently outperforms existing data imputation and distance calibration techniques.

6 Conclusion and Future Work

In this paper, we propose an imputation-free framework for spectral clustering on incomplete data. Our framework directly learns high-quality affinity matrices via kernel correction and affinity learning, improving clustering performance. We introduce a novel kernel correction method with a theoretical guarantee, new affinity learning methods, and an adaptive extension that simultaneously learns high-quality kernel and affinity matrices. Extensive experiments demonstrate the effectiveness of our proposed methods on various benchmark datasets, showing superior performance compared to existing data imputation and distance calibration approaches. Our work provides a systematic solution to clustering incomplete data, with future work focused on integrating with deep clustering methods for various types of incomplete data and enhancing the speed and scalability of the algorithms.

Acknowledgments

We appreciate the anonymous reviewers for their insightful feedback that greatly enhanced this paper. The work of Fangchen Yu and Wenye Li was supported in part by Guangdong Basic and Applied Basic Research Foundation (2021A1515011825), Guangdong Introducing Innovative and Entrepreneurial Teams Fund (2017ZT07X152), Shenzhen Science and Technology Program (CUHKSZWDZC0004), and Shenzhen Research Institute of Big Data Scholarship Program. The work of Jicong Fan was partially supported by the Youth program 62106211 of the National Natural Science Foundation of China. The work of Yicheng Zeng was supported by the Shenzhen Science and Technology Program (Grant No. RCBS20221008093336086), the Youth Program (Grant No. 12301383) of National Natural Science Foundation of China, and the Internal Project Fund from Shenzhen Research Institute of Big Data (Grant No. J00220230012). The work of Jianfeng Mao was supported in part by the National Natural Science Foundation of China (NSFC) under grant U1733102, in part by the National Natural Science Foundation of China (NSFC) under grant 72394362, in part by the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen) under grant B10120210117, and in part by CUHK-Shenzhen under grant PF.01.000404.

References

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems , 14, 2001.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17:395-416, 2007.

Jicong Fan, Yiheng Tu, Zhao Zhang, Mingbo Zhao, and Haijun Zhang. A simple approach to automated spectral clustering. Advances in Neural Information Processing Systems , 35:99079921, 2022.

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22(8):888-905, 2000.

Stella X Yu and Jianbo Shi. Multiclass spectral clustering. In IEEE International Conference on Computer vision , volume 2, pages 313-313. IEEE Computer Society, 2003.

Víctor Mijangos, Gerardo Sierra, and Azucena Montes. Sentence level matrix representation for document spectral clustering. Pattern Recognition Letters , 85:29-34, 2017.

Tae Jin Park, Kyu J Han, Manoj Kumar, and Shrikanth Narayanan. Auto-tuning spectral clustering for speaker diarization using normalized maximum eigengap. IEEE Signal Processing Letters , 27:381-385, 2019.

Marcilio CP De Souto, Ivan G Costa, Daniel SA de Araujo, Teresa B Ludermir, and Alexander Schliep. Clustering cancer gene expression data: a comparative study. BMC Bioinformatics , 9(1):1-14, 2008.

Seyoung Park and Hongyu Zhao. Spectral clustering based on learning similarity matrix. Bioinformatics , 34(12):2069-2076, 2018.

Nauman Shahid, Vassilis Kalofolias, Xavier Bresson, Michael Bronstein, and Pierre Vandergheynst. Robust principal component analysis on graphs. In Proceedings of the IEEE International Conference on Computer Vision , pages 2812-2820, 2015.

Jicong Fan and Tommy W.S. Chow. Sparse subspace clustering for data with missing entries and high-rank matrix completion. Neural Networks , 93:36-44, 2017.

Jicong Fan. Large-scale subspace clustering via k-factorization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 342-352, 2021.

Jicong Fan and Tommy W. S. Chow. Exactly robust kernel principal component analysis. IEEE Transactions on Neural Networks and Learning Systems , 31(3):749-761, 2020.

Jicong Fan, Chengrun Yang, and Madeleine Udell. Robust non-linear matrix factorization for dictionary learning, denoising, and clustering. IEEE Transactions on Signal Processing , 69:1755-1770, 2021.

Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(11):2765-2781, 2013.

Guangcan Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(1):171-184, 2012.

Can-Yi Lu, Hai Min, Zhong-Qiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and efficient subspace segmentation via least squares regression. In European Conference on Computer Vision , pages 347-360. Springer, 2012.

Pan Ji, Mathieu Salzmann, and Hongdong Li. Efficient dense subspace clustering. In IEEE Winter Conference on Applications of Computer Vision , pages 461-468. IEEE, 2014.

Vishal M Patel and René Vidal. Kernel sparse subspace clustering. In 2014 IEEE International Conference on Image Processing , pages 2849-2853. IEEE, 2014.

Shijie Xiao, Mingkui Tan, Dong Xu, and Zhao Yang Dong. Robust kernel low-rank representation. IEEE Transactions on Neural Networks and Learning Systems , 27(11):2268-2281, 2015.

Bradley Efron. Missing data, imputation, and the bootstrap. Journal of the American Statistical Association , 89(426):463-475, 1994.

Wei-Chao Lin and Chih-Fong Tsai. Missing value imputation: a review and analysis of the literature (2006-2017). Artificial Intelligence Review , 53:1487-1509, 2020.

Tressy Thomas and Enayat Rajabi. A systematic review of machine learning-based missing value imputation techniques. Data Technologies and Applications , 55(4):558-585, 2021.

Susanne Rässler, Donald B Rubin, and Elizabeth R Zell. Imputation. Wiley Interdisciplinary Reviews: Computational Statistics , 5(1):20-29, 2013.

Ton De Waal, Jeroen Pannekoek, and Sander Scholtus. Handbook of Statistical Data Editing and Imputation , volume 563. John Wiley & Sons, 2011.

Lorenzo Beretta and Alessandro Santaniello. Nearest neighbor imputation algorithms: a critical evaluation. BMC Medical Informatics and Decision Making , 16(3):197-208, 2016.

Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization , 20(4):1956-1982, 2010.

Laura Balzano, Robert Nowak, and Benjamin Recht. Online identification and tracking of subspaces from highly incomplete information. In 48th Annual Allerton Conference on Communication, Control, and Computing , pages 704-711. IEEE, 2010.

Jicong Fan and Madeleine Udell. Online high rank matrix completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8690-8698, 2019.

Jicong Fan. Dynamic nonlinear matrix completion for time-varying data imputation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6587-6596, 2022.

Fangchen Yu, Yicheng Zeng, Jianfeng Mao, and Wenye Li. Online estimation of similarity matrices with incomplete data. In Uncertainty in Artificial Intelligence , pages 2454-2464. PMLR, 2023.

Elzbieta Pekalska and Robert PW Duin. The dissimilarity representation for pattern recognition - foundations and applications. Series in Machine Perception and Artificial Intelligence , 64, 2005.

Frank-Michael Schleif and Peter Tino. Indefinite proximity learning: A review. Neural Computation , 27(10):2039-2096, 2015.

Suvrit Sra, Joel Tropp, and Inderjit Dhillon. Triangle fixing algorithms for the metric nearness problem. Advances in Neural Information Processing Systems , 17, 2004.

Justin Brickell, Inderjit S Dhillon, Suvrit Sra, and Joel A Tropp. The metric nearness problem. SIAM Journal on Matrix Analysis and Applications , 30(1):375-396, 2008.

Wenye Li, Fangchen Yu, and Zichen Ma. Metric nearness made practical. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8648-8656, 2023.

Arijit Biswas and David W Jacobs. An efficient algorithm for learning distances that obey the triangle inequality. In British Machine Vision Conference , pages 10-1, 2015.

Andrej Gisbrecht and Frank-Michael Schleif. Metric and non-metric proximity transformations at linear costs. Neurocomputing , 167:643-657, 2015.

Wenye Li and Fangchen Yu. Calibrating distance metrics under uncertainty. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 219-234. Springer, 2022.

Fangchen Yu, Rui Bao, Jianfeng Mao, and Wenye Li. Highly-efficient robinson-foulds distance estimation with matrix correction. In 26th European Conference on Artificial Intelligence , pages 2914-2921, 2023.

Fangchen Yu, Zhan Shi, Yuqi Ma, Jianfeng Mao, and Wenye Li. From incompleteness to unity: Aframework for multi-view clustering with missing values. International Conference on Neural Information Processing , pages 101-112, 2023.

Liu Yang and Rong Jin. Distance metric learning: A comprehensive survey. Michigan State Universiy , 2(2):4, 2006.

Nathan Krislock and Henry Wolkowicz. Euclidean distance matrices and applications. In Handbook on Semidefinite, Conic and Polynomial Optimization , pages 879-914. Springer, 2012.

John Clifford Gower. Euclidean distance geometry. Mathematical Scientist , 7(1):1-14, 1982.

Warren S Torgerson. Multidimensional scaling: I. theory and method. Psychometrika , 17(4):401419, 1952.

Isaac J Schoenberg. Metric spaces and positive definite functions. Transactions of the American Mathematical Society , 44(3):522-536, 1938.

Wenye Li. Estimating Jaccard index with missing observations: a matrix calibration approach. Advances in Neural Information Processing Systems , 28, 2015.

Wenye Li. Scalable calibration of affinity matrices from incomplete observations. In Asian Conference on Machine Learning , pages 753-768. PMLR, 2020.

Rafic Nader, Alain Bretto, Bassam Mourad, and Hassan Abbas. On the positive semi-definite property of similarity matrices. Theoretical Computer Science , 755:13-28, 2019.

Frank R Deutsch. Best Approximation in Inner Product Spaces . Springer Science & Business Media, New York, US, 2012.

Richard L Dykstra. An algorithm for restricted least squares regression. Journal of the American Statistical Association , 78(384):837-842, 1983.

James P Boyle and Richard L Dykstra. A method for finding projections onto the intersection of convex sets in hilbert spaces. In Advances in Order Restricted Statistical Inference , pages 28-47. Springer, 1986.

Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review , 53(2):217-288, 2011.

Norbert Gaffke and Rudolf Mathar. A cyclic projection algorithm via duality. Metrika , 36(1):2954, 1989.

Alex J Smola, SVN Vishwanathan, and Thomas Hofmann. Kernel methods for missing variables. In International Workshop on Artificial Intelligence and Statistics , pages 325-332. PMLR, 2005.

Uwe Dick, Peter Haider, and Tobias Scheffer. Learning from incomplete data with infinite imputations. In Proceedings of the 25th International Conference on Machine Learning , pages 232-239, 2008.

Xinwang Liu, Xinzhong Zhu, Miaomiao Li, Lei Wang, En Zhu, Tongliang Liu, Marius Kloft, Dinggang Shen, Jianping Yin, and Wen Gao. Multiple kernel k -means with incomplete kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(5):1191-1204, 2019.

Shichao Zhang, Zhi Jin, Xiaofeng Zhu, and Jilian Zhang. Missing data analysis: a kernel-based multi-imputation approach. Transactions on Computational Science III , pages 122-142, 2009.

Chiu-Hsieh Hsu, Yulei He, Yisheng Li, Qi Long, and Randall Friese. Doubly robust multiple imputation using kernel-based techniques. Biometrical Journal , 58(3):588-606, 2016.

Xiaojun Chen, Fengmin Xu, and Yinyu Ye. Lower bound theory of nonzero entries in solutions of ℓ 2 -ℓ p minimization. SIAM Journal on Scientific Computing , 32(5):2832-2852, 2010.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Fanhua Shang, Yuanyuan Liu, and James Cheng. Unified scalable equivalent formulations for schatten quasi-norms. arXiv preprint arXiv:1606.00668 , 2016.

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.

Peter N. Belhumeur, Joao P Hespanha, and David J. Kriegman. Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 19(7):711-720, 1997.

Daniel B Graham and Nigel M Allinson. Characterising virtual eigensignatures for general purpose face recognition. Face Recognition: From Theory To Applications , pages 446-456, 1998.

Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence , 16(5):550-554, 1994.

Ron Cole, Yeshwant Muthusamy, and Mark Fanty. The isolet spoken letter database, 1990.

Todd K Moon. The expectation-maximization algorithm. IEEE Signal Processing Magazine , 13(6):47-60, 1996.

Changyi Ma, Fangchen Yu, Yueyao Yu, and Wenye Li. Learning sparse binary code for maximum inner product search. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 3308-3312, 2021.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Richard V Kadison. The pythagorean theorem: I. the finite case. Proceedings of the National Academy of Sciences , 99(7):4178-4184, 2002.

Stephen Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, Cambridge, UK, 2004.

David G Luenberger. Linear and nonlinear programming second edition. Columbus, Ohio: Addison-Wesley , 1984.

Otherwise, we obtain:

which implies:

Supplementary Material

Boosting Spectral Clustering on Incomplete Data via Kernel Correction and Affinity Learning

This document serves as supplementary material for the paper entitled 'Boosting Spectral Clustering on Incomplete Data via Kernel Correction and Affinity Learning'. It contains a theoretical analysis with proofs and formula derivation of theorems and algorithms presented in the main paper, a detailed description of experimental settings, and additional numerical results. Specifically,

Sec. A and B provide the proof of Theorem 1 and the formula derivations of Kernel Selfexpressive Learning (KSL) algorithms with an adaptive extension, respectively;

Sec. C illustrates experimental settings and implementation details;

Sec. D validates the benefits of the kernel correction algorithm on kernel and distance estimation;

Sec. E and F present comprehensive results and in-depth analysis on clustering performance.

A Theoretical Property of Kernel Correction

Theorem 1. ∥ K ∗ -ˆ K ∥ F ≤ ∥ K ∗ -K 0 ∥ F . The equality holds if and only if K 0 ⪰ 0 , i.e., K 0 = ˆ K .

Proof. Denote the feasible region as F , which is a closed convex set. Let K 0 be the initial kernel matrix. Let ˆ K be the projection of K 0 onto F ( ˆ K = min K ∈F ∥ K -K 0 ∥ 2 F ). Let K ∗ be the unknown ground-truth with positive semi-definiteness (PSD) in the feasible region F . By definition, we have:

Adding and subtracting ˆ K in the right-hand side, we get:

By the Pythagorean theorem [71], we have:

where ⟨· , ·⟩ F denotes the Frobenius inner product. Since ˆ K is the projection of K 0 onto the feasible region F , we have ⟨ K -ˆ K,K 0 -ˆ K ⟩ F ≤ 0 for all PSD matrices K in F [72]. Therefore, we have:

If K 0 is already a PSD matrix in F , we have:

Thus, we have ∥ K ∗ -ˆ K ∥ F ≤ ∥ K ∗ -K 0 ∥ F , which provides an improved estimate ˆ K to K ∗ .

B Formula Derivations of Kernel Self-expressive Learning

B.1 Formula Derivations of Kernel Self-expressive Learning with Proximal p -norm

Due to the non-convex nature of the ∥ Z ∥ p p term and the penalty term, finding a closed-form solution for the Z -update is not straightforward. However, we can relax the matrix optimization to an elementwise optimization without loss of generality, thanks to the format of our chosen objective function. Below, we present the deriving process:

where I ( z ij > 1) = { 1 , if z ij > 1 , 0 , otherwise. is an indicator function and L p = L + ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F .

We can use a solver to find out the Z -update via

Using the Alternating Direction Method of Multipliers (ADMM) approach [61], we can update the variables as follows.

C -update: Update C by minimizing the augmented Lagrangian with respect to C :

where K := ϕ ( X ) ⊤ ϕ ( X ) and I n is the identity matrix of size n × n .

Z -update: Update Z via solving

U -update: Update U via

We employ the ADMM process to solve the problem using the augmented Lagrangian function. However, despite the existence of a derivation method for this process, the computation proves to be excessively slow due to the following factors:

Matrix inversion: Inverting (2 K + 1 ρ I n ) -1 has a computational complexity of O ( n 3 ) .

Solver for Z -update: The golden-section solver is chosen to update z ij , resulting in an approximate computational complexity of log( 1 ϵ ) for each entry [73], which is relatively high.

Additionally, some potential complex solutions may arise when performing the element-wise update. As a result, we provide the algorithm for future researchers to further investigate and optimize.

B.2 Formula Derivations of Kernel Self-expressive Learning with Schatten p -norm

For 1 2 < p < 1 , we can define a relaxation of the optimization problem as:

With this definition, we can employ gradient descent to update the variables iteratively. The main steps of the algorithm are as follows:

U -update: Compute the gradient of L S p with respect to U :

where K := ϕ ( X ) ⊤ ϕ ( X ) . Update U according to the gradient:

where α is the learning rate.

V -update: Use the closed-form solution

Convergence condition:

If the difference between the updated U and the old U is less than a pre-defined tolerance ( ∥ U -U old ∥ 2 F < tol ), the algorithm is considered as converged, and we can terminate the loop. Otherwise, continue iterating with the new U and V values.

Upon convergence, the affinity matrix can be computed as A = ( | C | + | C | ⊤ ) / 2 , where C = UV ⊤ . This algorithm provides an efficient way to update the Schatten p -norm in the proposed KSL-Sp model, enhancing its robustness and clustering accuracy.

B.3 Formula Derivations of Adaptive Kernel Least-Squares Representation

The problem that we are doing with is:

Then we derive the augmented Lagrange equation in preparation for ADMM.

By introducing an auxiliary variable A, we have:

Augmented Lagrange equation is calculated by

where U is Lagrange multiplier, ρ is the updating step size.

ADMMconsists of iteratively updating K , A , C , and U as follows:

K -update: K = arg min K L ( K,A t , C t , U t ) . By First-Order Necessary Condition (FONC), we have

K -PSD: K t +1 ←P 1 ( K ) by projecting K onto the PSD set via Eq. (6).

A -update: A t +1 = arg min A L ( K t +1 , A, C t , U t ) . By FONC, we have

C -update: C t +1 = arg min C L ( K t +1 , A t +1 , C, U t ) . By FONC, we have

U -update:

C Experimental Settings

C.1 Datasets

To evaluate the performance of our proposed approaches, we select four benchmark datasets commonly used in machine learning and computer vision, including:

Yale64 [64] 3 : The Yale64 dataset consists of face images of 15 different individuals, each with different facial expressions, lighting conditions, and facial details. It contains a total of 165 images of size 64 × 64 pixels in 4,096-dimensional vectors.

Umist [65] 4 : The Umist dataset consists of face images of 20 different individuals, each with different facial expressions, lighting conditions, and facial details. It contains a total of 575 images of size 32 × 32 pixels in 1,024-dimensional vectors.

USPS [66] 5 : The USPS dataset consists of handwritten digit images of size 16 × 16 pixels in 10 classes. For the five repeated experiments in this paper, we use five different random seeds to randomly select 1,000 images with 256-dimensional vectors as a subset.

Isolet [67] 6 : The Isolet dataset is a speech dataset that contains recordings of 26 different speakers pronouncing the names of the English alphabet. It contains 1,560 recordings and each recording is of 617-dimensional vectors.

C.2 Baselines

The proposed approaches are evaluated against a range of baseline methods, including

ZERO : This statistical imputation method replaces missing values with zeros.

MEAN : This statistical imputation method replaces missing values with the mean value of all observed values in the corresponding features.

k NN [26]: This statistical imputation method predicts missing values based on the values of the k -nearest neighbors in the available data. (default value k = 10 )

EM [68]: This statistical imputation method estimates the missing values based on the Expectation Maximization algorithm.

SVT [27]: This matrix completion method uses a singular value thresholding algorithm to find the low-rank matrix that best fits the available data.

GR [28]: This matrix completion method uses an online algorithm to estimate the low-rank subspace of the data and then updates the subspace with new data points.

KFMC [29]: This matrix completion method uses a kernelized factorization approach to estimate the high-rank matrix that best fits the available data. (default: polynomial kernel)

DC [38]: This distance calibration method calibrates an initial distance matrix through double centering algorithm and transformation between similarity and distance.

TRF [35]: This distance calibration method calibrates an initial distance matrix to satisfy all the triangle inequalities by solving a convex optimization problem.

EE [39]: This distance calibration method calibrates an initial distance matrix to a Euclidean embeddable matrix by solving a convex optimization problem. (default value γ = 0 . 02 max { d 0 ij } )

C.3 Implementation Details

Data Preprocessing. To obtain better clustering performance, we conduct different preprocessing strategies. In the Yale64 and USPS datasets, we use the original data without preprocessing; in the Umist and Isolet datasets, we perform the normalization to normalize all data samples into [ -1 , 1] .

Generation of Incomplete Data. For experiments in Sections 5.1, 5.2 and 5.3, we generate an incomplete data matrix X 0 by randomly replacing each value in the data matrix X ∈ R d × n with an NA value with a given probability r (missing completely at random). We repeat this process five times and report the average performance of the baseline methods using X 0 as input.

Data Imputation Methods. We first impute the missing values in X 0 and then calculate the kernel matrix based on the imputed matrix ˆ X . We then perform spectral clustering on the kernel matrix.

Distance Calibration Methods. We first estimate an initial distance matrix D 0 on the incomplete data matrix X 0 via Eq. (1). We then calibrate D 0 to ˆ D and calculate the kernel matrix based on ˆ D . We finally perform spectral clustering on the resulting kernel matrix.

D Validation of Kernel Correction on Kernel and Distance Estimation

Two errors, relative-mean-square error ( RMSE = ∥ ˆ A -A ∗ ∥ 2 F ∥ A 0 -A ∗ ∥ 2 F ) and relative error ( RE = ∥ ˆ A -A ∗ ∥ F ∥ A ∗ ∥ F ), are used to evaluate the estimation accuracy, where A 0 is the naive kernel/distance matrix obtained by Eq. (1), ˆ A is the estimated one using different methods, and A ∗ as the ground-truth. Besides, Recall measures the search accuracy of local relationships, which is defined as the overlapping ratio of two sets of top-10 similar items obtained by ˆ A and A ∗ , respectively.

E Comprehensive Experiments on Standard Spectral Clustering

The robustness analysis is depicted in Fig. 1 in the main text, with accompanying numerical results in Table 6. Further, Fig. 2 displays clustering performance using ACC, NMI, and PUR metrics, underscoring the KC method's superiority over imputation baselines, particularly at large missing ratios. Notably, as the missing ratio escalates, imputation methods' accuracy and clustering efficacy decline sharply. Specifically, at missing ratios above 50%, their reliability wanes due to the paucity of data for imputation. In contrast, our KC method maintains consistent, and often superior, performance even with large missingness, exemplified by the results on the Yale64 and Umist datasets.

ZERO

MEAN

EM

SVT

GR

KFMC

KC

20%

40%

60%

80%

Missing Ratio r

0.2

0.3

0.4

0.5

0.6

ACC

Yale64

20%

40%

60%

80%

Missing Ratio r

0.2

0.3

0.4

0.5

0.6

0.7

NMI

Yale64

20%

40%

60%

80%

Missing Ratio r

0.2

0.3

0.4

0.5

0.6

PUR

Yale64

20%

40%

60%

80%

Missing Ratio r

0.2

0.3

0.4

0.5

0.6

ACC

Umist

20%

40%

60%

80%

Missing Ratio r

0.2

0.4

0.6

0.8

NMI

Umist

20%

40%

60%

80%

Missing Ratio r

0.2

0.3

0.4

0.5

0.6

0.7

PUR

Umist

F Comprehensive Experiments on Self-expressive Affinity Learning

Hyperparameters of Affinity Learning Algorithms. All kernel self-expressive affinity learning algorithms in this study adhere to a unified framework, i.e., min C 1 2 ∥ ϕ ( X ) -ϕ ( X ) C ∥ 2 F + λ R ( C ) , where λ is a regularization hyperparameter. Empirically, we choose λ from the interval [0 , 30] to roughly achieve better clustering performance, as detailed in Table 7. Note that the optimal λ may vary across algorithm implementations under different settings.

Clustering Performance on the Exponential Kernel. We investigate the benefits of the corrected Exponential kernel for kernel self-expressive affinity learning, including KSSC [19] and KLSR [3]. Tables 8 shows that our KC algorithm outperforms baselines regarding the NMI metric on the Exponential kernel based clustering, offering a systematic solution to clustering incomplete data.

Trade-off between Efficiency and Performance of the KC Algorithm. To enhance the efficiency of the KC algorithm, we explore the use of randomized singular value decomposition (rSVD) [53] a method that focuses on identifying the topk singular values (refer to Section 3.3). We compare the performance of the KC method with SD or rSVD on the USPS dataset. The results, presented in Table 9, reveal that rSVD greatly enhances the operational efficiency of the KC algorithm. However, it is crucial to choose an appropriate value for k (number of top singular values), as it heavily impacts the quality of clustering. Smaller k values (e.g., 10 or 20) result in poorer clustering due to the loss of important singular value information. On the other hand, selecting a suitable k value (e.g., 50 or 100) significantly reduces running time while maintaining clustering performance comparable to SD, particularly for the standard spectral clustering algorithm (SC). Striking a trade-off between efficiency and performance remains an intriguing avenue for future research.

Comparisons of KSL-Sp and AKLSR. AKLSRoutpaces KSL-Sp in execution speed. Algorithm 4 delineates each AKLSR update step explicitly, ensuring quicker convergence, whereas KSL-Sp in Algorithm 3 relies on gradient descent for U updates, leading to extended convergence times. Moreover, AKLSR's performance can vary with the presence or absence of a PSD constraint. Table 10 showcases the efficiency of AKLSR without PSD and its effectiveness with PSD on the Umist dataset.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. Firstly, the authors introduce a new kernel correction method that enhances the quality of the kernel matrix estimated on incomplete data with a theoretical guarantee, benefiting classical spectral clustering on pre-defined kernels. Secondly, they develop a new affinity learning method that equips the self-expressive framework with ℓp-norm to construct an intrinsic affinity matrix with adaptive extensions.\n\nSTRENGTHS:\nThe originality of this paper is satisfying, which proposes an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data and the significance is OK.\n\nThe quality and clarity of this paper are satisfying based on the clear presentation of the imputation-free framework with two novel approaches.\n\nWEAKNESSES:\n1. The authors propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. However, the advantages of the proposed method in dealing with incomplete data are not clearly stated based on Section 3, i.e., considering how to recover the missing data. Thus, the authors are expected to analyze the merits of the proposed method in dealing with the incomplete data.\n\n2. In section 3.1, the authors give different methods to provide a calibrated distance matrix with benefits for distance-based kernels and are not a universal solution for dealing with incomplete data in spectral clustering tasks. However, these separated methods may make the novelty of this paper separated and not focused. I think the authors can better stating the revisiting distance calibration methods in this part.\n\n3. The adopted datasets in the paper are almost with small scales in the experiments. The authors can add one or more multi-view datasets with large scales for validating the clustering performance.\n\n4. The improvements of the proposed method compared with other methods are not significant in the experiments, i.e., the RE_K of KC is just 0.217 on USPS.\n\n5. The authors can add one or more recent methods for comparison in the experiment, which make the comparson of experimental results more comprehensively.\n\nQUESTIONS:\n1. The authors propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. However, the advantages of the proposed method in dealing with incomplete data are not clearly stated based on Section 3, i.e., considering how to recover the missing data. Thus, the authors are expected to analyze the merits of the proposed method in dealing with the incomplete data.\n\n2. In section 3.1, the authors give different methods to provide a calibrated distance matrix with benefits for distance-based kernels and are not a universal solution for dealing with incomplete data in spectral clustering tasks. However, these separated methods may make the novelty of this paper separated and not focused. I think the authors can better stating the revisiting distance calibration methods in this part.\n\n3. The adopted datasets in the paper are almost with small scales in the experiments. The authors can add one or more multi-view datasets with large scales for validating the clustering performance.\n\n4. The improvements of the proposed method compared with other methods are not significant in the experiments, i.e., the RE_K of KC is just 0.217 on USPS.\n\n5. The authors can add one or more recent methods for comparison in the experiment, which make the comparson of experimental results more comprehensively.\n\nLIMITATIONS:\n1. The authors propose an imputation-free framework with two novel approaches to improve spectral clustering on incomplete data. However, the advantages of the proposed method in dealing with incomplete data are not clearly stated based on Section 3, i.e., considering how to recover the missing data. Thus, the authors are expected to analyze the merits of the proposed method in dealing with the incomplete data.\n\n2. In section 3.1, the authors give different methods to provide a calibrated distance matrix with benefits for distance-based kernels and are not a universal solution for dealing with incomplete data in spectral clustering tasks. However, these separated methods may make the novelty of this paper separated and not focused. I think the authors can better stating the revisiting distance calibration methods in this part.\n\n3. The adopted datasets in the paper are almost with small scales in the experiments. The authors can add one or more multi-view datasets with large scales for validating the clustering performance.\n\n4. The improvements of the proposed method compared with other methods are not significant in the experiments, i.e., the RE_K of KC is just 0.217 on USPS.\n\n5. The authors can add one or more recent methods for comparison in the experiment, which make the comparson of experimental results more comprehensively.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the spectral clustering problem when there is missing data. The paper proposes a new algorithm for correcting the computed kernel matrix by projecting the matrix to the nearest symmetric PSD matrix, and using this corrected kernel for clustering. The paper also combines the new kernel correction algorithm with affinity learning to develop a new technique for learning affinity matrices for incomplete data.\n\nExperimental evaluations show that the newly developed techniques outperform alternative methods for handing incomplete data with respect to standard clustering metrics.\n\nSTRENGTHS:\nThe newly proposed algorithm for kernel correction is novel and interesting. It has potential applications and could inspire future research directions. The algorithm is conceptually quite simple and captures the theoretical intuition quite naturally. The experimental results suggest that the new algorithm outperforms alternative methods.\n\nWEAKNESSES:\nThe running time of the kernel correction algorithm is quite slow - it requires computing the spectral decomposition of the kernel matrix at every iteration. It is good that the authors discuss this limitation in the paper, although it is a weakness of the algorithm. Improving the running time could be a future research direction.\n\nIt would be interesting to compare experimentally the benefit that kernel correction brings over simply doing nothing to correct the missing data (or doing something very naive to construct the k-NN graph for example).\n\nQUESTIONS:\nEvery iteration of the KC algorithm requires computing the spectral decomposition. Given that only the top k eigenvectors are needed for spectral clustering, could it be possible to speed up the algorithm by, for example, computing only k eigenvectors at each iteration? \n\nWhat is the performance experimentally of performing spectral clustering on the naive k-NN graph constructed from the incomplete data? Given that this will be much faster than running KC, it would be interesting to see the trade-off between running time and accuracy.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an imputation-free framework for correcting a kernel obtained from incomplete data. They propose the corrected kernel to be a PSD matrix with bounded elements that is closest to the initial kernel (calculated from incomplete data) in Frobenius norm. They show that the corrected kernel is guaranteed to be closer to the ground truth than the initial kernel. In the case of the Gaussian kernel, Dykstra's projection algorithm is presented to obtain the corrected kernel. They then extend the existing self-expressive affinity learning framework by incorporating the proximal p-norm (0 < p < 1) and the Schatten p-norm (½ < p < 1) penalties on the affinity matrix. An algorithm (KSL-Pp) based on augmented Lagrangian and ADMM is proposed for the proximal p-norm penalty. Another algorithm (KSL-Sp) based on gradient descent is proposed for the Schatten p-norm penalty. Finally, the authors combine the kernel correction and the self-expressive affinity learning frameworks to jointly learn the corrected kernel and the affinity matrix. An algorithm (AKLSR) based on augmented Lagrangian and ADDM is proposed to solve the joint optimization problem.\n\nFor numerical experiments, they perform a comparative analysis on synthetic and image datasets and show their proposed method produces a more accurate Gaussian kernel than the existing techniques. The performance of spectral clustering improves when their proposed corrected kernel is used against the corrected kernels obtained by competing techniques. For self expressive affinity learning, they show that more accurate affinity can be estimated when their proposed corrected kernel is used against the corrected kernels obtained by other techniques.\n\nSTRENGTHS:\nSignificance: The Gaussian kernel plays a central role in several tasks including clustering, dimensionality reduction, graph neural networks etc. A comparative analysis on image datasets is provided that showed improved quality of the corrected kernel using the proposed approach against those obtained by existing techniques. The same is also reflected in the improved performance of spectral clustering and the improved quality of self-expressive affinity matrices based on their proposed corrected kernel. On a high level, these improvements could be helpful in downstream applications.\n\nQuality: The authors adapted Dykstra's projection algorithm which comes with linear convergence guarantees, to correct the Gaussian kernel obtained from incomplete data.\n\nWEAKNESSES:\nQuality / clarity:\nThe authors extended the existing self-expressive affinity learning framework by incorporating p-norm based penalties (0 < p < 1). However, it is not clear how p-norm based penalties are more effective than the conventional 1-norm, Frobenius norm and nuclear norm based penalties. The provided empirical analysis seems insufficient to reach any conclusion.\n\nThe authors also proposed an algorithm that jointly optimizes the corrected kernel and the self-expressive affinity matrices using ADMM. However, there seems to be some issue in the formulation (see questions), and no empirical analysis is provided. This gives the sense that the paper is incomplete: the authors propose Adaptive Kernel Self-expressive Learning in 4.2 to tie between the kernel correction in section 3 and the self expressive affinity in 4.1, however do not include any simulations for this approach. Without demonstrating its efficacy, the inclusion of the proposed algorithm AKLSR seems unnecessary, and then the paper is essentially two unrelated approaches (kernel correction, self expressive learning) grouped together.   \n\nMissing references:\n* Gilbert, A. C., & Jain, L. (2017, October). If it ain't broke, don't fix it: Sparse metric repair. In 2017 55th Annual Allerton Conference on Communication, Control, and Computing\n* Shahid, N., Kalofolias, V., Bresson, X., Bronstein, M., & Vandergheynst, P. (2015). Robust principal component analysis on graphs. In Proceedings of the IEEE International Conference on Computer Vision \n* Biswas, Arijit, and David W. Jacobs. \"An Efficient Algorithm for Learning Distances that Obey the Triangle Inequality.\" BMVC. 2015.\n\nQUESTIONS:\nMajor comments:\n* Line 42. “Incomplete data” - the authors never define what incomplete setting they are solving, I assume missing at random, but this should be clarified.\n* Line 128 “With a small missing ratio of features, $D_0$ already satisfies most triangle inequalities,..., then the algorithm typically yields only marginal improvement” - wouldn’t this be true of all methods? If the ratio of missing features is low, the impact on kernel construction is small and fixing it should yield marginal improvement. \n* line 140, an explanation of “the quality of $\\hat{D}$ cannot be guaranteed” would be helpful.\n* Line 221 “the corrected distance obtained from the corrected Gaussian kernel will also be more accurate than the calibrated distance from the Euclidean embedding method.” is there a proof for this or just empirical evidence in the simulation?  \n* Why do the elements of \\hat{C} on line 11 and 10 of algorithms 2 and 3 lie in [0,1]?\n* The constraints on K in Eq. (9), seem incorrect. Why are they not consistent with those in Eq.(5)? Why is there no constraint that K is PSD?\n* Line 201: O(n^2) storage limits the size of data that can be analyzed. Are the estimated kernels dense or sparse?\n* Is the inverse of the formula for k_{ij} on line 260 used to obtain d_{ij}?\n* The future work on line 287 “with future work on the potential of deep learning clustering techniques” should be made a bit more clear.\n\nSimulations: My main concern is with the experiments. \n1. The asymptotic time complexity provided by the authors is appreciated. It would be helpful if the actual time taken (in seconds) by the kernel correction and the affinity learning algorithms are also reported.\n2. Numerical experiments for KLRR and KSL-Pp in Table 3 are missing.\n3. Numerical experiments for KLRR, KSL-Sp and KSL-Pp in Table 3 and 4 in the supplementary material are missing.\n4. Numerical experiments for the proposed AKLSR algorithm that jointly corrects the kernel and learns the affinity matrix are missing.\n5. Gains are modest compared to EE - how do the two compare in terms of runtime/storage?\n6. A popular measure to evaluate clustering is the adjusted Rand index (ARI). The authors should add this since it is more informative than ACC. \n7. An additional non-image dataset would be useful to add if there is time (word-document, recommendation systems).\n8. std values are missing from all the tables. \n9. the experiments are performed only for 80% missing values. A wider range of missingness should be explored (plot can be used instead of tables) to demonstrate how robust the method is.\n10. How are the hyperparameters of competing methods determined?  \n\nMinor comments:\n* Although it seems intuitive, a reference for line 22 “incomplete data is commonly seen in practice, leading to inaccurate affinities and degraded clustering performance” would be helpful.\n* A minus is missing in the formula for K_{ij} on line 87.\n* The presentation in section 3 of “first method, “second method”,... seems odd. Are these the only relevant algorithms? Also, instead of using “first method” as a paragraph title, it would be better to use the name of the algorithm for the title of the paragraphs starting on lines 122, 130 and 142.\n* A more precise reference (theorem number etc.) of Property 2 on line 146 would be helpful.\n* Theorem 1 on line 193 may be placed after equation 5 as it holds in the general setting.\n* The transition to section 4 is rather abrupt and 1-2 introductory sentences can help better tie the paper together. \n* Which algorithm (SVD or rSVD) in Section 3.3 is used in the numerical experiments?\n* What value of p for KSL-Sp used in Section 5.3?\n\nLIMITATIONS:\nlimitations aren't discussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new kernel correction method to address the issue of incomplete data. Existing approaches aim to recover the distance matrix of complete data, starting from that of the incomplete data. In contrast, the proposed method (Section 3.2) formulates the problem as finding a positive semi-definite matrix that is closest to kernel matrix of incomplete data. The optimisation is solved using a iterative approach.\nThe paper further extends the approach to (kernel) affinity learning problems to propose 3 further algorithms, and finally, experimentally shows that the proposed methods outperform existing distance completion methods in the setting of kernel spectral clustering.\n\nSTRENGTHS:\n- The proposed approach is technically sound, and based on a rather intuitively simple idea of projecting onto the space of positive semi-definite matrices.\n- Extensions to affinity learning are proposed\n- The experiments show clear improvement over existing approaches for completing distance matrices\n\nWEAKNESSES:\nLiterature:\nThe paper focus only on kernel spectral clustering and affinity learning, and hence, misses the broad and older literature on kernel methods for supervised learning. A quick search of Google scholar reveals a considerable literature on this topic. Few papers are noted but the literature is quite large, and it is not clear why the paper does not compare with such approaches.\n- Smola, Alex J., S. V. N. Vishwanathan, and Thomas Hofmann. \"Kernel methods for missing variables.\" International Workshop on Artificial Intelligence and Statistics. PMLR, 2005.\n- Dick, U., Haider, P., & Scheffer, T. (2008, July). Learning from incomplete data with infinite imputations. In Proceedings of the 25th international conference on Machine learning (pp. 232-239).\n- Liu, X., Zhu, X., Li, M., Wang, L., Zhu, E., Liu, T., ... & Gao, W. (2019). Multiple kernel $ k $ k-means with incomplete kernels. IEEE transactions on pattern analysis and machine intelligence, 42(5), 1191-1204.\nThe paper needs to positioned well in context of this literature, and the proposed algorithm should be compared with existing kernel completion methods.\n\nWeak theory, and insufficient experiments on different missing data:\nTheorem 1 is rather weak since it only guarantees that if the incomplete kernel matrix was derived from a some true psd matrix, then the returned projection is closer to ground truth that the incomplete kernel matrix. However, the algorithm/theorem does not guarantee that the output kernel matrix is significantly better than the given incomplete one. A general guarantee cannot be provided without assumptions. \nHowever, the work would significantly improve if there is a guarantee on recovery of ground truth kernel assuming that incomplete kernel matrix is obtained from data with features missing uniformly at random (setting considered in experiments)\n\nOn a similar note, the experiments only assume that features/values a missing at random. However, in practice, data is often systematically missing (some features tend to have higher missing values rate; there is often correlation between missing entries). Extensive experiments are needed to demonstrate that the proposed methods are robust to different types of missing entries, other than uniformly random.\n\nQUESTIONS:\n- The fundamental goal of the paper seems to be kernel completion. What is special about spectral clustering (or its affinity learning) in this context? Why cannot one use the methods for supervised learning?\n- It is well known that spectral clustering does not require a positive semi-definite kernel (for instance, one can apply spectral clustering also on a graph adjacency). Then isn't it better to run any other matrix completion approach that does not impose psd?\n\nLIMITATIONS:\nThe paper does not discuss any limitation, and the reported experimental results suggest that the proposed methods clearly outperform existing methods.\nHowever, as noted in weakness, the paper needs comparison with existing kernel completion works and needs discussion on what kind of missing data can drastically impact the proposed methods.\n\nPossible negative societal impact is neither evident nor discussed in the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}