Likelihood adjusted semidefinite programs for clustering heterogeneous data
Authors: Yubo Zhuang
Xiaohui Chen
Yun Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: clustering
likelihood inference
semidefinite programming
alternating maximization

Abstract: Clustering is a widely deployed unsupervised learning tool. Model-based clustering is a flexible framework to tackle data heterogeneity when the clusters have different shapes. Likelihood-based inference for mixture distributions often involves non-convex and high-dimensional objective functions, imposing difficult computational and statistical challenges. The classic expectation-maximization (EM) algorithm is a computationally thrifty iterative method that maximizes a surrogate function minorizing the log-likelihood of observed data in each iteration, which however suffers from bad local maxima even in the special case of the standard Gaussian mixture model with common isotropic covariance matrices. On the other hand, recent studies reveal that the unique global solution of a semidefinite programming (SDP) relaxed $K$-means achieves the information-theoretically sharp threshold for perfectly recovering the cluster labels under the standard Gaussian mixture model. In this paper, we extend the SDP approach to a general setting by integrating cluster labels as model parameters and propose an iterative likelihood adjusted SDP (iLA-SDP) method that directly maximizes the \emph{exact} observed likelihood in the presence of data heterogeneity. By lifting the cluster assignment to group-specific membership matrices, iLA-SDP avoids centroids estimation -- a key feature that allows exact recovery under well-separateness of centroids without being trapped by their adversarial configurations. Thus iLA-SDP is less sensitive than EM to initialization and more stable on high-dimensional data. Our numeric experiments demonstrate that iLA-SDP can achieve lower mis-clustering errors over several widely used clustering methods including $K$-means, SDP and EM algorithms.

Under review as a conference paper at ICLR 2023
LIKELIHOOD ADJUSTED SEMIDEFINITE PROGRAMS FOR CLUSTERING HETEROGENEOUS DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Clustering is widely deployed unsupervised learning tool. Model-based clustering is flexible framework to tackle data heterogeneity when the clusters have different shapes. Likelihood- based inference for mixture distributions often involves non- convex and high- gigh--dmensional objective functions, imposing difficult computational and statistical challenges. The classic snpectationmmaximzzann is computationally thrifty iterative method that maximizes surrogate function minorizing the log-likellhooe of observed data in each iteration, which however suffers from bad local maxima even in the special case of the standard Gaussian mixture model with common isotropic covariance matrices. On the other hand, recent studies reveal that the unique global solution of a semidefinite programming (SDP) relaxed K-means achieves the cnormatiion-heoretiiiii perfectly recovering the cluster labels under the standard Gaussian mixture model. In this paper, we extend the SDP approach to a general setting by integrating cluster labels model parameters and propose an iterative likelihood adjusted SDP (iLA-SDP) method that directly maximizes the exact observed likelihood in the presence data heterogeneity. By lifting the cluster assignment group-specific membership matrices, iLA-SDP avoids centroids estimation a key feature that allows exact recovery under well-separateness of centroids without being trapped by their adversarial configurations. Thus iLA-SDP is less sensitive than EM to initialization and more stable on high-dimensional data. Our numeric experiments demonstrate that iLA-SDP achieve lower mis mis-clustering errors over several widely used clustering methods including K K-means, SDP and EM algorithms.
(EM) algorithm
sharp threshold for
INTRODUCTION
Clustering analysis has been widely studied and regularly used in machine learning and its applications
in network science (Girvan & Newman, 2002), computer vision (Shi & Malik 2000 Joulin et al. 2010), manifold learning (Chen & Yang 2021a) and bioinformatics (Karim et al. 2020). Perhaps by far the most popular clustering method is the K K-means (MacQueen) 1967) partially because there are computationally convenient algorithms such as Lloyd's algorithm and K-means++ for heuristic approximation (Lloyd] 1982, Arthur & Vissilvitskii, 2007). Mathematically, K-means aims to find the optimal partition of data to minimize the total within-cluster squared Euclidean distances, which equivalent the maximum profile likelihood estimator under the standard Gaussian mixture model (GMM) with common isotropic covariance matrices (Chen & Yang 20211). Nevertheless, real data usually exhibit various degrees of heterogeneous features such the cluster shapes may vary from component to component, which renders K-means as a sub-optimal clustering method.
Another popular clustering method the classic expectation maximization (EM) algorithm, which is a computationally thrifty method based on the idea of data augmentation to iteratively optimize the non-convex observed data likelihood (Dempster et al. 1977). Theoretical investigations reveal that the EM algorithm suffers from bad local maxima even in the one-dimensional standard GMM with well-separated cluster centers [ et al., 2016). Thus practically even when applied in highly favorable separation-to-nois ratio settings, careful initialization, often through multiple random initializations or a Warm-start by another heuristic method such as hierarchical clustering (Fraley & Raftery| 2002), is the key for the EM algorithm find the correct cluster labels and model parameters. With a reasonable initial start, the EM algorithm has been shown to achieve good statistical properties (Balakrishhan et al. 2017 Wu & Zhou 2019

---

Under review as a conference paper at ICLR 2023
In this paper, we consider the likelihood- inference to tackle the problem of recovering cluster labels the presence of data heterogeneity. Our motivation stems from the recent progress in understanding the computational and statistical limits for convex relaxation methods of the K- means clustering. Since K-means is a worst-case NP-hard problem (Aloise et al. 2009), various
heuristic approximation algorithms such as Lloyd's algorithm (Lloyd 1982 & Zhou [2066), and computationally tractable relaxations such as spectral clustering (Meila & Shii|20011 Ng el al. 2001
Vempala & Wang 2004 AAhhiiotas & McSherry 2005; von Luxburg 2007] von Luxburg al.
2008) and semidefinite programs (SDP) (Peng & Weil [ei][2000]]
et al.
2016 Li et al. 2017 Fei
& Chen 2018 Chen & Yang 202 Royer 2017] Giraud & Verzelen, 2018] Bunea et al. 2016
Zhuang et al., 2022a), have been proposed in literature. Among the existing solutions, the SDP approach particularly attractive in that it attains wwermaaioon-thoreticn
optimal threshold on centroid separations for exact recovery of cluster labels (Chen & Yang 2021b).
Our contributions. We extend the SDP approach to a general setting with heterogeneous features by integrating cluster labels model parameters (together with other component-specife parameters) and propose an iterative likelihood adjusted SDP (iLA-SDP) method that directly maximizes the exact observed data likelihood. Our idea is to tailor the strength of SDP relaxation of the K K-means clustering method in the isotropic covariance case for likelihood fkklihooddawareeess awareness inference. On one hand, iLA-SDP has a similar flavor the EM algorithm by maximizing the likelihood function of the observed data. On the other hand, different from the EM framework, iLA-SDP treats the cluster labels as unknown parameters while profiles out the cluster centers (i.e., centroids), which brings several statistical and algorithmic ad advantages.
First in the arguably simplest one-dimensional GMM setting, EM known fail in certain configura- tions of centroids even when they are well-separated (Jin et al. 2016). In other words, EM is sensitive initialization and model configuration. The main reason is due to the effort for estimating the cluster centers during the EM iterations. In iLA-SDP, cluster centers are regarded as nuisance parameters and profiled out to obtain a likelihood function in component-speciff parameters including only the cluster covariance matrices. Thus iLA-SDP is more stable and performs empirically better than FM.
Second, cluster labels in EM are latent variables that are estimated by their posterior probabilities and the observed likeliool for component parameters and mixing weights are optimized through minorizing functions during iterations. In iLA-SDP, cluster labels are regarded as parameters optimized through the likelihood function jointly in the labels and covariance matrices. Thus iLA- SDP is a more direct approach than EM for taming the non-convexity in the observed log- objective and prove that it perfectly recovers the true clustering structure if the clusters well-separated under a lower bound without concerning the configurations of centroids.
The rest of the paper is organized as follows. In Section 2 review some background on partition- based formulation for basel- clustering. In Section R we introduce the likelihood adjusted SDP for recovering the true partition structure and discuss its connection to the EM algorithm. In Section 4 we compare the performance of several widely used clustering methods on two real datasets.
2 MODEL-BASED CLUSTERING: A PARTITION FORMULATION
We consider the model- based clustering problem. Suppose the data points X X,, E RP are independent random variables sampled from K-component Gaussian mixture model (GMM). Specifically, let ,i,..., G be the true partition of the index set [n] == (1,...,m) such that if i E G then
X,
= μ4 + C;+
(I)
where l E RR is the center of the k-th cluster and e; is an i.i.d. random noise term following the common distribution ((0, Ex). Here we focus the most general and realistic scenario where the within-cluster covariance matrices E....., Σg are heterogeneous. In our formulation of the GMM, the true partition (Gt) is treated as unknown parameter in model (20). along with the component-wise parameters (μa, _ With this parameterization (Gx, H&, ) the log- ligeeiili function for observing the data X = (XI...., Xn} } is given by
K
G 2
1 Σ
E((GR, pk. k X)
log (T (x1)
Σ 2 k=
(X+
T (x, pb))
2

---

Under review as a conference paper at ICLR 2023
where [G&] is the cardinality of G1 and |Σx| is the determinant of matrix Σg. Since are primarily interested in recovering the clustering labels (or equivalently the assignment matrix, cf. Section 3.1 below) in the presence of cluster heterogeneity, we can first profile out the nuisance parameters 14 in closed form and the resulting objective function as a profile log-likelihood for the remaining parameters (after dropping constants) is given by
K
K
K
E((GK, Ix)
X)
Σ
G& log (Ex)
Σ GEG
Σ
X
(X,,
Xihxe'>
G
AjEGG
(2)
where 4)y := Eu and || == (u, u)y for any u R' and Σ > 0. This leads us combinatorial optimization problem for the profile log- likelihood function
max {e((Gs, E()
x)
L G+ [n], Σx > >0}.
(3)
where the disjoint union - G2 = [n] means that U Gx = [m] and G3 n. == Note that the constrained optimization problem in (3) in the special case Σ₁ reduces to the K K-means clustering method, which known to be worst-case NP-hard (Dasgupta) 2007. Mahajan et al., 2009). To overcome such computational difficulty, semidefinite program (SDP) relaxation a tractable solution that achieves under the standard GMM with identical and isotropic covariance matrices (Chen & Yang. 2021a). Nevertheless, all existing formulations of various SDP relaxations of the standard GMM critically
= if
Σn Id
thcoreticallly optimal exact recovery
depend on the assumption that En =.... = Di = Id,,, with a known noise variance parameter
o² (Fei & Chen 2018 Li et al. 2017 Peng & Wei 2007 Chen & Yang 2021a). This motivates
us seek alternative SDP formulations adjusting the (full) information coming from the likelihood function for the observed data X.
3 LIKELIHOOD ADJUSTED SDP FOR CLUSTERING HETEROGENEOUS DATA
In this section, we introduce the likelihood adjusted SDP (LA-SDP) for recovering the true partition structure GT, ... by applying convex relaxation to the profile log-likelihood function
3.1 ORACLE LA- UNDER KNOWN COVARIANCE MATRICES
In this subsection, we consider the oracle case where the covariance matrices E....., Σx are known. Let us start with a well-studied SDP relaxation formulation (Peng & Wei 2007) for approximating the combinatorial optimization problem of maximizing the profile log-likelihood function under the isotropic setting with known Σ₁ attain the information-thhorrticalll theoretically optimal threshold on centroid separations for exact recovery of cluster labels. Note that there is a one-to-one correspondence between any given partition (G&) ) of [n] and a binary assignment matrix H (haa) {0, 1] | (up to cluster labels permutation) such that hąs = 1 if i E G5 and his = 0 otherwise for E [r+] and & E [K]. Because each row of H contains exactly one non-zero entry, the recovery of the true clustering structure (or its associated assignment matrix) by maximizing the profile log-likelihood function (after dropping constants) can be re-expressed (non-convex) mixed integer program:
Σx = Idld,, which is known (Chen & Yang 2021b) to
K
max (A, HBH 11
Σ (Xi, subject to H E (0, 1janK and H1K = 1ns (4) iJEGr
GK
k=
where A X is the n similarity matrix. 1, denotes the n-dimensional vector of all ones, and B is the K K diagonal matrix whose k-th diagonal component is |G₂ -1 (En ha)== Here, we have used the key identity Σx= Wk Eisec, ai (A, HBH that holds for any diagonal matrix B- diag(w), wx) and similarity matrix A (a:r)), Relaxing the above mixed
integer program by lifting the assignment matrix H into Z = HBH we arrive at its SDP relaxation
Ż = arg max (), subject to Z 2 0, tr(2) = K, Z1, 1,, Z > 0, Z
(5)
3

---

Under review as a conference paper at ICLR 2023
where Z > 0 means each entry Zi > 0 and Z ž means the matrix Z is symmetric and positive semi-definite. This SDP formulation relaxes the integer constraint on into two linear constraints tr( tr(2)= = K and Z > 0 that are satisfied by any Z = HBH as H ranges over feasible solutions of problem
Now let consider the general heterogeneous setting with (possibly) different and non-isotropic covariance matrices Σi,..., ......,,K, Σx. and extend the SDP relaxation to this setting. Two technical difficulties arise by examining the previous argument. First, the first two terms in the profile log- likelihood function (3) are no longer independent of the assignment matrix, and is therefore not negligible. particular, they also provide partial information about the cluster labels when the covariance matrices are different: ||xx,| in the second term quantifies how well X; aligns with
the covariance matrix Σg encoding second-order information of the k-th cluster; while the first term plays the role of balancing the cluster sizes and favors assigning more points to clusters with smaller shapes (since density is expected to be high). Second, the similarity (X;, s within cluster Gk
in the third term now depends on k. making the key identity Σ Ei= Wk Eijea, Oij (A, HBH for connecting the profile log- likeiiid function with the objective function of the mixed integer program (4) no longer applicable.
To solve the two aforementioned difficulties, we propose to augment the single variable Z in the SDP relaxation 5 to variables (7x) where Zg can be interpreted as the lifting of the k-th column H2 of the assignment matrix H via Z [G*] Hy H |Gx = Σi has H 1n. that encodes the cluster membership associated with the k-th cluster. More specifically, by extending the key identity in the isotropic setting to Σx= We Exjec, dij > (A(h), Hgw ) for any weight vector
(k)
(wx) k and K similarity matrices (As (a: maximizing profile log- ligelikei problem as the following (non-convex) mixed integer program:
(k) )2 - k=1' we can analogously express the
K
K
max H
(M)), Σ
Hawe Hw),,
subject to Hs E [0,1) and Σ
H& = las
(6)
where wk= = |Gxl -1 (Ein han)" and the k-thh cluster-specific similarity matrix A is
(A)) ; = log( 11 2 [diag(X x)1? + dini (X 2x x) + x M X. (7)
Here, diag(A) stands for the column vector composed of all diagonal entries of a matrix A. Now by lifting H₃ into Z H we arrive at the following SDP relaxation for the profile log-likelihood objective function
K
(2....., (.,
arg max
(Ax, Zz),
Z
ER
(8)
K (r(Zx) = K,
K
subject to Zu 2 0, km
Zx 1n = 1n, Zu > 0, V [
kml
which relaxes the integer constraint on H = (H,, H2,..., HR) into (K + 1) linear constraints Σk_ (r() = K and Zu > 0 for k E [K] that are satisfied by any Z₁- = H2 H.W.H as H ranges over feasible solutions of problem (6).
Since solving g requires the knowledge of the true covariance matrix for each component, we call the oracle likelihood adjusted SDP (LA-SDP) for estimating the cluster membership matrix of data points. the special case of isotropic covariance matrices Σ,
the solution (Z
k=
ΣK orid,, Proposition I below shows that LA-SDP reduces to become equivalent to the previous
SDP formulation (5).
Proposition 1 (SDP relaxation for K K-means is a special case of LA-SDP). Suppose Σ₁ = o I'Idp for all & E [K]. Let Ż be the solution to (5) that achieves maximum M2 and Ze, k 1,..., K, be
the solution to 5 with maximum M2. Then M1 = M2. And 2 Σ 2g, if 2 is unique in 5).
4

---

Under review as a conference paper at ICLR 2023
Note that the SDP relaxed K X-means in (5) originally proposed in (Peng & Wei
LA-SDP output
2-2+2+++2
2007) and has been extensively studied in literature. In particular, it achieves the information-theoretee
Overal
limit for exact covery under the standard GMM (Chen & Yang, 2021a) and robust against liers and adversarial attack (Fei & Chen 2018). In the case of exact recovery where 7 = and 7* the true cluster mem- bership matrix such that 2 - if 2,j Gj and Zi = otherwise, then we can easily recover the true partition struc- ture Gi,..., G or its associated assign- ment matrix from the block diagonal ma- trix 2. Thus it is interesting theoretical question of when the partition structure induced by 2 Σx Zx from the LA-SDP (see Figure for an illustration) can achieve exact recovery. Theorem
Z
72
Eigenvectors
Rounding
7/
74
Figure 1: LA-SDP membership matrices to cluster la- bels via spectral rounding.
2 below gives lower bound of the separation signal-to-noise ratio for achieving exact recovery in the presence of data heterogeneity.
For each distinct pair (k, E [K], let D(a,)) = (o characterize the closeness pmax, [A,
between Σg and Σg, where AI....,>p enumerate all eigenvalues of (2) > - Idp). If
A; 0, Vi [p], we let ((t,)) = 0. Let A == minayu ||_- (1z #=)l denote covariance adjusted centroid separation, R& [G the size of true cluster Gi. m ming 2 the least cluster sizes,
na+ ming ng the minimal cluster size, and M
pairwise harmonic mean maxe31 >>>>>>=>>> |> lop (matrix operator norm).
Theorem (Exact recovery for LA-SDP). Suppose there exist constants ó > 3 E (0, 1) and 1) E (0, such that
(1 3)2 (1- (1)))(1))k²
Cin 85
8
C2M
4(1 + 6)²
log n 2 max
B2
B² max - 1)3,1) ʃ m
- 31-3)2 K m
82
Then the LA-SDP achieves exact recovery, or 2 = Z", with probability at least 1 - Cn
A 2 (E; - E2) log n, and min ((.,)) 2 Cs(1 + log f p/n),
where concrete expressions of E₁ and Eg (depending on 8, 8, 17) are provided in Appendix A.4, and Ci..... C7 are universal constants.
Our definition of the centroid separation A extends the separation-to-noise ratio (SNR) for the exact recovery under the isotropic covariance setting (Chen & Yang (222a)) to the heterogeneous setting by taking into account the cluster shapes (i.e. second order information). From (18) we that our thcoretical centroid separation lower bound consists of two parts Ej and Eg: Ej reduced to the information-thhorrtteall] optimal threshold when M 1, corresponding matrices; E2 tends to vanish for small M close to one and satisfying M = 1 + o(1/ /n log n) or remains extra term for large M. From numerical results summarized in Figure 2 observe that our defined centroid separation A indeed captures the accuracy of cluster label recovery using LA-SDP-1hh mis-clustering error curves display almost identical patterns under different settings of the GMM. In comparison, the performance of the (original) SDP (5) and the K K-means clustering method designed for the isotropic case become significantly worse the condition number of the cluster covariance matrices increases. More details about implementation and model setups are provided in Appendix A.3
same covariance
3.2 ITERATIVE LA- ADSD UNDER UNKNOWN COVARIANCE MATRICES: AN ALTERNATING MAXIMIZATION ALGORITHM
Since the oracle LA-SDP relies on the knowledge covariance matrices Ej...., Ex, k. we propose simple and practical data-driven algorithm for approximating LA-SDP when these covariance
5

---

Under review as a conference paper at ICLR 2023
Mis-cluste error when changes
Mis-icustoiing error when A changes
6.7
0.74
50P
<
0.6
0,5
MIS--cusEenng ornor
0.5
0.4
0.4
0.3
0.3
-SDP
-4-KK
02
0.2
0.1
0
4
Figure 2: Mis-clustering error (with shaded error bars) vs centroid separation A under different conditional numbers of cluster covariance matrices Σ₁ Σg plot corresponds to moderate (large) condition number of the common covariance matrix. Here, KM refers to K -means method; SDP refers to the original SDP 5).
Σx (M = 1). The left (right)
matrices are unknown. The idea is to alternate between the SDP relaxation given a current estimate of the component covariance matrices and updating covariance matrices according to the maximum (penalized) likelihood given the new membership estimate. The next lemma gives a closed-form formula for updating covariance matrices given current estimate of the assignments Zi,..., Zx based on their (unconstrained) MLEs on the observed data.
Lemma 3 (Updating formula for covariance matrices under alternating maximization). For any feasible matrices Z1, ...., ZK satisfying the constraints of (8),
Σ
Σ
12
}
(X,X + +x,x]) xxx3 ZResi k [K],
(10)
1 ITZ
1,j=1
solve the following optimization problem
K Σ (Ap. Zk);
Σ.....,
EK = arg max Ex.....
(11)
2
1=1
where recall that (A) == ((((EE)) is the Σg dependeentt similarity matrix defined in 7
Based on the lemma, we propose an iterative LA-SDP (iLA-SDP) by alternating maximization of the profile log-likelihood (3) for estimating the lifted cluster membership matrices (Zx) k from
LA-SDP (8) and the component covariance matrices (Σx) as summarized in Algorithm In the special case where the lifted membership matrix Zg is of rank one, which holds for true lifted cluster membership matrices (Z the covariance matrices produced by iLA-SDP can be interpreted as within-cluster sample covariance matrices under soft clustering.
Proposition 4 (Covariance estimation in iLA-SDP via soft clustering). If rank 1, then there exists weights (Wk,1,... ..., Wk,n) such that these Ex in Lemma can be written as
WK. wX. px) T where Pu
Th Σ i and ne
Σ Wkis (12)
T2g.
is further noted from the proof of Proposition that when Z3 has rank one, the weights Wk.l..... Wk.n are proportional to the leading non-zero eigenvector of Zz. Thus the alternating maximization step (17) for updating the covariance matrices in iLA-SDP can be interpreted as a soft clustering technique that resembles the EM algorithm. Specifically, the E-step estimates the (hard) cluster label Y, E (0, 1 associated with X; by the posterior probabilities Tek = p(Yix X,, 0()
where (()) = (#()) (1) (1)
, ((()) K denotes the estimated GMM parameters at the 1-th iteration in the
EM. Then the M-step updates the parameters via म (1+1) m /n with m (t)
Σm= T
(t)
1 (1+1)
Σ
(t) X, and H m
((++1)
1
ak (X₃
p/ (++1))(X, (. (13)
(0)
sk
mi i=1
i=1

---

Under review as a conference paper at ICLR 2023
Note that 177 and 13 represent different weighting schemes in the soft clustering rule for obtaining an estimate for the cluster labels. In iLA-SDP, the weight wk, for X; belonging to component is calculated, remaining parameter updates in both iLA-SDP and EM boil down to simple averages with effective component sample sizes nig and mk- respectively. In Section 3.3 to follow, we provide deeper comparison between iLAASDP and EM.
determined by the SDP in (8) Once the weights
Remark 5. In Appendix A.2 we further propose two variations of iLA-SDP that can handle high-dimensional and sizge-size data with better computational and statistical efficiency. For high- dimensional data, we apply Fisher's LDA with an initial estimate of the cluster labels to find an optimal feature subspace that increases the SNR for better clustering, and for large-size data we combine the subsampling idea with iLA-SDP to reduce computational cost (Zhuang ct al. 2022b).
Algorithm 1:
The iterative likelihood adjusted SDP (iLA-SDP) algorithm
Input: Data matrix E RPx= containing n points. Initialization of assignments G [()) or covariance matrices _ ..... =K The stopping criterion parameters c, S.
GK
(0)
(Assignments to covariance matrices) If we have the initialization of assignments, let
2 (-) = | Eieato) (X; -x)((x); - Xx,)" to be the sample covariance of each cluster
-1 & E [K], where x= | 1 Eieogo X
for 1, ..... S do
(Adjusted-SDPP Solve the Adjusted-SDP solution -/00
using X and 2 (x(-1)
(>(=1)
to get
[(0).
Compute the sum 7(4) = Σx
-== % and the relative norm
p(+)) || (x) 2(a-1)| /|| /|| for 2. We will break the loop if r()) < c.
(Assignments to covariance matrices) Use formula in Lemma 3 to get covariance matrices
Σ(1) ...... x( from z(²) ..... 2/22
Perform the spectral decomposition (S) and take the top eigenvectors (ū).... uk).
Run K-means clustering on (i1...., 0x) and extract the cluster labels GI...., Gx as partition estimate for [n].
Output: A partition estimate GI,..., GK for [n]]
3.3 CONNECTIONS BETWEEN ILA-SDP AND EM ALGORIHTM
It is interesting to observe that our proposed il ILA-SDP algorithm is closely connected to the classic EM algorithm, which approximates the maximum likelihood estimation (MLE) of the observed data in statistical models with latent variables (Dempster et al., 1977). The key idea of EM algorithm in the model-based clustering context is data augmentation where the latent variables represent the cluster labels. More specifically, for each data point X; E RP, we associate with an unobserved one-hot encoded cluster label Y; := {Yi....,, YiR) E (0, Then the EM algorithm aims to iteratively maximize the expected log-likelihood of the complete data (X,, Y:) given by
g(t+1) arg max {Q(0) ((() Ex=g(11x,00)) (e X, Y
(14)
where 9 ((TK: Hkk Σx k=1) contains parameters in the GMM,
K are the weight parameters such that T > O and Σ k=l Th = 1, and the complete log-likelihood function
(
K
(0 X,Y)
Ys [log(2r[Ex)) - (xx-)))) T
(X, (%X,-)4)]
Am
Alternatively, the EM algorithm 14 can be interpreted as minorize-maximization (MM) that maxi- mizes best lower bound for the likelikeli of the observed data
p(X, Y 0) =: C(g. 8)
{(0) X) == log X 0) = log Σ X.
0) > Σ
Y
X) log
q x)
7

---

Under review as a conference paper at ICLR 2023
Mis-clustaring error when changes
Mis-clustering error when a changes
0.7
S0P
0.35
LA-SDP
KM
Mis-cluteriig eror
0.5
ASSPP
0.25
0.2
0.3
0.15
0.2
0.05
0
0.4
Figure 3: Mis-clustering error (with shaded eπror bars) vs (captures the signal strength of GMM) and a (perturbation percentage of initialization). mEM (SDP) refers to the reduced version of EM (LA-SDP) where we consider covariance matrices as fixed and equal to identity. The first plot compares the performance of mEM and SDP when separation is large with random initialization; the second plot compares all methods when we enlarge the perturbation percentage applied to the random initialization from hierarchical clustering (HC).
for any posterior distribution q( Y X). Under this perspective, the EM algorithm can be expressed as an alternating maximizationn algorithm on L(q, 0) between E-step g ((++1) = arg max, L(g, 0(0)
and M-step 1(++1) = arg max, [(g 0). Thus, give any q X), the M-step maximizes the
expected complete log- likelihood as surrogate function that minorizes {(O X) because L(q.
(r)g((1x)(ffll X, Y)] H ( Y X)) where Hg)) denotes the relative entropy of distribution
q. while given the current parameter estimate g(t), the E-step is maximized at q
((+++) (
p(Y | ((1) because
e(0(t) Xx) > C(p(Y X, 0(")),00))
Σ P(Y X, (())) logp(X ((0) = E(((() X),
where the first inequality is actually an equality at p X, (())). Even though the EM and iLA-SDP are both alternating maximization algorithms aiming to solve the MLE for the observed data log- likelihood and both can be viewed soft clustering methods (cf. Proposition 4 there are several important differences we would like to highlight.
First, cluster labels are (random) latent variables and they are estimated via posterior prohabilities in the EM algorithm, while the labels are treated as unknown parameters in iLA-SDP that are estimated via direct maximization of the observed data likelihood.
Second, the EM algorithm is special case of the minorization-maxinne &Lange 2000 by iteratively performing the coordinate ascent on the expected complete data log- likelihood minorizing surrogate function, while our iLA-SDP is exact in the sense that it directly optimizes the observed data log-likelihood via a convex relaxation formulation. Thus iLA-SDP is a more direct approach than EM for tackling the non-convex observed log-likelihood objective and it is principled to perfectly recover the true clustering structure the clusters are well-separated under an SNR lower bound in Theorem 2 As in the EM algorithm, iLA-SDP monotonically maximizes the observed data log-likelihood over iterations; of. Figure 7 in Appendix.
(MM) algorithm (Hunter
) Third, the EM algorithm in each iteration must estimate the cluster center parameters (pn) k= while our iLA-SDP profiles out the effect of centroid estimation and leverages only pairwise Mahalanobis distances between data to accommodate the heterogeneity of cluster shapes. Partly because the error in estimating the centroids propagates to other parameters, EM is more sensitive to initialization with inaccurate labels and the centroid configurations even in the standard GMM {Jin al. (016)). and ILA-SDP behaves better than EM, an observation we empirically verify in our simulation experiments; cf. Figure 3 for comparison between iLA-SPD and EM algorithms. From the first plot we can observe that LA-SDP with isotropic known covariance matrices, which reduces to the K- K-means SDP in (5), performs stable and achieves exact recovery when the separation is large. However, EM fails with random initialization in this adversarial centroids configuration. Moreover, from the second figure can see that LA-SDP is fairly stable with perturbation of initialization if the separation is large while EM can go worse the perturbation percentage of initialization a approaches 1, i.e., all the labels
8

---

Under review as a conference paper at ICLR 2023
Difference of mis-clustering error to iLA-SDP
Difference of mis-clustering emrr to iLA-SDP
Difference of mis cluustrruing errr
Difference of mis cl----uuuttnnngg erro
0.3
Noa
02-
0.1
-02
-0.5
4.3
H0
EM
I.A-SPP
KM
SDP
H0
FM
1.4.5DP
KM
SDP
SC
Figure 4: Box plots of difference of mis-clustering error (with means) for different methods to iLA-SDP. The left (right) plot summarizes the results for the banknote authentication dataset (landsat satellite dataset). Here SC refers to spectral clustering method.
are selected randomly. In other words, EM is more sensitive to initialization and iL.A-SDP is more stable if the signal is strong. More details of the settings in Figure can be found in Appendix A.3
4 REAL-DATA APPLICATIONS
In this section, we test the performance of iLA-SDP against several widely used clustering methods on two real datasets from the UCI machine learning repository.
Banknote authentication dataset. We first look at the performances of our methods for a banknote authentication dataset where the separation of clusters are large. The images were taken from genuine and forged banknote-like specimens, where the features were extracted by Wavelet Transform tool. It contains 1372 samples and p 4 attributes with K 2 clusters. We choose total 1000 samples randomly and equally from two clusters to make cluster sizes balanced among total 200 replicates. HC is used as initialization for EM, KM and iLA-SDP. If the initialization for the assignments ( (())) is highly unbalanced, i.e., ||G| | > if ||G > ||G|"), []. then the covariance matrices of two clusters should differ significantly and calculate the covariance estimation for unconstrained optimization problem; Otherwise we will calculate the estimation of the covariance matrices through graphical lasso with parameter À since the similarity shows that we could reduce the estimation of parameters. Then we run Algorithm to get the results for iLA-SDP. The comparison of those four methods can be found from the left plot in Figure where can observe that the separation between two clusters is not well in the sense that the medians of all methods are similar. Nevertheless, iLA-SDP can achieve better performance than other methods achieve
for most of the time. The reason iLA-SDP has lower mean is that sometimes iLA-SDP nearly exact recovery in the sense that there are 36 out of 200 times when the mis-clustering error for iLA-SDP are below 0.05, while EM can only hit around 0.25. This indicates that there more chances for iLD-SDP to sort two clusters with fairly good performance even the separation is problematic.
Landsat satellite dataset. This database was generated from landsat Multi-Spectral Scanner image data. The test set includes 2000 satellite images, 6 different clusters with 36 attributes (36 spectral bands x 9 pixels in neighbourhood), Every attribute is an integer from 0 to 255 indicating the color for certain pixel. We performed 4 methods on the transformed dataset with total 300 replicates, For each repetition, we draw total n = 1200 samples randomly and equally from clusters to make cluster sizes balanced. And for each attribute, we scale its range to (0, 1] and then take the function f(x) log(1 1/xx= 1) entry-wise to transform the range to R.. Then, we run Algorithm on the transformed dataset X to get the results for iLA-SDP with
10-2 Po K 6, and R = 50. From the results we can see that our method iLA-SDP performs the best for all four methods. Especially, ILA-SDP out-performs EM since the initialization (HC) is rough, which results in both biases the estimations of group means and covariance matrices for EM while iLA-SDP only uses the group covariance matrices as its initialization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dimitris Achlioptas and Frank McSherry. On spectral learning of mixtures of distributions. In Peter Auer and Ron Meir (eds.), Learning Theory, pp. 458 458-469, Berlin, Heidelberg. 2005. Springer Berlin Heidelberg, ISBN 378-3-540-31892-77
Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. Np-hardness of cuclidean sum-of- squares clustering. Machine learning, 75(2):245-248, 2009.
David Arthur and Sergei Vassilvitskii. K-means++: The advantages of careful seeding. In Proceed- ings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '07, pp. 1027-1035, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245.
Sivaraman Balakrishnan, Martin J. Wainwright, and Bin Yu. Statistical guarantees for the EM algorithm: From population to sample- based analysis. The Annals of Statistics,, 45(1):77 120, 2017. doi: 10.1214/1--AOS URL httpps://dolt org
0ttp://:///eii..11111112222. 11t.110..1111111 A081435
Florentina Bunea, Christophe Giraud, Martin Royer, and Nicolas Verzelen. PECOK: a convex optimization approach
variable clustering. 1rXiv:1606.05100 2016.
Xiaohui Chen and Yun Yang. Diffusion k-means clustering on manifolds: Provable exact recovery via semidefinite relaxations. Applied and Computational Harmonic Analysis, 52:303-347, 2021a. ISSN 1063-5203.
Xiaohui Chen and Yun Yang. Cutoff for exact recovery of gaussian mixture models. IEEE Transac- tions on Information Theory, 67(6):4223-422) 4238, 2021b.
Sanjoy Dasgupta. The hardness of k-means clustering. Technical Report CS2007-0890, University of California, San Diego, 2007.
A. P. Dempster, N. Laird, and D. B. Rubin. Maximum likelihood from incom- plete data via the em algorithm. Journal of the Royal Statistccul Society: Series B (Methodological) 1-)1) 1977. doi: htpps//oi.argg/01111//111772 tb01600.x. URL https: rss www.oonwwoarr.....o.......... wiley com. doi htsei//abs//ii httpp//awww...lllloo...coo/////a 2517- 1 1977 tb01600..0600 1977
Yingjic Fei and Yudong Chen. Hidden integrality of sdp relaxation for sub- tab-gaussian mixture models. a8X3::1803.065 10, 2018.
Chris Fraley and Adrian E Raftery. Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association, 97(458):611-631, 2002. doi: 10.1 10.1198/ 016214502760047131, URL
Christophe Giraud and Nicolas Verzelen. Partial recovery bounds for clustering with the relaxed kmeans. meei:1807.00444 2018
M. Girvan and M. E. J. Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences, 99(12);7821 2002. doi: 10. 1073/0m7/ 122653799. URL httph:: //ww. pr... orw.aas.rr//oooo wtt.://////ooo//bo/.eoooo
abs/ 10 1073/ 122653799
David R. Hunter and Kenneth Lange. Quantile regression via an mm algorithm. Journal of Computa- 10.1000000111000000000 200...0000..0........... tional and Graphical Statistics, 9(1):60 77, 2000 doi: 10.1080/1061
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin J. Wainwright, and Michael I. Jordan. Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences. Proceedings of the 30th International Conference on Neural Information Processing Systems. NIPS' 16, pp. 4123 4131. Red Hook, NY. USA, 2016. Curran Associates Inc.
ISBN 97815 10838898
Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1950, June 2010. doi: 10.1 20.11099..2.20005555520
10

---

Under review as a conference paper at ICLR 2023
Md Rezaul Karim, Oya Beyan, Achille Zappa, Ivan G Costa, Dietrich Rebholz-Schuhmann, Michael Cochez, and Stefan Decker. Deep learning- based clustering approaches for bioinformatics. Brief- ings in Bioinformattis, 22(1):333 415, 02 2020. ISSN 1477-4054. doi: 10.1093/bib/bbzz17. URL
https: //oii .org www.//////bbbbb/ http://bbbbbbb////
Xiaodong Li, Yang Li, Shuyang Ling, Thomas Stohmer, and Ke Wei. When do birds of a feather flock together? k-means, proximity, and conic programming. arXiv:1710.06008, 2017.
Stuart Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28: 129-137, 1982.
Yu Lu and Harrison Zhou. Statistical and computational guarantees of lloyd's algorithm and its variants. arXiv:16/2.02099, 2016.
J.B. MacQueen. Some methods for classification and analysis of multivariate observations. Proc. Fifth Berkeley Sympos. Math. Statist. and Probability. pp. 281 281-297, 1967.
Meena Mahajan, Prajakta Nimbhorkar, and Kasturi Varadarajan. The planar k-means problem is np-hard. In Sandip Das and Ryuhei Uehara (eds.), WALCOM: Algorithms and Compulation, pp. 274-285, Berlin, Heidelberg. 2009. Springer Berlin Heidelberg. ISBN 978-3-642--00202-1.
Marina Meila and Jianbo Shi. Learning segmentation by random walks. In In Advances in Neural Information Processing Systems, pp. 873 973-879. MIT Press, 2001.
Dustin G Mixon, Soledad Villar, and Rachel Ward. Clustering subgaussian mixtures by semidefinite programming. arXiv: 1000: 2v2, 2016.
Andrew Y. Ng. Michael I. Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm.
In Advances in Neural Information Processing Systems, pp. 849-856. MIT Press, 2001.
Jiming Peng and Yu Wei. Approximating k-means-type clustering via semidefinite programming.
SIAM J. OPTM,, ((1))188-2008 2007.
Martin Royer. Adaptive clustering through semidefinite programming. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30. pp. 1795-1803. Curran Associates, Inc., 2017.
Jianbo Shi and Malik. Normalized cuts and image segmentation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 22(8):888-900 Aug 2000. ISSN 1939-3539. doi: 10. 868688.
Santosh Vempala and Grant Wang. A spectral algorithm for learning mixture models. J. Comput. Syst. Sci, 68:2004, 2004.
Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4)3395-416, 2007.
Ulrike von Luxburg, Mikhail Belkin, and Olivier Bousquet. Consistency of spectral clustering. Annals of Statistics, 36(2):555-586, 2008
Yihong Wu and Harrison H. Zhou. Randomly initialized em algorithm for two-component gaussian mixture achieves near optimality in on iterations, 2019. URL htt.tpa://aaxiii.o////////
1908. 10935
Yubo Zhuang, Xiaohui Chen, and Yun Yang. Wasserstein k-means for clustering probability distri- butions. In Proceedings of Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Yubo Zhuang, Xiaohui Chen, and Yun Yang. Sketch-and-lift: scalable subsampled semidefinite program for k-means clustering. In Gustau Camps-Vall Francisco R. Ruiz, and Isabel Valera (eds.). Proceedings of The 25th International Conference Artificial Intelligence and Statistics (AISTATS), volume 151 of Proceedings of Machine Learning Research, pp. 9214-9246. PMLR, 28-30 Mar 2022b.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 FURTHER RESPONSE TO REVIEWERS
Sample complexity bound. To verify the sample complexity bound for LA-SIPP in Theorem (O(log(n))) is tight, we will change n and adjust the squared distance between clusters by multiplying log(n). More precisely, we let d = Avlog(n). X > O. The diagonal of the covariance matrices are placed at a simplex of IP that are not identical to the corresponding centers. i.e. j4 A ek+ This guarantees the symmetry of the construction. We set 10, p 4, K 4. Each time we draw the 120/240/ results of the simulation for the second plot in Figure are obtained through 20 total replicates, where we can observe the same pattern across different settings for n.. This shows that the order log(n) for separation bound in Theorem 2 should be tight.
Σg L. deagdiagsss (x+1), AA E [K] where ex+1
00////00448 /480 data from the GMM. The
Mis-clustering error when A changes for different
0.4
n-120
0.35
n=240
n-480
Mis his-clustering emar
0.25
0.2
0.15
0.05
1.5
2
2.5 X
3.5
Figure 5: Mis-clustering error (with shaded error bars for the left plot) vs À for iLA-SDP for different n.
Computational complexity for banknote authentication dataset. Now if we look at the results of time cost for clustering banknote authentication dataset in Table we can observe that the time cost for iLA-SDP is relatively high and to reduce the time cost, we could consider sub-sampling methods, e.g., the subsampling idea (Zhuang al. 20226). This will be set as our future goal.
Table 1: Time cost (SD) for clustering banknote authentication dataset for 20 replicates.
EM
KM
iLA-SDP
SC
0.1 (0.0853)
0.0013 (0.0013)
2100 (1882) 0.0395 (0.0959)
A.2 ENHANCED ILA-SDPS FOR HIGH-DIMENSIONAL. AND LARGE-SIZE DATA
In this section, propose two variations of iLA-SDP that can handle high- dimenssiin and large-size data with better computational and statistical efficiency.
High dimensional data. If the number attributes of the data are large, would be hard to approximate the true covariance matrices since there are O(p²) many unknown parameters. Thus, we propose two dimension reduction procedures that based hierarchical clustering, Fisher's LDA and F-test. The detailed algorithm have been shown in Algorithm 2 and Algorithm To reduce the dimension, we proposed two procedure.
1. If the number clusters K is small and the difference between centers are sparse, we shall use HC as benchmark method for feature selection and assume the group means according to HC
12

---

Under review as a conference paper at ICLR 2023
ground true. Specifically, for i-th attribute, we caleulate the F-statistics and its p-value based on the Hy that all group means w.r.t. i-th attribute are the same. At last, each attribute would likely to be selected if the p-value P; for i-th attribute is significantly small among p p-values for all attributes.
First we use the hierarchical clustering to get the clustering results for all possible input cluster number K E [p]. we assume all the clusters have identical covariance matrices, then we may use the assignments from HC to estimate the within-cluster covariance W (with group means (ny) and get the signal-to-noise ratio A == mink4 WW/1 - --)ll- Here, HC serves as a benchmark method for data initial processing. We will then choose the largest K within target range such that the signal-to-noise ratio A(K) (') is maximized. Then it will lead to the new dataset with dimension K after running Fisher's LDA on the assignments from HC with clusters number equals K. Finally we perform Algorithm | on the new dataset and extract the cluster labels.
Large-size data. As we know that the time complexity for solving SDP is high as ((n3-)). We might use subsampling methods to bring down the time cost while maintain the superior behavior for LA-SDP (Zhuang al. 2022b). The proposed algorithm is shown in Algorithm 4
Algorithm 2: Likelihood adjusted SDP based iterative algorithm with unknown covariance matrices Ei,..., Ex for large p.
Input: Data matrix X E Rpx= containing n points. Cluster numbers K. The stopping criterion parameters po. and S. a E [0, C'> 0.
Run hierarchical clustering with data X, clusters number K and extract the cluster labels
G(D) ...., G prior assignments for [n]. Suppose the assignments have true centers
Pk. A E [K].
for ,, do
(0) (0) = ... - |K where Pk.i Calculate the p-value P; of the F-test Fi under Ho: Mi,
corresponds to the i-th component of μi
Keep Po attributes with po smallest p- p-values P..
if there is no clear cutoff between P,; 's, i.e. max Pi/ min Pi < C, then
L we further keep other Po attributes with probability a 0.
Get dimension reduced data X.
Run Algorithm on X with initialization obtained from K clusters of HC and stopping criterion parameters c and S. Then extract the cluster labels GI,... GR as a partition estimate for [n].
Output: A partition estimate c,,..., Gx for [n]]
A.3 EXPERIMENT RESULTS
In this section, we provide more details of the settings and post the results for simulation experiments. For all the dimension reduction procedures used in the simulation experiments,
perform step 1-7 in Algorithm followed by Algorithm with input parameters a = 0.7, C = =00 10 Pu = 2K, P1 = 13 = 10 S 50. The initialization use is hierarchical clustering from mclust package in R. Here we test our algorithm on Gaussian mixture models and real datasets. We compared our algorithm iLA-SDP (HC initialization) with HC, EM algorithm (HC initialization), K-means (HC as initialization) and original SDP.
Improvements of iLA-SDP over SDP. Recall in Theorem 2 we define the signal-to-noiss ratio as a == min | 1/2 1/2(p) - 2. To verify the validity of the definition and compare iLA-SDP and SDP, we change the conditional number for covariance matrices Σ, ....... Σx. Here choose n = 200, p 4, K = 4. Recall M := max || 1 we choose all the covariance matrices to be the same such that M is fixed. The covariance matrices set to be identity matrix except that the first entry at the diagonal are set to be + 1, which refers to the condition number of matrices. We consider two cases where T. = 10, 100. Now denote ea IRP the vector with k-th
13

---

Under review as a conference paper at ICLR 2023
Algorithm 3: Likelihood adjusted SDP based iterative algorithm with unknown covariance
matrices Ei...., EK for large p.
Input: Data matrix X E _xx= containing n points. Cluster numbers K. The stopping criterion
parameters P1, and S.
Select a bench mark clustering method (HC) as a way to provide a prior assignments.
for K = K, L..... P1 do
Run hierarchical clustering with data X, clusters number K and extract the cluster labels
G ..... G(R) ..k.,, as prior assignments for [n] and get the group means Me (K) E [K].
Calculate the within-cluster covariance matrix W, then get the signal-to-noise ratio (K)
A(K) : miny ||W (R) P (1)
Choose K such that A(K*) is maximized for K K,K + 1........ P-
Perform the Fisher's 's LDA with data X, assignments G (KK) ...., Gk and get the transformed
data X E R with = K" 11 1.
Run Algorithm D on X with initialization obtained from K clusters of HC and stopping criterion parameters and S. Then extract the cluster labels Gl...., GK as a partition estimate for [n]. Output: A partition estimate ,I...., Gx for [n]]
Algorithm 4: Sketch and lift: Likelihood adjusted SDP based iterative algorithm with unknown covariance matrices Σ ..... Σx for large
Input: Data matrix X E R containing points. Cluster numbers K. The stopping criterion parameters P, and S. Sampling weights (Wi,..., w.) with
W1
Wn, 7 (0, 1) being the subsampling factor.
(Sketch) Independent sample an index subset T c: [n]] via Ber(w,) and store the subsampled data matrix = (XT)
Run subroutine Algorithm| I with input V to get a partition estimate Ri...., , fxx for T.
Compute the centroids X = |R -1 Σ ji Xj and within-group sample covariance matrices
Σg = ||++1~1 Rj (X, Xx) Xx)² for & E [K].
(Lift) For each E [n]] T, assign Gx if
log |Σg] 1/2 (X; Xx) < log EE + +1 1/ X.1I3 VI + k, le [K]. And
randomly assign to any K clusters if such doesn't 't exist.
Output: A partition estimate GI...., .......Gr GK for [n].
entry as 1, and O otherwise. The centers of clusters p1,-.., Ma are placed on vertices of a regular simplex, i.c.,
A/1++ (1 + Te,, E [K]. This ensures that for any L, A = X, Vλ. From Figures / we can observe that the signal-to-noise ratio we defined is reasonable. On the other hand, the performance of SDP becomes worse as condition number of the group covariance matrices grows since the assumption of isotropy group covariance matrices for SDP violated and same reason for K K-means.
Impact of dimension reduction. Here we want to see the performance of iLA-SDP after dimension reduction. 'he covariance matrices of GMM are drawn independently following Σg U Vk E [K]. Here Us is random orthogonal matrix, A diagonal matrix with entries drawn from Z = + BZ 1(2 0), where Z is standard Gaussian distribution, 3 0 controls the condition number of Σ.. Here we choose n = 200, p 20, K = 4, 3 5. The covariance matrices fixed once chosen and perform Algorithm on the dataset directly to get the results of iLA-SDP for each replicates. For dimension reduction, we follow the procedure of dimension reduction introduced in Algorithm 2 and Algorithm 3 in Appendix A.2 and get the transformed dataset X with lower dimension. Then the results of pLA-SDP is obtained from running Algorithm with HC as initialization on X. The results in Figure 6 shows that after reduction of dimension in our procedure, the performance of iLA-SDP becomes significantly better when the separation is large. This is because in setting, the difference hetween centers d(k == 14k M4, is
14

---

Under review as a conference paper at ICLR 2023
sparse for all distinct pairs. And after performing the F-test on the covariates, the noisy terms get eliminated which results in better performance.
Failure of EM SDP. Recall the failure of EM for random initialization (Jin al. 2016) in the special case that covariance matrices equal to identity matrix and it assumes equal weights. Both covariance matrices and weights known. In this case, EM algorithm would be reduced to the version that the weights and the mean update interactively. Meanwhile, iLA-SDP would be reduced SDP. The random initialization indicates that we pick any data point initialization of the centers uniformly. Following the same setting from the construction of the pitfall, we choose one dimension GMM with three clusters such that the distance between others. More concisely, we let = 300, K = 3, results can be observed from the first plot in Figure with 300 replicates, where we denote the reduced version of EM as mEM. From the figure we can observe that the reduced version of iLA-SDP, which SDP, performs stable and achieves exact recovery when the separation is large. However, EM would fail for random initialization.
of the centers is much smaller than 1. A11 7 H2 -%, H3 10 % The
Perturbation initialization assignments. To see how the performance of EM and iLA-SDP will change when perturbing the initialization, we set HC initialization and proportion (ax E [0, I) of the initialization labels will be perturbed. The diagonal of the covariance matrices are placed at simplex of I² that are not identical to the corresponding centers. i.c. μ4 X Eg. Σx J. diag(e:=aaa (c++1), vl E [K], where cx+1 = 61- This guarantees the symmetry of the construction. We set L = 10, p 4, K = and the distance between centers d 8. Each time draw the 200 data from the GMM and run HC as initialization. Then we randomly assign a proportion of the labels from HC to any cluster uniformly. The results of the simulation for the second plot in Figure are obtained through 300 total replicates, where we can observe that iLA-SDP is fairly stable with perturbation of initialization if the separation is large while EM can go worse as approaches i.e., all the labels are selected randomly. In other words, EM is more sensitive to initialization and iLA-SDP more stable if the signal is strong.
Empirical evidence for monotone increasing of objective function for iLA-SDP. Here we provide examples based on previous experiment settings where we set the distance between centers d 1/3/5/10. and try to how the log-likelihood function of given data changes as the iteration proceeds. From Figure Appendix we can see that our algorithm guarantees that the log-likelihood function of given data increases over iteration empirically. What is more, by construction show that the log- ligliikll function will increase after each step for iLA-SDP theoretically.
Mis-clustering when D changes
0.7
-LA-SDP
PLA-SDP
0.6
0.5
Mis-clustering enror
0.4
0.3
0.2
0.1
0
4
8
10
12
14
16
D
Figure 6: Mis-clustering error (with shaded error bars for the left plot) center distance D for iLA-SDP before and after dimension reduction. pLA-SDP denotes the iLA-SDP after dimension reduction.
15

---

Under review as a conference paper at ICLR 2023
log-likellhood when iteration grows
-200
-400
-460
-480
lby ikellhood
-500
-520
-540
-560
550
10
15
25
Figure 7: liglliillhhod (up to some constant) iteration grows for iDA
A.4 PROOF OF THE THEOREMS AND PROPOSITIONS
In this section, we provide the proofs for the Proposition Proposition 4 and sketch proof of Theorem 2 The proof of the main theorem follows the track from the paper solving the exact recovery for original SDP (Chen Yang 202Tb) and we will show the main differences in our proof.
16

---

Under review as a conference paper at ICLR 2023
First, we provide explicit expressions of some constants appearing in Theorem 2
below:
4(1++0)
M + M + (1-8) + CsR
E1 (1 (1-B)ar7
(1 5) log
with
(1 (1- (1-
VVllog
log n2 n
Rn
(1 + 8) log
and
Cs(M - M
logn +1) 1) + CoK2(1 - 3)
F2
(1 (1-8)(1 7))
(15)
n
log
p
(M -1)M²
(Vir Voogg +
pd
n
min
+
log
8(M 1)a m
log n2
logn'
8
m
A.4.1 PROOF PROPOSITION
Proposition (SDP relaxation for K-means a special case of LA-SDP). Suppose E3 = Id for all & E [K] Let 2 be the solution to (5) that achieves maximum M1 and Zz, k = 1..... K, be the solution with maximum M2. Then M1 M2. And 2 = - Zn, if 2 is unique in 5)
K
Proof of Proposition If Σ = ldld.. vk [K]. Then from 7 we have
A = 2 [iag(( x x'1 + di (xtiaagX""X"" x x, Vk E [K].
This implies that can be written as
K Xx'X, (Σ 7x) k=1
2,,- 2x = arg
max
Z ... Zx ER
(16)
subject to Zx 2 0, tr
2x) = K,
Zn 1n Zx 0, v k [K],
k=1
k=1
Since diag X (Σk Zz)
ur(x")) which is constant in the optimization problem
16 Now suppose is solution to that achieves maximum Mi and 2u, k 1,.... K, is the solution to (16) that achieves maximum M2, then we have
K
X
X,
Zx) S
M1,
k=1
x X.
Zx)) < M2,
where %,
2, 12
2x 0. In other words, M1 = M2, which finishes the proof. If 2 is
unique in 5 then we have 2 = ΣK kk= Zx since both of them achieve the maximum in 5
A.4.2 PROOF OF PROPOSITION 4
Proposition (iLA-SDP is soft clustering method). If (Zz) 1, then there exists weights (Wk,1y... Wk,n) such that Eg in Lemma can be written
1
1
Σg =
Wk (X, -xx) - (N)T with A = n
Wp. X
(17)
nh 1=1
t=l
where ng Ei= Wk.i.
17

---

Under review as a conference paper at ICLR 2023
Proof of Proposition 4 If Zi is rank 1, then there exists R such that Z₁ aa". Let wy Q"1 1 a, then we have
2x
W 1
i.e., Zn,ūj
we Wk.s Finally, by plugging in the expression of Zk.i with Wk,i can get the target Σi-1 wa,s
expression for Ex.
A.4.3 SKETCH PROOF OF THEOREM 2
Theorem
2 (Exact recovery for LA-SDP). Suppose there exist constants 8 and (0,1) such
that
(1-3)² (1- (1 k2 } Cin 8 83
C2M2/2
4(1 + +)
logn n 2 max
m 2
82 B max (M - 1)2, m
- (1-3) K
5²
A 2 2 EE E2) log and min (ia,)) 2 Cs(1 + logn/p + p/n): ky1
(18)
where
4(1 4(1+20)
(1 8)2 p
E
M+
M
C4
(1-8)2p²
(1 + 5) m logn
with
(1-B)²
vv Vplogn
log
Rn (1 + ++))log 8)log n
12
and
Cs(M -
M
CBK2(1 -3)
E2 (1 (1-3)(1-)) (1)
logn + 1
+
3
(19)
minn
1
n
1
log p
p log n
(M
1)M²
(Vier +
Vplogn
n
3(M - 1)2 m log n
8
m
then the LA-SDP achieves exact recovery, or 2 = Z with probability at least some universal constants CI....,,y; C7.
Cr n for
Sketch of the proof. Recall that we let Gi...., Gj be the true partition of the index set [n] [I) such that if i E G then
= μp + Eii
(20)
where S Rp is the true center of the k-th cluster G (G₃ for simplicity) and & is i.i.d. random Gaussian noise N(, Ex). First we can write down the dual problem:
1 min AK + a In, subject to B 2 O, Ald,, + / (a1 + lnaa) - A& B& 2 0, Ak E [K]. XER, BRE
Denote - Te 1G. 16 Vk E [K] then can be shown that the sufficient conditions for the
solution of SDP to be 7x= Zi, Vk [K]
B₁ 2 0:
(CI)
W4 = Md, + 2
(a1 + +1,aa
A₁ B₁ 2 0;
(C2)
tr(W22?) 0;
(C3)
tr(BaZt) 0.
(C4)
be verified that if can find symmetric Bş such that
BLGG, = 0;
18

---

Under review as a conference paper at ICLR 2023
nk + nM
[B,GG lo,l:
A
2nd
Tuk [[12 (Xx -Xx- x,)] log Ex() (1 x.) log [>]]] +
2
[BG,,, |A, lailj [AR,C,G,
BR.G, Gili = G Lal + [A. (AR.
for any triple pairs (k: (.) that are mutually distinct and E Gk. j E Gt. Then (C3) and (C4) hold. In fact, the target matrices can be defined through
Bi. == BRGG Iai Br,G,
(21)
G Be,
for any k E [K], (1,1) # k). Furthermore, the construction BB shows that Bgla. = 0, V(k, pairs.
The following two lemma gives the sufficient conditions for (C1). Lemma 6 (Separation bound on the covariance matrices). Let A Ap correspond to the eigenvalues of (- Ilpp and define D(k.t) >T (A: [o](1+))) there exists constant C' such that
pmax, [A,]
min D(K.)) 2 C(1 + logn/p p/T), kg1
then
PP P(AA,,,,, - [Ak,G,, Lc,l 2 0, for all (k,1) [K]² and E Gi) 2 - CK2/n
Lemma') (Separation bound the centers). Let 8 0, e (0, 1), 1 E (0, If we have
4(1 + 8)M2 82
M3/2 + MM+ (1 (1- 8)2M p 2 (lgg(kk)) + log(kk) m logn
A
logn,
7)
(1+
and
MM 1)2
2 2 (1 () 17)
1+ 2(1 - B)( 11
(3 log M 4M( 1)) 2V1(=+2/plonnn (log(nK))]
M
then
11 1/2 1/2 -xi -1)xx,)11 100 [zgl - -x))11 log (2rl)
2 [AI.o,,; |Ar 2
M |اا 1/2 (m ur 11 (nz + ny " )p TR.N
B
mh
for all triple (k,4,1) E [K] with (k,,, # (k: ki k)) and j E Giv)
CK3
no
where
log(nK)
TKLLP"
=
|52 1/2
(14 1 2/
14:) + 2(n + n) 2p log(nk)) + 4nn log(nK).
for some large constant C.
The proof of Lemma 7 follows the similar steps from the original paper (Chen & Yang 2021b). The two lemmas imply that (C1) can hold with high probability the separation condition in the assumption holds. The remaining part is verify the (C2).
19

---

Under review as a conference paper at ICLR 2023
Denote r = spanflo. : k E [K]} be the othogonal complement of the linear space spanned by 1G2, & E [K]. Note that WilG, = 0, v(k. 1) E []], we only need to check for e E r,
v² W' 2 0, Ak [K].
Note that o' 10. O, we have
2 W = 1/1i/² Sx()) Th (v),
where Sk(U) = o. AgU o X Xe, and (v) v Be. By concentration bound we can get
K
P(Sx(0) & (K(M V + v 220gg n), for all k E [K] 2
For Th (v), first we define
(1I k.1 = (2// 2 (mW 14) Σ 1ys):
JEGP
V(2) (ēr E9 1/2
En
E11
Ugtz);
jEGn
(18 =
2 Σ
x (2) - En | / Ej2yi
R,ll"
jEGP
(44 = (AAGG,,, - [A Laib)) 1(1 + 1¹). k.W
JEGU
Then we can write Te(()) as
Tạ(u)= = Σ
k2l + (2(2)
+ (3)
+ (T(4
k,1l' + T
1 Bx
k.ll
k.Wl
1410"
where
T-(1) ((I) V(I)
k.ll"
K."'" KIT
T (:2)) = (222) (22).
T:33 = (1)) (222) + (2) v(1).
k.W
k.10'
ki
(14) = (V + (44
k.1l (V + ((2)
+ (V + (2(2)
) -(V + vil));
T: kJ = (3)
+ (4( (3)
k,ll"
k.Wl k.P + k.r.
Now we choose = p +
4M m
8
which implies that
B41,
ntpp B
[|x (/ (w pr )11. |12, 1/2 - 33
1/2,,1/221
8
M
From concentration bounds for Gaussians have for all triple 2,, [K]3 such that (R:, 1,1') +
(k, k, k).
CM2
Σ b
niy
(1)) kil
S
(n + 2nn logn + 2K 2Klog log n)]l
1 1g
3
CM3
Thrr
(:(2)
(8 mmploon + mplog" n/n)|lu|13
1 B
1
Σ Bipr
((5)
CK2
1 - B
(p + pM logp/ logn) ) + M(M 1))m?
Ba1n
B
(M - 1)2M
Or
n:m
(3) k.ll'
CK2M(1 B)( (-
(V
pmm
log + pm log m)]l((l
121 1"1
3
20

---

Under review as a conference paper at ICLR 2023
with probability 2 - C--KK//r /n' for some constant C.
Note that by assumption we have A z
C(M-1)"M (1-B)[1-1) (p + log n) + (1-ø] /(1 + 8)p log n/m and
CM
the fact that the remaining terms of T can be bounded by the above inequalities up to multiplied by some constant, we can directly verify that (C2) is true under our assumptions.
Lemma 6 (Separation bound on the covariance matrices). Let Ai,.... Ap correspond to the (A)) there exists constant M Idp) and define D(k.)) = pmax, [Aa1
eigenvalues of (E C' such that
min D(k.)) 2 C(1 + logn/p p/n), Agl
then
P(AA,aaialll - AA, 2 0, for all (k,1) [K] and j E Gi) 2 CK2//K/
Sketch of the proof. Let T [A, lab [A,,G,G;llll B = D/ 12 Idp then by definition we have
P
T
Σ log(A: + 1) + Σ A
im1
+
(B, Cg. - -ldp)
B 1 Σ
Σ c)) b))
GES
T + 111 sEG:
TM LEG:
+ $
B, 6E
Idp)
tEG:
where the last three terms can be bounded by concentration bounds for Gaussians.
21


Reviews:
[{"review_id": "qpuq8gKo4g-", "reviewer": "Reviewer_BdFu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends an SDP relaxation for k-means clustering (Gaussian mixture modeling) from isotropic clusters (with identical scaled identity covariances) to non-isotropic clusters, where each cluster can have its own covariance matrix.  When these cluster covariances are unknown (which is most often the case) the paper proposes an iterative EM-like algorithm which alternates between covariance estimation with known cluster-labels, and SDP-relaxed model-based clustering with known covariances.  Limited simulations show improvements w.r.t. k-means, EM and hierarchical clustering.", "summary_of_the_review": "Interesting extension of SDP clustering and its analysis to the model-based clustering setting with non-isotropic covariances.  The proposal is likely only of theoretical interest, as the computational cost is very high.  Also in the practical setting of unknown covariances the method looses its convex-relaxation elegance, and starts looking more like EM that it tries to replace, requiring various heuristics.  Scholarship, and experiments need improvement. \n\n", "strength_and_weaknesses": "Strengths:  The paper proposes an interesting SDP relaxation for model-based clustering with known covariance matrices, and extends the information theoretic analysis from the isotropic case.  In the usual case where within-cluster covariances are now known,  the paper proposes an iterative approach which alternates between covariance estimation with known clusters, and SDP-relaxad covariance estimation with known covariances.  The approach is limited to small data (timing and computational complexity is not discussed in the paper, but SDP is very expensive and the largest simulation has 1000 data-points) -- but perhaps there could be critical small-data problems where any gains in clustering quality could be worth the computational cost.  The appendix mentions that it may be possible to extend the method to large data (larger dimensions, and more samples) -- but it's unclear how valuable/practical that is -- and it uses standard ideas like sub-sampling. \n\nWeaknesses:\n1) Computational complexity or timing is not discussed in the paper (which should definitely be addressed) -- and it's likely very high.  The authors mention SDP solvers to be O(N^3.5), and the relevant dimension N seems to be n*K  where n is data-points and K is the number of clusters. So it's useful at best for very small scale problems. When covariances are unknown -- the method is iterative, solving many such SDPs, and no information is provided of how many iterations are typically needed until progress slows down. It's worth reporting the timing of the proposed method and contrast with standard k-mean + k-means++ / EM -- to know the price to pay for the clustering accuracy improvements.  \n\n2) In the common case of unknown covariances -- the method appears more akin to EM, which the authors aim to avoid in the first place.  (There's discussion of why it still may be better than EM -- but it's not very convincing -- definitely the elegance of one-shot SDP is lost). Convergence or rates are no longer guaranteed. Also in practice the method uses a lot of other clustering methods (sometimes HC is used for initialization, and spectral-clustering and k-means are used for post-processing non-integral partitioning solutions) -- so it seems quite sensitive to initialization.  So while it's discussed as a clean one-step SDP relaxation -- in practice it's quite complex and uses various heuristics. \n\n3) Experiments do show an improvement -- but they are on the weaker side, and important details are not provided.  Did you use k-means++ initialization for k-means, and something like that for EM?  What is the computational cost of your method w.r.t. alternatives?  If k-means runs say 1000 times faster -- you could probably use this budget for random restarts, some more accurate heuristic initialization schemes -- how would it compare then?  Other common clustering methods (e.g. spectral clustering are not evaluated), even though they're used in post-processing of SDP solutions.  In figure 4 -- the bulk of the box-plot for LA-SDP is in-line with other methods,  but it has a small set of outliers which do better -- this seems an artifact of this specific problem, and perhaps better initialization for k-means could find these configurations. Ideally more than 2 datasets would be used.  Mis-clustering error is not defined in the body of the paper.  Also some discussion in the short experimental section is confusing. You subsample clusters to keep them 'balanced' -- but there's a discussion of 'highly unbalanced'...  \n3-a) \"Perturbation percentage of initaliation\" is not defined. \n3-b) Also why do you need graphical Lasso when you have 4 attributes?  Plain sample covariance should work very well even with relatively few samples that you have. \n\n4) Insufficient scholarship.  It's unclear where exactly the SDP k-means method was proposed -- the main citation is from 2021, but it seems to have a much longer history, including other earlier paper analyzing its performance, and many relevant references are missing. If there are many qualitatively different SDP relaxations -- then they should be mentioned and contrasted, otherwise key papers introducing and analyzing the method should also be mentioned.  Or does your paper propose a new isotropic SDP? \n* Iguchi, Takayuki, et al. \"On the tightness of an SDP relaxation of k-means.\", 2015. \n* Moses Charikar, Rachel Ward, Awasthi, Pranjal, et al. \"Relax, no need to round: Integrality of clustering formulations.\" Proc. Conference on Innovations in Theoretical Computer Science. 2015.\n\n5) The paper claims that profiling out the cluster-centers (i.e. not keeping them explicit) is a major advantage.  It's a key step in the SDP formulation, but once you have a (soft) cluster assignment -- cluster centers are trivially computed, why is it so important?\n\n 6) Term \"profiled-out\" commonly used in the paper is never defined -- setting nuisance parameters to their maximum-likelihood values (rather than being integrated out).  \n\n\n\n\n\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LFjYG9CjOF", "reviewer": "Reviewer_FBAZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends an SDP approach for Gaussian mixture models clustering by introducing cluster labels as model parameters.", "summary_of_the_review": "The idea of the paper is interesting, but the lack of experimental results with popular and successful clustering methods diminishes the paper's overall contribution to the community.", "strength_and_weaknesses": "Strength\n======\n\n- The paper is well written, except for the references/citation style. It's really hard to find the papers that you cited in the main manuscript in the References section as citations are done with last name, e.g., (Balakrishnan et al 2017) but in the Reference section they start with the author's first name, e.g., \"Sivaraman Balakrishnan, Martin J. Wainwright...\"\n- The idea of optimizing cluster labels, while not novel (see, e.g., \"Discrete Optimal Graph Clustering\" by\nYudong Han et al), is interesting.\n\nWeaknesses\n=========\n\n- Experiments with real data are very limited and do not consider classical methods for clustering such as Spectral Clustering.\n\nMinor comments\n=============\n\n- This sentence: \"Another popular clustering method is the classic expectation-maximization (EM) algorithm\" does not make sense. EM is not a clustering method. EM is an iterative algorithm for finding local minima of (generally) maximum likelihood problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5rGHxFy5ezJ", "reviewer": "Reviewer_cBBj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of clustering samples generated from multiple Gaussian mixtures with possibly non-identical and non-isotropic covariance matrices. The authors start with a mixed integer program (MIP) take the standard semidefinite program (SDP) relaxation approach. They first consider an oracle model, in which the covariances are known. For this oracle model the paper provides the sample complexity bound for exact recovery using SDP. Next the authors propose an iterative algorithm, which updates the estimated covariances and solves the new SDPs iteratively. Some real-world experiments are provided as well.", "summary_of_the_review": "In the rebuttal it will be great if the authors could highlight their technical contributions, which seem unclear / lacking in the current shape. \n", "strength_and_weaknesses": "- The MIP - SDP relaxation is very standard (see e.g. [1]).\n\n- The authors highlight multiple times in the paper, that the technical novelty comes from extending the isotropic Gaussian covariances, to a more general setting where there is no such uniform and identical assumption. To me this sounds technically incremental, because the whole SDP framework stays the same, and it only requires some changes about the concentration inequalities. But I think this could also due to writing. Could the authors elaborate on this? Specifically, what are the technical novelties in the oracle LA-SDP analysis?\n\n- Do the authors know whether the SDP relaxation is tight or not? More specifically, if the sample complexity condition in Theorem 2 is violated, are we immediately in a regime such that exact recovery is guaranteed to fail with large probability (information-theoretic lower bound)? Or we have something statistically possible but cannot be solved efficiently by any algorithm (computational lower bound)?  \n\n- Theorem 2: Can the authors provide some example settings, in which the sample complexity condition is fulfilled or not fulfilled? In other words, how should we interpret the sample complexity requirement?\n\n- Is there any convergence rate guarantee on the iterative algorithm iLA-SDP, or is it purely heuristical? It is known that solving a single SDP is very time consuming (O(n^6) if I remember correctly), not to mention solving SDPs iteratively.\n\n- Experiments: Do these datasets follow the rate condition in Theorem 2, or they only serve to show the effectiveness of the proposed algorithm in practice?\n\n\n\nReferences:\n- [1] Amini, Arash A., and Elizaveta Levina. \"On semidefinite relaxations for the block model.\" The Annals of Statistics 46.1 (2018): 149-179.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject