Title: Achieving Optimal Clustering in Gaussian Mixture Models with Anisotropic Covariance Structures
Venue: NeurIPS 2024
Accepted: yes
Keywords: Minimax rates

Mixture model

Lloyd’s algoirhtm

Clustering
Abstract: We study clustering under anisotropic Gaussian Mixture Models (GMMs), where covariance matrices from different clusters are unknown and are not necessarily the identity matrix. We analyze two anisotropic scenarios: homogeneous, with identical covariance matrices, and heterogeneous, with distinct matrices per cluster. For these models, we derive minimax lower bounds that illustrate the critical influence of covariance structures on clustering accuracy. To solve the clustering problem, we consider a variant of Lloyd's algorithm, adapted to estimate and utilize covariance information iteratively. We prove that the adjusted algorithm not only achieves the minimax optimality but also converges within a logarithmic number of iterations, thus bridging the gap between theoretical guarantees and practical efficiency.

Achieving Optimal Clustering in Gaussian Mixture Models with Anisotropic Covariance Structures

Abstract

We study clustering under anisotropic Gaussian Mixture Models (GMMs), where covariance matrices from different clusters are unknown and are not necessarily the identity matrix. We analyze two anisotropic scenarios: homogeneous, with identical covariance matrices, and heterogeneous, with distinct matrices per cluster. For these models, we derive minimax lower bounds that illustrate the critical influence of covariance structures on clustering accuracy. To solve the clustering problem, we consider a variant of Lloyd's algorithm, adapted to estimate and utilize covariance information iteratively. We prove that the adjusted algorithm not only achieves the minimax optimality but also converges within a logarithmic number of iterations, thus bridging the gap between theoretical guarantees and practical efficiency.

1 Introduction

Clustering is a fundamentally important task in statistics and machine learning [7, 2]. The most widely recognized and extensively studied model for clustering is the Gaussian Mixture Model (GMM) [17, 19], which is formulated as

Here Y = ( Y 1 , . . . , Y n ) are the observations with n being the sample size. We define the set [ n ] = { 1 , 2 , . . . , n } . Assume k is the known number of clusters. Let { θ ∗ a } a ∈ [ k ] represent the unknown centers, and Σ ∗ a denote the corresponding unknown covariance matrices. Define z ∗ ∈ [ k ] n as the cluster assignment vector, where for each index j ∈ [ n ] , the value of z ∗ j specifies which cluster the j -th data point is assigned to. The goal is to recover z ∗ from Y . For any estimator ˆ z , its clustering performance is measured by the misclustering error rate h (ˆ z, z ∗ ) , which will be introduced later in (4).

̸

There has been increasing interest in theoretical and algorithmic analysis of clustering under GMMs. In a scenario where a GMM is isotropic, meaning that all covariance matrices { Σ ∗ a } a ∈ [ k ] are equal to the identity matrix, [15] obtained the minimax rate for clustering, which takes the form of exp( -(1 + o (1))(min a = b ∥ θ ∗ a -θ ∗ b ∥ ) 2 / 8) , with respect to the misclustering error rate. A diverse range of methods has been explored in the context of the isotropic setting. Among these, Lloyd's algorithm [13] stands out as a particularly effective clustering algorithm, renowned for its extensive success in a myriad of disciplines. [15, 8] establish computational and statistical guarantees for the Lloyd's algorithm. Specifically, they showed it achieves the minimax optimal rates after a few iterations provided with some decent initialization. Another popular approach to clustering especially for high dimensional data is the spectral clustering [21, 18, 20], which is an umbrella term for clustering after a dimension reduction through a spectral decomposition. [14] proves the spectral clustering also achieves the optimality under the isotropic GMM. Semidefinite programming (SDP)

is also used for clustering by exploiting its low-rank structure, and its statistical properties have been studied in literature, for example, [5].

Despite the numerous compelling findings, most existing research primarily focuses on isotropic GMMs. The understanding of clustering in an anisotropic context, where the covariance matrices are not constrained to be identity matrices, remains relatively limited. Some studies, including [15, 5, 16, 1, 9, 24], present results for sub-Gaussian mixture models, wherein the errors ϵ j are assumed to follow some sub-Gaussian distributions with the variance proxy σ 2 . At first glance, it might appear that these results encompass the anisotropic case, as distributions of the form {N (0 , Σ ∗ a ) } a ∈ [ k ] are indeed sub-Gaussian distributions. However, from a minimax perspective, the least favorable scenario among all sub-Gaussian distributions with variance proxy σ 2 -and thus the most challenging for clustering-is when the errors are distributed as N (0 , σ 2 I ) . Therefore, the minimax rate for clustering under the sub-Gaussian mixture model essentially equals the one under the isotropic GMM, and methods like Lloyd's algorithm, which require no covariance matrix information, can be rate-optimal. As a result, the aforementioned findings primarily pertain to isotropic GMMs.

A few studies have explored the direction of clustering under anisotropic GMMs. [3] presents a polynomial-time clustering algorithm that provably performs well when Gaussian distributions are well-separated by hyperplanes. This idea is further developed in [11], which extends the approach to allow overlapping Gaussians, albeit only in two-cluster scenarios. [22] proposes a novel method for clustering under a balanced mixture of two elliptical distributions. They establish a provable upper bound on their clustering performance. Nevertheless, the fundamental limit of clustering under anisotropic GMMs, and whether a polynomial-time procedure can achieve it, remains unknown.

In this paper, we investigate the clustering task under two anisotropic GMMs. In Model 1, all covariance matrices are equal (i.e., homogeneous) to some unknown matrix Σ ∗ . Model 2 offers more flexibility, with covariance matrices that are unknown and not necessarily identical (i.e., heterogeneous). The contribution of this paper is two-fold, summarized as follows:

Our first contribution is on the minimax rates. We obtain minimax lower bounds for clustering under anisotropic GMMs with respect to the misclustering error rate. We show they take the form of

̸

where the signal-to-noise ratio under Model 1 is equal to min a,b ∈ [ k ]: a = b ∥ ( θ ∗ a -θ ∗ b ) T Σ ∗-1 2 ∥ . The signal-to-noise ratio for Model 2 is more intricate and will be introduced in Section 3. For both models, we can see the minimax rates depend not only on the centers but also on the covariance matrices. This is different from the isotropic case, whose signal-to-noise ratio is min a = b ∥ θ ∗ a -θ ∗ b ∥ . Our results precisely capture the role that covariance matrices play in the clustering problem. This shows that covariance matrices impact the fundamental limits of the clustering problem through complex interactions with the centers, especially in Model 2. We obtain the minimax lower bounds by drawing connections with Linear Discriminant Analysis (LDA) [6] and Quadratic Discriminant Analysis (QDA).

̸

Our second and more important contribution is on the computational side. We give a computationally feasible procedure and rate-optimal algorithm for the anisotropic GMM. Lloyd's algorithm, developed for the isotropic case, is no longer optimal as it only considers distances among centers [3]. We study an adjusted Lloyd's algorithm which estimates the covariance matrices in each iteration and adjusts the clusters accordingly. It can also be seen as a hard EM algorithm [4]. Here, we modify the E-step of the soft EM by implementing a maximization step that directly assigns data points to clusters, rather than calculating probabilities. As an iterative algorithm, we demonstrate that it achieves the minimax lower bound within log n iterations. This offers both statistical and computational guarantees, serving as valuable guidance for practitioners. Specifically, if we let z ( t ) denote the output of the algorithm after t iterations, it holds with high probability that

for all t ≥ log n . The algorithm can be initialized using popular methods like spectral clustering or Lloyd's algorithm. In our numerical studies, we demonstrate that our algorithm

significantly improves over the two aforementioned methods under anisotropic GMMs, and matches the optimal exponent specified in the minimax lower bound.

Paper Organization. The remaining paper is organized as follows. In Section 2, we study Model 1 where the covariance matrices are unknown but homogeneous. In Section 3, we consider Model 2 where covariance matrices are unknown and heterogeneous. For both cases, we establish the minimax lower bound for the clustering and give a computationally feasible and rate-optimal procedure. In Section 4, we provide a numerical comparison with other popular methods. Proofs are included in the supplement.

Notation. For any matrix X ∈ R d × d , we denote λ 1 ( X ) as its smallest eigenvalue and λ d ( X ) as its largest eigenvalue. In addition, we denote ∥ X ∥ as its operator norm. For any two vectors u, v of the same dimension, we denote ⟨ u, v ⟩ = u T v as their inner product. For any positive integer d , we denote I d as the d × d identity matrix. We denote N ( µ, Σ) as the normal distribution with mean µ and covariance matrix Σ . We denote I {·} as the indicator function. For two positive sequences { a n } and { b n } , a n ⪯ b n and a n = O ( b n ) both mean a n ≤ Cb n for some constant C > 0 independent of n . We also write a n = o ( b n ) or b n a n →∞ when lim sup n a n b n = 0 .

2 GMMwith Unknown but Homogeneous Covariance Matrices

2.1 Model

We first consider the GMM where the covariance matrices of different clusters are unknown but are assumed to be equal to each other. Then the data-generating process can be displayed as follows:

Throughout the paper, we call it Model 1 for simplicity and to distinguish it from a different and more complicated one that will be introduced in Section 3. The goal is to recover the underlying cluster assignment vector z ∗ . If Σ ∗ were known, then (1) can be converted into an isotropic GMM by a linear transformation (Σ ∗ ) -1 2 Y j . However, the unknown nature of Σ ∗ makes clustering under this model more challenging than under isotropic GMMs.

Signal-to-noise Ratio. Define the signal-to-noise ratio

̸

which is a function of all the centers { θ ∗ a } a ∈ [ k ] and the covariance matrix Σ ∗ . As we will show later in Theorem 2.1, SNR captures the difficulty of the clustering problem and determines the minimax rate. We defer the geometric interpretation of SNR until after presenting Theorem 2.2.

A quantity closely related to SNR is the minimum distance among the centers. Define ∆ as

̸

Then we can see SNR and ∆ are of the same order if all eigenvalues of the covariance matrix Σ ∗ are assumed to be constants. If Σ ∗ is further assumed to be σ 2 I d , then SNR equals ∆ /σ . As a result, in [15, 8, 14] where the isotropic GMMs are studied, ∆ /σ plays the role of signal-to-noise ratio and appears in their rates. Since (2) represents a direct generalization, we refer to it as the signal-to-noise ratio for Model 1.

Loss Function. To measure the clustering performance, we consider the following loss function. For any z, z ∗ ∈ [ k ] n , we define

̸

where Ψ = { ψ : ψ is a bijection from [ k ] to [ k ] } . Here, the minimum is taken over all permutations of [ k ] to address the identifiability issues of the labels 1 , 2 , . . . , k . The loss function measures the

proportion of coordinates where z and z ∗ differ, modulo any permutation of label symbols. Thus, it is referred to as the misclustering error rate in this paper. Another loss that will be used is ℓ ( z, z ∗ ) defined as

It measures the clustering performance of z considering the distances among the true centers. It is related to h ( z, z ∗ ) as h ( z, z ∗ ) ≤ ℓ ( z, z ∗ ) / ( n ∆ 2 ) and provides more information than h ( z, z ∗ ) . We will mainly use ℓ ( z, z ∗ ) in the technical analysis but will present results using h ( z, z ∗ ) which is more interpretable.

2.2 Minimax Lower Bound

We first establish the minimax lower bound for the clustering problem under Model 1.

Theorem 2.1. Under the assumption SNR √ log k →∞ , we have

If SNR = O (1) instead, we have inf ˆ z sup z ∗ ∈ [ k ] n E h (ˆ z, z ∗ ) ≥ c for some constant c > 0 .

Theorem 2.1 allows the cluster numbers k to grow with n and shows that SNR →∞ is a necessary condition to have a consistent clustering. If k is a constant, then SNR → ∞ is also a sufficient condition. Theorem 2.1 holds for any arbitrary configurations of { θ ∗ a } a ∈ [ k ] and Σ ∗ , with the minimax lower bound depending on these through SNR. The parameter space is only for z ∗ while { θ ∗ a } a ∈ [ k ] and Σ ∗ are held fixed. Hence, (6) can be interpreted as a case-specific result, precisely capturing the explicit dependence of the minimax rates on { θ ∗ a } a ∈ [ k ] and Σ ∗ .

Theorem 2.1 is closely related to the LDA. If there are only two clusters with known centers and a covariance matrix, then estimating each z ∗ j becomes exactly the task of the LDA: we aim to determine from which of two normal distributions, each with a different mean but the same covariance matrix, the observation Y j is generated. In fact, this approach is also how Theorem 2.1 is proved: We first reduce the estimation problem of z ∗ to two-point hypothesis testing for each individual z ∗ j . The error of these tests is analyzed in Lemma A.1 using the LDA, and we then aggregate all these testing errors together.

-2

0

2

4

-2

-1

0

1

2

3

4

y

N

(

q

1

*

,

S

*

)

N

(

q

2

*

,

S

*

)

f =

0

f =

1

-2

0

2

4

-2

-1

0

1

2

3

4

y

SNR/2

N

( 0,

I d)

N

( (

S

*

)

-1/2

(

q

2

*

- q

1

*

) ,

I d)

f =

0

f =

1

x

With the help of Lemma A.1, we have a geometric interpretation of SNR. In the left panel of Figure 1, we have two normal distributions N ( θ ∗ 1 , Σ ∗ ) and N ( θ ∗ 2 , Σ ∗ ) that X follows. The black line represents the optimal testing procedure ϕ displayed in Lemma A.1, dividing the space into two half-spaces. To calculate the testing error, we can make the transformation X ′ = (Σ ∗ ) -1 2 ( X -θ ∗ 1 ) so that the two normal distributions become isotropic: N (0 , I d ) and N ((Σ ∗ ) -1 2 ( θ ∗ 2 -θ ∗ 1 ) , I d ) as displayed in the right panel. Then the distance between the two centers is ∥ (Σ ∗ ) -1 2 ( θ ∗ 2 -θ ∗ 1 ) ∥ , and the distance from a center to the black curve is half of that. Then, the probability that N (0 , I d ) falls within the

grayed area equals exp( -(1 + o (1)) ∥ (Σ ∗ ) -1 2 ( θ ∗ 2 -θ ∗ 1 ) ∥ 2 / 8) , according to Gaussian tail probability. As a result, ∥ (Σ ∗ ) -1 2 ( θ ∗ 2 -θ ∗ 1 ) ∥ is the effective distance between the two centers of N ( θ ∗ 1 , Σ ∗ ) and N ( θ ∗ 2 , Σ ∗ ) for the clustering problem, taking into account the geometry of the covariance matrix. Since we have multiple clusters, SNR defined in (2) can be interpreted as the minimum effective distance among the centers { θ ∗ a } a ∈ [ k ] , considering the anisotropic structure of Σ ∗ . This measure captures the intrinsic difficulty of the clustering problem.

2.3 Rate-Optimal Adaptive Procedure

In this section, we give a computationally feasible and rate-optimal procedure for clustering under Model 1. Summarized in Algorithm 1, it is a variant of Lloyd's algorithm. Starting with an initial setup, it iteratively updates the estimates of the centers { θ ∗ a } a ∈ [ k ] (in (7)), the covariance matrix Σ ∗ (in (8)), and the cluster assignment vector z ∗ (in (9)). This algorithm differs from Lloyd's algorithm in that the latter is designed for isotropic GMMs and does not incorporate the covariance matrix update outlined in (8). Furthermore, (9) updates the estimation of z ∗ j using argmin a ∈ [ k ] ( Y j -θ ( t ) a ) T ( Y j -θ ( t ) a ) instead. To differentiate clearly, we refer to the classic form as the vanilla Lloyd's algorithm and our modified version, which accommodates the unknown and anisotropic covariance matrix, as the adjusted Lloyd's algorithm .

Algorithm 1 can also be interpreted as a hard EM algorithm. When applying Expectation Maximization (EM) to Model 1, the M step estimates the parameters { θ ∗ a } a ∈ [ k ] and Σ ∗ , while the E step estimates z ∗ . It turns out the updates on the parameters (7) - (8) are identical to those in the EM's M step. However, the update of z ∗ in Algorithm 1 differs from that in the EM. Instead of computing a conditional expectation typical of the E step, the algorithm performs maximization in (9). As a result, Algorithm 1 effectively consists solely of M steps for both parameters and z ∗ , characterizing it as a hard EM algorithm.

Algorithm 1: Adjusted Lloyd's Algorithm for Model 1.

Input: Data Y , number of clusters k , an initialization z (0) , number of iterations T . Output: z ( T ) 1 for t = 1 , . . . , T do

2

3

Update the centers:

Update the covariance matrix:

4 Update the cluster assignment vector:

In Theorem 2.2, we give a computational and statistical guarantee of Algorithm 1. We show that starting from a decent initialization, within log n iterations, Algorithm 1 achieves the error rate exp ( -(1 + o (1)) SNR 2 / 8 ) which matches the minimax lower bound given in Theorem 2.1. As a result, Algorithm 1 is a rate-optimal procedure. In addition, the algorithm is fully adaptive to the unknown { θ ∗ a } a ∈ [ k ] and Σ ∗ . The sole piece of information presumed to be known is k , the number of clusters, as commonly assumed in clustering literature [15, 8, 14]. The theorem also shows that the number of iterations needed to achieve the optimal rate is at most log n , providing implementation guidance to practitioners.

Theorem 2.2. Assume k = O (1) , d = O ( √ n ) , and min a ∈ [ k ] ∑ n j =1 I { z ∗ j = a } ≥ αn k for some constant α > 0 . Assume SNR → ∞ and λ d (Σ ∗ ) /λ 1 (Σ ∗ ) = O (1) . For Algorithm 1, suppose z (0) satisfies ℓ ( z (0) , z ∗ ) = o ( n ) with probability at least 1 -η . Then with probability at least 1 -η -n -1 -exp( -SNR ) , we have

We make the following remarks on the assumptions of Theorem 2.2: When k is constant, the assumption that SNR →∞ is a necessary condition for consistent recovery of z ∗ , as outlined in the minimax lower bound presented in Theorem 2.1. The assumption on Σ ∗ ensures that the covariance matrix is well-conditioned. The dimensionality d is assumed to be O ( √ n ) , a stronger assumption than in [15, 8, 14], where d = O ( n ) is sufficient. This is because, unlike these studies, our work requires estimating the covariance matrix Σ ∗ and controlling the estimation error ∥ Σ ( t ) -Σ ∗ ∥ .

Theorem 2.2 needs a decent initialization z (0) in the sense that it is sufficiently close to the ground truth such that ℓ ( z (0) , z ∗ ) = o ( n ) . This is because our theoretical analysis requires the initialization being within a specific proximity to the true parameters. The requirement can be fulfilled by simple procedures. An example is the vanilla Lloyd's algorithm whose performance is studied in [15, 8]. Though [15, 8] are for isotropic GMMs, their results can be extended to sub-Gaussian mixture models with nearly identical proof. Since ϵ j are sub-Gaussian random variables with proxy variance λ d (Σ ∗ ) , [8] implies the vanilla Lloyd's algorithm output ˆ z satisfies ℓ (ˆ z, z ∗ ) ≤ n exp( -(1 + o (1))∆ 2 / (8 λ d (Σ ∗ ))) with probability at least 1 -exp( -∆ / √ λ d (Σ ∗ )) -n -1 , under the assumption that ∆ 2 / ( k 2 ( kd/n +1) λ d (Σ ∗ )) → ∞ . Then we have ℓ (ˆ z, z ∗ ) = o ( n ) with high probability under the assumptions of Theorem 2.2, and hence it can be used as an initialization for the algorithm.

3 GMMwith Unknown and Heterogeneous Covariance Matrices

3.1 Model

In this section, we study the GMM where the covariance matrices of each cluster are unknown and not necessarily equal to each other. The data-generation process can be displayed as follows,

We refer to this as Model 2 throughout the paper to distinguish it from Model 1, as discussed in Section 2. The key difference between (10) and (1) is that here we have distinct covariance matrices { Σ ∗ a } a ∈ [ k ] for each cluster, instead of a single shared Σ ∗ . We use the same loss function as defined in (4).

̸

Signal-to-noise Ratio. The signal-to-noise ratio for Model 2 is defined as follows. We use the notation SNR ′ to distinguish it from the SNR used for Model 1. Compared to SNR, SNR ′ is much more complicated and does not have an explicit formula. We first define a set B a,b ⊂ R d for any a, b ∈ [ k ] such that a = b :

We then define SNR ′ a,b = 2min x ∈ B a,b ∥ x ∥ and

̸

The form of SNR ′ is closely connected to the testing error of the QDA, which we will give in Lemma 3.1. The interpretation of the SNR ′ , particularly from a geometric perspective, will be

̸

deferred until after the presentation of Lemma 3.1. Here let us consider a few special cases where we are able to simplify SNR ′ : (1) When Σ ∗ a = Σ ∗ for all a ∈ [ k ] , by simple algebra, we have SNR ′ a,b = ∥ ( θ ∗ a -θ ∗ b ) T Σ ∗-1 2 ∥ for any a, b ∈ [ k ] such that a = b . Hence, SNR ′ = SNR and Model 2 effectively reduces to Model 1. (2) When Σ ∗ a = σ 2 a I d for any a ∈ [ k ] where σ 1 , . . . , σ k > 0 are large constants, we have SNR ′ a,b , SNR ′ b,a both close to 2 ∥ θ ∗ a -θ ∗ b ∥ / ( σ a + σ b ) . From these examples, we can see SNR ′ is determined by both the centers { θ ∗ a } a ∈ [ k ] and the covariance matrices { Σ ∗ a } a ∈ [ k ] .

3.2 Minimax Lower Bound

We first establish the minimax lower bound for the clustering problem under Model 2.

Theorem 3.1. Assume d = O (1) and max a,b ∈ [ k ] λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) . Under the assumption SNR ′ √ log k →∞ , we have

If SNR ′ = O (1) instead, we have inf ˆ z sup z ∗ ∈ [ k ] n E h (ˆ z, z ∗ ) ≥ c for some constant c > 0 .

Although the statement of Theorem 3.1 appears similar to that of Theorem 2.1, the two minimax lower bounds differ due to the varying dependencies of the centers and covariance matrices on SNR ′ versus SNR. Using the same argument as in Section 2.2, the minimax lower bound established in Theorem 3.1 closely relates to the QDA between two normal distributions with different means and different covariance matrices.

Lemma 3.1 (Testing Error for the QDA) . Consider two hypotheses H 0 : X ∼ N ( θ ∗ 1 , Σ ∗ 1 ) and H 1 : X ∼ N ( θ ∗ 2 , Σ ∗ 2 ) . Define a testing procedure

Then we have inf ˆ ϕ ( P H 0 ( ˆ ϕ = 1) + P H 1 ( ˆ ϕ = 0)) = P H 0 ( ϕ = 1) + P H 1 ( ϕ = 0) . Assume d = O (1) and max a,b ∈{ 1 , 2 } λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) . If min { SNR ′ 1 , 2 , SNR ′ 2 , 1 } →∞ , we have

Otherwise, inf ˆ ϕ ( P H 0 ( ˆ ϕ = 1) + P H 1 ( ˆ ϕ = 0)) ≥ c for some constant c > 0 .

-2

0

2

4

-2

-1

0

1

2

3

4

y

N

(

q

1

*

,

S

1

*

)

N

(

q

2

*

,

S

2

*

)

f =

0

f =

1

-2

0

2

4

-2

-1

0

1

2

3

4

y

SNR'/2

N

( 0,

I d)

N

( (

S

1

*

)

-1/2

(

q

2

*

- q

1

*

) ,  (

S

1

*

)

-1/2

S

2

*

(

S

1

*

)

-1/2

)

f =

0

f =

1

x

Lemma 3.1 provides a geometric interpretation of SNR ′ . In the left panel of Figure 2, we have two normal distributions N ( θ ∗ 1 , Σ ∗ 1 ) and N ( θ ∗ 2 , Σ ∗ 2 ) from which X can be generated, and the black curve represents the optimal testing procedure ϕ , as detailed in Lemma 3.1. Since Σ ∗ 1 is not necessarily equal to Σ ∗ 2 , the black curve is not necessarily a straight line. If H 0 is true, the probability that X is incorrectly classified occurs when X falls into the gray area, represented by P H 0 ( ϕ = 1) . To

3

calculate this, we transform X to X ′ = (Σ ∗ 1 ) -1 2 ( X -θ ∗ 1 ) , standardizing the first distribution. Then, as displayed in the right panel of Figure 2, the two distributions become N (0 , I d ) and N ((Σ ∗ 1 ) -1 2 ( θ ∗ 2 -θ ∗ 1 ) , (Σ ∗ 1 ) -1 2 Σ ∗ 2 (Σ ∗ 1 ) -1 2 ) , and the optimal testing procedure ϕ becomes I { X ′ ∈ B 1 , 2 } . As a result, in the right panel of Figure 2, B 1 , 2 represents the space colored by gray, and the black curve is its boundary. Then P H 0 ( ϕ = 1) is equal to P ( N (0 , I d ) ∈ B 1 , 2 ) . Under the assumption d = O (1) and max a,b ∈{ 1 , 2 } λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) , in Lemma C.10, we can show P ( N (0 , I d ) ∈ B 1 , 2 ) = exp( -(1 + o (1)) SNR ′ 2 1 , 2 / 8) . As a result, SNR ′ can be interpreted as the minimum effective distance among the centers { θ ∗ a } a ∈ [ k ] , considering the anisotropic and heterogeneous structure of { Σ ∗ a } a ∈ [ k ] , and it captures the intrinsic difficulty of the clustering problem under Model 2.

3.3 Optimal Adaptive Procedure

In this section, we give a computationally feasible and rate-optimal procedure for clustering under Model 2. Similar to Algorithm 1, Algorithm 2 is a variant of Lloyd's algorithm, adjusted to accommodate unknown and heterogeneous covariance matrices. It can also be interpreted as a hard EM algorithm under Model 2. Algorithm 2 differs from Algorithm 1 in (13) and (14), as now there are k covariance matrices instead of a common one.

Algorithm 2: Adjusted Lloyd's Algorithm for Model 2.

Input: Data Y , number of clusters k , an initialization z (0) , number of iterations T . Output: z ( T )

1 for t = 1 , . . . , T do

2 Update the centers:

Update the covariance matrices:

4 Update the cluster assignment vector:

In Theorem 3.2, we give a computational and statistical guarantee for Algorithm 2. We demonstrate that, with proper initialization, Algorithm 2 achieves the minimax lower bound within log n iterations. The assumptions needed in Theorem 3.2 are similar to those in Theorem 2.2, except that we require stronger assumptions on the dimensionality d since now we have k (instead of one) covariance matrices to be estimated. In addition, by assuming max a,b ∈ [ k ] λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) , we ensure not only that each of the k covariance matrices is well-conditioned but also that they are comparable to one another.

Theorem 3.2. Assume k = O (1) , d = O (1) , and min a ∈ [ k ] ∑ n j =1 I { z ∗ j = a } ≥ αn k for some constant α > 0 . Assume SNR ′ → ∞ and max a,b ∈ [ k ] λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) . For Algorithm 2, suppose z (0) satisfies ℓ ( z (0) , z ∗ ) = o ( n ) with probability at least 1 -η . Then with probability at least 1 -η -5 n -1 -exp( -SNR ′ ) , we have

The vanilla Lloyd's algorithm can be used as the initialization for Algorithm 2. This is because Model 2 is also a sub-Gaussian mixture model. By the same argument as in Section 2.3, the output of the

vanilla Lloyd's algorithm ˆ z satisfies ℓ (ˆ z, z ∗ ) = o ( n ) with high probability under the assumptions of Theorem 3.2.

We conclude this section with a time complexity analysis of Algorithm 2. Compared to the vanilla Lloyd's algorithm, our method introduces additional computational overhead due to the need for computing the inverse and determinant of covariance matrices. Specifically, the time complexity of Algorithm 2 is O ( nkd 3 T ) . In contrast, the vanilla Lloyd's algorithm has a lower time complexity of O ( nkdT ) . The increase in complexity stems from matrix operations in d dimensions, as both matrix inversion and determinant computation scale as O ( d 3 ) .

4 Numerical Studies

In this section, we compare the performance of our methods with other popular clustering methods on synthetic and real datasets under different settings.

Model 1. The first simulation is designed for the GMM with unknown but homogeneous covariance matrices (i.e., Model 1). We independently generate n = 1200 samples with dimension d = 50 from k = 30 clusters. Each cluster has 40 samples. We set Σ ∗ = U T Λ U , where Λ is a 50 × 50 diagonal matrix with diagonal elements selected from 0.5 to 8 with equal space and U is a randomly generated orthogonal matrix. The centers { θ ∗ a } a ∈ [ n ] are orthogonal to each other with ∥ θ ∗ 1 ∥ = . . . = ∥ θ ∗ 30 ∥ = 9 . We consider four popular clustering methods: (1) the spectral clustering method in [14] (denoted as 'spectral'), (2) the vanilla Lloyd's algorithm in [15] (denoted as 'vanilla Lloyd'), (3) Algorithm 1 initialized by the spectral clustering (denoted as 'spectral + Alg 1'), and (4) Algorithm 1 initialized by the vanilla Lloyd (denoted as 'vanilla Lloyd + Alg 1'). The comparison is presented in the left panel of Figure 3.

Model 2. We also compare the performances of four methods (spectral, vanilla Lloyd, spectral + Alg 2, and vanilla Lloyd + Alg 2) for the GMM with unknown and heterogeneous covariance matrices (i.e., Model 2). In this case, we take n = 1200 , k = 2 , and d = 9 . We set Σ ∗ 1 = I d and Σ ∗ 2 = Λ 2 , a diagonal matrix where the first diagonal entry is 0.5 and the remaining entries are 5. We set the cluster sizes to be 900 and 300, respectively. To simplify the calculation of SNR ′ , we set θ ∗ 1 = 0 and θ ∗ 2 = 5 e 1 , with e 1 being the vector that has a 1 in its first entry and 0s elsewhere. The comparison is presented in the right panel of Figure 3.

-6

-5

-4

-3

-2

2

4

6

log(error)

method

spectral

spectral + Alg 1

-8

-7

-6

-5

2

4

6

log(error)

method

spectral

spectral + Alg 2

iteration iteration

In Figure 3, the x -axis is the number of iterations and the y -axis is the logarithm of the misclustering error rate, i.e., log( h ) . Each of the curves plotted is an average of 100 independent trials. We can see both Algorithm 1 and Algorithm 2 outperform the spectral clustering and the vanilla Lloyd's algorithm significantly. Additionally, the dashed lines in the left and right panels represent the optimal exponents -SNR 2 / 8 and -SNR ′ 2 / 8 of the minimax bounds, respectively. It is observed that both Algorithm 1 and Algorithm 2 meet these benchmarks after three iterations. This justifies the conclusion that both algorithms are rate-optimal.

Real Data. To further demonstrate the effectiveness of our methods, we conduct experiments using the Fashion-MNIST dataset [23]. In the first analysis, we use a total of 12,000 28 × 28 grayscale images, consisting of 6,000 images each from the T-shirt/top class and the Trouser class. The left panel of Figure 4 gives a visualization of the data points using their first two principal components, showing the anisotropic and heterogeneous covariance structures. Since a large number of pixels have zero across most images, we apply PCA to reduce dimensionality from 784 to 50 by retaining the top 50 principal components. Our Algorithm 2 achieves a misclustering error of 5.71%, outperforming the vanilla Lloyd's algorithm, which has an error of 8.24%. In the second analysis, we incorporate an additional class, the Ankle boot class, increasing the total to 18,000 images across three classes. Following the same preprocessing steps, the visualization of the dataset's structure in the right panel of Figure 4 again confirms the presence of anisotropic and heterogeneous covariances. Here, Algorithm 2 achieves an error of 3.97%, an improvement over the 5.64% error rate observed with the vanilla Lloyd's algorithm.

5 Conclusion

This paper focuses on clustering methods and theory for GMMs, with anisotropic covariance structures, presenting new minimax bounds and an adjusted Lloyd's algorithm tailored for varying covariance structures. Our theoretical and empirical analyses demonstrate the algorithm's ability to achieve optimality within a logarithmic number of iterations. Despite these advances, our results have some limitations that are worth addressing in future work:

High-Dimensional Settings: Current results are restricted to dimensions d growing at a rate slower than n , specifically d = O ( √ n ) as stated in Theorem 2.2. Section 3 further requires a stronger assumption d = O (1) . These constraints stem from technical challenges in estimating covariance matrices accurately and in controlling matrix determinant. Adopting more sophisticated analytical tools could potentially relax these bounds to d = O ( n ) . In scenarios where d exceeds n , the misclustering error deviates from the simpler exponential decay observed under isotropic GMMs, as shown in [16]. This suggests that our model might also exhibit similar complexities, warranting further exploration into the technique used in [16] for potential extensions.

Ill-Conditioned Covariance Structures: Our analysis relies on the assumption of wellconditioned covariance matrices, where max a,b ∈ [ k ] λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) . This condition is crucial for the current analytical framework, as it helps manage the estimation errors of covariance matrices and their inverses. While more advanced techniques may allow for a relaxation of this assumption, handling ill-conditioned or degenerate covariance matrices remains challenging, particularly due to the difficulty of working with matrix inverses in such cases. While minimax lower bounds suggest that clustering is still possible even when the covariance matrix is degenerate, it raises computational challenges for our current algorithms. This highlights the need for developing new algorithms that can function effectively under less restrictive conditions.

References

Emmanuel Abbe, Jianqing Fan, and Kaizheng Wang. An ℓ p theory of PCA and spectral clustering. The Annals of Statistics , 50(4):2359-2385, 2022.

Christopher M Bishop. Pattern recognition and machine learning . springer, 2006.

S Charles Brubaker and Santosh S Vempala. Isotropic PCA and affine-invariant clustering. In Building Bridges , pages 241-281. Springer, 2008.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, 1977.

Yingjie Fei and Yudong Chen. Hidden integrality of SDP relaxations for sub-Gaussian mixture models. In Conference On Learning Theory , pages 1931-1965. PMLR, 2018.

Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics , 7(2):179-188, 1936.

Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning , volume 1. Springer series in statistics New York, 2001.

Chao Gao and Anderson Y Zhang. Iterative algorithm for discrete structure recovery. The Annals of Statistics , 50(2):1066-1094, 2022.

Christophe Giraud and Nicolas Verzelen. Partial recovery bounds for clustering with the relaxed k -means. Mathematical Statistics and Learning , 1(3):317-374, 2019.

Daniel Hsu, Sham Kakade, Tong Zhang, et al. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability , 17, 2012.

Adam Tauman Kalai, Ankur Moitra, and Gregory Valiant. Efficiently learning mixtures of two Gaussians. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 553-562, 2010.

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , pages 1302-1338, 2000.

Stuart Lloyd. Least squares quantization in PCM. IEEE transactions on information theory , 28(2):129-137, 1982.

Matthias Löffler, Anderson Y Zhang, and Harrison H Zhou. Optimality of spectral clustering in the Gaussian mixture model. The Annals of Statistics , 49(5):2506-2530, 2021.

Yu Lu and Harrison H Zhou. Statistical and computational guarantees of Lloyd's algorithm and its variants. arXiv preprint arXiv:1612.02099 , 2016.

Mohamed Ndaoud. Sharp optimal recovery in the two component Gaussian mixture model. The Annals of Statistics , 50(4):2096-2126, 2022.

Karl Pearson. Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London. A , 185:71-110, 1894.

Daniel A Spielman and Shang-Hua Teng. Spectral partitioning works: Planar graphs and finite element meshes. In Proceedings of 37th Conference on Foundations of Computer Science , pages 96-105. IEEE, 1996.

D Michael Titterington, Adrian FM Smith, and Udi E Makov. Statistical analysis of finite mixture distributions . Wiley' 1985.

S. Vempala and G. Wang. A spectral algorithm for learning mixture models. J. Comput. Syst. Sci. , 68(4):841-860, 2004.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17(4):395-416, 2007.

Kaizheng Wang, Yuling Yan, and Mateo Díaz. Efficient clustering for stretched mixtures: Landscape and optimality. Advances in Neural Information Processing Systems , 33:2130921320, 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.

Anderson Y Zhang and Harrison H Zhou. Leave-one-out singular subspace perturbation analysis for spectral clustering. arXiv preprint arXiv:2205.14855 , 2022.

The appendices are organized as follows. Appendix A is dedicated to proving the results in Section 2. To be more specific, we prove the lower bound, Theorem 2.1, in Appendix A.1 and the upper bound, Theorem 2.2, in Appendix A.2. For the upper bound proof, we first give a high-level idea in Appendix A.2.1, followed by a detailed proof in Appendix A.2.2. Appendix B includes proofs of the results in Section 3: the proof of the lower bound, Theorem 3.1, is in Appendix B.1, and the proof of the upper bound, Theorem 3.2, is in Appendix B.2. We include all technical lemmas and their proofs in Appendix C.

A Proofs in Section 2

A.1 Proofs for the Lower Bound

In the following lemma, we give a sharp and explicit formula for the testing error of the LDA. Here we have two normal distributions N ( θ ∗ 1 , Σ ∗ ) and N ( θ ∗ 2 , Σ ∗ ) and an observation X that is generated from one of them. We are interested in estimating from which distribution the observation is drawn. By the Neyman-Pearson lemma, it is known that the likelihood ratio test I { 2( θ ∗ 2 -θ ∗ 1 ) T (Σ ∗ ) -1 X ≥ θ ∗ T 2 (Σ ∗ ) -1 θ ∗ 2 -θ ∗ T 1 (Σ ∗ ) -1 θ ∗ 1 } is the optimal testing procedure. Then by using the Gaussian tail probability, we are able to obtain the optimal testing error, with its lower bound given in Lemma A.1.

Lemma A.1 (Testing Error for the LDA) . Consider two hypotheses H 0 : X ∼ N ( θ ∗ 1 , Σ ∗ ) and H 1 : X ∼ N ( θ ∗ 2 , Σ ∗ ) . Define a testing procedure

Then inf ˆ ϕ ( P H 0 ( ˆ ϕ = 1)+ P H 1 ( ˆ ϕ = 0)) = P H 0 ( ϕ = 1)+ P H 1 ( ϕ = 0) . If ∥ ( θ ∗ 2 -θ ∗ 1 ) T (Σ ∗ ) -1 2 ∥ → ∞ , we have

Otherwise, inf ˆ ϕ ( P H 0 ( ˆ ϕ = 1) + P H 1 ( ˆ ϕ = 0)) ≥ c for some constant c > 0 .

Proof. Note that ϕ is the likelihood ratio test. By the Neyman-Pearson lemma, it is the optimal procedure. That is, inf ˆ ϕ ( P H 0 ( ˆ ϕ = 1)+ P H 1 ( ˆ ϕ = 0)) = P H 0 ( ϕ = 1)+ P H 1 ( ϕ = 0) . Let ϵ ∼ N (0 , I d ) . By Gaussian tail probability, we have

for some constant C > 0 . The proof is complete.

Proof of Theorem 2.1. We adopt the idea from [15]. Without loss of generality, assume the minimum in (2) is achieved at a = 1 , b = 2 so that SNR = ( θ ∗ 1 -θ ∗ 2 ) T (Σ ∗ ) -1 ( θ ∗ 1 -θ ∗ 2 ) . Consider an arbitrary ¯ z ∈ [ k ] n such that |{ i ∈ [ n ] : ¯ z i = a }| ≥ ⌈ n k -n 8 k 2 ⌉ for any a ∈ [ k ] . Then for each a ∈ [ k ] , we can choose a subset of { i ∈ [ n ] : ¯ z i = a } with cardinality ⌈ n k -n 8 k 2 ⌉ , denoted by T a . Let T = ∪ a ∈ [ k ] T a . Then we can define a parameter space

̸

̸

Notice that for any z = ˜ z ∈ Z , we have 1 n ∑ n i =1 I { z i = ˜ z i } ≤ k n n 8 k 2 = 1 8 k and 1 n ∑ n i =1 I { ψ ( z i ) = ˜ z i } ≥ 1 n ( n 2 k -n 8 k 2 ) ≥ 1 4 k for any permutation ψ on [ k ] . Thus we can conclude

̸

̸

We notice that

̸

̸

Now consider a fixed i ∈ T c . Define Z a = { z ∈ Z : z i = a } for a = 1 , 2 . Then we can see Z = Z 1 ∪ Z 2 and Z 1 ∩ Z 2 = ∅ . What is more, there exists a one-to-one mapping f ( · ) between Z 1 and Z 2 , such that for any z ∈ Z 1 , we have f ( z ) ∈ Z 2 with [ f ( z )] j = z j for any j = i and [ f ( z )] i = 2 . Hence, we can reduce the problem to a two-point testing probe and then apply Lemma A.1. We first consider the case that SNR →∞ . We have

̸

̸

̸

̸

̸

for some η = o (1) . Here the second inequality is due to Lemma A.1. Then,

for some other η ′ = o (1) , where we use SNR 2 / log k →∞ .

The proof for the case SNR = O (1) is similar and hence is omitted here.

A.2 Proofs for the Upper Bound

A.2.1 High-level Idea

In this section, we provide a high-level idea for the proof of Theorem 2.2. The detailed proof is technical and is given later in Appendix A.2.2.

The key idea for establishing the statistical guarantees of Algorithm 1, an iterative algorithm, is to perform a 'one-step' analysis [8]. That is, assume we have an estimation z for z ∗ . Then we can apply (7), (8), and (9) on z to obtain { ˆ θ a ( z ) } a ∈ [ k ] , ˆ Σ( z ) , and ˆ z ( z ) sequentially, which all depend on z . Thus, ˆ z ( z ) can be seen as a refined estimate of z ∗ . We will first build the connection between ℓ ( z, z ∗ ) with ℓ (ˆ z ( z ) , z ∗ ) as in Lemma A.2, which informally states that under certain conditions, with high probability, we have

holds for any z ∈ [ k ] n such that ℓ ( z, z ∗ ) is small. Here ξ ideal ( δ ) refers to the ideal error, which eventually leads to the upper bound in Theorem 2.2. Lemma A.2 tells us ˆ z ( · ) has a 'contraction' property. That is, after one iteration of (7), (8), and (9), ℓ (ˆ z ( z ) , z ∗ ) is at most a half of ℓ ( z, z ∗ ) , up to an additive term ξ ideal ( δ ) .

To establish Lemma A.2, we decompose the loss ℓ (ˆ z ( z ) , z ∗ ) into several errors according to the difference in their behaviors. Next, we will introduce several conditions (Conditions 1 - 3), under

which we demonstrate that these errors are either negligible, well-controlled by ℓ ( z, z ∗ ) , or connected to ξ ideal ( δ ) . Once Lemma A.2 is established, we will show in Lemma A.3 that the connection can be extended to multiple iterations, under two more conditions (Conditions 4 - 5). Lemma A.3 states informally that, under certain conditions and with high probability, we have

for all t ≥ 1 . This implies ℓ ( z ( t ) , z ∗ ) is eventually at most ξ ideal ( δ ) , up to some constant factor. Last, we will show all these conditions hold with high probability. Although the algorithmic guarantees in Lemma A.2 and Lemma A.3 are established with respect to the ℓ ( · , · ) loss, we will use the relationship between h ( · , · ) and ℓ ( · , · ) to convert this result to one involving h ( · , · ) . Hence, we prove Theorem 2.2.

A.2.2 Detailed Proofs

In the statement of Theorem 2.2, the covariance matrix Σ ∗ is assumed to satisfy λ d (Σ ∗ ) /λ 1 (Σ ∗ ) = O (1) . Without loss of generality, we can replace it by assuming Σ ∗ satisfies

where λ min , λ max > 0 are two constants. This is due to the following simple argument using the scaling properties of normal distributions. Let { Y j } be some dataset generated according to Model 1 with parameters { θ ∗ a } a ∈ [ k ] , Σ ∗ , and z ∗ . The assumption λ d (Σ ∗ ) /λ 1 (Σ ∗ ) = O (1) is equivalent to assuming there exist some constants λ min , λ max > 0 and some quantity σ > 0 that may depend on n such that λ min σ 2 ≤ λ 1 (Σ ∗ ) ≤ λ d (Σ ∗ ) ≤ λ max σ 2 . By performing a scaling transformation, we obtain another dataset Y ′ j = Y j /σ . Note that: 1) { Y ′ j } can be seen as generated from Model 1 with parameters { θ ∗ a /σ } a ∈ [ k ] , Σ ∗ /σ 2 , and z ∗ . 2) Clustering on { Y j } is equivalent to clustering on { Y ′ j } . 3) By the definition in (2), the SNRs that are associated with the data-generating processes of { Y ′ j } and { Y j } are exactly equal to each other. 4) We have λ min ≤ λ 1 (Σ ∗ /σ 2 ) ≤ λ d (Σ ∗ /σ 2 ) ≤ λ max . Thus, for the remainder of this section, we assume that (15) holds without any loss of generality.

In the proof, we will mainly use the loss ℓ ( · , · ) for convenience. Recall ∆ is defined as the minimum distance among centers in (3). We have

The algorithmic guarantees Lemma A.2 and Lemma A.3 are established with respect to the ℓ ( · , · ) loss. Eventually, we will use (16) to convert it into a result with respect to h ( · , · ) in the proof of Theorem 2.2.

Error Decomposition for the One-step Analysis: Consider an arbitrary z ∈ [ k ] n . Apply (7), (8), and (9) on z to obtain { ˆ θ a ( z ) } a ∈ [ k ] , ˆ Σ( z ) , and ˆ z ( z ) :

̸

For simplicity, we denote ˆ z as shorthand for ˆ z ( z ) . Let j ∈ [ n ] be an arbitrary index with z ∗ j = a . According to (9), z ∗ j will be incorrectly estimated after one iteration in ˆ z if a = argmin b ∈ [ k ] ( Y j -

ˆ

θ

b

(

z

))

T

( ˆ

Σ(

z

))

-

1

(

Y

j

-

ˆ

θ

b

(

z

))

. Therefore, it is important to analyze the event

for any b ∈ [ k ] \ { a } . Note that Y j = θ ∗ a + ϵ j . After some rearrangements, we can see (17) is equivalent to

where

In the above decomposition, the expression ⟨ ϵ j , ( ˆ Σ( z ∗ )) -1 ( ˆ θ a ( z ∗ ) -ˆ θ b ( z ∗ )) ⟩ ≤ -1 2 ⟨ θ ∗ a -θ ∗ b , (Σ ∗ ) -1 ( θ ∗ a -θ ∗ b ) ⟩ does not involve z . Roughly speaking, it corresponds to the event that z ∗ j will be incorrectly estimated in ˆ z ( z ∗ ) . This is considered the main part of (17) and will contribute to ξ ideal. The difference between (17) and the main term is expressed through the terms F j , G j , H j : F j includes terms related to noise ϵ j , illustrating the impact of measurement noise; G j covers estimation errors for cluster centers ( ˆ θ a ( z ) -ˆ θ a ( z ∗ ) ) and covariance matrices ( ˆ Σ( z ) -ˆ Σ( z ∗ ) ), showing the effect of the parameter estimation inaccuracies; H j contains all other terms from additional error sources. Readers can refer to [8] for more information about the decomposition.

Conditions and Guarantees for One-step Analysis. We continue to analyze the event (17). We first define a quantity independent of z , which we refer to as the ideal error:

When δ = 0 , it is determined by the main term in (17), namely ⟨ ϵ j , ( ˆ Σ( z ∗ )) -1 ( ˆ θ a ( z ∗ ) -ˆ θ b ( z ∗ )) ⟩ ≤ -1 2 ⟨ θ ∗ a -θ ∗ b , (Σ ∗ ) -1 ( θ ∗ a -θ ∗ b ) ⟩ . Roughly speaking, ξ ideal (0) relates to the performance of ˆ z ( z ∗ ) . Due to the presence of the terms F j , G j , H j in the decomposition of (17), what appears in the analysis of (17) is ξ ideal ( δ ) instead of ξ ideal (0) where hopefully δ > 0 is some small number.

To establish the guarantee for one-step analysis, we next give several conditions on the error terms F j ( a, b ; z ) , G j ( a, b ; z ) and H j ( a, b ; z ) .

Condition 1. Assume that

holds with probability at least 1 -η 1 for some τ, δ, η 1 > 0 .

Condition 2. Assume that

holds with probability at least 1 -η 2 for some τ, δ, η 2 > 0 .

Condition 3. Assume that

holds with probability at least 1 -η 3 for some τ, δ, η 3 > 0 .

Lemma A.2. Assumes Conditions 1 - 3 hold for some τ, δ, η 1 , η 2 , η 3 , > 0 . We then have

̸

Proof. Consider any j ∈ [ n ] such that z ∗ j = a . We notice that for any b ∈ [ k ] such that b = a ,

where the second inequality comes from Conditions 1 and 3. Note that we can multiply I { ˆ z j = b } on both sides of the above display and the inequality still holds. Hence,

Thus, we have

which implies Lemma A.2. Here the last inequality uses Condition 2.

Conditions and Guarantees for Multiple Iterations. In the above, we establish a statistical guarantee for the one-step analysis. Now we will extend the result to multiple iterations. That is, starting from some initialization z (0) , we will characterize how the losses ℓ ( z (0) , z ∗ ) , ℓ ( z (1) , z ∗ ) , ℓ ( z (2) , z ∗ ) , . . . , decay. We impose conditions on ξ ideal ( δ ) and the initialization z (0) .

Condition 4. Assume that

holds with probability at least 1 -η 4 for some τ, δ, η 4 > 0 .

Finally, we need a condition on the initialization.

Condition 5. Assume that

holds with probability at least 1 -η 5 for some τ, η 5 > 0 .

With these conditions satisfied, we can give a lemma that shows the convergence of our algorithm.

Lemma A.3. Assume Conditions 1 - 5 hold for some τ, δ, η 1 , η 2 , η 3 , η 4 , η 5 > 0 . We then have

for all t ≥ 1 , with probability at least 1 -η , where η = ∑ 5 i =1 η i .

Proof. By Conditions 4, 5 and a mathematical induction argument, we can easily conclude ℓ ( z ( t ) , z ∗ ) ≤ τ for any t ≥ 0 . Thus, Lemma A.3 is a direct extension of Lemma A.2.

With-high-probability Results for the Conditions and Proof of Theorem 2.2. Recall the definition of ∆ in (3). Recall that in (15) we assume λ min ≤ λ 1 (Σ ∗ ) ≤ λ d (Σ ∗ ) ≤ λ max for two constants λ min , λ max > 0 . Hence we have ∆ is of the same order as SNR. Specifically, we have

Hence the assumption SNR →∞ in the statement of Theorem 2.2 is equivalently ∆ →∞ . Next, we give two with-high-probability lemmas. The first lemma is for Conditions 1-3, providing upper bounds for the quantities involved in these conditions, showing that δ can be taken as some o (1) term. The second lemma shows that for any δ = o (1) , ξ ideal ( δ ) is upper bounded by the desired minimax rate multiplied by the sample size n .

Lemma A.4. Under the same conditions as in Theorem 2.2, for any constant C ′ > 0 , there exists some constant C > 0 only depending on α and C ′ such that

with probability at least 1 -n -C ′ . As a result, Conditions 1-3 hold for some δ = o (1) .

Proof. Under the conditions of Theorem 2.2, the inequalities (33)-(38) hold with probability at least 1 -n -C ′ . In the remaining proof, we will work on the event these inequalities hold. Denote

ˆ Σ a ( z ) = ∑ j ∈ [ n ] ( Y j -ˆ θ a ( z ))( Y j -ˆ θ a ( z )) T I { z j = a } ∑ j ∈ [ n ] I { z j = a } and Σ ∗ a = Σ ∗ for any a ∈ [ k ] . Then we have the equivalence

Hence, we can use the results from Lemma C.7 and Lemma C.8. By (43) and (44), we have

and

By the assumption that kd = O ( √ n ) , ∆ k →∞ and τ = o ( n/k ) , we have ∥ ˆ Σ( z ∗ ) -Σ ∗ ∥ , ∥ ˆ Σ( z ) -ˆ Σ( z ∗ ) ∥ = o (1) , which implies ∥ ( ˆ Σ( z ∗ )) -1 ∥ , ∥ ( ˆ Σ( z )) -1 ∥ ⪯ 1 . Thus, we have

and similarly

Now we start to prove (19)-(21). Let F j ( a, b, z ) = F (1) j ( a, b, z ) + F (2) j ( a, b, z ) + F (3) j ( a, b, z ) where F (1) j ( a, b, z ) := ⟨ ϵ j , ( ˆ Σ( z )) -1 ( ˆ θ b ( z ) -ˆ θ b ( z ∗ )) ⟩ - ⟨ ϵ j , ( ˆ Σ( z )) -1 ( ˆ θ a ( z ) -ˆ θ a ( z ∗ )) ⟩ , F (2) j ( a, b, z ) := -⟨ ϵ j , (( ˆ Σ( z )) -1 -( ˆ Σ( z ∗ )) -1 )( θ ∗ a -θ ∗ b ) ⟩ , F (3) j ( a, b, z ) := -⟨ ϵ j , (( ˆ Σ( z )) -1 -( ˆ Σ( z ∗ )) -1 )( θ ∗ b -ˆ θ b ( z ∗ )) ⟩ + ⟨ ϵ j , (( ˆ Σ( z )) -1 -( ˆ Σ( z ∗ )) -1 )( θ ∗ a -ˆ θ a ( z ∗ )) ⟩ . Notice that

where we use (34), (23), and the fact that ℓ ( z, z ∗ ) ≤ τ and kd = O ( √ n ) for the last inequality. Here the second to last inequality is due to the following argument: for any w ∈ R d , we have ∑ j |⟨ ϵ j , w ⟩| 2 = ∑ j w T ϵ j ϵ T j w = w T ( ∑ j ϵ j ϵ T j ) w ≤ ∥ w ∥ 2 ∥ ∑ j ϵ j ϵ T j ∥ . From (41) we have max a ∈ [ k ] ∥ θ ∗ a -ˆ θ a ( z ∗ ) ∥ = o (1) under the assumption kd = O ( √ n ) . By the similar analysis as in F (2) j ( a, b, z ) , we have

Similarly, we have

where we use (42) and the fact that ( ˆ Σ( z )) -1 has bounded operator norm. Combining these terms together, we obtain (20).

Next, for (19), by (41) we have

and

By (22) we have

Using the results above we can get (19).

Finally we are going to establish (21). Recall the definition of G j ( a, b, z ) which has four terms. For the third and fourth terms, we have

and

We can easily verify that the other two terms are smaller than the above two terms. Then, by using (42) and (23), we have

Lemma A.5. With the same conditions as in Theorem 2.2, for any δ = o (1) , we have

with probability at least 1 -n -C ′ -exp( -SNR ) .

Proof. Under the conditions of Theorem 2.2, the inequalities (33)-(38) hold with probability at least 1 -n -C ′ . In the remaining proof, we will work on the event these inequalities hold. Recall the definition of ξ ideal. We can write

where ¯ δ = ¯ δ n is some sequence to be chosen later. We bound the four terms sequentially. Suppose ϵ j = (Σ ∗ ) 1 / 2 w j , where w j ind ∼ N (0 , I d ) . By (22), we know

̸

where C is a constant which may vary from line by line. Recall that kd = O ( √ n ) , min a = b ∥ θ ∗ a -θ ∗ b ∥ → ∞ , and ∆ /k →∞ by assumption. Let n -1 4 = o ( ¯ δ ) . Using the χ 2 tail probability in Lemma C.1, we have for any a = b ∈ [ k ] ,

̸

We can obtain similar bounds on M 3 and M 4 by using (41). For M 1 , the Gaussian tail bound leads to the inequality

Thus,

Overall, we have E ξ ideal ⪯ n exp ( -(1 + o (1)) SNR 2 8 ) . By the Markov's inequality, we have

In other words, with probability at least 1 -exp( -SNR ) , we have

Proof of Theorem 2.2. By Lemmas A.3 - A.5, we have that Conditions 1 - 5 are satisfied with probability at least 1 -η -n -1 -exp( -SNR ) . Then applying Lemma A.3, we have

By (16), and since there exists a constant C such that ∆ ≤ C SNR, we can conclude

Notice that h ( · , · ) takes value in the set { j/n : j ∈ [ n ] ∪ { 0 }} , the term 2 -t in the above inequality should be negligible as long as 2 -t = o ( n -1 ) . Thus, we can claim

B Proofs in Section 3

B.1 Proofs for the Lower Bound

Proof of Lemma 3.1. The Neyman-Pearson lemma tells us the likelihood ratio test ϕ is the optimal procedure. Following the proof of Lemma A.1, we have

where the last inequality is by Lemma C.10.

Proof of Theorem 3.1. The proof is identical to the proof of Theorem 2.1 and is omitted here.

B.2 Proofs for the Upper Bound

We adopt a similar proof idea as in Section 2 for Model 1. We first present an error decomposition for the one-step analysis for Algorithm 2. In Lemma B.1, we show the loss decays after a one-step iteration under Conditions 6 - 11. Then in Lemma B.2 we extend the result to multiple iterations, under two extra Conditions 12 - 13. Finally, we show that all the conditions are satisfied with high probability and thus prove Theorem 3.2.

In the statement of Theorem 3.2, we assume max a,b ∈ [ k ] λ d (Σ ∗ a ) /λ 1 (Σ ∗ b ) = O (1) for the covariance matrices { Σ ∗ a } a ∈ [ k ] . Without loss of generality, we can replace it by assuming { Σ ∗ a } a ∈ [ k ] satisfy

where λ min , λ max > 0 are two constants. This is due to the scaling properties of the normal distributions. The reasoning is the same as that in (15) for Model 1 and is omitted here. For the remainder of this section, we will assume that (24) holds for the covariance matrices.

Error Decomposition for the One-step Analysis: Consider an arbitrary z ∈ [ k ] n . Apply (12), (13), and (14) on z to obtain { ˆ θ a ( z ) } a ∈ [ k ] , { ˆ Σ a ( z ) } a ∈ [ k ] , and ˆ z ( z ) :

̸

For simplicity, we denote ˆ z as shorthand for ˆ z ( z ) . Let j ∈ [ n ] be an arbitrary index with z ∗ j = a . According to (14), z ∗ j will be incorrectly estimated after one iteration in ˆ z if a = argmin b ∈ [ k ] ( Y j -ˆ θ b ( z )) T ( ˆ Σ b ( z )) -1 ( Y j -ˆ θ b ( z )) + log | ˆ Σ b ( z ) | . That is, it is important to analyze the event

for any b ∈ [ k ] \ { a } . After some rearrangements, we can see (25) is equivalent to

where

Among these terms, F j , G j , H j are nearly identical to their counterparts in Section A.2.2 with ˆ Σ( z ) replaced by ˆ Σ a ( z ) or ˆ Σ b ( z ) . There are three extra terms not appearing in Section A.2.2: Q j is a quadratic term of ϵ j and K j , L j are terms involving matrix determinants.

Conditions and Guarantees for One-step Analysis. To establish the guarantee for the one-step analysis, we first give several conditions on the error terms.

Condition 6. Assume that

holds with probability at least 1 -η 1 for some τ, δ, η 1 > 0 .

Condition 7. Assume that

holds with probability at least 1 -η 2 for some τ, δ, η 2 > 0 .

Condition 8. Assume that

holds with probability at least 1 -η 3 for some τ, δ, η 3 > 0 .

Condition 9. Assume that

holds with probability at least 1 -η 4 for some τ, δ, η 4 > 0 .

Condition 10. Assume that

holds with probability at least 1 -η 5 for some τ, δ, η 5 > 0 .

Condition 11. Assume that

holds with probability at least 1 -η 6 for some τ, δ, η 6 > 0 .

We next define a quantity referred to as the ideal error,

Lemma B.1. Assumes Conditions 6 - 11 hold for some τ, δ, η 1 , . . . , η 6 > 0 . We then have

where η = ∑ 6 i =1 η i .

Proof. The proof of this lemma is quite similar to the proof of Lemma A.2. The additional terms Q j and K j can be handled in the same way as F j while L j can be handled similarly to H j . We omit the details here.

Conditions and Guarantees for Multiple Iterations. In the above, we establish a statistical guarantee for the one-step analysis. Now we will extend the result to multiple iterations. That is, starting from some initialization z (0) , we will characterize how the losses ℓ ( z (0) , z ∗ ) , ℓ ( z (1) , z ∗ ) , ℓ ( z (2) , z ∗ ) , . . . , decay. We impose conditions on ξ ideal ( δ ) and the initialization z (0) .

Condition 12. Assume that

holds with probability at least 1 -η 7 for some τ, δ, η 7 > 0 .

Finally, we need a condition on the initialization.

Condition 13. Assume that

holds with probability at least 1 -η 8 for some τ, η 8 > 0 .

With these conditions satisfied, we can give a lemma that shows the convergence of our algorithm.

Lemma B.2. Assumes Conditions 6 - 13 hold for some τ, δ, η 1 , . . . , η 8 > 0 . We then have

for all t ≥ 1 , with probability at least 1 -η , where η = ∑ 8 i =1 η i .

Proof. The proof of this lemma is the same as the proof of Lemma A.3.

With-high-probability Results for the Conditions and Proof of Theorem 3.2. Lemma B.3 and Lemma B.4 are the counterparts of Lemmas A.4 and A.5 in Appendix A.2.2. Recall that (24) is assumed. By Lemma C.10, we have ∆ is of the same order as SNR ′ , which will play a similar role as (18) in Section A.2.2.

Lemma B.3 and Lemma B.4 are counterparts of Lemmas A.4 and A.5 in Section A.2.2. The first lemma is for Conditions 6-11, providing upper bounds for the quantities involved in these conditions, showing that δ can be taken as some o (1) term. The second lemma shows that for any δ = o (1) , ξ ideal ( δ ) is upper bounded by the desired minimax rate multiplied by the sample size n .

Lemma B.3. Under the same conditions as in Theorem 3.2, for any constant C ′ > 0 , there exists some constant C > 0 only depending on α, C ′ , λ min , λ max such that

(30)

with probability at least 1 -n -C ′ -4 nd . As a result, Conditions 6-11 hold for some δ = o (1) .

Proof. Under the conditions of Theorem 3.2, the inequalities (33)-(38) hold with probability at least 1 -n -C ′ . In the remaining proof, we will work on the event these inequalities hold. Hence, we can use the results from Lemma C.7 and C.8. Using the same arguments as in the proof of Lemma A.4, we can get (26), (27) and (28).

As for (29), we first use Lemma C.2 to have ∑ n j =1 ∥ ϵ j ∥ 4 ≤ 3 nd with probability at least 1 -4 / ( nd ) . Then, we have

where the last inequality is due to (53) and the fact that ℓ ( z, z ∗ ) ≤ τ .

Next for (30), notice that by (43), (44), and SNR ′ → ∞ , we have for any 1 ≤ i ≤ d , λ min 2 ≤ λ i ( ˆ Σ a ( z ∗ )) ≤ 2 λ max and

Thus by Weyl's inequality, we know

where the last inequality is due to the fact that λ i ( ˆ Σ a ( z ∗ )) is at the constant rate, ∥ ˆ Σ a ( z ) -ˆ Σ a ( z ∗ ) ∥ 2 = o (1) and the inequality log(1 + x ) ≤ x for any x > 0 . (32) yields to the inequality

Finally for (31), by (43) and the similar argument as (32), we can get

which implies (31). We complete the proof.

Lemma B.4. With the same conditions as Theorem 3.2, for any sequence δ n = o (1) , we have

with probability at least 1 -n -C ′ -exp( -SNR ′ )

Proof. Under the conditions of Theorem 3.2, the inequalities (33)-(38) hold with probability at least 1 -n -C ′ . In the remaining proof, we will work on the event these inequalities hold. Similar to the proof of Lemma A.5, we have a decomposition ξ ideal ≤ ∑ 6 i =1 M i where

is the main term and

Using the same arguments as the proof of Lemma A.5, we can choose some ¯ δ = ¯ δ n = o (1) which is slowly diverging to zero satisfying

As for M 5 , by (43) we have

where C is a constant and w j iid ∼ N (0 , I d ) . Since there exists some constant C ′ such that SNR ′ ≤ C ′ ∆ , we can choose appropriate ¯ δ = o (1) such that

M 6 is essentially the same with M 5 and can be proved similarly. Finally for M 1 , using Lemma C.10, we have

Then we have

Using the Markov's inequality we complete the proof of Lemma B.4.

Proof of Theorem 3.2. By Lemmas B.2-B.4, we can obtain the result by arguments used in the proof of Theorem 2.2 and hence the proof is omitted here.

C Technical Lemmas

In this section, we present and prove technical lemmas used in this paper. Lemmas C.1 and C.2 are about χ 2 distributions. Appendix C.1 gives various upper bounds needed in the proofs of Appendix B. Appendix C.2 is devoted to the calculation related to SNR ′ .

Lemma C.1. For any x > 0 , we have

Proof. These results are Lemma 1 of [12].

Lemma C.2. Let W i iid ∼ χ 2 d for any i ∈ [ n ] where n, d are positive integers. Then we have

Proof. We have E ∑ n i =1 W 2 i = nd ( d + 2) and E ∑ n i =1 W 4 i = nd ( d + 2)( d + 4)( d + 6) . Then we have Var (∑ n i =1 W 2 i ) = 8 nd ( d +2)( d +3) . Then we obtain the desired result by Chebyshev's inequality.

C.1 With-High-Probability Bounds

Lemma C.3. For any z ∗ ∈ [ k ] n and k ∈ [ n ] , consider independent vectors ϵ j ∼ N (0 , Σ ∗ z ∗ j ) for any j ∈ [ n ] . Assume there exists a constant λ max > 0 such that ∥ Σ ∗ a ∥ ≤ λ max for any a ∈ [ k ] . Then, for

any constant C ′ > 0 , there exists some constant C > 0 only depending on C ′ , λ max such that

with probability at least 1 -n -C ′ . We have used the convention that 0 / 0 = 0 .

Proof. Note that ϵ j is sub-Gaussian with parameter λ max which is a constant. The inequalities (33) and (35) are respectively Lemmas A.4, A.1 in [15]. The inequality (34) is a slight extension of Lemma A.2 in [15]. This extension follows from a standard union bound argument. The proof of (36) is identical to that of (35).

Lemma C.4. Consider the same assumptions as in Lemma C.3. Assume additionally min a ∈ [ k ] ∑ n j =1 I { z ∗ j = a } ≥ αn k for some constant α > 0 and k ( d +log n ) n = o (1) . Then, for any constant C ′ > 0 , there exists some constant C > 0 only depending on α, C ′ , λ max such that

with probability at least 1 -n -C ′ .

Proof. Note that we have ϵ j = Σ ∗ 1 2 z ∗ j η j where η j iid ∼ N (0 , I d ) for any j ∈ [ n ] . Since max a ∥ Σ ∗ a ∥ ≤ λ max , we have

Define

Take S d -1 = { y ∈ R d : ∥ y ∥ = 1 } and N ϵ = { v 1 , · · · , v | N ϵ | } is an ϵ -covering of S d -1 . In particular, we pick ϵ < 1 4 , then | N ϵ | ≤ 9 d . By the definition of the ϵ -covering, we have

For any v ∈ N ϵ ,

Denote n a = ∑ n j =1 I { z ∗ j = a } . Then ∑ n j =1 I { z ∗ j = a } v T η j η T j v ∼ χ 2 n a . Using Lemma C.1, we have

Since k ( d +log n ) n = o (1) and n a ≥ αn/k where α is a constant, we can take t = C ′′ √ k ( d +log n ) n for some large constant C ′′ and the proof is complete.

Lemma C.5. Consider the same assumptions as in Lemma C.3. Then, for any s = o ( n ) and for any constant C ′ > 0 , there exists some constant C > 0 only depending on C ′ , λ max such that

with probability at least 1 -n -C ′ . We have used the convention that 0 / 0 = 0 .

Proof. Consider any a ∈ [ s ] and a fixed T ⊂ [ n ] such that | T | = a . Similar to the proof of Lemma C.4, we can take S d -1 = { y ∈ R d : ∥ y ∥ = 1 } and its ϵ -covering N ϵ with ϵ < 1 4 and | N ϵ | ≤ 9 d . Then we have

Note that w T ϵ j / √ λ max is a sub-Gaussian random variable with parameter 1. By the tail probability result for quadratic forms of sub-Gaussian random vectors [10], for any fixed w ∈ N ϵ , we have

Since a = o ( n ) , there exists a constant C 0 such that 2 a ≤ C 0 a log n a . We can take t = ˜ C ( a log n a + d ) with ˜ C = C 16 -C 0 4 , then a +2 √ at +2 t ≤ C 4 ( a log n a + d ) . Thus,

Hence, we have

As a result,

Since a log n a is an increasing function when a ∈ [1 , s ] and a log n a ≥ log n ≥ log s , a choice of ˜ C = 3 + C ′ , that is C = 16 C ′ +4 C 0 +48 , can yield the desired result.

Finally, to allow | T | = 0 , we note that d ≤ min { 1 , √ | T |} d . The proof is complete.

Lemma C.6. For any z ∗ ∈ [ k ] n and k ∈ [ n ] , assume min a ∈ [ k ] ∑ n j =1 I { z ∗ j = a } ≥ αn k and ℓ ( z, z ∗ ) = o ( n ∆ 2 k ) , then

Proof. For any z ∈ [ k ] n such that ℓ ( z, z ∗ ) = o ( n ) and any a ∈ [ k ] , we have

̸

̸

which implies

Thus, we obtain (39).

In the following lemma, we are going to analyze estimation errors of the centers and covariance matrices under the anisotropic GMMs. For any z ∈ [ k ] n and for any z ∈ [ k ] , recall the definitions

Lemma C.7. For any z ∗ ∈ [ k ] n and k ∈ [ n ] , consider independent vectors Y j = θ ∗ z ∗ j + ϵ j where ϵ j ∼ N (0 , Σ ∗ z ∗ j ) for any j ∈ [ n ] . Assume there exist constants λ min , λ max > 0 such that λ min ≤ λ 1 (Σ ∗ a ) ≤ λ d (Σ ∗ a ) ≤ λ max for any a ∈ [ k ] , and a constant α > 0 such that min a ∈ [ k ] ∑ n j =1 I { z ∗ j = a } ≥ αn k . Assume k ( d +log n ) n = o (1) and ∆ k →∞ . Assume (33) -(38) hold. Then for any τ = o ( n ) and for any constant C ′ > 0 , there exists some constant C > 0 only depending on α, λ max , C ′ such that

for all z such that ℓ ( z, z ∗ ) ≤ τ .

Proof. Using (33) we obtain (41). By the same argument of (118) in [8], we can obtain (42). By (33) and (37) and (41), we can obtain (43). In the remaining proof, we will establish (53).

Since k ( d +log n ) n = o (1) , we have ∥ ˆ Σ a ( z ∗ ) ∥ ⪯ 1 for any a ∈ [ k ] . The difference ˆ Σ a ( z ) -ˆ Σ a ( z ∗ ) will be decomposed into several terms. We notice that

where

and

Also, we notice that where

For L 1 , we have

By (36), (39), (40), we have uniformly for any a ∈ [ k ] ,

Since max a ∈ [ k ] ∥ ∥ ∥ ˆ θ a ( z ∗ ) -θ ∗ a ∥ ∥ ∥ = o (1) , by (39), (42), (41), (47), and (48), we have uniformly for any a ∈ [ k ] ,

̸

̸

To bound L 2 , we first give the following simple fact. For any positive integer m and any { u j } j ∈ [ m ] , { v j } j ∈ [ m ] ∈ R d , we have ∥ ∑ j ∈ [ m ] ( u j + v j )( u j + v j ) T ∥ ≤ 2 ∥ ∑ j ∈ [ m ] u j u T j ∥ + 2 ∥ ∑ j ∈ [ m ] v j v T j ∥ . Hence, for L 2 , we have the following decomposition

̸

where

̸

̸

Since max a ∈ [ k ] ∑ n j =1 I { z j = a, z ∗ j = a } ≤ ℓ ( z,z ∗ ) ∆ 2 , we have

̸

̸

By (38) and the fact that max a ∈ [ k ] ∑ n j =1 I { z i = a, z ∗ i = a } ≤ ℓ ( z,z ∗ ) ∆ 2 , we also have

̸

̸

We are going to simplify the above bounds for R 1 , R 2 . Under the assumption that k ( d +log n ) n = o (1) , ∆ /k → ∞ , and ℓ ( z, z ∗ ) ≤ τ = o ( n ) , we have max a ∈ [ k ] ∥ ˆ θ a ( z ) -ˆ θ a ( z ∗ ) ∥ = o (1) , max a ∈ [ k ] ∥ ˆ θ a ( z ∗ ) -θ ∗ a ∥ = o (1) , and kℓ ( z,z ∗ ) n ∆ 2 = o (1) . Hence R 2 ⪯ kℓ ( z,z ∗ ) n ∆ 2 . Also we have

where in the last inequality, we use the fact that x (log( n/x )) 2 is an increasing function of x when 0 < x = o ( n ) . Then,

Since L 3 is similar to L 2 , by (46) we have uniformly for any a ∈ [ k ]

To bound S 2 , by (70) in [8], we have uniformly for any a ∈ [ k ] ,

̸

̸

where we use (43). Since k n ℓ ( z,z ∗ ) ∆ 2 ⪯ k √ nℓ ( z,z ∗ ) n ∆ , by (45) and the facts that ℓ ( z, z ∗ ) ≤ τ = o ( n ) we have

Lemma C.8. Under the same assumption as in Lemma C.7, if additionally we assume kd = O ( √ n ) and τ = o ( n/k ) , there exists some constant C > 0 only depending on α, λ min , λ max , C ′ such that

Proof. By (43) we have max a ∈ [ k ] ∥ ˆ Σ a ( z ∗ ) ∥ , max a ∈ [ k ] ∥ ( ˆ Σ a ( z ∗ )) -1 ∥ ⪯ 1 . By (44) we also have max a ∈ [ k ] ∥ ˆ Σ a ( z ) ∥ , max a ∈ [ k ] ∥ ( ˆ Σ a ( z )) -1 ∥ ⪯ 1 . Hence,

C.2 Calculation Related to SNR ′

In the following lemmas, we study properties of { SNR ′ a,b } a = b . Consider any pair a = b ∈ [ k ] . Let η ∼ N (0 , I d ) and Ξ a,b = θ ∗ a -θ ∗ b . Define

̸

̸

for any δ ∈ R . Then B a,b ( δ ) ⊂ B a,b ( δ ′ ) for any δ ′ ≤ δ . In addition, we define

Recall the definitions of B a,b and SNR ′ a,b in Section 3. Then it is a special case of B a,b ( δ ) and SNR ′ a,b ( δ ) with δ = 0 . That is, we have B a,b = B a,b (0) and SNR ′ a,b = SNR ′ a,b (0) .

To understand these quantities, we first study a canonical setting that can be later applied to establish Lemma C.10.

Lemma C.9. Consider any θ ∈ R d \ { 0 } and any Σ ∈ R d × d that is positive semi-definite. Let λ max , λ min > 0 be the largest and smallest eigenvalue of Σ , respectively. For any t ∈ R , define

and s ( t ) = min x ∈ D ( t ) ∥ x ∥ . Then the following hold:

Under the assumption that -∥ θ ∥ 2 / (8 λ max ) < t < ∥ θ ∥ 2 / 8 , we have

If t ′ also satisfies -∥ θ ∥ 2 /λ max < t ′ < ∥ θ ∥ 2 / 8 , we have

If θ is further assumed to satisfy min { √ λ min / 8 , 1 / 2 } ∥ θ ∥ ≥ 2 λ max and ∥ θ ∥ ≥ λ min 32 min { √ λ min / 8 , 1 / 2 } , there exists a d -dimensional ball H ( t ) ∈ R d with radius ( λ min / 8) min { √ λ min / 8 , 1 / 2 } such that H ( t ) ⊂ D ( t ) and ∥ x ∥ ≤ ( λ min / 8) min { √ λ min / 8 , 1 / 2 } + λ max + s ( t ) for all x ∈ H ( t ) .

Proof. First, we check whether each of the following two points is contained in D ( t ) or not, under the assumption -∥ θ ∥ 2 /λ max < t < ∥ θ ∥ 2 .

When x = 0 , we have ( x -θ ) T Σ -1 ( x -θ ) + t ≥ ∥ θ ∥ 2 /λ max + t > 0 . Hence, 0 / ∈ D ( t ) .

When x = θ , we have ∥ θ ∥ 2 > t . Hence, θ ∈ D ( t ) .

As a result, D ( t ) is non-empty, s ( t ) is well defined, and 0 < s ( t ) < ∥ θ ∥ 2 . Next, we consider a few more points to sharpen upper and lower bounds on s ( t ) under the assumption -∥ θ ∥ 2 / (8 λ max ) < t < ∥ θ ∥ 2 / 8 .

For any x that satisfies ∥ x ∥ ≤ ∥ θ ∥ / (max { 2 , 2 √ 2 λ max } ) , we have ∥ x -θ ∥ ≥ ∥ θ ∥ / 2 and consequently, ( x -θ ) T Σ -1 ( x -θ ) + t ≥ ( ∥ θ ∥ / 2) 2 /λ max + t = ∥ θ ∥ 2 / (4 λ max ) + t . Under the assumption that t > -∥ θ ∥ 2 / (8 λ max ) , we can verify that ∥ θ ∥ 2 / (4 λ max ) + t > ∥ θ ∥ 2 / (8 λ max ) ≥ ( ∥ θ ∥ / max { 2 , 2 √ 2 λ max } ) 2 ≥ ∥ x ∥ 2 . Hence, such x / ∈ D ( t ) .

When x = (1 -min { √ λ min / 8 , 1 / 2 } ) θ , we have ∥ x ∥ ≥ ∥ θ ∥ / 2 and ( x -θ ) T Σ -1 ( x -θ ) + t ≤ ∥ θ ∥ 2 (min { √ λ min / 8 , 1 / 2 } ) 2 /λ min + t = ∥ θ ∥ 2 / max { 8 , 4 λ min } + t . Under the assumption that t < ∥ θ ∥ 2 / 8 , we have ∥ θ ∥ 2 / max { 8 , 4 λ min } + t < ∥ θ ∥ 2 / 8 + t ≤ ∥ θ ∥ 2 / 4 ≤ ∥ x ∥ 2 . Hence, such x ∈ D ( t ) .

As a result, we have ∥ θ ∥ / (max { 2 , 2 √ 2 λ max } ) < s ( t ) < (1 -min { √ λ min / 8 , 1 / 2 } ) ∥ θ ∥ .

̸

Define a ball S ( r ) = { x ∈ R d : ∥ x ∥ 2 ≤ r 2 } and define S 2 ( r ; t ) = { x ∈ R d : ( x -θ ) T Σ -1 ( x -θ ) ≤ r 2 -t } to be the part of R d that is inside the corresponding ellipsoid. Then we have s ( t ) = min { r ≥ 0 : S 1 ( r ) ∩ S 2 ( r ; t ) = ∅} . By the definition and bounds of s ( t ) and the convexity of S 1 ( s ( t )) and S 2 ( s ( t ); t ) , we must have |S 1 ( s ( t )) ∩ S 2 ( s ( t ); t ) | = 1 , meaning that S 1 ( s ( t )) and S 2 ( s ( t ); t ) touch each other externally at one point. This implies s ( t ) can be obtained by the following process: We let S 1 ( r ) and S 2 ( r ) grow by increasing r , starting from 0 . The first time they touch each other, we stop and the value of r is exactly s ( t ) .

Denote y ( t ) ∈ R such that { y ( t ) } = S 1 ( s ( t )) ∩ S 2 ( s ( t ); t ) . Then we must have y ( t ) ∈ ¯ S 2 ( s ( t ); t ) . By the first conclusion, we have

In addition, we have

̸

Now we are going to prove the second conclusion of the lemma. Without loss of generality, assume t ≤ t ′ . Then we have D ( t ) ⊃ D ( t ′ ) and s ( t ) ≤ s ( t ′ ) . We are going to establish a lower bound for s ( t ) . First by definition of s ( t ′ ) , we have |S 1 ( s ( t ′ )) ∩ S 2 ( s ( t ′ ); t ′ ) | = 1 . Since t ≤ t ′ , we have S 2 ( s ( t ′ ); t ) ⊃ S 2 ( s ( t ′ ); t ′ ) . We have S 1 ( s ( t ′ )) ∩ S 2 ( s ( t ′ ); t ) = ∅ . From here we can also see

s ( t ) ≤ s ( t ′ ) . Now consider any x ∈ ¯ S 2 ( s ( t ′ ); t ) . It satisfies ( x -θ ) T Σ -1 ( x -θ ) = s 2 ( t ′ ) -t . Then we have

meaning that θ + √ s 2 ( t ′ ) -t ′ s 2 ( t ′ ) -t ( x -θ ) ∈ ¯ S 2 ( s ( t ′ ); t ′ ) . Hence,

Since |S 1 ( s ( t ′ )) ∩ ¯ S 2 ( s ( t ′ ); t ′ ) | = 1 , we have min y ∈ ¯ S 2 ( s ( t ′ ); t ′ ) ∥ y ∥ = s ( t ′ ) . Since ( x -θ ) T Σ -1 ( x -θ ) ≥ λ -1 max ∥ x -θ ∥ 2 , we have ∥ x -θ ∥ 2 ≤ λ max ( s 2 ( t ′ ) -t ) . Hence,

As a result, for any r < s ( t ′ ) - √ λ max t ′ -t 2 √ s 2 ( t ′ ) -t ′ , we have S 1 ( r ) ∩S 2 ( s ( t ′ ); t ) = ∅ and consequently S 1 ( r ) ∩S 2 ( r ; t ) = ∅ . As a result, s ( t ) ≥ s ( t ′ ) - √ λ max t ′ -t 2 √ s 2 ( t ′ ) -t ′ . Since we have shown s 2 ( t ′ ) -t ′ ≥ (min { √ λ min / 8 , 1 / 2 } ∥ θ ∥ ) 2 /λ max , we have s ( t ) ≥ s ( t ′ ) -λ max t ′ -t 2 min { √ λ min / 8 , 1 / 2 }∥ θ ∥ .

For the third conclusion of the lemma, recall the definition of y ( t ) . Under the assumption that min { √ λ min / 8 , 1 / 2 } ∥ θ ∥ ≥ 2 λ max , we have ∥ y ( t ) -θ ∥ -λ max > 0 and λ max / ∥ y ( t ) -θ ∥ ≤ 1 / 2 .

Denote y ′ ( t ) = θ + y ( t ) -θ ∥ y ( t ) -θ ∥ ( ∥ y ( t ) -θ ∥ -λ max ) . Then

Consequently,

Denote H ( t ) to be the ball centered at y ′ ( t ) with radius ( λ min / 8) min { √ λ min / 8 , 1 / 2 } . Then for any x ∈ H ( t ) , we have

where the last inequality holds under the assumption ∥ θ ∥ ≥ λ min 32 min { √ λ min / 8 , 1 / 2 } . Hence, H ( t ) ⊂ S 2 ( s ( t ); t ) and consequently H ( t ) ⊂ D ( t ) . On the other hand, for any x ∈ H ( t ) , we have

The proof is complete.

Lemma C.10. Assume d = O (1) . Consider any a = b ∈ [ k ] . Assume there exist constants λ min , λ max such that 0 < λ min ≤ λ 1 (Σ ∗ j ) ≤ λ d (Σ ∗ j ) ≤ λ max for any j ∈ { a, b } . Then SNR ′ a,b and ∥ Ξ b,a ∥ are of the same order. When SNR ′ a,b = O (1) , we have P a,b (0) ≥ c for some constant c > 0 . When SNR ′ a,b →∞ , we have

̸

Proof. Recall the setting stated in Lemma C.9. By the definition of B a,b , we can take θ = (Σ ∗ a ) -1 2 Ξ b,a , Σ = (Σ ∗ a ) -1 2 Σ ∗ b (Σ ∗ a ) -1 2 , and t = log | (Σ ∗ a ) -1 2 Σ ∗ b (Σ ∗ a ) -1 2 | such that B a,b = D ( t ) . Due to d = O (1) and (24), we have that all eigenvalues of Σ are constants and that log | (Σ ∗ a ) -1 2 Σ ∗ b (Σ ∗ a ) -1 2 | is also a constant. Consequently, t is constant. In addition, ∥ θ ∥ is of the same order as ∥ Ξ b,a ∥ .

and for any δ = o (1) , we have

When ∥ Ξ b,a ∥ is a constant, ∥ θ ∥ is a constant, then SNR ′ a,b must be a constant as well. This is because D ( t ) is the set of points where one density is greater or equal to another. Then D ( t ) is non-empty as both densities have integral 1. Hence, s ( t ) must be finite, meaning SNR ′ a,b is a constant. In this case, we have P a,b (0) = P ( η ∈ D ( t )) being a constant as well.

When ∥ Ξ b,a ∥ → ∞ , we have ∥ θ ∥ → ∞ . Since all the assumptions needed in Lemma C.9 are satisfied, by its first conclusion, we have s ( t ) being of the same order as ∥ θ ∥ , and consequently being of the same order as ∥ Ξ b,a ∥ . As a result, SNR ′ a,b and ∥ Ξ b,a ∥ are of the same order. In addition, H ( t ) exists and its radius is some constant c 1 > 0 . Hence, its volume is c d 1 V d where V d is denoted as the volume of a d -dimensional unit ball. In addition, for any x ∈ H ( t ) , we have ∥ x ∥ ≤ s ( t ) + c 2 = SNR ′ a,b / 2 + c 2 for some constant c 2 > 0 . Recall η ∼ N (0 , I d ) . Then we have

Now let us consider B a,b ( δ ) where δ = o (1) . We can take θ, Σ same as before, but let t ′ = log | (Σ ∗ a ) -1 2 Σ ∗ b (Σ ∗ a ) -1 2 | -δ Ξ T b,a (Σ ∗ b ) -1 Ξ b,a . Then we have t ′ = o (1) ∥ θ ∥ 2 . Hence, by Lemma C.9, we have

Hence,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer:[Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Experiment results were averaged. Error bars were inappropriate from a visual standpoint.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:[No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper carried out foundational and theoretical analysis on clustering which has no direct social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper states an hard-association EM algorithm for Gaussian mixture estimation, where the Gaussian components can be different and anisotropic. They also state theoretic bounds of the misclustering error rate.\n\nSTRENGTHS:\nOriginality: The theoretical association bound for different and anisotropic clustering is novel.\n\nQuality: Contribution is technically sound. Claims are very well supported by theoretical analysis and also some experimental results. Methods used are appropriate. It is a complete piece of work.\n\nClarity: Submission is clearly written and well organized. It adequately informs the reader and states the proposed algorithms in concise form to be easily reproduced.\n\nSignificance: Key contribution to the community are the theorems and very detailed and elaborate proofs about the error rate that can theoretically achieved in anisotropic clustering. Overcoming the constraint of isotropic components (or anisotropic but identical components) is a major step.\n\nWEAKNESSES:\nOriginality: A related work is \"Frisch, Hanebeck, Gaussian Mixture Estimation from Weighted Samples\" (not telling you to include this, just check if it's relevant; its soft-EM algorithm may be similar to your hard-EM).\n\nSuggestions:\n- in Line 108 give some inutition about the loss function, e.g. \"ratio of mis-associations\".\n- mention that Model 1 is equivalent to isotropic components in linearly transformed space with root of covariance (if that's correct)\nTypos:\n- line 102 displayed as follow*s*\n- line 136: \"Figure 1\" should be in same line, use \\sim or cleveref package\n- line 150 computational*ly* feasible\n- Theorem (and elsewhere): too big whitespace in \"exp (\". Use \"exp\\!(\" or \\mathopen{}, \\mathclose{} instead.\n- SNR has a non-math font in formulas and is sometimes italic and sometimes not\n- References: check capitalization, e.g. [11] PCM etc. I recommend:\n\\usepackage[style=numeric-verb, maxbibnames=20, sorting=none, eprint=false]{biblatex}\n\nQUESTIONS:\nWhat about individual weights (mixing proportions) of the Gaussian components? E.g. from line 271, must the 30 clusters each have 1200/30=40 samples?\n\nLIMITATIONS:\nLimitations are stated but maybe should be summarized in a dedicated section.\n- \"decent\" initial guess required. What does that mean? Were you able to reliably get the rate-optimal result without taking into account prior knowledge about the ground truth? At least state the qualitative dependencies, like \"decent\" initial guesses are more restricted for lower SNR / overlapping components, fewer samples, higher dimensions, more components etc.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the minimax error rate in clustering anisotropic Gaussian mixture models. The authors establish a universal lower bound in two different models: different means, same covariance matrix (resp., different means, different covariance matrix) for every cluster and different covariance. For both models, they prove that a simple iterative algorithm (a variant of Lloyd's) attains the minimax rate.\n\nSTRENGTHS:\nStrong paper. The technical proofs are impressive and the results quite general with a minimum amount of assumptions (clusters all not too small, dimension O(\\sqrt(n)), covariances well conditioned; the only strong assumption is over the initialisation).\n\nWEAKNESSES:\n* The paper misses some high-level intuition, and several proofs are tough to parse. For example, the error decomposition with the terms F, G, H etc appears mysterious and a bit out of nowhere. While the ideal error has a natural explanation (at least when delta = 0), I did not find it properly mentioned. For example, line 435 embodies exactly my feeling while reading the proofs: \"We next define a quantity that refers to as the ideal error\"; yes this quantity is important, yet I do not tell explicitly why and I leave the reader to understand himself why this is indeed an ideal error. \n\n* The interpretation of the minimax rate for model 2 is quite complex. Because it involves hypothesis testing, one should naively expect to find the Chernoff information. Quick computations show that the Chernoff information between N( $\\theta_1, \\Sigma$ ) and N( $\\theta_2, \\Sigma$ ) equals  $ 1 / 8 \\times \\|| \\theta_1 - \\theta_2 \\||^2_{\\Sigma}$  and indeed recovers the minimax rate of Model 1. Expression is more complex for Model 2. That could help interpret the quantity SNR' (which is not truly a signal-to-noise ratio, as it is hard from the expression of SNR'_{ab} to find where is the signal, the noise and the ratio...). \n\n* The assumption on the loss of the initialisation is quite strong (\\hat z^{(0)} should already be a consistent estimator). Especially, the authors consider spectral clustering, which does not scale as well as iterative methods such as Lloyd's. Moreover, spectral clustering is rate-optimal in an isotropic mixture model. A natural question is: what happens if one takes the best over 10 runs over random initialisation \\hat z^{(0)} (or something more clever such as k-means++)? Numerical simulations could show whether a strong condition on the initialisation is indeed needed or not. In particular, Paper [1] had weaker conditions on the initialisation (but stronger conditions on everything else). \n\n* In large dimensions (say d >> n), the lower bound derived here cannot be attained by algorithms agnostic on the model parameters. While this is obviously out of the scope of the paper, the authors should mention it more clearly (key reference 14 is a bit diluted in the intro, other recent work such as [2], albeit posterior to the author's submission, could also appear).  \n\n[1] Lu, Y., & Zhou, H. H. (2016). Statistical and computational guarantees of Lloyd's algorithm and its variants. arXiv preprint arXiv:1612.02099.\n\n[2] Even, B., Giraud, C., & Verzelen, N. (2024). Computation-information gap in high-dimensional clustering. arXiv preprint arXiv:2402.18378.\n\n\nMinor comments:\n* line 377: \"where we use k = O(1)\". No, you use SNR / log k >> 1.\n* Typo in line 189: the with probability 1-exp(-0.08) should be 1-exp(-0.08n) according to [12, Proposition D.1]. This typo appears several times (Corollary 2.1, 3.1 at least) and is important to correct. \n* line 290: I assume the log is in basis 10. \n* Line 641: typo in the ref.\n* Line 661: I don't think the SE(a,b) is properly defined (namely, a proper definition to explain what the a and b in SE(a,b) exactly stands for).\n\nQUESTIONS:\n* Is the assumption SNR / log k \\gg 1 needed? It seems to appear in the lower bound because of the choice of the space \\mathcal{Z}. Would a more refined analysis delete this assumption? (By this comment, I am specifically referring to the paragraph \"If our goal is only to obtain a lower bound ... there exists a much simpler way\" in [1].) I am not sure if this assumption is needed in the upper bound (for example Lemma A.5 for the ideal error does not seem to require it?). \n\n* Line 42 the sentence: 'From a minimax perspective, the least favorable scenario among all sub-Gaussian distributions with variance proxy \\sigma^2—and thus the most challenging for clustering—is when the errors are distributed as N (0, \\sigma^2 I)\". Does it mean that the SNR obtained in Model 2 is always larger than the SNR of Model 1 (with the same centres \\theta_a but all covariances equal to, say, \\Sigma_1)? \n\n* I am not sure how to obtain the second to last inequality of the inequations starting line 473. I get that Cauchy-Schwarz is applied, but it leads to something slightly different (more precisely, I don't know how to obtain the $\\|| \\sum_{a} 1(z_j = a) \\epsilon_j \\epsilon_j^T \\||$ as Cauchy-Schwarz gives $\\|| \\epsilon_j \\||$. I may miss a simple one-liner showing that $ \\sum_{a} 1(z_j = a)  \\|| \\epsilon_j \\|| = \\|| \\sum_{a} 1(z_j = a) \\epsilon_j \\epsilon_j^T \\||$ . \n\n* I do not understand the last statement \"Future work will explore broader covariance structures and refine these methods to further bridge theoretical robustness with computational feasibility in complex clustering scenarios.\" For the first part of the sentence, I believe Model 2 already encompasses the broadest covariance structure (besides some assumptions of \\Sigma^* being well-conditioned). For the second part, if the authors mean robustness to noise, they should cite recent work addressing this in isotropic Gaussian mixture models (for example [1,2]). \n\nRef: \n\n[1]  Anderson Y. Zhang. Harrison H. Zhou. \"Minimax rates of community detection in stochastic block models.\" Ann. Statist. 44 (5) 2252 - 2280, October 2016. https://doi.org/10.1214/15-AOS1428\n\n[2] Liu, Allen, and Ankur Moitra. \"Robustly learning general mixtures of Gaussians.\" Journal of the ACM 70.3 (2023): 1-53.\n\n[3] Patel, D., Shen, H., Bhamidi, S., Liu, Y., & Pipiras, V. (2023). Consistency of Lloyd's Algorithm Under Perturbations. arXiv preprint arXiv:2309.00578.\n\nLIMITATIONS:\n* Albeit this is already a long and dense paper, I would have enjoyed a section motivating anisotropic mixture models in real-data examples. For example: standard Lloyd versus Algorithm 1 versus Algorithm 2. I believe Algorithm 2 obtains a higher accuracy in large data sets, but may not in smaller data sets (where the estimation of the covariance) may lead to less reliable predictions. Having (even heuristics) guidelines on which algorithm one should use depending on the data set size is important. \n\n* While I went quite far down the proof, I found them hard to read. Appendix C is just a dump of technical Lemmas over 17 (!!) pages with no structure. Come on, this is not really serious...", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper provides a minimax lower bound of misclustering error rate for clustering under the anisotropic GMM. Then, the paper designs an adjusted Lloyd's algorithm which can obtain the minimax lower bound within log(n) iterations. The paper also conducts some experiments to show the performance of the proposed method.

STRENGTHS:
1. The paper tackles a more difficult setting of GMM, i.e., anisotropic GMM. The paper provides the bound of the misclustering error rate in this setting and designs a rate-optimal algorithm, the adjusted Lloyd's algorithm. 

2. The paper is theoretically sound and solid.

WEAKNESSES:
1. My major concern is about its experiments. The paper only conducts experiments on synthetic data sets. In machine learning community, we often care more about the performance on the real-world data sets.

2. In Algorithm 2, it needs to compute the inverse and determinant of the covariance matrices, which seems time-consuming. Could you give the time complexity analysis and experiments on how much the overhead has increased compared to the vanilla Lloyd's algorithm?

QUESTIONS:
Please see above.

LIMITATIONS:
The paper claimed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09