Title: Mini-batch kernel $k$-means
Venue: NeurIPS 2025
Accepted: no
Keywords: minibatch

kernel

k-means
Abstract: We present the first mini-batch kernel $k$-means algorithm, offering an order of magnitude improvement in running time compared to the full batch algorithm. A single iteration of our algorithm takes $\widetilde{O}(kb^2)$ time, significantly faster than the $O(n^2)$ time required by the full batch kernel $k$-means, where $n$ is the dataset size and $b$ is the batch size. Extensive experiments demonstrate that our algorithm consistently achieves a 10-100x speedup with minimal loss in quality, addressing the slow runtime that has limited kernel $k$-means adoption in practice. We further complement these results with a theoretical analysis under an early stopping condition, proving that with a batch size of $\widetilde{\Omega}(\max \set{\gamma^{4}, \gamma^{2}}\cdot k\epsilon^{-2} )$, the algorithm terminates in $O(\gamma^2/\epsilon)$ iterations with high probability, where $\gamma$ bounds the norm of points in feature space and $\epsilon$ is a termination threshold. Our analysis holds for any reasonable center initialization, and when using $k$-means++ initialization, the algorithm achieves an approximation ratio of $O(\log k)$ in expectation. 
For normalized kernels, such as Gaussian or Laplacian it holds that $\gamma=1$. Taking $\epsilon = O(1)$ and $b=\Theta(k\log n)$, the algorithm terminates in $O(1)$ iterations, with each iteration running in $\widetilde{O}(k^3)$ time.

Abstract

We present the first mini-batch kernel k -means algorithm, offering an order of magnitude improvement in running time compared to the full batch algorithm. A single iteration of our algorithm takes ˜ O ( kb 2 ) time, significantly faster than the O ( n 2 ) time required by the full batch kernel k -means, where n is the dataset size and b is the batch size. Extensive experiments demonstrate that our algorithm consistently achieves a 10-100x speedup with minimal loss in quality, addressing the slow runtime that has limited kernel k -means adoption in practice. We further complement these results with a theoretical analysis under an early stopping condition, proving that with a batch size of ˜ Ω(max { γ 4 , γ 2 } · kϵ -2 ) , the algorithm terminates in O ( γ 2 /ϵ ) iterations with high probability, where γ bounds the norm of points in feature space and ϵ is a termination threshold. Our analysis holds for any reasonable center initialization, and when using k -means++ initialization, the algorithm achieves an approximation ratio of O (log k ) in expectation. For normalized kernels, such as Gaussian or Laplacian it holds that γ = 1 . Taking ϵ = O (1) and b = Θ( k log n ) , the algorithm terminates in O (1) iterations, with each iteration running in ˜ O ( k 3 ) time.

1 Introduction

Mini-batch methods are among the most successful tools for handling huge datasets for machine learning. Notable examples include Stochastic Gradient Descent (SGD) and mini-batch k -means [30]. Mini-batch k -means is one of the most popular clustering algorithms used in practice [24].

While k -means is widely used due to it's simplicity and fast running time, it requires the data to be linearly separable to achieve meaningful clustering. Unfortunately, many real-world datasets do not have this property. One way to overcome this problem is to project the data into a high, even infinite , dimensional space (where it is hopefully linearly separable) and run k -means on the projected data using the 'kernel-trick'. A toy example is given in Figure 1 and a more realistic example is given in Figure 2.

Kernel k -means achieves significantly better clustering compared to k -means in practice. However, its running time is considerably slower. Surprisingly, prior to our work there was no attempt to speed up kernel k -means using a mini-batch approach.

Problem statement We are given an input (dataset), X = { x i } n i =1 , of size n and a parameter 33 k representing the number of clusters. A kernel for X is a function K : X × X → R that can be 34 realized by inner products. That is, there exists a Hilbert space H and a map ϕ : X →H such that 35 ∀ x, y ∈ X, ⟨ ϕ ( x ) , ϕ ( y ) ⟩ = K ( x, y ) . We call H the feature space and ϕ the feature map . 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

Mini-batchk-means

ARI: 0.069

Image

In kernel k -means the input is a dataset X and a kernel function K as above. Our goal is to find a set C of k centers (elements in H ) such that the following goal function is minimized: 1 n ∑ x ∈ X min c ∈C ∥ c -

ϕ

(

x

)

∥

2

.

Equivalently we may ask for a partition of

X

into

k

parts, keeping

C

implicit.

1

Lloyd's algorithm The most popular algorithm for (non kernel) k -means is Lloyd's algorithm, often referred to as the k -means algorithm [20]. It works by randomly initializing a set of k centers and performing the following two steps: (1) Assign every point in X to the center closest to it. (2) Update every center to be the mean of the points assigned to it. The algorithm terminates when no point is reassigned to a new center. This algorithm is extremely fast in practice but has a worst-case exponential running time [4, 33].

Mini-batch k -means To update the centers, Lloyd's algorithm must go over the entire input at every iteration. This can be computationally expensive when the input data is extremely large. To tackle this, the mini-batch k -means method was introduced by Sculley [30]. It is similar to Lloyd's algorithm except that steps (1) and (2) are performed on a batch of b elements sampled uniformly at random with repetitions, and in step (2) the centers are updated slightly differently. Specifically, every center is updated to be the weighted average of its current value and the mean of the points (in the batch) assigned to it. The parameter by which we weigh these values is called the learning rate , and its value differs between centers and iterations. The larger the learning rate, the more a center will drift towards the new batch cluster mean.

Lloyd's algorithm in feature space Implementing Lloyd's algorithm in feature space is challeng55 ing as we cannot explicitly keep the set of centers C . Luckily, we can use the kernel function together 56 with the fact that centers are always set to be the mean of cluster points to compute the distance from 57

any point x ∈ X in feature space to any center c = 1 | A | ∑ y ∈ A ϕ ( y ) as follows: 58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

where A can be any subset of the input X . While the above can be computed using only kernel evaluations, it makes the update step significantly more costly than standard k -means. Specifically, the complexity of the above may be quadratic in n [11].

Mini-batch kernel k -means Applying the mini-batch approach for kernel k -means is even more difficult because the assumption that cluster centers are always the mean of some subset of X in feature space no longer holds.

In Section 4 we first derive a recursive expression that allows us to compute the distances of all points to current cluster centers (in feature space). Using a simple dynamic programming approach that maintains the inner products between the data and centers in feature space, we achieve a running time of O ( n ( b + k )) per iteration compared to O ( n 2 ) for the full-batch algorithm. However, a true mini-batch algorithm should have a running time sublinear in n , preferably only polylogarithmic. We show that the recursive expression can be truncated , achieving a fast update time of ˜ O ( kb 2 ) while only incurring a small additive error compared to the untruncated version 2 .

While our main contribution is practical - achieving an order-of-magnitude speedup for kernel k -means - we also provide theoretical guarantees for our algorithm (deferred to Appendix B). This is somewhat tricky for mini-batch algorithms due to their stochastic nature, as they may not even converge to a local-minima. To overcome this hurdle, we take the approach of Schwartzman [29] and answer the question: how long does it take truncated mini-batch kernel k -means to terminate with an early stopping condition . Specifically, we terminate the algorithm when the improvement on the batch drops below some user provided parameter, ϵ . Early stopping conditions are very common in practice (e.g., sklearn [24]). We show that applying the k -means++ initialization scheme [5] for our initial centers implies we achieve the same approximation ratio, O (log k ) in expectation, as the full-batch algorithm.

While our general approach is similar to [29], we must deal with the fact that H may have an infinite dimension. The guarantees of [29] depend on the dimension of the space in which k -means is executed, which is unacceptable in our case. We overcome this by parameterizing our results by a new parameter γ = max x ∈ X ∥ ϕ ( x ) ∥ . We note that for normalized kernels, such as the popular Gaussian and Laplacian kernels, it holds that γ = 1 . We also observe that it is often the case that γ ≪ 1 for various other kernels used in practice (see Appendix C). We show that if the batch size is Ω(max { γ 4 , γ 2 } kϵ -2 log 2 ( γn/ϵ )) then w.h.p. our algorithm terminates in O ( γ 2 /ϵ ) iterations. Our theoretical results are summarised in Theorem 1.1 (where Algorithm 2 is presented in Section 4).

Theorem 1.1. The following holds for Algorithm 2: (1) Each iteration takes O ( kb 2 log 2 ( γ/ϵ )) time, (2) If b = Ω(max { γ 4 , γ 2 } kϵ -2 log 2 ( γn/ϵ )) then it terminates in O ( γ 2 /ϵ ) iterations w.h.p, (3) When initialized with k -means++ it achieve a O (log k ) approximation ratio in expectation.

Our result improves upon [29] significantly when a normalized kernel is used since Theorem 1.1 doesn't depend on the input dimension. Our algorithm copes better with non linearly separable data and requires a smaller batch size ( ˜ Ω(1 /ϵ 2 ) vs ˜ Ω(( d/ϵ ) 2 )) ) 3 for normalized kernels. This is particularly apparent with high dimensional datasets such as MNIST [18] where the dimension squared is already nearly ten times the number of datapoints.

The learning rate we use, suggested in [29], differs from the standard learning rate of sklearn in that it does not go to 0 over time. Unfortunately, this new learning rate is non-standard and [29] did not present experiments comparing their learning rate to that of sklearn. We fill the experimental gap left in [29] by evaluating (non-kernel) mini-batch k -means with their new learning rate compared to that of sklearn. Following our experimental evaluation, the sklearn team accepted a pull request implementing this learning rate in future versions.

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

In Section 5 we extensively evaluate our results experimentally both with the learning rate of [29] and that of sklearn. To allow a fair empirical comparison, we run each algorithm for a fixed number of iterations without stopping conditions. Our results are as follows: 1) Truncated mini-batch kernel k -means is significantly faster than full-batch kernel k -means, while achieving solutions of similar quality, which are superior to the non-kernel version, 2) The learning rate of [29] results in solutions with better quality both for truncated mini-batch kernel k -means and (non-kernel) mini-batch k -means.

2 Related work

Until recently, mini-batch k -means was only considered with a learning rate going to 0 over time. This was true both in theory [32, 30] and practice [24]. Recently, [29] proposed a new learning which does not go to 0 over time, and showed that if the batch is of size ˜ Ω( k ( d/ϵ ) 2 ) 4 , mini-batch k -means must terminate within O ( d/ϵ ) iterations with high probability, where d is the dimension of the input, and ϵ is a threshold parameter for termination.

A popular approach to deal with the slow running time of kernel k -means is constructing a coreset of the data. A coreset for kernel k -means is a weighted subset of X with the guarantee that the solution quality on the coreset is close to that on the entire dataset up to a (1 + ϵ ) multiplicative factor. There has been a long line of work on coresets for k -means an kernel k -means [28, 12, 6], and the current state-of-the-art for kernel k -means is due to [15]. They present a coreset algorithm with a nearly linear (in n and k ) construction time which outputs a coreset of size poly ( kϵ -1 ) .

In [8] the authors only compute the kernel matrix for uniformly sampled set of m points from X . Then they optimize a variant of kernel k -means where the centers are constrained to be linear combinations of the sampled points. The authors do no provide worst case guarantees for the running time or approximation of their algorithm.

Another approach to speed up kernel k -means is by computing an approximation for the kernel matrix. This can be done by computing a low dimensional approximation for ϕ (without computing ϕ explicitly)[26, 9, 7], or by computing a low rank approximation for the kernel matrix [22, 34].

Kernel sparsification techniques construct sparse approximations of the full kernel matrix in subquadratic time. For smooth kernel functions such as the polynomial kernel, [25] presents an algorithm for constructing a (1 + ϵ ) -spectral sparsifier for the full kernel matrix with a nearly linear number of non-zero entries in nearly linear time. For the gaussian kernel, [21] show how to construct a weaker, cluster preserving sparsifier using a nearly linear number of kernel density estimation queries.

We note that our results are complementary to coresets, dimensionality reduction, and kernel sparsification, in the sense that we can compose our method with these techniques.

To the best of our knowledge, the only approach which cannot be directly composed with our work is kernel sketching [19, 35]. Here the kernel matrix is used to compute an embedding of the points into a low dimensional Euclidean space, followed by running the standard (non-kernel) k -means algorithm. We compare our algorithm with the state of the art results [35] and observe that our algorithm achieves solutions of superior quality for most datasets.

3 Preliminaries

Throughout this paper we work with ordered tuples rather than sets, denoted as Y = ( y i ) i ∈ [ ℓ ] , where [ ℓ ] = { 1 , . . . , ℓ } . To reference the i -th element we either write y i or Y [ i ] . It will be useful to use set notations for tuples such as x ∈ Y ⇐⇒ ∃ i ∈ [ ℓ ] , x = y i and Y ⊆ Z ⇐⇒ ∀ i ∈ [ ℓ ] , y i ∈ Z . When summing we often write ∑ x ∈ Y g ( x ) which is equivalent to ∑ ℓ i =1 g ( Y [ i ]) .

We borrow the following notation from [16] and generalize it to Hilbert spaces. For every x, y ∈ H let ∆( x, y ) = ∥ x -y ∥ 2 . We slightly abuse notation and and also write ∆( x, y ) = ∥ ϕ ( x ) -ϕ ( y ) ∥ 2 when x, y ∈ X and ∆( x, y ) = ∥ ϕ ( x ) -y ∥ 2 when x ∈ X,y ∈ H (similarly when x ∈ H , y ∈ X ). For every finite tuple S ⊆ X and a vector x ∈ H let ∆( S, x ) = ∑ y ∈ S ∆( y, x ) . Let us denote

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

γ = max x ∈ X ∥ ϕ ( x ) ∥ . Let us define for any finite tuple S ⊆ X the center of mass of the tuple as cm ( S ) = 1 | S | ∑ x ∈ S ϕ ( x ) .

Kernel k -means We are given an input X = ( x i ) n i =1 and a parameter k . Our goal is to (implicitly) find a tuple C ⊆ H of k centers such that the following goal function is minimized: 1 n ∑ x ∈ X min C ∈C ∆( x, C ) .

Let us define for every x ∈ X the function f x : H k → R where f x ( C ) = min C ∈C ∆( x, C ) . We can treat H k as the set of k -tuples of vectors in H . We also define the following function for every tuple A = ( a i ) ℓ i =1 ⊆ X : f A ( C ) = 1 ℓ ∑ ℓ i =1 f a i ( C ) . Note that f X is our original goal function.

We make extensive use of the notion of convex combination :

Definition 3.1. We say that y ∈ H is a convex combination of X if y = ∑ x ∈ X p x ϕ ( x ) , such that ∀ x ∈ X,p x ≥ 0 and ∑ x ∈ X p x = 1 .

4 Our Algorithm

We start by presenting a slower algorithm that will set the stage for our truncated mini-batch algorithm and will be useful during the analysis. We present our pseudo-code in Algorithm 1. It requires an initial set of cluster centers such that every center is a convex combination of X . This guarantees that all subsequent centers are also a convex combination of X . Note that if we initialize the centers using the kernel version of k -means++, this is indeed the case.

Algorithm 1 proceeds by repeatedly sampling a batch of size b (the batch size is a parameter). For the i -th batch the algorithm (implicitly) updates the centers using the learning rate α i j for center j . Note that the learning rate may take on different values for different centers, and may change between iterations. Finally, the algorithm terminates when the progress on the batch is below ϵ , a user provided parameter. While our termination guarantees (Appendix B) require a specific learning rate, it does not affect the running time of a single iteration, and we leave it as a parameter for now.

Recursive distance update rule Unlike k -means, the center updates and assignment of points to clusters is tricky for kernel k -means and even harder for mini-batch kernel k -means. Specifically, how do we overcome the challenge that we do not maintain the centers explicitly?

To assign points to centers in the ( i + 1) -th iteration, it is sufficient to know ∥ ϕ ( x ) - C j i +1 ∥ 2 for every j . This is because we are interested in the closest center to x in kernel space. If we can keep track of this quantity through the execution of the algorithm, we are done. Let us derive a recursive expression for the distances: ∥ ϕ ( x ) -C j i +1 ∥ 2 = ⟨ ϕ ( x ) , ϕ ( x ) ⟩ -2 ⟨ ϕ ( x ) , C j i +1 ⟩ + ⟨C j i +1 , C j i +1 ⟩ .

Let us expand ⟨ ϕ ( x ) , C j i +1 ⟩ and ⟨C j i +1 , C j i +1 ⟩ : 181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

The above is all we need to compute the distances. Furthermore, it is possible to use dynamic programming to update the center for every iteration in O ( n ( b + k )) time and O ( nk ) space (proof deferred to Appendix A). This is a considerable speedup compared to the best known quadratic update time. Next, we go a step further and show that it is possible to get an update time with only polylogarithmic dependence on n .

4.1 Truncating the centers

The issue with the above approach is that each center is written as a linear combination of potentially all points in X . We now present a simple way to overcome this issue. We maintain C j i +1 as an explicit sparse linear combination of X . Let us expand the recursive expression of C j i +1 for t terms, assuming t < i :

The idea behind our truncation technique is that when t is sufficiently large, the term C j i -t Π t ℓ =0 (1 -α j i -ℓ ) becomes very small and can be discarded. The rate by which this term decays depends on the learning rates, which in turn depend on the number of elements assigned to the cluster in each of the previous iterations.

Let us start with some definitions. Let us denote b j i = ∣ ∣ ∣ B j i ∣ ∣ ∣ . We would like to trim the recursive expression such that every cluster center is represented using about τ points, where τ is a parameter to be set later. We define Q j i to be the set of indices from i to i -t , where t is the smallest integer such that ∑ ℓ ∈ Q j i b j i ≥ τ holds. If no such integer exists then Q j i = { i, i -1 , . . . , 1 } . It is the case that ∑ ℓ ∈ Q j i b j i ≤ τ + b . Intuitively, Q j i is the most recent window of updates to cluster j that contains enough points (at least τ ) to serve as a sufficient approximation of the current cluster center.

Next we define the truncated centers , for which the contributions of older points to the centers are forgotten after about τ points have been assigned to the center:

From the above definition it is always the case that either ̂ C j i +1 = C j i +1 or ∑ ℓ ∈ Q j i b j i ≥ τ . The following lemma shows that when τ is sufficiently large ∥ ̂ C j i +1 -C j i +1 ∥ is small. Intuitively, this implies that the truncated algorithm should achieve results similar to the untruncated version (we formalize this intuition in Appendix B).

Proof. We assume that ∑ ℓ ∈ Q j i b j i ≥ τ , as otherwise the claim trivially holds.

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

Algorithm implmentation and runtime To implement this, we simply need to swap C j i in Algorithm 1 with ̂ C j i (Lines 7 and 8). As before, the main bottleneck of each iteration is assigning points in the batch to their closest center. Once this is done, updating the truncated centers is straightforward by simply adjusting the coefficients in (1), removing the last element from the sum and adding a new element to the sum 5 . If min { Q j i } is 1, then we also need to add C j 1 Π ℓ ∈ Q j i (1 -α j ℓ ) which guarantees that ̂ C j i = C j i . The pseudo code is provided in Algorithm 2.

As before, let us consider assigning all points in the ( i +1) iteration to their closest centers. Unlike the previous approach, when computing distances between points in B i +1 and ̂ C i +1 we can do this directly (without recursion) and it is now sufficient to consider a much smaller set of inner products.

As before, the terms we are interested in computing are: ⟨ ϕ ( x ) , ̂ C j i +1 ⟩ and ⟨ ̂ C j i +1 , ̂ C j i +1 ⟩ . However, there are several differences to the previous approach. We no longer need ⟨ ϕ ( x ) , ̂ C j i +1 ⟩ for all x ∈ X , but only for x ∈ B i +1 . Furthermore, ̂ C j i +1 can be simply written as a weighted sum of at most ∑ ℓ ∈ Q j i b j ℓ ≤ τ + b terms. Summing over all element in B i +1 and k centers we get O ( kb ( b + τ )) time to compute ⟨ ϕ ( x ) , ̂ C j i +1 ⟩ . For ⟨ ̂ C j i +1 , ̂ C j i +1 ⟩ using the bound on the number of terms we directly get O ( k ( τ + b ) 2 ) time. We conclude that every iteration of Algorithm 2 requires O ( k ( τ + b ) 2 ) = ˜ O ( kb 2 ) time. The additional space required is O ( kτ ) = ˜ O ( kb ) .

5 Experiments

We evaluate our algorithms on the following datasets:

MNIST: The MNIST dataset [18] has 70,000 grayscale images of handwritten digits (0 to 9), each image being 28x28 pixels. When flattened, this gives 784 features. PenDigits: The PenDigits dataset [1] has 10992 instances, each represented by an 16-dimensional vector derived from 2D pen movements. The dataset has 10 labelled clusters, one for each digit. Letters: The Letters dataset [31] has 20,000 instances of letters from 'A' to 'Z', each represented by 16 features. The dataset has 26 labelled clusters, one for each letter. HAR: The HAR dataset [2] has 10,299 instances collected from smartphone sensors, capturing human activities like walking, sitting, and standing. Each instance is described by 561 features. It has 6 labeled clusters, corresponding to different types of physical activities.

We compare the following algorithms: full-batch kernel k -means, truncated mini-batch kernel k -239 means, and mini-batch k -means (both kernel and non-kernel) with learning rates from [29] and sklearn. 240 We also implement the three kernel sketching algorithms of Yin et al [35] that use either sub-Gaussian, 241 randomized orthogonal system (ROS), or Nyström sketches. After sketching, we run k -means. We 242 set the dimension of the sketch to 150, the same as in the experiments of [35]. We evaluate our results 243 with batch sizes : 2048 , 1024 , 512 , 256 and τ : 50 , 100 , 200 , 300 . We execute every algorithm for 200 244 iterations. For the results below, we apply the Gaussian kernel: K ( x, y ) = e -∥ x -y ∥ 2 /κ , where the κ 245 parameter is set using the heuristic of [34] followed by some manual tuning (exact values appear in 246 the supplementary materials). We also run experiments with heat and knn kernels in Appendix C. 247 We repeat every experiment 10 times and present the average Adjusted Rand Index (ARI) [13, 27], 248 Normalized Mutual Information (NMI) [17] and Accuracy (ACC) 6 scores for every dataset. All 249 experiments were conducted using an AMD Ryzen 9 7950X CPU with 128GB of RAM and a Nvidia 250 GeForce RTX 4090 GPU. We present partial results in Figure 3 and the full results in Appendix C. 251 Error bars in the plot measure the standard deviation.

pendigits

mnist_784

102

0.8

103

101

Time (s) log scale

0.6

102

Score

100

101

0.4

100

10-1

0.2

10-1

10-2

0.0

ARI

NMI

ACC

time

ARI

NMI

ACC

time

letter

0.8

102

101

Time (s) log scale

0.6

101

Score

100

0.4

100

0.2

10-1

10-1p

0.0

10-2

10-2

ARI

NMI

ACC

time

ARI

NMI

ACC

time

X

Kernel

MiniBatchKernel

Kernel sketching (ROs)

Truncatedβ-MiniBatchKernel

β-MiniBatch

Kernel sketching (Nystrom)

β-MiniBatchKernel

MiniBatch

Kernel sketching (Sub-Gaussian)

Discussion Throughout our results, we consistently observe that the truncated algorithm achieves performance on par with the non-truncated version with a running time which is often an order of magnitude faster. Surprisingly, this often holds for tiny values of τ (e.g., 50) far below the theoretical threshold (i.e., τ ≪ b ). We also achieve considerably better quality solutions on most datasets compared to kernel sketching. We believe that our approach achieves a good balance between speed and performance, and is a valuable addition to the tool-box of clustering algorithms.

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

References

E. Alpaydin and Fevzi. Alimoglu. Pen-Based Recognition of Handwritten Digits. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C5MG6K. License: CC BY 4.0 DEED, available at https://creativecommons.org/licenses/by/4.0/ .

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. A public domain dataset for human activity recognition using smartphones. In Esann , volume 3, page 3, 2013. License: CC BY-NC-SA 4.0 DEED, available at https://creativecommons. org/licenses/by-nc-sa/4.0/ .

Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence , 33(5):898-916, 2010.

David Arthur and Sergei Vassilvitskii. How slow is the k -means method? In SCG , pages 144-153. ACM, 2006.

David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In SODA , pages 1027-1035. SIAM, 2007.

Artem Barger and Dan Feldman. Deterministic coresets for k-means of big sparse data. Algorithms , 13(4):92, 2020.

Di Chen and Jeff M Phillips. Relative error embeddings of the gaussian kernel distance. In International Conference on Algorithmic Learning Theory , pages 560-576. PMLR, 2017.

Radha Chitta, Rong Jin, Timothy C. Havens, and Anil K. Jain. Approximate kernel k-means: solution to large scale kernel clustering. In KDD , pages 895-903. ACM, 2011.

Radha Chitta, Rong Jin, and Anil K Jain. Efficient kernel clustering using random fourier features. In 2012 IEEE 12th International Conference on Data Mining , pages 161-170. IEEE, 2012.

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Inderjit S. Dhillon, Yuqiang Guan, and Brian Kulis. Kernel k-means: spectral clustering and normalized cuts. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '04, page 551-556, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138881. doi: 10.1145/1014052.1014118. URL https://doi.org/10.1145/1014052.1014118 .

Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constantsize coresets for k-means, pca, and projective clustering. SIAM Journal on Computing , 49(3): 601-657, 2020.

Alexander J Gates and Yong-Yeol Ahn. The impact of random models on clustering similarity. Journal of Machine Learning Research , 18(87):1-28, 2017.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Shaofeng H.-C. Jiang, Robert Krauthgamer, Jianing Lou, and Yubo Zhang. Coresets for kernel clustering. CoRR , abs/2110.02898, 2021.

Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. A local search approximation algorithm for k-means clustering. Comput. Geom. , 28(2-3):89-112, 2004.

Andrea Lancichinetti, Santo Fortunato, and János Kertész. Detecting the overlapping and hierarchical community structure in complex networks. New journal of physics , 11(3):033015, 2009.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 304 1998. License: CC0 1.0 DEED CC0 1.0 Universal, available at https://creativecommons. 305 org/publicdomain/zero/1.0/ . 306

Yong Liu. Refined learning bounds for kernel and approximate k -means. Advances in neural 307 information processing systems , 34:6142-6154, 2021. 308

Stuart P. Lloyd. Least squares quantization in PCM. IEEE Trans. Inf. Theory , 28(2):129-136, 309 1982. 310

Peter Macgregor and He Sun. Fast approximation of similarity graphs with kernel density 311 estimation. Advances in Neural Information Processing Systems , 36, 2024. 312

Cameron Musco and Christopher Musco. Recursive sampling for the nystrom method. Advances 313 in neural information processing systems , 30, 2017. 314

Assaf Naor. On the banach-space-valued azuma inequality and small-set isoperimetry of 315 alon-roichman graphs. Combinatorics, Probability and Computing , 21(4):623-634, 2012. 316

317

318

319

320

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

Kent Quanrud. Spectral Sparsification of Metrics and Kernels , pages 1445-1464. doi: 10.1137/1.9781611976465.87. URL https://epubs.siam.org/doi/abs/10.1137/1. 9781611976465.87 .

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association , 66(336):846-850, 1971.

Melanie Schmidt. Coresets and streaming algorithms for the k-means problem and related clustering objectives. 2014.

Gregory Schwartzman. Mini-batch k-means terminates within O ( d/ϵ ) iterations. In ICLR , 2023.

D. Sculley. Web-scale k-means clustering. In WWW , pages 1177-1178. ACM, 2010.

David Slate. Letter Recognition. UCI Machine Learning Repository, 1991. DOI: https://doi.org/10.24432/C5ZP40. License: CC BY 4.0 DEED, available at https:// creativecommons.org/licenses/by/4.0/ .

Cheng Tang and Claire Monteleoni. Convergence rate of stochastic k-means. In AISTATS , volume 54 of Proceedings of Machine Learning Research , pages 1495-1503. PMLR, 2017.

Andrea Vattani. k -means requires exponentially many iterations even in the plane. Discret. Comput. Geom. , 45(4):596-616, 2011.

Shusen Wang, Alex Gittens, and Michael W Mahoney. Scalable kernel k-means clustering with nystrom approximation: Relative-error bounds. Journal of Machine Learning Research , 20(12): 1-49, 2019.

Rong Yin, Yong Liu, Weiping Wang, and Dan Meng. Randomized sketches for clustering: 343 Fast and optimal kernel k -means. Advances in Neural Information Processing Systems , 35: 344 6424-6436, 2022. 345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

A Omitted proofs and Algorithms for Section 4

Runtime analysis of Algorithm 1 Assuming that ⟨C j i , C j i ⟩ and ⟨ ϕ ( x ) , C j i ⟩ are known for all j ∈ [ k ] and for all x ∈ X , we can compute ⟨C j i +1 , C j i +1 ⟩ and ⟨ ϕ ( x ) , C j i +1 ⟩ for all j ∈ [ k ] and x ∈ X , which implies we can compute the distances from any point in the batch to all centers.

We now bound the running time of a single iteration of the outer loop in Algorithm 1. Let us denote b j i = ∣ ∣ ∣ B j i ∣ ∣ ∣ and recall that cm ( B j i ) = 1 b j i ∑ y ∈ B j i ϕ ( y ) . Therefore, computing ⟨ ϕ ( x ) , cm ( B j i ) ⟩ = 1 b j i ∑ y ∈ B j i ⟨ ϕ ( x ) , ϕ ( y ) ⟩ requires O ( b j i ) time. Similarly, computing ⟨ cm ( B j i ) , cm ( B j i ) ⟩ requires O (( b j i ) 2 ) time. Let us now bound the time it requires to compute ⟨ ϕ ( x ) , C j i +1 ⟩ and ⟨C j i +1 , C j i +1 ⟩ .

Assuming we know ⟨ ϕ ( x ) , C j i ⟩ and ⟨C j i , C j i ⟩ , updating ⟨ ϕ ( x ) , C j i +1 ⟩ for all x ∈ X,j ∈ [ k ] requires O ( n ( b + k )) time. Specifically, the ⟨ ϕ ( x ) , C j i ⟩ term is already known from the previous iteration and we need to compute α j i ⟨ ϕ ( x ) , cm ( B j i ) ⟩ for every x ∈ X,j ∈ [ k ] which requires n ∑ j ∈ [ k ] b j i = nb time. Finally, updating ⟨ ϕ ( x ) , C j i +1 ⟩ for all x ∈ X,j ∈ [ k ] requires O ( nk ) time.

Updating ⟨C j i +1 , C j i +1 ⟩ requires O ( b 2 + kb ) time. Specifically, ⟨C j i , C j i ⟩ is known from the previous iteration and computing ⟨ cm ( B j ) , cm ( B j ) ⟩ for all j ∈ [ k ] requires O ( ∑ ( b j ) 2 ) = O ( b 2 )

Computing ⟨C i , cm ( B i ) ⟩ for all j ∈ [ k ] requires time O ( b ) using ⟨ ϕ ( x ) , C i ⟩ from the previous iteration. Therefore, the total running time of the update step (assigning points to new centers) is O ( n ( b + k )) . To perform the update at the ( i +1) -th step we only need ⟨ ϕ ( x ) , C j i ⟩ , ⟨C j i , C j i ⟩ , which results in a space complexity of O ( nk ) . This completes the first claim of Theorem 1.1.

i i j ∈ [ k ] i time. j j j

B Termination guarantee

In this section we prove the second claim of Theorem 1.1. For most of the section we analyze Algorithm 1, and towards the end we use the fact that the centers of the two algorithms are close throughout the execution to conclude our proof.

Section preliminaries We introduce the following definitions and lemmas to aid our proof of the second claim of Theorem 1.1.

Lemma B.1. For every y which is a convex combination of X it holds that ∥ y ∥ ≤ γ .

Lemma B.2. For any tuple of k centers C ⊂ H d which are a convex combination of points in X , it holds that ∀ A ⊆ X,f A ( C ) ≤ 4 γ 2 .

Proof. It is sufficient to upper bound f x . Combining that fact that every C ∈ C is a convex combination of X with the triangle inequality, we have that

We state the following simplified version of an Azuma bound for Hilbert space valued martingales from [23], followed by a standard Hoeffding bound.

Theorem B.3 ([23]) . Let H be a Hilbert space and let Y 0 , ..., Y m be a H -valued martingale, such that ∀ 1 ≤ i ≤ m, ∥ Y i -Y i -1 ∥ ≤ a i . It holds that Pr [ ∥ Y m -Y 0 ∥ ≥ δ ] ≤ e Θ ( δ 2 ∑ i m =1 a 2 i ) .

Theorem B.4 ([14]) . Let Y 1 , ..., Y m be independent random variables such that ∀ 1 ≤ i ≤ m,E [ Y i ] = µ and Y i ∈ [ a min , a max ] . Then Pr (∣ ∣ 1 m ∑ m i =1 Y k -µ ∣ ∣ ≥ δ ) ≤ 2 e -2 mδ 2 / ( a max -a min ) 2 .

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

The following lemma provides concentration guarantees when sampling a batch .

Lemma B.5. Let B be a tuple of b elements chosen uniformly at random from X with repetitions. For any fixed tuple of k centers, C ⊆ H which are a convex combination of X , it holds that: Pr [ | f B ( C ) -f X ( C ) | ≥ δ ] ≤ 2 e -bδ 2 / 8 γ 4 .

For any tuple S ⊆ X and some tuple of cluster centers C = ( C ℓ ) ℓ ∈ [ k ] ⊂ H , C implies a partition ( S ℓ ) ℓ ∈ [ k ] of the points in S . Specifically, every S ℓ contains the points in S closest to C ℓ (in H ) and every point in S belongs to a single C ℓ (ties are broken arbitrarily). We state the following useful observation:

Observation B.6 . Fix some A ⊆ X . Let C be a tuple of k centers, S = ( S ℓ ) ℓ ∈ [ k ] be the partition of A induced by C and S = ( S ℓ ) ℓ ∈ [ k ] be any other partition of A . It holds that ∑ k j =1 ∆( S j , C j ) ≤ ∑ k j =1 ∆( S j , C j ) .

Recall that C j i is the j -th center in the beginning of the i -th iteration of Algorithm 1 and ( B ℓ i ) ℓ ∈ [ k ] is the partition of B i induced by C i . Let ( X ℓ i ) ℓ ∈ [ k ] be the partition of X induced by C i .

We now have the tools to analyze Algorithm 1 with the learning rate of [29].

assume that the algorithm executes for at least

t

iterations, the learning rate is

α

j

i

Specifically, we

=

b

√

j

i

/b

, where b j i = ∣ ∣ ∣ B j i ∣ ∣ ∣ , and the batch size is b = Ω(max { γ 4 , γ 2 } kϵ -2 log( nt )) . We show that the algorithm must terminate within t = O ( γ 2 /ϵ ) steps w.h.p. Plugging t back into b , we get that a batch size of b = Ω(max { γ 4 , γ 2 } kϵ -2 log 2 ( γn/ϵ )) is sufficient. We assume that ϵ is chosen such that γ 2 /ϵ > 1 / 4 . Otherwise, the stopping condition immediately holds due to Lemma B.2.

Proof outline We note that when sampling a batch it holds w.h.p that f B i ( C i ) is close to f X i ( C i ) (Lemma B.5). This is due to the fact that B i is sampled after C i is fixed. If we could show that f B i ( C i +1 ) is close f X i ( C i +1 ) then combined with the fact that we make progress of at least ϵ on the batch we can conclude that we make progress of at least some constant fraction of ϵ on the entire dataset.

Unfortunately, as C i +1 depends on B i , getting the above guarantee is tricky. To overcome this issue we define the auxiliary value C j i +1 = (1 -α j i ) C j i + α j i cm ( X j i ) . This is the j -th center at step i +1 if we were to use the entire dataset for the update, rather than just a batch. Note that this is only used in the analysis and not in the algorithm. Note that C i +1 is almost independent of B i . Every C j i +1 depends only on C j i , X j i and α j i . While C j i , X j i are independent of B i , the learning α j i is not .

Nevertheless, the number of possible values of

{

α

j

i

}

j

∈

k

this issue by showing concentration for every possible learning rate configuration followed by a union bound. This allows us to use C i +1 instead of C i +1 in the above analysis outline. We show that for our choice of learning rate it holds that C i +1 , C i +1 are sufficiently close, which implies that f X ( C i +1 ) , f X ( C i +1 ) and f B i ( C i +1 ) , f B i ( C i +1 ) are also sufficiently close. That is, C i +1 acts as a proxy for C i +1 . Combining everything together we get our desired result for Algorithm 1.

We start with the following useful observation, which will allow us to use Lemma B.1 to bound the norm of the centers by γ throughout the execution of the algorithm.

Observation B.7 . If ∀ j ∈ [ k ] , C j 1 is a convex combination of X then ∀ i > 1 , j ∈ [ k ] , C j i , C j i are also a convex combinations of X .

is sufficiently small, and we can overcome

We state the following useful lemma. Although the original proof is for Euclidean spaces, it goes 428 through for Hilbert spaces. 429

Lemma B.8 ([16]) . For any set S ⊆ X and any C ∈ H it holds that ∆( S, C ) = ∆( S, cm ( S )) + 430 | S | ∆( C, cm ( S )) . 431

Proof.

where the last step is due to the fact that 432

433

434

435

436

We use the above to state the following useful lemma.

Lemma B.9. For any S ⊆ X and C, C ′ ∈ H which are convex combinations of X , it holds that: | ∆( S, C ′ ) -∆( S, C ) | ≤ 4 γ | S | ∥ C -C ′ ∥ .

Proof. Using Lemma B.8 we get that ∆( S, C ) = ∆( S, cm ( S )) + | S | ∆( cm ( S ) , C ) and that 437 ∆( S, C ′ ) = ∆( S, cm ( S )) + | S | ∆( cm ( S ) , C ′ ) . Thus, it holds that | ∆( S, C ′ ) -∆( S, C ) | = 438 | S | · | ∆( cm ( S ) , C ′ ) -∆( cm ( S ) , C ) | . Let us write 439

Where in the last transition we used the Cauchy-Schwartz inequality, the triangle inequality, and the 440 fact that C, C ′ , cm ( S ) are convex combinations of X and therefore their norm is bounded by γ . 441

When centers are sufficiently close, these lemmas imply their values are close for any f A . 442

Lemma B.10. Fix some A ⊆ X and let ( C j ) j ∈ [ k ] , ( C j ) j ∈ [ k ] ⊂ H be arbitrary centers such that 443 ∀ j ∈ [ k ] , ∥C j -C j ∥ ≤ ϵ/ 28 γ . It holds that ∀ i ∈ [ t ] , ∣ ∣ f A ( C i +1 ) -f A ( C i +1 ) ∣ ∣ ≤ ϵ/ 7 . 444

Proof. Let S = ( S ℓ ) ℓ ∈ [ k ] , S = ( S ℓ ) ℓ ∈ [ k ] be the partitions induced by C , C on A . Let us expand the 445 expression 446

The first inequality is due to Observation B.6, the second is due Lemma B.9 and finally we use the 447 assumption about the distances between centers together with the fact that ∑ k j =1 ∣ ∣ S j ∣ ∣ = | A | . Using 448 the same argument we also get that f A ( C ) -f A ( C ) ≤ ϵ/ 7 , which completes the proof. 449

450

Now we show that due to our choice of learning rate, C j i +1 and C j i +1 are sufficiently close.

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

Lemma B.11. It holds w.h.p that ∀ i ∈ [ t ] , j ∈ [ k ] , ∥C j i +1 -C j i +1 ∥ ≤ ϵ 28 γ .

Proof. Note that C j i +1 - C j i +1 = α j i ( cm ( B j i ) -cm ( X j i )) . Let us fix some iteration i and center j . To simplify notation, let us denote: X ′ = X j i , B ′ = B j i , b ′ = b j i , α ′ = α j i . Although b ′ is a random variable, in what follows we treat it as a fixed value (essentially conditioning on its value). As what follows holds for all values of b ′ it also holds without conditioning due to the law of total probabilities.

For the rest of the proof, we assume b ′ > 0 (if b ′ = 0 the claim holds trivially). Let us denote by { Y ℓ } b ′ ℓ =1 the sampled points in B ′ . Note that a randomly sampled element from X is in B ′ if and only if it is in X ′ . As batch elements are sampled uniformly at random with repetitions from X , conditioning on the fact that an element is in B ′ means that it is distributed uniformly over X ′ . Note that ∀ ℓ, E [ ϕ ( Y ℓ )] = 1 | X ′ | ∑ x ∈ X ′ ϕ ( x ) = cm ( X ′ ) and E [ cm ( B ′ )] = 1 b ′ ∑ b ′ ℓ =1 E [ ϕ ( Y ℓ )] = cm ( X ′ ) .

Let us define the following martingale: Z r = ∑ r ℓ =1 ( ϕ ( Y ℓ ) -E [ ϕ ( Y ℓ )]) . Note that Z 0 = 0 , and when r > 0 , Z r = ∑ r ℓ =1 ϕ ( Y ℓ ) -r · cm ( X ′ ) . It is easy to see that this is a martingale:

We bound the differences: ∥ Z r -Z r -1 ∥ = ∥ ϕ ( Y r ) -cm ( X ′ ) ∥ ≤ ∥ ϕ ( Y r ) ∥ + ∥ cm ( X ′ ) ∥ ≤ 2 γ.

Now we may use Azuma's inequality: Pr [ ∥ Z b ′ -Z 0 ∥ ≥ δ ] ≤ e -Θ( δ 2 γ 2 b ′ ) . Let us now divide both sides of the inequality by b ′ and set δ = b ′ ϵ 28 γα ′ . We get Pr [ ∥ cm ( B ′ ) -cm ( X ′ ) ∥ ≥ ϵ 28 γα ′ ] = Pr [ ∥ 1 b ′ ∑ b ′ ℓ =1 ϕ ( Y ℓ ) -cm ( X ′ ) ∥ ≥ ϵ 28 γα ′ ] ≤ e -Θ( b ′ ϵ 2 ( γα ′ ) 2 ) . Using the fact that α ′ = √ b ′ /b together with the fact that b = Ω(max { γ 4 , γ 2 } kϵ -2 log( nt )) (for an appropriate constant) we get that the above is O (1 /ntk ) . Finally, taking a union bound over all t iterations and all k centers per iteration completes the proof.

Let us state the following useful lemma.

Lemma B.12. It holds w.h.p that for every i ∈ [ t ] ,

Proof. The first two inequalities follow from Lemma B.10. The third is due to Lemma B.5 by setting 473 δ = ϵ/ 7 , B = B i : 474

The last inequality is a bit more involved 7 . Let ℓ ⃗ ∈ N k be a vector whose entries sum to b . For every ℓ ⃗ 475 we can define C i +1 ( ℓ ⃗ ) such that C j i +1 ( ℓ ⃗ ) = C j i (1 -√ ℓ j /b ) + √ ℓ j /b · cm ( X j i ) . For every choice of 476 ℓ ⃗ it holds that C i +1 ( ℓ ⃗ ) is independent of B i and we can apply Lemma B.5 for every possible C i +1 ( ℓ ⃗ ) 477 by setting δ = ϵ/ 7 , B = B i 478

where last inequality is due to the fact that b = Ω(max { γ 4 , γ 2 } kϵ -2 log( nt )) (for an appropriate constant). Finally, we take a union bound over all possible vectors ℓ ⃗ , a total of ( b + k -1 k -1 ) ≤ ( ( b + k -1) · e k -1 ) k -1 = O ( n k -1 ) . As C i +1 corresponds to at least one C i +1 ( ℓ ⃗ ) we are done.

Taking a union bound over t iterations, we obtain the result.

Putting everything together We wish to lower bound f X ( C i ) -f X ( C i +1 ) . We write the following, where the ± notation means we add and subtract a term:

Where the first inequality is due to inequality 4 in Lemma B.12 ( f X ( C i ) -f B i ( C i ) ≥ -ϵ/ 7 ), the second is due to the stopping condition of the algorithm ( f B i ( C i ) -f B i ( C i +1 ) > ϵ ), and the last is due to the remaining inequalities in Lemma B.12. The above holds w.h.p over all of the iterations of the algorithms. Using these guarantees for Algorithm 1 we can easily derive our main result for the truncated version.

Truncated termination Using Lemma 4.1 together with Lemma B.10 and the fact that f X ( C i ) -f X ( C i +1 ) ≥ 3 ϵ/ 7 we get that: f X ( ̂ C i ) -f X ( ̂ C i +1 ) ≥ f X ( C i ) -f X ( C i +1 ) -2 ϵ/ 7 ≥ ϵ/ 7 . We conclude that when b = Ω(max { γ 4 , γ 2 } kϵ -2 log 2 ( γn/ϵ )) , w.h.p. Algorithm 2 terminates within t = O ( γ 2 /ϵ ) iterations. This completes the second claim of Theorem 1.1. The final claim of Theorem 1.1 is due to the following lemma.

Lemma B.13. The expected approximation ratio of the solution returned by Algorithm 2 is at least the approximation guarantee of the initial centers provided to the algorithm.

Proof. Let p = 1 -O ( ϵ/nγ 2 ) = 1 -O (1 /n ) be the success probability of a single iteration. By 'success" we mean that all inequalities in Lemma B.12 hold. The value of p is due to the fact that we take t = O ( γ 2 /ϵ ) and that γ 2 /ϵ ≥ 1 / 4 .

With probability at least p , it holds that f X ( C i +1 ) ≤ f X ( C i ) -2 ϵ/ 7 . On the other hand, f X is upper bounded by 4 γ 2 . Let us denote Z = f X ( C i ) -f X ( C i +1 ) the change in the goal function after the i -th iteration. Consider the following:

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

We show that E [ Z ] = E [ f X ( C i ) -f X ( C i +1 )] ≥ 0 which implies that E [ f X ( C i +1 )] ≤ E [ f X ( C i )] 503 and completes the proof. Note that if E [ Z | Z < ϵ/ 7] > 0 then we are done as we simply have a 504 linear combination of two positive terms which is greater than 0. Let us focus on the case where 505 E [ Z | Z < ϵ/ 7] < 0 . 506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

Where the first inequality is due to the definition of p and the fact that E [ Z | Z < ϵ/ 7] < 0 , the second is due to the upper bound on f X , and the last inequality is by assuming n is sufficiently large.

C Full experimental results

We list our full experimental results in this section. We use the β prefix to denote that the algorithm uses the learning rate of [29]. τ denotes the maximum number of data points used to represent each truncated cluster center. We investigate 3 kernel functions: 1) The Gaussian kernel, as presented in Section 5, 2) The k-nearest-neighbor (k-nn) kernel, where the kernel matrix is D -1 AD -1 , A is a k-nn adjacency matrix of the data and D is the corresponding degree matrix, and 3) the heat kernel [10] where the kernel matrix is exp( -tD -1 / 2 AD -1 / 2 ) for some 0 < t < ∞ , A is a k-nn adjacency matrix and D is the corresponding degree matrix. All parameter settings can be found in the supplementary material.

Unlike for the Gaussian kernel where γ = 1 ; We observe empirically that for both the k-nn and heat kernels, γ ≪ 1 . In this case, the dependence on max { γ 4 , γ 2 } in the batch size required for Theorem 1.1 actually helps us. We found the parameters for these kernels to be easier to tune in practise than the Gaussian kernel parameter σ . For each kernel, we recorded the empirical value of gamma as follows:

Dataset:mnist_784

Batch size:256

Batch size:512

0.6

103

103

103

Time (s) log scale

0.4

102

102

102

Score

T: 50

101

101

101

0.2

100

100

100

/200

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

[10°

103

103

103

TY

Time (s) log scale

0.4

102

102

102

101

101

101

0.2

100

100

100

/200

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

101

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

103

103

103

103

Time (s) log scale

0.4

102

102

102

Score

T: 200

101

101

101

0.2

100

100

100

100

0.01

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

101

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

[10°

103

103

103

Time (s)log scale

0.4

102

102

102

Score

000:1

101

101

101

0.2

100

100

100

0.0

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

ARINMIACC

time

10-1

ARI

NMIACCtime

10-1

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

ZI

Truncatedβ-MiniBatchKernel

MiniBatch

Kernel sketching (Nystrom)

Dataset:mnist_784_knn

Batch size:256

Batch size:512

0.6

103

103

103

103

Time (s) log scale

Score

0.4

102

102

102

T: 50

101

101

101

0.2

100

100

100

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

101

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

103

103

103

103

Time (s)log scale

0.4

102

102

102

101

101

101

0.2

100

100

100

0.0

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

103

103

103

103

Time (s) log scale

Score

0.4

102

102

102

T:200

101

101

101

0.2

100

100

100

100

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

103

103

103

Time (s) log scale

Score

0.4

102

102

102

T:300

101

101

101

0.2

100

100

100

0.0

10-1

10-1

10-1

10-1

ARI

NMI

ACC

time

ARI

NMI

ACC

time

ARI

NMI

ACC

time

ARI

NMI

ACC

time

X

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

Dataset:mnist_784_heat

Batch size:256

Batch size:512

0.6

102

102

102

Time (s) log scale

Score

0.4

101

101

101

T: 50

0.2

100

100

100

100

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

102

102

102

102

Time (s) log scale

[10

0.4

101

101

101

0.2

100

100

100

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

T

102

102

102

102

Time (s) log scale

Score

0.4

101

101

101

102

T:200

0.2

100

100

100

100

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

102

102

102

102

Time (s) log scale

Score

0.4

101

101

101

102

000:1

0.2

100

100

100

/100

0.0

10-1

10-1

10-1

10-1

ARI

NMIACC

time

AR1

NMI

ACC

time

ARI

NMIACC

time

ARI

NMI

ACC

time

X

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

Dataset: har

Batch size:256

Batch size:512

0.6

101

101

101

101

Time (s) log scale

T: 50

Score

0.4

100

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

101

101

101

TT

Time (s) log scale

0.4

100

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

1

Time (s) log scale

101

10

101

0.4

T:200

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

Time (s) log scale

101

101

101

/20

000:1

Score

0.4

100

100

100

0.2

0.0

ARI

NMIACC

time

10-1

ARI

NMI

ACCtime

10-1

ARINMIACCtime

10-1

ARI

NMIACCtime

10-1

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

Dataset: har_knn

Batch size:256

Batch size:512

0.6

Time (s) log scale

101

101

101

101

T: 50

100

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

Time (s) log scale

101

101

101

T:100

100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

101

101

101

10

Time (s) log scale

T: 200

100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

Time (s) log scale

101

101

101

T:300

100

100

100

100

0.2

0.0

10-1

10-1

10-1

10-1

ARI

NMIACC

time

ARI

NMI

ACC

time

ARI

NMIACC

time

ARI

NMI

ACC

time

X

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

Truncatedβ-MiniBatchKernel

MiniBatchKernel

MiniBatch

Kernel sketching (Nystrom)

Dataset:har_heat

Batch size:256

Batch size:512

0.6

101

101

101

101

Time (s) log scale

Score

T: 50

0.4

100

100

100

100

0.2

0.0

AR1

NMI

ACC

time

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

101

101

101

10

Time (s) log scale

100

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

101

101

101

101

Time (s) log scale

T:200

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.6

101

101

101

Time (s) log scale

T:300

100

100

100

0.2

0.0

10-1

10-1

10-1

10-1

ARI

NMI

ACC

time

ARI

NMI

ACC

time

ARI

NMI

ACC

time

ARI

NMI

ACC

time

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

Truncatedβ-MiniBatchKernel

MiniBatch

Kernel sketching (Nystrom)

Dataset:letter

Batch size:256

Batch size:512

0.4

102

102

102

102

Time (s) log scale

0.3

T: 50

Score

0.2

101

101

101

[100

0.1

100

100

100

0.0

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

Score

101

101

101

T:100

0.2

0.1

100

100

100

100

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

102

Time (s) log scale

0.3

T:200

Score

0.2

101

101

101

[10

0.1

100

100

100

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

101

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

102

Time (s) log scale

0.3

Score

101

101

101

000:1

0.2

0.1

100

100

100

100

0.0

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

ARINMIACC

time

10-1

ARI

NMIACC

time

10-1

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

ZI

Truncatedβ-MiniBatchKernel

MiniBatch

Kernel sketching (Nystrom)

Dataset:letter_knn

Batch size:256

Batch size:512

0.4

102

102

102

102

Time (s) log scale

0.3

T: 50

Score

0.2

101

101

101

100

0.1

100

100

100

0.0

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

0.2

101

101

101

100

0.1

100

100

100

0.0

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

core

101

101

101

T:200

0.2

0.1

100

100

100

0.0

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

101

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

102

Time (s) log scale

0.3

T: 300

core

0.2

101

101

101

100

0.1

100

100

100

0.0

10-1

10-1

10-1

10-1

ARI

NMIACC

time

ARI

NMI

ACC

time

ARI

NMIACC

time

ARI

NMI

ACC

time

X

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

Dataset:letter_heat

Batch size:256

Batch size:512

0.4

102

102

102

Time (s) log scale

0.3

Score

101

101

101

T: 50

0.2

0.1

100

100

100

100

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

101

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

Score

101

101

101

T:100

0.2

0.1

100

100

100

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

Score

101

101

101

T:200

0.2

[20

0.1

100

100

100

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.4

102

102

102

Time (s) log scale

0.3

Score

101

101

101

T:300

0.2

10

0.1

100

100

100

0.0

10-1

10-1

10-1

10-1

ARI

NMIACC

time

AR1

NMI

ACC

time

AR1

NMIACC

time

ARI

NMI

ACC

time

X

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

Dataset: pendigits

Batch size:256

Batch size:512

0.8

Time (s) log scale

0.6

101

101

101

101

T: 50

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

Time (s) log scale

0.6

101

101

101

101

0.4

[100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

[10

Time (s) log scale

0.6

101

101

101

core

T:200

0.4

100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

Time (s) log scale

0.6

101

101

101

100

100

100

100

0.2

0.0

ARI

NMIACC

time

10-1

ARI

NMIACCt

time

10-1

ARINMIACCtime

10-1

ARI

NMIACCtime

10-1

Kernel

β-MiniBatchKernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

Truncated β-MiniBatch Kernel

MiniBatch

Kernel sketching (Nystrom)

Dataset: pendigits_knn

Batch size:256

Batch size:512

0.8

Time (s) log scale

0.6

101

101

101

101

Score

T: 50

0.4

100

100

100

/100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

T

0.6

101

101

101

[102

Time (s) log scale

0.4

[10°

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

0.6

101

101

101

[10

Time (s) log scale

Score

T:200

0.4

100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

T

Time (s) log scale

0.6

101

101

101

10

Score

000:1

0.4

100

100

100

100

0.2

0.0

10-1

10-1

10-1

10-1

ARI

NMIACC

time

ARI

NMI

ACCt

time

ARINMIACCt

time

ARI

NMIACCtime

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

Truncated β-MiniBatch Kernel

MiniBatch

Kernel sketching (Nystrom)

Dataset:pendigits_heat

Batch size:256

Batch size:512

0.8

Time (s) log scale

0.6

101

101

101

Score

T: 50

0.4

10

100

100

100

0.2

0.0

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

0.6

TT

101

101

101

101

Time (s) log scale

0.4

1

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

AR1

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

TT

Time (s) log scale

0.6

101

101

101

101

T:200

0.4

100

100

100

100

0.2

0.0

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

ARI

NMI

ACC

time

10-1

0.8

0.6

101

101

101

10

Time (s) log scale

Score

000:1

0.4

100

100

100

/100

0.2

0.0

ARI

NMIACC

time

10-1

ARI

NMI

ACC

time

10-1

ARINMIACCt

time

10-1

ARI

NMIACCtime

10-1

Kernel

β-MiniBatch Kernel

β-MiniBatch

Kernel sketching (ROS)

Kernel sketching (Sub-Gaussian)

MiniBatch

Kernel sketching (Nystrom)

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We prove our theoretical claims are true and show that the performance of our algorithm is competitive using experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: From our experimental results, each iteration of our mini-batch kernel k -means algorithm is still slower than that of mini-batch kmeans. This difference in speed may be alleviated by lazily enumerating the kernel matrix for very large datasets.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are complete and correct to the best of our knowledge.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed to replicate the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code, data, and instructions on how to reproduce our experimental results in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Full experimental details can be found within the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We explain how we report error bars using sample standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We give the runtime of each experiment and the type of machine they were performed on.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the code of ethics and believe that our research conforms to all the requirements.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. We believe the broader impact to be similar to that of mini-batch kmeans++.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets are properly credited and licenses and terms are mentioned and respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide documentation along with our code, which can automatically fetch and preprocess the data used in our experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the first mini-batch kernel k-means algorithm, addressing the computational bottleneck that has limited kernel k-means adoption in practice. The proposed method combines a recursive distance update rule with a truncation technique. The authors provide theoretical guarantees, showing that the algorithm converges in $\\mathcal{O}(\\gamma^2 / \\epsilon)$ iterations with high probability, where $\\gamma$ bounds the norm of points in feature space. Experimental results across multiple datasets demonstrate significant speedups over the full-batch kernel k-means while maintaining solution quality, and the method consistently outperforms kernel sketching approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis is the first mini-batch kernel k-means algorithm which addresses the computational bottleneck limiting kernel k-means adoption.\n2.\tThis method incorporates a recursive distance update rule with a truncation technique to avoid explicit center maintenance and achieve significant speedups while maintaining solution quality.\n3.\tAuthors provide convergence guarantees under early stopping conditions, and showing that the algorithm converges in $\\mathcal{O}(\\gamma^2 / \\epsilon)$ iterations with high probability.\n\nWeaknesses:\n\n1.\tYou don’t show the experimental results to demonstrate your method can converge quickly. \n2.\tCertain statements are not clear. For example, in line 253-255, does the truncated algorithm mean “Truncated beta-MiniBatch Kernel” and the non-truncated version mean the rest ones?\n3.\tSome typos and grammatical errors.\n\nQUESTIONS:\n1.\tSee weaknesses. \n2.\tIn line 94, you mentioned “Our algorithm copes better with non linearly separable data”. Can you provide some examples to prove it?\n3.\tIn Figure 3, can you clarify which is your method and the baselines?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper proposes a mini batch kernel k-means algorithm and it improves the running time by an order of magnitude compared to traditional batch algorithms. The experiment shows that the algorithm can consistently achieve a speed improvement of 10-100 times with minimal quality loss.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Provided theoretical analysis including termination conditions, iteration times, and expected approximation ratios, enhancing the credibility and predictability of the algorithm.
2. Through kernel methods, this algorithm can effectively handle non-linear separable data, expanding the application scope of k-means algorithm.
3. The new algorithm has an order of magnitude improvement in running time compared to traditional methods, making the kernel k-means algorithm more practical for processing large-scale datasets.

Weaknesses:
1. Besides the k-means++initialization method, are there any other better initialization strategies that can further improve the performance and convergence speed of the algorithm?
2. How to better integrate this algorithm with other data preprocessing or post-processing techniques (such as feature selection, dimensionality reduction, etc.) to further improve the efficiency and effectiveness of the entire clustering process?

QUESTIONS:
See Weaknesses for details.

LIMITATIONS:
The paper does not discuss its limitations or potential negative impacts. Including a brief discussion on these aspects is recommended.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the first minibatch-based acceleration method for kernel k-means. k-means is one of the most widely used unsupervised clustering methods, and various acceleration methods have been proposed for it. In recent years, theoretical results on the size of the mini-batch and the number of iterative updates have become known for empirical algorithms utilizing mini-batches that can be applied to large data sets. However, standard k-means only works ideally in limited scenarios, and kernel k-means are promising for more complex real data. This paper successfully elucidates empirically and theoretically that mini-batch speedup works for kernel k-means by using a recursive method to compute the distance between cluster centers and data represented by linear combinations.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n- This paper follows the recent theoretical analysis of the k-means algorithm using mini-batches [Schwartzman, ICLR2023], and is the first successful speed-up and theoretical analysis of the kernel k-means using mini-batches.\n\n- The paper theoretically demonstrates that kernel k-means not only qualitatively improves clustering with its kernel method (under natural scenarios), but also that mini-batch updates contribute significantly to speedup over standard (non-kernelized) k-means.\n\n[Weaknesse]\n\n- The technical advances in this paper are largely due to existing research, and the kernelization part of the advances appears to be a straightforward application of martingale theory of standard kernel methods.\n\nQUESTIONS:\nThis paper provides a solid improvement in the evolutionary flow of k-means and kernel k-means in terms of its speedup and theoretical transparency of the number of iterations required. We believe the scientific contribution of this paper, including its active outreach to open source, is of great value. On the other hand, I also have some concerns from two perspectives: the big picture and the local (technical) perspective.The first is that, in the big picture, this paper appears at first glance to be a straightforward kernel extension of the literature  [Schwartzman, ICLR2023]. From a very penetrating point of view, it seems that the part about k-means with mini-batches is a contribution to the literature  [Schwartzman, ICLR2023], and the kernel extension part is a straightforward application of established theory. In other words, the fusion of mini-batch k-means and kernelization does not seem to create any new technical challenges.Another concern is that the presentation of the theoretical analysis itself may have consciously or unconsciously introduced exhibits or assumptions that slightly favor the proposed method. This may be my misunderstanding, so I would appreciate it if you could confirm that my following questions are correct, just in case. Once again, I am grateful to the authors for sharing their excellent paper and for the opportunity to have this discussion.\n\n\n[Major concern 1: Independence of input data dimension]\n\nThe observations authors make in lines 93-97 are conceptually very impressive, but I am somewhat confused. My understanding is that in the proposed method, mini-batch k-means, once and for all, the computation of the inner product of pairs of data requires a computation time that depends on the dimension of the input data. However, I am convinced that there is indeed no dependence on the data dimension k, because for subsequent updates there is no need to perform this inner product calculation. Does the authors' argument implicitly assume that the computation of the prior Gram matrix is not a computational time bottleneck for subsequent iterative updates? Or is my understanding above incorrect, implying that the authors' method can omit even the computation of such a prior Gram matrix? I may have missed some important explanation and would be very grateful for clarification from the authors.\n\n\n[Major concern 2: Technical novelty]\n\nI would like to better understand the technical novelty/non-obviousness of this paper, so let me ask the following simple question. This paper certainly seems to be the first paper on speedup using mini-batch for kernel k-means (in fact, I have not been able to find a similar technique). However, the algorithms proposed by this paper (in particular Algorithm 1, not Algorithm 2 using truncation) also seem very straightforward. Put another way, it does not appear that Algorithm 1 overcomes any significant obstacles that have traditionally been unresolved over the years. So, at first glance, it would not seem surprising that a similar type of algorithm already existed empirically. Do the authors have any guesses as to why the same type of algorithm has not been utilized until now? For example, is there a reason that the implementation of Algorithm 1 with recursive computation is much more complicated than other algorithms for k-means and kernel k-means if one tries to implement it straightforwardly?\n\n\n\n[Minor comments]\n\n> (Lines 185-186) Next, we go a step further and show that it is possible to get an update time with only polylogarithmic dependence on n.\n\nI could not figure out why each iteration requires polylogarithm computation time of data size n in cluster-centered linear combination truncation. At first glance this does not seem to be consistent with the description in lines 226-227, but perhaps my understanding is incorrect?\n\n> (Line 95) ... requires a smaller batch size (Omega(1/epsilon^2) vs Omega((d/epsilon)^2))...\n\nIs the latter complexity Omega((d/epsilon)^2) correct? As stated in the 4-th footnote, authos point out that Omega((d/epsilon)^2) described in [Schwartzman, ICLR2023] contains mistakes and should be replaced with Omega(k(d/epsilon)^2). I would like authors to clarify the correct description, to be sure.\n\nLIMITATIONS:\nYes. Authors clearly discuss its potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents a new algorithm for mini-batched kernel k-means. The proposed algorithm offer an order of magnitude improvement in running time compared to the full batch algorithm. Theoretical analysis is provided to bound the running time and approximation ratio.

STRENGTHS AND WEAKNESSES:
The results of the paper are very promising. It has been tested on a wide range of datasets including MNIST, PenDigits, Letters, and HAR. The scores are maintained well, and significant speedup is observed.

Section 4 is pretty hard to parse. I wish the paper is written as an algorithm paper. With intuition and algorithms as the main focus. Currently, the paper’s claim is pretty theoretical. I do not have the background to check the correctness of the proof.

Overall, my feeling is that this paper is actually acceptable. I think with some revision, this paper can be a great one, since its result is very strong. My main problem about this paper is that the authors should clarify the main contribution. Currently, it is a combination of algorithm, theory, and implementation. I hope the paper can concentrate on one of them. I feel as an algorithm paper, I don't feel the paper gives reader enough intuition on the proposed. algorithm. As an implementation paper, I feel the evaluation section does not have enough information about the implementation and microbenchmarks (e.g., is any new CUDA kernel being developed, where is the current performance bottleneck).

Other feedback:
I am not sure why the paper needs figure 2. To me, a kernel-based method is better in accuracy than pure distance-based method is very clear.

QUESTIONS:
Can you provide more intuition about your algorithm?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02