Title: Discriminative Entropy Clustering and its Relation to K-means and SVM
Venue: NeurIPS 2023
Accepted: no
Keywords: discriminative clustering

Kmeans

entropy

SVM
Abstract: Maximization of mutual information between the model's input and output is formally related to "decisiveness" and "fairness'" of the softmax predictions, motivating such unsupervised entropy-based losses for discriminative models. Recent self-labeling methods based on such losses represent the state of the art in deep clustering. First, we discuss a number of general properties of such entropy clustering methods, including their relation to K-means and unsupervised SVM-based techniques. Disproving some earlier published claims, we point out fundamental differences with K-means. On the other hand, we show similarity with SVM-based clustering allowing us to link explicit margin maximization to entropy clustering. Finally, we observe that the common form of cross-entropy is not robust to 
pseudo-label errors. Our new loss addresses the problem and leads 
to a new EM algorithm improving the state of the art on many standard benchmarks.

Abstract

Maximization of mutual information between the model's input and output is formally related to 'decisiveness' and 'fairness' of the softmax predictions Bridle et al. (1991), motivating such unsupervised entropy-based losses for discriminative models. Recent self-labeling methods based on such losses represent the state of the art in deep clustering. First, we discuss a number of general properties of such entropy clustering methods, including their relation to K-means and unsupervised SVM-based techniques. Disproving some earlier published claims, we point out fundamental differences with K-means. On the other hand, we show similarity with SVM-based clustering allowing us to link explicit margin maximization to entropy clustering. Finally, we observe that the common form of cross-entropy is not robust to pseudo-label errors. Our new loss addresses the problem and leads to a new EM algorithm improving the state of the art on many standard benchmarks.

1 Introduction

Discriminative entropy-based loss functions, e.g. decisiveness and fairness , were proposed for 14 network training Bridle et al. (1991); Krause et al. (2010) and regularization Grandvalet & Bengio 15 (2004) and are commonly used for unsupervised and weakly-supervised classification problems 16 Ghasedi Dizaji et al. (2017); Hu et al. (2017); Ji et al. (2019); Asano et al. (2020); Jabi et al. (2021). 17 In particular, the state-of-the-art in unsupervised classification Asano et al. (2020); Jabi et al. (2021) 18 is achieved by self-labeling methods using extensions of decisiveness and fairness. 19

20

21

22

Section 1.1 reviews the entropy-based clustering with soft-max models and introduces the necessary notation. Then, Section 1.2 reviews the corresponding self-labeling formulations. Section 1.3 summarizes our main contributions and outlines the structure of the main parts of the paper.

23

24

25

26

27

28

29

30

31

32

33

34

1.1 Discriminative entropy clustering: background and notation

Consider neural networks using probability-type outputs, e.g. softmax σ : R K → ∆ K mapping K logits l k ∈ R to K -class probabilities σ k = exp l k ∑ c exp l c forming a categorical distribution σ = ( σ 1 , . . . , σ K ) ∈ ∆ K often interpreted as a posterior. We reserve superscripts to indicate classes or categories. For shortness, this paper uses the same symbol for functions or mappings and examples of their output, e.g. specific predictions σ . If necessary, subscript i can indicate values, e.g. prediction σ i or logit l k i , corresponding to any specific input example X i in the training dataset { X i } N i =1 .

The mutual information (MI) loss, proposed by Bridle et al. (1991) for unsupervised discriminative training of softmax models, trains the model output to keep as much information about the input as possible. They derived MI estimate as the difference between the average entropy of the output H ( σ ) = 1 N ∑ i H ( σ i ) and the entropy of the average output σ = 1 N ∑ i σ i , which is a distribution of class predictions over the whole dataset

where C is a random variable representing the class prediction for input X . Besides the motivating 35 information-theoretic interpretation of the loss, the right-hand side in (1) has a clear discriminative 36 interpretation that stands on its own: H ( σ ) encourages 'fair' predictions with a balanced support 37 of all categories across the whole training dataset, while H ( σ ) encourages confident or 'decisive' 38 prediction at each data point suggesting that decision boundaries are away from the training examples 39 Grandvalet & Bengio (2004). Our paper refers to unsupervised training of discriminative soft-max 40 models using predictions' entropies, e.g. see (1), as discriminative entropy clustering . This should 41 not be confused with generative entropy clustering methods where the entropy is used as a measure 42 of compactness for clusters' density functions 1 . 43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

Discriminative clustering loss (1) can be applied to deep or shallow models. For clarity, this paper distinguishes parameters w of the representation layers of the network computing features f w ( X ) ∈ R M for any input X . We separate the linear classifier parameters v in the output layer computing K -logit vector l = v ⊤ f for any feature f ∈ R M . As mentioned earlier, this paper uses the same notation for mapping f ( · ) and its values or (deep) features f produced by the representation layers. For shortness, we assume a 'homogeneous' representation of the linear classifier so that v ⊤ f includes the bias. The overall network model is defined as

A special 'shallow' case of the model in (2) is a basic linear discriminator

directly operating on given input features f ( X ) = X . In this case, M represents the input dimensions. Optimization of the loss (1) for the shallow model (3) is done only over linear classifier parameters v , but the deeper network model (2) is optimized over all network parameters [ v , w ] . Typically, this is done via gradient descent or backpropagation Rumelhart et al. (1986); Bridle et al. (1991).

In the context of deep models (2), the decision boundaries between the clusters of data points { X i } can be arbitrarily complex since the network learns high-dimensional non-linear representation map or embedding f w ( X ) . In this case, loss (1) is optimized with respect to both representation w and classification v parameters. To avoid overly complex clustering of the training data and to improve generality, it is common to use self-augmentation techniques Hu et al. (2017). For example, Ji et al. (2019) maximize the mutual information between class predictions for input X and its augmentation counterpart X ′ encouraging deep features invariant to augmentation.

To reduce the model's complexity, Krause et al. (2010) combine entropy-based loss (1) with regularization of all network parameters interpreted as their isotropic Gaussian prior

where c = represents equality up to an additive constant and u is a uniform distribution over K classes. The second loss formulation in (4) uses KL divergence motivated in Krause et al. (2010) by the possibility to generalize the fairness to any target balancing distribution different from the uniform.

1.2 Self-labeling methods for entropy clustering

Optimization of losses (1) or (4) during network training is mostly done with standard gradient descent 69 or backpropagation Bridle et al. (1991); Krause et al. (2010); Hu et al. (2017). However, the difference 70 between the two entropy terms implies non-convexity, which makes such losses challenging for 71 gradient descent. This motivates alternative formulations and optimization approaches. For example, 72 it is common to extend the loss by incorporating auxiliary or hidden variables y representing pseudo73 labels for unlabeled data points X , which are to be estimated jointly with optimization of the network 74 parameters Ghasedi Dizaji et al. (2017); Asano et al. (2020); Jabi et al. (2021). Typically, such 75 self-labeling approaches to unsupervised network training iterate optimization of the loss over pseudo76 labels and network parameters, similarly to Lloyd's algorithm for K -means or EM algorithm for 77 Gaussian mixtures Bishop (2006). While the network parameters are still optimized via gradient 78 descent, the pseudo-labels can be optimized via more powerful algorithms. 79

For example, Asano et al. (2020) formulate self-labeling using the following constrained optimization 80 problem with discrete pseudo-labels y tied to predictions by cross entropy function H ( y, σ ) 81

where ∆ K 0 , 1 are one-hot distributions, i.e. corners of the probability simplex ∆ K . Training of the network is done by minimizing cross entropy H ( y, σ ) , which is convex w.r.t. σ , assuming fixed pseudo-labels y . Then, model predictions get fixed and cross-entropy is minimized w.r.t variables y . Note that cross-entropy H ( y, σ ) is linear with respect to y , and its minimum over simplex ∆ K is achieved by one-hot distribution for a class label corresponding to arg max( σ ) at each training example. However, the balancing constraint ¯ y = u converts minimization of cross-entropy over all data points into a non-trivial integer programming problem that can be approximately solved via optimal transport Cuturi (2013). The cross-entropy in (5) encourages the network predictions σ to approximate the estimated one-hot target distributions y , which implies the decisiveness.

Self-labeling methods for unsupervised clustering can also use soft pseudo-labels y ∈ ∆ K as target distributions inside H ( y, σ ) . In general, soft targets y are commonly used with cross-entropy functions H ( y, σ ) , e.g. in the context of noisy labels Tanaka et al. (2018); Song et al. (2022). Softened targets y can also assist network calibration Guo et al. (2017); Müller et al. (2019) and improve generalization by reducing over-confidence Pereyra et al. (2017). In the context of unsupervised clustering, cross entropy H ( y, σ ) with soft pseudo-labels y approximates the decisiveness since it encourages σ ≈ y implying H ( y, σ ) ≈ H ( y ) ≈ H ( σ ) where the latter is the decisiveness term in (1). Inspired by (4), instead of the hard constraint ¯ y = u used in (5), self-labeling losses can represent the fairness using KL divergence KL (¯ y ∥ u ) , as in Ghasedi Dizaji et al. (2017); Jabi et al. (2021). In particular, Jabi et al. (2021) formulates the following entropy-based self-labeling loss

encouraging decisiveness and fairness, as discussed. Similarly to (5), the network parameters in loss (6) are trained by the standard cross-entropy term. Optimization over relaxed pseudo-labels y ∈ ∆ K is relatively easy since KL divergence is convex and cross-entropy is linear w.r.t. y . While there is no closed-form solution, the authors offer an efficient approximate solver for y . Iterating steps that estimate pseudo-labels y and optimize the model parameters resemble the Lloyd's algorithm for K-means. Jabi et al. (2021) also establish a formal relation with K-means objective.

1.3 Summary of our contributions

Our work is closely related to self-labeling loss (6) and the corresponding ADM algorithm proposed in Jabi et al. (2021). Their inspiring approach is a good reference point for our self-labeling loss formulation (13). It also helps to illuminate the limits in a general understanding of entropy clustering.

Our paper provides conceptual and algorithmic contributions. First of all, we examine the relations of discriminative entropy clustering to K-means and SVM. In particular, we disprove the main theoretical claim (in the title) of a recent TPAMI paper Jabi et al. (2021) wrongly stating the equivalence between the standard K-means objective and the entropy-based clustering losses. Our Figure 1 provides a simple counterexample to the claim, but we also show specific technical errors in their proof. We highlight fundamental differences with a broader generative group of clustering methods, which includes K-means, GMM, etc. On the other hand, we find stronger similarities between entropy clustering and discriminative SVM-based clustering. In particular, this helps to formally show the soft margin maximization effect when decisiveness is combined with a norm regularization term.

This paper also proposes a new self-labeling algorithm for entropy-based clustering. In the context of relaxed pseudo-labels y , we observe that the standard formulation of decisiveness H ( y, σ ) is sensitive to pseudo-label uncertainty/errors. We motivate the reverse cross-entropy formulation, which we demonstrate is significantly more robust to label noise. We also propose a zero-avoiding form of KL-divergence as a strong fairness term. Unlike standard fairness, it does not tolerate highly unbalanced clusters. Our new self-labeling loss allows an efficient EM algorithm for estimating pseudo-labels. We derive closed-form E and M steps. Our new algorithm improves the state-of-the-art on many standard benchmarks for deep clustering, which empirically validates our technical insights.

Our paper is organized as follows. Section 2 discusses the relation of entropy clustering to K-means and SVM. Section 3 motivates our self-labeling loss and derives an EM algorithm for estimating pseudo-labels. The experimental results for our entropy clustering algorithm are in Section 4.

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

TWO LINEAR DECISION FUNCTIONS OVER 2D FEATURES

X

∈ R

2

k

µ

(

X

) = arg min

k

∥

X

-

µ

k

∥

σ

v

(

X

) =

soft-max

(

v

⊤

X

)

(a) variance clustering

(b) entropy clustering

0v(X)

kμ(X) = 0

22

onehot 1

0v(X)

22

kμ(X) = 1

onehot o

compactnessofclusters

decisiveness&fairnessofpredictions

2 Relation to discriminative and generative clustering methods

2.1 Entropy-based clustering versus K-means

Discriminative entropy clustering (1) is not as widely known as K-means, but for no good reason. With linear models (3), entropy clustering (1) is as simple as K-means, e.g. it produces linear cluster boundaries. Both approaches have good approximate optimization algorithms for their non-convex (1) or NP-hard Mahajan et al. (2012) objectives. Two methods also generalize to non-linear clustering using more complex representations, e.g. learned f w ( X ) or implicit (kernel K-means).

There is a limited general understanding of how entropy clustering relates to more popular methods, such as K-means. The prior work, including Bridle et al. (1991), mainly discusses entropy clustering in the context of neural networks. K-means is also commonly used with deep features, but it is hard to understand the differences in such complex settings. An illustrative 2D example of entropy clustering in Krause et al. (2010) (Fig.1) is helpful, but it looks like a typical textbook example for K-means where it would work perfectly. Interestingly, Jabi et al. (2021) make a theoretical claim about algebraic equivalence between K-means objective and a regularized entropy clustering loss.

Here we show significant differences between K-means and entropy clustering. First, we disprove the claim by Jabi et al. (2021). We provide a simple counterexample in Figure 1 where the optimal solutions are different in a basic linear setting. Moreover, we point out a critical technical error in their Proposition 2 - its proof ignores normalization inside softmax. Symbol ∝ hides it in their equation (5), which is later treated as equality in the proof of Proposition 2. Equations in their proof do not work with normalization, which is critical for softmax models. The extra regularization term ∥ v ∥ 2 in their entropy loss is also important. Without softmax normalization, ln σ inside cross-entropy H ( y, σ ) turns into a linear term w.r.t. logits v ⊤ x and adding ∥ v ∥ 2 creates a quadratic form resembling squared errors ( x -v ) 2 in K-means. In contrast, Section 2.2 shows that regularization ∥ v ∥ 2 corresponds to the margin maximization controlling the width of the soft gap between the clusters, see our Fig.1(b).

In general, Figure 1 highlights fundamental differences between generative and discriminative approaches to clustering using two basic linear methods of similar parametric complexity (about

K

×

M

parameters).

K

-means (a) seeks balanced compact clusters of the least variance (squared errors). This can be interpreted 'generatively' Kearns et al. (1997) as MLE fitting of two (isotropic)

Gaussian densities, which also explains why K-means fails on highly anisotropic clusters (a). To fix this 'generatively', one should use non-isotropic Gaussian densities. In particular, 2-mode GMM

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

would produce soft clusters as in (b). But, this increases parametric complexity (two extra covariance matrices) and leads to quadratic decision boundaries. In contrast, discriminative entropy clustering in (b) simply looks for the best linear decision boundary giving balanced ('fair') clusters with data points away from the boundary ('decisiveness'), regardless of the data density model complexity.

2.2 Entropy-based clustering and SVM: margin maximization

This section discusses similarities between entropy clustering with soft-max models and unsupervised SVM methods Ben-Hur et al. (2001); Xu et al. (2004). First, consider the fully supervised setting, where the relationship between SVMs Vapnik (1995) and logistic regression is known. Assuming binary classification with target labels t = ± 1 , one standard soft-margin SVM loss formulation combines a margin-maximization term with the hinge loss penalizing margin violations, e.g. see Bishop (2006)

where the linear classifier norm ∥ v ∥ (excluding bias!) is the reciprocal of the decision margin and γ is the relative weight of the margin maximization term. For shortness and consistently with the notation introduced in Sec.1.1, logits v ⊤ f include the bias using 'homogeneous' representations of v and features f , and the 'bar' operator represents averaging over all training data points.

Instead of the hinge loss, soft-margin maximization (7) can use the logistic regression as an alternative soft penalty for margin violations, see Section 7.1.2 and Figure 7.5 in Bishop (2006),

where the second binary cross-entropy formulation in (8) replaces integer targets t ∈ {± 1 } with one-hot target distributions y ∈ { (1 , 0) , (0 , 1) } consistent with our general terminology in Sec.1.2. Our second formulation in (8) uses soft-max σ = { exp l 1 exp l 1 +exp l 2 , exp l 2 exp l 1 +exp l 2 } with logits l 1 = 1 2 v ⊤ f and l 2 = -1 2 v ⊤ f ; its one advantage is a trivial multi-class generalization. The difference between the soft-margin maximization losses (7) and (8) is that the flat region of the hinge loss leads to a sparse set of support vectors for the maximum margin solution, see Section 7.1.2 in Bishop (2006).

Now, consider the standard SVM-based self-labeling formulation of maximum margin clustering by Xu et al. (2004). They combine loss (7) with a linear fairness constraint -ϵ ≤ ¯ t ≤ ϵ

and treat labels t as optimization variables in addition to model parameters. Note that the hinge loss encourages consistency between the pseudo labels t ∈ {± 1 } and the sign of the logits v ⊤ f . Besides, loss (9) still encourages maximum margin between the clusters. Keeping data points away from the decision boundary is similar to the motivation for the decisiveness in entropy-based clustering.

It is easy to connect (9) to self-labeling entropy clustering. Similarly to (7) and (8), one can replace the hinge loss by cross-entropy as an alternative margin-violation penalty. As before, the main difference is that the margin may not be defined by a sparse subset of support vectors . We can also replace the linear balancing constraint in (9) by an entropy-based fairness term. Then, we get

which is a self-labeling surrogate for the entropy-based maximum-margin clustering loss

Losses (11) and (10) are examples of general clustering losses for K ≥ 2 combining decisiveness and fairness as in Sections 1.1, 1.2. The first term can be seen as a special case of the norm regularization in (4). However, instead of a generic model simplicity argument used to justify (4), the specific combination of cross-entropy with regularizer ∥ v ∥ 2 (excluding bias) in (11) and (10) is explicitly linked to margin maximization where 1 ∥ v ∥ corresponds to the margin's width 2 .

It was known that 'for a poorly regularized classifier' the combination of decisiveness and fairness 'alone will not necessarily lead to good solutions to unsupervised classification' (Bridle et al. (1991)) and that decision boundary can tightly pass between the data points (Fig.1 in Krause et al. (2010)). The formal relation to margin maximization above complements such prior knowledge. Our supplementary material (A) shows the empirical effect of parameter γ in (11) on the inter-cluster gaps.

205

206

207

208

209

210

211

212

213

214

215

216

217

218

(a)

strong

fairness

KL

(

u

∥

¯

σ

)

(b)

reverse

cross-entropy

H

(

σ, y

)

3 Our self-labeling entropy clustering method

The conceptual properties discussed in the previous section may improve the general understanding of entropy clustering, but their new practical benefits are limited. For example, margin maximization implicitly happens in prior entropy methods since norm regularization (weight-decay) is omnipresent.

This section addresses some specific limitations of prior entropy clustering formulations that do affect the practical performance. We focus on self-labeling (Sec.1.2) and observe that the standard cross-entropy formulation of decisiveness is sensitive to pseudo-label errors. Section 3.1 introduces our new self-labeling loss using the reverse cross-entropy , which we show is more robust to label noise. We also propose strong fairness . Section 3.2 derives an efficient EM algorithm for minimizing our loss w.r.t. pseudo-labels, which is a critical step of our self-labeling algorithm.

3.1 Our self-labeling loss formulation

Westart from the maximum-margin entropy clustering (10) where the entropy fairness can be replaced by an equivalent KL-divergence term explicitly expressing the target balance distribution u . This gives a self-labeling variant of the loss (4) in Krause et al. (2010) similar to (6) in Jabi et al. (2021)

We propose two changes to this loss based on several numerical insights leading to a significant 219 performance improvement over Krause et al. (2010) and Jabi et al. (2021). First, we reverse the order 220 of the cross-entropy arguments, see Fig.2(b). This improves the robustness of network predictions σ 221 to errors in estimated pseudo-labels y , as confirmed by our experiment in Figure 3. This reversal also 222 works for estimating pseudo-labels y as the second argument in cross-entropy is a standard position 223 for an 'estimated' distribution. Second, we also observe that the standard fairness term in (12,4,6) 224 is the reverse KL divergence w.r.t. cluster volumes, i.e. the average predictions ¯ σ . It can tolerate 225 highly unbalanced solutions where ¯ σ k = 0 for some cluster k , see the dashed curves in Fig.2(a). We 226 propose the forward , a.k.a. zero-avoiding , KL divergence KL ( u ∥ σ ) , see the solid curves Fig.2(a), 227 which assigns infinite penalties to highly unbalanced clusters. We refer to this as strong fairness . 228

The two changes above modify the clustering loss (12) into our formulation of self-labeling loss 229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

0.0

0.2

0.4

0.6

0.8

corruption level

25%

35%

45%

55%

65%

75%

85%

95%

accuracy

forward CE:

H

(

y

,

)

forward CE:

H

(

y

,

)

reverse CE:

H

(

,

y

)

3.2 Our EM algorithm for pseudo-labels

Minimization of a self-supervised loss w.r.t pseudolabels y for given predictions σ is a critical operation in iterative self-labeling techniques Asano et al. (2020); Jabi et al. (2021), see Sec.1.2. Besides wellmotivated numerical properties of our new loss (13), in practice it also matters that it has an efficient solver for pseudo-labels. While (13) is convex w.r.t. y , optimization is done over a probability simplex and a good practical solver is not a given. Note that H ( σ, y ) works as a log barrier for the constraint y ∈ ∆ K . This could be problematic for the first-order methods, but a basic Newton's method is a good match, e.g. Kelley (1995). The overall convergence rate of such second-order methods is fast, but computing the Hessian's inverse is costly, see Table 1. Instead, we derive a more efficient expectation-maximization (EM) algorithm.

Assume that model parameters and predictions in (13) are fixed, i.e. v and σ . Following variational inference Bishop (2006), we introduce K auxiliary latent variables, distributions S k ∈ ∆ N representing normalized support of each cluster k over N data points. In contrast, N distributions y i ∈ ∆ K show support for each class at every point X i . We

refer to each vector S k as a normalized cluster k . Note that here we focus on individual data points and explicitly index them by i ∈ { 1 , . . . , N } . Thus, we use y i ∈ ∆ K and σ i ∈ ∆ K . Individual components of distribution S k ∈ ∆ N corresponding to data point X i is denoted by scalar S k i .

First, we expand our loss (13) using our new latent variables S k ∈ ∆ N

Due to the convexity of negative log , we apply Jensen's inequality to derive an upper bound, i.e. (15), 259 to L our . Such a bound becomes tight when: 260

Then, we fix S k i as (16) and solve the Lagrangian of (15) with simplex constraint to update y as:

We run these two steps until convergence with respect to some predefined tolerance. Note that the minimum y is guaranteed to be globally optimal since (14) is convex w.r.t. y

The empirical convergence rate is within 15 steps on MNIST. The comparison of computation speed on synthetic data is shown in Table 1. While the number of iterations to convergence is roughly the same as Newton's methods, our EM algorithm is much faster in terms of running time and is extremely easy to implement using the highly optimized built-in functions from the standard PyTorch library that supports GPU.

.

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

Inspired by Springenberg (2015); Hu et al. (2017), we also adapted our EM algorithm to allow for updating y within each batch. In fact, the mini-batch approximation of (14) is an upper bound. Considering the first two terms of (14), we can use Jensen's inequality to get:

where B is the batch randomly sampled from the whole dataset. Now, we can apply our EM algorithm to update y in each batch, which is even more efficient. Compared to other methods Ghasedi Dizaji et al. (2017); Asano et al. (2020); Jabi et al. (2021) which also use the auxiliary variable y , we can efficiently update y on the fly while they only update once or just a few times per epoch due to the inefficiency to update y for the whole dataset per iteration. Interestingly, we found that it is actually important to update y on the fly, which makes convergence faster and improves the performance significantly (see supplementary material). We use this 'batch version' EM throughout all the experiments. Our full algorithm for the loss (13) is summarized in supplementary material.

4 Experimental results

Our experiments start from clustering on fixed features to joint training with feature learning. We test our approach on standard benchmark datasets with different network architectures. We also provide the comparison of different losses under weakly-supervised settings (see supplementary material).

Dataset For the clustering problem, we use four standard benchmark datasets: MNIST Lecun et al. (1998), CIFAR10/100 Torralba et al. (2008) and STL10 Coates et al. (2011). We follow Ji et al. (2019) to use the whole dataset for training and testing unless otherwise specified.

Evaluation As for the evaluation on clustering, we set the number of clusters to the number of ground-truth category labels and adopt the standard method Kuhn (1955) by finding the best one-to-one mapping between clusters and labels.

4.1 Clustering with fixed features

We compare our method against the state-of-the-art methods using fixed deep features generated by pre-trained (ImageNet) ResNet-50 He et al. (2016). We use a one-layer linear classifier for all losses except for K-means. We set λ in our loss to 100. We use stochastic gradient descent with learning rate 0.1 to optimize the loss for 10 epochs. The batch size was set to 250. The coefficients for the margin maximization terms are set to 0.001, 0.02, 0.009, and 0.02 for MNIST, CIFAR10, CIFAR100 and STL10 respectively. As stated in Section 2.2, such coefficient is important for the optimal decision boundary, especially when features are fixed. If we simultaneously learn the representation/feature and cluster the data, we observed that the results are less sensitive to such coefficient.

4.2 Joint clustering and feature learning

In this section, we train a deep network to jointly learn the features and cluster the data. We test our method on both a small architecture (VGG4) and a large one (ResNet-18). The only extra standard technique we add here is the self-augmentation, following Hu et al. (2017); Ji et al. (2019); Asano et al. (2020). The experimental settings and more details are given in the supplementary material.

To train the VGG4, we use random initialization for network parameters. From Table 3, it can be seen that our approach consistently achieves the most competitive results in terms of accuracy

(ACC). Most of the methods we compared in our work (including our method) are general concepts 309 applicable to single-stage end-to-end training. To be fair, we tested all of them on the same simple 310 architecture. But, these general methods can be easily integrated into other more complex systems. 311

As for the training of ResNet-18, we found that random initialization does not work well when we only use self-augmentation. We may need more training tricks such as auxiliary over-clustering, multiple heads, and more augmentations Ji et al. (2019). In the mean time, the authors from Van Gansbeke et al. (2020) proposed a three-stage approach for the unsupervised classification and we found that the pre-trained weight from their first stage is beneficial to us. For a fair comparison, we followed their experimental settings and compared ours to their second-stage results. Note that they split the data into training and testing. We also report two additional evaluation metrics, i.e. NMI and ARI.

In Table 4, we show the results using their pretext-trained network (stage one) as initialization for our entropy clustering. We use only our clustering loss together with the self-augmentation (one augmentation per image this time) to reach higher numbers than SCAN, as shown in the table below.

5 Conclusions

Our paper proposed a new self-labeling algorithm for discriminative entropy clustering, but we also clarify several important conceptual properties of this general methodology. For example, we disproved a theoretical claim in a recent TPAMI paper stating the equivalence between variance clustering (K-means) and discriminative entropy-based clustering. We also demonstrate that standard formulations of entropy clustering losses may lead to narrow decision margins. Unlike prior work on discriminative entropy clustering, we show that classifier norm regularization is important for margin maximization.

We also discussed several limitations of the existing self-labeling formulations of entropy clustering and propose a new loss addressing such limitations. In particular, we replace the standard (forward) cross-entropy by the reverse cross-entropy that we show is significantly more robust to errors in estimated soft pseudo-labels. Our loss also uses a strong formulation of the fairness constraint motivated by a zero-avoiding version of KL divergence. Moreover, we designed an efficient EM algorithm minimizing our loss w.r.t. pseudo-labels; it is significantly faster than standard alternatives, e.g Newton's method. Our empirical results improved the state-of-the-art on many standard benchmarks for deep clustering.

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

References

Asano, Y. M., Rupprecht, C., and Vedaldi, A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations , 2020.

Ben-Hur, A., Horn, D., Siegelman, H., and Vapnik, V. Support vector clustering. Journal of Machine Learning Research , 2:125 - 137, 2001.

Bishop, C. M. Pattern Recognition and Machine Learning . Springer, 2006.

Bridle, J. S., Heading, A. J. R., and MacKay, D. J. C. Unsupervised classifiers, mutual information and 'phantom targets'. In NIPS , pp. 1096-1101, 1991.

Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.

Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Ghasedi Dizaji, K., Herandi, A., Deng, C., Cai, W., and Huang, H. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision , pp. 5736-5745, 2017.

Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. Advances in neural information processing systems , 17, 2004.

Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning , pp. 1321-1330. PMLR, 2017.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.

Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning , pp. 1558-1567. PMLR, 2017.

Jabi, M., Pedersoli, M., Mitiche, A., and Ayed, I. B. Deep clustering: On the link between discriminative models and k-means. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43 (6):1887-1896, 2021.

Ji, X., Henriques, J. F., and Vedaldi, A. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 9865-9874, 2019.

Kearns, M., Mansour, Y., and Ng, A. Y. An information-theoretic analysis of hard and soft assignment methods for clustering. In UAI '97: Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, Brown University, Providence, Rhode Island, USA, August 1-3, 1997 , pp. 282-293. Morgan Kaufmann, 1997.

Kelley, C. T. Iterative methods for linear and nonlinear equations . SIAM, 1995.

Krause, A., Perona, P., and Gomes, R. Discriminative clustering by regularized information maximization. Advances in neural information processing systems , 23, 2010.

Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Mahajan, M., Nimbhorkar, P., and Varadarajan, K. The planar K-means problem is NP-hard. Theoretical Computer Science , 442:13-21, 2012.

Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

NSD. Natural Scenes Dataset [NSD]. https://www.kaggle.com/datasets/ nitishabharathi/scene-classification , 2020.

Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. ICLR workshop, arXiv:1701.06548 , 2017.

Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning representations by back-propagating errors. Nature , 323(6088):533-536, 1986.

Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.-G. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Springenberg, J. T. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In International Conference on Learning Representations , 2015.

Tanaka, D., Ikami, D., Yamasaki, T., and Aizawa, K. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5552-5560, 2018.

Torralba, A., Fergus, R., and Freeman, W. T. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence , 30 (11):1958-1970, 2008.

Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., and Van Gool, L. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X , pp. 268-285. Springer, 2020.

Vapnik, V. The Nature of Statistical Learning Theory . Springer, 1995.

Xu, L., Neufeld, J., Larson, B., and Schuurmans, D. Maximum margin clustering. In Saul, L., Weiss, Y., and Bottou, L. (eds.), Advances in Neural Information Processing Systems , volume 17. MIT Press, 2004.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors first presented an analysis of the relationship between the regularized information maximization (RIM) clustering framework to K-means and SVM-based clustering methods, showing stronger similarities to the SVM-based clustering than K-means. Then they proposed a new loss function and associated EM optimization algorithm for clustering leveraging the reverse cross-entropy/KL divergence to obtain more robust and fair clustering, which has been demonstrated to improve the performance on several balanced image classification benchmarks.\n\nSTRENGTHS:\n1. The authors identified an error about the missing normalization term in the proof of the equivalence theorem presented in Jabi et al. (2021).\n\n2. The replacement of the forward cross-entropy with the reverse counterpart in the RIM loss appears novel in the clustering scenarios and has the potential to effectively mitigate the impact of uncertain/noisy pseudo-labels.\n\n3. The proposed method showed improved performance on different image classification benchmarks over several baselines.\n\nWEAKNESSES:\n1. The manuscript was poorly written. The authors dedicated more than two pages to discuss the general background of the information maximization clustering framework and related methods. However, these discussions were confusing and largely limited the space for presenting the actual contributions of this work. Furthermore, many terms, including concepts like H and KL divergence, are not explicitly defined or explained, which may cause difficulties to understand the differences between the forward and reverse formulations. Additionally, the claimed conceptual and algorithmic contributions seem to be independent of each other. It is unclear if any of these conceptual clarifications contribute the discovery of the new loss function.\n\n2. The disproof of the equivalence theorem in Jabi et al. (2021) is not convincing. While the authors pointed out an error in the original proof, this does not necessarily eliminate the possibility that the equivalence itself remains valid. Furthermore, this work focused on the standard K-means objective (including Figure 1), whereas Jabi et al. (2021) considered a soft and regularized K-means loss.\n\n3. The authors' claim that the L2 regularization is linked to margin maximization seems questionable. [1] demonstrated that margin maximization is a property of the loss function itself rather than the regularization, which serves to control the model complexity. Indeed, certain combinations of loss function and regularization are not margin-maximizing.\n\n4. The experimental validation is limited. A more comprehensive evaluation of the proposed modifications to the loss function would involve investigating the individual and combined effects of these changes on selected benchmarks and then comparing the results with multiple established baselines. It is still unclear how each modification contributes to the final improvement. Although the authors presented the impact of the reverse cross-entropy modification, they did so within the fully supervised setting rather than unsupervised scenarios. Furthermore, the authors only considered balanced datasets and tested the clustering with the ground truth number of clusters. A more diverse set of experimental conditions, including unbalanced datasets and varying numbers of clusters, would provide a more robust evaluation of the proposed method. Both NMI and ARI metrics used in Table 4 are capable of handling different number of clusters. Additionally, the architecture used in Section 4.2 is different from that used in the baseline methods. It would be preferable to standardize the experimental settings, including model architecture, to be able to directly compare with the results in the literature. Lastly, the inability of the proposed method to properly train a deep network-based clustering model is a concern as well. Most of the tricks should be independent of the loss function modifications, especially the reverse cross-entropy term, thus can be naturally integrated together.\n\n[1] Rosset et al., 2003. Margin maximizing loss functions.\n\nQUESTIONS:\n1. Could the tolerance of \"highly unbalanced solutions where $\\bar{\\sigma}_k = 0$ for some cluster k\" be a good thing, especially when we over-cluster in practice?\n\n2. How were the coefficients of the norm regularization and KL divergence determined in the proposed method and relevant baselines? Were they shared across all the methods? How could we select these coefficients in practice in an objective way since the performance is very sensitive to these values?\n\n3. The K-means results reported in Table 2 for CIFAR10/100 do not match with those in Table 3 in the IMSAT paper, but both employed the pretrained ResNet-50 extracted features. This raises the question of the reliability of the reported quantitative results.\n\n4. Why only used the 20 coarse categories for the CIFAR100 benchmark?\n\n5. The authors mentioned that \"if we simultaneously learn the representation/feature and cluster the data, we observed that the results are less sensitive to such coefficient.\" But I could not find these results in the main text or supplementary material.\n\n6. Table 4 presents a comparison with SCAN but does not include any comparisons with the baseline methods featured in Table 3 following the same two-stage training.\n\nLIMITATIONS:\nThere is no discussion of limitations or potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

comments: SUMMARY:
This paper first discusses a number of general properties of  entropy clustering methods, including their relation to K-means and unsupervised SVM-based techniques.
Then the aurthors find that cross-entropy is not robust to pseudo-label errors in clustering.
Finally, this paper proposes a new loss function based on reverse KL divergence for clustering  to obtain more robust and fair clustering.

STRENGTHS:
(1) The proposed  loss function is interesting and and seems to be effective to obtain more robust and fair clustering.

(2) The authors try to establish connections between entropy clustering methods and classical methods.

WEAKNESSES:
(1) This paper is not well organized. There are too many details for the proposed method. Some of them can be moved to Appendix.

(2) There can be more descriptions and examples about the proposed method.

(3) The proposed method is merely tested on three image datasets.

QUESTIONS:
(1) Will the authors open-source the code?

LIMITATIONS:
I mentioned them in the Weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a very interesting analysis of discriminative entropy clustering in the literature and their use for self-labeling highlighting clear interpretation of the conditional and marginal entropy terms as decisiveness (push to have confident predictions) and fairness (to encourage desired proportions in clusters). The paper analyzes the variants of this kind of losses and their connections. They also discusses the relationship of this loss to K-means and refines the previously mistaken connection in a previous paper to point out the distinct difference. They further point out the effectiveness of reverse cross-entropy in case of uncertainty error and forward KL term to make them more effective  in the endpoints of softmax  interval. They also propose to use the regularization of classifier weights for margin maximization similar to SVM. A closed form update is derived to compute the pseudolabels from the combined weighted loss and shown its efficacy in clustering experiments.\n\nSTRENGTHS:\n- I think the paper discusses beautifully the intricacies and usage of Mutual information based entropy clustering loss, which is widely used in self labeling/self supervised/ weak supervised learning. The flow of the discussion is to the point and tries to give interpretation of each terms in the loss in a concise manner by drawing connection among the variants. \n\n- The paper discusses the the previous link of entropy clustering to K-means and identify the distinct difference to k-means which was in fault in previous literature. On top of that they find the usefulness of the classifier weight regularization used in previous proof to link to the loss explicitly for margin maximization similar to SVM based clustering. \n\n- The use of reverse cross-entropy and forward KL term and the motivation behind it is explained  very well with the aid of Figure 2, so that they are more robust in case of uncertainty around the corner of the softmax simplex.\n\n- The formulation of the EM algorithm is nice to make it work faster along with batchwise operation, showing its global optimal solution guarantee at convergence due to drawing convexity with formulated tight upper bound.\n- The experimental results shows clear improvement according to their claim.\n\nWEAKNESSES:\n1. I would say the results of joint clustering and feature learning in Table 3 is not encouraging even when showing a very shallow network of VGG4, the improvement is not significant apart from  MNIST. But in Resnet-18,  the inductive bias learned from pretraining is helping, then the improvement from the proposed loss might not improve very significantly with the proposed loss.  Also in Table 4, the regularization on the feature extractor $\\textbf{w}$ done or not in the loss or by weight decay? \n\n2. The experiments on semi supervised learning could also be shown to understand more when some supervision is available. How the idea of using reverse cross-entropy could be used in case of labeled one-hot y in equation 13? or it will be the regular cross-entropy for labeled set and reverse for pseudolabels with updated y_i?\n\n3. What if the loss in 13 is directly optimized with gradient descent instead of using the EM? Although, it seems if $y_i= \\sigma_i$ the reverse cross-entropy does not change anything if not updating y_i with closed-form update, is it?\n\nQUESTIONS:\n- In Algorithm 1, after updating the pseudolabels y_i in closed form solutions, the loss in the gradient descent is without the KL term in ALgorithm 1 in appendix. Is it a typo or the KL term is not required here?\n\nLIMITATIONS:\nThe limitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors consider discriminative entropy clustering and produce a discussions linking previous works. They have a version of the algorithm based on EM and a modified KL-divergence term. Experiments show the modified algorithm works better than competing methods with small networks.

STRENGTHS:
- The authors provide a good overview of discriminative entropy clustering and its development, from the work of Bridle et al to the regularized version by Krause et al, to the more recent work using deep learning and representation learning (Asano et al and Jabi et al).

WEAKNESSES:
- The contribution of this paper is somewhat limited: 
  1. The pointing out of a proof error in Jabi et al is helpful but is not significant on its own
  2. The discussion on SVM is based on previous works and simply replaces the logistic loss with margin loss, and is not particularly insightful 
  3. Section 3 is the authors' contributed new algorithm, but the main difference with previous works is changing the order of the KL term in the objective. 

- The improvements in empirical evaluations, compared to other methods, are somewhat limited. Many of the improvements are within standard error of competing methods. 

- The authors use a lot of space to discuss previous work (first 5.5 pages), and not enough space to explain what is new about their method and specifically what problems it addresses.

QUESTIONS:
- Is there any explanation on the joint clustering and feature learning working only on small networks like VGG4, but not on larger VGG or ResNet18?

LIMITATIONS:
Limitations not mentioned; potential negative societal impact not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06