Title: Collision Cross-entropy for Soft Class Labels and Entropy-based Clustering
Venue: NeurIPS 2024
Accepted: no
Keywords: collision cross entropy

entropy-based clustering
Abstract: We propose ''collision cross-entropy'' as a robust alternative to Shannon's cross-entropy (CE) loss when class labels are represented by soft categorical distributions y. In general, soft labels can naturally represent ambiguous targets in classification. They are particularly relevant for self-labeled clustering methods, where latent pseudo-labels $y$ are jointly estimated with the model parameters and uncertainty is prevalent. 
In case of soft labels $y$, Shannon's CE teaches the model predictions $\sigma$ 
to reproduce the uncertainty in each training example, which inhibits the model's ability to learn and generalize from these examples. As an alternative loss, we propose the negative log of ``collision probability'' that
maximizes the chance of equality between two random variables, predicted class and unknown true class, 
whose distributions are $\sigma$ and $y$. We show that it has the properties of a generalized CE. 
The proposed collision CE agrees with Shannon's CE for one-hot labels $y$, 
but the training from soft labels differs. For example, unlike Shannon's CE, data points where $y$ is a uniform distribution have zero contribution to the training. Collision CE significantly improves classification 
supervised by soft uncertain targets. 
Unlike Shannon's, collision CE is symmetric for $y$ and $\sigma$, which is particularly relevant when 
both distributions are estimated in the context of self-labeled clustering.
Focusing on discriminative deep clustering where self-labeling and entropy-based losses are dominant, 
we show that the use of collision CE improves the state-of-the-art. 
We also derive an efficient EM algorithm that significantly 
speeds up the pseudo-label estimation with collision CE.

Abstract

We propose 'collision cross-entropy' as a robust alternative to Shannon's crossentropy (CE) loss when class labels are represented by soft categorical distributions y. In general, soft labels can naturally represent ambiguous targets in classification. They are particularly relevant for self-labeled clustering methods, where latent pseudo-labels y are jointly estimated with the model parameters and uncertainty is prevalent. In case of soft labels y , Shannon's CE teaches the model predictions σ to reproduce the uncertainty in each training example, which inhibits the model's ability to learn and generalize from these examples. As an alternative loss, we propose the negative log of 'collision probability' that maximizes the chance of equality between two random variables, predicted class and unknown true class, whose distributions are σ and y . We show that it has the properties of a generalized CE. The proposed collision CE agrees with Shannon's CE for one-hot labels y , but the training from soft labels differs. For example, unlike Shannon's CE, data points where y is a uniform distribution have zero contribution to the training. Collision CEsignificantly improves classification supervised by soft uncertain targets. Unlike Shannon's, collision CE is symmetric for y and σ , which is particularly relevant when both distributions are estimated in the context of self-labeled clustering. Focusing on discriminative deep clustering where self-labeling and entropy-based losses are dominant, we show that the use of collision CE improves the state-ofthe-art. We also derive an efficient EM algorithm that significantly speeds up the pseudo-label estimation with collision CE.

1 Introduction and Motivation

Shannon's cross-entropy H ( y, σ ) is the most common loss for training network predictions σ from ground truth labels y in the context of classification, semantic segmentation, etc. However, this loss may not be ideal for applications where the targets y are soft distributions representing various forms of uncertainty. For example, this paper is focused on self-labeled classification [17, 1, 15, 16] where the ground truth is not available and the network training is done jointly with estimating latent pseudo-labels y . In this case soft y can represent the distribution of label uncertainty. Similar uncertainty of class labels is also natural for supervised problems where the ground truth has errors [26, 41]. In any cases of label uncertainty, if soft distribution y is used as a target in H ( y, σ ) , the network is trained to reproduce the uncertainty, see the dashed curves in Fig.1.

Our work is inspired by generalized entropy measures [33, 18]. Besides mathematical generality, the need for such measures 'stems from practical aspects when modelling real world phenomena though entropy optimization algorithms' [30]. Similarly to L p norms, parametric families of generalized entropy measures offer a wide spectrum of options. The Shannon's entropy is just one of them. Other measures could be more 'natual' for any given problem.

A simple experiment in Figure 2 shows that 37 Shannon's cross-entropy produces deficient so38 lutions for soft labels y compared to the pro39 posed collision cross-entropy . The limitation 40 of the standard cross-entropy is that it encour41 ages the distributions σ and y to be equal, see 42 the dashed curves in Fig.1. For example, the 43 model predictions σ are trained to copy the un44 certainty of the label distribution y , even when 45 y is an uninformative uniform distribution. In 46 contrast, our collision cross-entropy (the solid 47 curves) gradually weakens the training as y 48 gets less certain. This numerical property of 49 our cross-entropy follows from its definition 50 (9) - it maximizes the probability of 'colli51 sion', which is an event when two random 52 variables sampled from the distributions σ and 53 y are equal. This means that the predicted class 54 value is equal to the latent label. This is signif55 icantly different from the σ = y encouraged 56 by the Shannon's cross-entropy. For example, 57 if y is uniform then it does not matter what the 58 model predicts as the probability of collision 59 1 K would not change. 60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

y1 = (0.95,0.05)

ln 0.2

ln 0.5

ln 0.5

y3 = (0.5,0.5)

ln0.8

ln 0.95

0

0.5

0.8

0.95 1

b

Organization of the paper: After the summary of our contributions below, Section 2 reviews the relevant background on self-labeling models/losses and generalized information measures for entropy, divergence, and cross-entropy. Then, Section 3 introduces our collision cross entropy measure, discusses its properties, related formulations of Rényi cross-entropy, and relation to noisy labels in fully-supervised settings. Section 4 formulates our self-labeling loss by replacing the Shannon's cross entropy term in a representative state-of-the-art formulation using soft pseudo-labels [16] with our collision-cross-entropy. The obtained loss function is convex w.r.t. pseudo-labels y , which makes estimation of y amenable to generic projected gradient descent. However, Section 4 derives a much faster EM algorithm for estimating y . As common for self-labeling, optimization of the total loss w.r.t. network parameters is done via backpropagation. Section 5 presents our experiments, followed by conclusions.

Summary of Contributions: We propose the collision cross-entropy as an alternative to the standard Shannon's cross-entropy mainly in the context of self-labeled classification with soft pseudo-labels. The main practical advantage is its robustness to uncertainty in the labels, which could also be useful in other applications. The definition of our cross-entropy has an intuitive probabilistic interpretation that agrees with the numerical and empirical properties. Unlike the Shannon's crossentropy, our formulation is symmetric w.r.t. predictions σ and pseudo-labels y . This is a conceptual advantage since both σ and y are estimated/optimized distributions. Our cross-entropy allows efficient optimization of pseudo-labels by a proposed EM algorithm, that significantly accelerates a generic projected gradient descent. Our experiments show consistent improvement over multiple examples of unsupervised and semi-supervised clustering, and several standard network architectures.

2 Background Review 82

We study a new generalized cross-entropy measure in the context of deep clustering. The models are 83 trained on unlabeled data, but applications with partially labeled data are also relevant. Self-labeled 84 deep clustering is a popular area of research [5, 31]. More recently, the-state-of-the-art is achieved by 85 discriminative clustering methods based on maximizing the mutual information between the input and 86 the output of the deep model [3]. There is a large group of relevant methods [22, 10, 15, 17, 1, 16] 87 and we review the most important loss functions, all of which use standard information-theoretic 88 measures such as Shannon's entropy. In the second part of this section, we overview the necessary 89 mathematical background on the generalized entropy measures, which are central to our work. 90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

2.1 Information-based Self-labeled Clustering

The work of Bridle, Heading, and MacKay from 1991 [3] formulated mutual information (MI) loss for unsupervised discriminative training of neural networks using probability-type outputs, e.g. softmax σ : R K → ∆ K mapping K logits l k ∈ R to a point in the probability simplex ∆ K . Such output σ = ( σ 1 , . . . , σ K ) is often interpreted as a posterior over K classes, where σ k = exp l k ∑ i exp l i is a scalar prediction for each class k .

The unsupervised loss proposed in [3] trains the model predictions to keep as much information about the input as possible. They derived an estimate of MI as the difference between the average entropy of the output and the entropy of the average output

where c is a random variable representing class prediction, X represents the input, and the averaging is done over all input samples { X i } i M =1 , i.e. over M training examples. The derivation in [3] assumes that softmax represents the distribution Pr( c | X ) . However, since softmax is not a true posterior, the right hand side in (1) can be seen only as an MI loss. In any case, (1) has a clear discriminative interpretation that stands on its own: H ( σ ) encourages 'fair' predictions with a balanced support of all categories across the whole training data set, while H ( σ ) encourages confident or 'decisive' prediction at each data point implying that decision boundaries are away from the training examples [11]. Generally, we call clustering losses for softmax models 'information-based' if they use measures from the information theory, e.g. entropy.

Discriminative clustering loss (1) can be applied to deep or shallow models. For clarity, this paper distinguishes parameters w of the representation layers of the network computing features f w ( X ) ∈ R N for any input X and the linear classifier parameters v of the output layer computing K -logit vector v ⊤ f for any feature f ∈ R N . The overall network model is defined as

Aspecial 'shallow' case in (2) is a basic linear discriminator

directly operating on low-level input features f = X . Optimization of the loss (1) for the shallow model (3) is done only over linear classifier parameters v , but the deeper network model (2) is optimized over all network parameters [ v , w ] . Typically, this is done via gradient descent or backpropagation [35, 3].

Optimization of MI losses (1) during network training is mostly done with standard gradient descent or backpropagation [3, 22, 15]. However, due to the entropy term representing the decisiveness, such loss functions are

0.0

0.2

0.4

0.6

0.8

corruption level

25%

35%

45%

55%

65%

75%

85%

95%

accuracy

shannon CE:

H

(

y

,

)

shannon CE:

H

(

y

,

)

collision CE:

H

2(

y

,

)

non-convex and present challenges to the gradient descent. This motivates alternative formulations and optimization approaches. For example, it is common to incorporate into the loss auxiliary variables y representing pseudo-labels for unlabeled data points X and to estimate them jointly with optimization of the network parameters [10, 1, 16]. Typically, such self-labeling approaches to unsupervised network training iterate optimization of the loss over pseudo-labels and network parameters, similarly to the Lloyd's algorithm for K -means [2]. While the network parameters are still optimized via gradient descent, the pseudo-labels can be optimized via more powerful algorithms.

For example, self-labeling in [1] uses the following constrained optimization problem with discrete 140 pseudo-labels y 141

where ∆ K 0 , 1 are one-hot distributions, i.e. corners of the probability simplex ∆ K . Training the network predictions σ is driven by the standard cross entropy loss H ( y, σ ) , which is convex assuming fixed (pseudo) labels y . With respect to variables y , the cross entropy is linear. Without the balancing constraint ¯ y = u , the optimal y corresponds to the hard arg max( σ ) . However, the balancing constraint converts this into an integer programming problem that can be solved approximately via optimal transport [9]. The cross-entropy in (4) encourages the predictions σ to approximate one-hot pseudo-labels y , which implies the decisiveness.

Self-labeling methods for unsupervised clustering can also use soft pseudo-labels y ∈ ∆ K as target distributions in cross-entropy H ( y, σ ) . In general, soft targets y are common in H ( y, σ ) , e.g. in the context of noisy labels [41, 38]. Softened targets y can also assist network calibration [12, 26] and improve generalization by reducing over-confidence [29]. In the context of unsupervised clustering, cross-entropy H ( y, σ ) with soft pseudo-labels y approximates the decisiveness since it encourages σ ≈ y implying H ( y, σ ) ≈ H ( y ) ≈ H ( σ ) where the latter is the first term in (1). Instead of the hard constraint ¯ y = u used in (4), the soft fairness constraint can be represented by KL divergence KL (¯ y ∥ u ) , as in [10, 16]. In particular, [16] formulates the following self-labeled clustering loss

encouraging decisiveness and fairness as discussed. Similarly to (4), the network parameters in loss (5) are trained by the standard cross-entropy term, but optimization over relaxed pseudo-labels y ∈ ∆ K is relatively easy due to convexity. While there is no closed-form solution, the authors offer an efficient approximate solver for y . Iterating steps that estimate pseudo-labels y and optimize the model parameters resembles the Lloyd's algorithm for K-means. The results in [16] also establish a formal relation between the loss (5) and the K -means objective.

2.2 Generalized Entropy Measures

Below, we review relevant generalized formulations of the information-theoretic concepts: entropy, divergence, and cross-entropy. Rényi [33] introduced the entropy of order α > 0 for any probability distribution p

̸

derived as the most general measure of uncertainty in p satisfying four intuitively evident postulates. The entropy measures the average information and the order parameter α relates to the power of the corresponding mean statistic [44]. The general formula above includes the Shannon's entropy

as a special case when α → 1 . The quadratic or second-order Rényi entropy 164

is also known as a collision entropy since it is a negative log-likelihood of a 'collision' or 'rolling 165 double' when two i.i.d. samples from distribution p have equal values. 166

Basic characterization postulates in [33] also lead to the general Rényi formulation of the divergence , also known as the relative entropy , of order α > 0

̸

defined for any pair of distributions p and q . This reduces to the standard KL divergence when α → 1 167

168

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

169

170

171

172

173

174

175

176

177

and to the Bhattacharyya distance for α = 1 2 .

Optimization of entropy and divergence [24] is fundamental to many machine learning problems [37, 20, 19, 30], including pattern classification and cluster analysis [36]. However, the entropyrelated terminology is often mixed-up. For example, when discussing the cross-entropy minimization principle (MinxEnt), many of the references cited earlier in this paragraph define cross-entropy using the expression for KL-divergence (7). Nowadays, it is standard to define the Shannon's cross-entropy as

One simple explanation for the confusion is that KL-divergence D ( p, q ) and cross-entropy H ( p, q ) as functions of q only differ by a constant if p is a fixed known target, which is often the case.

3 Collision Cross-Entropy 178

Minimizing divergence enforces proximity between two distributions, which may work as a loss for training model predictions σ with labels y , for example, if y are ground truth one-hot labels. However, if y are pseudo-labels that are estimated jointly with σ , proximity between y and σ is not a good criterion for the loss. For example, highly uncertain model predictions σ in combination with uniformly distributed pseudo-labels y correspond to the optimal zero divergence, but this is not a very useful result for self-labeling. Instead, all existing self-labeling losses for deep clustering minimize Shannon's cross-entropy (8) that reduces the divergence and uncertainty at the same time

The entropy term corresponds to the 'decisiveness' constraint in unsupervised discriminative clus179 tering [3, 17, 1, 15, 16]. In general, it is recommended as a regularizer for unsupervised and 180 semi-supervised network training [11] to encourage decision boundaries away from the data points 181 implicitly increasing the decision margins. 182

We propose a new form of cross-entropy 183

that we call collision cross-entropy since it extends the collision entropy in (6). Indeed, (9) is the 184 negative log-probability of an event that two random variables with (different) distributions p and q 185 are equal. When training softmax σ with pseudo-label distribution y , the collision event is the exact 186 equality of the predicted class and the pseudo-label, where these are interpreted as specific outcomes 187 for random variables with distributions σ and y . Note that the collision event, i.e. the equality of 188 two random variables, has very little to do with the equality of distributions σ = y . The collision 189 may happen when σ = y , as long as σ · y > 0 . Vice versa, this event is not guaranteed even when 190 σ = y . It will happen almost surely only if the two distributions are the same one-hot. However, if 191 the distributions are both uniform, the collision probability is only 1 /K . 192

̸

As easy to check, the collision cross-entropy (9) can be equivalently represented as

where cos ( p, q ) is the cosine of the angle between p and q as vectors in R K and H 2 is the collision 193 entropy (6). The first term corresponds to a 'distance' between the two distributions: it is non194 negative, equals 0 iff p = q , and -ln cos ( · ) is a convex function of an angle, which can be interpreted 195 as a spherical metric. Thus, analogously to the Shannon's cross-entropy, H 2 is the sum of divergence 196 and entropy. 197

The formula (9) can be found as a definition of quadratic Rényi cross-entropy [30, 32, 46]. However, 198 we could not identify information-theoretic axioms characterizing a generalized cross-entropy. Rényi 199 himself did not discuss the concept of cross-entropy in his seminal work [33]. Also, two different 200 formulations of 'natural' and 'shifted' Rényi cross-entropy of arbitrary order could be found in 201 [44, 42]. In particular, the shifted version of order 2 agrees with our formulation of collision cross202 entropy (9). However, lack of postulates or characterization for the cross-entropy, and the existence of 203 multiple non-equivalent formulations did not give us the confidence to use the name Rényi. Instead, 204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

we use 'collision' due to its clear intuitive interpretation of the loss (9). But, the term 'cross-entropy' is used only informally.

The numerical and empirical properties of the collision cross-entropy (9) are sufficiently different from the Shannons cross-entropy (8). Figure 1 illustrates H 2 ( y, σ ) as a function of σ for different label distributions y . For confident y it behaves the same way as the standard cross entropy H ( y, σ ) , but softer low-confident labels y naturally have little influence on the training. In contrast, the standard cross entropy encourages prediction σ to be the exact copy of uncertainty in distribution y . Self-labeling methods based on H ( y, σ ) often 'prune out' uncertain pseudo-labels [4]. Collision cross entropy H 2 ( y, σ ) makes such heuristics redundant. We also demonstrate the 'robustness to label uncertainty' on an example where the ground truth labels are corrupted by noise, see Fig.2. This artificial fully-supervised test is used only to compare the robustness of (9) and (8) in complete isolation from other terms in the self-labeled clustering losses, which are the focus of this work.

Due to the symmetry of the arguments in (9), such robustness of H 2 ( y, σ ) also works the other way around. Indeed, self-labeling losses are often used for both training σ and estimating y : the loss is iteratively optimized over predictions σ (i.e. model parameters responsible for it) and over pseudolabel distribution y . Thus, it helps if y also demonstrates 'robustness to prediction uncertainty'.

Soft labels vs noisy labels: Our collision CE for soft labels, represented by distributions y , can be related to loss functions used for supervised classification with noisy labels [40, 28, 38], which assume some observed hard target labels l that may not be true due to corruption or 'noise'. Instead of our probability of collision

between the predicted class C and unknown true class T , whose distributions are prediction σ and soft target y , they maximize the probability that a random variable L representing a corrupted target equals the observed value l

where the approximation uses the model predictions σ k instead of true class probabilities Pr( T = k ) , which is a significant assumption. Vector Q l is the l -th row of the transition matrix Q , such that Q lk = Pr( L = l | T = k ) , that has to be obtained in addition to hard noisy labels l .

Our approach maximizing the collision probability based on soft labels y is a generalization of the methods for hard noisy labels. Their transitional matrix Q can be interpreted as an operator for converting any hard label l into a soft label y = Q ⊤ 1 l = Q l . Then, the two methods are numerically equivalent, though our statistical motivation is significantly different. Moreover, our approach is more general since it applies to a wider set of problems where the class target T can be directly specified by a distribution, a soft label y , representing the target uncertainty. For example, in fully supervised classification or segmentation the human annotator can directly indicate uncertainty (odds) for classes present in the image or at a specific pixel. In fact, class ambiguity is common in many data sets, though for efficiency, the annotators are typically forced to provide one hard label. Moreover, in the context of self-supervised clustering, it is natural to estimate pseudo-labels as soft distributions y . Such methods directly benefit from our collision CE, as this paper shows.

4 Our Self-labeling Loss and EM

Based on prior work (5), we replace the standard cross-entropy with our collision cross-entropy to formulate our self-labeling loss as follows:

To optimize such loss, we iterate between two alternating steps for σ and y . For σ , we use the standard stochastic gradient descent algorithms[34]. For y , we use the projected gradient descent (PGD) [7]. However, the speed of PGD is slow as shown in Table 1 especially when there are more classes. This motivates us to find more efficient algorithms for optimizing y . To derive such an algorithm, we made a minor change to (10) by switching the order of variables in the divergence term:

246

247

248

249

250

251

252

253

254

255

256

Such change allows us to use the Jensen's inequality on the divergence term to derive an efficient EM algorithm while the quality of the self-labeled classification results is almost the same as shown in the Appendix D.

EM algorithm for optimizing y We derive the EM algorithm introducing latent variables, K distributions S k ∈ ∆ M representing normalized support for each cluster over M data points. We refer to each vector S k as a normalized cluster k . Note the difference with distributions represented by pseudo-labels y ∈ ∆ K showing support for each class at a given data point. Since we explicitly use individual data points below, we will start to carefully index them by i ∈ { 1 , . . . , M } . Thus, we will use y i ∈ ∆ K and σ i ∈ ∆ K . Individual components of distribution S k ∈ ∆ M corresponding to data point i will be denoted by scalar S k i .

First, we expand (11) introducing the latent variables S k ∈ ∆ M

Due to the convexity of negative log , we apply the Jensen's inequality to derive an upper bound, i.e. 257 (13), to L CCE + . Such bound becomes tight when: 258

Next, we derive the M step. Introducing the hidden variable S breaks the fairness term into the sum of independent terms for pseudo-labels y i ∈ ∆ K at each data point i . The solution for S does not change (E step). Lets focus on the loss with respect to y . The collision cross-entropy (CCE) also breaks into the sum of independent parts for each y i . For simplicity, we will drop all indices i in variables y k i , S k i , σ k i . Then, the combination of CCE loss with the corresponding part of the fairness constraint can be written for each y = { y k } ∈ ∆ K as

First, observe that this loss must achieve its global optimum in the interior of the simplex if S k > 0 and u k > 0 for all k . Indeed, the second term enforces the 'log-barier' at the boundary of the simplex. Thus, we do not need to worry about KKT conditions in this case. Note that S k might be zero, in which case we need to consider the full KKT conditions. However, the Property 1 that will be mentioned later eliminates such concern if we use positive initialization. For completeness, we also give the detailed derivation for such case and it can be found in the Appendix B.

Adding the Lagrange multiplier γ for the simplex constraint, we get an unconstrained loss

that must have a stationary point inside the simplex. The following theorem indicates the way to solve the problem above. All the missing proofs can be found in Appendix A.

Theorem 1. [M-step solution]: The sum ∑ k y k as in (16) is positive, continuous, convex, and monotonically decreasing function of x on the specified interval. Moreover, there exists a unique solution { y k } ∈ ∆ k and x such that

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

The monotonicity and convexity of ∑ k y k with respect to x suggest that the problem (16) formulated 281 in Theorem 1 allows efficient algorithms for finding the corresponding unique solution. For example, 282

one can use the iterative Newton's updates to search for x in the specified interval. The following 283

Lemma gives us a proper starting point 284

Lemma 1. Assuming u k S k is positive for each k , then the reachable left end point in Theorem 1 can be written as

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

for Newton's method. The algorithm for M-step solution is summarized in Algorithm 1 in Appendix C. Note that we present the algorithm for only one data point, and we can easily and efficiently scale up for more data in a batch by using the Numba compiler. In the following, we give the property about the positivity of the solution. This property implies that if our EM algorithm has only (strictly) positive variables S k or y k at initialization, these variables will remain positive during all iterations.

Property 1 . For any category k such that u k > 0 , the set of strictly positive variables y k or S k can only grow during iterations of our EM algorithm for the loss (15) based on the collision cross-entropy.

Note that Property 1 does not rule out the possibility that y k may become arbitrarily close to zero during EM iterations. Empirically, we did not observe any numerical issues. The complete algorithm is given in Appendix C. Inspired by [39, 15], we also update our y in each batch. Intuitively, updating y on the fly can prevent the network from being easily trapped in some local minima created by the incorrect pseudo-labels.

5 Experiments

We apply our new loss to self-labeled classification problems in both shallow and deep settings, as well as semi-supervised modes. All the results are reproduced using either public codes or our own implementation under the same experimental settings for fair comparison. Our approach consistently achieves either the best or highly competitive results across all the datasets and is therefore more robust. All the missing details in the experiments can be found in Appendix E.

Dataset We use four standard datasets: MNIST [25], CIFAR10/100 [43] and STL10 [8]. The training and test data are the same unless otherwise specified.

Evaluation As for the evaluation of self-labeled classification, we set the number of clusters to the number of ground-truth categories. To calculate the accuracy, we use the standard Hungarian algorithm [23] to find the best one-to-one mapping between clusters and labels. We don't need this matching step if we use other metrics, i.e. NMI, ARI.

5.1 Clustering with Fixed Features

In this section, we test our loss as a proper clustering loss and compare it to the widely used Kmeans (generative) and other closely related losses (entropy-based and discriminative). We use the pretrained (ImageNet) Resnet-50 [14] to extract the features. For Kmeans, the model is parameterized by K cluster centers. Comparably, we use a one-layer linear classifier followed by softmax for all other losses including ours. Kmeans results were obtained using scikit-learn package in Python. To optimize the model parameters for other losses, we use

stochastic gradient descent. Here we report the average accuracy and standard deviation over 6 randomly initialized trials in Table 2.

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

5.2 Deep Clustering

In this section, we train a deep network to jointly learn the features and cluster the data. We test our method on both a small architecture (VGG4) and a large one (ResNet-18). The only extra standard technique we add here is self-augmentation following [15, 1, 6].

To train the VGG4, we use random initialization for network parameters. From Table 3, it can be seen that our approach consistently achieves the most competitive re-

sults in terms of accuracy (ACC). Most of the methods we compared in our work (including our method) are general concepts applicable to single-stage end-to-end training. To be fair, we tested all of them on the same simple architecture. But, these general methods can be easily integrated into other more complex systems with larger architecture such as ResNet-18.

In Table 4, we show the results using the pretext-trained network from SCAN [45] as initialization for our clustering loss as well as IMSAT and MIADM. We use only the clustering loss together with the self-augmentation (one augmentation per image). As shown in the table below, our method reaches a higher number with more robustness almost for every metric on all datasets compared to the SOTA method SCAN. More importantly, we consis-

tently improve over the most related method, MIADM, by a large margin, which clearly demonstrates the effectiveness of our proposed loss together with the optimization algorithm.

5.3 Semi-supervised Classification

Although our paper is focused on self-labeled classification, we find it also interesting and natural to test our loss under semi-supervised settings where partial data is provided with ground-truth labels. We use the standard cross-entropy loss for labeled data and directly add it to the self-labeled loss to train the network initialized by the pretext-trained network following [45].

6 Conclusion

We propose a new collision cross-entropy loss. Such loss is naturally interpreted as measuring the probability of the equality between two random variables represented by the two distributions σ and y , which perfectly fits the goal of self-labeled classification. It is symmetric w.r.t. the two distributions instead of treating one as the target, like the standard cross-entropy.

While the latter makes the network copy the uncertainty in estimated pseudo-labels, our cross-entropy naturally weakens the training on data points where pseudo labels are more uncertain. This makes our cross-entropy robust to labeling errors. In fact, the robustness works both for prediction and for pseudo-labels due to the symmetry. We also developed an efficient EM algorithm for optimizing the pseudo-labels. Such EM algorithm takes much less time compared to the standard projected gradient descent. Experimental results show that our method consistently produces top or near-top results on all tested clustering and semi-supervised benchmarks.

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

References

Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations , 2020.

Christopher M. Bishop. Pattern Recognition and Machine Learning . Springer, 2006.

John S. Bridle, Anthony J. R. Heading, and David J. C. MacKay. Unsupervised classifiers, mutual information and 'phantom targets'. In NIPS , pages 1096-1101, 1991.

Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In International Conference on Computer Vision (ICCV) , pages 5879-5887, 2017.

Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEEE international conference on computer vision , pages 5879-5887, 2017.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Yunmei Chen and Xiaojing Ye. Projection onto a simplex, 2011.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision , pages 5736-5745, 2017.

Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems , 17, 2004.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning , pages 1558-1567. PMLR, 2017.

Mohammed Jabi, Marco Pedersoli, Amar Mitiche, and Ismail Ben Ayed. Deep clustering: On the link between discriminative models and k-means. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(6):1887-1896, 2021.

Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9865-9874, 2019.

Jagat N. Kapur. Measures of Information and Their Applications . John Wiley and Sons, 1994. 417

Jagat N. Kapur and Hiremagalur K. Kesavan. Entropy Optimization Principles and Applications . 418 Springer, 1992. 419

Hiremagalur K. Kesavan and Jagat N. Kapur. Maximum Entropy and Minimum Cross-Entropy 420 Principles: Need for a Broader Perspective , pages 419-432. Springer, 1990. 421

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR 422 (Poster) , 2015. 423

Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized 424 information maximization. Advances in neural information processing systems , 23, 2010. 425

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics 426 quarterly , 2(1-2):83-97, 1955. 427

Solomon Kullback. Information Theory and Statistics . Wiley, New York, 1959. 428

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document 429 recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 430

431

432

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

433

434

435

436

437

438

NSD. Natural Scenes Dataset [NSD]. https://www.kaggle.com/datasets/ nitishabharathi/scene-classification , 2020.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) , pages 1944-1952, 2017.

Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regular439 izing neural networks by penalizing confident output distributions. 2017. 440

Jose C. Principe, Dongxin Xu, and John W. Fisher III. Information-theoretic learning. Advances 441 in unsupervised adaptive filtering , 2000. 442

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with 443 deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 , 2015. 444

Sudhir Rao, Allan de Medeiros Martins, and José C. Príncipe. Mean shift: An information 445 theoretic perspective. Pattern Recognition Letters , 30:222-230, 2009. 446

Alfréd Rényi. On measures of entropy and information. Fourth Berkeley Symp. Math. Stat. 447 Probab. , 1:547-561, 1961. 448

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint 449 arXiv:1609.04747 , 2016. 450

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by 451 back-propagating errors. Nature , 323(6088):533-536, 1986. 452

John E. Shore and Robert M. Gray. Minimum cross-entropy pattern classification and cluster 453 analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 11-17, 1982. 454

455

456

457

John E. Shore and Rodney W. Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on Information Theory , 26(1):547-561, 1980.

458

459

460

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical genera461 tive adversarial networks. In International Conference on Learning Representations , 2015. 462

Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training 463 convolutional networks with noisy labels. ICLR workshop , 2015. 464

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization 465 framework for learning with noisy labels. In Proceedings of the IEEE conference on computer 466 vision and pattern recognition , pages 5552-5560, 2018. 467

Ferenc C. Thierrin, Fady Alajaji, and Tamás Linder. Rényi cross-entropy measures for common 468 distributions and processes with memory. Entropy , 24(10), 2022. 469

470

471

472

Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence , 30(11):1958-1970, 2008.

473

474

475

476

477

478

Francisco J. Valverde-Albacete and Carmen Peláez-Moreno. The case for shifting the Rényi entropy. Entropy , 21(1), 2019.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X , pages 268-285. Springer, 2020.

Xiao-Tong Yuan and Bao-Gang Hu. Robust feature extraction via information theoretic learning. 479 In International Conference on Machine Learning, (ICML) , page 1193-1200, 2009. 480

A Missing proofs 481

Theorem 2. [M-step solution]: The sum ∑ k y k as in (17) is positive, continuous, convex, and 482 monotonically decreasing function of x on the specified interval. Moreover, there exists a unique 483 solution { y k } ∈ ∆ k and x such that 484

Proof. All y k in (17) are positive, continuous, convex, and monotonically decreasing functions of x 485 on the specified interval. Thus, ∑ y k behaves similarly. Assuming that max is the index of prediction 486 σ max , we have y max → + ∞ when approaching the interval's left endpoint x → σ max 1+ λu ⊤ S . Thus, 487 ∑ y k > 1 for smaller values of x . At the right endpoint x = σ max we have y k ≤ λu k S k λu ⊤ S for all k 488 implying ∑ y k ≤ 1 . Monotonicity and continuity of ∑ y k w.r.t. x imply the theorem. 489

Lemma 2. Assuming u k S k is positive for each k , then the reachable left end point in Theorem 1 can be written as

Proof. Firstly, we prove that l is (strictly) inside the interior of the interval in Theorem 1. For the left 490 end point, we have 491

For the right end point, we have 492

Therefore, l is a reachable point. Moreover, any σ max 1+ λu ⊤ S < x < l will still induce positive y k for any k and we will also use this to prove that x should not be smaller than l . Let

then we can substitute l into the x of y c . It can be easily verified that y c = 1 at such l . Since y c is monotonically decreasing in terms of x , any x smaller than l will cause y c to be greater than 1. At the same time, other y k is still positive as mentioned just above, so the ∑ k y k will be greater than 1. Thus, l is a reachable left end point.

493

494

495

496

497

498

Property 2 . For any category k such that u k > 0 , the set of strictly positive variables y k or S k can only grow during iterations of our EM algorithm for the loss (d) based on the collision cross-entropy.

Proof. As obvious from the E-step (14), it is sufficient to prove this for variables y k . If y k = 0 , then 499 the E-step (14) gives S k = 0 . According to the M-step for the case of collision cross-entropy, variable 500 y k may become (strictly) positive at the next iteration if σ k = σ max . Once y k becomes positive, the 501 following E-step (14) produces S k > 0 . Then, the fairness term effectively enforces the log-barrier 502 from the corresponding simplex boundary making M-step solution y k = 0 prohibitively expensive. 503 Thus, y k will remain strictly positive at all later iterations. 504

B Complete Solutions for M step 505

The main case when u k S k > 0 for all k is presented in the main paper. Here we derive the case when there exist some k such that u k S k = 0 . Assume a non-empty subset of categories/classes

̸

̸

In this case the second term (fairness) in our loss (d) does not depend on variables y k for k ∈ K o . Also, note that the first term ( collision cross-entropy) in (d) depends on these variables only via their linear combination ∑ k ∈ K o σ k y k . It is easy to see that for any given confidences y k for k ∈ ¯ K o it is optimal to put all the remaining confidence 1 -∑ k ∈ ¯ K o y k into one class c ∈ K o corresponding to the larges prediction among the classes in K o

so that

Then, our loss function (d) can be written as 506

that gives the Lagrangian function incorporating the probability simplex constraint 507

The stationary point for this Lagrangian function should satisfy equations 508

which could be easily written as a linear system w.r.t variables y k for k ∈ ¯ K o ∪ { c } . 509

We derive a closed-form solution for the stationary point as follows. Substituting γ from the right 510 equation into the left equation, we get 511

Summing over k ∈ ¯ K o we further obtain 512

giving a closed-form solution for σ ⊤ y and its non-empty complement

Substituting this back into (f) we get closed-form solutions for y k

Note that positivity and boundedness of y k requires σ c > σ k for all k ∈ ¯ K o . In particular, this means σ c = σ max , but it also requires that all σ k for k ∈ ¯ K o are strictly smaller than σ max . We can also write the corresponding closed-form solution for y c

Note that this solution should be positive y c > 0 as well. 513

In case any of the mentioned constraints ( σ c > σ k , ∀ k ∈ ¯ K o and y c > 0 ) is not satisfied, the 514 complimentary slackness (KKT) can be used to formally prove that the optimal solution is y c = 0 . 515 That is, y k = 0 for all k ∈ K o . This reduces the optimization problem to the earlier case focusing 516 on resolving y k for k ∈ ¯ K o . This case is guaranteed to find a unique solution in the interior of the 517 simplex ∆ ¯ K o . Indeed, since inequality u k S k > 0 holds for all k ∈ ¯ K o , the strong fairness enforces a 518 log-barrier for all the boundaries of this simplex. 519

C Optimization algorithms 520

Algorithm 1: Newton's method for M-step

Input :

Output:

Initialize calculate

{ y k }

x

f

(

x

)

k

1+

∑

σ

k

⊤

λu

S

λu

-

k

λu

S

k

k

λu

S

+1

-

k

S

x

end

Algorithm 2: Optimization for (11)

Input

: network parameters and dataset

Output: network parameters for each epoch do for each iteration do Initialize y by the network output at current stage as a warm start; while not convergent do E step: S k i = y k i ∑ j y k j ; Mstep: find y k i using Newton's method; end Update network using loss H 2 ( y, σ ) via stochastic gradient descent end end

D Self-supervision Loss Comparison 521

⊤

k

-

1

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

E Experiments

E.1 Network Architecture

The network structure of VGG4 is adapted from [17]. We used standard ResNet-18 from the PyTorch library as the backbone architecture for Figure 2. As for the ResNet-18 used for Table 4, we used the code from this repository 1 .

E.2 Experimental Settings

Here we present the missing details of experimental settings for Table 2 - 5. As for Table 2, the weight of the linear classifier is initialized by using Kaiming initialization [13] and the bias is all set to zero at the beginning. We use the l 2 -norm weight decay and set the coefficient of this term to 0.001, 0.02, 0.009, and 0.02 for MNIST, CIFAR10, CIFAR100 and STL10 respectively. The optimizer is stochastic gradient descent with a learning rate set to 0.1. The batch size is set to 250. The number of epochs is 10. We set λ in our loss to 100 and separately tuned the hyperparameters for other methods.

For Table 3, we use Adam [21] with learning rate 1 e -4 for optimizing the network parameters. We set batch size to 250 for CIFAR10, CIFAR100 and MNIST and we use 160 for STL10. We report the mean accuracy and Std from 6 runs with random initializations. We use 50 epochs for each run and all methods reach convergence within 50 epochs. The weight decay coefficient is set to 0.01.

As for the training of ResNet-18 in Table 4, we still use the Adam optimizer, and the learning rate is set to 5 e -2 for the linear classifier and 1 e -5 for the backbone. The weight decay coefficient is set to 1 e -4 . The batch size is 200 and the number of total epochs is 50. The λ is still set to 100. We only use one augmentation per image, and the coefficient for the augmentation term is set to 0.5, 0.2, and 0.4 respectively for STL10, CIFAR10, and CIFAR100 (20).

As for the semi-supervised settings, we made two changes compared to the above. First, we added the cross-entropy loss on the labeled images and set the weight to 2, and separately tuned the hyperparameters for other methods. Second, the pseudo-labels on the labeled images are constrained to be the ground truth during the optimization.

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This can be justified from reading the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are clearly stated and missing proofs can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the details can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code is released upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Mean and standard deviation are provided for most of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The datasets are not large. We used single P100 GPU card.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite them and put the links as well.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSoft labels are often used to represent ambiguous/noisy/uncertain targets in classification, particularly in self-labelled clustering, where pseudo-labels are estimated together with model parameters.\nThe authors propose an alternative to Shannon cross-entropy for a loss term, called the collision probability. \nThis term arises as a limiting case of a Renyi entropy, or as a probability that two random variables are equal.\nThe collision cross entropy admits several advantageous properties: it is robust to large deviations in the target data, it agrees with Shannon cross-entropy for one-hot labels, it is symmetric, and points that are labelled as uniform distribution have no contribution to training.\nThe authors provide an EM algorithm for pseudo-label estimation and show state of the art results.\n\nSTRENGTHS:\n- The paper is **very well written**, providing strong intuition and flowing prose. The intuition in Figure 1 is helpful, especially in showing that the proposed measure is robust to large target errors. \n- The main technical element appears to be an EM algorithm for solving the clustering problem obtained by using a collision cross-entropy in place of the Shannon entropy (which a swap in the arguments between equations (10) and (11)). This algorithm appears to be **technically sound**, and guarantees convergence of the subproblem in the M step. \n- The proposed term has **several nice properties** (as I mentioned earlier). Tt is robust to large deviations in the target data, it agrees with Shannon cross-entropy for one-hot labels, it is symmetric, and points that are labelled as uniform distribution have no contribution to training.\n\nWEAKNESSES:\n- My main concern is that **conceptually, the contributions are rather limited**. Generalisations of entropy are well-known, and as far as I understand (correct me if I am wrong), the main contribution is that authors use a different measure of entropy to Shannon entropy inside existing formulations. This leads the authors to investigate EM-algorithm and empirical performances, but as the paper is currently written (see further comments below), I cannot see whether these EM-algorithm and empirical performance benefits are actually real and beneficial. I also do not understand why this particular notion of entropy was used, compared with the other spectra of entropies. \n- An incomplete review of relevant generalized formulations of entropy is provided. This is not a weakness per se, however **perhaps the title in section 2.2 could be changed to something like Renyi Entropy**. Similarly tone down the discussion of generalised entropy measures throughout the paper. Alternatively, the authors might consider expanding their discussion and including more well-known entropy measures. For example, see section 8 and 11 of [1].\n- The bold numbers in table 2 require clarification. The caption doesn't mention the number of trials (however the text mentions 6 trials). Compared with MIGD, excluding MNIST, due to the high variance in the trials, the results do not appear to be **statistically significant**. Perhaps the authors could consider running more trials and performing a significance test, and/or also bold relevant entries in MIGD. \n- As above for Table 3, 4 and 5. \n- It is **not clear how long the method takes to run** compared with competitors. Does the EM algorithm outperform a naive marginalisation of the log likelihood (using e.g. MC), both in terms of time and in terms of predictive performance? \n- Related to the above, is the reason for specialising on $\\alpha \\to 1$ because it allows for the EM algorithm? If you consider other values of $\\alpha$, how do the results compare in terms of time and performance. Or is this setting intractable?\n\n\n[1] Generalized Thermostatistics, Jan Naudts, 2011.\n\n\nMinor:\n- The text in the tables is too small to read without zooming in a lot.\n- Recommend less active tense in the abstract: \"In case of soft labels y, Shannon’s CE teaches the model predictions σ to reproduce the uncertainty in each training example\" could be \"In case of soft labels y, Shannon’s CE results in model predictions σ which reproduce the uncertainty in each training example\".\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nThe author checklist appears to be incomplete. The authors answer NA to \"Does the paper discuss the limitations of the work performed by the authors?\", without a justification. I do see a small discussion around local minima and numerical instability towards the end of section 4, but I think these could be further elaborated on.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the concept of collision cross-entropy (CCE) as an alternative to Shannon's cross-entropy (SCE) for self-labeling in the context of unsupervised and semi-supervised learning. The primary motivation is to address the limitations of SCE, especially its sensitivity to label noise and uncertainty. CCE aims to enhance robustness to such uncertainties by defining a probabilistic interpretation that encourages collision events between predicted and true distributions. The paper provides theoretical foundations, describes an EM algorithm for efficient optimization, and presents experimental results demonstrating the superior performance of CCE over SCE on the task of deep clustering.\n\nSTRENGTHS:\nOriginality\n-The paper introduces a novel loss function, the collision cross-entropy, which is well-motivated by the need to handle soft and uncertain labels in classification tasks, particularly in self-labeled clustering. The idea of maximizing the collision probability is distinct from the traditional approach of minimizing the (implicit) KL divergence between distributions.\n\nQuality\n-The paper provides a solid theoretical foundation for the collision cross-entropy, including its properties and relationship to other entropy measures. The derivation of an efficient EM algorithm for pseudo-label estimation further strengthens the paper's technical contribution.\n\nClarity \n- The paper is generally well-written and organized. The motivation, theoretical analysis, and experimental results are presented clearly. The authors provide sufficient details for an expert reader to understand and potentially reproduce the work.\n\nSignificance\n- The proposed collision cross-entropy has the potential to be a valuable tool for handling soft and uncertain labels in various machine learning tasks.\n\nWEAKNESSES:\nQuality\n- The superiority of CCE seems to hinge on making the model capture the same \"decisions\" as the target distribution, without forcing the model to capture the entirety of the distribution, as well as de-weighting target distributions which are not spiky. While the properties of the loss are clear, it is not self-evident to me that the properties *of the loss function* translate into necessarily *better properties for models*, both as a function for training a classification model directly or for clustering. \n- In addition, the experiments were conducted on fairly old architectures (VGG, ResNet) and small datasets. Often improvements on small datasets do not translate into improvements on larger-scale models. I would encourage the authors to examine for full imagenet dataset at the very least. This also open up the capability to look at various robustness / calibration properties of the models on the various corrupted forms of ImageNet. \n\nClarity\n- Certain sections, the task to which this method is applied and the desired model properties for the task could be more clearly explained. It took me a while to get my head around the deep clustering task which the authors are solving. \n\nSignificance\n- The impact of CCE on real-world applications beyond the presented datasets and tasks could be further elaborated. This notion that CCE is better for noisy pseudo labels immediately suggests to me examining it as a loss function for doing distillation / noisy teacher-student training of a model on a pseudo-labelled corpus of data, however, I didn't see any links to the area of distillation / teacher-student training within this paper.\n- The significance would be bolstered by demonstrating CCE's performance on larger scale, more diverse and challenging datasets.\n\nQUESTIONS:\n- **Impact of Soft Label Quality:** How does the quality of the soft labels (e.g., the degree of uncertainty) affect the performance of the collision cross-entropy compared to the standard cross-entropy? Are there scenarios where the standard cross-entropy might be preferable? How strongly does this problem scale with the number of classes? \n- **Numerical Stability:** Are there any numerical stability issues in optimizing CCE? I can imaging the taking the log of the sum of a product of small numbers can cause problems.\n- **Applicability to Other Tasks:** The paper focuses on self-labeled clustering. Can the collision cross-entropy be applied to other tasks involving soft labels, such as semi-supervised learning, distillation and teacher-student training? What modifications or adaptations might be necessary? How would this work on sequence-based pseudo-labels, such as are encountered in ASR, machine translation and language modeling. It seems to me that CCE could be tried as a drop-in-replacement loss function.\n\nLIMITATIONS:\n**Strengths:**\n- The paper acknowledges the need for robustness to label noise and addresses this effectively through CCE.\n- The paper briefly mentions the potential increase in privacy disclosure risk with larger synthetic datasets but does not elaborate on this limitation or discuss potential mitigation strategies. It would be beneficial to include a more detailed discussion of the privacy implications of the proposed method and any potential negative societal impacts.\n\n**Weaknesses:**\n- The discussion on limitations could be more explicit, particularly regarding any assumptions made and potential edge cases where CCE may not perform optimally.\n- The experimental evaluation uses very old model architectures (VGG, ResNet) and small datasets (CIFAR-10, CIFAR-100, MNIST, STL-10) which feature images only as large as 96x96 pixels. I would be curious whether the advantages of this method translate to high-dimensional image data with more classes, and on more modern, transformer-based  architectures (eg: ViT) .\n- Similarly, could the author's consider extending this approach to deal with pseudo-labelled sequence data, for language models or for translation models, for example?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the choice of the loss function in problems with soft distributions of the labels, in particular in the context of pseudo-labeling for unsupervised or self-supervised problems such as clustering. In sections 1-2 the paper gives a thorough review of existing practices and relevant theoretical research. In section 3 the paper proposes a new collision cross-entropy loss as a replacement of the standard Shannon loss, and discusses various aspects of this new loss. In section 4 the paper proposes a new EM algorithm for pseudo-label estimation in connection with the new loss. Finally, in Section 5 the new algorithm is experimentally compared with existing ones and is shown to outperform them.\n\nSTRENGTHS:\nThe paper is generally well-written. Sections 1-3 contain a thorough discussion of entropies and losses suitable for self-labeled clustering, with abundant references. The main point of the paper, the new collision cross-entropy loss, is well explained and motivated. The paper provides an experimental comparison of the proposed algorithm with alternatives and shows its significant advantage.\n\nWEAKNESSES:\nI'm confused by mixing the discussion of losses and EM algorithms in section 4. The bulk of the paper is focused exclusively on the advantages of the proposed new collision cross-entropy loss. The main claim in the abstract and introduction is that the proposed loss is better than the standard Shannon loss. The EM algorithm is mentioned only in the last line of abstract, as if in passing. However, the experimental comparison in section 5 obviously crucially depends on the EM algorithm proposed in section 4. How can we tell if the experimentally demonstrated advantage is due to the new loss or the EM algorithm? Since the main claim is about the superiority of the loss, why not just take any existing soft-labeled clustering algorithms and replace the standard Shannon loss by the proposed new loss? In my opinion, the lack of such a direct comparison substantially weakens the main claim of the paper. The advantage shown experimentally is good, but the conceptual takeaway may be misleading.\n\nI found section 4 on the EM algorithm harder to read relative to the other sections (in fact, I'm not familiar with such algorithms and not even sure what EM stands for - apparently Expectation-Maximization, but this acronym is not explained in the paper). In constrast to the other sections, this one seems to assume familiarity of the reader with related algorithms. I didn't understand, for example, how equation (14) (E-step) was derived.\n\nAnother weakness I see is that the strongest results of the paper are largely experimental (not counting general arguments and auxiliary theoretical constructions in Section 4), but, as far as I understand, they are not easily verifiable since the code is not open-sourced.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the loss function for soft class labels and entropy-based clustering. In particular, it introduces a new loss function called 'collision cross-entropy' as an alternative to Shannon's cross-entropy when class labels are represented by soft categorical distributions. The motivation for this new loss function is to handle ambiguous targets/labels in classification. The authors provide an EM algorithm for pseudo-label estimation and conduct experiments to demonstrate that this approach leads to improvements in classification accuracy when models are trained with soft, uncertain targets.\n\nSTRENGTHS:\n- The proposed collision cross-entropy may have advantages over Shannon's cross-entropy when handling soft labels in certain scenarios.\n- Through experiments, the authors demonstrate that the proposed method achieves better robustness to label uncertainty, which is important for self-labeled clustering methods.\n\nWEAKNESSES:\n- [**Theory-1**] The main contribution of this paper is proposing the new loss function 'collision cross-entropy'. However, there is not much theoretical analysis about this loss function. From the current paper presentation, the Eqn. (9) can be interpreted as a modified version (or inspired by) Eqn. (6). For example, by minimizing the new objective for learning linear models, could this new loss lead to the right linear classification model?\n\n- [**Theory-2**] For the EM algorithm, is there any convergence analysis for the EM algorithm proposed in this paper? \n\n- [**Experiments**] State-of-the-art for comparison. The methods for comparison in Table 1/2/3 are not very recent. It is possible that the previous methods still work well and be the state-of-the-art. However, I found some recent papers could achieve much better results, for example, the ACC on CIFAR10 of [DTC+2023] is 92%+, however the result in this paper is <84%.\n\n\n[DTC+2023] Unsupervised Manifold Linearizing and Clustering. Tianjiao Ding, Shengbang Tong, Kwan Ho Ryan Chan, Xili Dai, Yi Ma, Benjamin D. Haeffele. ICCV 2023.\n\nQUESTIONS:\n- What's the definition of $u$ in Eqn. (4) (5) (10)? The presentation of this paper could be further improved, especially the part related to loss functions.\n\n- The font size of Eqn. (16) is a bit odd.\n\n- What's the first letter 'y' in Line 3? If it is $y$, then make it consistent with the label $y$ in Line 6.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}