Title: Modulate Your Spectrum in Self-Supervised Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: self-supervised learning

whitening

dimensional collapse

spectral transformation

iterative normalization
Abstract: Whitening loss provides theoretical guarantee in avoiding feature collapse for self-supervised learning (SSL)  using joint embedding architectures. One typical implementation of whitening loss is hard whitening that designs whitening transformation over embedding and imposes the loss on the whitened output. In this paper, we propose spectral transformation (ST) framework to map the spectrum of embedding to a desired distribution during forward pass, and to modulate the spectrum of embedding by implicit gradient update during backward pass. We show that whitening transformation is a special instance of ST by definition, and there exist other instances that can avoid collapse by our empirical investigation. Furthermore, we propose a new instance of ST, called IterNorm with trace loss (INTL). We theoretically prove that INTL can avoid collapse and modulate the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization. Moreover, INTL achieves 76.6\% top-1 accuracy in linear evaluation on ImageNet using ResNet-50, which exceeds the performance of the supervised baseline, and this result is obtained by using a batch size of only 256. Comprehensive experiments show that INTL is a promising SSL method in practice. 

Abstract

Whitening loss provides theoretical guarantee in avoiding feature collapse for self-supervised learning (SSL) using joint embedding architectures. One typical implementation of whitening loss is hard whitening that designs whitening transformation over embedding and imposes the loss on the whitened output. In this paper, we propose spectral transformation (ST) framework to map the spectrum of embedding to a desired distribution during forward pass, and to modulate the spectrum of embedding by implicit gradient update during backward pass. We show that whitening transformation is a special instance of ST by definition, and there exist other instances that can avoid collapse by our empirical investigation. Furthermore, we propose a new instance of ST, called IterNorm with trace loss (INTL). We theoretically prove that INTL can avoid collapse and modulate the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization. Moreover, INTL achieves 76.6% top-1 accuracy in linear evaluation on ImageNet using ResNet-50, which exceeds the performance of the supervised baseline, and this result is obtained by using a batch size of only 256. Comprehensive experiments show that INTL is a promising SSL method in practice.

1 Introduction

Self-supervised learning (SSL) via joint embedding architectures to learn visual representations has made significant progress over the last several years [1, 18, 7, 9, 2, 30], almost outperforming their supervised counterpart on many downstream tasks [28, 23, 32]. This paradigm addresses to train a dual pair of networks to produce similar embeddings for different views of the same image [9]. One main challenge with the joint embedding architectures is how to prevent a collapse of the representation, in which the two branches ignore the inputs and produce identical and constant outputs [9]. A variety of methods have been proposed to successfully avoid collapse , including contrastive learning methods [41, 18, 34] that attract different views from the same image (positive pairs) while pull apart different images (negative pairs), and non-contrastive methods [16, 9] that directly match the positive targets without introducing negative pairs.

The collapse problem is further generalized into collapse

dimensional collapse

[20, 24] (or informational

[2]), where the embedding vectors only span a lower-dimensional subspace and would be

highly correlated. In this case, the covariance matrix of embedding has certain zero eigenvalues, which 30

degenerates the representation in SSL. To prevent dimensional collapse , a theoretically motivated 31

paradigm, called whitening loss, is proposed by minimizing the distance between embeddings of 32

positive pairs under the condition that embeddings from different views are whitened [12, 20]. 33

One typical implementation of whitening loss is hard whitening [12, 40] that designs whitening 34

transformation over mini-batch data and imposes the loss on the whitened output [12, 20, 40]. We note 35

that the whitening transformation is a function over embedding during forward pass, and modulates 36

the spectrum of embedding implicitly during backward pass when minimizing the objective. This 37

raises questions whether there exist other functions over embedding can avoid collapse? If yes, how 38

the function affects the spectrum of embedding? 39

This paper proposes spectral transforma40 tion (ST), a framework to modulate the 41 spectrum of embedding in joint embed42 ding architecture. ST maps the spec43 trum of embedding to a desired distribu44 tion during forward pass, and modulates 45 the spectrum of embedding by implicit 46 gradient update during backward pass 47 (Figure 1). This framework provides a 48 way to seek for functions beyond whiten49 ing transformation that can avoid dimen50 sional collapse. We show that whitening 51 transformation is a special instance of ST 52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

X: input

GLYPH(cmap:d835)~GLYPH(cmap:d835)

GLYPH(cmap:d835)1

GLYPH(cmap:d835)2

joint architecture

GLYPH(cmap:d835)1

GLYPH(cmap:d835)2

GLYPH(cmap:d835)1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(∙)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

′

′

(∙)

GLYPH(cmap:d835)

:

embedding

ST

ST

ℒ

GLYPH(cmap:d835)

:

transformed output

GLYPH(cmap:d835)GLYPH(cmap:d835)2

GLYPH(cmap:d835)GLYPH(cmap:d835)1

GLYPH(cmap:d835)Z1

GLYPH(cmap:d835)Z2

GLYPH(cmap:d835)2

using a power function by definition, and there exist other power functions that can avoid dimensional collapse by our empirical investigation (see Section 3.2 for details). We demonstrate that IterNorm [22], an approximating whitening method by using Newton's iterations [3, 42], is also an instance of ST, and show that IterNorm with different iteration number corresponds to different ST (see Section 3.3 for details). We further theoretically characterize how the spectrum evolves as the increasing of iteration number of IterNorm.

We empirically observe that IterNorm suffers from severe dimensional collapse and mostly fails to train the model in SSL unexpectedly, unlike its benefits in approximating whitening for supervised learning [22]. We thus propose IterNorm with trace loss (INTL), a simple solution to address the failure of IterNorm, by adding an extra penalty on the transformed output. Moreover, we theoretically demonstrate that INTL can avoid dimensional collapse, and reveal its mechanism in encouraging the covariance matrix of embedding to have equal eigenvalues. We conduct comprehensive experiments and show that INTL is a promising SSL method in practice. E.g. , INTL achieves 76.6% top-1 accuracy in linear evaluation on ImageNet using ResNet-50, which exceeds the performance of the supervised baseline, and this result is obtained by using a batch size of only 256. Our main contributions are summarized as follows:

We propose spectral transformation, a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show there exist other functions that can avoid dimensional collapse by empirical observations.

We propose a new instance of ST, called IterNorm with trace loss (INTL). We theoretically prove that INTL can avoid collapse and modulate the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization.

INTL is a promising SSL method in practice. INTL is on par with or outperforms the stateof-the-art SSL methods on standard benchmarks. Furthermore, these results are obtained by using a relatively small batch size.

2 Related Work

Our work is related to the SSL methods that address the feature collapse problem when using joint embedding architectures. Contrastive learning prevents collapse by attracting positive samples closer, and spreading negative samples apart [41, 43]. In these methods, negative samples play an important role and need to be well designed [29, 1, 19]. MoCos [18, 8] builds a memory bank with a momentum encoder to provide consistent negative samples, while SimCLR [7] addresses that more negative samples in a batch with strong data augmentations perform better. Our proposed INTL can avoid collapse and work well without negative samples.

Non-contrastive methods by designing asymmetric network architecture avoid feature collapse without introducing negative pairs explicitly [4, 5, 26, 16, 9]. BYOL [16] appends a predictor after the online network and introduce momentum into the target network. SimSiam [9] further generalizes BYOL by empirically showing that stop-gradient is essential for preventing trivial solutions. Other progresses include a cluster assignment prediction using Sinkhorn-Knopp algorithm [5], and an asymmetric pipeline with a self-distillation loss for Vision Transformers [6]. It remains not clear how the asymmetric network avoids collapse without negative pairs, leaving the debates on batch normalization (BN) [13, 39, 33] and stop-gradient [9, 45], even though preliminary works have attempted to analyze the training dynamics [38] and build a connection between non-contrastive and contrastive methods [36, 14]. Our work addresses the more challenging dimensional collapse problem, and theoretically shows that our INTL can avoid dimensional collapse.

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

Whitening loss is a theoretically motivated paradigm to prevent dimensional collapse [12]. One typical implementation of whitening loss is hard whitening that designs whitening transformation over mini-batch data and imposes the loss on the whitened output. The designed whitening transformation includes batch whitening in W-MSE [12] and Shuffled-DBN [20], channel whitening in CW-RGP [40], and the combination of both in Zero-CL [47]. Our proposed ST generalizes whitening transformation and provides a frame to modulate the spectrum of embedding. Our proposed INTL can improve these work in training stability and performance, by replacing whitening transformation with IterNorm [22] and imposing an additional trace loss on the transformed output. Furthermore, we theoretically show that our proposed INTL encourages the covariance matrix of embedding having equal eigenvalues.

Another way to implement whitening loss is soft whitening that imposes a whitening penalty as regularization on the embedding, including Barlow Twins [44], VICReg [2] and CCA-SSG [46]. Different from these works, our proposed INTL imposes the trace loss on the approximated whitened output, which implicitly encourages the covariance matrix of embedding having equal eigenvalues to avoid dimensional collapse.

There are also theoretical works analyzing how dimensional collapse occurs [20, 24] and how it can be avoided by using whitening loss [20, 40]. The recent works [17, 15] further discuss how to characterize the magnitude of dimensional collapse, and connect the spectrum of a representation to a power law. They show the coefficient of the power law is a strong indicator for the effects of the representation. Different from these works, our theoretical analysis presents a new thought in demonstrating how to avoid dimensional collapse, which provides theoretical basis for our proposed INTL.

3 Spectral Transformation beyond Whitening

3.1 Preliminary and Notation

Joint embedding architectures. Let x denote the input sampled uniformly from a set of images D , and T denote the set of data transformations available for augmentation. We consider a pair of neural networks F θ and F ′ θ ′ , parameterized by θ and θ ′ respectively. They take as input two randomly augmented views, x (1) = T 1 ( x ) and x (2) = T 2 ( x ) , where T 1 , 2 ∈ T ; and they output the embedding z (1) = F θ ( x (1) ) and z (2) = F ′ θ ′ ( x (2) ) . The networks are trained with an objective function that minimizes the distance between embeddings obtained from different views of the same image:

where ℓ ( · , · ) is a loss function. The mean square error (MSE) of L 2 -normalized vectors as ℓ ( z (1) , z (2) ) = ∥ z (1) ∥ z (1) ∥ 2 -z (2) ∥ z (2) ∥ 2 ∥ 2 2 is usually used as the loss function [9]. This loss is also equivalent to the negative cosine similarity, up to a scale of 1 2 and an optimization irrelevant constant [9]. This architecture is also called Siamese Network [9], if F θ = F ′ θ ′ . Another variant distinguishes the networks into target network F ′ θ ′ and online network F θ , and updates the weight θ ′ of target network through exponential moving average (EMA) [8, 16] over θ of online network.

Feature collapse. While minimizing Eqn. 1, a trivial solution known as complete collapse could occur such that F θ ( x ) ≡ c , ∀ x ∈ D . Moreover, a weaker collapse condition called dimensional collapse can be easily arrived, for which the projected features collapse into a low-dimensional manifold. To express dimensional collapse more mathematically, we refer to dimensional collapse as the phenomenon that one or certain eigenvalues of the covariance matrix of feature vectors degenerate to 0. Therefore, we can determine the occurrence of dimensional collapse by observing the spectrum of the covariance matrix.

Whitening loss. To address the collapse problem, whitening loss [12] is proposed to minimize Eqn. 1, under the condition that embeddings from different views are whitened. Whitening loss provides theoretical guarantee in avoiding (dimensional) collapse, since the embedding is whitened with all axes decorrelated [12, 20]. Ermolov et al. [12] propose to whiten the mini-batch embedding Z ∈ R d × m using batch whitening (BW) [21, 35] and impose the loss on the whitened output ̂ Z ∈ R d × m , given the mini-batch inputs X with size of m , as follows:

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

where Σ = 1 m ZZ T is the covariance matrix of embedding 1 . Σ -1 2 is called the whitening matrix, and is calculated either by Cholesky decomposition in [12] or by eigen-decomposition in [20]. E.g. , zero-phase component analysis (ZCA) whitening [21] calculates Σ -1 2 = U Λ -1 2 U T , where Λ = diag ( λ 1 , . . . , λ d ) and U = [ u 1 , ..., u d ] are the eigenvalues and associated eigenvectors of Σ , i.e. , U Λ U T = Σ . One intriguing result shown in [40] is that hard whitening can avoid collapse by only constraining the embedding Z to be full-rank, but not whitened.

We note that the whitening transformation is a function over embedding Z during forward pass, and modulates the spectrum of embedding Z implicitly during backward pass when minimizing MSE loss imposed on the whitened output. This raises questions whether there are other functions over embedding Z can avoid collapse? If yes, how the function affects the spectrum of embedding Z ?

3.2 Spectral Transformation

In this section, we extend the whitening transformation to spectral transformation, a more general view to characterize the modulation on the spectrum of embedding, and empirically investigate the interaction between the spectrum of the covariance matrix of ̂ Z and collapse of the SSL model.

)

Definition 1. ( Spectral Transformation ) Given any one-variable mapping function g ( · ) in the definition domain λ ( Z ) = { λ 1 , λ 2 , . . . , λ d } , spectral transformation (ST) maps the spectrum λ ( Z to g ( λ ( Z )) = { g ( λ 1 ) , g ( λ 2 ) , . . . , g ( λ d ) } . Accordingly, for a d × d real symmetric matrix Σ , spectral transformation g ( · ) on Σ = d ∑ i =1 λ i u i u T i is defined as g (Σ) = d ∑ i =1 g ( λ i ) u i u T i . We denote g (Λ) = diag ( g ( λ ( Z ))) , and Φ ST = g (Σ) = U g (Λ) U T is the transformation matrix.

The output of spectral transformation is calculated by ̂ Z = Φ ST Z = U g (Λ) U T Z . The covariance matrix of ̂ Z is:

Based on this formula, the essence of spectral transformation is mapping the spectrum λ ( Z ) = { λ 1 , λ 2 , . . . , λ d } to λ ( ̂ Z ) = { λ 1 g 2 ( λ 1 ) , λ 2 g 2 ( λ 2 ) , . . . , λ d g 2 ( λ d ) } .

ST using power functions. Whitening is a special instance of spectral transformation, where g ( · ) is a power function g ( λ ) = λ -1 2 . We further study the mechanism of this power transformation, where we consider a more general transformation g ( λ ) = λ -p , p ∈ ( -∞ , + ∞ ) for ST. Based on Definition. 1, this general power transformation is mapping the spectrum λ ( Z ) to λ ( ̂ Z ) = { λ 1 1 -2 p , λ 2 1 -2 p , . . . , λ d 1 -2 p } , e.g. , λ ( ̂ Z ) = { 1 , 1 , . . . , 1 } when using whitening with p = 1 2 .

We first conduct experiments on the 2D dataset with varying p and visualize outputs of the toy models in Figure 2(a). We observe the toy model seems to perform well to avoid collapse although the trans-

GLYPH(cmap:d835) = 1.50

GLYPH(cmap:d835) = 0.55

GLYPH(cmap:d835) = 0.50

GLYPH(cmap:d835) = 0.45

GLYPH(cmap:d835) = 0.40

GLYPH(cmap:d835) = 0.30

(a)





























8

6

4

2

0



1





1

(b)

formed output is not ideally whitened, when p is in the neighborhood of 0 . 5 , e.g. 0 . 45 ∼ 0 . 55 . But when p gradually deviates from 0 . 5 , collapse occurs. We then conduct experiments on real-world datasets to confirm these phenomena. The results shown in Figure 2(b) are consistent with the above phenomena. When p is set to 0 . 45 or 0 . 55 , the model remains high evaluation performance as the one of p = 0 . 5 . When p is in the neighborhood of 0 . 5 , the transformed output has a well-conditioned spectrum that each eigenvalue approaches 1. When p deviates from 0 . 5 to a certain extent, the

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

spectrum of the transformed output is not well-conditioned, which is closely related to collapse of the embedding. Therefore, we empirically show that if a certain ST could obtain a well-conditioned spectrum of transformed output, collapse could be avoided.

Based on the above results, we empirically observe that the spectral transformation g ( λ ) = λ -p with p around 0.5 can avoid collapse. Therefore, we can design new algorithms based on our framework to avoid collapse.

3.3 Implicit Spectral Transformation using Newton's Iteration

One problem of ST using power functions g ( λ ) = λ -p ( p is around 0.5) is the numerical instability, when calculating eigenvalues λ and eigenvectors U using eigen-decomposition if the covariance matrix is ill-conditioned [31]. We provide detailed experiments and analysis in supplementary materials to confirm the existence of this problem in SSL.

Naturally, if we can implement a spectral transformation that can modulate the spectrum without explicitly calculating λ or U , this problem can be solved. Indeed, we note an approximate whitening method by using Newton's iteration, called iterative normalization (IterNorm) [22], is proposed to address the numerical problem of batch whitening in supervised learning. Specifically, given the centralized embedding Z , iteration number T and the trace-normalized covariance matrix Σ N = Σ /tr (Σ) , it performs Newton's iteration as follows.

The whitening matrix Σ -1 2 is approximated by Φ T = P T / √ tr (Σ) and we have the whitened output ̂ Z = Φ T Z . When T → + ∞ , Φ T → Σ -1 2 and the covariance matrix of ̂ Z will be an identity matrix. Here, we theoretically show that IterNorm is also an instance of spectral transformation as follows.

Theorem 1. Define one-variable iterative function f T ( x ) , satisfying

The mapping function of IterNorm is

Without calculating λ or U , IterNorm implicitly maps ∀ λ i ∈ λ ( Z ) to ̂ λ i = λ i tr (Σ) f T 2 ( λ i tr (Σ) ) .

The proof is shown in supplementary materials . For simplicity, we define the T-whitening function of IterNorm h T ( x ) = xf T 2 ( x ) , which obtains the spectrum of transformed output. Based on the fact that the covariance matrix of transformed output will be identity when T of IterNorm increases to infinity [3], we thus have

Different iteration numbers T of IterNorm imply different T-whitening functions h T ( · ) . It is interesting to analyze the characteristics of h T ( · ) .

Proposition 1. Given x ∈ (0 , 1) , ∀ T ∈ N we have h T ( x ) ∈ (0 , 1) and h ′ T ( x ) > 0 .

The proof of Proposition 1 is shown in supplementary materials . Proposition 1 states h T ( x ) is a monotone increasing function for x ∈ (0 , 1) and its range is also in (0 , 1) . Since λ i tr (Σ) ∈ (0 , 1) , ∀ λ i > 0 , we have

Formula 5 indicates that IterNorm maps all non-zero eigenvalues to (0 , 1) and preserves monotonicity. Proposition 2. Given x ∈ (0 , 1) , ∀ T ∈ N , we have h T +1 ( x ) > h T ( x ) .

The proof of Proposition 2 is shown in supplementary materials . Proposition 2 indicates that IterNorm gradually stretches the eigenvalues towards one as the iteration number T increases. This property of IterNorm theoretically shows that the spectrum of ̂ Z will have better condition if we use a larger iteration number T of IterNorm.

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

In summary, our analyses theoretically show that IterNorm gradually stretches the eigenvalues towards one as the iteration number T increases, and the smaller the eigenvalue is, the larger T is required to approach one.

4 Iterative Normalization with Trace Loss

It is expected that IterNorm, as a kind of spectral transformation, can avoid collapse and obtain good performance in SSL, due to its benefits in approximating whitening for supervised learning [22]. However, we empirically observe that IterNorm suffers severe dimensional collapse and mostly fails to train the model in SSL (we postpone the details in Section 4.2.). Based on the analyses in Section 3.2 and 3.3, we propose a simple solution by adding an extra penalty named trace loss on the transformed output ̂ Z by IterNorm to ensure a well-conditioned spectrum. It is clear that the sum of eigenvalues of Σ ̂ Z is less than or equal to d , we thus propose a trace loss that encourages the trace of Σ ̂ Z to be its maximum d , when d ≤ m . In particular, we design a new method called IterNorm with trace loss (INTL) for optimizing the SSL model as 2 :

where Z = F θ ( · ) and ̂ Z = IterNorm ( Z ) . Eqn. 6 can be viewed as an optimization problem over θ to encourage the trace of ̂ Z to be d .

4.1 Theoretical Analysis

In this section, we theoretically prove that INTL can avoid collapse, and INTL modulates the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization.

Note that Σ ̂ Z can be expressed using the T-whitening function h T ( · ) as Σ ̂ Z = d ∑ i =1 h T ( x i ) u i u T i ,

d

∑

i

=1

problem over θ (Eqn. 6) can be transformed as the following optimization problem over x (Eqn. 7) without changing the optimal value (please see supplementary materials for the details of derivation):

where u ji is the j -th elements of vector u i . In this formulation, we can prove that our proposed INTL can theoretically avoid collapse, as long as the iteration number T of IterNorm is larger than zero.

Theorem 2. Let x ∈ [0 , 1] d , ∀ T ∈ N + , INTL ( x ) shown in Eqn. 7 is a strictly convex function. x ∗ = [ 1 d , · · · , 1 d ] T is the unique minimum point as well as the optimal solution to INTL ( x ) .

The proof is shown in supplementary materials . Based on Theorem 2, INTL promotes the equality of all eigenvalues of the covariance matrix of embedding Z during the course of optimization, which provides a theoretical guarantee to avoid dimensional collapse.

Connection to hard whitening. Hard whitening methods, like W-MSE [12] and shuffle-DBN [20], design a whitening transformation over each view and minimize the distances between the whitened outputs from different views. This mechanism encourages the covariance matrix of embedding to be full-rank [40]. Our INTL designs an approximated whitening transformation using IterNorm and imposes an additional trace loss penalty on the (approximately) whitened output. This encourages the covariance matrix of embedding having equal eigenvalues.

Connection to soft whitening. Soft whitening methods, like Barlow-Twins [44] and VICReg [2] directly impose a whitening penalty as a regularization on the embedding . This encourages the covariance matrix of the embedding to be identity (with a fixed scalar γ , e.g. , γ I ). Our INTL imposes

where

x

i

=

λ

i

/tr

(Σ)

0

and

x

i

= 1

. When the range of

F

θ

(

·

)

is wide enough, the optimization

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312





































(a)































(b)





























(c)



























(d)

the penalty on the transformed output, but can be viewed as implicitly encouraging the covariance matrix of the embedding to be identity with a free scalar ( i.e. , having equal eigenvalues).

Intuitively, INTL provides the equal-eigenvalues constraint on the covariance matrix of embedding , which is a stronger constraint than hard whitening (the full-rank constraint), but a weaker constraint than soft whitening (the whitening constraint). This preliminary but new comparison provides a new way to understand the whitening loss in SSL.

4.2 Empirical Analysis

In this section, we empirically show that IterNorm-only fails to avoid collapse, but IterNorm with trace loss can well avoid collapse.

IterNorm fails to avoid collapse. In theory, IterNorm can map all non-zero eigenvalues to approach one, with a large enough T . In practice, it usually uses a fixed T , and it is very likely to encounter small eigenvalues during training. In this case, IterNorm cannot ensure the transformed output has a well-conditioned spectrum (Figure 3(a)), which potentially results in dimensional collapse. One may use a large T , however, IterNorm will encounter numeric divergence upon further increasing the iteration number T , even though it has converged. E.g. , IterNorm suffers from numeric divergence in Figure 3(d) when using T = 9 , since the maximum eigenvalue of whitened output is around 10 7 , significantly large than 1 (we attribute to the numeric divergence, since this result goes against Proposition 1 and 2, and we further validate it by monitoring the transformed output). It is noteworthy that when T ≥ 11 , the loss values are all NAN , making the model unable to be trained. These problems make IterNorm difficult to avoid dimensional collapse in practice.

The magic of trace loss for IterNorm. IterNorm with trace loss works significantly different from IterNorm-only. Our experimental results (Figure 3(b)) empirically show that INTL avoid dimensional collapse, which is consistent with our Theorem 2. INTL encourages the equality of all eigenvalues of the covariance matrix of embedding Z and achieve good evaluation performance (Figure 3(c)) even when the iteration number T is only 1. This equal-eigenvalues optimal solution is derived from the combination of IterNorm and trace loss . Benefiting from trace loss, IterNorm can obtain well-conditioned spectra for the transformed output during training (Figure 3(a)), which helps to avoid collapse.

5 Experiments on Standard SSL Benchmark

In this section, we conduct experiments on standard SSL benchmarks to validate the effectiveness of our proposed INTL. We first evaluate the performance of INTL for classification on ImageNet [11], CIFAR-10/100 [25] and ImageNet-100 [37]. Then we evaluate the effectiveness in transfer learning, for a pre-trained model using INTL. We provide details of implementation and training protocol as well as computational overhead and the full PyTorch-style algorithm in supplementary materials .

5.1 Evaluation for Classification

Evaluation on ImageNet. We train our INTL using ResNet-50 backbone and evaluate the performance using the common linear evaluation protocol on ImageNet. The results are shown in Table 1. Our INTL achieves a top-1 accuracy of 76 . 6% that exceeds the performance of the supervised baseline [7] and other SSL methods. Moreover, this result is obtained under a batch size of only 256,

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

achieving one goal of SSL community that seeks for obtaining good performance with a small batch size.

Semi-supervised training on ImageNet. Furthermore, we fine-tune our pre-trained INTL model on a subset of ImageNet. We use subsets of size 1% and 10% using the same split as SimCLR. The semi-supervised results obtained on the ImageNet validation set are also reported in Table 1. It shows that INTL outperforms other baselines with a significant margin.

Evaluation on small and medium size datasets. In order to further test the generality of INTL, we also provide the linear evaluation results of INTL on CIFAR-10/100 [25] and ImageNet-100 [37] with ResNet-18 as the backbone. We strictly follow the experimental settings in solo-learn [10] for these datasets. As shown in Table 2, INTL achieves a top-1 accuracy of 92 . 60% on CIFAR-10, 70 . 88% on CIFAR-100 and 81 . 68% on ImageNet-100 which is on par with or exceeds the state-of-the-art methods reproduced by solo-learn. Meanwhile, INTL outperforms other baselines with a significant margin when using 5-nearest neighbors classifier which also indicates that INTL has learned good representations.

5.2 Transfer to Downstream Tasks

We examine the representation quality by transferring our pre-trained model to other tasks, including COCO[27] object detection and instance segmentation. We use the baseline of the detection codebase from MoCo [18] for INTL. The results of baselines shown in Table 3 are mostly inherited from [9]. We observe that INTL performs much better than other state-of-the-art approaches on COCO object detection and instance segmentation, which shows the great potential of INTL in transferring to downstream tasks.

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

5.3 Ablation Study

Batch size. Most SSL methods, including certain whitening-based methods, are known to be sensitive to batch sizes, e.g. SimCLR [7], SwAV [5] and WMSE [12] all require a large batch size (e.g. 4096) to work well. We then test the robustness of INTL to batch sizes. We train INTL on ImageNet for 100 epochs with various batch sizes ranging from 32 to

1024. As shown in Table. 4, even if the batch size is as low as 32 or 64, INTL still maintains good performance. At the same time, when the batch size increases, the accuracy of INTL is also improved. These results indicate that INTL has good robustness to batch sizes and can adapt to various scenarios that constrain the training batch size.

Embedding dimension. Embedding dimension, the output dimension of the projection, is also a key element for most self-supervised learning methods, which may have a significant impact on training results. As illustrated in [44], Barlow Twins is very sensitive to embedding dimension and it requires a large dimension (e.g. 8192 or 16384) to work well. We also test the robustness of INTL to embedding dimensions. Following the setup of [7] and [44], we train INTL on ImageNet for 300 epochs with the dimension ranging from 64 to 16384. As shown in Figure. 4, even when the embedding dimension is low as 64 or 128, INTL still achieves good results. These results show that INTL also has strong robustness to embedding dimensions.

6 Conclusion and Limitation

In this paper, we proposed spectral transformation (ST) framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. Our proposed IterNorm with trace loss (INTL) is well-motivated, theoretically demonstrated, and empirically validated in avoiding dimension collapse. Comprehensive experiments have shown the merits of INTL for achieving state-of-the-art performance for SSL in practice. We showed that INTL provides the equal-eigenvalues constraint on the covariance matrix of embedding , which is a stronger constraint than hard whitening (the full-rank constraint), but a weaker constraint than soft whitening (the whitening constraint). This preliminary but new results provides a potential way to understand and compare SSL methods.

Limitation. Our work only explores the mechanism of ST using power function and Newton's iteration for SSL. As a general concept, we believe that more functions in our ST framework can be designed to avoid collapse in the future. Besides, our theoretical work mainly revolves around the ST using Newton's iteration, without providing theoretical analysis for more general ST. There are still mysteries about modulation of the spectrum in more general ST during backpropagation.











































376

References

Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual 377 information across views. In: NeurIPS (2019) 378

Bardes, A., Ponce, J., LeCun, Y.: Vicreg: Variance-invariance-covariance regularization for 379 self-supervised learning. In: ICLR (2022) 380

Bini, D.A., Higham, N.J., Meini, B.: Algorithms for the matrix pth root. Numerical Algorithms 381 (2005) 382

Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of 383 visual features. In: ECCV (2018) 384

385

386

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020)

387

388

Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)

Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning 389 of visual representations. In: ICML (2020) 390

Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive 391 learning. arXiv preprint arXiv:2003.04297 (2020) 392

Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR (2021) 393

da Costa, V.G.T., Fini, E., Nabi, M., Sebe, N., Ricci, E.: solo-learn: A library of self-supervised 394 methods for visual representation learning. Journal of Machine Learning Research 23 (56), 1-6 395 (2022), http://jmlr.org/papers/v23/21-1155.html 396

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale 397 Hierarchical Image Database. In: CVPR (2009) 398

Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N.: Whitening for self-supervised representation 399 learning. In: ICML (2021) 400

Fetterman, A., Albrecht, J.: Understanding self-supervised and contrastive learning with 'boot401 strap your own latent' (byol). Technical Report (2020) 402

Garrido, Q., Chen, Y., Bardes, A., Najman, L., LeCun, Y.: On the duality between contrastive 403 and non-contrastive self-supervised learning. In: ICLR (2023) 404

Ghosh, A., Mondal, A.K., Agrawal, K.K., Richards, B.A.: Investigating power laws in deep 405 representation learning. arXiv preprint arXiv:2202.05808 (2022) 406

Grill, J.B., Strub, F., Altch´ e, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., 407 Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: 408 Bootstrap your own latent - a new approach to self-supervised learning. In: NeuraIPS (2020) 409

He, B., Ozay, M.: Exploring the gap between collapsed and whitened features in self-supervised 410 learning. In: ICML (2022) 411

He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual 412 representation learning. In: CVPR (2020) 413

Henaff, O.: Data-efficient image recognition with contrastive predictive coding. In: ICML 414 (2020) 415

Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H.: On feature decorrelation in self416 supervised learning. In: ICCV (2021) 417

Huang, L., Yang, D., Lang, B., Deng, J.: Decorrelated batch normalization. In: CVPR (2018) 418

Huang, L., Zhou, Y., Zhu, F., Liu, L., Shao, L.: Iterative normalization: Beyond standardization 419 towards efficient whitening. In: CVPR (2019) 420

Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., Makedon, F.: A survey on contrastive 421 self-supervised learning. arXiv preprint arXiv:2011.00362 (2020) 422

Jing, L., Vincent, P., LeCun, Y., Tian, Y.: Understanding dimensional collapse in contrastive 423 self-supervised learning. In: ICLR (2022) 424

Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep. (2009) 425

Li, J., Zhou, P., Xiong, C., Hoi, S.C.: Prototypical contrastive learning of unsupervised repre426 sentations. In: ICLR (2021) 427

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, L.: 428 Microsoft coco: Common objects in context. In: ECCV (2014) 429

430

431

Liu, Y., Pan, S., Jin, M., Zhou, C., Xia, F., Yu, P.S.: Graph self-supervised learning: A survey. arXiv e-prints pp. arXiv-2103 (2021)

432

433

Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., 434 Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without 435 supervision. arXiv preprint arXiv:2304.07193 (2023) 436

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., 437 Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning 438 library. Advances in neural information processing systems 32 (2019) 439

Ranasinghe, K., Naseer, M., Khan, S., Khan, F.S., Ryoo, M.: Self-supervised video transformer. 440 In: CVPR (2022) 441

Richemond, P.H., Grill, J.B., Altch´ e, F., Tallec, C., Strub, F., Brock, A., Smith, S., De, 442 S., Pascanu, R., Piot, B., et al.: Byol works even without batch statistics. arXiv preprint 443 arXiv:2010.10241 (2020) 444

Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., Khandeparkar, H.: A theoretical analysis of 445 contrastive unsupervised representation learning. In: ICML (2019) 446

Siarohin, A., Sangineto, E., Sebe, N.: Whitening and coloring batch transform for gans. In: 447 ICLR (2019) 448

Tao, C., Wang, H., Zhu, X., Dong, J., Song, S., Huang, G., Dai, J.: Exploring the equivalence of 449 siamese self-supervised learning via A unified gradient framework. In: CVPR (2022) 450

Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: European conference on 451 computer vision (2020) 452

Tian, Y., Chen, X., Ganguli, S.: Understanding self-supervised learning dynamics without 453 contrastive pairs. In: ICML (2021) 454

Tian, Y., Yu, L., Chen, X., Ganguli, S.: Understanding self-supervised learning with dual deep 455 networks. CoRR abs/2010.00578 (2020) 456

Weng, X., Huang, L., Zhao, L., Anwer, R.M., Khan, S., Khan, F.: An investigation into 457 whitening loss for self-supervised learning. In: NeurIPS (2022) 458

Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance 459 discrimination. In: CVPR (2018) 460

Ye, C., Evanusa, M., He, H., Mitrokhin, A., Goldstein, T., Yorke, J.A., Fermuller, C., Aloimonos, 461 Y.: Network deconvolution. In: ICLR (2020) 462

Ye, M., Zhang, X., Yuen, P.C., Chang, S.F.: Unsupervised embedding learning via invariant and 463 spreading instance feature. In: CVPR (2019) 464

Zbontar, J., Jing, L., Misra, I., Lecun, Y ., Deny, S.: Barlow twins: Self-supervised learning via 465 redundancy reduction. In: ICML (2021) 466

Zhang, C., Zhang, K., Zhang, C., Pham, T.X., Yoo, C.D., Kweon, I.S.: How does simsiam avoid 467 collapse without negative samples? a unified understanding with self-supervised contrastive 468 learning. In: ICLR (2022) 469

Zhang, H., Wu, Q., Yan, J., Wipf, D., Yu, P.S.: From canonical correlation analysis to self470 supervised graph neural networks. In: NeurIPS (2021) 471

Zhang, S., Zhu, F., Yan, J., Zhao, R., Yang, X.: Zero-CL: Instance and feature decorrelation for 472 negative-free symmetric contrastive learning. In: ICLR (2022) 473

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFeature collapse is a major problem in contrastive-base self-supervised learning. To address this issue, the paper introduces the spectral transformation framework, which aims to mitigate the aforementioned problem. Generally speaking, the spectral transformation expands upon the extensively employed whitening transformation discussed in the literature and treats it as a special case. Furthermore, the paper proposes the IterNorm approach featuring trace loss to enhance the performance of self-supervised models. The experimental results validate the effectiveness of the IterNorm with trace loss (INTL) on various widely-used SSL-evaluation benchmarks, including but not limited to linear evaluation, classification, and object detection.\n\nSTRENGTHS:\n1)\tThe paper addresses a fundamental challenge in the context of contrastive-based self-supervised learning, and proposes an elegant method with theoretical guarantees. Specifically, the proposed framework extends the widely used whitening operation discussed in prior literature, treating it as a special instance, thereby enhancing the theoretical contribution of this work.\n\n2)\t The experimental results are promising. The proposed method is evaluated on several SSL-evaluation benchmarks, including ImageNet linear classification (achieving 76.6% accuracy on ResNet-50), COCO object detection (achieving a score of 41.2 on ResNet50-C4), and significantly surpasses the previous state-of-the-art methods.\n\n3)\tThe presentation of the paper is good, with a clear elaboration of the motivation and contribution of the proposed method.\n\nWEAKNESSES:\n1)\tThe comparison between the proposed method and baseline is unfair. Firstly, the proposed framework employs a 3-layer projection head with a substantially larger dimension (8192), which significantly increases both computation and memory costs. This discrepancy makes it hard to conduct a fair comparison between the proposed method and the baseline approach. \n\n2)\tAs mentioned earlier, the proposed method incurs nearly twice the computation time per epoch and peak memory per GPU compared to the naive contrastive baseline method MoCo/MoCo-v2.\n\n3)\tIt is worth noting that the multi-crop strategy employed in the paper is highly similar to that used in [a], including the crop-nums and crop-sizes. It is unclear why this was not cited in the paper.\n\n4)\tIt should be noted that the multi-crop strategy employed in [a] can also enhance the detection and classification results. Therefore, the absence of this information in the comparison makes it difficult to conduct a fair evaluation.\n\n[a] Wang, X., & Qi, G. J. (2022). Contrastive learning with stronger augmentations. IEEE transactions on pattern analysis and machine intelligence, 45(5), 5549-5560.\n\nQUESTIONS:\n1)\tThe experiments conducted in the paper are based solely on the ResNet-50 network architecture, it would be beneficial if the authors could provide results based on transformer backbones, given their widespread usage in the self-supervised learning domain.\n \n2)\tAs a theoretical-style paper, it is important to have a fair and direct comparison between the proposed method and the baseline approach. However, the utilization of several tricks and unfair experimental settings in the paper makes it challenging to determine the actual degree of improvement gained from the original technical contribution. Furthermore, such experimental settings make it hard to follow the paper's methodology.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor self-supervised learning, this paper proposes a spectral transformation (ST) framework to modulate embedding, seeking functions other than whitening that can avoid dimensional collapse. The authors propose IterNorm with trace loss and provide a lot of theoretical and experimental analysis. The results show its effectiveness and advancement.\n\nSTRENGTHS:\n1. This paper provides intuitive experimental verification, extending the whitening transformation to a more general spectral transformation.\n2. For the proposed IterNorm with trace loss, this paper conducts rigorous theoretical analysis and practice, showing that it is effective for avoiding collapse.\n3. The proposed method can achieve good results without relying on large batches, which can be comparable to supervised performance under regular settings.\n\nWEAKNESSES:\n1.\tThe motivation for the proposed IterNorm with trace loss is unclear. The authors empirically observe that IterNorm suffers from severe dimensionality collapse, but do not state the limitations of existing SSL methods in this case.\n2.\tFor the ablation study of batch size, Table 4 shows that the performance of the proposed method increases significantly as the batch size increases, indicating that the method is also sensitive to the batch size. In addition, SimCLR, SwAV and other methods require a batch size of 4096. Table 4 should provide the corresponding results with a batch size of 4096 and the sensitivity of other methods to the batch size.\n3.\tAmong the existing SSL methods compared in the experimental part, few new methods since 2022 are covered. More state-of-the-art method comparisons need to be added.\n\nQUESTIONS:\nPlease see above weaknesses.\n\nLIMITATIONS:\nThe authors have explained some of the limitations in the paper, and others can be seen in weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a spectral transformation for redundancy-reduction-based (a.k.a. whitening-based) self-supervised learning (SSL).\nSpectrum-domain modulation is helpful in preventing collapsing caused by representations' rank deficiency.\nSince the whitening operation can be seen as a square-root transformation on spectrum, it can be generalized to other\npower-function transformation.\nTo avoid numerical instability of matrix decomposition in computing spectra, an approximation based on Newton’s iteration method IterNorm\nis proposed. While IterNorm by itself suffers from dimensional collapse, its combination with trace loss, the term to encourage the spectrum to have equal-eigenvalue distribution, works well to avoid collapse.\nExperiments on ImageNet, CIFAR-10/100, and COCO show that the proposed method outperforms existing state-of-the-art methods.\n\nSTRENGTHS:\n- The idea of spectrum-domain modulation seems intuitive to avoid dimensional collapse caused by rank deficiency of representations.\n- State-of-the-art SSL performances using ResNet50 were achieved.\n- Especially excellent performance avoiding collapse in small batch sizes is practical for low-resource training.\n\nWEAKNESSES:\n- The difference between existing whitening-based methods and power-function modulation with p = 0.5: Sec 3.2 empirically shows that p == 0.5 (== 1/2) is a good value. At the same time, p4 l 170 says, \"whitening is a special instance of spectral transformation, where g(·) is a power function g(λ) = λ^{− 1/2}\". This confused me with what difference this generalization made from usual whitening.\n\n- Trace loss + BarlowTwins/VICReg?: It is said that IterNorm works well only when combined with trace loss.\nThis naturally implies that the source of success is trace loss itself rather than IterNorm. Is it possible to combine trace loss with existing whitening-based methods part from IterNorm?\n\nQUESTIONS:\n- p4 l151 \"a function over embedding Z during forward pass, and modulates the spectrum of embedding Z implicitly during backward pass when minimizing MSE loss\": I could not follow this part; what kind of modulation is done in the backward path? More concrete explanation is preferable.\n\nLIMITATIONS:\nLimitations or social impact are not discussed, but I do not have particular concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the dimension collapse problem in image self-supervised learning. Previously, people use image space data augmentation/momentum/stop gradient and so on to address this problem. This paper proposes to address the problem via spectrum transformation (ST) in the feature space instead (aka balancing the eigenvalue of covariance matrix while keeping eigenvectors the same in mini-batches).\n\nSTRENGTHS:\n1.\tIts ST generalized previous whitening approach (aka make the covariance matrix to be identity), and extends it to power functions (aka p \\in R instead of p = 0.5).  \n2.\tFor the calculation of whitening matrix, the paper proposes a less expensive approach to calculate (IterNorm). And the paper analyzes how the singular values will change using IterNorm. But to really make things work, the paper adds a trace loss (INTL). \n3.\tThe paper shows INTL reaches SOTA performance on image classification and object detection tasks. And INTL is less sensitive to batch size.\n\nWEAKNESSES:\n1. The **major concern** for this paper is that, it resembles quite a few similarities in the key ideas to a recent AAAI work: \"Spectral Feature Augmentation for Graph Contrastive Learning and Beyond\" (https://arxiv.org/abs/2212.01026), but the AAAI work has not been cited or discussed in this work, making the contribution of this work a bit incremental:\n\na. The motivations are similar: deal with feature collapse/dimension collapse via change feature space’s spectrum.\n\nb. Similar pipeline, the AAAI paper apply SFA after encoder and before projection, this work applies ST in the end of the pipeline. the AAAI work injects noise for better augmentation while the paper does not. This paper finally needs to add an additional trace loss to make ST work, which may play a similar role to the injected noise. \n\nc. Iterative normalization to calculate \\Sigma_{-0.5} is like appendix C8 algorithm 2. But with a slightly different formulation. The AAAI paper calculates \\Sigma_{-0.5} and \\Sigma_{0.5} at the same time. This work only calculates \\Sigma_{0.5} with a slightly different iteration function.\n\nd. This work’s power functions for ST are like AAAI’s MaxExp(F) and Matrix Square Root, but this work’s approach does not inject a Gaussian noise.\n\ne.Both experiments on SSL on ImageNet. The AAAI work explored both graphs and images. This work focuses more on images, so it also explores CIFAR, and also object detection tasks. But AAAI work deals with contrastive SSL (InfoNCE) while this work uses a non-contrastive approach (normalized MSE loss).\n\n2. It states to explore other possible spectrum transformation approaches but only extends from whitening (p=0.5) to power (p \\in R) with a limited range of working p around 0.5. This does not validate its statement to generalize ST.\n\n3. Directly using ST (IterNorm) does not work, and finally needs to add trace norm (INTL). It is not shown that if only using the implicit trace norm without explicit ST, whether the performance will still be good. This does not validate its need to do explicit ST.\n\nQUESTIONS:\nThe notation in Figure 1 is not clear: are X, Z matrices? The notation of Z in Figure 1 and Equation are inconsistent. It would also be better to add notation section since there are many similar symbols.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper addresses the dimensional collapse problem in self-supervised learning and proposes a framework to moderate the spectrum of embedding. Besides, a new spectral transformation variant, called IterNorm with trace loss (INTL), is proposed that can avoid collapse and modulate the spectrum of embedding towards an equal-eigenvalue distribution. Experiments on various tasks verify the effectiveness of the proposed method.

STRENGTHS:
- The proposed method looks technically sound. 

- Extensive experiments have been conducted to verify the effectiveness by comparing with the existing methods in the field. 

- The presentation is great.

WEAKNESSES:
This looks like a good paper. I do not have much to complain about (possibly due to I do not work in this direction). I just have two minor questions. 

- How do you balance the MSE loss and IterNorm trace loss? Are there coefficients in the two loss terms? If there are, how do you set the coefficient values? Do they keep fixed across all the datasets? 

- The proposed method INTL is based on the empirical finding that when p is in the neighborhood of 0.5, e.g. 0.45 ∼ 0.55, the model can avoid collapse. Is there any intuitive explanation or theoretic analysis for this phenomenon?

QUESTIONS:
See the weakness above.

LIMITATIONS:
See the weakness above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the dimension collapse problem in self-supervised learning. The authors propose spectral transformation which can be served as an alternative for the whitening function in avoiding dimensional collapse. Further, they propose a new instance of ST, called IterNorm with trace loss (INTL), and prove that INTL can avoid collapse. Results on several SSL benchmarks show promising improvements over previous methods.\n\nSTRENGTHS:\n- The paper is well-organized with a clear story and justification.\n- The paper aims at analyzing the dimension collapse problem and prove the previous whitening function is a special case of spectrum transformation (ST).\n- The authors claim that the numerical instability can be solved if a spectral transformation can modulate the spectrum without explicitly calculating λ or U. Then, they propose Iterative Normalization with Trace Loss as a solution.\n-  The performance on Imagenet, CIFAR and COCo shows improvements over previous methods such as Btwins, and VICreg.\n\nWEAKNESSES:\n1. Comparing methods in the Experiment section:\nAs the proposed method is claimed to be the general form of whitening function in SSL, I am wondering why the authors did not put the performance comparison against whitening loss [20, 40]. As [20, 40] have demonstrated the performance gains over Btwins, it is hard to tell how the INTL it better than [20, 40].\n\n2. Concerns about novelty. \nIn my view, section 3.2 Spectral Transformation proves the effectiveness of Whitening. The general form of ST is $g(\\lambda) = \\lambda^{-p}$ and the authors claim that \n> it seems to perform well to avoid collapse although the transformed output is not ideally whitened when p is in the neighborhood of 0.5, e.g. 0.45 ∼ 0.55. But when p gradually deviates from 0.5, collapse occurs.\"\nThe whitening function is the case where $p=0.5$.\n\nIn section 3.3, the iterative normalization has also been explored by [40] in https://github.com/PatrickHua/FeatureDecorrelationSSL/blob/main/models/utils/iterative_normalization.py\n\nThe major contribution of this paper in my view is to propose a trace loss in iterative normalization to avoid dimension collapse. Yet, as the quantitative comparison to [20,40] is missing, it is hard to evaluate the contribution.\n\nQUESTIONS:\nPlease refer to the weakness\n\nLIMITATIONS:\nI think the authors well address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}