Title: Explore Positive Noise in Deep Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: Deep learning

Positive Noise
Abstract: In computer vision, noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different tasks like image classification and transfer learning. However, this paper aims to rethink whether the conventional proposition always holds. We demonstrate that specific noise can boost the performance of various deep architectures under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the complexity of the task. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieve an unprecedented top 1 accuracy over 95$\%$ on ImageNet. Both theoretical analysis and empirical evidence have confirmed the presence of positive noise, which can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us to utilize noise rather than suppress noise.

Abstract

In computer vision, noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different tasks like image classification and transfer learning. However, this paper aims to rethink whether the conventional proposition always holds. We demonstrate that specific noise can boost the performance of various deep architectures under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the complexity of the task. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieve an unprecedented top 1 accuracy over 95 % on ImageNet. Both theoretical analysis and empirical evidence have confirmed that the presence of positive noise, can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us to utilize noise rather than suppress noise.

1 Introduction

Noise, conventionally regarded as a hurdle in machine learning and deep learning tasks, is universal 22 and unavoidable due to various reasons, e.g., environmental factors, instrumental calibration, and 23 human activities [23] [37]. In computer vision, noise can be generated from different phases: (1) 24 Image Acquisition: Noise can arise from a camera sensor or other imaging device [33]. For example, 25 electronic or thermal noise in the camera sensor can result in random pixel values or color variations 26 that can be visible in the captured image. (2) Image Preprocessing: Noise can be introduced during 27 preprocessing steps such as image resizing, filtering, or color space conversion [1]. For example, 28 resizing an image can introduce aliasing artifacts, while filtering an image can result in the loss of 29 detail and texture. (3) Feature Extraction: Feature extraction algorithms can be sensitive to noise 30 in the input image, which can result in inaccurate or inconsistent feature representations [2]. For 31 example, edge detection algorithms can be affected by noise in the image, resulting in false positives 32 or negatives. (4) Algorithms: algorithms used for computer vision tasks, such as object detection or 33 image segmentation, can also be sensitive to noise in the input data [6]. Noise can cause the algorithm 34 to learn incorrect patterns or features, leading to poor performance. 35

Since noise is an unavoidable reality in engineering tasks, existing works usually make the assumption 36

that noise has a consistently negative impact on the current task [30] [24]. Nevertheless, is the above 37

assumption always valid? As such, it is crucial to address the question of whether noise can ever 38

have a positive influence on deep learning models. This work aims to provide a comprehensive 39 answer to this question, which is a pressing concern in the deep learning community. We recognize 40 that the imprecise definition of noise is a critical factor leading to the uncertainties surrounding the 41 identification and characterization of noise. To address these uncertainties, an in-depth analysis 42 of the task's complexity is imperative for arriving at a rigorous answer. By using the definition of 43 task entropy, it is possible to categorize noise into two distinct categories: positive noise (PN) and 44 harmful noise (HN). PN decreases the complexity of the task, while HN increases it, aligning with 45 the conventional understanding of noise. 46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

1.1 Scope and Contribution

Our work aims to investigate how various types of noise affect deep learning models. Specifically, the study focuses on three common types of noise, i.e., Gaussian noise, linear transform noise, and salt-and-pepper noise. Gaussian noise refers to random fluctuations that follow a Gaussian distribution in pixel values at the image level or latent representations in latent space [29]. Linear transforms, on the other hand, refer to affine elementary matrix transformations to the dataset of original images or latent representations, where the elementary matrix is row equivalent to an identity matrix [36]. Salt-and-pepper noise is a kind of image distortion that adds random black or white values at the image level or to the latent representations [7].

This paper analyzes the impact of these types of noise on the performance of deep learning models for image classification and domain adaptation tasks. Two popular model families, Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs), are considered in the study. Image classification is one of the most fundamental tasks in computer vision, where the goal is to predict the class label of an input image. Domain adaptation is a practically meaningful task where the training and test data come from different distributions, also known as different domains. By investigating the effects of different types of noise on ViTs and CNNs for typical deep learning tasks, the paper provides insights into the influences of noises on deep models. The findings presented in this paper hold practical significance for enhancing the performance of various types of deep learning models in real-world scenarios.

The contributions of this paper are summarized as follows:

We re-examined the conventional view that noise, by default, has a negative impact on deep learning models. Our theoretical analysis and experimental results show that noise can be a positive support for deep learning models and tasks.

We implemented extensive experiments with different deep models, such as CNNs and ViTs, and on different deep learning tasks. Empowered by positive noise, we achieved state-of-the-art (SOTA) results in all the experiments presented in this paper.

Instead of operating on the image level, our injecting noise operations are performed in the latent space. We theoretically analyze the difference between injecting noise on the image level and in the latent space.

The theory and framework of reducing task complexity via positive noise in this work can be applied to any deep learning architecture. There is great potential for exploring the application of positive noise in other deep-learning tasks beyond the image classification and domain adaptation tasks examined in this study.

1.2 Related Work

Positive Noise In fact, within the signal-processing society, it has been demonstrated that random 81 noise helps stochastic resonance improve the detection of weak signals [4]. Noises can have positive 82 support and contribute to less mean square error compared to the best linear unbiased estimator when 83 the mixing probability distribution is not in the extreme region [28]. Also, it has been reported that 84 noise could increase the model generalization in natural language processing (NLP) [27]. Recently, 85 the perturbation, a special case of positive noise, has been effectively utilized to implement self86 refinement in domain adaptation and achieved state-of-the-art performance [36]. The latest research 87 shows that by proactively adding specific noise to partial datasets, various tasks can benefit from the 88 positive noise [19]. Besides, noises are found to be able to boost brain power and be useful in many 89 neuroscience studies [21] [22]. 90

Deep Model Convolutional Neural Networks have been widely used for image classification, object 91 detection, and segmentation tasks, and have achieved impressive results [18][15]. However, these 92 networks have limitations in terms of their ability to capture long-range dependencies and extract 93 global features from images. Recently, Vision Transformers has been proposed as an alternative to 94 CNNs [13]. ViT relies on self-attention mechanisms and a transformer-based architecture to enable 95 global feature extraction and modeling of long-range dependencies in images [40]. The attention 96 mechanism allows the model to focus on the most informative features of the input image, while 97 the transformer architecture facilitates information exchange between different parts of the image. 98 ViT has demonstrated impressive performance on a range of image classification tasks and has the 99 potential to outperform traditional CNN-based approaches. However, ViT currently requires a large 100 number of parameters and training data to achieve state-of-the-art results, making it challenging to 101 implement in certain settings [45]. 102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

2 Preliminary

In information theory, the entropy [32] of a random variable x is defined as:

where p ( x ) is the distribution of the given variable x . And the mutual information (MI) of two random discrete variables ( x, y ) is denoted as [8]:

where D KL is the Kullback-Leibler divergence [16], and p ( x, y ) is the joint distribution. The conditional entropy is defined as:

The above definitions can be readily expanded to encompass continuous variables through the substitution of the sum operator with the integral symbol. In this work, the noise is denoted by ϵ if without any specific statement.

Before delving into the correlation between task and noise, it is imperative to address the initial crucial query of the mathematical measurement of a task T . With the assistance of information theory, the complexity associated with a given task T can be measured in terms of the entropy of T . Therefore, we can borrow the concepts of information entropy to explain the difficulty of the task. For example, a smaller H ( T ) means an easier task and vice versa.

Since the entropy of task T is formulated, it is not difficult to define the mutual information of task T and noise ϵ ,

Formally, if the noise can help reduce the complexity of the task, i.e., H ( T | ϵ ) < H ( T ) then the noise has positive support. Therefore, a noise ϵ is defined as positive noise (PN) when the noise satisfies MI ( T , ϵ ) > 0 . On the contrary, when MI ( T , ϵ ) ≤ 0 , the noise is considered as the conventional noise and named harmful noise (HN). The positive noise can be perceived as an augmentation of information gain brought by ϵ .

Moderate Model Assumption : The positive noise may not work for deep models with severe 124 problems. For example, the model is severely overfitting where models begin to memorize the 125 random fluctuations in the data instead of learning the underlying patterns. In that case, the presence 126 of positive noise will not have significant positive support in improving the models' performance. 127 Besides, when the models are corrupted under brute force attack, the positive noise also can not work. 128

129

Input (Images)

Predicted Classes

CNN (ResNet)

ViT

Inject Noise

conv

pool

conv

conv

conv

conv

conv

conv

conv

conv

Average Pool

FC

...

Norm

Multi-Head

Attention

Norm

MLP

1

×

Transformer Layer

Transformer Layer

...

FC

L

×

Transformer Layer

Class token

FC: Fully Connected Layer

Noise

Noise

Class 1

Class 2

Class 3

3 Methods

The idea of exploring the influence of noise on the deep models is straightforward. The framework is 130 depicted in Fig. 1. This is a universal framework where there are different options for deep models, 131 such as CNNs and ViTs. Through the simple operation of injecting noise into a randomly selected 132 layer, a model has the potential to gain additional information to reduce task complexity, thereby 133 improving its performance. It is sufficient to inject noise into a single layer instead of multiple layers 134 since it imposes a regularization on multiple layers simultaneously. 135

136

137

138

139

140

For a classification problem, the dataset ( X , Y ) can be regarded as samplings derived from D X , Y , where D X , Y is some unknown joint distribution of data points and labels from feasible space X and Y , i.e., ( X , Y ) ∼ D X , Y [31]. Hence, given a set of k data points X = { X 1 , X 2 , ..., X k } , the label set Y = { Y 1 , Y 2 , ..., Y k } is regarded as sampling from Y ∼ D Y|X . The complexity of T on dataset X is formulated as [19]:

The operation of adding noise at the image level can be formulated as: 141

142

While the operation of proactively injecting noise in the latent space can be formulated as:

Step ⋆ differs from the conventional definition of conditional entropy, as our method injects the noise 143 into the latent representations instead of the original images. The Gaussian noise is additive, the 144 linear transform noise is also additive, and the salt-and-pepper is a multiplicative noise. 145

Gaussian Noise The Gaussian noise is one of the most common additive noises that appeared in 146 computer vision tasks. The Gaussian noise is independent and stochastic, obeying the Gaussian 147 distribution. Without loss of generality, defined as N ( µ, σ 2 ) . Since our injection happens in the 148 latent space, therefore, the complexity of the task is: 149

According to the definition in Equation 4, and making the distribution of X and Y multivariate 150 normal distribution [5] [14], the mutual information with Gaussian noise is: 151

152

153

154

155

156

157

158

159

160

161

162

163

164

where λ = σ 2 ϵ 1+ ∑ k i =1 1 σ 2 Xi , σ 2 ϵ is the variance of the Gaussian noise, cov( X i , Y i ) is the covariance of sample pair X i , Y i , σ 2 X i and σ 2 Y i are the variance of data sample X i and data label Y i , respectively. The detailed derivations can be found in section 1.1.2 of the supplementary. Given a dataset, the variance of the Gaussian noise, and statistical properties of data samples and labels control the mutual information, we define the function:

Since ϵ 2 ≥ 0 and λ ≥ 0 , σ 2 X i σ 2 Y i -cov 2 ( X i , Y i ) = σ 2 X i σ 2 Y i (1 -ρ 2 X i Y i ) ≥ 0 , where ρ X i Y i is the correlation coefficient, the sign of f ( σ 2 ϵ ) is negative. We can conclude that Gaussian noise injected into the latent space is harmful to the task. More details and the Gaussian noise added to the image level are provided in the supplementary.

Linear Transform Noise This type of noise is obtained by elementary transformation of the features matrix, i.e., ϵ = Q X , where Q is an elementary matrix. We name the Q the quality matrix since it controls the property of linear transform noise and determines whether positive or harmful. In the linear transform noise injection in the latent space case, the complexity of the task is:

The mutual information is then formulated as: 165

Since we want the mutual information to be greater than 0, we can formulate Equation 13 as an 166 optimization problem: 167

̸

where ∼ means the row equivalence. The key to determining whether the linear transform is positive 168 noise or not lies in the matrix of Q . The most important step is to ensure that I + Q is reversible, 169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

̸

which is | ( I + Q ) | = 0 . The third constraint is to make the trained classifier get enough information about a specific image and correctly predict the corresponding label. For example, for an image X 1 perturbed by another image X 2 , the classifier obtained dominant information from X 1 so that it can predict the label Y 1 . However, if the perturbed image X 2 is dominant, the classifier can hardly predict the correct label Y 1 and is more likely to predict as Y 2 . The fourth constraint is to maintain the norm of latent representations. More in-depth discussion and linear transform noise added to the image level are provided in the supplementary.

Salt-and-pepper Noise The salt-and-pepper noise is a common multiplicative noise for images. The image can exhibit unnatural changes, such as black pixels in bright areas or white pixels in dark areas, specifically as a result of the signal disruption caused by sudden strong interference or bit transmission errors. In the Salt-and-pepper noise case, the mutual information is:

Obviously, the mutual information is smaller than 0, which indicates the complexity is increasing when injecting salt-and-pepper noise into the deep model. As can be foreseen, the salt-and-pepper noise is pure detrimental noise. More details and Salt-and-pepper added to the image level are in the supplementary.

4 Experiments

In this section, we conduct extensive experiments to explore the influence of various types of noises on deep learning models. We employ popular deep learning architectures, including both CNNs and ViTs, and show that the two kinds of deep models can benefit from the positive noise. We employ deep learning models of various scales, including ViT-Tiny (ViT-T), ViT-Small (ViT-S), ViT-Base (ViT-B), and ViT-Large (ViT-L) for Vision Transformers (ViTs), and ResNet-18, ResNet-34, ResNet-50, and ResNet-101 for ResNet architecture. The details of deep models are presented in the supplementary. Without specific instructions, the noise is injected at the last layer of the deep models. Note that for ResNet models, the number of macro layers is 4, and for each macro layer, different scale ResNet models have different micro sublayers. For example, for ResNet-18, the number of macro layers is 4, and for each macro layer, the number of micro sublayers is 2. The noise is injected at the last micro sublayer of the last macro layer for ResNet models. More experimental settings for ResNet and ViT are detailed in the supplementary.

4.1 Noise Setting

We utilize the standard normal distribution to generate Gaussian noise in our experiments, ensuring that the noise has zero mean and unit variance. Gaussian noise can be expressed as:

For linear transform noise, we use a quality matrix of as:

where I is the identity matrix, α represents the linear transform strength and f is a row cyclic shift operation switching row to the next row, for example, in a 3 × 3 matrix, f will move Row 1 to Row 2, Row 2 to Row 3, and Row 3 to Row 1. For salt-and-pepper noise, we also use the parameter α to control the probability of the emergence of salt-and-pepper noise, which can be formulated as:

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

where p is a probability generated by a random seed, α ∈ [0 , 1) , and X is the representation of an image.

4.2 Image Classification Results

We implement extensive experiments on large-scale datasets such as ImageNet [11] and small-scale datasets such as TinyImageNet [17] using ResNets and ViTs.

4.2.1 CNN Family

The results of ResNets with different noises on ImageNet are in Table 1. As shown in the table, with the design of linear transform noise to be positive noise (PN), ResNet improves the classification accuracy by a large margin. While the salt-and-pepper, which is theoretically harmful noise (HN), degrades the models. Note we did not utilize data augmentation techniques for ResNet experiments except for normalization. The significant results show that positive noise can effectively improve classification accuracy by reducing task complexity.

4.2.2 ViT Family

The results of ViT with different noises on ImageNet are in Table 2. Since the ViT-L is overfitting on the ImageNet [13] [34], the positive noise did not work well on the ViT-L. As shown in the table, the existence of positive noise improves the classification accuracy of ViT by a large margin compared to vanilla ViT. The comparisons with previously published works, such as DeiT [38], SwinTransformer [20], DaViT [12], and MaxViT [39], are shown in Table 3, and our positive noise-empowered ViT achieved the new state-of-the-art result. Note that the JFT-300M and JFT-4B datasets are private and not publicly available [35], and we believe that ViT large and above will benefit from positive noise significantly if trained on larger JFT-300M or JFT-4B, which is theoretically supported in section 4.4.

4.3 Ablation Study

We also proactively inject noise into variants of ViT, such as DeiT [38], Swin Transformer [20], BEiT [3], and ConViT [9], and the results show that positive noise could benefit various variants of ViT by improving classification accuracy significantly. The results of injecting noise to variants of ViT are reported in the supplementary. We also did ablation studies on the strength of linear transform noise and the injected layer. The results are shown in Fig. 2. We can observe that the deeper layer the positive noise injects, the better prediction performance the model can obtain. There are reasons behind this phenomenon. First, the latent features of input in the deeper layer have better

238

representations than those in shallow layers; second, injection to shallow layers obtain less mutual 235 information gain because of trendy replacing Equation 8 with Equation 7. More results on the small 236 dataset TinyImageNet can be found in the supplementary. 237

(a) The relationship of the CNN family between the strength of

linear transform noise and Top 1 accuracy.

(b) The relationship of the CNN family between the injected

layer of linear transform noise and Top 1 accuracy.

(c) The relationship of the ViT family between the strength of

linear transform noise and Top 1 accuracy.

(d) The relationship of the ViT family between the injected

layer of linear transform noise and Top 1 accuracy.

0.8

0.80

0.7

0.75

0.5

0.70

0.4

resnet18

resnet18

resnet34

0.65

resnet34

0.3

resnet50

resnet50

0.2

resnet101

0.60

resnet101

0.0

0.1

0.2

0.3

0.40.5

0.6

0.7

0.8

0.9

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.90

0.90

ViT-T

0.88

0.88

ViT-S

0.86

0.86

ViT-B

ViT-L

0.84

0.84

0.82

0.82

ViT-T

A

ViT-S

0.80

0.80

ViT-B

0.78

0.78

ViT-L

0.76

0.0

0.1

0.2

0.3

0.40.5

0.6

0.7

0.8

0.9

0

2

4

6

8

10

12

InjectLayer

4.4 Optimal Quality Matrix

As shown in Equation 14, it is interesting to learn about the optimal quality matrix of Q that maximizes 239 the mutual information while satisfying the constraints. This equals minimizing the determinant of 240 the matrix sum of I and Q . Here, we directly give out the optimal quality matrix of Q as: 241

where k is the number of data samples. And the corresponding upper boundary of the mutual 242 information as: 243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

The details are provided in the supplementary. We find that the upper boundary of the mutual information of injecting positive noise is determined by the number of data samples, i.e., the scale of the dataset. Therefore, the larger the dataset, the better effect of injecting positive noise into deep models. With the optimal quality matrix and the top 1 accuracy of ViT-B on ImageNet can be further improved to 95 % , which is shown in Table 4.

4.5 Domain Adaption Results

Unsupervised domain adaptation (UDA) aims to learn transferable knowledge across the source and target domains with different distributions [25] [42]. Recently, transformer-based methods achieved SOTA results on UDA, therefore, we evaluate the ViT-B with the positive noise on widely used UDA benchmarks. Here the positive noise is the linear transform noise identical to that used in the classification task. The positive noise is injected into the last layer of the model, the same as the classification task. The datasets include Office Home [41] and VisDA2017 [26]. Detailed datasets introduction and experiments training settings are in the supplementary. The objective function is borrowed from TVT [44], which is the first work that adopts Transformer-based architecture for UDA. The results are shown in Table 5 and 6. The ViT-B with positive noise achieves better performance than the existing works. These results show that positive noise can improve model generality, therefore, benefit deep models in domain adaptation tasks.

5 Conclusion

This study presents a comprehensive investigation into the influence of various common noise types 262 on deep learning models, including Gaussian noise, linear transform noise, and salt-and-pepper noise. 263 We demonstrate that, under certain conditions, linear transform noise can have a positive effect on 264 deep models. Our experiments show that injecting the positive noise in latent space can significantly 265 enhance the prediction performance of deep models on image classification tasks, leading to new 266 state-of-the-art results on ImageNet. The findings of this study have a broad impact on future research 267 and may contribute to the development of more accurate models and their improved performance in 268 real-world applications. Moreover, we are excited to explore the potential of positive noise in more 269 deep learning tasks. 270

271

272

273

References

Osama K. Al-Shaykh and Russell M. Mersereau. Lossy compression of noisy images. IEEE Transactions on Image Processing , 7(12):1641-1652, 1998.

Wissam A. Albukhanajer, Johann A. Briffa, and Yaochu Jin. Evolutionary multiobjective image 274 feature extraction in the presence of noise. IEEE Transactions on Cybernetics , 45(9):1757-1768, 275 2014. 276

Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. arXiv 277 preprint arXiv:2106.08254 , 2021. 278

Roberto Benzi, Alfonso Sutera, and Angelo Vulpiani. The mechanism of stochastic resonance. 279 Journal of Physics A: mathematical and general , 14(11):L453, 1981. 280

George EP Box and David R. Cox. An analysis of transformations. Journal of the Royal 281 Statistical Society: Series B (Methodological) , 26(2):211-243, 1964. 282

Sebastian Braun, Hannes Gamper, Chandan KA Reddy, and Ivan Tashev. Towards efficient mod283 els for real-time deep noise suppression. In ICASSP 2021-2021 IEEE International Conference 284 on Acoustics, Speech and Signal Processing (ICASSP) , pages 656-660, 2021. 285

Raymond H. Chan, Chung-Wa Ho, and Mila Nikolova. Salt-and-pepper noise removal by 286 median-type noise detectors and detail-preserving regularization. IEEE Transactions on image 287 processing , 14(10):1479-1485, 2005. 288

Thomas M. Cover. Elements of information theory. John Wiley & Sons , 1999. 289

Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent 290 Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv 291 preprint arXiv:2103.10697 , 2021. 292

293

294

295

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, and et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 (2023) , 2023.

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Feifei Li. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition , pages 248-255, 2009.

Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jindong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In In Computer Vision-ECCV 2022: 17th European Conference , pages 74-92, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv preprint arXiv:2010.11929 , 2020.

Changyong Feng, Hongyue Wang, Naiji Lu, Tian Chen, Hua He, Ying Lu, and Xin M. Tu. Log-transformation and its implications for data analysis. Shanghai archives of psychiatry , 26(2):105, 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Solomon Kullback and Richard A. Leibler. On information and sufficiency. The annals of mathematical statistics , 22(1):79-86, 1951.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N 7 , (7), 2015. 314

Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series. 315 The handbook of brain theory and neural networks , 3361(10), 1995. 316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

Xuelong Li. Positive-incentive noise. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Peter McClintock. Can noise actually boost brain power? Physics World , 15(7), 2002.

Toshio Mori and Shoichi Kai. Noise-induced entrainment and stochastic resonance in human brain waves. Physical review letters , 88(21), 2002.

Rich Ormiston, Tri Nguyen, Michael Coughlin, Rana X. Adhikari, and Erik Katsavounidis. Noise reduction in gravitational-wave data via deep learning. Physical Review Research , 2(3):033066, 2020.

J. S. Owotogbe, T. S. Ibiyemi, and B. A. Adu. A comprehensive review on various types of noise in image processing. int. J. Sci. eng. res , 10(10):388-393, 2019.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering , 22(10):1345-1359, 2009.

Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

Lis Kanashiro Pereira, Yuki Taya, and Ichiro Kobayashi. Multi-layer random perturbation training for improving model generalization efficiently. Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , 2021.

Kamiar Radnosrati, Gustaf Hendeby, and Fredrik Gustafsson. Crackling noise. IEEE Transactions on Signal Processing , 68:3590-3602, 2020.

Fabrizio Russo. A method for estimation and filtering of gaussian noise in images. IEEE Transactions on Instrumentation and Measurement , 52(4):1148-1154, 2003.

James P. Sethna, Karin A. Dahmen, and Christopher R. Myers. Crackling noise. Nature , 410(6825):242-250, 2001.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, Cambridge, 2014.

Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review , 5(1):3-55, 2001.

Jan Sijbers, Paul Scheunders, Noel Bonnet, Dirk Van Dyck, and Erik Raman. Quantification and improvement of the signal-to-noise ratio in a magnetic resonance image acquisition procedure. Magnetic resonance imaging , 14(10):1157-1163, 1996.

Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. In arXiv preprint arXiv:2106.10270 , 2021.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In In Proceedings of the IEEE international conference on computer vision , pages 843-852, 2017.

Tao Sun, Cheng Lu, Tianshuo Zhang, and Harbin Ling. Safe self-refinement for transformerbased domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7191-7200, 2022.

Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd-Yusof. Combating label noise in deep learning using abstention. In arXiv preprint arXiv:1905.10964 , 2019.

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357, 2021.

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In In Computer Vision-ECCV 2022: 17th European Conference , pages 459-479, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , 2017.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. CVPR , pages 5018-5027, 2017.

Ying Wei, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer. ICML , pages 5085-5094, 2018.

Tongkun Xu, Weihua Chen, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. ICLR , pages 520-530, 2022.

Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. WACV , pages 520-530, 2023.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104-12113, 2022.

Checklist

The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description. For example:

Did you include the license to the code and datasets? [Yes] Yes

Did you include the license to the code and datasets? [No] The code and the data are proprietary.

Did you include the license to the code and datasets? [N/A]

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

For all authors...

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [TODO] Yes

(b) Did you describe the limitations of your work? [TODO] Yes

(c) Did you discuss any potential negative societal impacts of your work? [TODO] No

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [TODO] Yes

If you are including theoretical results...

(a) Did you state the full set of assumptions of all theoretical results? [TODO] Yes

(b) Did you include complete proofs of all theoretical results? [TODO] Yes

If you ran experiments...

(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [TODO] Yes

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [TODO] Yes

(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [TODO] No

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? No [TODO]

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

(a) If your work uses existing assets, did you cite the creators? [TODO] N/A

(b) Did you mention the license of the assets? [TODO] N/A

(c) Did you include any new assets either in the supplemental material or as a URL? [TODO] N/A

(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [TODO] N/A

(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [TODO] N/A

If you used crowdsourcing or conducted research with human subjects...

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [TODO] N/A

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [TODO] N/A

(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [TODO] N/A

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the influence of adding three types of noise to the images or the latent representation of the images on the performance of CNN and ViT. The authors theoretically and empirically show that the family of random noise can be divided to positive noise and harmful noise regarding their impacts on the deep learning models.\n\nSTRENGTHS:\n1. This paper is well-motivated and inspiring because it shows that the noise, which is normally perceived as harmful to the DNNs, can actually be good to the performance of DNNs in many ways. \n2. The authors have done extensive theoretical analysis to support their claims with great novelty. \n3. The presentation of this paper is clear and well-written.\n\nWEAKNESSES:\n1. Although the authors propose a novel idea of positive noise and harmful noise, the current content of this paper is not enough to support their claims. \n2. My major concern is that there are some fundamental errors in the theoretical analysis (see the questions for details), which makes me question whether the theorems really hold. \n3. The experiments are also not enough to support the theorems.\n\nQUESTIONS:\n1.\tWhat is the relationship between the proposed method and the existing methods that add stochasticity into the network, such as SGD. And I think adding noise to the latent space must have some connection to the dropout method, can the author explain more about the connections and do some experiments to see if the proposed method has  superior performance than dropout?\n\n2.\tThe definition of the difficulty of the task $\\tau$ is obscure. If I understand correctly, it is defined as the conditional entropy H(Y|X). This makes sense to me because it indicates the uncertainty of Y given X. Then I have the following questions: (1) Eq. (6) should also have expectation over X. (2) The definition in Eq. (8) is incorrect. According to the chain rule for entropy, it should be H(Y|X,X+e)=H(X,Y,X+e)-H(X+e|X)-H(X). \n\n3.\tIn the analysis of salt-and-pepper noise, the author mentioned that the mutual information between the task and the noise is negative. But we know that mutual information is always non-negative. So there must be some mistakes.\n\n4.\tThe theoretical analysis of adding noise to the image indicates adding Gaussian noise and salt-and-pepper noise to the image are harmful to the performance of the model, while no experiment has been done to support this conclusion. As far as I know, there are many studies in data augmentation and adversarial learning suggesting adding Gaussian noise or some other noise to the image can be beneficial to the model’s robustness [1,2]. Why is the finding in this paper contradictive to these studies?\n\n[1] Yin D, Gontijo Lopes R, Shlens J, et al. A fourier perspective on model robustness in computer vision[J]. Advances in Neural Information Processing Systems, 2019, 32.\n[2] Cohen J, Rosenfeld E, Kolter Z. Certified adversarial robustness via randomized smoothing[C]//international conference on machine learning. PMLR, 2019: 1310-1320.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the accuracy of image classification models by injecting noise into the latent representations during training. Please be aware that I do not understand key aspects of the proposed method, so my summary and interpretation below may be wrong.\n\nThe paper argues that injecting noise into the latent representations of neural networks can reduce or improve performance, depending on the kind of noise. The paper analyzes different kinds of noise using an abstract information-theoretical framework of task complexity that I do not understand. Three kinds of noise are analyzed:\n\nAdditive noise (e.g. Gaussian): This is determined to be always detrimental according to the theoretical framework.\nMultiplicative noise (e.g. salt-and-pepper noise): This is also determined to be always detrimental.\nLinear transform noise: As far as I understand, this noise consist of randomly permuting rows of the feature matrix my multiplying with an elementary matrix Q. Depending on the choice of Q, this type of noise can improve model performance.\n\nThe paper goes on to show that across many model architectures and classification tasks, injecting linear transform noise leads to huge gains in performance. Here, I must be missing important aspects of the method, since the gains are so large that they are basically impossible. Perhaps these are theoretical upper performance bounds when the noise is optimized for a given test set? I am not sure.\n\nSTRENGTHS:\nThe performance improvements presented in the paper would be spectacular if true, but I have serious doubts about their validity.\n\nWEAKNESSES:\n1. The theoretical parts of the paper are hard to follow. For example, equation 4 introduces the concept of “task entropy”, but no mathematical definition for a “task” is given, and no explanation what “task entropy” is in the context of image classification. Another example is the matrix Q. This is described as an “elementary matrix”, which, according to Wikipedia, is a matrix that differs from the identity matrix by a single elementary row operation. The paper must be using the term in a different sense, since the elements of Q can have values between 0 and 1 in the paper (e.g. Equation 19). This needs to be explained. In general, more intuition and relation to image classification should be given in the theoretical part.\n\n2. The experimental aspects of the paper are also insufficiently described. When and where exactly is the noise injected during training? How much? Does the noise depend in some way on the data (this is suggested by statements in L246, which say that the effectiveness of the noise depends on the dataset size)? Is the validation/test set used to choose the noise?\n\n3. The accuracy improvements shown in the paper appear unrealistic and need to be discussed more. Table 1 reports 91.37% top-1 accuracy on ImageNet, Table 4 even reports 95.12%, both for ViT-B models. The value of 91.37% is already far above the best existing models (https://paperswithcode.com/sota/image-classification-on-imagenet), which are much larger than ViT-B. The value of 95.12% is very likely impossible without accessing the test data, given that ImageNet has significant label noise, which means that even humans or a perfect classification model could not reach 100% on ImageNet (see https://arxiv.org/abs/2006.07159). Therefore, I suspect that the presented results either represent some sort of upper performance bound when optimizing the injected noise directly for the test data (in which case it would be wrong to compare these numbers to models that have not seen the test data), or there are some other issues with the evaluation.\n\nQUESTIONS:\nSee Weaknesses.\n\nMinor comment: Appendix Figure 1: Legends seem wrong, second image from the left looks like salt and pepper but legend says “with Gaussian Noise”.\n\nLIMITATIONS:\nThe paper does not discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper challenges the conventional view that noise is usually detrimental in deep learning and proposes the concept of positive noise.  It explores how particular types of noise can, under specific circumstances, enhance performance, from both theoretical view and experimental view. It illustrates that injecting positive noise can facilitate learning and achieve promising improvements (sometimes even SOTA) on both image classification and domain adaptation tasks for both CNNs and ViTs.

STRENGTHS:
1. It proposes the concept of "positive noise", which is contradictory to the conventional view that noise is usually harmful to performance.
2. It provides solid theoretical analysis and comprehensive experiments to support its claims
3. Injecting positive noise to hidden layers as proposed by this work is applicable to any deep learning architectures. It holds significant potential for exploring the application of positive noise in diverse deep learning tasks beyond the scope of image classification and domain adaptation tasks investigated in this study.

WEAKNESSES:
1. The paper focuses on exploring three common noises: Gaussian noise, linear transform noise, and salt-and-pepper noise. Whether it can be generalized to other noises remains to be explored.

QUESTIONS:
1. There have been many works that adding noise to help data augmentation, model robustness, generalization, regularization, etc, whether to input or hidden layers. Besides scoping out the concept of "positive noise", are there any other novelties compared to those existing works?
2. In Fig 2(d), in many cases adding the noise is hurting the accuracy? How to interpret this?

LIMITATIONS:
The types of noise and the experimented tasks are limited to demonstrate its generalizability.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges the view that noise is always harmful in computer vision tasks and deep learning architectures. The authors demonstrate that specific types of noise, termed positive noise (PN), can actually boost the performance of deep architectures under certain conditions. The paper provides theoretical proof and empirical evidence of the benefits of positive noise, showcasing significant performance gains in large image datasets like ImageNet.\n\nSTRENGTHS:\n- The paper provides both theoretical analysis and empirical evidence to support the presence and effectiveness of positive noise.\n- The empirical experiments conducted on large image datasets, including ImageNet, demonstrate significant performance gains by proactively injecting positive noise. \n- The signal processing perspective on noise for neural networks is interesting.\n\nWEAKNESSES:\n- The main weakness is lack of novelty. Adding noise to neural networks and machine learning models in general has been proposed under various occasions as an effective way to improve the performance [1-5]. As such, I think the contribution of this paper is limited. \n\n- The authors of the paper should conduct a more careful set of experiments to compare against some of the previously proposed noise-based regularization techniques against their proposed method.\n\n- A more careful literature search should be done to address the difference between this work and prior works on noise-based regularizations. \n\n[1] Noh, Hyeonwoo, et al. \"Regularizing deep neural networks by noise: Its interpretation and optimization.\" Advances in neural information processing systems 30 (2017).\n\n[2] You, Zhonghui, et al. \"Adversarial noise layer: Regularize neural network by adding noise.\" 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019.\n\n[3] Bishop, Chris M. \"Training with noise is equivalent to Tikhonov regularization.\" Neural computation 7.1 (1995): 108-116.\n\n[4] Li, Yinan, and Fang Liu. \"Whiteout: Gaussian adaptive noise regularization in deep neural networks.\" arXiv preprint arXiv:1612.01490 (2016).\n\n[5] Li, Bai, et al. \"Certified adversarial robustness with additive noise.\" Advances in neural information processing systems 32 (2019).\n\nQUESTIONS:\n- It seems that Linear Transform Noise is the best among the three tested. Any insight on why this is the case?\n\nLIMITATIONS:\nThe authors might want to address limitations of the proposed method further.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies noise injection into popular image classification models (ResNets and ViTs) from a theoretical as well as experimental standpoints. The analyses are carried out for various noise types (Gaussian, salt-and-pepper or linear transform) as well as different points of application (at the input images or at the intermediate activations). Experimental results demonstrate a strong improvement over baseline and competing methods with the linear transform noise.\n\nSTRENGTHS:\n* Strong experimental results\n* Analysis of a broad range of design choices (noise types, noise injection points, vision architectures)\n* In-depth theoretical analysis of the effect of noise injection that aligns well with experimental results.\n\nWEAKNESSES:\n**Major**\n\n* I have to say that I found the paper to be difficult to digest. The main findings are somewhat masked by the breadth of analysis. For example, a considerable amount of time is spent on image-level noise injection or on salt-and-pepper noise, which end up not yielding significant improvements. I found the simultaneous consideration of both - image-level as latent-level noise - throughout the paper to be confusing. Similarly, the space allocated to salt-and-pepper and Gaussian noise could be repurposed to provide additional details on the linear transform noise.\n* At a high-level noise injection described in the paper is a form of augmentation and/or regularisation. Yet the connection to these techniques isn't explored in the paper. In this context, the statement on lines 1-3 about noise being viewed as a harmful perturbation isn't fully fair, as we do view augmentation as helpful and often necessary.\n* Details of how linear noise application are missing. For example, is it applied across the entire activation tensor? Or is it perhaps applied per spatial position / token?\n* The authors mention that models that are overfitting are not expected to benefit from noise injection. Could they please elaborate on this? When viewing noise injection as augmentation, it seems that noise injection would help battling overfitting.\n* I did not follow the derivations in the paper closely, but it appears that the authors assume that label and input images follow the standard normal distribution (lines 150-151). How realistic is that? If the theoretical analysis conclusions are based on this assumption, can we expect them to be valid for real-world scenarios?\n* There seems to be some confusion about the structure of the linear transform matrix Q. In line 162 it is set to be an elementary matrix (i.e. different from an identity matrix by a single elementary operation), where as in equation 14 it is allowed to be from a broader class of matrices.\n* Beyond the connection of the proposed methods to data augmentation techniques, it would be very interesting to understand whether the proposed noise injection scheme and SOTA augmentation methods (e.g. RandAugment) are complementary.\n* Noise injection in the last layer only should be justified further, also experimentally.\n\n\n\n**Minor**\n* Unclear text: line 83 \"[...] contribute to less mean [...]\", Tables 1 and 2 \"[...] in the bracket [...]\", line 206 \"[...] where $p$ is a probability generated by a random seed [...]\", line 236 \"[...] because of trendy replacing Equation [...]\"\n* Unclear what's meant by \"severe problems\" in lines 124-125, or by \"brute force attacks\" in line 128\n* Figure 1 talks about injecting noise at a random layer in the network, yet experimentally only input image and last layer were considered.\n* line 147 \"[...] Gaussian noise is independent and stochastic\" - Gaussian noise does not have to be independent (i.e. can have a non-diagonal covariance matrix)\n* In line 152, is the covariance taken between variables with a different number of dimensions?\n* In line 156 \"[...] we define the function [...]\" - unclear for what purpose.\n* Column names in Table 5 are ineligible\n\nQUESTIONS:\nCovered in Weakness, will not repeat here.\n\nLIMITATIONS:\nNot addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}