Data Augmentation in Training CNNs: Injecting Noise to Images
Authors: Murtaza Eren Akbiyik
Venue:  (2020)
Keywords: deep learning
data augmentation
convolutional neural networks
noise
image processing
SSIM

Abstract: Noise injection is a fundamental tool for data augmentation, and yet there is no widely accepted procedure to incorporate it with learning frameworks. This study analyzes the effects of adding or applying different noise models of varying magnitudes to Convolutional Neural Network (CNN) architectures. Noise models that are distributed with different density functions are given common magnitude levels via Structural Similarity (SSIM) metric in order to create an appropriate ground for comparison. The basic results are conforming with the most of the common notions in machine learning, and also introduces some novel heuristics and recommendations on noise injection. The new approaches will provide better understanding on optimal learning procedures for image classification.

Under review as a conference paper at ICLR 2020
DATA AUGMENTATION IN TRAINING CNNS: INJECTING NOISE TO IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Noise injection is a fundamental tool for data augmentation, and yet there is no widely accepted procedure incorporate it with learning frameworks. This study analyzes the effects of adding or applying different noise models of varying mag- nitudes to Convolutional Neural Network (CNN) architectures. Noise models that are distributed with different density functions are given common magnitude lev- els via Structural Similarity (SSIM) metric in order to create an appropriate ground for comparison. The basic results are conforming with the most of the common notions in machine learning, and also introduces some novel heuristics and rec- ommendations noise injection. The new approaches will provide better under- standing on optimal learning procedures for image classification.
INTRODUCTION
Convolutional Neural Networks (CNNs) find ever-growing field of application throughout image and sound processing tasks, since the success of AlexNet (Krizhevsky et al., (012) in the 2012 ImageNet competition. Yet, training these networks still keeps the need of an "artistic" touch: even the most cited state-of-the-ar studies employ wildly varying set of solvers, augmentation and regularization techniques (Domhan et al. 2015). In this study, one of the crucial data augmentation techniques, noise injection, will be thoroughly analysed to determine the correct way of application on image processing tasks.
Adding noise to the training data is not procedure that is unique to the training of neural architec- tures: additive and multiplicative noise has long been used in signal processing for regression-bast methods, in order create more robust models (Saiz et al. 2005). The technique is also one of the oldest data augmentation methods employed in the training of feed forward networks, as analysed by Holmstrom & Koistinen (1992), yet it is also pointed out in the same study that while using additive Gaussian noise is helpful, the magnitude of the noise cannot be selected blindly, as a badly-chosen variance may actually harm the performance of the resulting network (see Gu & Rigazio (0014) and Hussein et al. (2017) for more examples).
The main reasons for noise injection to the training data can be listed such in non-excluding manner: first of all, injection of any noise type makes the model more robust against the occurrence of that particular noise over the input data (see Braun al. (016)) and Saiz al. (2005) for further reference), such as the cases of Gaussian additive noise in photographs, and Gaussian-Poris Poisson noise on low-light charge coupled devices (Bovik) 2005). Furthermore, it is shown that the neural networks optimize on the noise magnitude they are trained on (Yin et al.) 2015). Therefore, it is important to choose the correct type and level of the noise to augment the data during training.
Another reason for noise addition is to encourage the model to learn the various aspects of each class by occluding random features. Generally, stochastic regularization techniques embedded inside the neural network architectures are used for this purpose, such as Dropout layers, yet it is also possible to augment the input data for such purposes as in the example of "cutout" regularization proposed by Devries & Taylor (2017). The improvement of the generalization capacity of network is highly correlated with its performance, which can be scored by the accuracy over predetermined test set.
There has been similar studies conducted on the topic, with the example of Koziarski & Cyganek (017)) which focuses on the effects of noise injection on the training of deep networks and the possible denoising methods, yet they fail to provide proper methodology determine the level of

---

Under review as a conference paper at ICLR 2020
noise to be injected into the training data, and use PSNR the comparison metrie between different noise types which is highly impractical (see Section 3). To resolve these issues, this study focuses on the ways to determine which noise types to combine the training data with, and which levels, in addition to the validity of active noise injection techniques while experimenting on a larger set of noise models.
In the structure of this work, the effect of injecting different types of noises into images for varying CNN architectures is assessed based their performance and noise robustness. Their interaction and relationship with each other are analyzed over (also noise-injected) validation sets. Finally as follow-up up study, proper ways on adding or applying noise CNN for image classification tasks are discussed.
DIFFERENT TYPES OF NOISE
Noise can be -somewhat broadly- defined as unwanted component the image (Bovik) 2005). It can be sourced from the environment at which the image is taken, the device utilized to take the image, or the medium of communication that is used to convey the information of image from source to the receiver. According to its properties and nature, noise and image can be analytically decomposed as additive or multiplicative, but some of the noise types cannot be described by neither of these classes
2.1 ADDITIVE NOISE
Let f(x denote an image signal. This signal can be decomposed into two components in an additive manner fx) s(x) + xx) where g(x) denoting the desired component of the image, and n(x) standing for the unwanted noise component. Most commonly encountered variant of this noise class is Gaussian noise, whose multivariate probability density function be written as:
(-1
1 2
fx(x)
exp
m) m)
(2T)
T
where and Σ denoting the n-dimensional mean vector and symmetric covariance matrix with rank n, respectively. In images, the mean vector m is generally zero, therefore the distribution is centered and the magnitude is controlled by the variance. This study also follows these assumptions.
2.2 MULTIPLICATIVE NOISE
Again, let fxx denote an image signal. This signal can also be decomposed into respective desired and noise components as f(x) g(x) + n())). The noise component in this model is called multiplicative noise. The most common variant this case called speckle noise, which may have different density functions, and in this study Gaussian is assumed. Similar with the additive noise, the mean is assumed to be 0 and the magnitude refers to the variance.
Speckle noise can be encountered in coherent light imaging such as in the cases of SAR images (Ding et al. 2016) and images with faser-based illumination, but they may also be observed in other digital images (Bovik 2005).
2.3 OTHER TYPES OF NOISE
There exists many other noise instances that cannot be modeled by additive or multiplicative de- compositions. Most common of these types are listed below, whose effects on the performances of CNNs are also analysed this study.
Salt and pepper (S&P) noise. This noise manifests itself as a basic image degradation, for which only few pixels in image noisy, but they are extremely noisy in way that pixels either completely black or white. The magnitude of this noise is the probability of pixel to be completely black (i.e. pepper), completely white (i.e. salt), or stay unchanged. The probabilities for pepper and
2

---

Under review as a conference paper at ICLR 2020
salt cases assumed to be equal, and total probability of the degradation of pixel is referred as the the magnitude of the noise.
Poisson noise. Also referred as photon counting noise or shot noise, this noise type has a particular probability density function:
A [n.k) k!
where À stands for both the variance and the mean of the distribution. As Poisson noise is signal- dependant, does not have a direct magnitude parameter similar to other noise types, therefore a magnitude factor e is used that divides the intensity values of all pixels from which the distribution is sampled, and returned to the original range by multiplying to the same factor.
Although it is not generally referred as a noise type, occlusion of important features can happen for number of reasons in image: image may be cropped, damaged particular object in it may be hidden by an obstacle such as tree. This noise is realized with zero- intensity squares appearing on the image, and the magnitude is determined according to the size of the square as the shape of the occluding object does not have a large impact on the final performance of the model (Devries
Occlusion noise.
Taylor 2017).
As listed in this section, five different types of noise and their various combinations are added or applied to the images, with varying magnitudes. Robustness against each of these noise types also assessed.
CHOOSING THE RIGHT METRIC: PSNR VS SSIM
Different types of noise are easy to compare when they are sampled from the same distribution, such in the case of additive Gaussian noise and speckle noise. However, it is sometimes impossible to assess two different metrics in the same context because of varying magnitude parameters and sensi- tivity of the model each of these types. In this case, becomes imperative to intermediary metric that will case the comparison process for noise types.
In general, Peak Signal-to-Noise Ratio (PSNR), and similarly Mean Squared Error (MSE) are the most commonly used quality metrics in the image processing field. For an 8-bit two-dimensional MxN image (m1 (2)) and its free counterpart (11, the MSE is defined as
1 M-1N-
MSF
Σ Σ
[2) (m),
(3)
MN
0
From above definition, PSNR (in dB) can be derived.
2552 MSSE
PSNR R 10 10
(4)
There are several limitations of using PSNR the image quality metric of data set: is shown that PSNR loses its validity as quality metric when the content and/or codec of the images are different as in that case the correlation between subjective quality and PSNR is highly reduced (Huynh- & Ghanbari 2008). Also, even though the sensitivity of PSNR to Gaussian noise is very high, the metric is unable to present similar performance for different types of perturbation (veiias et al.) 2002 (Horé & Ziou 2010).
There exists another widely accepted image quality metric called Structural Similarity (SSIM), which resolves or alleviates some of the above-listed problems. The Structural Similarity between two non negativee image signals x and y. whose means, standard deviations and covariance are de- noted by pars Pyy Zz By and azy respectively, can be expressed as:
3

---

Under review as a conference paper at ICLR 2020
21
212
24
2.4
X1
X0
(a) Gaussian
(b) Speckle
87
M4
110
270
(c) S&P
(d) Poisson
(e) Occlusion
Figure I: "hexbin" plots of the PSNR and SSIM values for different noise types over Imagewoof dataset, subset of ImageNet with training size of 13000 samples.
(2prlyy C2) + C2) (1² + 1 + ()) + o + C2)
SSTM
(5)
where C1 and C2 are constants to avoid instability (Zhou Wang et al. 2004). This metric combines luminance, contrast and structure information in order provide assessment of similarity in the range from 0 to 1, where 1 stands for highest similarity.
In image classification tasks, the objective is mostly to classify the depicted objects beings ac- cording to the human perception. Therefore, it is crucial for the used metric to be consistent with human opinions: it is shown in several studies that SSIM provides quality metric closer average opinion of public than PSNR (Kim et al. 2019) Zhou Wang et al., 2004).
Furthermore, when sampled from the same noise distributions over identical images of a data set, distribution of PSNR values of the noisy images have significantly high kurtosis than their SSIM counterparts every noise type except S&P. This behavior is demonstrated over a subset of ImageNet dataset (Deng et al.) (2099) called Imagewoof (can be accessed via github. hotaaaaccoo/sssaaamaaeettte with 10 classes and 12454 total number of images on the training set. Each type of noise listed in Section 2 is applied to each image, sampling from the same distribu- tion, and quality metries are recorded. The utilized noise magnitudes are 0.2 variance for Gaussian noise, 0.2 variance for speckle noise, 0.05 total probability for S&P noise, 0.2 scaling factor for Poisson noise, and 0.3 relative side length for occlusion noise; which are all chosen to provide sen- sible values from the metries. The distribution of the metries can be seen Figure over the axes
Gaussian
Speckle
S&P
Poisson
Ocelusion
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
4.263
0.074
4.426
0.644
0.253
0.278
6.221
2.030
7.286
-0.830
Table I: Kurtosis values of quality metric distributions over different types of noise
4

---

Under review as a conference paper at ICLR 2020
This is interpreted PSNR having propensity to produce more outliers than SSIM for the same levels of noise (Westfall 2014).
of each subfigure, and the kurtosis values noted in Table 1
For the reasons listed above, SSIM will be used as the primary metric throughout this study.
4 EXPERIMENTS AND RESULTS
Effects of injecting different noise types into training data are evaluated for different magmitudes and types of the noise as listed in Section 2. The chosen datasets are two different subsets of ImageNet dataset, namely Imagenette and Imagewoof, each consisting of 10 different classes with 12894 and 12454 training samples respectively and 500 test samples (both can be accessed via github. comp/astai/imaaenett)) Former dataset contains ten easily classified classes of the original set, while the latter task is the classification of ten different dog breeds and require the network to successfully learn the particularly small features the classes. Image data range is henceforth from 0 to 1.
In order to select the magnitudes of cach noise component, a sweep for mean SSIM (MSSIM) of an array of noise magnitudes for each noise type is conducted over 200 images of Imagewoof dataset. Resulting graph can be seen Figure Very similar results are also observed when the same procedure is conducted polynomial is fitted to the relative side length of occlusion noise, and logarithmic polynomials are fitted to the rest. Look-up table for the fittings be seen Table which are also the degradations applied in the experiments. Exemplary application of the noise can be seen at Figure
Imagenette dataset. According the shapes of the curves, quadratic
As the training model, a 18-layer deep residual network (ResNet18V2) as proposed by He et al. (2016) is chosen, because of the fact that it is a well-known architecture and also sufficiently deep: Bengio et al. (2011) demonstrate that performance
the deep networks are more sensitive to data augmentation than their more shallow counterparts. The residual connections and layering structure of ResNet1882 exhibit similar architectural properties of the most often utilized CNNs the field.
Adam solver with learning rate le-4 is preferred for training. Models are trained for 20 epochs. No dropout or weight decay used for regularization purposes, and Batch Normalization layers are used in accordance with He et al. (206)).
1.0
hotse Type
Noise Types
567
Proaon Deruuoon
Gaussian
Speckle
S&P
Poisson
Occlus.
07
MSSIM
var.
prob.
magn.
length
MS5M
0.25
0.0341
0.3355
0.1288
0.1046
1.1908
0.5
0.0085
0.0515
0,0461
0.0222
0.9078
0.7
0.0028
0.0115
0.0203
0.0064
0.6308
0.8
0.0016
0.0054
0.0134
0.0035
0,4753
52
0.9
0.0009
0.0026
0.0089
0.0019
0.3086
32
20
22
Negrituda Stndl Laved
Figure 2: MSSIM vs Noise
Table 2: Look-up table for the noise levels and MSSIM values
Pc Sobaa
Gaumiaa Hoine
Speciee
Hons
Figure 3: Exemplary noise applications for MSSIM = 0.5 from Table
5

---

Under review as a conference paper at ICLR 2020
0.8
Noise Type
Gaussian
0.6
Speckle
S&P
Accuracy
Poisson
0.4
Occlusion
No Noise
0.2
0.0
0.25 SSIM
0.5 SSIM
0.7 SSIM Noise Level
SSIM
0.9 SSIM
Figure 4: Accuracy of models over clean test set for Imagenette dataset.
0.6
Noise Type
0.5
Gaussian
Speckle
0.4
S&I
Accuracy
Poisson
0.3
Occlusion
No Noise
0.2
0.1
0.0
0.25 SSIM
0.5 SSIM
0.7 SSIM Noise Level
0.8 SSIM
0.9 SSIM
Figure 5: Accuracy of models over clean test set for Imagewoof dataset.
4.1 COMPARISON OF NOISY AND VANILLA TRAINING
Chosen CNN architecture trained with noise injected to the training data for noise models described in Section 2 for magnitudes corresponding to respective MSSIM values from Table 2 Total number of 52 networks are trained (25 on noisy data and on original data for each dataset). The categorical accuracy of each trained network the validation set can be seen at Figures 4 and 5) and the accuracies of the CNNs depending on the noise they are trained with for noisy test set can be observed at Figure The latter results can be considered as the robustness test of the trained networks. One of the most important features of these heatmaps, robustness of the models against the noise injected to their input, is also plotted for each dataset individually in Figures and
5 DISCUSSION
There are several confirmations to acquire from this set of results for the literature: first of all, there exists a trade-off between noise robustness and clean set accuracy. Yet contrary to the common notion, we believe that the data presents a highly valid optimum for this exchange in our study, As can be seen from Figures 6 and 7 in order to create a robust model against particular kind of noise while maintaining the performance of the model, one must apply a level of degradation that results in 0.8 MSSIM over training data. We believe that as long as the noise perturbation is somewhat homogeneously distributed, this rule of thumb will hold for all image classification tasks. However, the same thing cannot be said for non non-homogencously distributed noise models, as SSIM (and also PSNR as demonstrated in Section 3) fails to capture the level of degradation appropriately for such verdict (see Occlusion results in Figures and 7).
A second confirmation of the current literature is the fact that the neural networks optimize on the noise level they trained with, as seen again at Figures 6 and 7 and also the diagonals of Figure g Yet, the level of this optimization quite small after 0.5 MSSIM, featuring similar robustness for each trained model. Therefore, it is not particularly necessary determine the noise level of

---

Under review as a conference paper at ICLR 2020
(a) Gaussian
(b) Speckle
(c) S&P
(d) Poisson
(e) Occlusion
Figure 6: Accuracy (robustness) plots of the models their noise type, Imagewoof dataset.
(a) Gaussian (b) Speckle
(c) S&P
(d) Poisson (e) Occlusion
Figure 7: Accuracy (robustness) plots of the models their noise type, Imagenette dataset.
St
Dr
2100e
Inode
Octeoebd
Imagenette
(b) Imagewoof
Figure 8: Accuracy (Robustness) heatmaps of each model to augmented with different noise types. High accuracy on the diagonals are signs of noise level optimization of neural networks.
dataset, or sample the noise from a predetermined interval, as long as the MSSIM does drop below 0.5, in which case noise removal techniques need to be considered for better models.
As noted above, occlusion noise type will be thoroughly analyzed in this section because of the fact that the quality metric has failed to provide sufficient comparative data for this discussion. Yet, the performance data and the lack of robustness the other models exhibit towards this particular noise type shows that "cutout" regularization as presented by Devries & Taylor (017)) is a crucial part of data augmentation to further extend the contribution of this method would be to alternate the intensity level of the patches from 0 to 255 for 8-bit images, which can be a topic of another research.
addition any other perturbation or noise injection technique. A way
For the rest of the noise types; Gaussian, speckle and Poisson noises are observed to increase the performance of the model while boosting the robustness, and their effects exhibit the possibility of interchangeable usage. For image classification tasks involving RGB images of daily objects, in- jection only one of these noise types with above-mentioned level is believed to be sufficient as repetition of the clusters be observed in Figure Among these three, Gaussian noise is recom-

---

Under review as a conference paper at ICLR 2020
mended considering the results of model performance. S&P noise contamination, on the other hand, may not resolved by injection the former noise types as the other models are not sufficiently robust against it. Therefore, at this point one of the two methodologies noise can be removed by simple filtering techniques, or S&P noise can be applied an alternating manner with Gaussian noise during data augmentation. Former approach is recommended for the simplicity of the training procedure.
suggested: either S&P
The constant behaviour of the models towards occlusion noise in Figures 6 does not have a satisfactory explanation, despite several diagnostics of the training procedure. A longer training procedure, which was not feasible in our experiment because of the model count, may resolve these undesirable results.
7 and 8 unfortunately
6 CONCLUSION
In this study, extensive analysis of noise injection training data has conducted. The results confirmed some the notions the literature, while also providing new rule of thumbs for CNN training. As further targets of research, extension of "cutout" regularization as described in the above paragraphs, and the distribution behavior of the SSIM and PSNR metrics in Figure 2 with regards to the work of Horé Ziou 2010) may be pursued.
REFERENCES
Ismail Avcibas, Bulent Sankur, and K. Sayood. Statistical evaluation of quality measures. j electron imaging. J. Electronic Imaging, 11:206-223, 04 2002. doi: 10.1117/1.145500
Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger-Leewandowske Thomas M. Breuel, Youssouf Chherawala, Moustapha Cissé, Myriam Côté, Dumitru Erhan, Jeremy Eustache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, Franois Savard, and Guillaume Sicard. Deep learners benefit more from -iitt--disttibution examples. In AISTATS, 2011.
Alan C. Bovik. Handbook of Image and Video Processing (Communications, Networking and Mul- timedia). Academic Press, Inc., Orlando, FL., USA, 2005. ISBN 0121197921.
Stefan Braun, Dan Neil, and Shih-Chii Liu. A curriculum learning method for improved noise robustness in automatic speech recognition. 06 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPRO9, 2009.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv, abs/1 had/1708.04552... 04552, 2017.
Ding, Chen, H. Liu, and M. Huang. Convolutional neural network with data augmentation for sar target recognition. IEEE Geoscience and Remote Sensing Letters, 13(3):364-33 36886666,, March 2016. doi: 10. 111199......001555335.44.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation learning curves. In Proceedings of the 24th International Conference Artificial Intelligence, IJCAI' pp. 3460-3468. AAAI Press, 2015, ISBN 978-1-57735-738-4, URL WRLLhttp://ddl at..g//aaaa.......ttttiiio... ifmmii cfmiid= 2832581 2832731
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. 12 2014.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas. Nicu Sebe, and Max Welling (eds.), Computer Viston ECCV 2016, pp. 630-645, Cham, 2016. Springer International Publishing. ISBN 978-3-319- 46493-0.
8

---

Under review as a conference paper at ICLR 2020
L. Holmstrom and P. Koistinen. Using additive noise in paok-propngation training. IEEE Transac- tions on Neural Networks, 3(1):24-38) 38, Jan 1992. doi: 10.1109/72.105415.
Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs, ssim. pp. 2366-2369, 08 2010. doi: N0.1109/CCPR22100.579
Sarfaraz Hussein, Robert Gillies, Kunlin Cao, Qi Song, and Ulas Bagei. Tumornet: Lung nodule characterization using multi-view convolutional neural network with gaussian process. CoRR, 00645
abs/1703.000455, 2017. DRtppppp:/aaxxi.orrr org.
Q. Huynh- Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics Letters, 44(13):800-800) 801, June 2008. doi: 10. 20000
Kim, A. Nguyen, and S. Lee. Deep enn-based blind image quality predictor. JEEE Transactions on Neural Networks and Learning Systems, (0(1):11-24 Jan 2019. doi: 10.1109/TNLSS.0115 2829819.
Micha Koziarski and Boguslaw Cyganek. Image recognition with deep neural networks in presence of noise dealing with and taking advantage of distortions. Integrated Computer-Aided Engineer- doi: 10.3233/ICA-1700 170551.
ing, 24:1-13, 08 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Proceedings of the 25th International Conference Neural Infor- mation Processing Systems- Volume 1, NIPS 12, pp. 1097-1105, USA, 2012. Curran Associates htt.:aatt.o...cc..m......99.1... 2999999999992999977
Inc. URL httpt://t//..... cm.
Maria Saiz, Bjrm-Helge Mevik, V.H. Segtnan, and Ns. Ensemble methods and data augmentation by noise addition applied to the analysis of spectroscopic data. Analytica Chimica Acta, 533: 147-159, 03 2005. doi: 10. 10.1016/j.aca.200 2004. 10.086.
Peter Westfall. Kurtosis as peakedness, 1905-2014. rip. The American Statistician, 68, 07 2014. doi: 10. 101080000031305.204..9000
Shi Yin, Chao Liu, Zhiyong Zhang, Yiye Lin, Dong Wang, Javier Tejedor, Fang Zheng, and Yinguo Li. Noisy training for deep neural networks in speech recognition. EURASIP Journal on Audio, Speech, and Music Processing, 2015, 12 2015. doi: 10.11866/133333-014-0044474
Zhou Wang, C. Bovik, Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-612, April 2004. doi: 10. 1000


Reviews:
[{"review_id": "ryxfxYLKYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the effect of various data augmentation methods on image classification tasks. The Authors propose the Structural Similarity (SSIM) as a measure of the magnitude of the various types of data augmentation noise they consider. The Authors argue that SSIM is superior to PSNR as a measure of the intensity of the noise, across various noise types.\n\nThe idea of using SSIM as a unified measure for noise in the context data augmentation in images is novel AFAIK and is neat IMO, because as the authors point, SSIM provides a more perceptually-driven distance measure between images than RMS or PSNR. One of the results of the paper is that a SSIM value of 0.8 is a good rule of thumb for choosing the magnitude of the noise irrespectively to the type of noise. This is a useful and interesting result.\n\nNevertheless, at this stage I am inclined to reject the paper, because I feel that the main claim is not sufficiently substantiated. The argument that SSIM provides a more universal (less noise-type-dependent) measure of strength than, say, PSNR in the context of data augmentation is not substantiated in the experiments. While intuitively the claim makes sense to me, it is hard to draw this conclusion when SSIM is not compared to any other metric (RMS, PSNR) as measure of strength for data augmentation. \n\nWhy is a larger Kurtosis detrimental for measuring the strength of data argumentation? Is there any evidence that links low Kurtosis to a better measure data augmentation strength? This is another question that could be addressed if SSIM were compared to other data augmentation strength metrics.\n\nIMHO the paper could have been made much stronger if it had the analog of Figure 5 for other measures of the noise (e. g. PSNR, RMS). If the results showed that SSIM is superior to them, I would learn a useful and insightful lesson form the paper. In its current form, I feel that the Authors made the first step in a very interesting direction but did not go far enough to substantiate their claims.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJxOAIYJcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at analyzing the effect of injecting noise to images as data augmentation in training CNN for the image classification task. Based on the SSIM metric (which is shown to be a better metric than PSNR), different noise level on a set of different kinds of noise are explored. Experimental results on two sub-datasets of ImageNet suggest that Speckle noise would lead to better CNN models.\n\nEven though the simulations appear seemingly convincing, and the conclusion is somewhat interesting to me: speckle noise is recommended which contradicts the general usage of Gaussian noise. The results is too specific to both the model chosen resnet18v2 and also in the chosen dataset. Besides, my bigger concern is that the contribution of this work is highly limited, since there are a bunch of data augmentation techniques: cropping, flipping, color space transformation, rotation, noise injection, etc. Given this broad selection of data augmentation, as far as I know, noise injection is not the most effective nor popular one.  In fact, random cropping is the mostly used one that established past a few benchmark CNN models in the imageNet classification task, e.g, ResNet, DenseNet, etc. As such, it would be more convincing if it can be shown that proper noise injection can boost the recognition performance on the ImageNet task. \n\nMinors:\nAbstract line 8, and also introduces -> and also introduce   ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylWZ0s6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the different types of noises that could be added to the training image dataset while training an CNN model for classification. They study 5 different types of noise functions: Gaussian, Speckle, Salt and Pepper, Poisson, Occlusion.\n \nPros:\n1. Rigorously studying how to augment training data for CNN is important. \n\n\nCons:\n1. The primary question is novelty . - what is the research contribution of this dataset? They are running experiments of 5 different known noise functions, on two image datasets, on a single deep learning models. There are no fundamental research questions or hypothesis. This is a mere running of few experiments - known methods and known approaches. \n\n2. Are the results generalizable? Answer is no! The results on shown on two subsamples of Imagenet datasets for only ResNet 18 model. Maybe for this combination speckle noise (and not Gaussian, as pointed out in the comments by the authors) is better. How can the assure that for a different dataset, model, task combination the same speckle noise would perform better ? \n\n3. Improvement suggestion: What I would ideally look in this topic, is a method to automatically study the properties of the training data images (study the distribution) and conditional on this distribution recommend the best noise type and noise intensity. Thus, the whole story of noise injection could be made dynamic for a dataset, model, task combination\n\n4. Writing of the paper could be improved: 1. The need for noise based augmentation of is well known (Section 1). 2. The different kinds of noise functions are mostly text book knowledge (Section 2). 3. The different image quality metrics written here - MSE, PSNR, SSIM are also text book knowledge (Section 3). Overall, the first 4 pages of the paper are redundant and could be compressed into 1 page. Would like to read more on the experiments, analysis, and maybe automation of noise selection techniques in different kinds of tasks - segmentation, text classification, seq2seq etc.\n\n5. Choose very naive noise (or augmentation) functions: In general, the approach of augmentation of training data has evolved so much in the literature, that adding noise is hardly in practice. \n1. \"The Effectiveness of Data Augmentation in Image Classification using Deep Learning\" - Style Transformation\n2. \"Improving Deep Learning using Generic Data Augmentation\" - Geometric and Affine Transformation\nThus, the study on data augmentation should be performed across all these different transformation functions on training data and using only noise function is naive and is incomplete.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effect of various data augmentation methods on image classification tasks. The authors propose the structural similarity as a measure of the magnitude of the various types of data augmentation noise they consider and argue that it is outperforms PSNR as a measure of the intensity of the noise. The authors performed an empirical analysis showing that speckle noise leads to improved CNN models on two subsets of ImageNet. While there is merit in thoroughly analysing data augmentation schemes for training CNNs, the reviewers argued that the main claims of the work were not substantiated and the raised issues were not addressed in the rebuttal. I will hence recommend rejection of this paper.

Decision: Reject