Data Augmentation in Training CNNs: Injecting Noise to Images

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) find an ever-growing field of application throughout image and sound processing tasks , since the success of AlexNet ( Krizhevsky et al. , 2012 ) in the 2012 ImageNet competition . Yet , training these networks still keeps the need of an ” artistic ” touch : even the most cited state-of-the-art studies employ wildly varying set of solvers , augmentation and regularization techniques ( Domhan et al. , 2015 ) . In this study , one of the crucial data augmentation techniques , noise injection , will be thoroughly analysed to determine the correct way of application on image processing tasks . Adding noise to the training data is not a procedure that is unique to the training of neural architectures : additive and multiplicative noise has long been used in signal processing for regression-based methods , in order to create more robust models ( Saiz et al. , 2005 ) . The technique is also one of the oldest data augmentation methods employed in the training of feed forward networks , as analysed by Holmstrom & Koistinen ( 1992 ) , yet it is also pointed out in the same study that while using additive Gaussian noise is helpful , the magnitude of the noise can not be selected blindly , as a badly-chosen variance may actually harm the performance of the resulting network ( see Gu & Rigazio ( 2014 ) and Hussein et al . ( 2017 ) for more examples ) . The main reasons for noise injection to the training data can be listed as such in a non-excluding manner : first of all , injection of any noise type makes the model more robust against the occurrence of that particular noise over the input data ( see Braun et al . ( 2016 ) and Saiz et al . ( 2005 ) for further reference ) , such as the cases of Gaussian additive noise in photographs , and Gaussian-Poisson noise on low-light charge coupled devices ( Bovik , 2005 ) . Furthermore , it is shown that the neural networks optimize on the noise magnitude they are trained on ( Yin et al. , 2015 ) . Therefore , it is important to choose the correct type and level of the noise to augment the data during training . Another reason for noise addition is to encourage the model to learn the various aspects of each class by occluding random features . Generally , stochastic regularization techniques embedded inside the neural network architectures are used for this purpose , such as Dropout layers , yet it is also possible to augment the input data for such purposes as in the example of ” cutout ” regularization proposed by Devries & Taylor ( 2017 ) . The improvement of the generalization capacity of a network is highly correlated with its performance , which can be scored by the accuracy over a predetermined test set . There has been similar studies conducted on the topic , with the example of Koziarski & Cyganek ( 2017 ) which focuses on the effects of noise injection on the training of deep networks and the possible denoising methods , yet they fail to provide a proper methodology to determine the level of noise to be injected into the training data , and use PSNR as the comparison metric between different noise types which is highly impractical ( see Section 3 ) . To resolve these issues , this study focuses on the ways to determine which noise types to combine the training data with , and which levels , in addition to the validity of active noise injection techniques while experimenting on a larger set of noise models . In the structure of this work , the effect of injecting different types of noises into images for varying CNN architectures is assessed based on their performance and noise robustness . Their interaction and relationship with each other are analyzed over ( also noise-injected ) validation sets . Finally as a follow-up study , proper ways on adding or applying noise to a CNN for image classification tasks are discussed . 2 DIFFERENT TYPES OF NOISE . Noise can be -somewhat broadly- defined as unwanted component of the image ( Bovik , 2005 ) . It can be sourced from the environment at which the image is taken , the device utilized to take the image , or the medium of communication that is used to convey the information of image from source to the receiver . According to its properties and nature , noise and image can be analytically decomposed as additive or multiplicative , but some of the noise types can not be described by neither of these classes . 2.1 ADDITIVE NOISE . Let f ( x ) denote an image signal . This signal can be decomposed into two components in an additive manner as f ( x ) = g ( x ) +n ( x ) where g ( x ) denoting the desired component of the image , and n ( x ) standing for the unwanted noise component . Most commonly encountered variant of this noise class is Gaussian noise , whose multivariate probability density function can be written as : fn ( x ) = 1√ ( 2π ) n|Σ| exp ( −1 2 ( x−m ) TΣ−1 ( x−m ) ) ( 1 ) where m and Σ denoting the n-dimensional mean vector and symmetric covariance matrix with rank n , respectively . In images , the mean vector m is generally zero , therefore the distribution is centered and the magnitude is controlled by the variance . This study also follows these assumptions . 2.2 MULTIPLICATIVE NOISE . Again , let f ( x ) denote an image signal . This signal can also be decomposed into respective desired and noise components as f ( x ) = g ( x ) ( 1 + n ( x ) ) . The noise component in this model is called multiplicative noise . The most common variant in this case is called speckle noise , which may have different density functions , and in this study Gaussian is assumed . Similar with the additive noise , the mean is assumed to be 0 and the magnitude refers to the variance . Speckle noise can be encountered in coherent light imaging such as in the cases of SAR images ( Ding et al. , 2016 ) and images with laser-based illumination , but they may also be observed in other digital images ( Bovik , 2005 ) . 2.3 OTHER TYPES OF NOISE . There exists many other noise instances that can not be modeled by additive or multiplicative decompositions . Most common of these types are listed below , whose effects on the performances of CNNs are also analysed in this study . Salt and pepper ( S & P ) noise . This noise manifests itself as a basic image degradation , for which only a few pixels in an image are noisy , but they are extremely noisy in a way that pixels are either completely black or white . The magnitude of this noise is the probability of a pixel to be completely black ( i.e . pepper ) , completely white ( i.e . salt ) , or stay unchanged . The probabilities for pepper and salt cases are assumed to be equal , and total probability of the degradation of a pixel is referred as the the magnitude of the noise . Poisson noise . Also referred as photon counting noise or shot noise , this noise type has a particular probability density function : fn ( k ) = e−λλk k ! ( 2 ) where λ stands for both the variance and the mean of the distribution . As Poisson noise is signaldependant , it does not have a direct magnitude parameter similar to other noise types , therefore a magnitude factor c is used that divides the intensity values of all pixels from which the distribution is sampled , and returned to the original range by multiplying to the same factor . Occlusion noise . Although it is not generally referred as a noise type , occlusion of important features can happen for a number of reasons in an image : image may be cropped , damaged or a particular object in it may be hidden by an obstacle such as a tree . This noise is realized with zerointensity squares appearing on the image , and the magnitude is determined according to the size of the square as the shape of the occluding object does not have a large impact on the final performance of the model ( Devries & Taylor , 2017 ) . As listed in this section , five different types of noise and their various combinations are added or applied to the images , with varying magnitudes . Robustness against each of these noise types is also assessed . 3 CHOOSING THE RIGHT METRIC : PSNR VS SSIM . Different types of noise are easy to compare when they are sampled from the same distribution , such as in the case of additive Gaussian noise and speckle noise . However , it is sometimes impossible to assess two different metrics in the same context because of varying magnitude parameters and sensitivity of the model to each of these types . In this case , it becomes imperative to use an intermediary metric that will ease the comparison process for noise types . In general , Peak Signal-to-Noise Ratio ( PSNR ) , and similarly Mean Squared Error ( MSE ) are the most commonly used quality metrics in the image processing field . For an 8-bit two-dimensional MxN image f̂ ( n1 , n2 ) and its noise-free counterpart f ( n1 , n2 ) , the MSE is defined as MSE = 1 MN M−1∑ n1=0 N−1∑ n2=0 [ f ( n1 , n2 ) − f̂ ( n1 , n2 ) ] 2 . ( 3 ) From above definition , PSNR ( in dB ) can be derived . PSNR = 10 log10 { 2552 MSE } ( 4 ) There are several limitations of using PSNR as the image quality metric of a data set : it is shown that PSNR loses its validity as a quality metric when the content and/or codec of the images are different as in that case the correlation between subjective quality and PSNR is highly reduced ( Huynh-Thu & Ghanbari , 2008 ) . Also , even though the sensitivity of PSNR to Gaussian noise is very high , the metric is unable to present similar performance for different types of perturbation ( Avcibas et al. , 2002 ) ( Horé & Ziou , 2010 ) . There exists another widely accepted image quality metric called Structural Similarity ( SSIM ) , which resolves or alleviates some of the above-listed problems . The Structural Similarity between two non-negative image signals x and y , whose means , standard deviations and covariance are denoted by µx , µy , σx , σy and σxy respectively , can be expressed as : SSIM = ( 2µxµy + C1 ) ( 2σxy + C2 ) ( µ2x + µ 2 y + C1 ) ( σ 2 x + σ 2 y + C2 ) ( 5 ) where C1 and C2 are constants to avoid instability ( Zhou Wang et al. , 2004 ) . This metric combines luminance , contrast and structure information in order to provide an assessment of similarity in the range from 0 to 1 , where 1 stands for highest similarity . In image classification tasks , the objective is mostly to classify the depicted objects or beings according to the human perception . Therefore , it is crucial for the used metric to be consistent with human opinions : it is shown in several studies that SSIM provides a quality metric closer to average opinion of public than PSNR ( Kim et al. , 2019 ) ( Zhou Wang et al. , 2004 ) . Furthermore , when sampled from the same noise distributions over identical images of a data set , distribution of PSNR values of the noisy images have significantly high kurtosis than their SSIM counterparts in every noise type except S & P . This behavior is demonstrated over a subset of ImageNet dataset ( Deng et al. , 2009 ) called Imagewoof ( can be accessed via github.com/fastai/imagenette ) , with 10 classes and 12454 total number of images on the training set . Each type of noise listed in Section 2 is applied to each image , sampling from the same distribution , and quality metrics are recorded . The utilized noise magnitudes are 0.2 variance for Gaussian noise , 0.2 variance for speckle noise , 0.05 total probability for S & P noise , 0.2 scaling factor for Poisson noise , and 0.3 relative side length for occlusion noise ; which are all chosen to provide sensible values from the metrics . The distribution of the metrics can be seen at Figure 1 over the axes of each subfigure , and the kurtosis values are noted in Table 1 . This is interpreted as PSNR having propensity to produce more outliers than SSIM for the same levels of noise ( Westfall , 2014 ) . For the reasons listed above , SSIM will be used as the primary metric throughout this study .

Summary: This paper aims at analyzing the effect of injecting noise to images as data augmentation in training CNN for the image classification task. Based on the SSIM metric (which is shown to be a better metric than PSNR), different noise level on a set of different kinds of noise are explored. Experimental results on two sub-datasets of ImageNet suggest that Speckle noise would lead to better CNN models.