Data Augmentation in Training CNNs: Injecting Noise to Images

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) find an ever-growing field of application throughout image and sound processing tasks , since the success of AlexNet ( Krizhevsky et al. , 2012 ) in the 2012 ImageNet competition . Yet , training these networks still keeps the need of an ” artistic ” touch : even the most cited state-of-the-art studies employ wildly varying set of solvers , augmentation and regularization techniques ( Domhan et al. , 2015 ) . In this study , one of the crucial data augmentation techniques , noise injection , will be thoroughly analysed to determine the correct way of application on image processing tasks . Adding noise to the training data is not a procedure that is unique to the training of neural architectures : additive and multiplicative noise has long been used in signal processing for regression-based methods , in order to create more robust models ( Saiz et al. , 2005 ) . The technique is also one of the oldest data augmentation methods employed in the training of feed forward networks , as analysed by Holmstrom & Koistinen ( 1992 ) , yet it is also pointed out in the same study that while using additive Gaussian noise is helpful , the magnitude of the noise can not be selected blindly , as a badly-chosen variance may actually harm the performance of the resulting network ( see Gu & Rigazio ( 2014 ) and Hussein et al . ( 2017 ) for more examples ) . The main reasons for noise injection to the training data can be listed as such in a non-excluding manner : first of all , injection of any noise type makes the model more robust against the occurrence of that particular noise over the input data ( see Braun et al . ( 2016 ) and Saiz et al . ( 2005 ) for further reference ) , such as the cases of Gaussian additive noise in photographs , and Gaussian-Poisson noise on low-light charge coupled devices ( Bovik , 2005 ) . Furthermore , it is shown that the neural networks optimize on the noise magnitude they are trained on ( Yin et al. , 2015 ) . Therefore , it is important to choose the correct type and level of the noise to augment the data during training . Another reason for noise addition is to encourage the model to learn the various aspects of each class by occluding random features . Generally , stochastic regularization techniques embedded inside the neural network architectures are used for this purpose , such as Dropout layers , yet it is also possible to augment the input data for such purposes as in the example of ” cutout ” regularization proposed by Devries & Taylor ( 2017 ) . The improvement of the generalization capacity of a network is highly correlated with its performance , which can be scored by the accuracy over a predetermined test set . There has been similar studies conducted on the topic , with the example of Koziarski & Cyganek ( 2017 ) which focuses on the effects of noise injection on the training of deep networks and the possible denoising methods , yet they fail to provide a proper methodology to determine the level of noise to be injected into the training data , and use PSNR as the comparison metric between different noise types which is highly impractical ( see Section 3 ) . To resolve these issues , this study focuses on the ways to determine which noise types to combine the training data with , and which levels , in addition to the validity of active noise injection techniques while experimenting on a larger set of noise models . In the structure of this work , the effect of injecting different types of noises into images for varying CNN architectures is assessed based on their performance and noise robustness . Their interaction and relationship with each other are analyzed over ( also noise-injected ) validation sets . Finally as a follow-up study , proper ways on adding or applying noise to a CNN for image classification tasks are discussed . 2 DIFFERENT TYPES OF NOISE . Noise can be -somewhat broadly- defined as unwanted component of the image ( Bovik , 2005 ) . It can be sourced from the environment at which the image is taken , the device utilized to take the image , or the medium of communication that is used to convey the information of image from source to the receiver . According to its properties and nature , noise and image can be analytically decomposed as additive or multiplicative , but some of the noise types can not be described by neither of these classes . 2.1 ADDITIVE NOISE . Let f ( x ) denote an image signal . This signal can be decomposed into two components in an additive manner as f ( x ) = g ( x ) +n ( x ) where g ( x ) denoting the desired component of the image , and n ( x ) standing for the unwanted noise component . Most commonly encountered variant of this noise class is Gaussian noise , whose multivariate probability density function can be written as : fn ( x ) = 1√ ( 2π ) n|Σ| exp ( −1 2 ( x−m ) TΣ−1 ( x−m ) ) ( 1 ) where m and Σ denoting the n-dimensional mean vector and symmetric covariance matrix with rank n , respectively . In images , the mean vector m is generally zero , therefore the distribution is centered and the magnitude is controlled by the variance . This study also follows these assumptions . 2.2 MULTIPLICATIVE NOISE . Again , let f ( x ) denote an image signal . This signal can also be decomposed into respective desired and noise components as f ( x ) = g ( x ) ( 1 + n ( x ) ) . The noise component in this model is called multiplicative noise . The most common variant in this case is called speckle noise , which may have different density functions , and in this study Gaussian is assumed . Similar with the additive noise , the mean is assumed to be 0 and the magnitude refers to the variance . Speckle noise can be encountered in coherent light imaging such as in the cases of SAR images ( Ding et al. , 2016 ) and images with laser-based illumination , but they may also be observed in other digital images ( Bovik , 2005 ) . 2.3 OTHER TYPES OF NOISE . There exists many other noise instances that can not be modeled by additive or multiplicative decompositions . Most common of these types are listed below , whose effects on the performances of CNNs are also analysed in this study . Salt and pepper ( S & P ) noise . This noise manifests itself as a basic image degradation , for which only a few pixels in an image are noisy , but they are extremely noisy in a way that pixels are either completely black or white . The magnitude of this noise is the probability of a pixel to be completely black ( i.e . pepper ) , completely white ( i.e . salt ) , or stay unchanged . The probabilities for pepper and salt cases are assumed to be equal , and total probability of the degradation of a pixel is referred as the the magnitude of the noise . Poisson noise . Also referred as photon counting noise or shot noise , this noise type has a particular probability density function : fn ( k ) = e−λλk k ! ( 2 ) where λ stands for both the variance and the mean of the distribution . As Poisson noise is signaldependant , it does not have a direct magnitude parameter similar to other noise types , therefore a magnitude factor c is used that divides the intensity values of all pixels from which the distribution is sampled , and returned to the original range by multiplying to the same factor . Occlusion noise . Although it is not generally referred as a noise type , occlusion of important features can happen for a number of reasons in an image : image may be cropped , damaged or a particular object in it may be hidden by an obstacle such as a tree . This noise is realized with zerointensity squares appearing on the image , and the magnitude is determined according to the size of the square as the shape of the occluding object does not have a large impact on the final performance of the model ( Devries & Taylor , 2017 ) . As listed in this section , five different types of noise and their various combinations are added or applied to the images , with varying magnitudes . Robustness against each of these noise types is also assessed . 3 CHOOSING THE RIGHT METRIC : PSNR VS SSIM . Different types of noise are easy to compare when they are sampled from the same distribution , such as in the case of additive Gaussian noise and speckle noise . However , it is sometimes impossible to assess two different metrics in the same context because of varying magnitude parameters and sensitivity of the model to each of these types . In this case , it becomes imperative to use an intermediary metric that will ease the comparison process for noise types . In general , Peak Signal-to-Noise Ratio ( PSNR ) , and similarly Mean Squared Error ( MSE ) are the most commonly used quality metrics in the image processing field . For an 8-bit two-dimensional MxN image f̂ ( n1 , n2 ) and its noise-free counterpart f ( n1 , n2 ) , the MSE is defined as MSE = 1 MN M−1∑ n1=0 N−1∑ n2=0 [ f ( n1 , n2 ) − f̂ ( n1 , n2 ) ] 2 . ( 3 ) From above definition , PSNR ( in dB ) can be derived . PSNR = 10 log10 { 2552 MSE } ( 4 ) There are several limitations of using PSNR as the image quality metric of a data set : it is shown that PSNR loses its validity as a quality metric when the content and/or codec of the images are different as in that case the correlation between subjective quality and PSNR is highly reduced ( Huynh-Thu & Ghanbari , 2008 ) . Also , even though the sensitivity of PSNR to Gaussian noise is very high , the metric is unable to present similar performance for different types of perturbation ( Avcibas et al. , 2002 ) ( Horé & Ziou , 2010 ) . There exists another widely accepted image quality metric called Structural Similarity ( SSIM ) , which resolves or alleviates some of the above-listed problems . The Structural Similarity between two non-negative image signals x and y , whose means , standard deviations and covariance are denoted by µx , µy , σx , σy and σxy respectively , can be expressed as : SSIM = ( 2µxµy + C1 ) ( 2σxy + C2 ) ( µ2x + µ 2 y + C1 ) ( σ 2 x + σ 2 y + C2 ) ( 5 ) where C1 and C2 are constants to avoid instability ( Zhou Wang et al. , 2004 ) . This metric combines luminance , contrast and structure information in order to provide an assessment of similarity in the range from 0 to 1 , where 1 stands for highest similarity . In image classification tasks , the objective is mostly to classify the depicted objects or beings according to the human perception . Therefore , it is crucial for the used metric to be consistent with human opinions : it is shown in several studies that SSIM provides a quality metric closer to average opinion of public than PSNR ( Kim et al. , 2019 ) ( Zhou Wang et al. , 2004 ) . Furthermore , when sampled from the same noise distributions over identical images of a data set , distribution of PSNR values of the noisy images have significantly high kurtosis than their SSIM counterparts in every noise type except S & P . This behavior is demonstrated over a subset of ImageNet dataset ( Deng et al. , 2009 ) called Imagewoof ( can be accessed via github.com/fastai/imagenette ) , with 10 classes and 12454 total number of images on the training set . Each type of noise listed in Section 2 is applied to each image , sampling from the same distribution , and quality metrics are recorded . The utilized noise magnitudes are 0.2 variance for Gaussian noise , 0.2 variance for speckle noise , 0.05 total probability for S & P noise , 0.2 scaling factor for Poisson noise , and 0.3 relative side length for occlusion noise ; which are all chosen to provide sensible values from the metrics . The distribution of the metrics can be seen at Figure 1 over the axes of each subfigure , and the kurtosis values are noted in Table 1 . This is interpreted as PSNR having propensity to produce more outliers than SSIM for the same levels of noise ( Westfall , 2014 ) . For the reasons listed above , SSIM will be used as the primary metric throughout this study .

Summary: The paper studies the effect of various data augmentation methods on image classification tasks. The Authors propose the Structural Similarity (SSIM) as a measure of the magnitude of the various types of data augmentation noise they consider. The Authors argue that SSIM is superior to PSNR as a measure of the intensity of the noise, across various noise types.