Image Denoising with Feature Attention
Tags: Keras, Eam, Tensorflowjs, Image Processing, Computer Vision

This is my Second medium Story. Hope you have fun reading it as much as I enjoyed writing it for the audience.

Table of Contents:

Introduction Business Problem Mapping to ML/DL problem Source of data Existing approaches Model Explanation Implementation details Results Future Work Conclusion References Profile

1.Introduction

With the Explosion of in the number of images taken everyday, the demand for more accurate and pleasing Images is Increasing.However,the Images Captured by modern cameras are inevitable degraded by noise, which leads to deteriorated visual image quality. Therefore, work is required to reduce noise without losing image features. With the presence of noise, possible subsequent Image processing tasks, such as video processing, image analysis and tracking are adversely affected. Therefore Image denoising plays a vital role in modern image processing system. Image denoising is to remove noise from the image, so as to restore the true image.

However noise, edge and texture are the high frequency components, it is difficult to distinguish them in the process of denoising and denoise image can inevitably lose some details. Overall recovering meaningful information from noisy image in the process of noise removal to obtain high quality images is an important problem nowadays.

2 Business Problem:

We are capturing the Images on day to day basis via Camera or Mobile phone, but sometimes the pictures quality get blurred and not properly visible to the person, so in that case the pictures gets noisy and for that we get the picture to be clean and properly visible, so for this task we are designing the use case where we are getting the Noisy images and we design an algorithm which can denoised the images captured by Mobile phone or Cameras.It can be used where we are capturing the bunch of images and from that more number of images contribute to noise then in that case if we have automated solution then we get all the noisy images to be clean.

Challenges to solve:

Provide a different type of Noise to an Image of required size and then remove the noise from the image, so can we built an algorithm that automatically converts a Noisy Image to denoised image?Challenging,right?

But if solved correctly , then it can eliminate the human interference as well as used any tool for converting the noisy images to be denoised.

3.Mapping the real world problem to a ML/DL Problem:

For given a Noisy Images of different Noise we have to denoised it after applying some of the techniques of deep learning to solve this problem. It is a deep learning problem, where we have multiple Noisy Images as a Input and result as a denoised images.

Performance Metric:

Here, we are using performance metrics as accuracy and loss as mean absolute error and optimizer as Adam.

Real-world/Business objectives and constraints:

low-latency requirement is not a problem.

Interpretability is important.

parallel image processing, improving the throughput time of algorithms to process large and heterogeneous data sets to make them usable for real-time or low-latency procedures.

4.Source of Data:

Data:

We have a data of color Images and we make them as Input file path as Noisy Image path and clean Image path.

Data file’s information:

Noisy Image file path:It is the path of the Noisy Image file.

Clean Image file path:It is the path of the Clean Image file.

5.Existing approaches:

CNN based approaches for real image denoising employ two-stage models; we present the first model that provides state-of-the-art results using only one stage. our model is the first to incorporate feature attention in denoising. Most current models connect the weight layers consecutively; and so increasing the depth will not help improve performance Also, such networks can suffer from vanishing gradients. We experiment on three synthetic image datasets and four real-image noise datasets to show that our model achieves state-of-the-art results on synthetic and real images quantitatively and qualitatively. The popularity of convolutional neural networks (CNNs), image denoising algorithms have achieved a performance boost Notable denoising neural networks, DnCNN and IrCNN predict the residue present in the image instead of the denoised image as the input to the loss function is ground truth noise as compared to the original clean image. Both networks achieved better results despite having a simple architecture where repeated blocks of convolutional, batch normalization and ReLU activations are used. Furthermore, IrCNN and DnCNN are dependent on blindly predicted noise i.e. without taking into account the underlying structures and textures of the noisy image.

6.Model Explanation:

1.We are Enhancing the performance of the Model using EAM archietecture where we have Five convolution layer and build the model .

2.In the EAM architecture there are mainly three main modules like feature extraction, feature learning residual on the residual module, and reconstruction.

3.Let us consider x is a noisy input image and yˆ is the denoised output image. Our feature extraction module is composed of only one convolutional layer to extract initial features f0 from the noisy input

4.it performs convolution on the noisy input image. Next, f0 is passed on to the feature learning residual on the residual module.Our network has small depth, but provides a wide receptive field through kernel dilation in each EAM initial two branch convolutions. The output features of the final layer are fed to the reconstruction module, which is again composed of one convolutional layer.

5. Each EAM is composed of D blocks followed by feature attention. Due to the residual on the residual architecture, very deep networks are now possible that improve denoising performance; however, we restrict our model to four EAM modules only. The first part of EAM covers the full receptive field of input features, followed by learning on the features; then the features are compressed for speed, and finally a feature attention module enhances the weights of important features from the maps. The first part of EAM is realized using a novel merge-and-run unit. The input features branched and are passed through two dilated convolutions, then concatenated and passed through another convolution. Next, the features are learned using a residual block of two convolutions while compression is achieved by an enhanced residual block (ERB) of three convolutional layers. The last layer of ERB flattens the features by applying a 1×1 kernel. Finally, the output of the feature attention unit is added to the input of EAM.

7. Implementation Details:

First of all we are loading all the images from the location in google colab . Create the structured data for Noisy Images from the original Images. Creating the 12 Noisy Images from the Original Image and create the training data. Used the Image generator for Original Images and Noisy Images for Train and Validation and test. Using this above generator feed that to neural network and create the CNN network with EAM architecture discussed in the paper. Fitting the model and predict from the saved model. Saved the weights of the model.

7.1)Noise Creation:

Here we are creating different types of noise on the ground truth image like Black Pixel Noise, WhitePixelNoise,LeftUpper Noise, RightUpperNoise, LeftBottomNoise, RightBottomNoise,FullNoise etc.

BlackPixelNoise:It is random Noise on the ground truth image with Black pixels on them.

WhitePixelNoise:It is random Noise on the ground truth image with Whitepixels on them.

LeftUpperNoise:It is random Position on the Top left side of the image with White pixels on it.

RightUpperNoise:It is a random Position on the top right side of the image with White pixels on it.

LeftBottomNoise:It is a random Position on the Botttom left of the image with White Pixels on it.

RightBottomNoise:It is a random Position on the Botttom rightof the image with White Pixels on it.

FullNoise:It is type of noise which covers pixels noise all over the image.

7.2)Applying Black Pixel Noise to the Image:

def black_pixels(image,n1): Sp_NoiseImg=image SP_NoiseNum=int(n1*image. Shape[0]*image.shape[1]) for i in range(SP_NoiseNum): randX=np.random.randint(0,image.shape[0]-1) randY=np.random.randint(0,image.shape[1]-1) Sp_NoiseImg[randX,randY]=0 return Sp_NoiseImg

7.3)Applying White Pixel Noise to the Image:

def White_pixels(image,n1): Sp_NoiseImg=image SP_NoiseNum=int(n1*image.shape[0]*image.shape[1]) for i in range(SP_NoiseNum): randX=np.random.randint(0,image.shape[0]-1) randY=np.random.randint(0,image.shape[1]-1) Sp_NoiseImg[randX,randY]=255 return Sp_NoiseImg

7.4)Applying UpperLeft Side Noise to the Image:

def UpperLeft(img,p1, p2): input_shape = (218,178,3) orig_image_upper_left=np.zeros((input_shape)) orig_image_upper_left[:p1,:p2,:]=np.random.random((p1,p2,3)) orig_img_noisy=img+orig_image_upper_left return orig_img_noisy

7.5)Applying UpperRight Side Noise to the Image:

def UpperRight(img, s1,s2): input_shape = (218,178,3) orig_image_noisy_Up_Right=np.zeros((input_shape)) orig_image_noisy_Up_Right[:s1,s2:,:]=np.random.random((s1,s2,3)) orig_img_noisy_new1=img+orig_image_noisy_Up_Right return orig_img_noisy_new1

7.6)Applying LowerLeft Side Noise to the Image:

def LowerLeft(img, r1,r2):

input_shape = (218,178,3) orig_image_noisy_Bottom_Left=np.zeros((input_shape))

orig_img_noisy_new2=img orig_image_noisy_Bottom_Left return orig_img_noisy_new2

7.7)Applying LowerRight Side Noise to the Image:

def LowerRight(img,a1,a2): input_shape = (218,178,3) orig_image_noisy_Bottom_Right=np.zeros((input_shape) orig_image_noisy_Bottom_Right[a1:,a2:,:]=

np.random.random((a1,a2,3)) orig_img_noisy_new3=img+orig_image_noisy_Bottom_Right

return orig_img_noisy_new3

7.8)Applying Total Noise to all over the Image:

def total_noise(img,pr): orig_img_noisy=img+pr*np.random.normal(loc=0.0,scale=1.0,

size=img.sh ape) orig_img_noisy=np.clip(orig_img_noisy,0,1) return orig_img_noisy

After creating the Noise we are making the structured data for Training the model and split the data into Train,test and validation. And we are creating 12 different types of noise per single original image, so in my total data i have 48000 images for training the model and in test and validation i have 6000 and 6000 images respectively for prediction. after getting the data from train, test and validation i stored them as a csv and put it into the drive.

7.9)Reading Training data from the stored Csv and make the label as None:

Here, reading the training data of 48000 images from the file located at google drive folder and set the label column as None , as we don’t require the label for this task to predict, so in the below code I read the file from the location and set the label as none.

7.10)Reading Training Data from the CSV file:

After making the label as None we divide the training data into types Pure Images and Noisy Images, where Pure Images file contain the Pure Image file path and label as None and Noisy Images file contain the Noisy Image file path and label as None.

7.11)Reading the Validation data from the csv:

Same thing we are doing for Validation data, where we divide the whole data into two types Pure validation Images, where we have Pure Image file path and label as None and also Noisy validation Images,where we have Noisy Image file path as below.

7.12)Creating ImageDataGenerators for Train and validation data:

In this Step I am creating the Imagedatagenerator for Noisy Image file path and Pure Image file path using Imagedatagenerator, which contains the name of the dataframe of Pure Image file path from the directory and it is also same for Noisy Image file path using Imagedatagenerator , which contains the name of the dataframe of Noisy Image file path. In this same way we are creating the Imagedatagenerators for validation as well as test data as below.

Pure_generator = train_datagen.flow_from_dataframe(train_Pure_df, directory='/content/drive/My Drive/Noisy/', x_col='PureImagePath', y_col='label', batch_size=1, seed=42, shuffle=False, class_mode="raw", target_size=(218,178), color_mode='rgb', #rescale=1.0/255, interpolation='nearest') Noisy_generator=train_datagen.flow_from_dataframe(train_Noisy_df, directory='/content/drive/My Drive/Noisy/', x_col='NoisyImagePath', y_col='label', batch_size=1, seed=42, shuffle=False, class_mode="raw", target_size=(218,178), color_mode='rgb', #rescale=1.0/255, interpolation='nearest')

7.13)Combining the Training Generator for Original Image and Noisy Image:

Here , we are combining the Training Imagedatagenerator of PureImagefilepath and NoisyImagefilepath, which creates the pair of images as one image of noisy with their respective pure Image and shuffle the data and prefetch it, same thing we are doing for validation data using validation Imagedatagenerator as below.

def generator3(): while True: g1=Pure_generator.next() g2=Noisy_generator.next() yield g2[0].reshape(218,178,3),g1[0].reshape(218,178,3)

gen4=generator3() #Calling the generator object

temp1=next(gen4) data1=tf.data.Dataset.from_generator(generator3,output_types=(tf.float32,tf.float32),output_shapes=((218,178,3),(218,178,3))) data1_series=data1.shuffle(1000).repeat().batch(32,drop_remainder=True).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)

7.14)Creating a Basic CNN archietecture for fitting this model:

Here we are creating the basic architecture of CNN sequential model with 3 layers with Maxpool and dropout layers as below.

from tensorflow.keras import layers model = tf.keras.Sequential([ layers.Conv2D(32, 3, input_shape=(218,178,3),padding='same',activation='relu'), layers.Conv2D(32, 3, padding='same'), layers.MaxPool2D(1), layers.Dropout(0.5), layers.Conv2D(64, 3, input_shape=(218,178,3),padding='same',activation='relu'), layers.Conv2D(64, 3, padding='same',activation='relu'), layers.MaxPool2D(1), layers.Dropout(0.5), layers.Conv2D(3, 3, batch_input_shape=(218,178,3),padding='same',activation='relu'), layers.Dense(3,activation='sigmoid') ]) model.summary()

7.15)Compile the Model:

Here we are compiling the Model using categorical crossentropy and metrics which evaluate the performance of the Model as accuracy.

model.compile(loss = "categorical_crossentropy", optimizer = optimizer, metrics = ["accuracy"])

7.16)ModelCheckpoint and Early Stopping Criteria:

We are saving the model on each iteration using the Mode checkpoint and early stopping criteria as below.

checkpointer=ModelCheckpoint(filepath='/content/Noisy/Image_Noisy/weights.hdf5',verbose=1,save_best_only=True ) early_stopping = EarlyStopping(monitor = 'val_loss', patience = 50)

7.17)Train the CNN Model:

Here we are training the model with batch size as 32 with eatch iteration with 62 steps per epoch and 5 epochs i am iterated to make the model accuracy better as below.

7.18)Save the Model:

After training the model we are saving the model on the directory in the google drive.

model.save('/content/drive/MyDrive/Noisy/model_Basic_CNN_Updated.h5')

7.19)Visualize the Prediction on the Testing data:

On the above Prediction it is not properly denoised the image so we are enhancing the Network using EAM architecture(Enhanced Attention Mechanism).

8.Results:

Here we are getting the results as one image Noisy can be denoised using the EAM architecture that we are implementing above. Most of the noise are cleared from the Noisy image and it gives result as a denoised and clean image as compared to the basic CNN network, where we are implementing the CNN network on the Noisy Image and results as denoised Image.

Create the EAM archietecture:

Below we are creating the EAM archietecture as discussed above.

def EAMArchietecture(input): Conv_1_1=layers.Conv2D(32, (3,3),dilation_rate=1,padding='same',activation='relu')(input) Conv_1_2=layers.Conv2D(32,

(3,3),dilation_rate=2,padding='same',activation='relu')(Conv_1_1) Conv_2_1=layers.Conv2D(32,

(3,3),dilation_rate=3,padding='same',activation='relu')(input) Conv_2_2=layers.Conv2D(32,

(3,3),dilation_rate=4,padding='same',activation='relu')(Conv_2_1) Concat_1_2=Concatenate(axis=-1)([Conv_1_2,Conv_2_2]) Concat_1_2_conv=layers.Conv2D(32,

(3,3),padding='same',activation='relu')(Concat_1_2) add_1=layers.Add()([Concat_1_2_conv,input]) Conv_3_1=layers.Conv2D(32,(3,3),padding='same',activation='relu')

(add_1) Conv_3_2=layers.Conv2D(32,(3,3),padding='same')(Conv_3_1) add_2=layers.Add()([Conv_3_2,add_1]) add_2=layers.Activation('relu')(add_2) Conv_4_1=layers.Conv2D(32,(3,3),padding='same',activation='relu')

(add_2) Conv_4_2=layers.Conv2D(32,(3,3),padding='same',activation='relu')

(Conv_4_1) Conv_4_3=layers.Conv2D(32,(1,1),padding='same')(Conv_4_2) add_3=layers.Add()([Conv_4_3,add_2]) add_3=layers.Activation('relu')(add_3) Conv_5_1=layers.GlobalAveragePooling2D()(add_3) Conv_5_1=tf.expand_dims(Conv_5_1,1) Conv_5_1=tf.expand_dims(Conv_5_1,1) Conv_5_2=layers.Conv2D(4,(3,3),padding='same',activation='relu')

(Conv_5_1) Conv_5_3=layers.Conv2D(32,

(3,3),padding='same',activation='sigmoid')(Conv_5_2) multiply = layers.Multiply()([Conv_5_3, add_3]) return multiply input=layers.Input(shape=(218,178,3)) f_1=layers.Conv2D(32,(3,3),padding='same')(input) eam_1=EAMArchietecture(f_1) eam_2=EAMArchietecture(eam_1) eam_3=EAMArchietecture(eam_2) eam_4=EAMArchietecture(eam_3) conv=layers.Conv2D(3,(3,3),padding='same')(eam_4) add_4=layers.Add()([conv,input])

After creating the EAM architecture, based on that we are training the model , where we passed the training data as training image generator and for validation data, we are passing the validation generator as below

Visualize the Prediction& PSNR Calculation:

After running the epochs of 100 times, we are visualize the output of the EAM model using the Noisy Image which will be predicted as denoised Images as below.

The EAM archietecture is much more efficient in denoised images as compated to the Basic CNN archietecture.

Calculate the PSNR value:

Here we are calculate the PSNR value between Noisy Image and Ground truth Image, which gives the result in the percentage of the portion has denoised from the Noisy Image.

Predict the Images based on the PSNR value:

Here,getting the 33%PSNR value from the list of Images as baseline and then check for the greater then PSNR and less then PSNR value belonging to which of this Images and visualize the belonging images within the PSNR value using the histogram by creating the bins.

Divide the PSNR values into the bins:

Here, we are diving the PSNR values into the bins with respective to the baseline value as 0.33 and get the less then that and greater then that and visualize.

_, bins = pd.cut(d1[d1.T_PSNR < s3].T_PSNR, bins=10,retbins=True) new_bins = pd.np.append(bins,float(max(d1.T_PSNR))) out = pd.cut(d1.T_PSNR,bins=new_bins) d1.T_PSNR.hist(bins=new_bins)

_, bins1 = pd.cut(d1[d1.T_PSNR > s3].T_PSNR, bins=10,retbins=True) new_bins1 = pd.np.append(bins1,float(max(d1.T_PSNR))) out1 = pd.cut(d1.T_PSNR,bins=new_bins1,duplicates='drop') d1.T_PSNR.hist(bins=new_bins1)

9.Future Work:

Right Now We are Improvising the Basic CNN Model to EAM archietecture and it gives the Better Image denoising but not fully so now again we are using another approach like DNCNN, FFDNET, CBDNET etc to enhance the performance of the Images.

10.Conclusion:

This is My Second Deep Learning Case Study and it was based on the Research paper and i got a pretty decent result of clean images. I got to learn many techniques of Computer Vision via this use case while optimizing the architecture. It always feels great to read from books and blogs about the methodologies, but unless you do it on your own and learn things from scratch, you won’t get a good idea about how you can solve this in practical.

While some code snippets are included within the blog, for the full code you can check out this https://github.com/Kenil706/Image-Denoising/tree/master

11.References:

12.Profile:

You can also find and connect with me on Linkdin:https://www.linkedin.com/in/kenil-shah-06613b97/

and Github:https://github.com/Kenil706/Image-Denoising/tree/master