Training a Neural Network to identify Pneumonia in X-rays
By: Dipta Roy
Tags: Convolution Neural Net, Convolutional Network, Neural Networks, Deep Learning, Pneumonia

During the COVID era, especially in the beginning of the pandemic there were no widespread testing to tell if a person had the virus. One way doctors were able to guess if a person had COVID was through x-ray of the lung and seeing if the person had signs of pneumonia.

What is Pneumonia?

Pneumonia is an infection that inflames the air sacs in one or both lungs. The air sacs may fill with fluid or pus, which can cause coughing, fever, chills, and difficulty breathing. Organisms, including bacteria, viruses, and fungi can cause pneumonia.

Signs of potential Pneumonia:

The picture below shows the difference in x-ray between a normal lung and a pneumonic lung. Usually lungs are full of air and does not absorb the X-rays so it appears black, which you can see in the normal image. Bones absorb X-rays and appear white. Fluid or tissues show up as grey-ish.

Importing Libraries and downloading data:

Let’s download the data from from kaggle: https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia

We need to then import all of the necessary libraries we will be using for the project! We’ll use keras to do the meat of the Neural net stuff.

import matplotlib.pyplot as plt

import seaborn as sns

import keras

from keras.models import Sequential

from keras.layers import Dense, Conv2D , MaxPool2D , Flatten , Dropout , BatchNormalization

from keras.preprocessing.image import ImageDataGenerator

from sklearn.model_selection import train_test_split

from sklearn.metrics import classification_report,confusion_matrix

from keras.callbacks import ReduceLROnPlateau

import os

import numpy as np # linear algebra

import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

import cv2

For this project I’m using google collab, as it has faster training time. Compared to 2 minutes per epoch on my computer, collab was able to do it for 12s per epoch.

So we need to create a helper function to actually pull in the data. We’ll use CV2 for this. And we’ll also pull in the images in a 150 x 150.

labels = ['PNEUMONIA', 'NORMAL']

img_size = 150

def get_data(data_dir):

data = []

for label in labels:

path = os.path.join(data_dir, label)

class_num = labels.index(label)

for img in os.listdir(path):

try:

img_arr = cv2.imread(os.path.join(path, img), cv2.IMREAD_GRAYSCALE)

resized_arr = cv2.resize(img_arr, (img_size, img_size)) # Reshaping images to preferred size

data.append([resized_arr, class_num])

except Exception as e:

print(e)

return np.array(data) train = get_data('/content/drive/My Drive/chest_xray/train')

test = get_data('/content/drive/My Drive/chest_xray/test')

val = get_data('/content/drive/My Drive/chest_xray/val')

PREPARING THE DATA:

In image classification we pull in images into an array of numbers. These numbers represent the pixel intensity, and it’s a number between 0 and 255, 255 being white and 0 being black.

So a few key points before we actually prepare the data:

We need to pair the arrays of the image with their labels (if pneumonia or not)

In a neural net, there’s a lot of math that happens in the backend. If we use numbers like 255 the computer is forced to work with really huge numbers which may increase computational power as well as slow down the epochs. To rectify this we can just divide every pixel by 255 that way we’re left with numbers between 0 and 1, 1 being white and 0 being black.

Lastly when we feed the images to keras, we need to reshape our dimensions. So we’ll use the x_train.reshape(-1, image_size, image_size, 1). The numbers mean [batch_size, height, width, channels]. The -1 means that the length in the dimension is inferred so we don’t have to specify it. The 1 is because we’re using a black and white picture so we’ll only have one layer image.

x_train = []

y_train = []



x_val = []

y_val = []



x_test = []

y_test = []



for feature, label in train1:

x_train.append(feature)

y_train.append(label)



for feature, label in test:

x_test.append(feature)

y_test.append(label)



for feature, label in val:

x_val.append(feature)

y_val.append(label) # Normalize the data

x_train = np.array(x_train) / 255

x_val = np.array(x_val) / 255

x_test = np.array(x_test) / 255 # resize data for deep learning

x_train = x_train.reshape(-1, img_size, img_size, 1)

y_train = np.array(y_train)



x_val = x_val.reshape(-1, img_size, img_size, 1)

y_val = np.array(y_val)



x_test = x_test.reshape(-1, img_size, img_size, 1)

y_test = np.array(y_test)

We also have a big imbalance of data so we’ll use Data Augmentation to combat this. What does this mean? Basically we’ll take the pictures that we have now and create duplicates of them with doing certain changes to the images which can include cropping, rotating, and flipping. Let’s set this up and fit it to our x_train. Notice that we’re not going to flip the images since it does matter how we show our images to the computer and the right lung is always on the left side. The computer might get confused =\

datagen = ImageDataGenerator(

featurewise_center=False, # set input mean to 0 over the dataset

samplewise_center=False, # set each sample mean to 0

featurewise_std_normalization=False, # divide inputs by std of the dataset

samplewise_std_normalization=False, # divide each input by its std

zca_whitening=False, # apply ZCA whitening

rotation_range = 30, # randomly rotate images in the range (degrees, 0 to 180)

zoom_range = 0.2, # Randomly zoom image

width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)

height_shift_range=0.1, # randomly shift images vertically (fraction of total height)

horizontal_flip = False, # randomly flip images

vertical_flip=False) # randomly flip images





datagen.fit(x_train)

WE”RE READY TO GO!

So we have everything set up. We can go through baseline convolutions and try out different things but I’ll save you the trouble. After playing with some of the parameters the best results came out with 6 convolution layers.

model = Sequential()

model.add(Conv2D(32 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu' , input_shape = (150,150,1)))

# model.add(BatchNormalization())

model.add(MaxPool2D((2,2)))



model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))

model.add(Dropout(0.2))

# model.add(BatchNormalization())

model.add(MaxPool2D((2,2)))



model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))

# model.add(Dropout(0.3))

# model.add(BatchNormalization())

model.add(MaxPool2D((2,2)))



model.add(Conv2D(128 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))

model.add(Dropout(0.3))

# model.add(BatchNormalization())

model.add(MaxPool2D((2,2)))



model.add(Conv2D(256 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))

model.add(Dropout(0.3))

# model.add(BatchNormalization())

model.add(MaxPool2D((2,2)))



model.add(Conv2D(512 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))

model.add(Dropout(0.3))



model.add(Flatten())

model.add(Dense(units = 128 , activation = 'relu'))

model.add(Dropout(0.3))

# model.add(Dense(units = 128 , activation = 'relu'))

# model.add(Dropout(0.3))

model.add(Dense(units = 1 , activation = 'sigmoid'))

model.compile(optimizer = "rmsprop" , loss = 'binary_crossentropy' , metrics = ['accuracy'])

model.summary()

That’s our model set up, now we’re going to have to actually run it using the model.fit() method.

history = model.fit(datagen.flow(x_train,y_train, batch_size=100),epochs = 12 , validation_data = datagen.flow(x_val, y_val) predictions = model.predict_classes(x_test)

predictions = predictions.reshape(1,-1)[0]

cm= confusion_matrix(y_test,predictions)



print(classification_report(y_test, predictions, target_names = ['Pneumonia (Class 0)','Normal (Class 1)']))

Our F1-Score came out really good! With a 92% accuracy. But during the tuning of our model I was more interested in the recall. Recall means that our of all of the positive cases that we had we were able to predict 96% of them. Which is awesome!! But looking at the normal we did well but not as well with only 85%. But this is okay because we’d rather tell people that they have pneumonia and start treatment right away instead of misclassifying it and telling them they don’t have pneumonia when they do, and prolonging the treatment might make it worse.

Lastly, we can visualize what our convolution layers are doing in the backend.

from keras.preprocessing import image

from keras.models import Model layer_outputs = [layer.output for layer in model.layers[:50]]

test_image = '/content/drive/My Drive/chest_xray/test_image.jpeg'



img = image.load_img(test_image, target_size=(img_size, img_size))

img_tensor = image.img_to_array(img)

img_tensor = img_tensor.reshape(-1, img_size, img_size, 1)

# img_tensor = np.expand_dims(img_tensor, axis=0)

img_tensor /= 255.



activation_model = Model(inputs=model.input, outputs=layer_outputs)

activations = activation_model.predict(img_tensor)



layer_names = ['conv2d_1', 'activation_1', 'conv2d_2', 'activation_2', 'conv2d_5', 'activation_5']

activ_list = [activations[1], activations[3], activations[11], activations[13]]



images_per_row = 16



for layer_name, layer_activation in zip(layer_names, activ_list):

n_features = layer_activation.shape[-1]

size = layer_activation.shape[1]

n_cols = n_features // images_per_row

display_grid = np.zeros((size * n_cols, images_per_row * size))



for col in range(n_cols):

for row in range(images_per_row):

channel_image = layer_activation[0, :, :, col * images_per_row + row]

channel_image -= channel_image.mean()

channel_image /= channel_image.std()

channel_image *= 64

channel_image += 128

channel_image = np.clip(channel_image, 0, 255).astype('uint8')

display_grid[col * size : (col + 1) * size, row * size : (row + 1) * size] = channel_image



scale = 1. / size

plt.figure(figsize=(scale * display_grid.shape[1], scale * display_grid.shape[0]))

plt.title(layer_name)

plt.grid(False)

plt.imshow(display_grid, aspect='auto', cmap='plasma')

plt.savefig(layer_name+"_grid.jpg", bbox_inches='tight')

It’s not the best representation of how the convolution is breaking it down. But we can kind of see what they’re focusing on.