Pneumonia Detection From X-ray Images Using Deep Learning Neural Network
By: Javier Herbas
Tags: Convolutional Neural Net, Flatiron School, Image Classification, Deep Learning, Data Science

This blog is probably going to be in terms of coding the most technically driven that I have written so far, and it will be about a project using Deep Learning Neural Networks to classify a set of structured X-rays images from pedriatic patients to identify whether or not, they have pneumonia. The Neural Network that I chose is the Convolutional Neural Network (CNN/ConvNet), mainly because it is known to perform really well with image classification.

This project is not intended as a study of Pneumonia itself, but for detecting it from X-rays images by using Deep Learning Neural Networks. Therefore only a short explanation of what the illness is and why it is important in today’s world will follow.

So what is Pneumonia?

Pneumonia is an infection of the lungs that may be caused by bacteria, viruses, or fungi. The infection causes the lung’s air sac, or alveoli, to become inflamed and fill up with fluid or pus. This progression then limits a person’s ability to take in oxygen. If there is a continuous deprivation of oxygen, many of the body’s organs can get damaged, causing kidney failure, heart failure, and other life threatening conditions.

The symptoms of pneumonia can range from mild to severe, and include cough, fever, chills, and trouble breathing (source)

Now, is there any relationship between Pneumonia and Covid-19?

According to the Center for Disease Control and Prevention (CDC), Covid-19 is a respiratory illness where a person can experience a dry cough, fever, muscle aches, and fatigue.

This virus can progress through the respiratory tract and into a person’s lungs causing pneumonia. People with severe cases of pneumonia may have lungs that are so inflamed that they cannot take in enough oxygen or expel enough carbon dioxide.

According to the World Health Organization (WHO), the most common diagnosis for severe COVID-19 is severe pneumonia. For people who do develop symptoms in the lungs, COVID-19 may be life threatening, therefore, the detection of Pneumonia through X-ray has increased its importance due to the worldwide spread of Covid-19 (source).

Now that we know what Pneumonia is, and why detecting it through X-rays is important, lets go into the dataset used and start analysing it before going into the modeling.

Dataset

The dataset used comes from Kaggle and it consists of 5856 X-ray images contained in a hierarchical file structure which is already split into three subsets folders (test, train, and val), each of which containing sub-folders where the X-ray images have been already labeled as NORMAL or PNEUMONIA (Figure 1).

Figure 1. Hierarchical file structure

Because this is a typical nested file structure, the first thing to do is to link each one of those subset folders (test, train and val) and then their internal labelled X-ray images (NORMAL and PNEUMONIA). This will be done with the following code:

# Directory paths: data_dir = "../chest_xray/"

train_dir = os.path.join(data_dir, "train/")

test_dir = os.path.join(data_dir, "test/")

val_dir = os.path.join(data_dir, "val/") # train folder data os.listdir(train_dir)

train_normal = train_dir + 'NORMAL/'

train_pneumo = train_dir + 'PNEUMONIA/' # test folder data os.listdir(test_dir)

test_normal = test_dir + 'NORMAL/'

test_pneumo = test_dir + 'PNEUMONIA/' # val folder data os.listdir(val_dir)

val_normal = val_dir + 'NORMAL/'

val_pneumo = val_dir + 'PNEUMONIA/'

Now, with access to all the data, I created a function that randomly selects a couple of images per subset and plots them. As an example of the output, Figure 2 shows a train subset pair.

Figure 2. X-ray image randomly selected from the train subset

The function is also display below:

def X_ray_QC(path, set_normal, set_pneumo):



# X-Rays classified as 'NORMAL' or equal to 'No-Pneumonia' rand_norm = np.random.randint(0, len(os.listdir(path)))

norm_xray = os.listdir(set_normal)[rand_norm]

print('Normal xray file name:', norm_xray) # X-rays classified as 'PNEUMONIA' meaning that the patient has pneumonia rand_pneumo = np.random.randint(0, len(os.listdir(path)))

pneumo_xray = os.listdir(set_pneumo)[rand_pneumo]

print('Pneumo xray file name:', pneumo_xray) # Image loading norm_xray_address = set_normal + norm_xray

pneumo_xray_address = set_pneumo + pneumo_xray normal_load = Image.open(norm_xray_address)

pneumonia_load = Image.open(pneumo_xray_address) # Plotting the X-ray files: figure = plt.figure(figsize=(14,6)) ax1 = figure.add_subplot(1,2,1)

xray_plot = plt.imshow(normal_load, cmap='gray')

plt.rcParams["figure.facecolor"] = "lightblue"

ax1.set_title('NORMAL', fontsize=14)

ax1.axis('on')



ax2 = figure.add_subplot(1,2,2)

xray_plot = plt.imshow(pneumonia_load, cmap='gray')

ax2.set_title('PNEUMONIA', fontsize=14)

ax2.axis('on')

plt.show()

Continuing with the hierarchical file structure, we already know that the dataset has been split into three subsets: train, test and val. This is great to save some time but we now need to know if the split has created a balanced distribution?

In order to answer that question I created another function that automatically creates dataframes with binary columns, where a 0 was assigned to the NORMAL cases, and a 1 to the PNEUMONIA cases, and then these are automatically plotted (Figure 3). Initially the val (validation) subset (validation subset) had only 16 X-rays images, so I modified it and added 100 X-rays from the test subset not to modify the train subset, as Deep Learning algorithms behave better with larger train dataset.

Figure 3. X-ray data distribution for data balance check

The final data split distributed the data with the following percentages:

Train subset: 89.07%

Test subset: 8.95%

Val subset: 1.98%

Now, these distribution might suggest that the data is unbalanced, but again, because we are doing Deep Learning, I want to keep the largest possible training set, therefore I will proceed as it is.

Data Scaling and Augmentation

Since neural networks receive inputs of the same size, all images need to be resized to a fixed size before inputting them to the CNN. The larger the fixed size, the less shrinking required. Less shrinking means less deformation of features and patterns inside the image.

These X-rays are all in a black and white scale, so just for the sake of avoiding any image difference size issues, I have rescaled all images to 256 colors (0–255). There is always a second option which is to re-scale even further to [0,1] and could accelerate the model, but I didn’t go that route

As mentioned above, it is well known that the performance of Deep Learning Neural Networks often improves with the amount of data available, therefore increasing the number of data samples could result in a more skillful model. This can be achieved by doing “Data Augmentation”, which is a technique to artificially create new training data from existing training data. This is done by applying domain-specific techniques to examples from the training data that create new and different training examples by simply shifting, flipping, rotating, modifying brightness, and zooming the training examples. Both, re-scaling and augmentation were performed with the code below:

rescale = 1./255

target_size = (150,150)

batch_size = 32

class_mode = "categorical"

#class_mode = 'binary' train_datagen = ImageDataGenerator(rescale=rescale,

shear_range=0.2,

zoom_range=0.2,

horizontal_flip=True) train_generator = train_datagen.flow_from_directory(train_dir,

target_size=target_size,

class_mode=class_mode,

batch_size=batch_size,

shuffle=True) validation_datagen = ImageDataGenerator(rescale=rescale) validation_generator = validation_datagen.flow_from_directory(val_dir,

target_size=target_size,

class_mode=class_mode,

batch_size=dir_file_count(val_dir),

shuffle=False) test_datagen = ImageDataGenerator(rescale=rescale) test_generator = test_datagen.flow_from_directory(test_dir,

target_size=target_size,

class_mode=class_mode,

batch_size=dir_file_count(test_dir),

shuffle=False)

Model Architecture

The next step is to define the architecture of the CNN model which is where it gets a bit more complicated. I tested a total of 15 different models/architectures which might sound like a lot, but I started from a very simple two layer one and with low epochs numbers to get results faster, and evolve from there into more complex ones by adding layers one at a time to test them. I will obviously just go over one of these models, which is the one that I consider to be the best, and for those interested on seeing the other architectures and results I will add my GitHub at the end of this blog. The final model architecture is displayed on Figure 4:

model_15 = models.Sequential() # 1st Convolution block

model_15.add(layers.Conv2D(16, (3, 3), activation='relu', padding='same', input_shape=(150, 150, 3)))

model_15.add(layers.Conv2D(16, (3, 3), activation='relu'))

model_15.add(layers.MaxPooling2D((2, 2))) # 2nd Convolution block

model_15.add(layers.Conv2D(32, (3, 3), activation='relu', padding='same'))

model_15.add(layers.Conv2D(32, (3, 3), activation='relu'))

model_15.add(layers.MaxPooling2D((2, 2))) # 3rd Convolution block

model_15.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same'))

model_15.add(layers.Conv2D(64, (3, 3), activation='relu'))

model_15.add(layers.MaxPooling2D((2, 2))) # 4th Convolution block

model_15.add(layers.Conv2D(96, (3, 3), dilation_rate=(2,2), activation='relu', padding='same'))

model_15.add(layers.Conv2D(96, (3, 3), activation='relu'))

model_15.add(layers.MaxPooling2D((2, 2))) # 5th Convolution block

model_15.add(layers.Conv2D(128, (3, 3), dilation_rate=(2, 2), activation='relu', padding='same'))

model_15.add(layers.Conv2D(128, (3, 3), activation='relu'))

model_15.add(layers.MaxPooling2D((2, 2))) # Flattened the layer

model_15.add(layers.Flatten()) # Fully connected layers

model_15.add(layers.Dense(64, activation='relu'))

model_15.add(layers.Dropout(0.4))

model_15.add(layers.Dense(2, activation='softmax')) model_15.summary()

I will not go over what each one of the layers of my model represent/mean to avoid making this an extremely long blog, so if interested just refer to my GitHub where you will find explanations of each one and good links to tutorials on how to manage them.

The one thing I do want to explain is the usage of the Softmax activation instead of a Sigmoid, which is normally the choice for a binary classification, and that is what I have on my hands. So what I did was to convert my binary problem into a category one with two categories, NORMAL or PNEUMONIA, and therefore the behaviour of the Softmax became the same as a Sigmoid. Now why did I do that? because the validation accuracy was stagnated at 50% regardless of the model architecture that I used, therefore I used this workaround as a solution because a 50% accuracy means tossing a coin and that is not acceptable regardless of the accuracy obtained with your train subset which in my case was always above 95%, so most probably overfitting.

Now to continue, below is the code for fitting/training the model that I used followed by the results of running it with the test subset

optimizer=optimizers.Adam()

loss='categorical_crossentropy'

metrics=['accuracy']

epochs = 50

steps_per_epoch=100

validation_steps=50 model.compile(optimizer, loss=loss, metrics=metrics) history = model_15.fit_generator(train_generator,

steps_per_epoch=steps_per_epoch,

epochs=epochs,

verbose=2,

validation_data=validation_generator,

validation_steps=validation_steps)

In the code above there is a parameter called epoch. This is a term used in Machine Learning (ML) to indicate the number of passes of the entire training dataset the ML algorithm has completed. Because of time constraints I chose 50 epochs, which is probably the minimum for these type of models. Therefore you could expect a better behaved and skillful model by increasing this number to 100 or even 150 but you need the hardware power and the time. Still with the 50 epochs the model ended up with a train accuracy of 96.69% (most probably overfitted) and validation accuracy of 94.83%.

A simple way to understand how the model behaves with each epoch is to look at a plot of the train and validation subset loss and accuracy (Figure 4)

Figure 4. Model loss and accuracy visualisation for train and validation data subsets

What the plots from Figure 4 is telling us is that for the train subset, with the increasing epochs, there is convergence happening as both blue curves get closer together, with a similar situation for the validation subset but not as smooth and fast as for the train subset, hence the need of increasing the epoch number to probably 100 as it seems that the validation curves start becoming less spiky towards the 40 epochs onwards.

Now that I am happy with this model’s performance, the next step is to evaluate the model using the test subset. The code to do so is in the cell below with the outputs on the last three rows.

test_generator = test_datagen.flow_from_directory(test_dir,

target_size=target_size,

class_mode=class_mode,

batch_size=dir_file_count(test_dir),

shuffle=False) test_loss, test_acc = model_5.evaluate_generator(test_generator, steps=50)

print('test accuracy:', round(test_acc*100, 2)

print('test loss:' , round(test_loss)) Found 524 images belonging to 2 classes.

test accuracy: 90.84

test loss: 0.3

The test subset resulted with an accuracy of 90.84% which is quite good. However, as a performance measure, accuracy is inappropriate for imbalanced classification problems. The reason for this is that the overwhelming number of samples from the majority class, in this case the train subset, will overwhelm the number of examples in the minority class. This means that even unskillful models can achieve accuracy scores of 90 percent, or 99 percent, depending on how severe the class imbalance happens to be.

An alternative to using classification accuracy is to use precision and recall metrics. For those not familiar with these metrics, here is a simple definition of what they are:

Precision: quantifies the number of positive class predictions that actually belong to the positive class.

Recall: quantifies the number of positive class predictions made out of all positive examples in the dataset.

F-Measure: provides a single score that balances both the concerns of precision and recall in one number.

If you are interested in knowing more about these metrics I recommend this link which has good explanations and a small project to run them.

With those definitions now in hand, what I did was to calculate them automatically by using the scikit-learn ML library. The code follows below with the outputs for each in the last three rows.

precision = precision_score(true_classes, predicted_classes)

recall = recall_score(true_classes, predicted_classes)

f1 = f1_score(true_classes, predicted_classes) print('Precision score:', round(precision*100, 2))

print('Recall score:', round(recall*100, 2))

print('F1 score:', round(f1*100, 2)) Precision score: 87.05

Recall score: 85.0

F1 score: 86.01

These three are better metrics for these type of imbalanced datasets. In this particular case the difference is not too big as the model ended up with an accuracy of 90.84%, and the Precision, Recall and F-score ended up being 87, 85, and 86% respectively. So not too far apart from each other making the model’s accuracy mode credible.

So there we go, I have successfully finished my first CNN model for Pneumonia prediction using X-ray images with an accuracy of 90.84%

I hope those who made it this far have enjoyed the reading, and as promised above, here is the associated GitHub and my linkedin just in case anyone has questions or wants to discuss these results.