Deep Learning for Detecting Pneumonia from X-ray Images | by Abhinav Sagar | Towards Data Science

Stuck behind the paywall? Click here to read the full story with my Friend Link!
The risk of pneumonia is immense for many, especially in developing nations where billions face energy poverty and rely on polluting forms of energy. The WHO estimates that over 4 million premature deaths occur annually from household air pollution-related diseases including pneumonia. Over 150 million people get infected with pneumonia on an annual basis especially children under 5 years old. In such regions, the problem can be further aggravated due to the dearth of medical resources and personnel. For example, in Africa’s 57 nations, a gap of 2.3 million doctors and nurses exists. For these populations, accurate and fast diagnosis means everything. It can guarantee timely access to treatment and save much needed time and money for those already experiencing poverty.
This project is a part of the Chest X-Ray Images (Pneumonia) held on Kaggle.
Build an algorithm to automatically identify whether a patient is suffering from pneumonia or not by looking at chest X-ray images. The algorithm had to be extremely accurate because lives of people is at stake.
scikit-learnkerasnumpypandasmatplotlib
scikit-learn
keras
numpy
pandas
matplotlib
The dataset can be downloaded from the kaggle website which can be found here.
Without much ado, let’s get started with the code. The complete project on github can be found here.
Let’s start with loading all the libraries and dependencies.
Next I displayed some normal and pneumonia images to just have a look at how much different they look from the naked eye. Well not much!
Then I split the data-set into three sets — train, validation and test sets.
Next I wrote a function in which I did some data augmentation, fed the training and test set images to the network. Also I created labels for the images.
The practice of data augmentation is an effective way to increase the size of the training set. Augmenting the training examples allow the network to “see” more diversified, but still representative, data points during training.
Then I defined a couple of data generators: one for training data, and the other for validation data. A data generator is capable of loading the required amount of data (a mini batch of images) directly from the source folder, convert them into training data (fed to the model) and training targets (a vector of attributes — the supervision signal).
For my experiments, I usually set the batch_size = 64. In general a value between 32 and 128 should work well. Usually you should increase/decrease the batch size according to computational resources and model’s performances.
After that I defined some constants for later usage.
The next step was to build the model. This can be described in the following 5 steps.
I used five convolutional blocks comprised of convolutional layer, max-pooling and batch-normalization.On top of it I used a flatten layer and followed it by four fully connected layers.Also in between I have used dropouts to reduce over-fitting.Activation function was Relu throughout except for the last layer where it was Sigmoid as this is a binary classification problem.I have used Adam as the optimizer and cross-entropy as the loss.
I used five convolutional blocks comprised of convolutional layer, max-pooling and batch-normalization.
On top of it I used a flatten layer and followed it by four fully connected layers.
Also in between I have used dropouts to reduce over-fitting.
Activation function was Relu throughout except for the last layer where it was Sigmoid as this is a binary classification problem.
I have used Adam as the optimizer and cross-entropy as the loss.
Before training the model is useful to define one or more callbacks. Pretty handy one, are: ModelCheckpoint and EarlyStopping.
ModelCheckpoint: when training requires a lot of time to achieve a good result, often many iterations are required. In this case, it is better to save a copy of the best performing model only when an epoch that improves the metrics ends.
EarlyStopping: sometimes, during training we can notice that the generalization gap (i.e. the difference between training and validation error) starts to increase, instead of decreasing. This is a symptom of overfitting that can be solved in many ways (reducing model capacity, increasing training data, data augumentation, regularization, dropout, etc). Often a practical and efficient solution is to stop training when the generalization gap is getting worse.
Next I trained the model for 10 epochs with a batch size of 32. Please note that usually a higher batch size gives better results but at the expense of higher computational burden. Some research also claim that there is an optimal batch size for best results which could be found by investing some time on hyper-parameter tuning.
Let’s visualize the loss and accuracy plots.
So far so good. The model is converging which can be observed from the decrease in loss and validation loss with epochs. Also it is able to reach 90% validation accuracy in just 10 epochs.
Let’s plot the confusion matrix and get some of the other results also like precision, recall, F1 score and accuracy.
CONFUSION MATRIX ------------------[[191  43] [ 13 377]]TEST METRICS ----------------------Accuracy: 91.02564102564102%Precision: 89.76190476190476%Recall: 96.66666666666667%F1-score: 93.08641975308642TRAIN METRIC ----------------------Train acc: 94.23
The model is able to achieve an accuracy of 91.02% which is quite good considering the size of data that is used.
Although this project is far from complete but it is remarkable to see the success of deep learning in such varied real world problems. I have demonstrated how to classify positive and negative pneumonia data from a collection of X-ray images. The model was made from scratch, which separates it from other methods that rely heavily on transfer learning approach. In the future this work could be extended to detect and classify X-ray images consisting of lung cancer and pneumonia. Distinguishing X-ray images that contain lung cancer and pneumonia has been a big issue in recent times, and our next approach should be to tackle this problem.
medium.com
becominghuman.ai
stanfordmlgroup.github.io
The corresponding source code can be found here.
github.com
Happy reading, happy learning and happy coding!
If you want to keep updated with my latest articles and projects follow me on Medium. These are some of my contacts details: