Automatic Impression Generation from Medical Imaging Report
By: Anand Pandiyan
Tags: Deep Learning, Artificial Intelligence, Image Captioning, Machine Translation, Computer Vision

8.3 EDA Findings

All the raw texts from xml files are parsed and created the dataset.

Each patient have multiple x-rays associated with them.

Major finding is that how the images are in sequence or number of images associated with each record.

We have mostly of 2 images per record frontal and lateral. and also, we have 1, 3, 4, 5 images associated with each record.

There is no missing files. We have total of 3955 records and 3 additional features (Comparison, Indication and Findings) which we will not use in this model and 1 Impression target variable.

Most occurring words: Impression - acute cardiopulmonary

Images are in different shapes.

All the X-Ray images are human upper body particularly about Chest part.

In text features there are some unknown values like XXXX XXXXX these are replaced with empty string.

8.4 Data Conflicts

There are only two image types Frontal and Lateral, but we have 1, 3, 4, and 5 images associated for each datapoints. we have met a conflict here that how we provide the data points to the model we build. Because of this conflict we need to come up with an idea to handle how we input the data to the model. Before building our model, we see some data point structuring methods that could help use handle this case.

9. Data point construction

As we have more than 2 image or some case less than 2 images associated with each data point. If we have no images, we dropped those data point.

Lets handle the data point which are having 1,3,4,5 images. Below is the data point counts with number of image sets.

Data point having 2 images is 3208

Data point having 1 images is 446

Data point having 3 images is 181

Data point having 4 images is 15

Data point having 1 images is 1

Total data point is 3851 data points

Approach,

Limiting the data point to 2 images per data point, if we have 5 images, its 4+1 (all image + last image) so make it as 4 data points as below.

Here last image should be Lateral if we have frontal as remaining images.

if i have 5 images then, here 5th image is Lateral other or frontal,

1st image + 5th image => Frontal + Lateral

2nd image + 5th image => Frontal + Lateral

3rd image + 5th image => Frontal + Lateral

4th image + 5th image => Frontal + Lateral

Increased to 4 data point from this single data point

likewise, for other data point,

if i have 4 images then,

1st (Frontal) + 4th (Lateral)

2nd (Frontal) + 4th (Lateral)

3rd (Frontal) + 4th (Lateral)

Increase to 3 data point from 1 data point

if i have 3 images then,

1st (Frontal) + 3rd (Lateral)

2nd (Frontal) + 3rd (Lateral)

Increased to 2 data point from this single data point

If we have only one image then,

1st images either (Frontal or Lateral) + Duplicate 1st image

Same data point count. We need to make sure this duplicating data point should be equally split among the train test and validation sets. If we don’t have Lateral images, then keep the frontal as last image data points.

So with this data constructing method we could also increase the data point and come up with fine input data points. Code for the above explained data structuring.

After constructing the data point we will add the <start> and <end> token to text data.

Final datapoints,

10. Train Test and Validation split

We have a separate data one is without duplicate data points other is with duplicate data points. We need to split the data points as the duplicate data points are equally available in all three splits.

After taking the two different data set we need to concatenate the dataset equally.

We get the final data point shape as above.

11. Tokenization and Dataset preparation

11.1. Tokenization

We cannot feed raw text to our deep learning model. Text data need to be encoded as numbers and then used in both machine learning and deep learning models. The Keras deep learning library provides some basic tools to perform this operation.

Total vocabulary (vocab_size) present is 1339 and maximum length of the output sentence is taken as 60.

11.2. Dataset Preparation

For the dataset preparation we will be using the transfer learning method for image to feature vector conversion and text data tokenization.

Please refer this blog on why I have chosen the inception model over others https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/

I will be using the InceptionV3 model trained on ImageNet dataset. Initially I will be doing a xray classification task using below mentioned dataset. https://www.kaggle.com/yash612/covidnet-mini-and-gan-enerated-chest-xray. It’s a three-class classification task where we need to classify the whether the xray of the patient is belongs to one of these 3 class Corona or Normal or Pneumonia.

Once the classification is done, I will save the weights of the trained model and use this model with removed top layer of shape(1, 2048) as feature vector for our model and prepare the dataset.

Below is the model architecture for this classification task.

Transfer Learning

Accuracy plot on the classification task.

Accuracy plot

Model weights are saved for future use as hdf5 file.

I have trained this model using ImageNet weights and without ImageNet weights with image net weights performed well in this classification.

With the trained weights I will use like below for feature extraction for our image data.

Loading weights

I will create a image tensor for all available images using the inception feature vectorization like below.

Creating image tensor

These image tensors are used in TensorFlow dataset preparation basically I am doing a caching mechanism here for future use.

Create TensorFlow using tf.data

Refer the link for further reading on tf.data: https://www.tensorflow.org/guide/data

Tutorials to read: https://adventuresinmachinelearning.com/tensorflow-dataset-tutorial/

Now that we have our image tensor and text vectors we can build the tf.data dataset

tf.data creation

Multi_image() function converts the two-input tensor of shape (1,2048) & (1,2048) to (2,1,2048). Batch_size, embedding dimension, and units size are mentioned these are the hyperparameters that we can tune according to our model.

So we have done the feature extraction and tokenization for our model to work, And we have the tf.data dataset now lets build the required model.

12. Basic Model

12.1. Model Architecture

As I have already explained about the subclass model. I will directly jump into the model architecture.

I have built the functional Api model for checking the model architecture

Basic model architecture

12.1.1 Encoder architecture:

Have single fully connected layer linear output. Before we pass to the FC layer, we add the two image tensor and pass to FC layer. This layer outputs shape of (batch_size, 1, embedding_dimension)

Encoder

12.1.2 Decoder Architecture:

In this part we have an embedding layer LSTM layer and dense layer which outputs shape (batch_size, vocab_size)

LSTM layer is Long Short-Term Memory networks — usually just called “LSTMs” — are a special kind of RNN, capable of learning long-term dependencies.

To know more about LSTM refer this link: Understanding LSTM Networks

Decoder

12.2 Model metric and optimizer Initialization

12.3 Model Training

For the training phase we use the Teacher forcing. Teacher forcing is a strategy for training recurrent neural networks that uses model output from a prior time step as an input.

In the Training, a “start-of-sequence” token can be used to start the process and the generated word in the output sequence is used as input on the subsequent time step, perhaps along with other input like an image or a source text.

This same recursive output-as-input process is used till the model converge to better result. Below I have mentioned the source.

Further readings about teacher forcing: link to Teacher forcing

Training step

12.4. Model Performance visualized in TensorBoard

We have logged the loss and accuracy using tf.summary

Accuracy

Loss

12.5. Model Evaluation

In the evaluation or testing stage I have used the argmax search based teacher forcing to find the output sentence. In time step t we generated a word using <start> token and the predicted word again fed back to the next step and it become the input of the decoder in time t+1. Code for argmax search is mentioned below.

Evaluation stage

Sample outputs are shown below

Lets try a longer sentence word first

Prediction is not perfect in the longer sentence lets see the shorter sentence.

Even in the short sentence model not performing well.

12.6. Basic Model Conclusion

This model is built on a simple encoder decoder with LSTM.

getting not perfect or not worst predictions

validation accuracy is not improving much but loss is converging

we could even fine tune this model for perform well.

We will see a better performing and modified architecture having bidirectional LSTM layer with Additive Attention mechanism.

13. Main Model

13.1 Model Architecture

The Model Architecture is reimplemented using one of the research paper I came across Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Please refer the paper before we understand the model architecture of this model.

In this paper they proposes Attention-BLSTM model in detail.

As shown in below figure, the model proposed in this paper contains five components:

Input layer: input sentence to this model summed with the image feature vectors

Embedding layer: map each word into a low dimension vector

LSTM layer: utilize BLSTM to get high level features from step (2) this BLSTM layers is repeated twice for more in depth feature understanding

Attention layer: produce a weight vector, and merge word-level features from each time step into a sentence-level feature vector, by multiplying the weight vector

Output layer: the sentence-level feature vector is finally used for relation classification. These components will be presented in detail functional view in coming sections.

Lets see the model functional layers using Functional Api. In this model the hyper parameters are same as the basic model only change is with the maximum sentence length is take as 80.

The same below functional model will be implemented using model subclass as I have already mentioned the subclass model is easier while debugging your architecture. We can have the control over each layer.