# Framework to Segment and Evaluate Multiple Sclerosis Lesion in MRI Slices Using VGG-UNet

Authors: Sujatha Krishnamoorthy; Yaxi Zhang; Seifedine Kadry; Weifeng Yu
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Multiple sclerosis (MS) is an autoimmune disease that causes mild to severe issues in the central nervous system (CNS). Early detection and treatment are necessary to reduce the harshness of the disease in individuals. The proposed work aims to implement a convolutional neural network (CNN) segmentation scheme to extract the MS lesion in a 2D brain MRI slice. To achieve a better MS detection, this work implemented the VGG-UNet scheme in which the pretrained VGG19 is considered as the encoder section. This scheme is tested on 30 patient images (600 images with dimension512×512×3 pixels), and the experimental outcome confirms that this scheme provides a better result compared to traditional UNet, SegNet, VGG-UNet, and VGG-SegNet. The experimental investigation implemented on axial, coronal and sagittal plane 2D slices of Flair modality confirms that this work provides a better value of Jaccard (>85%), Dice (>92%), and accuracy (>98%).


---

## Body

## 1. Introduction

The brain is one of the prime internal organs in human physiology, and the abnormality in the brain is a major medical emergency. The abnormality in the brain causes various behavioural disorders, and appropriate diagnosis and treatment will help to reduce these issues [1, 2]. Multiple sclerosis (MS) is one of the brain-connected diseases that happens mainly in the central nervous system (CNS) [3–5].The major cause of MS is the damage to the myelin of the nerve fibres, which creates mild to severe communication issues between the brain and other body organs. The individuals who are affected with MS will experience one or all of the following issues based on the severity of the disease, vision-related issues, speech issues, body muscle-related, and dizziness. Furthermore, the individuals affected with MS may develop epilepsy frequently compared to healthy individuals. The major cause of MS is due to the malfunction in the immune system, and other factors include age, race, climatic conditions, and vitamin-D deficiency [6, 7]. A recent survey confirms that around 2.8 million individuals are affected with MS globally, and nearly 1 million individuals are living in the United States [8, 9]. Furthermore, this survey also confirms that every five minutes, an individual is diagnosed with MS globally, and hence, the number of infected persons is steadily rising. Due to its rapid happening rate, a number of research works are proposed to detect MS in its early phase.The clinical level diagnosis of MS is commonly performed with the help of magnetic resonance imaging (MRI) recorded using a chosen modality. After recording the image, a personal check by the doctor or a chosen computer algorithm is employed to inspect the MS lesion in the brain. In most of the earlier works, the computer algorithm-based examination is implemented to segment and evaluate the MS lesion from the chosen brain MRI slice [10–14].The earlier MS examination works confirm that the convolutional neural network (CNN) schemes perform well in detecting the disease with appropriate accuracy [15, 16]. The proposed work in this study aims to implement a CNN-supported MS detection framework to examine the 2D brain slices with axial, coronal, and sagittal planes. In this work, initially, the performance of the pretrained CNN segmentation schemes, such as UNet, SegNet, VGG-UNet, and VGG-SegNet, are analyzed. This experimental investigation confirms that the UNet scheme helps to achieve better MS detection compared to SegNet, and hence, VGG-UNet developed using the VGG19 architecture is considered and its performance is verified.The proposed VGG-UNet scheme is considered to segment the MS lesion from the chosen brain MRI slices. During this task, the CNN segmentation system is trained with the test images and the ground truth (GT) images of the chosen MS database. After segmentation, the segmented image is then compared with GT, and the necessary performance metrics, such as Jaccard, Dice, accuracy, precision, sensitivity, and specificity, are computed, and based on these values, the merit of the proposed framework is verified. The proposed work is implemented using Python®, and the outcome of the proposed work confirms that this scheme works well on axial, coronal, and sagittal plane 2D MRI slices.The main contributions of the proposed work are as follows:(i)
VGG19 supported CNN segmentation scheme is developed and executed to achieve an improved result over the existing VGG-UNet scheme(ii)
This work considered the Flair modality brain MRI slices for the examination, and the experiment is executed separately on axial, coronal, and sagittal planes.The remaining part of this research is organized as follows: Sections2 and 3 demonstrate the related earlier works and methodology. Sections 4 and 5 present the experimental outcome and conclusion, respectively.


## 2. Related Work

The happening rate of MS is steadily increasing globally, and well-organized screening and treatment are essential to detect the disease in its early phase. Because of its significance, a number of MS detection schemes are proposed to support the efficient segmentation of the MS lesion from the brain MRI slice of chosen modality. The earlier works in the literature also confirm that the CNN segmentation procedures help to achieve a better result compared to the conventional methods. Existing MS lesion detection methods and its outcome are summarised in this section to demonstrate the existing methods, their merits, and limitations. This section presents the CNN segmentation supported extraction and examination of the MS lesions in the earlier works.Ghosh et al. presented a CNN segmentation scheme to extract the MS lesion in a brain MRI slice [10]. This work implemented CNN schemes such as UNet and UNet++ to extract the MS lesion in 2D MRI slices with modalities T1, T2, and Flair and achieved a Dice score >75%. Birenbaum and Greenspan implemented a multiview long CNN scheme to extract the MS lesion from ISBI2015 benchmark test images and achieved an overall Dice value of 62.7% [11]. Gabr et al. performed fully convolutional neural network- (FCNN-) based segmentation of the MS lesion on clinical images and confirmed that the proposed scheme helps to achieve 82% of the Dice score during MS lesion extraction [12]. Aslani et al. presented a customized encoder-decoder CNN scheme to extract the MS lesion and achieved a maximum Dice score of 50.01% on the chosen test images [13]. Ansari et al. discussed the segmentation of the MS lesion using the CNN scheme and achieved a Dice score of 63.06% [14, 15].Weeda et al. presented a detailed review of the segmentation procedures implemented for MS lesion extraction and confirmed that the CNN schemes help to achieve a better result compared to the alternatives [15]. Zeng et al. presented a detailed review of the MS lesion segmentation methods existing in the literature for ISBI and MICCAI challenge datasets and confirmed the merit of the CNN segmentation to examine the MS lesion in the MRI slices [16].All these works confirm that CNN segmentation provides better results when MRI slices are considered to examine the MS lesions. Furthermore, in all these works, the Dice score achieved is less, and hence, it is necessary to implement a chosen CNN scheme to segment the MS lesion in the chosen test image with better performance metrics. To overcome this problem, a novel VGG-UNet is proposed and implemented to extract the MS lesion in Flair modality MRI. The experimental result confirms that the proposed scheme helps to achieve a better Dice value on axial, coronal, and sagittal planes.


## 3. Methodology

This research aims to implement a CNN segmentation scheme to extract the MS lesion from the 2D MRI slices. The procedure implemented in this scheme is shown in Figure1. Usually, the brain MRI collected from a patient is in the form of 3D, and hence, a 3D to 2D conversion is initially implemented using ITK-SNAP, and the extracted 2D slice is then resized into 512×512×3 pixels. A similar procedure is then implemented for the ground truth (GT) image. The image slice along with GT is then considered to train and test the segmentation performance of the chosen CNN scheme. In the proposed work, the segmentation task is implemented with UNet, SegNet, VGG-UNet (with VGG16 encoder), VGG-SegNet (with VGG16 encoder), and VGG-UNet using the VGG19 encoder. After segmentation, a comparative assessment is then employed to compute the necessary image measures, and based on this value, the MS lesion detection performance is verified.Figure 1
Structure of the proposed MS lesion segmentation work.


### 3.1. Image Database

The necessary images for this research work are collected from [17]. In this work, 30 patients’ Flair modality database is considered for the assessment. These images are in 3D form, and the 3D to 2D conversion is achieved using ITK-SNAP [18, 19], which helped to extract 20 slices from every patient. After extraction, these 2D images are resized to 512×512×3 pixels. In this work, 600 images along with their related ground truth (GT) are separately collected from axial, coronal, and sagittal planes, and these images are individually tested on the considered MS detection framework. The sample test images and the associated GT are shown in Figure 2.Figure 2
Sample test images and related GT.


### 3.2. CNN Scheme

Extraction of the MS lesion from the MRI slice is necessary to identify the severity of the disease, which plays a key role during the clinical level treatment planning process. In this work, pretrained CNN schemes such as UNet [20], SegNet [21], VGG-UNet [22, 23], and VGG-SegNet [24] are initially employed, and the segmentation performance is then verified. Later, VGG-UNet is then enhanced by considering the VGG19 encoder and decoder section, and its merit in detecting the MS lesion is verified. Figure 3 shows the proposed scheme and the concept behind this scheme, and the discussion can be found in [25].Figure 3
Proposed VGG-SegNet scheme to evaluate MS lesion.In this work, the MRI slice and associated GT are considered to train and validate the CNN segmentation scheme. The initial parameters assigned to the CNN scheme are as follows: initial weights = ImageNet, optimizer = Adam, pooling = average, hidden layer activation = ReLU, classifier activation = sigmoid, learning rate = 0.001, monitoring metric = Dice and accuracy, training images = 480, and testing images = 120.All the CNN models help to get a binary version of the MS lesion image, and after extraction, a pixelwise comparative analysis is implemented with GT, and the necessary performance metrics are computed. Based on the achieved performance metric value, the merit of the proposed scheme is verified.


### 3.3. Execution

The proposed framework is implemented using Python® using a workstation of Intel i7 processor with 24 GB RAM and 4 GB VRAM. Every CNN scheme is separately tested and verified. During execution, the test images (512×512×3 pixels) and GT (512×512×3 pixels) are considered to train the network using 80% of the input data (480 images), and the remaining 20% of input data (120 images) are considered to validate the segmentation performance of the CNN scheme.Figure4 shows the results achieved for the proposed VGG-UNet scheme in which Figure 4(a) shows the training images, Figure 4(b) shows the convergence of the training and validation outcomes with respect to epochs, and Figure 4(c) shows the segmented image. In this image, the first and second columns denote the test image and its GT and the third column depicts the extracted (predicted) MS lesion. To present the images with better visibility, the JET color map is considered in this demonstration. After training, the proposed scheme helps to extract the MS lesions from all 600 images and stores them as binary images. In the binary image, the MS lesion is assigned with a value of 1 (white in color), and the background is assigned with a value of 0 (dark color). This binary image is then compared with GT, and the necessary performance metrics are computed.Figure 4
An experimental result achieved with the proposed VGG-UNet. (a) Training data. (b) Convergence of VGG-UNet training. (c) Validation outcome.
(a)(b)(c)


### 3.4. Performance Evaluation

The merit of the automatic MS detection system is verified by computing the necessary image metrics. During this comparison, the common measures such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN) are initially computed.(i)
TP is the number of binary 1’s correctly predicted as 1(ii)
TN is the number of binary 0’s correctly predicted as 0(iii)
FP is the number of binary 0’s predicted as 1(iv)
FN is the number of binary 1’s predicted as 0The above measures help to get the related values, such as the true positive rate (TPR), true negative rate (TNR), false positive rate (FPR), and false negative rate (FNR) as discussed in [26–30]. Along with these measures, the other vital metrics, such as Jaccard, Dice, accuracy, precision, sensitivity, specificity, and negative predictive values (NPV), are also computed, and based on these values, the merit of the CNN is confirmed.The measures considered in this work are presented in equations (1)–(9):(1)TPR=sensitivity=TPTP+FN,(2)TNR=specificity=TNTN+FP,(3)FPR=FPFP+TN,(4)FNR=FNFN+TP,(5)Jaccard=TPTP+FP+FN,(6)Dice=2TP2TP+FP+FN,(7)Accuracy=TP+TNTP+TN+FP+FN,(8)Precision=TPTP+FP,(9)NPV=TNTN+FN.


## 3.1. Image Database

The necessary images for this research work are collected from [17]. In this work, 30 patients’ Flair modality database is considered for the assessment. These images are in 3D form, and the 3D to 2D conversion is achieved using ITK-SNAP [18, 19], which helped to extract 20 slices from every patient. After extraction, these 2D images are resized to 512×512×3 pixels. In this work, 600 images along with their related ground truth (GT) are separately collected from axial, coronal, and sagittal planes, and these images are individually tested on the considered MS detection framework. The sample test images and the associated GT are shown in Figure 2.Figure 2
Sample test images and related GT.


## 3.2. CNN Scheme

Extraction of the MS lesion from the MRI slice is necessary to identify the severity of the disease, which plays a key role during the clinical level treatment planning process. In this work, pretrained CNN schemes such as UNet [20], SegNet [21], VGG-UNet [22, 23], and VGG-SegNet [24] are initially employed, and the segmentation performance is then verified. Later, VGG-UNet is then enhanced by considering the VGG19 encoder and decoder section, and its merit in detecting the MS lesion is verified. Figure 3 shows the proposed scheme and the concept behind this scheme, and the discussion can be found in [25].Figure 3
Proposed VGG-SegNet scheme to evaluate MS lesion.In this work, the MRI slice and associated GT are considered to train and validate the CNN segmentation scheme. The initial parameters assigned to the CNN scheme are as follows: initial weights = ImageNet, optimizer = Adam, pooling = average, hidden layer activation = ReLU, classifier activation = sigmoid, learning rate = 0.001, monitoring metric = Dice and accuracy, training images = 480, and testing images = 120.All the CNN models help to get a binary version of the MS lesion image, and after extraction, a pixelwise comparative analysis is implemented with GT, and the necessary performance metrics are computed. Based on the achieved performance metric value, the merit of the proposed scheme is verified.


## 3.3. Execution

The proposed framework is implemented using Python® using a workstation of Intel i7 processor with 24 GB RAM and 4 GB VRAM. Every CNN scheme is separately tested and verified. During execution, the test images (512×512×3 pixels) and GT (512×512×3 pixels) are considered to train the network using 80% of the input data (480 images), and the remaining 20% of input data (120 images) are considered to validate the segmentation performance of the CNN scheme.Figure4 shows the results achieved for the proposed VGG-UNet scheme in which Figure 4(a) shows the training images, Figure 4(b) shows the convergence of the training and validation outcomes with respect to epochs, and Figure 4(c) shows the segmented image. In this image, the first and second columns denote the test image and its GT and the third column depicts the extracted (predicted) MS lesion. To present the images with better visibility, the JET color map is considered in this demonstration. After training, the proposed scheme helps to extract the MS lesions from all 600 images and stores them as binary images. In the binary image, the MS lesion is assigned with a value of 1 (white in color), and the background is assigned with a value of 0 (dark color). This binary image is then compared with GT, and the necessary performance metrics are computed.Figure 4
An experimental result achieved with the proposed VGG-UNet. (a) Training data. (b) Convergence of VGG-UNet training. (c) Validation outcome.
(a)(b)(c)


## 3.4. Performance Evaluation

The merit of the automatic MS detection system is verified by computing the necessary image metrics. During this comparison, the common measures such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN) are initially computed.(i)
TP is the number of binary 1’s correctly predicted as 1(ii)
TN is the number of binary 0’s correctly predicted as 0(iii)
FP is the number of binary 0’s predicted as 1(iv)
FN is the number of binary 1’s predicted as 0The above measures help to get the related values, such as the true positive rate (TPR), true negative rate (TNR), false positive rate (FPR), and false negative rate (FNR) as discussed in [26–30]. Along with these measures, the other vital metrics, such as Jaccard, Dice, accuracy, precision, sensitivity, specificity, and negative predictive values (NPV), are also computed, and based on these values, the merit of the CNN is confirmed.The measures considered in this work are presented in equations (1)–(9):(1)TPR=sensitivity=TPTP+FN,(2)TNR=specificity=TNTN+FP,(3)FPR=FPFP+TN,(4)FNR=FNFN+TP,(5)Jaccard=TPTP+FP+FN,(6)Dice=2TP2TP+FP+FN,(7)Accuracy=TP+TNTP+TN+FP+FN,(8)Precision=TPTP+FP,(9)NPV=TNTN+FN.


## 4. Results and Discussion

This section presents the experimental outcomes and their discussion. Initially, the proposed work is executed using the pretrained CNN schemes, such as UNet, SegNet, VGG-UNet, and VGG-SegNet, using the 2D MRI slices and GT with a dimension of512×512×3 pixels. Every CNN scheme is individually trained and validated using axial, coronal, and sagittal plane 2D slices, and the achieved outcomes are recorded. Later, a comparative assessment is executed between the predicted MS lesion with GT, and the performance metrics are computed.Figure5 shows the sample test image, GT, and extracted MS lesion for chosen 2D MRI slices of Flair modality achieved using the proposed VGG-UNet scheme. Figures 5(a)–5(c) show the 2D planes, such as axial, coronal, and sagittal, respectively. A visual comparison confirms that the segmented image is similar to GT. The pixelwise evaluation of the segmented and GT provided the necessary performance metrics, and these values are given in Tables 1 and 2.Figure 5
Sample test images and the segmented MS lesion with VGG-UNet. (a) Sagittal. (b) Axial. (c) Coronal.
(a)(b)(c)Table 1
Initial performance metrics achieved for the sample test image.
2D planeTPFNTNFPTPRFNRTNRFPRSagittal75528192532295440.90220.09780.99790.0021Axial46911062568544930.97790.02210.99810.0019Coronal77451622535826550.97950.02050.99740.0026Table 2
Prime performance metrics computed for the sample test image.
2D planeJaccardDiceAccuracyPrecisionSensitivitySpecificityNPVSagittal84.711291.722899.480193.280690.216299.785699.6776Axial88.676793.998699.771590.490097.790399.808499.9587Coronal90.457894.989999.688392.202497.951299.742499.9362Table1 provides the initial metrics in which TPR and TNR are better and FPR and FNR are having a lesser value for all the 2D planes shown in Figure 5. The result of Table 2 confirms that the proposed approach helps to get a segmentation accuracy of >99% on the considered 2D Flair modality MRI slices. In order to confirm the overall performance of the CNN segmentation scheme, a glyph plot is constructed as shown in Figure 6, and this confirms that the overall metrics achieved with axial and coronal planes is better compared to the sagittal plane.Figure 6
Glyph plot demonstrating the overall performance.The proposed investigation is implemented on all 600 numbers of test images with axial, coronal, and sagittal planes, and the performance metrics are individually computed, and the overall value (mean and standard deviation) is then considered to validate the merit of the considered CNN segmentation schemes. Table3 provides the average metrics achieved with every approach, and to represent this comparison graphically, a spider plot is constructed as shown in Figure 7. Figure 7(a) shows the plot for sagittal plane images and Figures 7(b) and 7(c) show the plots for axial and coronal planes. All these results confirm that the overall result achieved with the proposed CNN segmentation scheme (VGG19 based VGG-UNet) helps to achieve a better result compared to the traditional UNet, SegNet, VGG-UNet, and VGG-SegNet.Table 3
Overall performance metrics computed for the test images of this study.
CNN scheme2D planeJaccardDiceAccuracyPrecisionSensitivitySpecificityUNetSagittal82 ± 1.1887 ± 2.6198 ± 0.6491 ± 2.9288 ± 2.7797 ± 1.08Axial81 ± 2.1488 ± 3.0698 ± 0.7191 ± 3.5288 ± 3.0898 ± 0.51Coronal85 ± 2.2988 ± 2.0598 ± 0.3291 ± 1.5587 ± 4.1697 ± 0.96SegNetSagittal82 ± 1.8489 ± 1.1697 ± 0.7291 ± 2.3787 ± 2.8496 ± 1.38Axial83 ± 3.0689 ± 1.5596 ± 1.2893 ± 2.1688 ± 3.0697 ± 0.52Coronal85 ± 1.1689 ± 2.7398 ± 0.5191 ± 1.3787 ± 3.3397 ± 0.68VGG-UNetSagittal83 ± 1.2291 ± 2.4798 ± 0.3891 ± 1.5789 ± 3.0398 ± 0.49Axial88 ± 1.0692 ± 2.1898 ± 0.1890 ± 2.5593 ± 1.6898 ± 0.52Coronal89 ± 1.8492 ± 3.1798 ± 0.2293 ± 2.0794 ± 2.4999 ± 0.06VGG-SegNetSagittal82 ± 2.0590 ± 2.2298 ± 0.2790 ± 2.0789 ± 2.5598 ± 0.28Axial87 ± 2.5391 ± 2.7198 ± 0.3190 ± 2.3291 ± 2.1698 ± 0.14Coronal87 ± 3.1591 ± 1.4598 ± 0.1691 ± 2.1792 ± 1.1898 ± 0.23VGG-UNet (proposed)Sagittal85 ± 3.0592 ± 2.1899 ± 0.0492 ± 2.1795 ± 0.4899 ± 0.03Axial91 ± 2.2194 ± 1.0599 ± 0.1094 ± 1.2698 ± 0.1299 ± 0.11Coronal90 ± 2.6293 ± 1.1199 ± 0.0393 ± 1.9497 ± 1.0599 ± 0.08Figure 7
Spider plot demonstrating the overall performance of the CNN segmentation schemes on the chosen images. (a) Sagittal plane images. (b) Axial plane images. (c) Coronal plane images.
(a)(b)(c)In order to validate the merit of the proposed scheme with the existing work, the Dice score is considered and is then compared with the dice value available in other earlier works in the literature [10–14]. Figure 8 confirms that the Dice score achieved with the proposed scheme is better compared to other works found in the literature, and this comparison confirms the significance of the proposed VGG-UNet in detecting the MS lesion from MRI slices.Figure 8
Comparison of Dice score of VGG-UNet with other schemes.In this work, the Flair modality MRI without skull section is considered for examination, and better values of image metrics, such as Jaccard, Dice, accuracy, precision, sensitivity, and specificity, are achieved for all the 2D planes with the proposed CNN segmentation scheme. In the future, this VGG-UNet can be considered to examine the MS lesion in Flair modality image with skull section and other MRI modalities available in the considered image database.


## 5. Conclusion

Early detection of the MS lesion is necessary to plan the treatment to reduce its harshness. In this work, the CNN segmentation approach is implemented to extract and evaluate the MS lesion from the Flair modality MRI slice. During this study, the various 2D MRI slices, such as axial, coronal, and sagittal planes, are separately evaluated. VGG19 supported VGG-UNet is implemented in this research to enhance the segmentation performance. The result of the proposed scheme is verified with the pretrained schemes, such as UNet, SegNet, VGG-UNet, and VGG-SegNet. The experimental result of the proposed scheme is implemented on 30 patient images (600 images in each 2D plane), and the proposed scheme helps to achieve Jaccard of >85%, Dice of >92%, and accuracy of >98%. In the future, the proposed VGG-UNet can be tested on other MRI modalities, such asT1 and T2, and it also can be considered to extract the MS lesion in images with skull section.



---

*Source: 4928096-2022-06-02.xml*