# Automatic semantic segmentation and detection of vertebras and intervertebral discs by neural networks

Sylvain Guinebert, Eric Petit, Valérie Bousson, Sylvain Bodard, Nicolas

Amoretti, Bruno Kastler

► To cite this version:

Sylvain Guinebert, Eric Petit, Valérie Bousson, Sylvain Bodard, Nicolas Amoretti, et al. Automatic semantic segmentation and detection of vertebras and intervertebral discs by neural networks. Computer Methods and Programs in Biomedicine Update, 2022, 2, pp.100055. 10.1016/j.cmpbup.2022.100055. hal-03636537

HAL Id: hal-03636537

https://hal.science/hal-03636537

Submitted on 11 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Automatic semantic segmentation and detection of vertebras and intervertebral discs by neural networks

Sylvain Guinebert<sup>a, *</sup>, Eric Petit<sup>a</sup>, Valérie Bousson<sup>d</sup>, Sylvain Bodard<sup>b</sup>, Nicolas Amoretti<sup>c</sup>, Bruno Kastler<sup>b</sup>

## ARTICLE INFO

### Keywords:

Artificial intelligence

Deep neural network learning

Magnetic resonance imaging

Spinal pathologies

Feasibility analysis

## ABSTRACT

**Objective:** To realize a tool for automatic segmentation and detection of lumbar disc degenerative disease and fractures in MRI images using convolutional neural networks.

**Material and methods:** We developed, on one hand, a Picture Archiving and Communication System (PACS) with a DICOM viewer (Digital Imaging and Communications in Medicine) to extract training data and implement 2 CNN networks dedicated to the segmentation task and on the other hand, the analysis of simple lumbar pathologies on the other hand. Two hundred forty-four MRI scans T2 weighted sagittal were selected at the university hospital Pasteur 2 of Nice - France. After accomplishing segmentation and classification task for all structures of interest, we trained two neural networks (U-Net++ and Yolov5x) to segment and detect discs and vertebras.

**Results:** Neural networks allowed semantic segmentations with a good accuracy of the order of 0.96 and 0.93 DICE index for intervertebral discs and vertebral bodies respectively. They are less effective in detecting of common pathologies (degenerative disc disease, disc herniation, vertebral fractures) with an area under the precision-recall curve of 0.88 for fractures and 0.76 for degenerative disc disease.

**Conclusion:** Our work has shown good efficiency for the segmentation of vertebral bodies and intervertebral discs, but progress remains to be made in the detection of degenerative discopathy and vertebral fractures. On this last point, we believe that the lower performance can be attributed to a lack of training images and that better performances will be achieved with the increase of the training data set on our home made PACS. We will also provide other regions of interest and conditions.

## Introduction

The purpose of computer vision is to give an interpretation of the information present in digital images. These techniques have shown some results especially in the medical field (diagnosis of certain diseases, surgery or microbiology).

It therefore seemed appropriate to apply these techniques to our specific field, namely spinal structures.

A concrete application could be the pre-operative planning in radio-intervention, neuro-surgery or orthopaedic surgery [10]. An important task for diagnosis, treatment and clinical research is the segmentation of vertebral bodies, spinal canal, dural sac, discs and roots on MRI images.

Accurate localization and tracing of these structures can be difficult, even when performed manually.

In addition, manual labelling of each MRI in multiple 3D regions is extremely time consuming and daunting.

For this initial work, we focused on degenerative spine disease and osteoporotic vertebral fractures, whose high prevalence, particularly in the elderly and working-age population, results in significant societal costs related to treatment and disability [23]. It is a major cause of low back pain [24] and a significant portion of these costs come from medical imaging, which is used for both initial diagnosis and follow-up evaluation in conservative and surgical treatments.

Magnetic resonance imaging (MRI) is the imaging modality of choice

<footnote><sup>a</sup> Laboratoire Images, Signaux et Systèmes Intelligents (LISSI) EA 3956, France</footnote>

<footnote><sup>b</sup> Department of Radiology, Necker CHU, Paris, France</footnote>

<footnote><sup>c</sup> Department of Radiology, Pasteur 2 CHU, Nice, France</footnote>

<footnote><sup>d</sup> Department of Radiology, Lariboisière CHU, Paris, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: sylvain.guinebert.sg@gmail.com (S. Guinebert).</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Classes histograms: left disk related labels, right vertebrae related labels (abciss: class indices – table 1, ordinate: number).</figcaption></figure>

<table><caption>**Table 1**
classes.</caption><thead><tr><th>class</th><th>disc</th><th>Vertebra</th></tr></thead><tbody><tr><td>0</td><td>L5-S1 normal</td><td>S1 normal</td></tr><tr><td>1</td><td>L5-S1 moderate degenerative discopathy</td><td>L5 normal</td></tr><tr><td>2</td><td>L5-S1 MODIC1 discopathy</td><td>L5 recent fracture</td></tr><tr><td>3</td><td>L5-S1 severe degenerative discopathy</td><td>L4 normal</td></tr><tr><td>4</td><td>L4-L5 normal</td><td>L4 recent fracture</td></tr><tr><td>5</td><td>L4-L5 moderate degenerative discopathy</td><td>L3 normal</td></tr><tr><td>6</td><td>L4-L5 MODIC1 discopathy</td><td>L3 recent fracture</td></tr><tr><td>7</td><td>L4-L5 severe degenerative discopathy</td><td>L2 normal</td></tr><tr><td>8</td><td>L3-L4 normal</td><td>L2 recent fracture</td></tr><tr><td>9</td><td>L3-L4 moderate degenerative discopathy</td><td>L1 normal</td></tr><tr><td>10</td><td>L3-L4 MODIC1 discopathy</td><td>L1 recent fracture</td></tr><tr><td>11</td><td>L3-L4 severe degenerative discopathy</td><td>T12 normal</td></tr><tr><td>12</td><td>L2-L3 normal</td><td>T12 recent fracture</td></tr><tr><td>13</td><td>L2-L3 moderate degenerative discopathy</td><td>T11 normal</td></tr><tr><td>14</td><td>L2-L3 MODIC1 discopathy</td><td>T11 recent fracture</td></tr><tr><td>15</td><td>L2-L3 severe degenerative discopathy</td><td>T10 normal</td></tr><tr><td>16</td><td>L1-L2 normal</td><td>T10 recent fracture</td></tr><tr><td>17</td><td>L1-L2 moderate degenerative discopathy</td><td>T9 normal</td></tr><tr><td>18</td><td>L1-L2 MODIC1 discopathy</td><td>T9 recent fracture</td></tr><tr><td>19</td><td>L1-L2 severe degenerative discopathy</td><td>L5 old fracture</td></tr><tr><td>20</td><td>T12-L1 normal</td><td>L4 old fracture</td></tr><tr><td>21</td><td>T12-L1 moderate degenerative discopathy</td><td>L3 old fracture</td></tr><tr><td>22</td><td>T12-L1 MODIC1 discopathy</td><td>L2 old fracture</td></tr><tr><td>23</td><td>T12-L1 severe degenerative discopathy</td><td>L1 old fracture</td></tr><tr><td>24</td><td>T11-T12 normal</td><td>T12 old fracture</td></tr><tr><td>25</td><td>T11-T12 moderate degenerative discopathy</td><td>T11 old fracture</td></tr><tr><td>26</td><td>T11-T12 MODIC1 discopathy</td><td>T10 old fracture</td></tr><tr><td>27</td><td>T11-T12 severe degenerative discopathy</td><td></td></tr><tr><td>28</td><td>T10-T11 normal</td><td></td></tr><tr><td>29</td><td>T10-T11 moderate degenerative discopathy</td><td></td></tr><tr><td>30</td><td>T10-T11 MODIC1 discopathy</td><td></td></tr><tr><td>31</td><td>T10-T11 severe degenerative discopathy</td><td></td></tr><tr><td>32</td><td>L5-S1 herniated disk</td><td></td></tr><tr><td>33</td><td>L4-L5 herniated disk</td><td></td></tr><tr><td>34</td><td>L3-L4 herniated disk</td><td></td></tr><tr><td>35</td><td>L2-L3 herniated disk</td><td></td></tr><tr><td>36</td><td>L1-L2 herniated disk</td><td></td></tr></tbody></table>

<caption>**Table 2**</caption>

Coefficients DICE for training/test datasets and binary cross entropy for validation dataset.

<table><caption>**Table 3**
Results of YOLOv5x.</caption><thead><tr><th></th><th>Training DICE</th><th>Validation/Test DICE</th><th>Validation BCE</th></tr></thead><tbody><tr><td>Vertebrae</td><td>0.87</td><td>0.93</td><td>0.02</td></tr><tr><td>Disks</td><td>0.91</td><td>0.96</td><td>0.01</td></tr></tbody></table>

<table><thead><tr><th></th><th>F1-score</th><th>mAP - 0.5</th><th>mAP - 0.95</th></tr></thead><tbody><tr><td>Vertebrae</td><td>0.81</td><td>0.881</td><td>0.601</td></tr><tr><td>Disks</td><td>0.68</td><td>0.765</td><td>0.701</td></tr></tbody></table>

to evaluate both spinal degenerative diseases and osteoporotic vertebral fracture due to its superior ability to characterize soft tissue detail such as neural tissue but also to identify the symptomatic vertebral fracture (even with a moderate height loss). It allows radiologists to identify the location, aetiology, and severity of these frequent diseases and report level-by-level these findings to referring physicians and their patients. Imaging reports thus inform clinical decision-making with regard to different therapeutic approaches and are also used to assess response to treatment.

It is virtually impossible to segment and compute residual volume of each disk and vertebra in a normal workflow based on our own experience as MSK radiologists.

To address this complicated challenge of spinal imaging segmentation and diagnosis screening, a variety of computer-aided diagnosis techniques have been explored over the past decade for potential applicability. Many of the previous works used computer vision techniques, such as histogram of orientated gradients [25, 26, 27], probabilistic models [28, 29, 30], and GrowCut [31], to perform segmentation or localize vertebral bodies and discs.

For automated diagnosis of spinal degeneration, Zhang et al [32], proposed a weakly-supervised approach to extract salient features for lumbar spinal stenosis in MRI using window filters and pathological labels only.

Indeed, convolutional neural networks (CNN) has led to deep learning approaches where algorithms automatically learn representative features from raw data at multiple different levels of abstraction to perform classification tasks at a high level of performance [33, 34, 35].

Recently, the U-Net [6] deep learning algorithm has proven to be effective in segmentation tasks even with limited data and has been used for vertebral segmentation in spinal CT [11] and X-ray imaging [12]. For example, DeepSpine [21] is a method based on a sagittal U-net pass to segment the vertebrae and then a neural network classifier to grade spinal canal and foraminal stenosis based on the sagittal and axial reconstruction. For automated diagnosis of spinal degeneration, Jamaudin et al [7, 8], made a model in which a multi-task VGG-M architecture was developed for Pfirrmann grading of disc degeneration, central canal stenosis, and other spinal diseases.

In this study, we tried to bring a simpler and faster one-pass approach
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Bounding boxes sizes histograms interdependency between box position (x, y), and box dimensions (width, height). All axes are normalized dimensions of the width and height of the image.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Example of manual segmentation with our software.</figcaption></figure>

in order to label the stage and condition of discs and vertebrae, and then used a second pass to obtain precise metrics of these structures of interest. This allows end-to-end trainable algorithm (no spine-curve fitting technique for example) and avoids multi-input heavy neural networks techniques. This tool allows to segment the anatomical regions of the spine (vertebral bodies and intervertebral discs) and to locate the thoracolumbar spinal level as well as to detect the most frequent abnormalities (degenerative disc disease, herniated discs, fractures of the vertebral bodies).

## Materials and methods

The premise of this work was based on the ability of deep learning

neural network models to identify features in MRI data that represent varying intensities or severities of pathologies or injuries in patients.

## Patients and data

This retrospective study was conducted in accordance with the tenets of the Declaration of Helsinki. Written consent was waived because of the retrospective nature of the study. The imaging data sets were provided by lumbar non-contrast MRI examinations of outpatients from the radiology department of Nice University Hospital over the period 2016–2018. The non-contrast imaging protocol is typically used for evaluation of low back pain and/or radiculopathy. We screened 980 studies but we excluded the majority of patients undergoing evaluation for potential spinal trauma, infection, inflammation, tumors, or post-surgical follow-up. These indications are outside the scope of our objective and require case-specific interpretation and/or measurements.

Our dataset was composed of 244 MRI acquisition volumes from 244 patients (14 images of 768 × 768 pixels, i.e. 3416 images), which were acquired in T2 STIR (Short Tau Inversion Recovery) sequence with a spatial resolution of 768 × 768px × 3 mm (Width × Height × Thickness) on 1.5T strength (GE Healthcare) MR scanners.

The average age of the 244 patients was 41 years, with a standard deviation of 15 years. There were 48% male and 52% female patients.

The volumes were partitioned into two sets: the training/validation set (204 vol, 80% of the images), and, in order to test our approach and compare the capabilities of our CNNs, a test set of 40 vol (20% of the images). Performing segmentation and detection on more than 40 test volumes is a good indicator of the real clinical applicability of CNN-
<figure><img src="image_5.png" /><figcaption>Fig. 4. Example of Multiplanar reconstruction used to check the quality of each segmentation.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. (from Zongwei Zhou et al. [36]): global representation of Unet++ (each node represents layers conv block). The backbone being the encoder.</figcaption></figure>

based automatic segmentation.

The study of the class distribution (level + condition) in the Fig. 1 below showed a non-uniform distribution between some classes of the training set. This is due to the different frequency of disc diseases and fractures at each thoracolumbar level in the population [37]. For example, the herniated disc is more frequent on the last level L5-S1 than in L1-L2.

The use of a weighted loss function (see Chapter model and architecture) was therefore necessary here in view of this imbalance.

## Preparation of training data

All volumes were acquired using an MRI modality and represent spinal structures in sagittal section in T2 STIR weighting.

We developed a centralized image archiving system on a server

capable of distributing the images over the network to a DICOM viewer (desktop and web version) allowing to read the exams and to perform all the usual reconstructions.

Each image in DICOM format contained 16-bit unsigned grayscale format and was exploited as normalized image with a rescale of intensity [-1.0, 1.0] for the neural network.

A radiologist was required to manually segment and label each structure of interest into one of the classes of Table 1. An example of the structures of interest is given in Fig. 7 and 8.

The annotation of the training images was carried out from our viewer, first manually and then semi-automatically. On the server side, once the networks were pre-trained with the first segmented samples, the segmentation was then performed in a semi-automatic way, i.e. with an AI run applied directly by the archiving server.

The server analysed each image twice, once for the vertebrae and
<figure><img src="image_7.png" /><figcaption>**Fig. 6.** The network architecture of Yolov5. It consists of three parts: (1) Backbone: CSPDarkNet, (2) Neck: PANet, and (3) Head: Yolo Layer. The data are first input to CSPDarknet for feature extraction, and then fed to PANet for feature fusion. Finally, head outputs detection results (class, score, location, size).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** Example of vertebra segmentation (normal condition): colors correspond to labels of table 1.</figcaption></figure>

then a second time for the discs by the two networks independently:

- A pass used the U-Net++ to segment into two classes (background/object of interest).

<figure><img src="image_9.png" /><figcaption>**Fig. 8.** Example of vertebra segmentation (fracture of D12 vertebral body): colors correspond to labels of table 1.</figcaption></figure>

- Another pass used YOLOv5x (You Only Look Once v5x) to automatically detect vertebral level, fractures, disc disease and disc herniation.

After that, the radiologist could apply manual correction to correct neural networks errors.

The final annotated images (ground truth) were then transferred on the archiving server.

We were therefore able to correctly classify the pixels of each instance in one of the 64-targeted classes.
<figure><img src="image_10.png" /><figcaption>**Fig. 9.** results on disk condition detection. F1-score on left with 0.68, with $p$>0.29 for all classes (blue line). Each grey line represents one class. Area Under Curve (AUC): 0.765 for all classes (blue line, min box overlap 0.5).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Fig. 10.** results on vertebrae condition detection. F1-score on left with 0.81, with $p$ > 0.44 for all classes (blue line). Each grey line represents one class. Area Under Curve (AUC): 0.881 for all classes (blue line) (min box overlap 0.5).</figcaption></figure>

For the development and validation of the algorithm, various splits of the dataset were used to ensure that the model was tested against cases that it had not seen during training. Train-validation percentages of 70%–30%, 80%–20%, and 90%–10% were used for various models.

## Design considerations

Our workflow consists of four independent tasks, which are segmentation and level-condition detection on disc and vertebra.

These tasks require both accurate segmentation at the pixel level and global identification of the structure of interest.

For precise segmentation, U-Net, being a reference in the field of biomedical segmentation, could have been suitable. However, we know that the most recent approaches [11, 4] propose to multiply the links between downstream and upstream layers to improve localization.

We have therefore chosen a more recent architecture, called U-Net++ [38] which is an improved version of the U-Net architecture with feedback of the semantic information extracted in the lowest layers to the higher layers.

For the detection task, we chose to work with a YOLOv5x type network because of its speed and lightness while offering a level of

accuracy of the same order as EfficientDet-D4 on Coco2017 (generalist image set). The network architecture of Yolov5 is shown in Fig. 5.

This efficiency is mainly due to the integration of cross stage partial network (CSPNet) [22] into Darknet, which solves the problems of repeated gradient information in large-scale backbones, and integrates the gradient changes into the feature map, thereby decreasing the parameters of model, which not only ensures the inference speed and accuracy, but also reduces the model size. Another advantage is that the head of Yolov5, namely the Yolo layer, generates 3 different sizes (18 × 18, 36 × 36, 72 × 72) of feature maps to achieve multi-scale [18] prediction, enabling the model to handle small (for example disc hernia) and big objects (for example vertebrae levels and discs)).

The combination of the results of the detection task locates the anomaly so that the expert can assess its reality, but also accurately labels the segmented structures (2 classes (background/object) -> 64 classes (vertebrae levels and discs)).

## Transfer learning

To overcome the lack of data, we used pre-trained weights on generalist datasets IMAGENET for U-Net++ and COCO 2017 for
<figure><img src="image_12.png" /><figcaption>**Fig. 11.** Example of detection of vertebrae level and condition by yolo. N for normal, Fract. for recent fracture (with oedema), A-fract. for old fracture (without oedema).</figcaption></figure>

Yolov5x (a technique known as transfer learning).

Therefore, rather than training a completely blank network, the weights of the lower levels were optimized by a feed-forward approach to recognize structures found in images in general, and by recycling the weights of the higher levels using backpropagation. This method allows for better results and faster learning.

We justified our approach based on results obtained on classification tasks where the pre-trained backbone, which extracts low level features from images, can be used on specialized datasets and has shown good results [39, 40].

For Unet++, we froze the backbone corresponding to the encoder part of the network, which extracts low-level features learned from ImageNet. The decoder and skip-connection layers were all fine-tuned to fit the segmentation task.

Following the same principle, the first 10 layers of yolov5 were frozen, which corresponds to the CSPDarknet backbone (Fig. 6) dedicated to image feature extraction (pre-trained with COCO2017). All the layers of the neck and the head of yolov5 were fine-tuned.

## Hyperparameters configuration

The model coefficients were optimized using Adam Optimization Algorithm in both cases.

Adam's initial learning rate was 0.01, and beta coefficient was 1 for both models.

We used a weight decay (L2 regularization) strategy $5.10^{-5}$ to avoid overfitting.

For yolo, we used different ponderations for each loss terms (see below), 0.05 for the box dimensions loss terms, 0.5 for the class loss term, 1.0 for the object loss term with a minimal interlap over union $> 0.20$ to consider valid object detection. In addition, we used a 1.5 focal loss coefficient.

## Loss function

For the U-Net++ network we chose a loss function based on the DICE index corresponding to $Dice = \frac{2|X \cap Y|}{|X| + |Y|}$ (X is the computed pixel and Y the ground truth) which is commonly used and recommended over cross entropy to train and evaluate the segmentation's quality.
As an indication only, we also computed the cross-entropy index (CEI).

For the YOLOv5x network, a complex cross entropy function between the results and the ground truth was used to represent the error of the boxes, of the presence of an object and finally of the probability of the classification of the object (binary cross entropy (BCE) on the class).

We used a focal loss function to compensate for the imbalance of the different classes:

$$Position\ Loss = \lambda_{coord} \sum_{i=0}^{S^2} \sum_{j=0}^{B} 1_{ij}^{obj} \left[ (x_{ij} - \hat{x}_{ij})^2 + (y_{ij} - \hat{y}_{ij})^2 \right]$$

$$Size\ Loss = \lambda_{coord} \sum_{i=0}^{S^2} \sum_{j=0}^{B} 1_{ij}^{obj} \left[ (w_{ij} - \hat{w}_{ij})^2 + (h_{ij} - \hat{h}_{ij})^2 \right]$$

$$Object\ Loss = \sum_{i=0}^{S^2} \sum_{j=0}^{B} 1_{ij}^{obj} (C_{ij} - \hat{C}_{ij})^2 + \lambda_{noobj} \sum_{i=0}^{S^2} \sum_{j=0}^{B} 1_{ij}^{noobj} (C_{ij} - \hat{C}_{ij})^2$$

$$Class\ Loss = \sum_{i=0}^{S^2} 1_{ij}^{obj} \sum_{c \in classes} (p_i(c) - \hat{p}_i(c))^2$$

$$Loss = position\ loss + size\ loss + object\ loss + class\ loss$$

The index i represents the grid number S in the cut image; the index j represents the number of box B; x, y, w, h represent respectively the coordinates of the centre and the width, height of the box.

$1_{ij}^{obj}$: 1 if there is an object 0 else. $1_{ij}^{noobj}$: 1 if there is no object 0 else. $\lambda_{coord}$ and $\lambda_{noobj}$ represents focal loss coefficient.

C represents the probability of an object being present in each box (the box confidence). $p_i$ represents the class probability in each cell.

The first 2 terms represent the loss as a function of the size difference between the selected boxes and the ground truth.

The 3rd and 4th terms represent the loss according to the presence or not of an object in each box (0 or 1).

The last term represents the loss according to the classes assigned as probable to each box.

## Data augmentation

To have a good output, high volumes of data are needed for CNN models. Data augmentation is a process consisting in artificially increasing the training dataset by performing a series of transformations to the image, such as rotations, reflections, and elastic deformations [22].

These transformations are usually selected based on the analysed data and subjective criteria based on researchers' experience [22].

Data augmentation is also an effective method to reduce the “over-fitting” of CNNs caused by limited training samples.

This was achieved through a combination of random image translation (max 30% of dimensions), central cropping (max 20% of dimensions), horizontal flipping (0.5 probability) and rotation (max 45°). Moreover, Gaussian noise augmentation was used to mimic low-quality noisy images. We also used specific mosaic data augmentation for yolo training [20].

The images were normalized using the mean and standard deviation of each image, since MRI datasets and image interpretation are very different from the ImageNet and coco2017 datasets. It would not have been relevant to classify image by global mean or standard deviation difference since they can differ from each constructor and since they are processed to match a standard window width and centre adapted for the human eye [19].

# Results

## Results of semantic segmentation tests

We tested U-Net++ on 40 vol (600 images), using disc and vertebra annotations.

Table 2 shows the average performance in terms of estimated DICE coefficients compared with ground truth annotations.

We also verified by visual inspection of an expert radiologist that the annotation adequately delineated the regions of interest, even in the case of severe fractures or advanced disc disease. The localization failure rate (min overlap > 0.50) of the U-Net++ segmentation was zero in our experiment.

Visual examples of MRI segmentation results can be seen in Fig. 5, 6 and following.

It should be noted that the automatic CNN segmentation was extremely flexible and capable of correctly localizing/segment these structures despite high anatomical variability, shape or signal variants of the structures of interest that are difficult even to consider in manual approaches.

## Level detection test results and pathologies

We tested YOLOv5x on all 40 vol (600 images), using the respective annotations of the discs and vertebrae.

The results of the YOLO network were as follows:

For disc identification and disc level/pathology status:

- AP95 (min box overlap > 0.95): 0.601

- AP50 (min box overlap > 0.5): 0.765

For identification of vertebrae and their level/state:

- AP95: 0.701

- AP50: 0.881

A precision of 100% can be obtained if the activation probability threshold is very high (>0.90).

The F1-score was:

- 0.81 maximum with a probability threshold > 0.440 for vertebral bodies and

- 0.68 with a probability threshold > 0.3 for the intervertebral discs.

Object detection and box localization was excellent (mAP > 0.99) when the experiment was performed with all classes treated as one (background vs object).

The errors were therefore mainly due to classification errors, in particular on the unbalanced classes in the dataset (and this despite the use of the "focal loss").

Figs. 9 and 10 show the average performance of the YOLOv5x network in terms of Mean Average Precision (mAP) coefficients, which represents the area under the inverse of the precision as a function of the recall, and of the F1-Score estimated in relation to the ground truth annotations.

# Discussion

Regarding the detection task, the results of this work on 244 lumbar MRIs show that the YOLOv5x network approach is able to produce fast detection (>25frames per second [9]) of discs and vertebra as well as their state.

To our knowledge, no study has adopted our approach with a segmentation and a detection network. The main expected interest was a gain in time and simplicity, if not in quality. We were able to ascertain the location of each disease in the image and to quantify volumes or areas of the structure of interest.

Other authors have attempted image analysis for disc herniation with a recall of 81.8% and specificity of 87.4% [1, 7, 8, 13] and for radiography (X-ray) fractures detection with a recall of 89%, 83%, and 95% [2, 3, 4].

In comparison, we obtained a recall of 76.5% for a precision of 75%
for the disc disease detection and obtained a recall of 88% for a precision of 90% for the vertebral fracture.

The modest overall accuracy of YOLOv5x neural networks may have
several explanations:

- A lack of data to train the model, evidence being its tendency to overlearn on the training dataset seen on the loss function curves.

- An intrinsic limitation of these neural networks unable to provide very high-quality object detection on generalist datasets (COCO2017 - IMAGENET). It can be noted that our results with the YOLOv5x network are consistent with those obtained on "mainstream" image sets such as COCO2017 (AP95: 0.55; AP50: 0.69) [9].

- Another possible problem may lie in the underlying DICOM datasets, which are obtained on 1.5 T machines that are often noisy compared to high field MRI.

However, compared to other work on MRI spine [1, 5], we believe
that the main problem of object misclassification is mainly due to the
insufficient volume of available data.

Regarding segmentation, we obtained satisfying results, equivalent to the results obtained by other teams [11, 12, 14, 15, 16, 17, 18] and in particular with the DeepSpine network [5], which obtained 0.93 DICE index for vertebrae and discs in 2018.

We find it difficult to do better with current technologies and limited
datasets in size.

In general, we found that U-Net++ gave better DICE coefficients when the region was larger and with high contrast. When smaller or very low contrast we used, it was in all cases localized and classified, but somewhat more difficult to segment.

Training CNNs usually requires a large amount of data in order to
obtain satisfying results in terms of pixel/voxel classification and to
perform satisfactory semantic segmentation. However, as mentioned, it
is rarely possible to obtain such large annotated datasets in a clinical
setting. Using a strategy based on a derivative of the U-Net++ made it
possible to locate the anatomy of interest with good accuracy and an
acceptable amount of data in a clinical setting.

Concerning real application, the validation test on 40 MRIs showed
maximum recall measures above 99% for detecting discs, degenerative
disc disease/herniated discs, and for vertebral numbering, presence of
vertebral fracture for a low probability threshold *p* > 0.20 (which gen-
erates false positives and poor positive predictive value).

The interest of these techniques could lie in the possibility of sorting the examinations (in the manner of a diagnostic test) to improve the order of reading from the most urgent, to the least urgent, i.e. to avoid ignoring a pathological image during a repetitive task. This may attract the attention of the radiologist, knowing that it implies a low diagnostic value of the test (and thus a severe sorting of false positives).(Fig 11, 2, 3)

Conclusions

In this study, we applied the neural network technique (Artificial
Intelligence) to the segmentation of spinal MR images and to the iden-
tification of pathologies.

We consider that if the objective was reached on the task of seg-
mentation of the discs and vertebrae, there is still progress to be made on
the task of detecting pathologies of these discs and vertebrae.

It is important to note that despite the lack of data to train the model,
our PACS system is now operational and will be used to augment the
dataset and be extended to other common diseases.

The objective of the level identification task is achievable with yolo-
like networks, but requires more data.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] K.U. Lewandrowski, N. Muraleedharan, S.A. Eddy, V. Sobti, B.D. Reece, J. F. Ramirez León, S Shah, Feasibility of Deep Learning Algorithms for Reporting in Routine Spine Magnetic Resonance Imaging, Int. J. Spine Surg. 14 (s3) (2020), https://doi.org/10.14444/7131. DecS86-S97PMID: 33298549; PMCID: PMC7735442.

[2] Y.C. Li, H.H. Chen, H. Hornig-Shing Lu, H.T. Hondar Wu, M.C. Chang, P.H Chou, Can a Deep-learning Model for the Automated Detection of Vertebral Fractures Approach the Performance Level of Human Subspecialists? Clin. Orthop. Relat. Res. 479 (7) (2021) 1598-1612, https://doi.org/10.1097/ CORR.0000000000001685. Jul 1PMID: 33651768; PMCID: PMC8208416.

[3] C. Rainey, J. McConnell, C. Hughes, R. Bond, S. McFadden, Artificial intelligence for diagnosis of fractures on plain radiographs: a scoping review of current literature, Intelligence-Based Med. 5 (2021), 100033, https://doi.org/10.1016/j. ibmed.2021.100033.

[4] D.H. Kim, T. MacKinnon, Artificial intelligence in fracture detection: transfer learning from deep convolutional neural networks, Clin. Radiol. 73 (5) (2018) 439–445, https://doi.org/10.1016/j.crad.2017.11.015. MayEpub 2017 Dec 18. PMID: 29269036.

[5] T. Lu, S. Pedemonte, B. Bizzo, et al., DeepSPINE: automated lumbar vertebral segmentation, disc-level designation, and spinal stenosis grading using deep learning, Comput. Sci. (2018). Preprint posted on July 26, 2018. arXiv:1807.10215 [cs.CV].

[6] Olaf Ronneberger, Philipp Fischer, Thomas Brox, U-Net: convolutional Networks for Biomedical Image Segmentation, LNCS 9351 (2015) 234–241, https://doi.org/10.1007/978-3-319-24574-4_28.

[7] Amir Jamaludin, Timor Kadir, Andrew Zisserman, SpineNet: automated classification and evidence visualization in spinal MRIs, Med. Image Anal. 41 (2017) 63–73, a.

[8] Amir Jamaludin, Meelis Lootus, Timor Kadir, Andrew Zisserman, Jill Urban, Michele C Battie, Jeremy Fairbank, Iain McCall, Genodisc Consortium, et al., Issls prize in bioengineering science 2017: automation of reading of radiological features from magnetic resonance images (MRIs) of the lumbar spine without human intervention is comparable with an expert radiologist, Eur. Spine J. 26 (5) (2017) 1374–1383, b.

[9] ultralYTICS/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations DOI: https://doi.org/10.5281/zenodo.3908559.

[10] Reginald Fayssoux, Neil I Goldfarb, Alexander R Vaccaro, James Harrop, Indirect costs associated with surgery for low back pain-a secondary analysis of clinical trial data, Popul. Health Manag. 13 (1) (2010) 9–13, https://doi.org/10.1089/ pop.2009.0061.

[11] Rens Janssens, Guodong Zeng, Guoyan Zheng, Fully automatic segmentation of lumbar vertebrae from CT images using cascaded 3D fully convolutional networks, In Biomed. Imaging (ISBI 2018), 2018 IEEE 15th Int. Symp. (2018) 893–897, pagesIEEE.

[12] S.M. Masudur Rahman Al Arif, Karen Knapp, Greg Slabaugh, Shape-aware deep convolutional neural network for vertebrae segmentation. International Workshop and Challenge on Computational Methods and Clinical Applications in Musculoskeletal Imaging, Springer, 2017, pp. 12–24, pages.

[13] Fu M.C., Buerba R.A., Long W.D. 3rd, Blizzard D.J., Lischuk A.W., Haims A.H., Grauer J.N. Interrater and intrarater agreements of magnetic resonance imaging findings in the lumbar spine: significant variability across degenerative conditions. Spine J.. 2014 Oct 1;14(10):2442-8. doi: 10.1016/j.spinee.2014.03.010.

[14] Ghosh, Subarna & Malgireddy, Manavender & Chaudhary, Vipin & Dhillon, Gurmeet. (2012). A new approach to automatic disc localization in clinical lumbar MRI: combining machine learning with heuristics. Proceedings - International Symposium on Biomedical Imaging. 114–117. 10.1109/ISBI.2012.6235497.

[15] Lootus, Meelis & Kadir, Timor & Zisserman, Andrew. (2014). Vertebrae Detection and Labelling in Lumbar MR Images. Lecture Notes in Computational Vision and Biomechanics. 17. 219–230. 10.1007/978-3-319-07269-2_19.

[16] Milletari, Fausto & Navab, Nassir & Ahmadi, Seyed-Ahmad. (2016). V-Net: fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. 565–571. 10.1109/3DV.2016.79.

[17] A.B. Oktay, Y.S. Akgul, Simultaneous localization of lumbar vertebrae and intervertebral discs with SVM-based MRF, IEEE Trans. Biomed. Eng. 60 (9) (2013) 2375-2383. Sep10.1109/TBME.2013.2256460.

[18] Alomari R.S., Corso J.J., Chaudhary V. Labeling of lumbar discs using both pixel- and object-level features with a two-level probabilistic model. IEEE Trans. Med. Imaging. 2011, Jan;30(1):1–10. doi: 10.1109/TMI.2010.2047403.

[19] Hansen M.S., Kellman P. Image reconstruction: an overview for clinicians. J. Magn. Reson. Imaging. 2015 Mar;41(3):573–85. doi: 10.1002/jmri.24687.

[20] Bag of Freebies for Training Object Detection Neural Networks Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li Amazon Web Services arXiv:1902.04103v3 2020.

[21] Deep Spine: Automated Lumbar Vertebral Segmentation, Disc-Level Designation, And Spinal Stenosis Grading Using Deep Learning Jen-Tang Lu, Stefano Pedemonte, Bernardo Bizzo, Sean Doyle, Mark H. Michalski, R.Gilberto Gonzalez, Stuart R. Pomerantz arXiv:1807.10215v1.2020.

[22] L.F. Sánchez-Peralta, L. Bote-Curiel, A. Picón, F.M. Sánchez-Margallo, J.B. Pagador, Deep learning to find colorectal polyps in colonoscopy: a systematic literature review, Artif. Intell. Med. 108 (2020), 101923, https://doi.org/10.1016/j. artmed.2020.101923.

[23] R.A. Deyo, S.K. Mirza, B.I. Martin, W. Kreuter, D.C. Goodman, J.G. Jarvik, Trends, major medical complications, and charges associated with surgery for lumbar
spinal stenosis in older adults, JAMA 303 (13) (2010) 1259-1265, https://doi.org/
10.1001/jama.2010.338.

[24] Fayssoux R., Goldfarb N.I., Vaccaro A.R., Harrop J. Indirect costs associated with surgery for low back pain-a secondary analysis of clinical trial data. Popul. Health Manag. 2010 Feb;13(1):9-13. doi: 10.1089/pop.2009.0061.

[25] Subarna Ghosh, Manavender Malgireddy, Vipin Chaudhary, Gurmeet Dhillon, A new approach to automatic disc localization in clinical lumbar MRI: combining machine learning with heuristics, in: Proceedings - International Symposium on Biomedical Imaging, 2012, pp. 114-117, 10.1109/ISBI.2012.6235497.

[26] Ayse Oktay, Yusuf Akgul, Simultaneous Localization of Lumbar Vertebrae and Intervertebral Discs With SVM-Based MRF, IEEE Trans. Biomed. Eng. (2013) 60, 10.1109/TBME.2013.2256460.

[27] Meelis Lootus, Timor Kadir, Andrew Zisserman, Vertebrae Detection and Labelling in Lumbar MR Images, Lecture Notes in Comput. Vis. Biomech. 17 (2014) 219-230, 10.1007/978-3-319-07269-2_19.

[28] Jason J Corso, Alomari RajaS, Vipin Chaudhary, Lumbar disc localization and labeling with a probabilistic model on both pixel and object features, Int. Conf. Med. Image Comput. Comput.-Assist. Interv. (2008) 202-210, pagesSpringer.

[29] Melih Aslan, Asem Ali, Rara Rara, Aly Farag, An automated vertebra identification and segmentation in CT images, in: Proceedings - International Conference on Image Processing, 2010. ICIP. 233 - 236. 10.1109/ICIP.2010.5651959.

[30] R.S. Alomari, J.J. Corso, V. Chaudhary, Labeling of lumbar discs using both pixel- and object-level features with a two-level probabilistic model, IEEE Trans. Med. Imaging 30 (1) (2011) 1-10, https://doi.org/10.1109/TMI.2010.2047403. Jan.

[31] J. Egger, C. Nimsky, X. Chen, Vertebral body segmentation with GrowCut: initial experience, workflow and practical application, SAGE Open Med. 5 (2017), https://doi.org/10.1177/2050312117740984. Nov 132050312117740984.

[32] Zhang, Qiang & Bhalerao, Abhir & Hutchinson, Charles. (2017). Weakly-Supervised Evidence Pinpointing and Description. 10.1007/978-3-319-59050-9_17.

[33] Varun Gulshan, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al., Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, JAMA 316 (22) (2016) 2402-2410.

[34] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes van Diest, Bram van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM van der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al., Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer, JAMA 318 (22) (2017) 2199-2210.

[35] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, Sebastian Thrun, Dermatologist-level classification of skin cancer with deep neural networks, Nature 542 (7639) (2017) 115.

[36] A. Jamaludin, T. Kadir, Zisserman A. SpineNet, Automated classification and evidence visualization in spinal MRIs, Med. Image Anal. 41 (2017) 63-73, https://doi.org/10.1016/j.media.2017.07.002, OctEpub 2017 Jul 21. PMID: 28756059.

[37] J. Jordan, K. Konstantinou, J O'Dowd, Herniated lumbar disc, BMJ Clin. Evid. (2009). Mar 26;2009:1118. PMID: 19445754; PMCID: PMC2907819.

[38] UNet++: A Nested U-Net Architecture for Medical Image Segmentation Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang (2018). arXiv:1807.10165v1 [cs.CV] 18 Jul.

[39] M.A. Morid, A. Borjali, G Del Fiol, A scoping review of transfer learning research on medical image analysis using ImageNet, Comput. Biol. Med. (2020), 104115.

[40] Hosseinzadeh Taher M.R., Haghighi F., Feng R., Gotway M.B., Liang J. (2021) A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis. In: Albarqouni S. et al. (eds) Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI For Resource Diverse Global Health. DART 2021, FAIR 2021. Lecture Notes in Computer Science, vol 12968. Springer, Cham. doi:10.1007/978-3-030-87722-4_1.