Region‐based two‐stage MRI bone tissue segmentation of the knee joint

IET Image ProcessingVolume 16, Issue 13 p. 3458-3470 ORIGINAL RESEARCHOpen Access Region-based two-stage MRI bone tissue segmentation of the knee joint Jianping Mao, Jianping Mao The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this authorPeng Men, Peng Men The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this authorHao Guo, Corresponding Author Hao Guo [email protected] orcid.org/0000-0001-5834-9529 The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, China Correspondence Hao Guo, No.1 Linghai Street, 116026 Dalian, P. R. China. Email: [email protected]Search for more papers by this authorJubai An, Jubai An The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this author Jianping Mao, Jianping Mao The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this authorPeng Men, Peng Men The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this authorHao Guo, Corresponding Author Hao Guo [email protected] orcid.org/0000-0001-5834-9529 The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, China Correspondence Hao Guo, No.1 Linghai Street, 116026 Dalian, P. R. China. Email: [email protected]Search for more papers by this authorJubai An, Jubai An The Department of Computer Science and Technology, Dalian Maritime University, Dalian, Liaoning, ChinaSearch for more papers by this author First published: 04 July 2022 https://doi.org/10.1049/ipr2.12475AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In medical image segmentation, the neural network structure of the U-Net family has demonstrated sufficient advantages. However, MRI images have different scan parameters and different scan times, resulting in different feature representation of the images. Furthermore, there is a great class imbalance between bone and cartilage tissues in MRI knee images. To address these issues, a region-based two-stage MRI knee bone tissue segmentation network is proposed in this paper. The segmentation network makes full use of the location characteristics of the three types of bone tissue in the knee joint and uses a two-stage network architecture with a modified U2-Net backbone network to segment MRI knee bone tissue. The neural network structure is divided into two phases, the first phase with a simple coded decoding structure for saliency detection to obtain the positional regional relationships of different bone tissues, and the second phase with a segmentation network consisting of 2 modified U2-Net, one for segmenting the patella and associated cartilage and the other for segmenting the femur, tibia and associated cartilage. The algorithm was tested with a variety of MRI knee data to verify the effectiveness of the algorithm. 1 INTRODUCTION Semantic segmentation is one of the fundamental tasks in computer vision, where the goal of semantic segmentation is to assign semantic labels to each pixel in an image. Compared to traditional semantic segmentation algorithms, convolutional neural networks based on deep learning [1] have shown significant advances in the semantic segmentation task. In clinical and medical research applications, high-fidelity semantic segmentation of MRI images is important for estimating histomorphometric and relaxation parameters. The segmentation of knee bone tissue is the basis for 3D modelling of the knee joint, which in turn allows for the quantitative analysis and evaluation of common diseases such as osteoarthritis, bone tumours and cartilage damage. For example, when cartilage is damaged in the knee joint, surgical repair of the cartilage tissue using a replacement material is required. The volume or surface area of the filled material can be obtained visually with the help of a reconstructed 3D model. However, currently, the 3D modeling fails due to missegmentation of the knee tissue caused by different MRI modalities and cartilage adhesions (as shown in Figure 1a,b). For example, segmentation errors can lead to mislocalisation of bone tumour lesions and thickening of the cartilage tissue in the knee after reconstruction. FIGURE 1Open in figure viewerPowerPoint Different types of bone tissue have different characteristics. (a),(b) Chondral adhesions on PD-weighted imaging. (c), (d) Different grey composition of the same bone tissue on T1-weighted and PD-weighted imaging. (e),(f) Different bone tissues appear to have the same grey-scale composition during T2-weighted imaging Unlike natural light images, MRI images mainly have the following characteristics: (1) Patellar cartilage and femoral cartilage are adherent to each other in most MRI images and are difficult to distinguish, as shown in the circles in Figure 1a,b. (2) Different MRI imaging modalities lead to large differences in the grey scale of the same tissue in different images, making the generalisation ability of the neural network much weaker, as shown in Figure 1c,d. (3) In the same MRI image, the imaging features of different bone tissues are the same, as in Figure 1e,f, the femur and tibia have the same pixel grey scale, and it is difficult to distinguish the femur from the tibia without texture information features. Traditional automated segmentation methods employ 3D statistical shape modelling modulated by anisotropic regularisation [2] or multi atlas segmentation. However, this technique is very sensitive to deformations in knee shape, which may be caused by changes in the size of the patient's knee and the incidence and progression of pathology. In recent years, various automatic segmentation methods for MRI knee bone tissue have been proposed [3, 4]. Shan [5] et al. proposed a new atlas-based automatic cartilage segmentation method for automatic OA (osteoarthritis) studies. The method proposes a multi-atlas segmentation strategy based on label fusion of non-local patches that can reliably identify candidate regions of cartilage. Dam [6] et al. used data from the Osteoarthritis Initiative tissue for knee image quantification to automatically segment the knee joint in both high and low fields of view. This paper presents a fully automated segmentation framework for knee MRI. The framework incorporates multi-atlas rigid alignment based on voxel classification and is trained on manual segmentation with different bone, cartilage and meniscal conformations. With the development of deep learning in recent years, convolutional neural networks (CNNs) have greatly contributed to the advancement of various semantic segmentation networks. Szegedy [7] et al. presented a deep convolutional neural network architecture codenamed Inception at ILSVRC14. The main feature of this architecture is to improve the utilisation of computational resources in the network. A typical example of this is GoogLeNet. Subsequently, Long [8]et al. proposed the first deep learning-based end-to-end semantic segmentation network, FCN, over state-of-the-art traditional semantic segmentation. FCN improves classification neural networks (AlexNet [9], VGG [10] and GoogleLeNet) into fully convolutional neural networks and passes their learned representations to the segmentation task by fine-tuning. After FCNs were proposed, deep learning-based neural networks have made great progress in semantic segmentation. Examples include, SegNet [11], PSPNet [12] and a series of DeepLab algorithms [13-15]. For the same knee tissue, there are many modalities in the MRI knee image due to different scanning methods and different thicknesses. With traditional methods (e.g., watershed algorithms, threshold-based segmentation algorithms etc.), a threshold value needs to be set for each modality. The process of setting thresholds requires several experiments to get a suitable threshold. This problem can be solved effectively if deep learning is used. The deep learning segmentation model has the ability to generalise to different modalities of data, so that we can segment MRI images of different modalities with only one training session. Moreover, for the problem of cartilage adhesions to each other, traditional segmentation methods cannot set a suitable threshold due to the similar pixel composition of cartilage tissue in the adhesion region, which in turn results in incorrect segmentation. A deep learning-based segmentation method will determine the different cartilage tissues based on the contextual information of the image (e.g., cartilage near the femur is femoral cartilage and cartilage near the patella is patellar cartilage), and thus obtain good segmentation results. Unlike ordinary optical images, medical images have small numbers, small segmentation targets, uneven categories, and after multiple convolution operations some fine categories are lost and the segmentation targets drift etc. Olaf et al. proposed a convolutional neural network named U-Net [16] for biomedical image segmentation, which successfully trained deep neural networks requiring thousands of annotated training samples. U-Net proposes neural network and training strategy that can be trained end-to-end with very few images. Subsequently, Fausto et al. proposed a volume-based, fully convolutional, neural network for 3D image segmentation, V-Net [17]. V-Net introduced a new objective function, optimized in training, based on dice coefficients. Abdulkadir et al. proposed 3D U-net for learning dense volumetric segmentation from sparse annotations [18], extending the previous U-Net framework by replacing 2D operations with 3D operations. Liu [19] et al. proposed a deep convolutional neural network and 3D deformation method for tissue segmentation in musculoskeletal magnetic resonance imaging. The method uses CNN and three-dimensional (3D) simplex deformation modelling to improve the accuracy and efficiency of cartilage and bone segmentation within the knee joint. Zhou et al. proposed a deeply supervised Encoder-Decoder UNet++ [20], where the Encoder-Decoder sub-networks are connected through a series of nested dense skip paths. Arjun [21] et al. used convolutional neural networks for MRI semantic segmentation to quantify the impact of three factors related to CNN segmentation performance: network architecture, training loss function and training data features to assess the impact of these variations on femoral cartilage segmentation. Huang et al. proposed UNet3+ [22], which makes use of full-scale jump connectivity and deep supervision. Wang [23] et al. constructed a 2D-3D hierarchical feature fusion network based on a convolutional codec structure for automatic segmentation of knee joints. Zhang [24] et al. added an attention mechanism to the U-Net for improving the comprehensive performance of knee cartilage. Compared to conventional segmentation algorithms, medical image segmentation networks based on deep learning possess stronger competitive performance. The prevailing semantic segmentation network is a Encoder-Decoder structure, and these Encoder-Decoder structures for segmentation all have one commonality: namely, downsampling followed by upsampling and jump connectivity. These commonalities have been shown to be effective in learning high-level semantic information as well as shallow texture information of the image. In the field of medical image segmentation, the most representative neural networks are the CNN-based U-Net series, such as U-Net, UNet++, U-Net3+ etc. Semantic segmentation of MRI medical images requires information about lesion regions, soft tissue morphology and related categories to be learned from high-level semantic information and local location information. Neural networks composed of CNN can make great progress in the application of images because CNNs can extract not only texture local information of images but also high-level semantic features. In this paper, the pixels in the high-level feature map formed after deep convolution are mapped to the bone tissue type represented by the perceptual field of the original map region as high-level semantic features. For example, if the original map is 224 × 224 and the feature map is 28 × 28 after a 3 × 3 convolution with a 3-layer step size of 2 and a padding of 1, the perceptual field of each pixel in the 28 × 28 feature map is 8 × 8, where the high-level semantic features refer to which type of bone tissue is in the 8 × 8 region or which types of bone tissue are in the region. In order to obtain richer semantic information, the number of layers of CNN needs to be continuously deepened, but as the number of layers of CNN increases, so does the loss of local location information from its generalisation, which is important for the prediction of focal regions and the organisation of edge morphology in MRI images. In this paper, one of the important tasks is the need to predict and classify the location regions of MRI knee bone tissue and corresponding cartilage tissue, as shown in Figure 2a. The cartilage region accounts for a very small percentage of the entire MRI knee image, which makes local location information have a great impact on the prediction of cartilage. The difficulty in this study is not only the small percentage of cartilage region, which is difficult to segment in detail, but also the fact that the femur and tibia are similar in their pixel information during MRI imaging, and the neural network tends to confuse the two, as shown in Figure 2b. After U-Net segmentation, it is common for the segmentation maps to have femurs interspersed with tibias, and for the cartilage tissue to be very poorly predicted. FIGURE 2Open in figure viewerPowerPoint Mixing of bone tissue and mis-segmentation of cartilage tissue during U-Net-based segmentation. (a) Original image (T1-weighted imaging). (b) U-Net-based segmentation map, (c)Hand-labelled images The current state-of-the-art medical segmentation networks are all based on U-Net improvements and are single stage segmentation models. Current state-of-the-art medical segmentation networks work well on many datasets (e.g., brain CT images, heart CT images etc.) because they have a single structure (e.g., brain tissues are all left-right) and few tissue classes (e.g., brain only has brain tissue and non-brain tissue). However, MRI images of the knee have a greater variety of bone tissues, including femur, tibia and patella, as well as the corresponding cartilage. Moreover, the most important feature of knee images is the problem of cartilage adhesions, which cannot be solved by a single-stage segmentation network. We therefore propose a new network architecture specifically for the segmentation of MRI knee bone tissue. The contributions of this study are as follows: We propose a two-stage semantic segmentation architecture based on U2-Net [25]. The first stage of this network uses a simple convolutional neural network-based coding and decoding structure to perform saliency detection on MRI images to infer the approximate locations of different bone tissues and cartilage tissues; The bone tissue is then cut out separately according to the different areas. The cropped images are segmented semantically in the second stage of the network using separate U2-Net with different degrees of improvement. We have solved the problem of patellar and femoral cartilage adhering to each other by segmenting different regions of bone tissue using different networks. Placing the femur and tibia in the same segmentation network simplifies the segmentation task and solves the problem of mixed femurs in the tibia. Numerous experiments have shown that our two-stage segmentation network is effective and yields more significant performance gains than the improved one-stage segmentation networks based on U-Net (e.g., UNet++,UNet3+,Att U-Net). In this paper, we improve the structure of U2-Net and use the improved U2-Net as the backbone of the segmentation network to alleviate the overfitting problem arising from MRI knee images during segmentation, and solve the problem of bone tissue segmentation errors caused by different parameters and different MRI imaging modes. A deep-supervised approach is also used to solve the target shift problem arising from existing networks in predicting fine cartilage regions. The remainder of this paper is organized as follows. Section 2 presents the relevant dataset. Section 3 describes the two-stage segmentation network and related work in this paper. Section 4 gives an experimental comparison of the current single-stage segmentation network with the two-stage network of this paper. Finally, the conclusions of this study are summarized in Section 5. 2 DATASET The data for this study are divided into two main parts, the first of which is from the Osteoarthritis Initiative (OAI), a longitudinal study examining the progression of osteoarthritis [26]. The relevant scan parameters are shown in Table 1, and as many of the images in this dataset were blurred and unusable, we selected 240 of them that were usable. The second part of the data was scanned by us using the uPMR790 and Signa HDXT devices using different parameters on the knee joints of 20 people, respectively, a total of 1239 images with the scanning parameters shown in Table 1. We set different scan parameters such as layer Size, thickness, FOV (field of view), Number and imaging modality. We used 1000 of the 1479 images as the training set and the other 479 as the validation set, and we tested with 504 MRI knee images that were different from these 1479 images. TABLE 1. Parameters associated with MRI knee bone tissue data from different devices using different modalities of imaging Dataset Layer thickness Size FOV Number Imaging methods DESS 0.7 mm 384 × 384 14 cm 240 T1 uPMR790-1 0.5–0.7 mm 384 × 384 14∼20 cm 303 T1 uPMR790-2 0.5–0.7 mm 384 × 384 14∼20 cm 311 PD uPMR790-3 0.5–0.7 mm 384 × 384 14∼20 cm 321 T2 Signa HDXT-1 0.5–0.7 mm 384 × 384 14∼20 cm 101 T1 Signa HDXT-2 0.5–0.7 mm 384 × 384 14∼20 cm 96 PD Signa HDXT-3 0.5–0.7 mm 384 × 384 14∼20 cm 107 T2 3 FRAMEWORK Introduction to the framework MRI knee images are characterized by similar internal tissue structure and large tissue structure differences between images. The cartilaginous tissue of the knee is a small, elongated target with a small proportion of the figure (uneven categories) and the patellar cartilage is adherent to the femoral cartilage in many MRI images, making it difficult to distinguish them with the current one-stage segmentation network. In this paper, we propose a two-stage semantic segmentation network based on convolutional neural networks. We segment the MRI knee images into six categories, among which three are cartilage tissues, and the cartilage tissues of the knee joint occupy a relatively small portion of the whole MRI images. If we use a deep neural network to downsample them several times, the following problems will occur: (1) The semantic information of the cartilage in the high-level semantic information will be lost in the deep neural network. (2) During the upsampling process, the position of the cartilage tissues will appear to drift. (3) After multiple convolutions, the patellar cartilage and femoral cartilage will fuse together in the high-level feature map, and the patellar cartilage and femoral cartilage cannot be accurately segmented. (4) If the number of convolution layers of the neural network is too small, the key feature information of the MRI knee image, such as semantic information, detailed texture information etc., cannot be learned. Multiple downsampling leads to a decreasing scale of the feature map and some small-scale important information of the original map (e.g., cartilage tissue) is lost. For example, in a 224 × 224 image, only 30 × 30 pixels are cartilage regions. If the 30 × 30 regions are deeply downsampled, they will eventually become 1 × 1 pixel points, and it is difficult to upsample a pixel point to the previous texture as well as semantic information, resulting in the loss of semantic information. Also, when a 1 × 1 pixel point is upsampled multiple times to a size of 30 × 30, the 30 × 30 area after upsampling may drift by 60 pixels or more if the 1 × 1 pixel point is shifted by two pixels. If there is both femoral and patellar cartilage in the 30 × 30 area, downsampling to a 1 × 1 size and upsampling to a 30 × 30 size, we cannot restore which pixels are femoral cartilage and which are patellar cartilage. Therefore we cannot simply increase or decrease the width and depth of the neural network. Simply Encoder-Decoder with jump connectivity will not achieve our desired results, but rather we need to improve the overall model architecture of the deep neural network. To address these problems posed by deep neural networks, we designed a two-stage region based segmentation network, segmenting the patella and corresponding cartilage using one segmentation network alone and the rest of the bone tissue together using another segmentation network. As shown in Figure 3, the neural network consists of two parts, the first part is the S module (significance detection branch), which is used for significance detection of bone tissue and the corresponding cartilage tissue and background to obtain the approximate locations of different bone tissues. After determining the approximate location of the different bone tissues, the corresponding areas of bone tissue are cut out and fed into the second section for segmentation. The second part is the P module (Patellar branch P) and the FT module (femur and tibia regions branch FT), which are used to segment the different regions of bone tissue and the corresponding cartilage tissue from the first part. After the segmentation is complete, the individual Patches are stitched together to form the complete segmentation. A two-stage region-based segmentation is used, with the P module focusing on the patella and corresponding cartilage and the FT on the patella and corresponding cartilage and the FT module on the femur and tibia and corresponding cartilage tissue. Segmenting different regions of bone tissue using different segmentation networks not only provides a good solution to the problem of missegmentation of patellar and femoral cartilage due to adhesions, but also makes the task of segmenting each segmentation network easier and solves the problem of missegmentation of the femur and tibia due to similar tissue structure. FIGURE 3Open in figure viewerPowerPoint Two-stage region-based segmentation network for MRI knee bone tissue: S Module for significant detection of bone tissue and corresponding cartilage tissue and background to obtain the approximate position of different bone tissues; P Module for segmentation of patella and corresponding cartilage; FT Module for segmentation of femur and tibia and corresponding cartilage tissue The structural significance of the Module is as follows. The merit of the S Module can be used to obtain the approximate areas of various types of bone tissue. With the P Module, the patellar region can be segmented separately to solve the problem of patellar cartilage and femoral cartilage sticking to each other. The FT Module can reduce the complexity of the segmentation process and use the existing position information (The input of the FT Module is the upper part of the femur and the lower part of the tibia) to solve the problem of mixing the tibia and femur during segmentation. S module Based on the inherent characteristics of the MRI knee bone tissue location, we segmented the patella and its corresponding cartilage separately, while the remaining four bone tissues (femur and tibia and their corresponding cartilage tissues) were segmented in a separate segmentation network. Before segmentation we need to detect the target regions of the different bone tissues, but a direct use of target detection algorithms (YOLO series, Fast-RCNN series etc.) would require relabeling the anchor for the different bone tissues, which is a time-consuming and labor-intensive task, and this idea is abandoned in this paper. Since we have already labelled the segmentation labels, there is no need to re-label the anchor for target detection. Using the pre-annotated segmentation labels, we designed a saliency target detection module, S Module, to identify the femur, tibia and patella regions using the saliency detection results. The S Module is a streamlined coding and decoding structure for saliency detection, which is a simple task that can be achieved without the use of a complex neural network structure. As shown in Figure 5, the specific design of S Module adopts the basic structure of U2-Net, the basic network structure of each layer of module is a V-shaped structure, the basic structure is shown as En_1 in Figure 5. we define this V-shaped structure as V-Module, Throughout the S-Module, we use maxpooling to downsample the images and the V-Module to extract features from the images. We change the image from 3 channels to 32 channels in the first layer of the V-Module, and in each subsequent layer of the downsampled V-Module, the input and output channels are 32 channels. When the image is downsampled to 1/16, we use bilinear interpolation to upsample it until it is upsampled to its original size. For the upsampling process, we use a jump connectivity with 64 input channels and 32 output channels in each of the upsampled V-Modules. we crop the MRI images centrally, with a size of 224 × 224, as input to the S Module. After the S Module to obtain the significance test results, the different regions of bone tissue and corresponding cartilage tissue were then cropped out for the second stage of segmentation based on the significance test results. After the S Module, the significance detection results are obtained, which in turn identify the different bone tissue regions in preparation for the second stage of the segmentation process. As shown in Figure 4, the cropping process is as follows: the edge coordinates of the femur and tibia regions are calculated from the saliency target detection results, as shown in Figure 4, that is, the leftmost (x,y) and rightmost (x',y') coordinates of the saliency target detection results, and the edge coordinates are used to calculate the starting point of the FT Module's input image The image size can be set by the user. For the P Module input, we can use the leftmost coordinates (x,y) of the detection result to determine the patellar region. FIGURE 4Open in figure viewerPowerPoint MRI knee bone tissue images in a two-stage segmentation network after different modules of the result map P module The P Module is used to segment the patella and the corresponding cartilage region from the S Module output. Compared to the simultaneous processing of seven different types of MRI bone tissue, the P Module only processes the patella and the corresponding cartilage region, making the segmentation task simpler. By segmenting the patellar region separately, the problem of adhesions between patellar and femoral cartilage is also solved. We conducted several experiments using U-Net, UNet++, Attention U-Net (Att U-Net) [37] and U2-Net as the backbone of P Module to compare the experimental results of different neural networks, and U2-Net using the V-shaped structure performed better in the segmentation of the patellar region. We further improved the U2-Net by adjusting the number of channels in each layer of the module to prevent the complexity of the whole neural network from becoming too large. The improved U2-Net is shown in Figure 6. We use a form of deep supervision to address the problem of drift in predicting cartilaginous tissue by calculating the cross-entropy loss for each layer of the feature map with ground truth during upsampling. The P Module has only three target categories, which is relatively easy for the learning task of the neural network, and is prone to overfitting if the neural network is too large. The input data of the P Module is a cropped patella and the corresponding cartilage region, which is narrow and has a small percentage of target categories. In order to prevent the target categories from disappearing in deep convolution, we have reduced the depth when designing the neural network. The segmentation task of P Module is simple, and it is easy to generate redundant feature layers in the inference process of the whole neural network, which will not only affect the inference speed of the neural network, but also may bring some overfitting problems. In the whole P Module, we set the output channels of each layer of the V-structure to 64 channels, so that the different features of the MRI knee can be fully learned and the complexity of the whole P Module is not too large. FT module The FT Module is used to segment the femur, tibia and corresponding cartilage areas from the S Module output. there are five segmentation targets for the FT Module, namely the femur and corresponding cartilage, the tibia and corresponding cartilage and the background. In order to de