Hierarchical conditional random field model for multi‐object segmentation in gastric histopathology images

Electronics LettersVolume 56, Issue 15 p. 750-753 Biomedical technologyFree Access Hierarchical conditional random field model for multi-object segmentation in gastric histopathology images Changhao Sun, Changhao Sun orcid.org/0000-0001-9087-0349 MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorChen Li, Corresponding Author Chen Li lichen201096@hotmail.com MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorJinghua Zhang, Jinghua Zhang orcid.org/0000-0003-3303-4752 MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorFrank Kulwa, Frank Kulwa MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorXiaoyan Li, Xiaoyan Li Liaoning Cancer Hospital and Institute, Shenyang, People's Republic of ChinaSearch for more papers by this author Changhao Sun, Changhao Sun orcid.org/0000-0001-9087-0349 MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorChen Li, Corresponding Author Chen Li lichen201096@hotmail.com MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorJinghua Zhang, Jinghua Zhang orcid.org/0000-0003-3303-4752 MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorFrank Kulwa, Frank Kulwa MBIE College, Northeastern University, Shenyang, People's Republic of ChinaSearch for more papers by this authorXiaoyan Li, Xiaoyan Li Liaoning Cancer Hospital and Institute, Shenyang, People's Republic of ChinaSearch for more papers by this author First published: 01 July 2020 https://doi.org/10.1049/el.2020.0729Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this Letter, a hierarchical conditional random field (HCRF) model-based gastric histopathology image segmentation (GHIS) method is proposed, which can localise abnormal (cancer) regions in gastric histopathology images to assist histopathologists in medical work. First, to obtain pixel-level segmentation information, the authors retrain a convolutional neural network (CNN) to build up their pixel-level potentials. Then, to obtain abundant spatial segmentation information in patch level, they fine tune another three CNNs to build up their patch-level potentials. Thirdly, based on the pixel- and patch-level potentials, their HCRF model is structured. Finally, a graph-based post-processing is applied to further improve their segmentation performance. In the experiment, a segmentation accuracy of is achieved on a haematoxylin and eosin stained gastric histopathological dataset with 560 images, showing the effectiveness and future potential of the proposed GHIS method. Introduction Gastric cancer is one of the five most common types of malignant tumours among men and women according to the World Health Organisation. It represents of all cases and of all deaths among cancer patients. Almost of new cases occur in Asia, and occur in China [1]. The patients almost have 12 months of disease-specific survival and of all cases dying within the first five years, meantime, it is one of the most aggressive and deadliest cancer types. So, it is very important for medical professionals to accurately estimate patient prognoses [2]. In recent years, image analysis plays a more and more important role in the intelligent diagnosis of gastric cancer, where a variety of deep learning methods, especially convolutional neural networks (CNNs), are developed and applied to gastric histopathology image segmentation (GHIS) tasks. However, the existing approaches usually focus on individual characteristics and network framework, such as layer numbers and network structures, without a strategy to describe the integral information. Therefore, some advanced methods are proposed to integrate these individual existing methods to obtain an even better performance. Especially, because of conditional random fields (CRFs) [3] can characterise the spatial relationship of images, they are effective and robust methods for analysing the contents of complex images. Whilst, medical doctors usually are too busy to draw all abnormal (cancer) regions, an effective multi-object segmentation approach is expected. Hence, we propose a hierarchical CRF (HCRF) model to address the GHIS problem, where the gastric histopathological images are mixed with the complicated nucleus, interstitial, and tissue fluid. Furthermore, because cellular visual features in a histopathological image can be observed on patch scales, when training the classifier, it is better to use the patches cut from the image than the whole image. The workflow of the proposed HCRF model is shown in Fig. 1. Fig 1Open in figure viewerPowerPoint Workflow of proposed HCRF model for GHIS There are three main contributions of our work: first, we heuristically apply a CRF framework to a new research field of the GHIS. Second, we propose a novel HCRF model using higher-order potentials in this Letter. Thirdly, we obtain high segmentation performance by the HCRF model for gastric histopathology images. Image dataset In this Letter, we use a public haematoxylin and eosin (HE) stained gastric histopathological image dataset [4] to test the effectiveness of our proposed novel HCRF model in the GHIS task. Image description: practical histopathologists mark most of the abnormal regions in the images; HE staining; 20 magnification; pixels image size; '*.tiff' or '*.png' image format. Image type: abnormal images, positive (cancer) regions appear in the sections; ground truth images (GT images), the positive regions are labelled in the sections. Image number: 560 abnormal images and 560 GT images. Furthermore, for the GHIS task, these 560 images are randomly divided into training, validation, and test sets with a ratio of 1:1:2 (140, 140 and 280 images). In the pixel-level potentials, we augment the training and validation datasets six times. Furthermore, because the size of the gastric histopathology images is too large to process, we crop the original and the GT images into pixels. So, we obtain 53,760 training, 53,760 validation, and 107,520 test images. In the patch-level potential, we mesh the original and GT images into patches ( pixels) and the area of a patch is 4096 pixels. When the sum of the pixel numbers in a GT image is over 2048, we give a positive label (1, foreground) to the corresponding original image patch; otherwise, we give it a negative label (0, background). However, when we do this operation, the positive image patches are much less than the negative image patches. So, to balance the data during training, we augment the positive image patches in the training and validation set. Hence, for both positive and negative patches, they both have 121,251 training and 119,151 validation images, respectively. Proposed HCRF model Most of the CRF models have so far used only unary and binary potentials [5, 6]. However, potentials defined on higher-order cliques have been shown to be useful in previous work, such as [7, 8]. Since we focus on the visual features of cellular scales in histopathological images [9], we introduce two types of higher-order potentials, one is a patch-unary potential to represent the information of cells and the other is a patch-binary potential to describe the surrounding spatial relationship among different cells. Hence, based on the basic definition of CRFs, our HCRF is expressed by (1) (1)where (2)is the normalisation factor; V is the set of all nodes in graph G, corresponding to the image pixels; E is the set of all edges in graph G. is one patch divided from an image; represents the surrounding patches of a single patch. The usual clique potential function consists of two parts (terms): the pixel-unary potential function is used to measure the probability that a pixel node i is labelled as , which takes values from a given set of classes , for a given observation vector [10]; the pixel-binary potential function is used to describe the adjacent nodes i and j in the graph. The spatial context relationship between them is related not only to the label of node i but also to the label of its neighbour node j. Furthermore, and are the newly introduced higher-order potentials. The patch-unary potential function is used to measure the probability that a patch node m is labelled as for a given observation vector ; the patch-binary potential function is used to describe the adjacent nodes m and n in the patch. , , and are the weights of the four potentials, , , and , respectively. and are the weights of the and , respectively. These weights are used to find the largest posterior label and to further improve the image segmentation performance. To give a visualised understanding, the structure of our HCRF model is shown in Fig. 2. Fig 2Open in figure viewerPowerPoint Overview of HCRF framework for analysing HE stained gastric histopathological image a Structure of proposed novel HCRF model. Left part denotes terms of pixel-level potentials, and right part shows terms of patch-level potentials b 48 neighbourhood 'lattice' layout of pixel-binary potential. Average of unary probabilities of 48 neighbourhood pixels is used as probability of pixel (central pixel in green) c Eight neighbourhood 'lattice' layout of patch-binary potential. Average of unary probabilities of eight neighbourhood patches is used as probability of target patch (central patch in green) In Fig. 2, the structure of our novel HCRF model is as follows: layer I shows the real labels of pixels and real labels of patches in an image. Layer II represents the original image pixels or patches , which correspond with labels or . Considering the effectiveness of U-Net in the medical image segmentation task, we retrain an U-Net to get pixel-level feature maps. From the pixel-level side, layer III represents the U-Net [11], which is applied in pixel-level potentials using images pixels . To extract abundant spatial information, we fine-tune different structure networks to obtain patch-level feature maps. From the patch-level side, layer III denotes three types of CNNs that are applied in patch-level potentials using image patches , including VGG-16 [12], Inception-V3 [13], and ResNet-50 [14] networks. In layer IV, we get one type of pixel-unary potential, corresponding to U-Net; and three types of patch-unary potentials, corresponding to VGG-16, Inception-V3 and ResNet-50 networks. To obtain the pixel-level binary potentials, we calculate the potentials of their surrounding image pixels according to the layout in Fig. 2b. Similarly, to obtain the patch-level binary potentials, we calculate the potentials of their surrounding image patches according to the 'lattice' layout in Fig. 2c. Especially, there is an additional layer in the patch-level side, namely the layer IV+, where we give weights to optimise the patch-unary and -binary potentials. In layer V, we give weights to the obtained pixel-unary, pixel-binary, patch-unary, and patch-binary potentials, respectively; and calculate the joint probability of these four potentials to structure the final HCRF model. In layer VI, to further improve the segmentation result from layer V, we use Markov random field (MRF) [15] and morphological operations [16] as post-processing in our work. Evaluation of proposed HCRF model To give a quantitative evaluation, we use seven criteria which are Sórensen–Dice coefficient (Dice), relative volume difference (RVD), intersection over union (IoU), precision, recall, specificity, and accuracy to measure the medical segmentation result. Based on the pixel- and patch-level image segmentation results, the final result by the HCRF model is obtained, where we use the pixel-unary, pixel-binary, patch-unary, and patch-binary segmentation accuracies (, , , and ) as the weights to optimise the HCRF model. An example of the HCRF segmentation results on the validation dataset is shown in Fig. 3a and the numerical evaluation is shown in Fig. 3c. We only use one pinkish-orange colour to represent the positive regions and another black colour to represent the negative regions. From Fig. 3b, we can find that our HCRF model obtains good image segmentation results on the test set, where most of the positive (cancer) regions are segmented, and the edges of the regions are smooth. The comparison in Fig. 3c denotes that although the test set has 280 images which are twice than the validation set, our proposed HCRF model also obtains good segmentation performance, where the values of all seven evaluation indexes on the test set are closed to that on the validation set, showing high stability and strong robustness of our method. Fig 3Open in figure viewerPowerPoint Image segmentation results on validation and test sets a (a)–(f) original gastric histopathology images, corresponding GT images and HCRF segmentation results on validation set b (a)–(f) original gastric histopathology images, corresponding GT images and HCRF segmentation results on test set c Comparison between image segmentation performance of proposed HCRF model on validation and test sets (in %) Comparison to existing methods To show the potential of the proposed HCRF method for the GHIS task, we compare it with seven existing methods, including three state-of-the-art methods (DenseCRF [6], SegNet [17], and U-Net) and four classical methods (Otsu thresholding [18], Watershed [19], k-means clustering [20], and MRF). A comparison of the image segmentation results on test dataset is shown in Fig. 4 and Table 1. We can find that our HCRF method has an overall better image segmentation performance than other existing methods. Fig 4Open in figure viewerPowerPoint Comparison of image segmentation results of our HCRF and other existing methods on test seta, b two typical examples of image segmentation results using different methods Table 1. Numerical comparison between different methods (in ) Criterion HCRF DenseCRF U-Net SegNet Otsu Watershed k-means MRF Dice 46.29 45.78 45.57 20.08 25.34 26.13 25.34 23.96 IoU 32.59 32.12 31.91 13.00 15.05 15.85 15.06 14.32 precision 41.58 40.47 40.04 38.85 21.59 29.30 21.65 18.39 recall 65.59 68.89 68.96 31.71 42.77 35.41 42.84 49.91 specificity 81.33 78.12 77.95 84.12 70.82 79.42 70.78 53.36 RVD 141.35 164.87 167.36 206.60 288.59 194.34 289.53 458.78 accuracy 78.91 77.02 76.84 75.31 65.98 72.05 65.93 54.41 Computational time In our experiment, we use a workstation with Intel® Core i7-8700k CPU 3.20 GHz, 32 GB RAM and GeForce RTX 2080 8 GB. The testing time of our HCRF model on 280 images ( pixels) within two working stages. The first stage is the 'main body' of the HCRF model, which is the stage from layer I to layer V in Fig. 2. The second stage is the 'post-processing' stage, which denotes layer VI in Fig. 2. The mean testing time for one ( pixels) image of the whole HCRF model is 61.6 s, where the 'main body' stage takes 28.1 s and the 'post-processing' stage takes 33.5 s. Conclusions In this Letter, we propose a novel HCRF model to the gastric histopathology image segmentation task. This HCRF model not only uses traditional unary and binary potentials but also applies higher-order potentials to improve the segmentation quality. In pixel-level potentials, we retrain the U-Net; in patch-level potentials, we fine-tune the VGG-16, Inception-V3, and ResNet-50 networks. Furthermore, when joining the pixel-level and patch-level potentials, we use different weights for different potentials to optimise the model. In the experiment, the proposed HCRF method is finally tested on a gastric HE histopathological test set and obtains a 78.91% segmentation accuracy, which is nearly close to the 79.83% segmentation accuracy on the validation set, showing the robustness and potential of our method. Acknowledgment This study was supported by the National Natural Science Foundation of China (grant no. 61806047). References 1Bernard, W.S., Christopher, P.W.: ' World cancer report 2014' ( International Agency for Research on Cancer (IARC), World Health Organization, France, 2014) 2Garcia, E., Hermoza, R., Castanon, C.B. et. al.,: ' Automatic lymphocyte detection on gastric cancer IHC images using deep learning'. Computer-Based Medical Systems (CBMS), Thessaloniki, Greece, 2017, pp. 200– 204 3Lafferty, J., McCallum, A., Pereira, F.C.: ' Conditional random fields: probabilistic models for segmenting and labeling sequence data'. Int. Conf. on Machine Learning (ICML), Williamstown, MA, USA, 2001, pp. 282– 289 4Zhang, Z., Gao, J., Lv, G. et. al.,: 'Pathological image classification of gastric cancer based on depth learning', Comput. Sci., 2018, 45, (S2), pp. 273– 278 5Zheng, S., Jayasumana, S., Romera-Paredes, B. et. al.,: ' Conditional random fields as recurrent neural networks'. Int. Conf. on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 1– 17 6Chen, L.-C., Papandreou, G., Kokkinos, I. et. al.,: 'Deeplab: semantic image segmentation with Deep convolutional nets, atrous convolution, and fully connected CRFs', IEEE Trans. Pattern Anal. Mach. Intell., 2017, 40, (4), pp. 834– 848 (https://doi/org/10.1109/TPAMI.2017.2699184) 7Vineet, V., Warrell, J., Torr, P.H.: 'Filter-based mean-field inference for random fields with higher-order terms and product label-spaces', Int. J. Comput. Vis., 2014, 110, (3), pp. 290– 307 (https://doi/org/10.1007/s11263-014-0708-6) 8Arnab, A., Jayasumana, S., Zheng, S. et. al.,: ' Higher order conditional random fields in deep neural networks'. European Conf. on Computer Vision (ECCV), Amsterdam, the Netherlands, 2016, pp. 524– 540 9Hou, L., Samaras, D., Kurc, T.M. et. al.,: ' Patch-based convolutional neural network for whole slide tissue image classification'. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 2424– 2433 10Kosov, S., Shirahama, K., Li, C. et. al.,: 'Environmental microorganism classification using conditional random fields and deep convolutional neural networks', Pattern Recognit., 2018, 77, pp. 248– 261 (https://doi/org/10.1016/j.patcog.2017.12.021) 11Ronneberger, O., Fischer, P., Brox, T.: ' U-net: convolutional networks for biomedical image segmentation'. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 2015, pp. 234– 241 12Simonyan, K., Zisserman, A.: 'Very deep convolutional networks for large-scale image recognition', arXiv preprint arXiv:1409.1556, 2014 13Szegedy, C., Vanhoucke, V., Ioffe, S. et. al.,: ' Rethinking the inception architecture for computer vision'. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 2818– 2826 14He, K., Zhang, X., Ren, S. et. al.,: ' Deep residual learning for image recognition'. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770– 778 15Li, S.Z.: ' Markov random field models in computer vision'. European Conf. on Computer Vision (ECCV), Stockholm, Sweden, 1994, pp. 361– 370 16Gonzalez, R.C., Woods, R.E., Eddins, S.L.: ' Digital image processing using matlab' ( Pearson/Prentice Hall, USA, 2004) 17Badrinarayanan, V., Kendall, A., Cipolla, R.: 'Segnet: a deep convolutional encoder-decoder architecture for image segmentation', IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (12), pp. 2481– 2495 (https://doi/org/10.1109/TPAMI.2016.2644615) 18Otsu, N.: 'A threshold selection method from gray-level histograms', IEEE Trans. Syst. Man Cybern., 1979, 9, (1), pp. 62– 66 (https://doi/org/10.1109/TSMC.1979.4310076) 19Vincent, L., Soille, P.: 'Watersheds in digital spaces: an efficient algorithm based on immersion simulations', IEEE Trans. Pattern Anal. Mach. Intell., 1991, 13, (6), pp. 583– 598 (https://doi/org/10.1109/34.87344) 20Hartigan, J.A., Wong, M.A.: 'Algorithm as 136: a K-means clustering algorithm', J. R. Stat. Soc. C, Appl. Stat., 1979, 28, (1), pp. 100– 108 Citing Literature Volume56, Issue15July 2020Pages 750-753 FiguresReferencesRelatedInformation