A Deep Learning-Based Approach for Glomeruli Instance Segmentation from Multistained Renal Biopsy Pathologic Images

Glomeruli instance segmentation from pathologic images is a fundamental step in the automatic analysis of renal biopsies. Glomerular histologic manifestations vary widely among diseases and cases, and several special staining methods are necessary for pathologic diagnosis. A robust model is needed to segment and classify glomeruli with different staining methods and apply in cases with various glomerular pathologic changes. Herein, pathologic images from renal biopsy slides stained with three basic special staining methods were used to build the data sets. The snapshot group included 1970 glomeruli from 516 patients, and the whole-slide image group included 8665 glomeruli from 148 patients. Cascade Mask region-based convolutional neural net architecture was trained to detect, classify, and segment glomeruli into three categories: i) GN, structural normal; ii) global sclerosis; and iii) glomerular with other lesions. In the snapshot group, total glomeruli, GN, global sclerosis, and glomerular with other lesions achieved an F1 score of 0.914, 0.896, 0.681, and 0.756, respectively, which were comparable with those in the whole-slide image group (0.940, 0.839, 0.806, and 0.753, respectively). Among the three categories, GN achieved the best instance segmentation effect in both groups, as determined by average precision, average recall, F1 score, and Mask mean Intersection over Union. The present model segments and classifies multistained glomeruli with efficiency and robustness. It can be applied as the first step for more detailed glomerular histologic analysis. Glomeruli instance segmentation from pathologic images is a fundamental step in the automatic analysis of renal biopsies. Glomerular histologic manifestations vary widely among diseases and cases, and several special staining methods are necessary for pathologic diagnosis. A robust model is needed to segment and classify glomeruli with different staining methods and apply in cases with various glomerular pathologic changes. Herein, pathologic images from renal biopsy slides stained with three basic special staining methods were used to build the data sets. The snapshot group included 1970 glomeruli from 516 patients, and the whole-slide image group included 8665 glomeruli from 148 patients. Cascade Mask region-based convolutional neural net architecture was trained to detect, classify, and segment glomeruli into three categories: i) GN, structural normal; ii) global sclerosis; and iii) glomerular with other lesions. In the snapshot group, total glomeruli, GN, global sclerosis, and glomerular with other lesions achieved an F1 score of 0.914, 0.896, 0.681, and 0.756, respectively, which were comparable with those in the whole-slide image group (0.940, 0.839, 0.806, and 0.753, respectively). Among the three categories, GN achieved the best instance segmentation effect in both groups, as determined by average precision, average recall, F1 score, and Mask mean Intersection over Union. The present model segments and classifies multistained glomeruli with efficiency and robustness. It can be applied as the first step for more detailed glomerular histologic analysis. Histologic evaluation of glomeruli from renal biopsy tissues plays a key role in the diagnosis of various kidney diseases, and important in making treatment strategies and predicting prognosis.1Puelles V.G. Bertram J.F. Counting glomeruli and podocytes: rationale and methodologies.Curr Opin Nephrol Hypertens. 2015; 24: 224-230PubMed Google Scholar,2Lees G.E. Cianciolo R.E. Clubb F.J. Renal biopsy and pathologic evaluation of glomerular disease.Top Companion Anim Med. 2011; 26: 143-153Crossref PubMed Scopus (19) Google Scholar Currently, histopathologic assessment of renal biopsies depends on manual judgment by pathologists. It is time-consuming and has unsatisfactory reproducibility, especially in situations involving grading or scoring the histologic features, in spite of efforts to improve the definitions, criteria, and classification methods.3Bajema I.M. Wilhelmus S. Alpers C.E. Bruijn J.A. Colvin R.B. Cook H.T. D'Agati V.D. Ferrario F. Haas M. Jennette J.C. Joh K. Nast C.C. Noel L.H. Rijnink E.C. Roberts I.S.D. Seshan S.V. Sethi S. Fogo A.B. Revision of the International Society of Nephrology/Renal Pathology Society classification for lupus nephritis: clarification of definitions, and modified National Institutes of Health activity and chronicity indices.Kidney Int. 2018; 93: 789-796Abstract Full Text Full Text PDF PubMed Scopus (177) Google Scholar,4Trimarchi H. Barratt J. Cattran D.C. Cook H.T. Coppo R. Haas M. Liu Z.H. Roberts I.S. Yuzawa Y. Zhang H. Feehally J. IgAN Classification Working Group of the International IgA Nephropathy Network and the Renal Pathology SocietyOxford classification of IgA nephropathy 2016: an update from the IgA Nephropathy Classification Working Group.Kidney Int. 2017; 91: 1014-1021Abstract Full Text Full Text PDF PubMed Scopus (368) Google Scholar In recent years, deep-learning methods, especially convolutional neural nets (CNNs), have been applied often in medical image analysis and have achieved good results.5Gadermayr M. Dombrowski A.-K. Klinkhammer B.M. Boor P. Merhof D. CNN cascades for segmenting sparse objects in gigapixel whole slide images.Comput Med Imaging Graphics. 2019; 71: 40-48Crossref PubMed Scopus (23) Google Scholar, 6Esteva A. Robicquet A. Ramsundar B. Kuleshov V. DePristo M. Chou K. Cui C. Corrado G. Thrun S. Dean J. A guide to deep learning in healthcare.Nat Med. 2019; 25: 24-29Crossref PubMed Scopus (695) Google Scholar, 7Wang S. Yang D.M. Rong R. Zhan X. Xiao G. Pathology image analysis using segmentation deep learning algorithms.Am J Pathol. 2019; 189: 1686-1698Abstract Full Text Full Text PDF PubMed Scopus (76) Google Scholar, 8Komura D. Ishikawa S. Machine learning approaches for pathologic diagnosis.Virchows Arch. 2019; 475: 131-138Crossref PubMed Scopus (31) Google Scholar Renal biopsy pathology, as an important branch of pathology, has unique complexity in image analysis and urgently needs this new technology to improve the efficiency and accuracy of diagnosis. Glomeruli extraction and recognition from pathologic images is a fundamental step in the automatic analysis of renal biopsies. Most of the recent work can generally be classified into two categories, object detection9Maree R. Dallongeville S. Olivo-Marin J.C. Meas-Yedid V. An approach for detection of glomeruli in multisite digital pathology. IEEE International Symposium on Biomedical Imaging.2016: 1033-1036Google Scholar, 10Temerinac-Ott M. Forestier G. Schmitz J. Hermsen M. Brasen J.H. Feuerhake F. Wemmert C. Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.2017: 19-24Google Scholar, 11Simon O. Yacoub R. Jain S. Tomaszewski J.E. Sarder P. Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images.Sci Rep. 2018; 8: 2032Crossref PubMed Scopus (40) Google Scholar, 12Bukowy J.D. Dayton A. Cloutier D. Manis A.D. Staruschenko A. Lombard J.H. Solberg Woods L.C. Beard D.A. Cowley Jr., A.W. Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.J Am Soc Nephrol. 2018; 29: 2081-2088Crossref PubMed Scopus (41) Google Scholar and semantic segmentation13Kato T. Relator R. Ngouv H. Hirohashi Y. Takaki O. Kakimoto T. Okada K. Segmental H.O.G. new descriptor for glomerulus detection in kidney microscopy image.BMC Bioinformatics. 2015; 16: 1-16Crossref PubMed Scopus (47) Google Scholar, 14Sarder P. Ginley B. Tomaszewski J.E. Automated renal histopathology: digital extraction and quantification of renal pathology. SPIE Medical Imaging.2016Google Scholar, 15Sheehan S. Mawe S. Cianciolo R.E. Korstanje R. Mahoney J.M. Detection and classification of novel renal histologic phenotypes using deep neural networks.Am J Pathol. 2019; 189: 1786-1796Abstract Full Text Full Text PDF PubMed Scopus (8) Google Scholar, 16Ginley B. Lutnick B. Jen K.-Y. Fogo A.B. Jain S. Rosenberg A. Walavalkar V. Wilding G. Tomaszewski J.E. Yacoub R. Rossi G.M. Sarder P. Computational segmentation and classification of diabetic glomerulosclerosis.J Am Soc Nephrol. 2019; 30: 1953-1967Crossref PubMed Scopus (55) Google Scholar, 17Hermsen M. de Bel T. den Boer M. Steenbergen E.J. Kers J. Florquin S. Roelofs J.J.T.H. Stegall M.D. Alexander M.P. Smith B.H. Smeets B. Hilbrands L.B. van der Laak J.A.W.M. Deep learning–based histopathologic assessment of kidney tissue.J Am Soc Nephrol. 2019; 30: 1968-1979Crossref PubMed Scopus (74) Google Scholar, 18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google Scholar, 19Zeng C. Nan Y. Xu F. Lei Q. Li F. Chen T. Liang S. Hou X. Lv B. Liang D. Luo W. Lv C. Li X. Xie G. Liu Z. Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.J Pathol. 2020; 252: 53-64Crossref PubMed Scopus (12) Google Scholar (Table 1). For object detection, Temerinac-Ott et al10Temerinac-Ott M. Forestier G. Schmitz J. Hermsen M. Brasen J.H. Feuerhake F. Wemmert C. Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.2017: 19-24Google Scholar used CNN to get classification results from differently stained sections. They used it to improve glomeruli detection on one staining and achieved 10% to 20% higher F1 scores than those with the HOG detector. Bukowy et al12Bukowy J.D. Dayton A. Cloutier D. Manis A.D. Staruschenko A. Lombard J.H. Solberg Woods L.C. Beard D.A. Cowley Jr., A.W. Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.J Am Soc Nephrol. 2018; 29: 2081-2088Crossref PubMed Scopus (41) Google Scholar used region-based convolutional neural net (R-CNN) and CNN for final classification as glomerulus or background objects, and achieved an average precision and recall of 96.94% and 96.79%, respectively. With the development of deep-learning methods, DeepLab-V2,17Hermsen M. de Bel T. den Boer M. Steenbergen E.J. Kers J. Florquin S. Roelofs J.J.T.H. Stegall M.D. Alexander M.P. Smith B.H. Smeets B. Hilbrands L.B. van der Laak J.A.W.M. Deep learning–based histopathologic assessment of kidney tissue.J Am Soc Nephrol. 2019; 30: 1968-1979Crossref PubMed Scopus (74) Google Scholar SegNet,18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google Scholar and U-Net18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google Scholar,19Zeng C. Nan Y. Xu F. Lei Q. Li F. Chen T. Liang S. Hou X. Lv B. Liang D. Luo W. Lv C. Li X. Xie G. Liu Z. Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.J Pathol. 2020; 252: 53-64Crossref PubMed Scopus (12) Google Scholar have been used to deal with segmentation tasks in semantic segmentation. Object detection tasks can detect and classify glomeruli, but the boundary of glomeruli is unknown. Although semantic segmentation can determine the boundary, the glomeruli number cannot be calculated and different glomeruli cannot be distinguished in the same image. Therefore, to overcome the above limitations, the current method attempts to achieve the goal of instance segmentation. The algorithm of the Cascade Mask R-CNN20Cai Z. Vasconcelos N. Cascade R.-C.N.N. High Quality Object Detection and Instance Segmentation.High Quality Object Detection and Instance Segmentation. 2021; 43: 1483-1498Google Scholar model was improved to better fit the data set so that sufficient image information can be acquired for further research.Table 1Previous Research on Glomeruli Detection and SegmentationSourceData/slideApproachPerformanceSource of renal tissue imagesStaining methodDetection Maree et al9Maree R. Dallongeville S. Olivo-Marin J.C. Meas-Yedid V. An approach for detection of glomeruli in multisite digital pathology. IEEE International Symposium on Biomedical Imaging.2016: 1033-1036Google ScholarTotal: 200Ellipsoidal shape + decision tree87% F1Human (renal biopsies)Masson trichrome Temerinac-Ott et al10Temerinac-Ott M. Forestier G. Schmitz J. Hermsen M. Brasen J.H. Feuerhake F. Wemmert C. Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.2017: 19-24Google ScholarTrain: 16Test: 4Mutual information + CNN66.38%–81.75% F11. Human (nephrectomy specimens)Jones H&E, PAS, Sirius Red, and immunostained for CD102. Human (needle biopsies from transplanted kidneys)Jones H&E, PAS, and Sirius Red Simon et al11Simon O. Yacoub R. Jain S. Tomaszewski J.E. Sarder P. Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images.Sci Rep. 2018; 8: 2032Crossref PubMed Scopus (40) Google ScholarTrain: 25Test: 9Multiradial LBP + SVM90.4% Precision,76.7% recall1. A standard streptozocin mouse modelH&E2. RatsH&E, PAS, Jones silver, Gömöri trichrome, and Congo Red3. Human (DN patients with chronic kidney disease stage II and stage III; control: normal kidneys)PAS Bukowy et al12Bukowy J.D. Dayton A. Cloutier D. Manis A.D. Staruschenko A. Lombard J.H. Solberg Woods L.C. Beard D.A. Cowley Jr., A.W. Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.J Am Soc Nephrol. 2018; 29: 2081-2088Crossref PubMed Scopus (41) Google ScholarTrain: 72Test: 13Faster R-CNN96.9% Precision,96.8% recallRats of various genetic backgroundsMasson trichromeSegmentation Kato et al13Kato T. Relator R. Ngouv H. Hirohashi Y. Takaki O. Kakimoto T. Okada K. Segmental H.O.G. new descriptor for glomerulus detection in kidney microscopy image.BMC Bioinformatics. 2015; 16: 1-16Crossref PubMed Scopus (47) Google ScholarTrain:∗Not mentioned.Test: 20S-HOG + SVM86.6% F1SD and SDT ratsImmunostained for desmin Sarder et al14Sarder P. Ginley B. Tomaszewski J.E. Automated renal histopathology: digital extraction and quantification of renal pathology. SPIE Medical Imaging.2016Google ScholarTrain:∗Not mentioned.Test: 15Gabor filters87.8% AccuracyNormal healthy untreated rats and micePAS and H&E Sheehan et al15Sheehan S. Mawe S. Cianciolo R.E. Korstanje R. Mahoney J.M. Detection and classification of novel renal histologic phenotypes using deep neural networks.Am J Pathol. 2019; 189: 1786-1796Abstract Full Text Full Text PDF PubMed Scopus (8) Google ScholarTotal: 90AlexNet + SVM92% Recall,90% specificityMice with different genotypesPAS Ginley et al16Ginley B. Lutnick B. Jen K.-Y. Fogo A.B. Jain S. Rosenberg A. Walavalkar V. Wilding G. Tomaszewski J.E. Yacoub R. Rossi G.M. Sarder P. Computational segmentation and classification of diabetic glomerulosclerosis.J Am Soc Nephrol. 2019; 30: 1953-1967Crossref PubMed Scopus (55) Google ScholarTrain: 41Test: 13DeepLab version 293% AccuracyHuman (patients with DN)PAS Hermsen et al17Hermsen M. de Bel T. den Boer M. Steenbergen E.J. Kers J. Florquin S. Roelofs J.J.T.H. Stegall M.D. Alexander M.P. Smith B.H. Smeets B. Hilbrands L.B. van der Laak J.A.W.M. Deep learning–based histopathologic assessment of kidney tissue.J Am Soc Nephrol. 2019; 30: 1968-1979Crossref PubMed Scopus (74) Google ScholarTrain: 37Test: 35 U-Nets ensemble79% Dice1. Human (transplant biopsies)PAS2. Human (nephrectomy specimens) Bueno et al18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google ScholarTrain: 38Test: 9SegNet-VGG1999.8% Precision,99.2% F1Human (AIDPATH kidney database)PAS Zeng et al19Zeng C. Nan Y. Xu F. Lei Q. Li F. Chen T. Liang S. Hou X. Lv B. Liang D. Luo W. Lv C. Li X. Xie G. Liu Z. Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.J Pathol. 2020; 252: 53-64Crossref PubMed Scopus (12) Google ScholarTest: 22Train: 360U-Net-SSIM + marked watershed93% Specificity,93.1% precision,94.0% F1,94.9% recall,90.1% DiceHuman (patients with IgA nephropathy)PASAIDPATH, Academia and Industry Collaboration for Digital Pathology; CNN, convolutional neural net; Dice, dice coefficient; DN, diabetes nephropathy; F1, F1 score; H&E, hematoxylin-eosin; LBP, local binary patterns; PAS, periodic acid–Schiff; R-CNN, region-based CNN; SD, sprague-dawley; SDT, spontaneously diabetic torii; SVM, support vector machine; S-HOG, segmental histogram of oriented gradient.∗ Not mentioned. Open table in a new tab AIDPATH, Academia and Industry Collaboration for Digital Pathology; CNN, convolutional neural net; Dice, dice coefficient; DN, diabetes nephropathy; F1, F1 score; H&E, hematoxylin-eosin; LBP, local binary patterns; PAS, periodic acid–Schiff; R-CNN, region-based CNN; SD, sprague-dawley; SDT, spontaneously diabetic torii; SVM, support vector machine; S-HOG, segmental histogram of oriented gradient. Glomerular lesions can involve any part of a glomerulus, including endothelial cells, mesangial cells, podocytes, and mesangial matrix.21Sethi S. Haas M. Markowitz G.S. D'Agati V.D. Rennke H.G. Jennette J.C. Bajema I.M. Alpers C.E. Chang A. Cornell L.D. Cosio F.G. Fogo A.B. Glassock R.J. Hariharan S. Kambham N. Lager D.J. Leung N. Mengel M. Nath K.A. Roberts I.S. Rovin B.H. Seshan S.V. Smith R.J.H. Walker P.D. Winearls C.G. Appel G.B. Alexander M.P. Cattran D.C. Casado C.A. Cook H.T. De Vriese A.S. Radhakrishnan J. Racusen L.C. Ronco P. Fervenza F.C. Mayo Clinic/Renal Pathology Society consensus report on pathologic classification, diagnosis, and reporting of GN.J Am Soc Nephrol. 2016; 27: 1278-1287Crossref PubMed Scopus (115) Google Scholar In addition, the severity of lesions varies widely. Hence, the appearance of different glomeruli varies greatly. An ideal model should be able to identify and segment all types of glomeruli. Most of the existing models analyzing human kidney tissues for glomeruli detection and segmentation are based on a single disease, such as diabetic nephropathy,16Ginley B. Lutnick B. Jen K.-Y. Fogo A.B. Jain S. Rosenberg A. Walavalkar V. Wilding G. Tomaszewski J.E. Yacoub R. Rossi G.M. Sarder P. Computational segmentation and classification of diabetic glomerulosclerosis.J Am Soc Nephrol. 2019; 30: 1953-1967Crossref PubMed Scopus (55) Google Scholar transplant biopsies,17Hermsen M. de Bel T. den Boer M. Steenbergen E.J. Kers J. Florquin S. Roelofs J.J.T.H. Stegall M.D. Alexander M.P. Smith B.H. Smeets B. Hilbrands L.B. van der Laak J.A.W.M. Deep learning–based histopathologic assessment of kidney tissue.J Am Soc Nephrol. 2019; 30: 1968-1979Crossref PubMed Scopus (74) Google Scholar and IgA nephropathy,19Zeng C. Nan Y. Xu F. Lei Q. Li F. Chen T. Liang S. Hou X. Lv B. Liang D. Luo W. Lv C. Li X. Xie G. Liu Z. Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.J Pathol. 2020; 252: 53-64Crossref PubMed Scopus (12) Google Scholar except in case of Bueno et al,18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google Scholar who used 47 whole-slide images (WSIs) from AIDPATH kidney database. In addition to hematoxylin-eosin staining, for light microscopy, a variety of histochemical stains, including periodic acid–Schiff (PAS), periodic acid–silver methenamine (PASM), and Masson trichrome stains are used to evaluate renal tissues. Each staining method has its unique value in elucidating specific histologic features and is essential for renal pathologic analysis. Both the variabilities of glomerular histologic manifestations and of staining methods increase the complexity of glomeruli detection and segmentation in pathologic images. Among the recent research analyzing human kidney tissues for glomeruli detection and segmentation, all results were obtained from PAS-stained sections.16Ginley B. Lutnick B. Jen K.-Y. Fogo A.B. Jain S. Rosenberg A. Walavalkar V. Wilding G. Tomaszewski J.E. Yacoub R. Rossi G.M. Sarder P. Computational segmentation and classification of diabetic glomerulosclerosis.J Am Soc Nephrol. 2019; 30: 1953-1967Crossref PubMed Scopus (55) Google Scholar, 17Hermsen M. de Bel T. den Boer M. Steenbergen E.J. Kers J. Florquin S. Roelofs J.J.T.H. Stegall M.D. Alexander M.P. Smith B.H. Smeets B. Hilbrands L.B. van der Laak J.A.W.M. Deep learning–based histopathologic assessment of kidney tissue.J Am Soc Nephrol. 2019; 30: 1968-1979Crossref PubMed Scopus (74) Google Scholar, 18Bueno G. Fernandez-Carrobles M.M. Gonzalez-Lopez L. Deniz O. Glomerulosclerosis identification in whole slide images using semantic segmentation.Comput Methods Programs Biomed. 2020; 184: 105273Crossref PubMed Scopus (25) Google Scholar, 19Zeng C. Nan Y. Xu F. Lei Q. Li F. Chen T. Liang S. Hou X. Lv B. Liang D. Luo W. Lv C. Li X. Xie G. Liu Z. Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.J Pathol. 2020; 252: 53-64Crossref PubMed Scopus (12) Google Scholar and there is no study that can segment the glomeruli from the images with multiple staining methods using a single model. In this research, an algorithmic model with the following features was constructed: i) it can recognize and segment glomeruli with various histologic appearances from different diseases, ii) it has good robustness so that it can be applied to different special stained slides, and iii) it can make a preliminary classification of the glomeruli. For the above purposes, we present a deep-learning–based approach for the object extraction and three-classification of glomeruli. Kidney biopsy samples from patients diagnosed with various kidney diseases from January 2018 to June 2019 in Department of Nephrology of Peking University People's Hospital (Beijing, China) were collected. All biopsy samples were processed according to standard techniques for light microscopy, immunofluorescence, and electron microscopy. For light microscopy, formalin-fixed, paraffin-embedded tissues were cut into sections (2 μm thick) and stained with one of the following techniques: hematoxylin-eosin, PAS, PASM, or Masson. The images from sections stained with PAS, PASM, and Masson were selected. The digital images used in this study were obtained from the two sources listed below. A total of 2 to 10 digital images were routinely captured in typical lesion areas from each case using Leica DFC550 digital camera (Singapore) attached to a Leica DM4000 B LED optical microscope (Weizla, Germany) and saved in the image library of our department. The average size of each image was approximately 1360 × 1024 × 3 pixels, corresponding to a field of 2.176 × 1.632 mm2, which resulted in a length scale of 0.85 mm/pixel. Images from patients diagnosed with various kidney diseases from January 2018 to April 2019, with high technical staining quality and containing at least one glomerular area, were selected. Slides were scanned at 40× using Precie 500B scanner (UNIC Technologies Inc., Beijing, China) for WSIs, with a 40× objective and numerical aperture = 0.8; the resolution was 0.12 μm/pixel for all acquired images. Images from patients diagnosed with various kidney diseases from May 2019 to June 2019, with high technical staining quality, were selected. In total, 1123 images from 516 patients, covering >30 pathologic types of kidney diseases, were collected (Tables 2 and 3). The diagnoses of the patients selected in this group are shown in Table 4. The total number of glomeruli was 1970 (PASM, 779; PAS, 492; and Masson, 699) (Table 5). The amplification multiples were 10× (65 glomeruli), 20× (1482 glomeruli), and 40× (423 glomeruli). Data are provided as number of images/number of patients (images/patient). WSI, whole-slide image. Masson, Masson trichrome stain; PAS, periodic acid–Schiff; PASM, periodic acid–silver methenamine; WSI, whole-slide image. ANCA, anti-neutrophil cytoplasmic antibody. GL, glomerulus with abnormal structure; GN, glomerulus with basically normal structure; GS, global sclerosis; WSI, whole-slide image. A total of 348 WSIs were obtained from 148 patients with >20 pathologic types of kidney diseases (Tables 2 and 3). The total number of glomeruli was 8665 (PASM, 3248; PAS, 2525; and Masson, 2892) (Table 5). The diagnoses of the patients selected in this group are shown in Table 6. ANCA, anti-neutrophil cytoplasmic antibody; WSI, whole-slide image. Because of the high resolution of WSIs and the limited computing resources, the entire WSIs cannot be fed into the model as input. So, the original WSIs were cropped into 2048 × 2048 pixel patches. In the cropping process, some glomeruli might be divided into two patches (incomplete cut). To avoid waste of data resources and to better use the data for getting a better model, the stride of the cutting window was three-fourths of the window's length. These images were then processed by script and checked manually to ensure that all irrelevant areas had been filtered out. The small patches were sent to the model, and the results were mapped back to WSIs, according to the cropped coordinates. Each image was labeled into one of the three categories:i)GN: The glomeruli with completely open capillary tufts were considered structural normal (GN). This category included normal glomeruli, glomeruli with mild lesions (such as mild mesangial hypercellularity or mild mesangial matrix expansion), and glomeruli with thickened glomerular basement membrane without other lesions.ii)GS: The glomeruli with complete obliteration of the entire glomerular tuft and with loss of their normal structure were considered as global sclerosis (GS).iii)GL: The glomeruli with lesions other than GS and leading to loss of any part of their structure were considered as glomerular with other lesions (GL). This category included segmental sclerosis, moderate to severe mesangial hypercellularity or expansion, crescents, and apparent endothelial proliferation. Each lesion was defined according to the multicenter Nephrotic Syndrome Study Network digital pathology scoring system.22Barisoni L. Troost J.P. Nast C. Bagnasco S. Avila-Casado C. Hodgin J. Palmer M. Rosenberg A. Gasim A. Liensziewski C. Merlino L. Chien H.P. Chang A. Meehan S.M. Gaut J. Song P. Holzman L. Gibson D. Kretzler M. Gillespie B.W. Hewitt S.M. Reproducibility of the NEPTUNE descriptor-based scoring system on whole-slide images and histologic and ultrastructural digital images.Mod Pathol. 2016; 29: 671-684Crossref PubMed Scopus (31) Google Scholar Moderate to severe mesangial hypercellularity was defined as six or more mesangial cells/mesangial area.23Roberts I.S. Cook H.T. Troyanov S. Alpers C.E. Amore A. Barratt J. Berthoux F. Bonsib S. Bruijn J.A. Cattran D.C. Coppo R. D'Agati V. D'Amico G. Emancipator S. Emma F. Feehally J. Ferrario F. Fervenza F.C. Florquin S. Fogo A. Geddes C.C. Groene H.J. Haas M. Herzenberg A.M. Hill P.A. Hogg R.J. Hsu S.I. Jennette J.C. Joh K. Julian B.A. Kawamura T. Lai F.M. Li L.S. et al.Working Group of the International Ig A Nephropathy Network and the Renal Pathology SocietyThe Oxford classification of IgA nephropathy: pathology definitions, correlations, and reproducibility.Kidney Int. 2009; 76: 546-556Abstract Full Text Full Text PDF PubMed Scopus (682) Google Scholar Each image was independently reviewed and labeled by two nephropathologists (L.J. and G.W.). If there was disagreement, a third nephropathologist (expert) (B.D.) reviewed the image and discussed the labeling with the other two pathologists, to reach the final agreement. The images were randomly split into two parts: 1803 glomeruli in the training set and 167 glomeruli in the test set. For the training set, data augmentation was used to increase the amount of training data and diversify the data distribution, so as to improve the robustness of the model. The specific transformation included random rotate 90 degrees, horizontal flip, vertical flip, transpose, and random crop. Thus, the final training set included a total of 10,818 glomeruli (Table 5). AP, average precision; AR, average recall; GL, glomerulus with abnormal structure; GN, glomerulus with basically normal structure; GS, global sclerosis; Mask mIoU, mean intersection/union ratio of the segmentation results; Masson, Masson trichrome stain; PAS, periodic acid–Schiff; PASM, periodic acid–silver methenamine. Similar to the snapshot images, cropped WSIs were randomly split into two parts: 7193 in the training set and 1472 in the test set. After data augmentation, the final training set included a total of 43,152 glomeruli (Table 5). The architectures of the model are shown in Figure 1. Cascade Mask R-CNN architecture, based on CNNs, was trained by using the glomeruli training set. The adopted Cascade Mask R-CNN, which is built by extending Cascade R-CNN20Cai Z. Vasconcelos N. Cascade R.-C.N.N. High Quality Object Detection and Instance Segmentation.High Quality Object Detection and Instance Segmentation. 2021; 43: 1483-1498Google Scholar by adding a branch for predicting an object mask