pc_fib search ranges. The second patent covers the malignant-to-benign breast lesion discrimination method, based on mapping in a low-dimensional feature space employing both the solidity feature and texture-based features, applied to ROIs extracted from MBI images.

**Author Contributions:** Conceptualization, A.F., L.D., B.M.M. and M.J.K.; methodology, A.F., L.D. and J.D.G.C.; software, A.F. and J.D.G.C.; validation, A.F., L.D., J.D.G.C.; formal analysis, A.F., B.M.M., L.D., J.D.G.C., S.M.A.E. and M.J.K.; investigation, B.M.M., S.M.A.E. and M.J.K.; resources, L.D., A.F., and M.J.K.; data curation, A.F., B.M.M., J.D.G.C.; writing—original draft preparation, A.F.; writing—review and editing, L.D., J.D.G.C., B.M.M. and M.J.K.; visualization, A.F. and J.D.G.C.; supervision, L.D., A.F. and M.J.K.; project administration, L.D. and A.F.; funding acquisition, L.D. and A.F. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board (or Ethics Committee) of Galway University Hospital.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Conflicts of Interest:** Authors L. Duchesne, A. Fasoula and J.D. Gil Cano are employed by MVG Industries, the company that has funded this study and is currently conducting clinical investigations of Wavelia, and have a financial interest in the outcome of those clinical investigations. Authors B. M. Moloney, S. M. Abd Elwahab and M.J. Kerin were investigators for the First-in-Human (FiH) clinical investigation of Wavelia and were funded by their institution. The authors declare no conflict of interest.

## References

1. Conceição, R.; Mohr, J.; O'Halloran, M. *An Introduction to Microwave Imaging for Breast Cancer Detection*; Conceição, R.C., Mohr, J.J., O'Halloran, M., Eds.; Biological and Medical Physics, Biomedical Engineering; Springer International Publishing: Cham, Switzerland, 2016; ISBN 978-3-319-27865-0.

2. Gabriel, C. Compilation of the Dielectric Properties of Body Tissues at RF and Microwave Frequencies. *Environ. Health* **1996**. Report N.A.L/OE-TR-1996-0037. Available online: http://www.brooks.af.mil/HSC/AL/OE/OER/Title/Title.html (accessed on 18 October 2021).

3. Campbell, A.M.; Land, D.V. Dielectric properties of female human breast tissue measured in vitro at 3.2 GHz. *Phys. Med. Biol.* **1992**, *37*, 193–210. [CrossRef] [PubMed]

4. Shere, M.; Lyburn, I.; Sidebottom, R.; Massey, H.; Gillett, C.; Jones, L. MARIA® M5: A multicentre clinical study to evaluate the ability of the Micrima radio-wave radar breast imaging system (MARIA®) to detect lesions in the symptomatic breast. *Eur. J. Radiol.* **2019**, *116*, 61–67. [CrossRef]

5. Sani, L.; Ghavami, N.; Vispa, A.; Paoli, M.; Raspa, G.; Ghavami, M.; Sacchetti, F.; Vannini, E.; Ercolani, S.; Saracini, A.; et al. Novel microwave apparatus for breast lesions detection: Preliminary clinical results. *Biomed. Signal Process. Control* **2019**, *52*, 257–263. [CrossRef]

6. Janjic, A.; Cayoren, M.; Akduman, I.; Yilmaz, T.; Onemli, E.; Bugdayci, O.; Aribal, M.E. SAFE: A Novel Microwave Imaging System Design for Breast Cancer Screening and Early Detection—Clinical Evaluation. *Diagnostics* **2021**, *11*, 533. [CrossRef]

7. Meaney, P.M.; Kaufman, P.A.; Muffly, L.S.; Click, M.; Poplack, S.P.; Wells, W.A.; Schwartz, G.N.; di Florio-Alexander, R.M.; Tosteson, T.D.; Li, Z.; et al. Microwave imaging for neoadjuvant chemotherapy monitoring: Initial clinical experience. *Breast Cancer Res.* **2013**, *15*, R35. [CrossRef]

8. Benny, R.; Anjit, T.A.; Mythili, P. An overview of microwave imaging for breast tumor detection. *Prog. Electromagn. Res. B* **2020**, *87*, 61–91. [CrossRef]

9. Moloney, B.M.; O'Loughlin, D.; Abd Elwahab, S.; Kerin, M.J. Breast Cancer Detection—A Synopsis of Conventional Modalities and the Potential Role of Microwave Imaging. *Diagnostics* **2020**, *10*, 103. [CrossRef]

10. O'Loughlin, D.; O'Halloran, M.; Moloney, B.M.; Glavin, M.; Jones, E.; Elahi, M.A. Microwave Breast Imaging: Clinical Advances and Remaining Challenges. *IEEE Trans. Biomed. Eng.* **2018**, *65*, 2580–2590. [CrossRef]

11. Kwon, S.; Lee, S. Recent Advances in Microwave Imaging for Breast Cancer Detection. *Int. J. Biomed. Imaging* **2016**, *2016*, 5054912. [CrossRef] [PubMed]

12. Moloney, B.M.; McAnena, P.F.; Abd Elwahab, S.M.; Fasoula, A.; Duchesne, L.; Gil Cano, J.D.; Glynn, C.; O'Connell, A.; Ennis, R.; Lowery, A.J.; et al. Microwave Imaging in Breast Cancer—Results from the First-In-Human Clinical Investigation of the Wavelia System. *Acad. Radiol.* **2021**. [CrossRef] [PubMed]

13. Fasoula, A.; Duchesne, L.; Gil Cano, J.; Lawrence, P.; Robin, G.; Bernard, J.-G. On-Site Validation of a Microwave Breast Imaging System, before First Patient Study. *Diagnostics* **2018**, *8*, 53. [CrossRef] [PubMed]
14. Moloney, B.M.; McAnena, P.F.; Elwahab, S.M.; Fasoula, A.; Duchesne, L.; Gil Cano, J.D.; Glynn, C.; O'Connell, A.; Ennis, R.; Lowery, A.J.; et al. The Wavelia Microwave Breast Imaging system-tumour discriminating features and their clinical usefulness. *Br. J. Radiol. 2021*. epub ahead of print. [CrossRef]

15. Fasoula, A.; Moloney, B.M.; Duchesne, L.; Cano, J.D.G.; Oliveira, B.L.; Bernard, J.; Kerin, M.J. Super-resolution radar imaging for breast cancer detection with microwaves: The integrated information selection criteria. In Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Berlin, Germany, 23-27 July 2019.

16. Fasoula, A.; Duchesne, L.; Moloney, B.M.; Gil Cano, J.D.; Chenot, C.; Oliveira, B.L.; Bernard, J.-G.; Abd Elwahab, S.M.; Kerin, M.J. Pilot patient study with the Wavelia Microwave Breast Imaging system for breast cancer detection: Clinical feasibility and identified technical challenges. In Proceedings of the 2020 14th European Conference on Antennas and Propagation (EuCAP), Copenhagen, Denmark, 15-20 March 2020.

17. Devaney, A. Super-resolution processing of multi-static data using time reversal and MUSIC. *J. Acoust. Soc. Am. 2000*. Available online: https://ece.northeastern.edu/fac-ece/devaney/preprints/paper02n_00.pdf (accessed on 18 October 2021).

18. Sugitani, T.; Kubota, S.; Kuroki, S.; Sogo, K.; Arihiro, K.; Okada, M.; Kadoya, T.; Hide, M.; Oda, M.; Kikkawa, T. Complex permittivities of breast tumor tissues obtained from cancer surgeries. *Appl. Phys. Lett. 2014*, *104*, 253702. [CrossRef]

19. Lazebnik, M.; Popovic, D.; McCartney, L.; Watkins, C.B.; Lindstrom, M.J.; Harter, J.; Sewall, S.; Ogilvie, T.; Magliocco, A.; Breslin, T.M.; et al. A large-scale study of the ultrawideband microwave dielectric properties of normal, benign and malignant breast tissues obtained from cancer surgeries. *Phys. Med. Biol. 2007*, *52*, 6093-6115. [CrossRef] [PubMed]

20. Martellosio, A.; Bellomi, M.; Pasian, M.; Bozzi, M.; Perregrini, L.; Mazzanti, A.; Svelto, F.; Summers, P.E.; Renne, G.; Preda, L. Dielectric Properties Characterization From 0.5 to 50 GHz of Breast Cancer Tissues. *IEEE Trans. Microw. Theory Tech. 2017*, *65*, 998-1011. [CrossRef]

21. Summers, P.E.; Vingiani, A.; Di Pietro, S.; Martellosio, A.; Espin-Lopez, P.F.; Di Meo, S.; Pasian, M.; Ghitti, M.; Mangiacotti, M.; Sacchi, R.; et al. Towards mm-wave spectroscopy for dielectric characterization of breast surgical margins. *The Breast 2019*, *45*, 64-69. [CrossRef]

22. Pertuz, S.; Puig, D.; Garcia, M.A. Analysis of focus measure operators for shape-from-focus. *Pattern Recognit. 2013*, *46*, 1415-1432. [CrossRef]

23. O'loughlin, D.; Krewer, F.; Glavin, M.; Jones, E.; O'halloran, M. Focal quality metrics for the objective evaluation of confocal microwave images. *Int. J. Microw. Wirel. Technol. 2017*, *9*, 1365-1372. [CrossRef]

24. Sickles, E.A.; D'Orsi, C.J.; Bassett, L.W.; Appleton, C.M.; Berg, W.A.; Burnside, E.S. ACR Bi-Rads® Mammography. In ACR BI-RADS® *Atlas, Breast Imaging Reporting and Data System*; American College of Radiology: Reston, VA, USA, 2013; Volume 5.

25. Chan, T.F.; Vese, L.A. Active contours without edges. *IEEE Trans. Image Process. 2001*, *10*, 266-277. [CrossRef]

26. Chen, Y.; Gunawan, E.; Low, K.S.; Wang, S.C.; Soh, C.B.; Putti, T.C. Effect of Lesion Morphology on Microwave Signature in 2-D Ultra-Wideband Breast Imaging. *IEEE Trans. Biomed. Eng. 2008*, *55*, 2011-2021. [CrossRef] [PubMed]

27. Davis, S.K.; Van Veen, B.D.; Hagness, S.C.; Kelcz, F. Breast Tumor Characterization Based on Ultrawideband Microwave Backscatter. *IEEE Trans. Biomed. Eng. 2008*, *55*, 237-246. [CrossRef]

28. Gerazov, B.; Conceicao, R.C. Deep learning for tumour classification in homogeneous breast tissue in medical microwave imaging. In Proceedings of the IEEE EUROCON 2017-17th International Conference on Smart Technologies, Ohrid, Macedonia, 6-8 July 2017; pp. 564-569.

29. Oliveira, B.; Godinho, D.; O'Halloran, M.; Glavin, M.; Jones, E.; Conceição, R. Diagnosing Breast Cancer with Microwave Technology: Remaining challenges and potential solutions with machine learning. *Diagnostics 2018*, *8*, 36. [CrossRef] [PubMed]

30. Conceição, R.C.; Medeiros, H.; Godinho, D.M.; O'Halloran, M.; Rodriguez-Herrera, D.; Flores-Tapia, D.; Pistorius, S. Classification of breast tumor models with a prototype microwave imaging system. *Med. Phys. 2020*, *47*, 1860-1870. [CrossRef]

31. Doshi, T.; Lyburn, I.; Sidebottom, R.; Gibbins, D. Radio-wave imaging: Frequency response as an aid to lesion characterization. Early concept work. In Proceedings of the Symposium Mammographicum, Liverpool, UK, 8-10 July 2018.

32. Rana, S.P.; Dey, M.; Tiberi, G.; Sani, L.; Vispa, A.; Raspa, G.; Duranti, M.; Ghavami, M.; Dudley, S. Machine Learning Approaches for Automated Lesion Detection in Microwave Breast Imaging Clinical Data. *Sci. Rep. 2019*, *9*, 10510. [CrossRef]

33. Mendelson, E.B.; Böhm-Vélez, M.; Berg, W.A. ACR BI-RADS® Ultrasound. In ACR BI-RADS® *Atlas, Breast Imaging Reporting and Data System*; American College of Radiology: Reston, VA, USA, 2013.

34. De Brito Silva, T.F.; de Paiva, A.C.; Silva, A.C.; Braz Júnior, G.; de Almeida, J.D.S. Classification of breast masses in mammograms using geometric and topological feature maps and shape distribution. *Res. Biomed. Eng. 2020*, *36*, 225-235. [CrossRef]

35. Haralick, R.M.; Shanmugam, K.; Dinstein, I. Textural Features for Image Classification. *IEEE Trans. Syst. Man. Cybern. 1973*, *SMC-3*, 610-621. [CrossRef]

36. Amadasun, M.; King, R. Textural features corresponding to textural properties. *IEEE Trans. Syst. Man. Cybern. 1989*, *19*, 1264-1274. [CrossRef]

37. Safdarian, N.; Hedyezadeh, M. Detection and Classification of Breast Cancer in Mammography Images Using Pattern Recognition Methods. *Multidiscip. Cancer Investig. 2019*, *3*, 13-24. [CrossRef]

38. Sadad, T.; Hussain, A.; Munir, A.; Habib, M.; Ali Khan, S.; Hussain, S.; Yang, S.; Alawairdhi, M. Identification of Breast Malignancy by Marker-Controlled Watershed Transformation and Hybrid Feature Set for Healthcare. *Appl. Sci. 2020*, *10*, 1900. [CrossRef]

39. Vallières, M.; Freeman, C.R.; Skamene, S.R.; El Naqa, I. A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. *Phys. Med. Biol. 2015*, *60*, 5471-5496. [CrossRef] [PubMed]
40. Parekh, V.; Jacobs, M.A. Radiomics: A new application from established techniques. *Expert Rev. Precis. Med. Drug Dev.* **2016**, *1*, 207–226. [CrossRef] [PubMed]

41. Coroller, T.P.; Grossmann, P.; Hou, Y.; Rios Velazquez, E.; Leijenaar, R.T.H.; Hermann, G.; Lambin, P.; Haibe-Kains, B.; Mak, R.H.; Aerts, H.J.W.L. CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma. *Radiother. Oncol.* **2015**, *114*, 345–350. [CrossRef] [PubMed]

42. Parekh, V.S.; Jacobs, M.A. Multiparametric radiomics methods for breast cancer tissue characterization using radiological imaging. *Breast Cancer Res. Treat.* **2020**, *180*, 407–421. [CrossRef]
<figure><img src="image_75.png" /></figure>
<figure><img src="image_76.png" /></figure>

Article

# Artificial Neural Network-Derived Cerebral Metabolic Rate of Oxygen for Differentiating Glioblastoma and Brain Metastasis in MRI: A Feasibility Study

Hakim Baazaoui<fnref n="1" />, Simon Hubertus<fnref n="1" />, Máté E. Maros<fnref n="2" />,<fnref n="3" />, Sherif A. Mohamed<fnref n="4" />, Alex Förster<fnref n="2" />, Lothar R. Schad<fnref n="1" /> and Holger Wenz<fnref n="2" />,*

**Featured Application:** MR-derived cerebral metabolic rate of oxygen in contrast-enhancing and peritumoral non-enhancing regions, as calculated by an artificial neural network, allows for robust differentiation of glioblastoma and brain metastasis.

**Abstract:** Glioblastoma may appear similar to cerebral metastasis on conventional MRI in some cases, but their therapies differ significantly. This prospective feasibility study was aimed at differentiating them by applying the quantitative susceptibility mapping and quantitative blood-oxygen-level-dependent (QSM + qBOLD) model to these entities for the first time. We prospectively included 15 untreated patients with glioblastoma (n = 7, median age: 68 years, range: 54–84 years) or brain metastasis (n = 8, median age 66 years, range: 50–78 years) who underwent preoperative MRI including multi-gradient echo and arterial spin labeling sequences. Oxygen extraction fraction (OEF), cerebral blood flow (CBF) and cerebral metabolic rate of oxygen (CMRO₂) were calculated in the contrast-enhancing tumor (CET) and peritumoral non-enhancing T2 hyperintense region (NET2), using an artificial neural network. We demonstrated that OEF in CET was significantly lower (p = 0.03) for glioblastomas than metastases, all features were significantly higher (p = 0.01) in CET than in NET2 for metastasis patients only, and the ratios of CET/NET2 for CBF (p = 0.04) and CMRO₂ (p = 0.01) were significantly higher in metastasis patients than in glioblastoma patients. Discriminative power of a support-vector machine classifier was highest with a combination of two features, yielding an area under the receiver operating characteristic curve of 0.94 with 93% diagnostic accuracy. QSM + qBOLD allows for robust differentiation of glioblastoma and cerebral metastasis while yielding insights into tumor oxygenation.

**Keywords:** brain metastasis; glioblastoma; machine learning; oxygenation; tumor infiltration

## 1. Introduction

Glioblastoma (GBM) and cerebral metastasis (cMET) are the most common brain tumors in adult patients [1]. Reliably differentiating GBM and cMET based on their conventional magnetic resonance imaging (MRI) characteristics has proven difficult [2,3], as both tumor types can show necrotic centers, contrast-enhancing peripheral areas and peritumoral edema (Figure 1) [4]. However, studies employing advanced MR-imaging techniques

**Citation:** Baazaoui, H.; Hubertus, S.; Maros, M.E.; Mohamed, S.A.; Förster, A.; Schad, L.R.; Wenz, H. Artificial Neural Network-Derived Cerebral Metabolic Rate of Oxygen for Differentiating Glioblastoma and Brain Metastasis in MRI: A Feasibility Study. *Appl. Sci.* **2021**, *11*, 9928.
https://doi.org/10.3390/app11219928

Academic Editor: Fabio La Foresta

Received: 26 August 2021

Accepted: 22 October 2021

Published: 24 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_77.png" /></figure>

Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

<footnote n="1">¹ Computer Assisted Clinical Medicine, Medical Faculty Mannheim, Heidelberg University, 68167 Mannheim, Germany; hb12@medma.ad.uni-heidelberg.de (H.B.); simon.hubertus@medma.uni-heidelberg.de (S.H.); lothar.schad@medma.uni-heidelberg.de (L.R.S.)</footnote>

<footnote n="2">² Department of Neuroradiology, Medical Faculty Mannheim, Heidelberg University, 68167 Mannheim, Germany; mate.maros@umm.de (M.E.M.); alex.foerster@umm.de (A.F.)</footnote>

<footnote n="3">³ Department of Biomedical Informatics at the Center for Preventive Medicine and Digital Health, Medical Faculty Mannheim, Heidelberg University, 68167 Mannheim, Germany</footnote>

<footnote n="4">⁴ Department of Diagnostic and Interventional Radiology, Heidelberg University Hospital, 69120 Heidelberg, Germany; sherif.mohamed@med.uni-heidelberg.de</footnote>

<footnote>* Correspondence: holgerwenz.hw@gmail.com</footnote>
focusing on the tumor microenvironment and hypoxia-induced changes in the microvasculature found that an elevated cerebral blood flow (CBF) and proxies for increased metabolic activity including a higher resulting cerebral metabolic rate of oxygen (CMRO₂) were associated with high-grade gliomas [5,6]. Therefore, we performed a prospective feasibility study to differentiate between GBM and cMET by using an artificial neural network (ANN) approach for non-invasive estimation of CMRO₂, combining quantitative susceptibility mapping (QSM) and the quantitative blood-oxygenation-level-dependent effect (qBOLD). To the best of the authors' knowledge, the concept of using the combined QSM + qBOLD approach for estimation of CMRO₂ is new for differentiating these two entities.

<figure><img src="image_78.png" /><figcaption>**Figure 1.** Axial contrast-enhanced magnetization prepared rapid gradient echo (MP-RAGE) brain images comparing (a) a right temporal metastasis from known esophageal cancer in a 50-year-old male patient and (b) a glioblastoma in the frontal lobe of the left hemisphere of a 71-year-old male patient. Both tumors show cystic elements and peripheral contrast enhancement, complicating a differentiation based on solely morphological criteria.</figcaption></figure>

GBMs constitute between 60% and 70% of all malignant gliomas [7]. Median survival for this highly malignant, infiltratively growing tumor is between 12 and 15 months with optimal treatment [7–9]. A higher median survival of 22 months has been reported for patients with recurrent glioblastoma undergoing a second surgical therapy [10]. The metabolism of GBM cells is adapted to the increased uptake of nutrients by utilizing aerobic glycolysis [11], thereby linking glucose metabolism to oxygen metabolism [12]. Although benefiting from a greater supply of components for cell upkeep and increased perfusion, GBMs frequently grow too fast for their vasculature, eventually resulting in the typical central necrosis with peripheral ring-enhancement [7,8].

Similarly, cMETs can also become ring-enhancing after developing a necrotic center due to them outgrowing their blood and nutrient supply [13,14]. Over 20% of cancer patients develop disseminations to the central nervous system [15]. One study identified that 55% of cMET cases had no known primary at diagnosis [16], while between 30% to 50% of cMETs have been found to first appear as solitary lesions, further complicating their correct identification [2,17,18]. Accurately discriminating between GBM and cMET is of great clinical importance because therapy approach and surgical decisions are quite different and directly affect patient outcomes [9,17,19]. The current diagnostic standard is an invasive tissue biopsy with subsequent histopathological examination [4], a procedure that is not without inherent risks with a complication rate of about 6% [20].
Hence, the discrimination of GBM and cMET has been attempted many times using a variety of different radiological approaches [4,19,21–24]. Perfusion-based studies showed relative cerebral blood volume (rCBV) in the proximal peritumoral non-enhancing T2 FLAIR hyperintensity (NET2) to be significantly higher in GBM than in cMET [21–24]. In the distal parts of NET2, GBM and cMET showed similar rCBV values which may reflect a lack of GBM cell infiltration and angiogenesis, supporting the notion that angiogenesis follows a gradient around the tumor and is highest on the surface of the contrast-enhancing tumor region (CET) [24,25]. With regards to oxygenation, tissue hypoxia is widely accepted as a predictor of therapy resistance to radiation and chemotherapy in gliomas [5]. Hypoxia has been found to stimulate the growth of new blood vessels via the induction and release of vascular endothelial growth factor [10]. This neovascularization leads to a dilated and tortuous vessel configuration, abnormal branching and arteriovenous shunts [26]. The resulting inefficiencies in the tumor vasculature were found to be correlated with greatly increased CBF, lower oxygen extraction fraction (OEF) and, in sum, a higher $CMRO_2$ [6].

In this work, tissue oxygenation was estimated using a combined QSM + qBOLD model that was introduced in 2018 and which utilizes both signal magnitude and phase of a 3D multi-gradient echo (mGRE) sequence [27–29]. We built upon an existing artificial neural network approach to perform the QSM + qBOLD analysis [29]. The artificial neural network was previously used only for mapping OEF in healthy individuals, making this study its first application in a clinical setting by employing it for the differential diagnosis of two brain tumor entities. Perfusion was measured with pseudocontinuous arterial spin labeling.

The purpose of this study was, therefore, to apply the QSM + qBOLD method for the first time to a prospectively recruited collective of GBM and cMET patients and to compare their cerebral oxygenation and perfusion. Based on the hypothesis that the infiltrative growth of GBM and the lack thereof in cMET would create differences in CET and NET2, a machine learning classifier was trained to differentiate the two entities.

## 2. Materials and Methods

### 2.1. Patients

Between December 2019 and October 2020, 15 patients with primary GBM ($n = 7$; median age: 68 years, range: 54–84 years) or cMET ($n = 8$; median age: 66 years, range: 50–78 years) before resection, radiation or chemotherapy were prospectively included in this study as a convenience sample. Ten patients were male, five were female. The cMETs, as determined by histopathology, originated from four lung carcinomas, one esophageal carcinoma and three cancers of unknown primary. Ethics committee approval was granted before recruitment of patients (reference: 2017-666N-MA). Written informed consent was obtained from every participant prior to MRI measurements and the acquired image data was anonymized before further processing. The study was conducted in compliance with the ethical standards of the Declaration of Helsinki of 2013.

### 2.2. Image Acquisition

All MRI scans were performed on a 3T MAGNETOM Trio system (Siemens Healthcare GmbH, Erlangen, Germany) at the Department of Neuroradiology of the Medical Faculty Mannheim. The perfusion data of the first two patients (one cMET, one GBM) were acquired with a 32-channel head coil. Due to practical reasons in clinical day-to-day MRI scans, a 12-channel head coil was used for all subsequent study participants. The MRI protocol included a 3D multi-gradient echo, an unbalanced axial two-dimensional pseudocontinuous arterial spin labeling (three-dimensional pseudocontinuous arterial spin labeling was employed for the first two patients that were scanned with a 32-channel head coil) and a magnetization prepared rapid gradient echo (MP-RAGE) sequence. The gadolinium-based contrast agent Dotarem® (Guerbet, Villepinte, France) was administered as bolus injection for the T1-weighted sequence at a dosage of 0.1 mL/kg body weight. The labeling plane for the
arterial spin labeling sequence was placed circa 85 mm inferior to the anterior commissure-posterior commissure line, approximately perpendicular to the feeding arteries, in line with the consensus recommendation for arterial spin labeling imaging [30]. The sequences and specific parameters used in this study have all been described in detail in a previously published study [29].

## 2.3. Image Processing

The MRI images were registered using the statistical parametric mapping software SPM12 (Wellcome Centre for Human Neuroimaging, UCL, London, UK) using default values. Correct registration was verified with the open-source medical image viewer ITK-SNAP (http://www.itksnap.org/pmwiki/pmwiki.php, accessed on 13 October 2021). Post-processing was performed with MATLAB (Mathworks, Natick, MA, USA). Three regions of interest (ROI) were outlined manually (H.B.) for calculation of oxygenation and perfusion parameters: CET and central necrosis on all slices of the acquired images where the respective region was present, as well as NET2 (Figure 2). The propensity of the central necrotic region to hemorrhage and its very low to non-existent perfusion can lead to an unphysiological spike in OEF (cf. Figure 3) [5]. Therefore, the necrotic region was subtracted from CET to exclude non-vital parts of the tumor in metabolic assessment. For the peritumoral edematous area, a 15–20 mm wide ROI was defined in NET2 on three consecutive FLAIR images with the largest peritumoral hyperintensity, leaving an approximately 3 mm wide margin to avoid partial volume effects [31]. All ROIs were adapted to the tumor border, subtracting a cerebrospinal fluid mask of ventricles and sulci generated automatically in SPM12. The ROIs were audited by an experienced neuroradiologist (H.W.).

<figure><img src="image_79.png" /><figcaption>**Figure 2.** Examples of the different regions of interest (ROIs) assessed in this study. (a) Axial T1-weighted contrast-enhanced image of a 71-year-old male patient with glioblastoma in the frontal lobe of the left hemisphere. The outer ROI marks contrast-enhancing tumor while the ROI inside the tumor indicates the central necrosis. (b) Axial FLAIR image of a 66-year-old male patient with a left hemispheric metastasis in the perirolandic region from unknown primary. The ROI marks the proximal edema bordering the solid-appearing tumor.</figcaption></figure>
<figure><img src="image_80.png" /><figcaption>Figure 3. Oxygen extraction fraction (OEF) map of a 54-year-old female patient with left occipito-temporal glioblastoma. Axes are in mm. The scale to the right of the image indicates OEF (e.g., 0.5 = 50%).</figcaption></figure>

## 2.4. Calculation of Perfusion and Oxygenation Parameters

In order to estimate cerebral perfusion, a quantitative perfusion map was created from the arterial spin labeling data using SPM12. Both the control and tag images from the pCASL sequence were averaged and used together with the proton density weighted image to calculate CBF in mL/100 g/min, using the consensus recommendation for implementation of arterial spin labeling perfusion MRI in clinical applications from Alsop et al. [30]. Time correction for multi-slice imaging was implemented to account for the different transit time of the labeled bolus depending on the time of slice acquisition [30].

CMRO₂ in µmol/100 g/min was calculated as follows:

$$CMRO_2 = CBF \cdot Y_a \cdot OEF \cdot [H], \qquad (1)$$

where $Y_a$ is the arterial oxygen saturation, assumed to be 98%, and [H] = 7.53 µmol/mL is the heme molar concentration in tissue blood assuming a hematocrit of Hct = 0.357 in arterioles [27,32].

An artificial neural network was employed for combined QSM + qBOLD analysis of the mGRE data to estimate $Y$ and calculate OEF [33]:

$$OEF = 1 - \frac{Y}{Y_a}, \qquad (2)$$

where $Y$ and $Y_a$ are venous and arterial blood oxygenation, the latter again assumed to be 98% [27].

## 2.5. Artificial Neural Network

The feed-forward artificial neural network used for estimation of $Y$ was designed in the Neural Network Toolbox in MATLAB (Mathworks, Natick, MA, USA), consisting of one input layer, one hidden layer with 10 nodes and one output layer. The normalized mGRE magnitude signal and the magnetic susceptibility from QSM were used as inputs. The artificial neural network has been described in detail in a previous study [26], where it was trained and used to emulate the solution of the qBOLD model for free induction decay [27], yielding an estimate of venous oxygen saturation $Y$, transverse relaxation rate
R2, deoxygenated blood volume v and non-blood susceptibility $\chi_{nb}$ [29]. After doing this for every voxel, OEF maps were created with Equation (2). In order to make the qBOLD model more robust, QSM was added by calculating magnetic susceptibility [27], using the MEDI toolbox (Cornell MRI Research Lab, Cornell University, New York, NY, USA). The resulting parameters were used for a last fitting step by giving starting values for a quasi-Newton optimization that was stopped once the relative change was smaller than 0.001 or a maximum of 50 iterations was reached (this was mostly the case after approx. 10 iterations) [34].

## 2.6. Statistical Analysis

The statistical analyses were descriptive and performed in MATLAB R2020b (Mathworks, Natick, MA, USA). Variables were summarized using their median, minimum–maximum and interquartile range. Outliers were included in the statistical tests. For all patients, means and standard deviations of OEF, CBF and CMRO₂ were calculated in the ROIs set out in Figure 2. In patients with multiple cMETs, perfusion and oxygenation parameters were not assessed on a metastasis-by-metastasis basis but averaged across all metastases present in the respective patient’s brain. GBM patients were not stratified according to IDH or MGMT promoter methylation status.

Non-parametric Mann–Whitney–Wilcoxon tests for two populations were applied to properly compare the distributions of explanatory variables between the two groups. Two-tailed Wilcoxon signed-rank tests were used for intra-individual comparisons between CET and NET2. *p*-values < 0.05 were considered significant. Due to the explorative nature of this study, *p*-values were not adjusted for multiple comparisons and power analysis for determination of required sample size was not performed [35].

The primary learning objective was the binary classification of GBM vs. cMET. For this, a well-established maximum margin classifier, a linear kernel support-vector machine [36], was fitted to the following features: OEF, CBF and CMRO₂ in CET and NET2. Additional secondary measures, i.e., the ratios of said features in CET divided by NET2, were also calculated and used for fitting [17]. We performed five-fold cross-validation with 10 repeats to assess the robustness of these explanatory variables [37]. Receiver operating characteristic analysis was performed to calculate the area under the receiver operating characteristic curve (AUC) accuracy metrics and derivatives of the confusion matrix, averaged over the five folds [37]. In an effort to compare the classification performance of the linear kernel support-vector machine with other common binary classifiers, ROC analysis was performed for naïve Bayes, weighted k-nearest neighbor, decision trees and for quadratic as well as Gaussian kernel support-vector machines.

## 3. Results

Analysis of oxygenation and perfusion maps in both groups revealed OEF to be lower and CBF to be generally higher than in normal-appearing brain, leading to a net positive effect on CMRO₂, meaning a higher oxygen metabolism in tumor tissue than in an unaffected contralateral brain, both in GBM and in cMET patients. The OEF map of a 54-year-old female patient with left occipito-temporal GBM supplied in Figure 3 gives an illustrative example of the oxygenation values emulated by the artificial neural network. The patient’s brain shows largely uniform OEF values across “healthy” brain matter. The solid-appearing region of the tumor displays a strong OEF signal compared to the surrounding tissue and the rest of the brain while OEF in the peritumoral edematous area appears to be lower than in contralateral white matter.

When comparing oxygenation and perfusion parameters between GBM and cMET, OEF in CET was found to be significantly (*p* = 0.03) lower in GBM than in cMET. No significant differences were found between CET of GBM and cMET in terms of CBF (*p* = 0.33) and CMRO₂ (*p* = 0.15). For cMET patients, all parameters, i.e., OEF, CBF and CMRO₂, were significantly (*p* = 0.01) higher in the CET region than in the NET2 region. Meanwhile, for the GBM group, neither the difference between CET and NET2 in OEF (*p* = 0.11), nor CBF
<figure><img src="image_81.png" /><figcaption>**Figure 5.** Five-fold cross-validated ROC curves for the support-vector machine classifier predicting binary outcome (GBM or cMET). Curves are shown for the ratios of oxygen extraction fraction (OEF<sub>CET/NET2</sub>), cerebral blood flow (CBF<sub>CET/NET2</sub>) and cerebral metabolic rate of oxygen (CMRO<sub>2</sub><sub>CET/NET2</sub>) in contrast-enhancing tumor (CET) divided by peritumoral non-enhancing T2 FLAIR hyperintensity (NET2) as well as for the multivariable fit to OEF<sub>CET</sub> and CMRO<sub>2</sub><sub>CET/NET2</sub>. OEF<sub>CET</sub>: OEF in contrast-enhancing tumor.</figcaption></figure>

For each parameter, values for accuracy, optimal sensitivity and specificity and AUC are listed in Table 1. CMRO₂ CET/NET2 emerged as the best single feature for differentiation of GBM from cMET. The resulting model had an AUC of 0.85 with an accuracy of 83% at an optimal sensitivity and specificity of 85% and 82%, respectively. The next best single classification features in terms of AUC were the ratio CBF*CET/NET2* (0.80) and OEF in CET (0.79). The highest discriminative power with the best diagnostic accuracy was achieved by combining OEF*CET* and CMRO₂*CET/NET2* for fitting a support-vector machine (AUC = 0.94). This allowed for an accurate classification of the tumors in 93% of cases, with a sensitivity of 99% and a specificity of 88%. Averaged over all iterations, out of 15 tumor patients included in this study, 14 were correctly diagnosed and only one case was misclassified.

We also assessed other machine learning classifiers that showed a lower discrimination performance including weighted k-nearest neighbor (AUC: 0.93, accuracy: 87%), naïve Bayes (AUC: 0.88, accuracy: 93%), and decision trees (AUC: 0.66, accuracy: 73%). Additionally, we investigated different variants of support-vector machines with quadratic (AUC: 0.89, accuracy: 87%) and Gaussian kernels (AUC: 0.86, accuracy: 87%). All of these achieved a smaller AUC with k-nearest neighbor coming closest to the linear kernel support-vector machine.
<table><caption>**Table 1.** Receiver operating characteristic analysis results of a linear kernel support-vector machine trained on oxygenation and perfusion parameters.</caption><thead><tr><th>Region</th><th>Feature</th><th>Accuracy</th><th>Sensitivity</th><th>Specificity</th><th>AUC (Range)</th></tr></thead><tbody><tr><td rowspan="3">CET</td><td>OEF</td><td>81%</td><td>87%</td><td>75%</td><td>0.79 (0.76-0.84)</td></tr><tr><td>CBF</td><td>71%</td><td>70%</td><td>71%</td><td>0.67 (0.55-0.73)</td></tr><tr><td>CMRO<sub>2</sub></td><td>63%</td><td>27%</td><td>95%</td><td>0.52 (0.41-0.68)</td></tr><tr><td rowspan="3">NET2</td><td>OEF</td><td>68%</td><td>70%</td><td>66%</td><td>0.65 (0.46-0.82)</td></tr><tr><td>CBF</td><td>73%</td><td>54%</td><td>89%</td><td>0.69 (0.64-0.75)</td></tr><tr><td>CMRO<sub>2</sub></td><td>73%</td><td>44%</td><td>99%</td><td>0.66 (0.61-0.71)</td></tr><tr><td rowspan="3">CET/NET2</td><td>OEF</td><td>69%</td><td>60%</td><td>78%</td><td>0.66 (0.55-0.77)</td></tr><tr><td>CBF</td><td>75%</td><td>63%</td><td>86%</td><td>0.80 (0.77-0.82)</td></tr><tr><td>CMRO<sub>2</sub></td><td>83%</td><td>85%</td><td>82%</td><td>0.85 (0.73-0.93)</td></tr><tr><td>Best combined</td><td>OEF<sub>CET</sub> + CMRO<sub>2</sub> <sub>CET/NET2</sub></td><td>93%</td><td>99%</td><td>88%</td><td>0.94 (0.88-0.96)</td></tr></tbody></table>

Values are shown for contrast-enhancing tumor (CET), peritumoral non-enhancing T2 FLAIR hyperintensity (NET2) and their ratio (CET/NET2). AUC range indicates the lowest and highest values over 10 iterations. OEF: oxygen extraction fraction; CBF: cerebral blood flow; CMRO₂: cerebral metabolic rate of oxygen; AUC: area under the receiver operating characteristic curve.

## 4. Discussion

In this feasibility study, we applied for the first time the MRI-based QSM + qBOLD approach for OEF and CMRO₂ estimation to a prospectively recruited collective of GBM and cMET patients in order to distinguish the two entities based on their respective cerebral oxygenation and perfusion. Since the MRI protocol employed in this study for estimating cerebral tissue oxygenation and perfusion does not require the administration of an intravenous contrast agent per se, it offers the potential to facilitate metabolic imaging of cerebral tumors as well as non-invasive differential diagnosis of GBM and cMET. We demonstrated for the two patient groups that (i) OEF in the enhancing tumor was significantly lower in GBM than in cMET, (ii) the differences in perfusion and oxygenation between CET and NET2 were only significant for the cMET group, and (iii) the ratios of CBF and CMRO₂ in CET divided by NET2 were significantly higher for cMET patients than for GBM patients.

In order to differentiate the two entities, a support-vector machine classifier was trained on oxygenation and perfusion parameters in CET and NET2. A support-vector machine was chosen since it emerged as the best classifier with the highest accuracy, in line with previous investigations that compared it to different approaches such as naïve Bayes, weighted k-nearest neighbor and decision trees for binary classification of glioblastoma and cerebral metastasis [4,23,38]. The described procedure could identify OEF in CET and the ratio of CMRO₂ in CET versus NET2 as the most promising features for distinguishing GBM from cMET, achieving the highest discriminative power. With an accuracy of 93% and an AUC of 0.94, the two entities could be successfully differentiated.

Our results suggest that differentiation of GBM and cMET based solely on OEF in CET is good but not sufficient for reliably distinguishing the two groups. AUC and accuracy were higher than for any other individual parameter in CET or NET2 but lower than those obtained from using the ratios of CBF or CMRO₂ in CET versus NET2. This may be explained by the tumor microenvironment, in particular angiogenesis, that is similar in the contrast-enhancing part of GBM and many hematogenous cMETs [17]. In both entities, a disruption of the blood-brain barrier is caused by tumor growth and angiogenesis: the hypoxic state of the tumor and its surroundings lead to hypoxia-inducible factors being activated and their gene product VEGF being expressed [39,40]. The improved classification performance with the ratios of CET versus NET2 is likely attributable to the dissimilarities in NET2 tissue structure between GBM and cMET. Higher vascularization and neoplastic cell growth are hallmarks of the region surrounding contrast-enhancing GBM [5,6,41], while the peritumoral edema around cMET is caused purely by vasogenic
edema and does not show extensive tumor cell infiltration [42]. This may explain why OEF, CBF and CMRO₂ were significantly higher in CET than in NET2 for cMET patients only.

We observed OEF in CET of GBM patients to be in line with the results of previous studies [26,27,43], reproducing the findings of low OEF, which is possibly a marker of less efficient oxygen extraction due to a physiologically inferior capillary configuration [27], coupled with high CBF. This gives plausibility to the parameter maps calculated with the QSM + qBOLD model that have in the past shown higher, more uniform OEF across the whole brain and more robust CMRO₂ estimates than QSM or qBOLD alone [27]. It was surprising, however, to find discrepancies between the OEF values computed in NET2 of GBM patients: While we found lower OEF in NET2 than in CET, reproducing the results of a previous study [43], two other studies found higher OEF in the edema surrounding the tumor [5,6]. This might either be a cause of incongruent definitions of the ROIs "edema" and "NET2" or another underlying phenomenon. A PET/MR study with a sufficiently large number of patients might be useful to shed light on true OEF in the peritumoral region of GBM.

In addition to the good classification accuracy, the CMRO₂ yielded by QSM + qBOLD serves as an important physiological parameter that has been shown to predict tumor response to antiangiogenic therapy as well as progression-free survival and overall survival in GBM patients [44] through association with intratumoral angiogenesis and oxygenation status [45]. Other possible applications of CMRO₂ that warrant further investigation consist of monitoring tumor response after initial radiation or chemotherapy and early detection of recurrence. With regard to preoperative imaging, CMRO₂ may also serve as a parameter to improve assessment of the surgical margins necessary for a more complete resection of glioblastomas, possibly adding complementary information to conventional MRI sequences for a "supratotal resection" [46].

This work is subject to a number of limitations. The QSM + qBOLD model requires a set of assumptions about physiologic parameters that were not measured for each patient individually. Among these assumptions are a constant tissue hematocrit and arterial oxygen saturation. Furthermore, the OEF values from QSM + qBOLD are prone to susceptibility artifacts from disturbances close to air-tissue bounds, e.g., the sinus frontalis, iron accumulations in deep grey matter or from blood degradation metabolites from hemorrhage, a phenomenon common in the necrotic centers of GBMs and cMETs, hence their exclusion from the assessment. Additional variance was introduced by changing the head coil after the first two patients. However, the perfusion values of these patients remained within a plausible range. Segmentation of ROIs was performed manually, adding a degree of intra- and inter-observer variability. Moreover, cMETs originating from various primary tumors may cause different oxygen metabolism characteristics. Subgroup analysis was not performed because of the relatively small sample size due to the explorative nature of this feasibility study. Thus, further research with larger patient populations is recommended before implementation of QSM + qBOLD into the clinical routine.

## 5. Conclusions

This study demonstrated that QSM + qBOLD allows for non-invasive differential diagnosis of GBM and cMET. In future studies, this differentiation could also be made without the use of an intravenous contrast agent. Going forward, our MRI approach for assessment of tissue oxygen metabolism might be helpful as a diagnostic tool that complements or replaces invasive stereotactic biopsies while yielding metabolic information about the tumor microenvironment and its surroundings, e.g., for predicting tumor response to therapy.

**Supplementary Materials:** The following is available online at https://www.mdpi.com/article/10.3390/app11219928/s1, Table S1: Oxygenation and perfusion parameters in CET and NET2 by individual patient.
**Author Contributions:** Conceptualization, H.B., A.F., L.R.S. and H.W.; methodology, H.B., S.H. and H.W.; software, H.B. and S.H.; validation, M.E.M. and S.A.M.; formal analysis, H.B., M.E.M. and S.A.M.; investigation, H.W.; resources, L.R.S. and H.W.; data curation, H.B. and S.A.M.; writing—Original draft preparation, H.B.; writing—Review and editing, S.H., S.A.M., M.E.M. and H.W.; visualization, H.B.; supervision, L.R.S. and H.W.; project administration, H.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of the University Hospital Mannheim (reference number: 2017-666N-MA, date of approval: 6 August 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. For a detailed breakdown of perfusion and oxygenation values by patient, please consult the supplementary material.

**Acknowledgments:** The authors are thankful for the technical help provided by the staff of the Institute for Computer Assisted Clinical Medicine at the Medical Faculty Mannheim.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Lee, E.J.; TerBrugge, K.; Mikulis, D.; Choi, D.S.; Bae, J.M.; Lee, S.K.; Moon, S.Y. Diagnostic value of peritumoral minimum apparent diffusion coefficient for differentiation of glioblastoma multiforme from solitary metastatic lesions. *AJR Am. J. Roentgenol.* **2011**, *196*, 71–76. [CrossRef]

2. Bauer, A.H.; Erly, W.; Moser, F.G.; Maya, M.; Nael, K. Differentiation of solitary brain metastasis from glioblastoma multiforme: A predictive multiparametric approach using combined MR diffusion and perfusion. *Neuroradiology* **2015**, *57*, 697–703. [CrossRef] [PubMed]

3. Li, X.; Wang, D.; Liao, S.; Guo, L.; Xiao, X.; Liu, X.; Xu, Y.; Hua, J.; Pillai, J.J.; Wu, Y. Discrimination between glioblastoma and solitary brain metastasis: Comparison of inflow-based vascular-space-occupancy and dynamic susceptibility contrast MR imaging. *AJNR Am. J. Neuroradiol.* **2020**, *41*, 583–590. [CrossRef]

4. Artzi, M.; Bressler, I.; Ben Bashat, D. Differentiation between glioblastoma, brain metastasis and subtypes using radiomics analysis. *J. Magn. Reason. Imaging* **2019**, *50*, 519–528. [CrossRef] [PubMed]

5. Preibisch, C.; Shi, K.; Kluge, A.; Lukas, M.; Wiestler, B.; Gottler, J.; Gempt, J.; Ringel, F.; Al Jaberi, M.; Schlegel, J.; et al. Characterizing hypoxia in human glioma: A simultaneous multimodal MRI and PET study. *NMR Biomed.* **2017**, *30*, e3775. [CrossRef] [PubMed]

6. Stadlbauer, A.; Zimmermann, M.; Kitzwogerer, M.; Oberndorfer, S.; Rossler, K.; Dorfler, A.; Buchfelder, M.; Heinz, G. MR imaging-derived oxygen metabolism and neovascularization characterization for grading and IDH gene mutation detection of gliomas. *Radiology* **2017**, *283*, 799–809. [CrossRef]

7. Wen, P.Y.; Kesari, S. Malignant gliomas in adults. *N. Engl. J. Med.* **2008**, *359*, 492–507. [CrossRef] [PubMed]

8. Noroxe, D.S.; Poulsen, H.S.; Lassen, U. Hallmarks of glioblastoma: A systematic review. *ESMO Open* **2016**, *1*, e000144. [CrossRef] [PubMed]

9. Stupp, R.; Mason, W.P.; van den Bent, M.J.; Weller, M.; Fisher, B.; Taphoorn, M.J.; Belanger, K.; Brandes, A.A.; Marosi, C.; Bogdahn, U.; et al. Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. *N. Engl. J. Med.* **2005**, *352*, 987–996. [CrossRef]

10. Montemurro, N.; Fanelli, G.N.; Scatena, C.; Ortenzi, V.; Pasqualetti, F.; Mazzanti, C.M.; Morganti, R.; Paiar, F.; Naccarato, A.G.; Perrini, P. Surgical outcome and molecular pattern characterization of recurrent glioblastoma multiforme: A single-center retrospective series. *Clin. Neurol. Neurosurg.* **2021**, *207*, 106735. [CrossRef]

11. Stadlbauer, A.; Oberndorfer, S.; Zimmermann, M.; Renner, B.; Buchfelder, M.; Heinz, G.; Doerfler, A.; Kleindienst, A.; Roessler, K. Physiologic MR imaging of the tumor microenvironment revealed switching of metabolic phenotype upon recurrence of glioblastoma in humans. *J. Cereb. Blood Flow Metab.* **2020**, *40*, 528–538. [CrossRef]

12. Montemurro, N.; Perrini, P.; Rapone, B. Clinical risk and overall survival in patients with diabetes mellitus, hyperglycemia and glioblastoma multiforme. A review of the current literature. *Int. J. Environ. Res. Public Health* **2020**, *17*, 8501. [CrossRef]

13. Pope, W.B. Brain metastases: Neuroimaging. *Handb. Clin. Neurol.* **2018**, *149*, 89–112. [PubMed]

14. Smirniotopoulos, J.G.; Murphy, F.M.; Rushing, E.J.; Rees, J.H.; Schroeder, J.W. Patterns of contrast enhancement in the brain and meninges. *Radiographics* **2007**, *27*, 525–551. [CrossRef] [PubMed]

15. Ostrom, Q.T.; Wright, C.H.; Barnholtz-Sloan, J.S. Brain metastases: Epidemiology. *Handb. Clin. Neurol.* **2018**, *149*, 27–42. [PubMed]

16. Giordana, M.T.; Cordera, S.; Boghi, A. Cerebral metastases as first symptom of cancer: A clinico-pathologic study. *J. Neurooncol.* **2000**, *50*, 265–273. [CrossRef]
17. Server, A.; Orheim, T.E.; Graff, B.A.; Josefsen, R.; Kumar, T.; Nakstad, P.H. Diagnostic examination performance by using microvascular leakage, cerebral blood volume, and blood flow derived from 3-T dynamic susceptibility-weighted contrast-enhanced perfusion MR imaging in the differentiation of glioblastoma multiforme and brain metastasis. *Neuroradiology* **2011**, *53*, 319–330.

18. Blasel, S.; Jurcoane, A.; Franz, K.; Morawe, G.; Pellikan, S.; Hattingen, E. Elevated peritumoural rCBV values as a mean to differentiate metastases from high-grade gliomas. *Acta Neurochir. 2010*, *152*, 1893–1899. [CrossRef]

19. Lee, E.J.; Ahn, K.J.; Lee, E.K.; Lee, Y.S.; Kim, D.B. Potential role of advanced MRI techniques for the peritumoural region in differentiating glioblastoma multiforme and solitary metastatic lesions. *Clin. Radiol. 2013*, *68*, e689–e697. [CrossRef]

20. Malone, H.; Yang, J.; Hershman, D.L.; Wright, J.D.; Bruce, J.N.; Neugut, A.I. Complications following stereotactic needle biopsy of intracranial tumors. *World Neurosurg. 2015*, *84*, 1084–1089. [CrossRef]

21. Swinburne, N.C.; Schefflein, J.; Sakai, Y.; Oermann, E.K.; Titano, J.J.; Chen, I.; Tadayon, S.; Aggarwal, A.; Doshi, A.; Nael, K. Machine learning for semi-automated classification of glioblastoma, brain metastasis and central nervous system lymphoma using magnetic resonance advanced imaging. *Ann. Transl. Med. 2019*, *7*, 232. [CrossRef]

22. Askaner, K.; Rydelius, A.; Engelholm, S.; Knutsson, L.; Latt, J.; Abul-Kasim, K.; Sundgren, P.C. Differentiation between glioblastomas and brain metastases and regarding their primary site of malignancy using dynamic susceptibility contrast MRI at 3T. *J. Neuroradiol. 2019*, *46*, 367–372. [CrossRef] [PubMed]

23. Tsolaki, E.; Svolos, P.; Kousi, E.; Kapsalaki, E.; Fountas, K.; Theodorou, K.; Tsougos, I. Automated differentiation of glioblastomas from intracranial metastases using 3T MR spectroscopic and perfusion data. *Int. J. Comput. Assist. Radiol. Surg. 2013*, *8*, 751–761. [CrossRef] [PubMed]

24. Lehmann, P.; Saliou, G.; de Marco, G.; Monet, P.; Souraya, S.E.; Bruniau, A.; Vallee, J.N.; Ducreux, D. Cerebral peritumoral oedema study: Does a single dynamic MR sequence assessing perfusion and permeability can help to differentiate glioblastoma from metastasis? *Eur. J. Radiol. 2012*, *81*, 522–527. [CrossRef] [PubMed]

25. Asgari, S.; Rohrborn, H.J.; Engelhorn, T.; Stolke, D. Intra-operative characterization of gliomas by near-infrared spectroscopy: Possible association with prognosis. *Acta Neurochir. 2003*, *145*, 453–459. [CrossRef] [PubMed]

26. Hardee, M.E.; Zagzag, D. Mechanisms of glioma-associated neovascularization. *Am. J. Pathol. 2012*, *181*, 1126–1141. [CrossRef] [PubMed]

27. Cho, J.; Kee, Y.; Spincemaille, P.; Nguyen, T.D.; Zhang, J.; Gupta, A.; Zhang, S.; Wang, Y. Cerebral metabolic rate of oxygen (CMRO2) mapping by combining quantitative susceptibility mapping (QSM) and quantitative blood oxygenation level-dependent imaging (qBOLD). *Magn. Reason. Med. 2018*, *80*, 1595–1604. [CrossRef] [PubMed]

28. Kurz, F.T.; Buschle, L.R.; Rotkopf, L.T.; Herzog, F.S.; Sterzik, A.; Schlemmer, H.P.; Kampf, T.; Bendszus, M.; Heiland, S.; Ziener, C.H. Dependence of the frequency distribution around a sphere on the voxel orientation. *Z. Med. Phys. 2021*. [CrossRef]

29. Hubertus, S.; Thomas, S.; Cho, J.; Zhang, S.; Wang, Y.; Schad, L.R. Using an artificial neural network for fast mapping of the oxygen extraction fraction with combined QSM and quantitative BOLD. *Magn. Reson. Med. 2019*, *82*, 2199–2211. [CrossRef]

30. Alsop, D.C.; Detre, J.A.; Golay, X.; Gunther, M.; Hendrikse, J.; Hernandez-Garcia, L.; Lu, H.; MacIntosh, B.J.; Parkes, L.M.; Smits, M.; et al. Recommended implementation of arterial spin-labeled perfusion MRI for clinical applications: A consensus of the ISMRM perfusion study group and the European consortium for ASL in dementia. *Magn. Reson. Med. 2015*, *73*, 102–116. [CrossRef]

31. Dong, F.; Li, Q.; Jiang, B.; Zhu, X.; Zeng, Q.; Huang, P.; Chen, S.; Zhang, M. Differentiation of supratentorial single brain metastasis and glioblastoma by using peri-enhancing oedema region-derived radiomic features and multiple classifiers. *Eur. Radiol. 2020*, *30*, 3015–3022. [CrossRef]

32. Ma, Y.; Mazerolle, E.L.; Cho, J.; Sun, H.; Wang, Y.; Pike, G.B. Quantification of brain oxygen extraction fraction using QSM and a hyperoxic challenge. *Magn. Reason. Med. 2020*, *84*, 3271–3285. [CrossRef]

33. Hubertus, S.; Thomas, S.; Cho, J.; Zhang, S.; Wang, Y.; Schad, L.R. Comparison of gradient echo and gradient echo sampling of spin echo sequence for the quantification of the oxygen extraction fraction from a combined quantitative susceptibility mapping and quantitative BOLD (QSM+qBOLD) approach. *Magn. Reason. Med. 2019*, *82*, 1491–1503. [CrossRef]

34. Cho, J.; Zhang, S.; Kee, Y.; Spincemaille, P.; Nguyen, T.D.; Hubertus, S.; Gupta, A.; Wang, Y. Cluster analysis of time evolution (CAT) for quantitative susceptibility mapping (QSM) and quantitative blood oxygen level-dependent magnitude (qBOLD)-based oxygen extraction fraction (OEF) and cerebral metabolic rate of oxygen (CMRO2) mapping. *Magn. Reason. Med. 2020*, *83*, 844–857. [CrossRef] [PubMed]

35. Wenz, H.; Maros, M.E.; Meyer, M.; Forster, A.; Haubenreisser, H.; Kurth, S.; Schoenberg, S.O.; Flohr, T.; Leidecker, C.; Groden, C.; et al. Image quality of 3rd generation spiral cranial dual-source CT in combination with an advanced model iterative reconstruction technique: A prospective intra-individual comparison study to standard sequential cranial CT using identical radiation dose. *PLoS ONE 2015*, *10*, e0136054. [CrossRef] [PubMed]

36. Cortes, C.; Vapnik, V. Support-vector networks. *Mach. Learn. 1995*, *20*, 273–297. [CrossRef]

37. Maros, M.E.; Capper, D.; Jones, D.T.W.; Hovestadt, V.; von Deimling, A.; Pfister, S.M.; Benner, A.; Zucknick, M.; Sill, M. Machine learning workflows to estimate class probabilities for precision cancer diagnostics on DNA methylation microarray data. *Nat. Protoc. 2020*, *15*, 479–512. [CrossRef]
38. Shrot, S.; Salhov, M.; Dvorski, N.; Konen, E.; Averbuch, A.; Hoffmann, C. Application of MR morphologic, diffusion tensor, and perfusion imaging in the classification of brain tumors using machine learning scheme. *Neuroradiology* **2019**, *61*, 757–765.
[CrossRef]

39. Fanelli, G.N.; Grassini, D.; Ortenzi, V.; Pasqualetti, F.; Montemurro, N.; Perrini, P.; Naccarato, A.G.; Scatena, C. Decipher the glioblastoma microenvironment: The first milestone for new groundbreaking therapeutic strategies. *Genes* **2021**, *12*, 445.
[CrossRef]

40. Guan, Z.; Lan, H.; Cai, X.; Zhang, Y.; Liang, A.; Li, J. Blood-brain barrier, cell junctions, and tumor microenvironment in brain metastases, the biological prospects and dilemma in therapies. *Front. Cell Dev. Biol.* **2021**, *9*, 722917. [CrossRef] [PubMed]

41. Sunwoo, L.; Yun, T.J.; You, S.H.; Yoo, R.E.; Kang, K.M.; Choi, S.H.; Kim, J.H.; Sohn, C.H.; Park, S.W.; Jung, C.; et al. Differentiation of glioblastoma from brain metastasis: Qualitative and quantitative analysis using arterial spin labeling MR imaging. *PLoS ONE* **2016**, *11*, e0166662. [CrossRef] [PubMed]

42. Fink, K.R.; Fink, J.R. Imaging of brain metastases. *Surg. Neurol. Int.* **2013**, *4*, S209–S219. [CrossRef] [PubMed]

43. Hubertus, S.; Thomas, S.; Cho, J.; Zhang, S.; Kovanlikaya, I.; Wang, Y.; Schad, L.R. In MRI-based oxygen extraction fraction and cerebral metabolic rate of oxygen mapping in high-grade glioma using a combined quantitative susceptibility mapping and quantitative blood oxygenation level-dependent approach. In Proceedings of the International Society for Magnetic Resonance in Medicine, Montréal, QC, Canada, 11–16 May 2019; p. 0391.

44. Bonekamp, D.; Mouridsen, K.; Radbruch, A.; Kurz, F.T.; Eidel, O.; Wick, A.; Schlemmer, H.P.; Wick, W.; Bendszus, M.; Ostergaard, L.; et al. Assessment of tumor oxygenation and its impact on treatment response in bevacizumab-treated recurrent glioblastoma. *J. Cereb. Blood Flow Metab.* **2017**, *37*, 485–494. [CrossRef] [PubMed]

45. Kickingereder, P.; Brugnara, G.; Hansen, M.B.; Nowosielski, M.; Pfluger, I.; Schell, M.; Isensee, F.; Foltyn, M.; Neuberger, U.; Kessler, T.; et al. Noninvasive characterization of tumor angiogenesis and oxygenation in bevacizumab-treated recurrent glioblastoma by using dynamic susceptibility MRI: Secondary analysis of the European Organization for Research and Treatment of Cancer 26101 trial. *Radiology* **2020**, *297*, 164–175. [CrossRef] [PubMed]

46. Yordanova, Y.N.; Duffau, H. Supratotal resection of diffuse gliomas—An overview of its multifaceted implications. *Neurochirurgie* **2017**, *63*, 243–249. [CrossRef] [PubMed]
<figure><img src="image_82.png" /></figure>
<figure><img src="image_83.png" /></figure>

Article

# Choroidal Neovascularization Screening on OCT-Angiography Choriocapillaris Images by Convolutional Neural Networks

Kawther Taibouni<fnref n="1" />, Alexandra Miere<fnref n="2" />, Abdourahmane Samake<fnref n="1" />, Eric Souied<fnref n="2" />, Eric Petit<fnref n="1" /> and Yasmina Chenoune<fnref n="1" />,<fnref n="3" />,*

**Abstract:** Choroidal Neovascularization (CNV) is the advanced stage of Age-related Macular Degeneration (AMD), which is the leading cause of irreversible visual loss for older people in developed countries. Optical Coherence Tomography Angiography (OCTA) is a recent non-invasive imaging technique widely used nowadays in diagnosis and follow-up of CNV. In this study, an automatic screening of CNV based on deep learning is performed using OCTA choriocapillaris images. CNV eyes (advanced wet AMD) are diagnosed among healthy eyes (no AMD) and eyes with drusen (intermediate AMD). An OCTA dataset of 1396 images is used to train and evaluate the model. A pre-trained convolutional neural network (CNN) is fine-tuned and validated on 80% of the dataset while the remaining 20% is used independently for predictions. The model can accurately detect CNV on the test set with an accuracy of 89.74%, precision of 0.96 and 0.99 area under the curve of the receiver operating characteristic. A good overall classification accuracy of 88.46% is obtained on a balanced test set. Detailed analysis of misclassified images shows that they are also considered ambiguous images for expert clinicians. This novel CNN-based application is truly a breakthrough to assist clinicians in the challenging task of screening for neovascular complications.

**Keywords:** age-related macular degeneration; choroidal neovascularization; convolutional neural networks; image classification; optical coherence tomography angiography

**Citation:** Taibouni, K.; Miere, A.; Samake, A.; Souied, E.; Petit, E.; Chenoune, Y. Choroidal Neovascularization Screening on OCT-Angiography Choriocapillaris Images by Convolutional Neural Networks. *Appl. Sci.* **2021**, *11*, 9313. https://doi.org/10.3390/app11199313

Academic Editors: Leonardo Rundo, Carmelo Militello and Andrea Tangherloni

Received: 15 September 2021

Accepted: 2 October 2021

Published: 8 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_84.png" /></figure>

Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

## 1. Introduction

Age-related Macular Degeneration (AMD) is the leading cause of irreversible blindness in the elderly population of developed countries. AMD is characterized by changes in the Retinal Pigment Epithelium (RPE), Bruch's Membrane (BM), or Choriocapillaris (CC) complex [1]. There are several staging systems for AMD, but the most widely used is the AREDS (Age Related Eye Disease Study) classification, distinguishing between early, intermediate and late AMD (see Figure 1) [2]. Early and intermediate AMD are characterized by the presence of drusen and pigmentary changes. Late AMD consists of wet AMD, characterized by choroidal neovascularization (CNV), and dry AMD, characterized by geographic atrophy (GA) in the macular area [3,4]. While both dry and wet AMD are visually threatening, in the particular case of wet AMD, CNV progression can result in rapidly deteriorating visual acuity, leading to scarring and irreversible visual loss [4,5]. Moreover, as hallmarks of early and intermediate AMD, drusen precede the progression to late AMD [6,7]. Hence, distinguishing between the early, intermediate, and late AMD plays a key role in both follow-up and treatment decisions, in order to preserve the visual prognosis.

<footnote n="1">¹ Laboratory of Images, Signals and Intelligent Systems (LISSI, EA N° 3956), University Paris-Est Créteil, Vitry sur Seine, 94400 Paris, France; kawther.taibouni@univ-paris-est.fr (K.T.); abdou.samake0908@gmail.com (A.S.); petit@u-pec.fr (E.P.)</footnote>

<footnote n="2">² Centre Hospitalier Intercommunal de Créteil, Department of Ophthalmology, 40, Avenue de Verdun, Créteil, 94010 Paris, France; alexandra.miere@chicreteil.fr (A.M.); Eric.Souied@chicreteil.fr (E.S.)</footnote>

<footnote n="3">³ ESME Sudria Research Lab., 34 Rue de Fleurus, 75006 Paris, France</footnote>

<footnote>* Correspondence: yasmina.chenoune@esme.fr</footnote>
<figure><img src="image_85.png" /><figcaption>**Figure 1.** Classification of three types of OCTA choriocapillaris images (no AMD, advanced wet AMD, intermediate AMD). On the top-left: the OCTA cross-sectional B-scan showing retinal layers.</figcaption></figure>

Novel imaging techniques, such as optical coherence tomography angiography (OCTA), contribute to the accurate diagnosis of early, intermediate and late AMD in a depth-resolved and non-invasive manner [8–10]. Besides the accurate detection of CNV or GA in the CC segmentation [11], OCTA has demonstrated CC alterations, i.e., flow deficits (FD) and signal attenuation associated with drusen, in early and intermediate AMD [12]. Moreover, recent literature has shown that choriocapillaris FD predict drusen enlargement, therefore being a significant risk factor for late forms of AMD [7]. Therefore, earlier diagnosis could ensure better follow-up of patients at high risk of conversion to advanced wet AMD. A computer-aided diagnosis (CAD) tool in ophthalmic clinical routine could be of significant assistance for clinicians at daily diagnosis and during follow-up.

The detection of CNV in the context of wet AMD on OCTA images is very challenging due to OCTA various artifacts [8,9]. Many recent papers dealt with the contributions of artificial intelligence (AI) and deep learning (DL) approaches in ophthalmology [13–15]. However, only few works have addressed CNV screening on OCTA images using DL methods [16]. Most of the existing works focused on the diagnosis of AMD using OCT or Color Fundus Photography. Moreover, published works on OCTA images involved other retinal diseases such as Diabetic Retinopathy (DR) [17].

In 2017, Rasti et al. [18] proposed a CAD system based on a multi-scale convolutional mixture of expert model to identify accurately dry AMD and diabetic macular edema using OCT images. Two different macular OCT datasets of 4142 and 3247 B-scans were used for the training step. A very good classification score of 99.85% was derived from the receiver operating characteristic curve (ROC-AUC).

In 2018, Burlina et al. [19] used a set of 67,401 color fundus images of AMD patients to estimate 5-year risk of progression to advanced stages of AMD by DL techniques. Two AMD severity scales (4-step and 9-step) were considered, and a human versus machine comparison was carried out. This study achieved weighted k scores of 0.77 for the 4-step and 0.74 for the 9-step AMD severity scales. The same year, Grassmann et al. [20] exploited a database of 120,656 color fundus images, manually graded in 13 AMD severity levels, to train several CNN architectures (AlexNet, GoogLeNet, VGG, Inception-V3, ResNet and I-ResNet-V2). A very good-weighted k of 92% was obtained. Govindaiah et al. [19] have also shown that deep CNN could be efficient to grade color fundus images in four classes: no AMD, early AMD, intermediate AMD, and advanced AMD. The study included
a comparison between training using the transfer learning approach and training from scratch. The obtained accuracies were 78.1% for transfer learning and 83% without transfer learning.

More recently, Russakoff et al. [21] developed a DL architecture so called “AMDnet” on OCT images to predict the conversion from early/intermediate AMD to advanced wet AMD. The study included 71 patients with confirmed early/intermediate AMD that were imaged with OCT three times over 2 years. Results showed a ROC-AUC of 0.89 at the B-scan levels and 0.91 for volumes. Hwang et al. [22] used 35,900 labeled OCT images from AMD patients to train three types of Convolutional Neural Networks (CNNs), VGG19, InceptionV3 and ResNet50, to perform AMD diagnosis. The authors developed an AI and cloud-based telemedicine interaction tool dedicated to diagnosis and therapeutic of AMD. The image discrimination rates obtained by expert clinicians (92.73% and 91.90%) and provided by the AI-based platform (above 90%) were almost the same.

Further works on OCT imaging, such as the recent study of Romo-Bucheli et al. [23], proposed a treatment predictive model using a densely connected neural network (DenseNet) and a recurrent neural network (RNN) on longitudinal OCT scans for neovascular AMD patients (281 patients for training and 69 for tests). The CNN model achieved 0.85 AUC in detecting patients with low treatment requirements and 0.81 AUC for patients with high treatment requirements.

In what concerns the use of OCTA in DL, Le et al. [17] tested the feasibility of using DL for DR detection from OCTA including 77 patients and 20 control subjects. The authors applied transfer learning on a VGG16 network for robust OCTA classification. The obtained results showed an accuracy of 87.27% in differentiating healthy, no DR and DR eyes. In the same period, Wang et al. [16] developed an algorithm based on two CNNs to classify input OCTA images (using structural volumes and enface retinal angiograms) as CNV or Non-CNV and then segment the CNV membrane when present. The proposed neural network included a cutoff threshold for CNV area to overcome the residual artifacts limitation that could be confounded with CNV. CNV binary classification ROC-AUC was 0.997.

In this work, we aim to fill the gap of CNV screening on OCTA images using DL by promoting a novel application of CNNs on OCTA images using the choriocapillaris slab. The main contribution of this paper is the deep learning-based solution to classify AMD on two major forms: advanced wet AMD (CNV) and intermediate AMD (drusen/pigmentary changes) including a healthy control group (no AMD) using choriocapillaris OCTA images. A second contribution is the adaptation of a pre-trained VGG19 model on non-medical ImageNet dataset to medical domain using an adapted densely connected classifier on our limited OCTA data. Additionally, class activation mapping is used to interpret the CNN prediction on choriocapillaris OCTA images, which is a promising DL application for CAD systems in retinal clinical routine.

## 2. Materials and Methods

### 2.1. Dataset and Study Population

Data from patients with AMD is collected from the Ophthalmology Department of Intercommunal Hospital Center of Créteil, France, between September 2014 and July 2019. A database of 1396 choriocapillaris OCTA images of size 304 × 304 with a pixel size of 9.87 × 9.87 µm is built from 787 eyes related to 508 patients (mean age 70.67 ± 17.74 years). All patients underwent a 3 × 3 mm OCTA examination (AngioVue, Optovue, Freemont, CA, USA). The choriocapillaris slab is extracted and there are no excluded images due to motion or projection artifacts.

A retina specialist (A.M.) classified the OCTA images into three classes (391 with no AMD images from healthy eyes of 156 subjects, 457 images with CNV from 187 AMD patients, and 548 images with intermediate AMD from 274 patients). Multiple images per patient are included in this database. On one hand, follow-up images acquired at different dates are considered in this study as they show notable and significant changes in the CNV progression or in the number and size of drusen. On the other hand, both eyes are
considered for some of the patients when both eyes' examinations are available. This study is performed in accordance with the Declaration of Helsinki and current French legislation and with approval of our local ethics committee.

## 2.2. CNV Screening on OCTA Images

Our goal is to discriminate from OCTA images the three predefined classes: No AMD (healthy CC), advanced wet AMD (CNV) and intermediate AMD (drusen or CC impairment) (see Figure 1). OCTA imaging allows physicians to visualize blood vessels in the individual layers of the retina and choroid without dye injection. Thus, CNV, drusen and impairment within the CC (pigmentary changes) can be clearly identified on OCTA images. The healthy choriocapillaris appears on OCTA images as a grainy texture with bright and dark spots corresponding to blood flow and flow deficits, respectively [7]. Drusen and CC impairment are characterized by black nonflow areas of different sizes related to flow deficits that can appear anywhere on the OCTA image surrounded by the grainy texture of the choriocapillaris. Regarding CNV, neovascular membranes harbor the aspect of a vascular branching, surrounded by the grainy texture of the choriocapillaris.

Nevertheless, these images are corrupted by speckle noise due to the physical principles of OCT, in addition to the image acquisition process and artifacts [24]. Moreover, included CNV lesions could have different sizes and locations with irregular shapes of neovascular membranes, thus small ones may be confused with the grainy texture of the choriocapillaris. This makes CNV detection on OCTA images a very challenging task. Figure 1 illustrates the OCTA choriocapillaris images of the three classes used for the classification in this work.

## 2.3. CNN Architecture and Transfer Learning

As depicted on Figure 2, our methodological approach consists of two parts: the VGG19 deep network [25] that provides the features extraction process on the OCTA image and a personalized densely connected network that represents the classification part.

<figure><img src="image_86.png" /><figcaption>**Figure 2.** Our proposed modified VGG19 architecture. OCTA images are mapped through the VGG19 feature extractor to build discriminant 9 × 9 feature maps that are used by the densely connected classifier to generate predicted probabilities and Grad-CAM heatmaps to visualize feature attribution for each class: No AMD (healthy choriocapillaris), advanced wet AMD (CNV), intermediate AMD (drusen/choriocapillaris impairment).</figcaption></figure>
VGG19 is a competition-winning model of the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) [26], that has a sequential pipeline architecture consisting of 16 convolutional and 3 fully connected trainable layers including five max-pooling layers. In this work, only the VGG19 convolutional part is used as a feature extractor where convolution layers include 3 × 3 convolution filters. A rectified linear unit (ReLU) activation function and a downsampling 2 × 2 max-pooling operation follow each convolutional stack. This convolutional part provides vectorized feature maps used as input to the densely connected classifier.

The densely connected classifier contains three fully connected layers, layers 1 and 2 are composed of 1024 nodes each. The last one consists of three nodes that provide the classification result into three types of OCTA images. A regularization dropout layer is included after the first dense layer to overcome overfitting the model by randomly dropping out 50% of the activations at that layer. ReLU activation function is used on the two first dense layers, whereas a softmax activation function is used on the last one.

According to the huge number of learnable VGG19 parameters (144 million) and the limited amount of OCTA data in our training dataset, transfer learning from non-medical data is applied in our approach [27]. Therefore, learned knowledge from the ImageNet dataset [26,28] is transferred to the model and adapted to our application by fine-tuning the convolutional part using OCTA images. The densely connected layers are trained from scratch on our OCTA data to classify OCTA images. Finally, feature maps from the last convolutional layer are mapped through the densely connected classifier to generate Gradient-weighted Class Activation Mapping (Grad-CAM) visualization [29]. The Grad-CAM produces a localization map that highlights the image's important features to the CNN for class predictions.

Additionally, to assess the impact of our approach (transfer learning on modified VGG19 model) on CNV detection accuracy, we trained the original VGG19 model independently from scratch with random initialization on our OCTA data.

## 2.4. *Implementation Details*

The dataset is divided into two independent subsets for training and testing. From the whole dataset, 80% (1115 images) is dedicated to fine-tuning, training, and validation. Subsequently, this first partition is further separated into 80% (892 images) for train and 20% (223 images) for validation. The remaining 20% (281 images) of the whole dataset is used for the performance evaluation and tests.

The whole network is trained end-to-end on 100 epochs for transfer learning and 200 epochs for training original VGG19. Stochastic Gradient Descent (SGD) optimization algorithm [30] and categorical cross entropy loss function are used. The learning-rate is set to $10^{-5}$ for transfer learning and to $10^{-4}$ for training original VGG19. Data augmentation is applied during training to reduce overfitting. Only random zoom is used in transfer learning to generate 16 OCTA images at each batch while rotation, horizontal and vertical flip are used in addition in training from scratch the original VGG19 model to generate 8 OCTA images at each batch.

The pipeline is implemented in Python with the Keras-TensorFlow library [31,32]. Training and testing are performed on a NVIDIA Corporation GP104GL [Quadro P4000] Graphics Processing Unit.

## 2.5. *Performance Evaluation*

Performance is evaluated using an independent balanced test set (78 images for each class) and there are no excluded images due to motion and projection artifacts or to image quality. The CNN prediction output is compared to the ground truth set by the expert reader (A.M.). Four statistical metrics are used to report classification performance [33]:
First, the accuracy that generally describes how the model performs across all classes. It is obtained as the ratio between the number of correct predictions to the total number of predictions:

$$ Accuracy = \frac{True_{positive} + True_{negative}}{True_{positive} + True_{negative} + False_{positive} + False_{negative}} $$

Then the precision, that is calculated as the ratio between the number of positive samples correctly classified to the total number of samples classified as positive. The precision measures the model's accuracy in classifying a sample as positive:

$$ Precision = \frac{True_{positive}}{True_{positive} + False_{positive}} $$

The recall is calculated as the ratio between the number of positive samples correctly classified as Positive to the total number of positive samples. The recall measures the model's ability to detect positive samples. The higher the recall, the more positive samples detected:

$$ Recall = \frac{True_{positive}}{True_{positive} + False_{negative}} $$

Finally, the F1-score is a way of combining the precision and recall:

$$ F1-score = \frac{2 \times Precision \times Recall}{Precision + Recall} $$

Confusion matrix and area under the curve (AUC) of the receiver operating characteristic (ROC) and precision-recall (PRC) curves (ROC-AUC and PRC-AUC) are supplied. Additionally, multiple class activation maps are generated to analyze the feature attribution and understand the CNN predictions.

### 3. Results

CNV screening evaluation, reported in Table 1 and Figure 3, shows that CNV detection on OCTA images achieves the best performance with a precision of 96%, recall of 90%, F1-score of 0.93 and an accuracy of 89.74%. In addition, ROC-AUC and PRC-AUC are 0.99 each. No AMD (healthy CC) OCTA images are also well classified by the proposed VGG19 modified model with a very good accuracy of 94.87% and F1-score of 0.90 (precision 0.85, recall 0.95). Regarding intermediate AMD (drusen/CC impairment) class, the images are, in some cases, confused with no AMD class. Sixty-three images from the intermediate AMD test dataset are correctly classified while 12 images are predicted as no AMD images and 3 as CNV images. This is summarized by the confusion matrix in Table 2.

<table><caption>**Table 1.** CNV screening performance of the modified VGG19. Accuracy, precision, recall, F1-score, area under the curve (AUC) for precision-recall (PRC) and receiver operation characteristic (ROC) curves for the three classes of OCTA images (HCC: healthy choriocapillaris, DCCI: drusen/choriocapillaris impairment, CNV: choroidal neovascularization).</caption><thead><tr><th></th><th>Accuracy (%)</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>PRC-AUC</th><th>ROC-AUC</th></tr></thead><tbody><tr><td>HCC</td><td>94.87</td><td>0.85</td><td>0.95</td><td>0.90</td><td>0.97</td><td>0.99</td></tr><tr><td>DCCI</td><td>80.77</td><td>0.85</td><td>0.81</td><td>0.83</td><td>0.94</td><td>0.97</td></tr><tr><td>CNV</td><td><strong>89.74</strong></td><td><strong>0.96</strong></td><td><strong>0.90</strong></td><td><strong>0.93</strong></td><td><strong>0.99</strong></td><td><strong>0.99</strong></td></tr></tbody></table>
<figure><img src="image_87.png" /><figcaption>**Figure 3.** Receiver operating characteristic (ROC) and precision-recall (PRC) curves for the three classes: advanced wet AMD (CNV), intermediate AMD (drusen/CCI: choriocapillaris impairment), no AMD (healthy choriocapillaris). Area under the curve (AUC) values for each class.</figcaption></figure>

<table><caption>**Table 2.** Confusion matrix of the modified VGG19 prediction on a balanced test dataset of 234 OCTA images (78 images for each class). CNV: choroidal neovascularization, DCCI: drusen/choriocapillaris impairment, HCC: healthy choriocapillaris.</caption><thead><tr><th rowspan="2">Ground Truth Class</th><th colspan="3">Predicted Class</th></tr><tr><th>CNV</th><th>DCCI</th><th>HCC</th></tr></thead><tbody><tr><th>CNV</th><td>70</td><td>7</td><td>1</td></tr><tr><th>DCCI</th><td>3</td><td>63</td><td>12</td></tr><tr><th>HCC</th><td>0</td><td>4</td><td>74</td></tr></tbody></table>

The overall classification accuracy using transfer learning on our modified VGG19 is 88.46% with a loss of 0.089, while the overall classification accuracy using original VGG19 is 83.76% with a loss of 0.37. The proposed approach on the modified VGG19 achieved better performance than original VGG19 for CNV detection. This is reported in classification reports (Tables 1 and 3) and confusion matrices (Tables 2 and 4) where CNV screening accuracy is 74.36% for original VGG19 against 89.74% for our proposed approach. In addition, PRC-AUC and ROC-AUC are 0.99 each using our modified VGG19, while PRC-AUC is 0.95 and ROC-AUC is 0.97 using original VGG19.

<table><caption>**Table 3.** CNV screening performance of the original VGG19. Accuracy, precision, recall, F1-score, area under the curve (AUC) for precision-recall (PRC) and receiver operation characteristic (ROC) curves for the three classes of OCTA images (HCC: healthy choriocapillaris, DCCI: drusen/choriocapillaris impairment, CNV: choroidal neovascularization).</caption><thead><tr><th></th><th>Accuracy (%)</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>PRC-AUC</th><th>ROC-AUC</th></tr></thead><tbody><tr><th>HCC</th><td>97.44</td><td>0.84</td><td>0.97</td><td>0.90</td><td>0.96</td><td>0.99</td></tr><tr><th>DCCI</th><td>79.49</td><td>0.75</td><td>0.79</td><td>0.77</td><td>0.83</td><td>0.93</td></tr><tr><th>CNV</th><td><strong>74.36</strong></td><td><strong>0.97</strong></td><td><strong>0.74</strong></td><td><strong>0.84</strong></td><td><strong>0.95</strong></td><td><strong>0.97</strong></td></tr></tbody></table>
<table><caption>**Table 4.** Confusion matrix of the original VGG19 prediction on a balanced test dataset of 234 OCTA images (78 images for each class). CNV: choroidal neovascularization, DCCI: drusen/choriocapillaris impairment, HCC: healthy choriocapillaris.</caption><thead><tr><th rowspan="2">Ground Truth Class</th><th colspan="3">Predicted Class</th></tr><tr><th>CNV</th><th>DCCI</th><th>HCC</th></tr></thead><tbody><tr><td>CNV</td><td>58</td><td>19</td><td>1</td></tr><tr><td>DCCI</td><td>2</td><td>62</td><td>14</td></tr><tr><td>HCC</td><td>0</td><td>2</td><td>76</td></tr></tbody></table>

Figures 4 and 5 display Grad-CAM visualizations for correct predictions and incorrect predictions of the three classes, respectively, (no AMD, intermediate AMD, and advanced wet AMD). Grad-CAM heatmaps are superimposed on original OCTA choriocapillaris images with warm colors (red, orange, and yellow) for discriminant features and cold colors (blue, cyan, and green) for non-discriminant features.

In Figure 4, expected discriminant features are correctly highlighted by Grad-CAM heatmaps for each class: grainy texture throughout the whole OCTA image for no AMD images (Figure 4A,B), flow deficits/nonflow areas for intermediate AMD images (Figure 4C,D), and high flow vascular networks (CNV) for advanced wet AMD images (Figure 4E,F,G,H). This is further supported by the CNN predicted probabilities for each image. Regarding no AMD images (Figure 4A,B), the CNN predicted high probabilities were 0.94 and 0.97, respectively, and were attributed to no AMD class. Predicted probabilities for intermediate AMD images (Figure 4C,D) were 0.70 and 0.99, respectively, and were attributed to the correct class. Finally, regarding advanced wet AMD images (Figure 4E,F,G,H) predicted probabilities were 0.99 for images Figure 4E,G,H and 0.63 for image Figure 4F, correctly attributed to CNV.

On the other hand, non-discriminant CNV features are highlighted by Grad-CAM heatmaps in Figure 5I,J, including the flow deficits/nonflow areas or regions in Figure 5I and grainy texture in Figure 5J.

The CNN predicted probabilities reinforce this observation where image Figure 5I is predicted as intermediate AMD with 0.84 probability and image Figure 5J as no AMD image with 0.90 probability. Regarding the Figure 5K, non-discriminant drusen features are highlighted by Grad-CAM heatmap, grainy texture is highlighted as discriminant features showing the CNN prediction as no AMD with 0.59 of probability against 0.41 for intermediate AMD. Conversely, Figure 5L represents a healthy CC (no AMD) image predicted as intermediate AMD with 0.58 of probability against 0.42 for no AMD. The Grad-CAM heatmap supports these probabilities by highlighting only drusen features as discriminant rather than those from grainy texture of no AMD images.
<figure><img src="image_88.png" /><figcaption>**Figure 4.** Grad-CAM visualizations for correct predictions by the proposed VGG19 modified model. Below each OCTA image, the corresponding Grad-CAM visualization of the CNN prediction. (A,B) Healthy choriocapillaris—no AMD. (C,D) Drusen/Choriocapillaris impairment-intermediate AMD. (E–H) Choroidal Neovascularization—advanced wet AMD.</figcaption></figure>
<figure><img src="image_89.png" /><figcaption>**Figure 5.** Grad-CAM visualizations for incorrect predictions by the proposed VGG19 modified model. To the right of each OCTA image, the corresponding Grad-CAM visualization of the CNN prediction. (I,J) advanced wet AMD (CNV) images, (K) intermediate AMD image and (L) No AMD (healthy CC) image.</figcaption></figure>

## 4. Discussion and Conclusions

CNV screening on OCT-angiography choriocapillaris images is challenging due to the high variance in neovascular membranes sizes, shapes, and locations. Additionally, the speckle noise and the image acquisition process makes the detection task even more difficult, particularly to differentiate CNV or early drusen from the grainy texture of the choriocapillaris. Our proposed VGG19 modified model achieved very good CNV detection performance with an accuracy of 89.74% and F1-score of 0.93 (as reported in Table 1 [0.96 of precision, 0.90 of recall]). The ROC-AUC and PRC-AUC were 0.99 each (Table 1 and Figure 3). The confusion matrix in Table 2 shows that the CNN could accurately detect the CNV lesion on 70 images out of 78 independent test images. Seven CNV images are confused with intermediate AMD images, only one CNV image is misclassified as healthy CC, and 3 images of intermediate AMD are misclassified as CNV.

Concerning the no AMD (healthy CC) classification performance by our model, the statistical analysis reported in Table 1 and Figure 3 shows that it is the second-best classified class after CNV. Precision and recall are 0.85 and 0.95, respectively, with a ROC-AUC of 0.99, a PRC-AUC of 0.97 and an accuracy of 94.87%. The confusion matrix in Table 2 indicates that 74 no AMD images are correctly classified, only 4 images are confused with intermediate AMD images and no healthy CC image is misclassified as CNV.

Finally, for intermediate AMD detection, statistics show a F1-score of 0.83 (precision of 0.85 and recall of 0.81), an accuracy of 80.77%, a ROC-AUC of 0.97 and a PRC-AUC of 0.94 (reported in Table 1 and Figure 3). These results are better explained by the confusion matrix attributions in Table 2, where 63 of the 78 intermediate AMD images are correctly classified, while 12 images are predicted as no AMD and 3 images confused with CNV.

Furthermore, the CNN predicted probabilities demonstrate, on one hand, the CNN certainty when predicting correct classes (probability higher than 0.90 for the images in Figure 4A,B,D,E,G,H), and on the other hand, the CNN uncertainty for more difficult cases, such as images C and F of Figure 4. Image C illustrates a flow impairment clustered only on one region of the image and surrounded by the grainy texture of the CC on the whole
image. Therefore, the CNN predicted probabilities for image C are 0.70 as intermediate AMD and 0.29 as no AMD. Regarding image F, the CNV membrane is covered by different drusen surrounded by the grainy texture of the CC. Consequently, the CNN predicted probabilities are 0.63 for CNV and 0.36 for intermediate AMD. This shows the CNN's high ability to discriminate wet AMD eyes from healthy and intermediate AMD eyes and to show uncertainty for ambiguous and confusing cases.

The Grad-CAM visualization (see Figure 4) provides a better understanding of these results. No matter the shape, size, and location of the CNV on images of Figure 4E-H; the CNN's predicted high probability is attributed to the correct region with high discriminative CNV features. Conversely, the CNN's low probability is attributed to the non-discriminative regions. Similarly, regardless of the presence of artifacts on images of Figure 4B,C); the CNN feature attribution is correctly highlighted by Grad-CAM heatmaps. This proves that the CNN prediction is based on relevant regions of the OCTA choriocapillaris images related to the three classes: no AMD, intermediate AMD, and advanced wet AMD, that are also considered by the expert reader to detect CNV on OCTA images.

The most difficult step of this classification problem is, on one hand, the detection of some confusing CNV where a small, indefinite vascular shape is visible and for which very small amount of OCTA images are available in our training dataset. On the other hand, the discrimination of early small drusen from healthy choriocapillaris remains problematic. These difficulties are addressed in Figure 5 that illustrates two misclassified CNV images (Figure 5I,J), a misclassified intermediate AMD image (Figure 5K) and a misclassified no AMD image (Figure 5L).

Image I is misclassified as intermediate AMD with 0.84 of probability and image J is the only CNV image of our test dataset misclassified as no AMD with 0.90 of probability. Feature attribution visualization for these two misclassified images illustrated by Grad-CAM heatmaps in Figure 5 helps understanding the CNN prediction for these cases. The Grad-CAM heatmap of image Figure 5I shows that discriminant features are those of drusen by only black nonflow regions which explains the CNN predicted high probability attributed to intermediate AMD class rather than CNV class. Only one small CNV is visible in Figure 5I and is considered as non-discriminant feature. The CNN prediction is thus based on the most present features on the image. Such images are also ambiguous for clinicians and additional imaging modalities are needed to establish a clear diagnosis.

The Grad-CAM heatmap of image Figure 5J explains again the CNN misclassification. Only grainy regions present throughout the whole image are considered as discriminant features by the CNN. The tiny CNV membrane visible on this image is considered as non-discriminant as it is hidden in the grainy texture. This explains the CNN's prediction with high probability (0.90) for image Figure 5J, attributed to no AMD class.

To overcome these classification errors, we should supply our training dataset with more ambiguous OCTA choriocapillaris images such as images I and J of Figure 5 to train the model to detect tiny CNV membranes when they are hidden and confused with drusen or CC grainy texture.

Drusen or significant flow impairment on OCTA choriocapillaris images appear as nonflow areas/flow deficits that are generally surrounded by the grainy texture of CC. When these areas are small and less important than the CC texture, the OCTA image is considered as ambiguous and confusing even by clinicians. This is the case of the 12 OCTA images misclassified by the CNN and predicted as no AMD images. Early small drusen are hardly visible on OCTA images and can be considered as flow deficits related to OCTA image acquisition process. Thus, the early small drusen manually classified as intermediate AMD images are, in some cases, predicted as no AMD by the CNN. These cases are illustrated by typical images in Figure 5K,L.

The above analysis is supported by the predicted probabilities for images K and L of Figure 5. Figure 5K represents an intermediate AMD image predicted as no AMD by the CNN with close probabilities for both classes (0.59 for no AMD against 0.41 for intermediate AMD). Similarly, Figure 5L is a no AMD image predicted as intermediate
AMD by the CNN with close probabilities (0.58 for intermediate AMD against 0.42 for no AMD). This last result reveals again the CNN's ability to show uncertainty in cases of confusing features. To resolve this ambiguity, it would be relevant to classify these images as uncertain images for which the clinician should use additional information from other imaging modalities or patient history to decide and make a diagnosis.

Despite these few misclassification errors, the CNN showed a great ability to screen and detect CNV on OCTA choriocapillaris images. This main finding is achieved through the transfer learning approach that is used to train the proposed VGG19 modified model to overcome the limitation of the small amount of training data. Fine-tuning the proposed modified VGG19 improved the overall classification accuracy compared to that obtained from training the original VGG19 from random initialization. The overall accuracy increased from 83.76% using original VGG19 to 88.46% using transfer learning on the modified VGG19 although it was applied from non-medical data.

This study is one of the few works dealing with CNV screening on OCTA data using only images at the choriocapillaris slab. Obtained results revealed a promising application of CAD systems to diagnose CNV on OCTA choriocapillaris images in clinical routine using DL-based methods. In order to produce more reliable results to clinicians and to help them quantify the CNN uncertainty, we aim to measure the CNN prediction uncertainty in further works to identify how much a CNN could be trusted in diagnosis [34,35] and to avoid using images not suitable for diagnosis when high uncertainty is detected [35].

Further studies on this topic will focus on data augmentation, as well as including more CNV images to work on larger datasets and different imaging modalities to improve classification performance for ambiguous cases.

**Author Contributions:** Conceptualization, K.T., A.M., E.S., E.P. and Y.C.; methodology, K.T.; validation, E.S., E.P. and Y.C.; investigation, K.T., A.M. and A.S.; resources, A.M., A.S. and E.S.; writing—original draft preparation, K.T.; writing—review and editing, K.T., A.M., E.S., E.P. and Y.C.; supervision, E.S., E.P. and Y.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and current French legislation and with approval of our local ethics committee.

**Informed Consent Statement:** Patient consent was waived due to the retrospective nature of the study.

**Data Availability Statement:** The data presented in this study are available on reasonable request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Biesemeier, A.; Taubitz, T.; Julien, S.; Yoeruek, E.; Schraermeyer, U. Choriocapillaris breakdown precedes retinal degeneration in age-related macular degeneration. *Neurobiol. Aging* **2014**, *35*, 2562-2573. [CrossRef] [PubMed]

2. Age-Related Eye Disease Study Research Group A Randomized, Placebo-Controlled, Clinical Trial of High-Dose Supplementation With Vitamins C and E, Beta Carotene, and Zinc for Age-Related Macular Degeneration and Vision Loss. *Arch. Ophthalmol.* **2001**, *119*, 1417-1436. [CrossRef]

3. Colijn, J.M.; Buitendijk, G.H.S.; Prokofyeva, E.; Alves, D.; Cachulo, M.L.; Khawaja, A.P.; Cougnard-Gregoire, A.; Merle, B.M.J.; Korb, C.; Erke, M.G.; et al. Prevalence of age-related macular degeneration in Europe: The past and the future. *Ophthalmology* **2017**, *124*, 1753-1763. [CrossRef] [PubMed]

4. Abramoff, M.D.; Garvin, M.K.; Sonka, M. Retinal Imaging and Image Analysis. *IEEE Rev. Biomed. Eng.* **2010**, *3*, 169-208. [CrossRef] [PubMed]

5. De Jong, P.T. Age-related macular degeneration. *N. Engl. J. Med.* **2006**, *355*, 1474-1485. [CrossRef]

6. Lipecz, A.; Miller, L.; Kovacs, I.; Czakó, C.; Csipo, T.; Baffi, J.; Csiszar, A.; Tarantini, S.; Ungvari, Z.; Yabluchanskiy, A.; et al. Microvascular contributions to age-related macular degeneration (AMD): From mechanisms of choriocapillaris aging to novel interventions. *GeroScience* **2019**, *41*, 813-845. [CrossRef] [PubMed]
7. Nassisi, M.; Tepelus, T.; Nittala, M.G.; Sadda, S.R. Choriocapillaris flow impairment predicts the development and enlargement of drusen. *Graefe's Arch. Clin. Exp. Ophthalmol.* **2019**, *25*, 2079–2085. [CrossRef]

8. de Carlo, T.E.; Romano, A.; Waheed, N.K.; Duker, J.S. A review of optical coherence tomography angiography (OCTA). *Int. J. Retin. Vitr.* **2015**, *1*, 1–15. [CrossRef]

9. Spaide, R.F.; Fujimoto, J.G.; Waheed, N.K. Image artifacts in optical coherence angiography. *Retina* **2015**, *35*, 2163. [CrossRef]

10. Ma, J.; Desai, R.; Nesper, P.; Gill, M.; Fawzi, A.; Skondra, D. Optical Coherence Tomographic Angiography Imaging in Age-Related Macular Degeneration. *Ophthalmol. Eye Dis.* **2017**, *9*, 1179172116686075. [CrossRef]

11. Miere, A.; Butori, P.; Cohen, S.Y.; Semoun, O.; Capuano, V.; Jung, C.; Souied, E.H. Vascular Remodeling of Choroidal Neovascularization after Anti-Vascular Endothelial Growth Factor Therapy Visualized on Optical Coherence Tomography Angiography. *Retina* **2019**, *39*, 548–557. [CrossRef] [PubMed]

12. Alten, F.; Lauermann, J.L.; Clemens, C.R.; Heiduschka, P.; Eter, N. Signal reduction in choriocapillaris and segmentation errors in spectral domain OCT angiography caused by soft drusen. *Graefe's Arch. Clin. Exp. Ophthalmol.* **2017**, *255*, 2347–2355. [CrossRef]

13. Schmidt-Erfurth, U.; Sadeghipour, A.; Gerendas, B.S.; Waldstein, S.M.; Bogunović, H. Artificial intelligence in retina. *Prog. Retin. Eye Res.* **2018**, *67*, 1–29. [CrossRef] [PubMed]

14. Ting, D.S.W.; Pasquale, L.R.; Peng, L.; Campbell, J.P.; Lee, A.Y.; Raman, R.; Tan, G.S.W.; Schmetterer, L.; Keane, P.A.; Wong, T.Y. Artificial intelligence and deep learning in ophthalmology. *Br. J. Ophthalmol.* **2018**, *103*, 167–175. [CrossRef] [PubMed]

15. Sarhan, M.H.; Nasseri, M.A.; Zapp, D.; Maier, M.; Lohmann, C.P.; Navab, N.; Eslami, A. Machine Learning Techniques for Ophthalmic Data Processing: A Review. *IEEE J. Biomed. Health Inform.* **2020**, *24*, 3338–3350. [CrossRef]

16. Wang, J.; Hormel, T.T.; Gao, L.; Zang, P.; Guo, Y.; Wang, X.; Bailey, S.T.; Jia, Y. Automated diagnosis and segmentation of choroidal neovascularization in OCT angiography using deep learning. *Biomed. Opt. Express* **2020**, *11*, 927–944. [CrossRef] [PubMed]

17. Le, D.; Alam, M.; Yao, C.K.; Lim, J.I.; Hsieh, Y.-T.; Chan, R.V.P.; Toslak, D.; Yao, X. Transfer Learning for Automated OCTA Detection of Diabetic Retinopathy. *Transl. Vis. Sci. Technol.* **2020**, *9*, 35. [CrossRef]

18. Rasti, R.; Rabbani, H.; Mehridehnavi, A.; Hajizadeh, F. Macular OCT Classification Using a Multi-Scale Convolutional Neural Network Ensemble. *IEEE Trans. Med. Imaging* **2018**, *37*, 1024–1034. [CrossRef]

19. Burlina, P.M.; Joshi, N.; Pacheco, K.D.; Freund, D.E.; Kong, J.; Bressler, N.M. Use of Deep Learning for Detailed Severity Characterization and Estimation of 5-Year Risk Among Patients With Age-Related Macular Degeneration. *JAMA Ophthalmol.* **2018**, *136*, 1359–1366. [CrossRef]

20. Grassmann, F.; Mengelkamp, J.; Brandl, C.; Harsch, S.; Zimmermann, M.E.; Linkohr, B.; Peters, A.; Heid, I.M.; Palm, C.; Weber, B.H. A Deep Learning Algorithm for Prediction of Age-Related Eye Disease Study Severity Scale for Age-Related Macular Degeneration from Color Fundus Photography. *Ophthalmology* **2018**, *125*, 1410–1420. [CrossRef]

21. Russakoff, D.B.; Lamin, A.; Oakley, J.D.; Dubis, A.M.; Sivaprasad, S. Deep Learning for Prediction of AMD Progression: A Pilot Study. *Investig. Ophthalmol. Vis. Sci.* **2019**, *60*, 712–722. [CrossRef]

22. Hwang, D.-K.; Hsu, C.-C.; Chang, K.-J.; Chao, D.; Sun, C.-H.; Jheng, Y.-C.; Yarmishyn, A.A.; Wu, J.-C.; Tsai, C.-Y.; Wang, M.-L.; et al. Artificial intelligence-based decision-making for age-related macular degeneration. *Theranostics* **2019**, *9*, 232–245. [CrossRef]

23. Romo-Bucheli, D.; Erfurth, U.S.; Bogunovic, H. End-to-End Deep Learning Model for Predicting Treatment Requirements in Neovascular AMD From Longitudinal Retinal OCT Imaging. *IEEE J. Biomed. Health Inform.* **2020**, *24*, 3456–3465. [CrossRef]

24. Baghaie, A.; Yu, Z.; D'Souza, R.M. State-of-the-art in retinal optical coherence tomography image analysis. *Quant. Imaging Med. Surg.* **2015**, *5*, 603–617. [CrossRef]

25. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. *arXiv* **2014**, arXiv:1409.1556.

26. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. Imagenet large scale visual recognition challenge. *Int. J. Comput. Vis.* **2015**, *115*, 211–252. [CrossRef]

27. Pan, S.J.; Yang, Q. A Survey on Transfer Learning. *IEEE Trans. Knowl. Data Eng.* **2010**, *22*, 1345–1359. [CrossRef]

28. Morid, M.A.; Borjali, A.; Del Fiol, G. A scoping review of transfer learning research on medical image analysis using ImageNet. *Comput. Biol. Med.* **2021**, *128*, 104115. [CrossRef] [PubMed]

29. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. *Int. J. Comput. Vis.* **2020**, *128*, 336–359. [CrossRef]

30. Ruder, S. An overview of gradient descent optimization algorithms. *arXiv* **2016**, arXiv:1609.04747.

31. Chollet, F. Keras. 2015. Available online: https://keras.io/ (accessed on 11 May 2021).

32. Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, C.S.; Davis, A.; Dean, J.; Devin, M.; et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. *arXiv Prepr.* **2016**, arXiv:1603.04467.

33. Hossin, M.; Sulaiman, M.N. A review on evaluation metrics for data classification evaluations. *Int. J. Data Min. Knowl. Manag. Process.* **2015**, *5*, 1.

34. Araújo, T.; Aresta, G.; Mendonça, L.; Penas, S.; Maia, C.; Carneiro, Â.; Mendonça, A.M.; Campilho, A. DR | GRADUATE: Uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images. *Med. Image Anal.* **2020**, *63*, 101715. [CrossRef]

35. Laves, M.-H.; Ihler, S.; Ortmaier, T. Uncertainty Quantification in Computer-Aided Diagnosis: Make Your Model say "I don't know" for Ambiguous Cases. *arXiv* **2019**, arXiv:1908.00792.
<figure><img src="image_90.png" /></figure>
<figure><img src="image_91.png" /></figure>

Article

# Deep Learning-Based Segmentation of Various Brain Lesions for Radiosurgery

Siangruei Wu <fnref n="1" />,†, Yihong Wu <fnref n="2" />,†, Haoyun Chang <fnref n="3" />, Florence T. Su <fnref n="4" />, Hengchun Liao <fnref n="2" />, Wanju Tseng <fnref n="5" />, Chunchih Liao <fnref n="6" />, Feipei Lai <fnref n="7" />, Fengming Hsu <fnref n="8" /> and Furen Xiao <fnref n="9" />,*

**Citation:** Wu, S.; Wu, Y.; Chang, H.; Su, F.T.; Liao, H.; Tseng, W.; Liao, C.; Lai, F.; Hsu, F.; Xiao, F. Deep Learning-Based Segmentation of Various Brain Lesions for Radiosurgery. *Appl. Sci.* **2021**, *11*, 9180. https://doi.org/10.3390/app11199180

Academic Editors: Leonardo Rundo, Carmelo Militello, Andrea Tangherloni and Donato Cascio

Received: 27 July 2021

Accepted: 24 September 2021

Published: 2 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Featured Application:** This study implemented deep learning methods to the task of segmentation of various brain lesions, facilitating the treatment planning process of neurosurgery and radiation oncology.

**Abstract:** Semantic segmentation of medical images with deep learning models is rapidly being developed. In this study, we benchmarked state-of-the-art deep learning segmentation algorithms on our clinical stereotactic radiosurgery dataset. The dataset consists of 1688 patients with various brain lesions (pituitary tumors, meningioma, schwannoma, brain metastases, arteriovenous malformation, and trigeminal neuralgia), and we divided the dataset into a training set (1557 patients) and test set (131 patients). This study demonstrates the strengths and weaknesses of deep-learning algorithms in a fairly practical scenario. We compared the model performances concerning their sampling method, model architecture, and the choice of loss functions, identifying suitable settings for their applications and shedding light on the possible improvements. Evidence from this study led us to conclude that deep learning could be promising in assisting the segmentation of brain lesions even if the training dataset was of high heterogeneity in lesion types and sizes.

**Keywords:** deep learning; image segmentation; brain tumors; radiosurgery; magnetic resonance imaging

<figure><img src="image_92.png" /></figure>

Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

## 1. Introduction

Stereotactic radiosurgery (SRS) is a treatment modality using ionizing radiation, focusing on precisely selected areas of tissue. It is usually delivered in a single session, but the radiation dose can also be fractionated. Targeting accuracy and anatomic precision are critical to successful SRS, but are historically secondary concerns in other types of radiation therapy [1]. Undoubtedly, as technology evolves, standards in this area will have to change. Nevertheless, when root mean square errors can be reduced to approximately 1 mm, a threshold of surgical possibilities is reached both in the brain and throughout the rest of

<footnote n="1">¹ Graduate Institute of Communication Engineering, National Taiwan University, Taipei 106, Taiwan; raywu0@gmail.com</footnote>

<footnote n="2">² School of Medicine, National Taiwan University, Taipei 100, Taiwan; patrickwu97@gmail.com (Y.W.); b05401014@ntu.edu.tw (H.L.)</footnote>

<footnote n="3">³ Graduate Institute of Electrical Engineering, National Taiwan University, Taipei 106, Taiwan; hyzhang@eda.ee.ntu.edu.tw</footnote>

<footnote n="4">⁴ College of Arts and Sciences, Santa Clara University, Santa Clara, CA 95053, USA; fsu1@scu.edu</footnote>

<footnote n="5">⁵ Data Intelligence and Application Division, QNAP Systems, Inc., New Taipei 221, Taiwan; rowantseng@qnap.com</footnote>

<footnote n="6">⁶ Department of Neurosurgery, Taipei Hospital, New Taipei 242, Taiwan; ns00360@tph.mohw.gov.tw</footnote>

<footnote n="7">⁷ Department of Computer Science & Information Engineering, National Taiwan University, Taipei 106, Taiwan; flai@csie.ntu.edu.tw</footnote>

<footnote n="8">⁸ Department of Oncology, National Taiwan University Hospital, Taipei 100, Taiwan; hsufengming@ntuh.gov.tw</footnote>

<footnote n="9">⁹ Department of Neurosurgery, National Taiwan University Hospital, Taipei 100, Taiwan</footnote>

<footnote>* Correspondence: xiao@ntuh.gov.tw; Tel.: +886-2-23123456</footnote>

<footnote>† These authors contributed equally to this work.</footnote>
the body. As the ACR-ASTRO guidelines suggest, a targeting accuracy is approximately 1 mm [2–4]. Although SRS can be performed in many parts of the body, it is best known to treat intracranial lesions. The common indications for intracranial SRS include many different types of brain tumors, vascular malformations (including arteriovenous malformation, AVM), and functional diseases such as trigeminal neuralgia (TN). Brain metastases, vestibular schwannomas, meningiomas, and pituitary adenomas are common tumor types treated by SRS.

Before the delivery of SRS to the target (e.g., a brain tumor), detailed treatment planning with precise contouring of the target is conducted by a neurosurgeon or a radiation oncologist. The contouring is performed on computed tomography (CT) or magnetic resonance images (MRI). Sometimes, both CT and MRI are used, depending on the devices and diseases. Normal organs or tissues sensitive to radiation are also contoured so that radiation dose and risk of injury can be estimated. These normal organs are called critical organs or organs at risk (OARs). In terms of image analysis, “precise” segmentation of targets and OARs is mandatory for SRS treatment planning. In current clinical practice, the segmentation is performed by professional personnel. The manual contouring process is time-consuming and prone to substantial inter-practitioner variability, even amongst experts, and may lead to large variation in care quality. Several pieces of research suggest computer assistance [5–10]. We expect an AI-based assistive tool could improve tumor detection, shorten mean contouring time, and increase inter-clinician agreement [11].

As convolutional neural networks (CNNs), the dominant deep learning models, are leading the breakthrough in computer vision recently, they also dominate MRI segmenta- tion tasks. Havaei et al. (2017) proposed the idea of using a deep learning model to perform brain tumor segmentation tasks on MRI images [12]. They pointed out that both local and global representations are essential to produce better results, and this intuition was later realized in various ways. Kamnitsas et al. (2017) later perfected this idea and achieved state-of-the-art performance with a two-path model [13]. On the other hand, U-Net was first proposed for the cell tracking task [14], but then became widely used in many other segmentation tasks [15,16]. In MICCAI BraTS 2017 competition [17], most participants used U-Net variants, as the winner [18] simply assembled three kinds of the most common deep learning models, namely FCN (fully convolutional network) [19], V-Net [20], and DeepMedic [13]. Other than deep learning, some studies on brain cancer segmentation took advantage of fuzzy c-means clustering [21–23], cellular automata [24], random walker [8], and so on [5, 7, 24, 26]. However, they are not deep learning by not possessing over two hidden layers and will not be further discussed.

However, few studies apply deep learning methods to the actual SRS datasets. Unlike the BraTS competitions, real applicable models may need to handle much more diversity rather than a single type of disease. Liu et al. (2017) proposed a modification of DeepMedic that outperformed its parent method in segmentation during SRS treatment planning by adding a subpath, with a dice score reaching 0.67 in a cohort of 240 patients [27]. Lu et al. (2019) assembled two neural networks, namely 3D U-Net and DeepMedic, which were trained with different hyper-parameters so that one neural network focused on small metastases with high sensitivity while the other one addressed overall tumor segmentation with high specificity, yielding a good performance on segmentation within 305 patients, with a median dice score of 0.74 [28]. Fong et al. (2019) trained the convoluted neural network with multiplanar slices, reducing false-positive predictions and yielding a dice score of 0.77 on the 248-patient dataset while maintaining competent 80% isodose coverage [29]. Lu et al. (2021) implemented deep learning in the treatment planning process, decreasing the plethora of time consumed during the planning process as well as enhancing the prediction overlap with ground truth significantly especially in the subgroup of non-experts. However, the cohort size was rather small [11]. Heterogeneity could have been considered a major problem for machine learning decades ago; however, it should be considered a real-world situation. A heterogeneous dataset could help the generalizability and transferability of trained models [11, 23, 25, 30–32]. However, in the
previously-mentioned studies, small sample sizes were important contributors to the lack of confidence to infer the generalization of deep-learning models in clinical practices with heterogeneous lesion types. For the technology to achieve satisfactory performance, we explored the behavior of deep-learning models in a realistic scenario. Therefore, we collected a relatively large dataset with 1688 patients and analyzed the performance of models with various types of settings and architectures. More specifically, we benchmarked the performance of different segmentation models previously proposed for other tasks and also compared the effectiveness of various sampling methods and the choice of loss functions. We used the BRATS dataset to evaluate whether our implementations of deep learning models were correct and comparable to their original implementations.

## 2. Materials and Methods

### 2.1. Dataset

#### 2.1.1. NTUH (National Taiwan University Hospital) Dataset

The data were extracted from a medical center in northern Taiwan. The SRS device used was CyberKnife (Accuray, Sunnyvale, CA, USA) and commenced operation in January 2008. In the decade until December 2017, there were 2578 treatment courses completed in 2411 patients. Among these, 2036 treatment courses of 1921 patients were intracranial.

We only selected patients undergoing first SRS with contrast-enhanced T1-weighted (T1+C) MRI images available. Finally, there were 1688 patients included in our dataset. Their data were randomly divided into training and test sets (Table 1). However, because treatment targets for patients with trigeminal neuralgia are neither tumor nor vascular malformation, their data were all assigned to the training set.

<table><caption>**Table 1.** Clinical diagnoses of 1688 patients in the final dataset.</caption><thead><tr><td>Types of Brain Lesions</td><td>Train</td><td>Test</td></tr></thead><tbody><tr><td>Metastases</td><td>504</td><td>53</td></tr><tr><td>Meningioma</td><td>314</td><td>29</td></tr><tr><td>Schwannoma</td><td>305</td><td>20</td></tr><tr><td>Pituitary tumor</td><td>147</td><td>8</td></tr><tr><td>Arteriovenous malformation</td><td>80</td><td>6</td></tr><tr><td>Trigeminal neuralgia</td><td>38</td><td>0</td></tr><tr><td>Other tumors</td><td>169</td><td>15</td></tr><tr><td>Total</td><td>1557</td><td>131</td></tr></tbody></table>

For each patient, the target was extracted from the treatment planning system together with axial T1+C MRI. Most of the time, the targets were contoured by a neurosurgeon and then reviewed by a radiation oncologist. Occasionally only one radiation oncologist contoured the targets without review by another physician. An image volume could contain more than one target, particularly in patients with brain metastases. NTUH volumes were registered to CT volumes before contouring, with tumor contours stored in CT coordinates. We had to make an inverse transformation to put the tumor labels in MRI coordinates. In other words, volumes were resampled on a common voxel grid instead of directly cropped on grids of different voxel numbers. The images were retrieved from DICOM format and saved in NIfTI-1 data format, where names, birth dates, and geographic data were removed. After registration and de-identification, these image/label pairs were used for the training and evaluation of deep neural networks. The images were presented in native axial slices, with 1–2 mm slice thicknesses. The number of slices varied from 30 to 233 since the slices did not necessarily cover the full cranial regions, instead, they could only include the region of interest. The in-plane resolution was usually 512 × 512, and the smallest resolution was 197 × 197. The field of view in the x–y plane was usually 300 mm, ranging from 200 to 350 mm. The pixel size was mostly 0.5859 × 0.5859 mm² and was 1.1719 × 1.1719 mm² for some images with lower resolution.
There were a total of 2568 distinct targets in these 1688 image sets. The target volumes ranged from 20 to 72,646 mm³, with a median of 1236 mm³ and a mean of 3696 ± 6637 mm³. In 1013 image sets, there was only one target. The number of targets may reach up to 34 in a single image set.

### 2.1.2. BraTS Dataset

The BraTS 2015 dataset is a standard benchmark dataset for MRI segmentation tasks. It includes 220 multi-modal scans of patients with high-grade glioma (HGG) and 54 with low-grade glioma (LGG). T1-weighted, contrast-enhanced T1+C, T2-weighted, and FLAIR images are available. The data had a common dimension of 240 × 240 × 155 with 1 mm³ resolution. The annotation contains five classes: 0 for background, 1 for necrotic core (NC), 2 for edema (OE), 3 for non-enhancing core, and 4 for enhancing core. The evaluation follows the rules of the competition by merging the predictions into three sets: whole tumor (classes 1,2,3,4), core (classes 1,3,4), and enhancing core (class 4). The train to test ratio was 10:1 in each of the experiments we conducted.

### 2.2. Preprocessing

The raw data of the NTUH dataset contains images of different resolutions and fields of view (FOVs). We first used the skull stripping function of Brain-Suite [33] to locate the brain, then utilized the information possessed by the brain masks for centration and cropping of the MRI to make sure the images contain fewer extracranial areas. Brain-Suite was used only to locate the brain center for better cropping. Everything from the scalp to skull remained on the cropped images for reasons that some of our lesions may locate extra-axially. The final input images size was 200 × 200 × 200 mm³. Finally, we normalized them by the z-scores.

Images in the BraTS dataset were already registered, cropped, and normalized with bias field corrections. We only normalized the data by the z-scores for every pulse sequence (T1, T2, T1+C, FLAIR).

### 2.3. Data Augmentation

To perform a fair comparison of the model architectures, we established the following standard data augmentation in the training phase. For 2D models, we performed data augmentation with translation, rotation, shear, zoom, brightness, and elastic distortion [15]. For 3D models, since data augmentation did not yield higher performances of segmentation in the preliminary experiment we performed, we did not perform any type of data augmentation.

### 2.4. Deep Learning Models

The design of the models we employed could be found online [34]. We will discuss the rationale and the architecture below.

#### 2.4.1. DeconvNet

DeconvNet is an architecture adopted from VGG16, a 16-layered CNN by the Visual Geometry Group, and is rather simple to implement [35]. The objective of this design is to overcome the limitations of FCN, which cannot detect objects that are bigger or smaller than a specific size. In this case, the object may be fragmented or mislabeled. Furthermore, FCN only uses one convolution transpose layer to construct its output, so the output loses much detail. As a consequence, DeconvNet uses several layers of transpose-convolution and up-pooling.

The model can be divided into two parts: the encoder and the decoder, which are formed by convolution and deconvolution operations, respectively. It is worth noting that we replaced the max-pooling and up-sampling operations by setting the stride of Conv and Deconv to 2 in our implementation. This is inspired by the recent proposition of generative adversarial networks.
2.4.2. DeepMedic

DeepMedic is another kind of 3D CNN [13]. It is special for taking two inputs, high
resolution, and low resolution. This design seeks to balance fine structures and high-level
information. High-resolution inputs for DeepMedic are patches from our preprocessed
data. Low-resolution inputs are downsampled using 3D average pooling from each corre-
sponding high-resolution patch. Both inputs go through a series of convolution layers with
skip connection, and then it constructs the output by fusing the features of both pathways.
This is a state-of-the-art model, and we expect the model to perform well on segmentation
of brain lesions based on previous benchmarks [27,36–38].

2.4.3. PSPNet

Pyramid scene parsing network, or PSPNet, is a state-of-the-art model in scene parsing tasks [39]. We included it because it is also suitable for our segmentation task. The PSPNet utilizes the high-level representations extracted by a pretrained network, and a novel design of the pyramid pooling module serves as a backend to predict the segmentations. The pyramid pooling modules pool the extracted feature maps to obtain features of different scales. The pretrained model is typically a ResNet trained on an ImageNet dataset [39]. However, on an MRI dataset, the features are not transferable due to the large consistency and the absence of common pretrained models to process MRI images. In our implementation, we randomly initialized the ResNet backend and also removed the deep supervision loss.

2.4.4. U-Net

U-Net tries to improve the fine structure of segmentations and increase the amount of context used [14]. Traditionally, when a certain amount of pooling is required, if one is intending to train with large patches, it unfortunately degrades the performance such as in FCN and DeconvNet. Hence, the U-Net model utilizes skip connections to forward the unpooled features, thus the model can utilize the information of various scales. In our implementation, we abandoned the max-pooling and up-sampling operations for the same reason as in DeconvNet.

2.4.5. V-Net

V-Net is the adaption of U-Net for 3-dimensional data to capture the relationships in
consecutive slices, which were omitted in the 2D models, addressing contiguity problems
and yielding better results in the segmentation of various 3D images [20]. It replaces the
convolution and pooling operations with 3D versions.

2.5. *Sampling Method*

Batch samplers were defined in the source code (see Supplementary Material) [40]. We had four batch sampler designs for our models to experiment with the most efficient and accurate settings. We tried each kind of sampling method for each model in our preliminary experiment, but only those sampling methods that did not cause much overfitting, excessive memory consumption, or lower performance than other sampling methods were benchmarked.

2.5.1. Two Dimensional

For two-dimensional models, we split the MRI data slice by slice and performed
predictions separately. This may result in noises along the sliced axis due to the loss of
spatial contiguity information.

2.5.2. Three Dimensional

For three-dimensional models, the basic strategy is to feed the whole brain image
data directly. In the preliminary experiment, three-dimensional patch resulted in high
memory consumption when we employed the DeepMedic. Thus, we did not perform a
three-dimensional patch for the DeepMedic. Furthermore, while we experimented with this setting on the BraTS2015 dataset, we found it caused overfitting, and we suspect that this is because many of the voxels are irrelevant and redundant for the prediction. Thus, we added two more three-dimensional sampling methods described below.

### 2.5.3. Uniform Patch

To reduce the redundant voxels and save memory usage, we sampled small patches within the brain regions. While inferencing, we simply reassembled the patch predictions together. The patch size used was 152 × 128 × 128. Worth noting, for the DeepMedic, uniform patch resulted in generally lower performance in segmentation, sensitivity, and precision compared with the center patch. Hence this method was not employed when training the DeepMedic.

### 2.5.4. Center Patch

It has been suggested that patches containing foreground regions are crucial to the training [35]. We thus deployed this sampling strategy, which guarantees at least one foreground voxel in the patch. The patch size was default to 64 × 64 × 64. However, we could set the patch size to 96 × 96 × 96 or the same as the size of the uniform patch when we applied the sampling method. When we experimented with the settings in the preliminary test, center patch sizes lower than 190 × 190 would result in tremendously low segmentation performance for V-Net in the NTUH dataset, so this sampling method was not used in the formal benchmark analysis.

### 2.6. Hyperparameters

We used fixed optimizer settings across all experiments. The optimizer chosen was Adam. The learning rate was initially $1 \times 10^{-4}$, with step decay of factor 0.1 at 50 and 70 epochs. Patch-wise methods were inferenced with patches cropped without overlap and excessive boundaries were filled with zero paddings. Samples were generated on the fly in a patch-wise method.

### 2.7. Loss Functions

Class imbalance is a major problem in most tumor segmentation problems, and it is even more severe in our task compared to the BraTS glioma dataset because of small target volumes. The imbalance would most likely lead the model to a trivial solution, which predicts all voxels as background. There are several ways to deal with this problem by modifying the loss function.

#### 2.7.1. Weighted Cross-Entropy

Re-weighting the sparse class is the most common solution to the class imbalance problem. In this study, we set the class weights inversely proportional to the ratio of the class. In particular:

$$C = - \sum_{c=1}^{M} \frac{g_{oc} \log(p_{oc})}{r_c} \quad (1)$$

where M is the number of classes and $r_c$ is the ratio of class c in the whole volume/dataset (as an implementation choice); $g_{oc}$ is the ground-truth label of a voxel; and $p_{oc}$ is the predicted label probability of a voxel of class c.

#### 2.7.2. Soft-Dice

Milletari et al. (2016) suggest using the differentiable version of the dice score, namely soft-dice, directly as the objective due to its resistibility to class imbalance [20]. It is fairly natural to use this loss function because the dice score is the most common evaluation metric in related tasks. There are two implementations of the soft-dice loss function.
Regarding the cardinality of sets, one can perform summation directly or with squaring. In particular:

$$D1 = \frac{2 \sum_{i}^{N} p_{i} g_{i}}{\sum_{i}^{N} p_{i}^{2} + \sum_{i}^{N} g_{i}^{2}} \quad \text{or} \quad D2 = \frac{2 \sum_{i}^{N} p_{i} g_{i}}{\sum_{i}^{N} p_{i} + \sum_{i}^{N} g_{i}} \qquad (2)$$

where $p_i$ is the predicted label probability and $g_i$ is the ground-truth label. We found the two versions producing almost identical performances. In this study, we refer to the second version as the soft-dice loss function.

## 2.8. Evaluation Metrics

### 2.8.1. Dice Score (Hard Dice)

Dice score is the standard metric for evaluating segmentation results. It is defined as

$$D = \frac{2|X \cap Y|}{|X| + |Y|} \qquad (3)$$

where X and Y are the sets of predicted and labeled lesion voxels.

The previously mentioned soft-dice loss is a modified differentiable version of the dice score. We, therefore, refer to the dice score metric as hard-dice to distinguish the two.

### 2.8.2. Precision and Sensitivity

Precision and sensitivity (also known as recall) are standard metrics of binary classification, which is a more general scheme for segmentation. Precision quantifies the volume ratio of correctly predicted lesion voxels (TP) to all predicted lesion voxels (TP + FP). It is defined as

$$Precision = \frac{TP}{TP + FP} \qquad (4)$$

Sensitivity quantifies the volume ratio of correctly predicted lesion voxels (TP) to all labeled lesion voxels (TP + FN). It is defined as

$$Sensitivity = \frac{TP}{TP + FN} \qquad (5)$$

## 2.9. Experiments

In all our experiments, the training and testing were conducted under a 10:1 train-test ratio.

### 2.9.1. Performances of Models on Segmentation of Brain Lesions in NTUH Dataset

We first experimented with the performances of different models on the segmentation of brain tumors after training with various brain lesions. Different batch samplers and loss functions were employed based on the models used. The hard dice, precision, and sensitivity were the outcomes we were interested in. The higher dice score, precision, and sensitivity were deemed as better segmentation performance.

### 2.9.2. Performances of Models on Segmentation of Brain Tumors in BraTS Dataset

To compare the variables contributing to the performances of the models trained with the NTUH dataset, we experimented with the segmentation of the brain tumors in the BraTS dataset. We trained our models with either 4-channel or only T1+C inputs to compare the performances of the models trained with the same imaging modality in the NTUH dataset. During training, the labels encompassed either five classes or only tumor cores. The segmentation performances were measured using the hard dice. The evaluation was based on predictions of the tumor cores.
### 3. Results

Three cases from the NTUH dataset showing representative results of different models were shown in Tables 2–4. The overall dice scores of these networks on the NTUH dataset ranged from 0.33 (DeepMedic) to 0.51 (V-Net). Table 5 shows the detailed performance of each network tested with the NTUH dataset.

<table><caption>**Table 2.** Predictions with low dice scores.</caption><thead><tr><th>Ground truth</th><th>DeconvNet</th><th>DeepMedic</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr><tr><td>PSPNet</td><td>U-Net</td><td>V-Net</td></tr><tr><td></td><td></td><td></td></tr></tbody></table>

<table><caption>**Table 3.** Predictions with average dice scores.</caption><thead><tr><th>Ground truth</th><th>DeconvNet</th><th>DeepMedic</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr></tbody></table>
<table><caption>Table 3. *Cont.*</caption><thead><tr><th>PSPNet</th><th>U-Net</th><th>V-Net</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 4. Predictions with high dice scores.</caption><thead><tr><th>Ground truth</th><th>DeconvNet</th><th>DeepMedic</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr><tr><th>PSPNet</th><th>U-Net</th><th>V-Net</th></tr><tr><td></td><td></td><td></td></tr></tbody></table>

On the NTUH datasets, the performance was also affected by the types of lesions. As shown in Figure 1, we obtained better results for brain metastases, meningiomas, and schwannomas, while all models performed poorly on pituitary tumors, AVMs, and other tumor types. Detailed tables are attached as Appendix A, Appendix B, Appendix C.
<table><caption>**Table 5.** Performance of different models on the NTUH dataset.</caption><thead><tr><th>Model</th><th>Numbers of Parameters</th><th>Batch Samplers</th><th>Loss Function</th><th>Precision</th><th>Sensitivity</th><th>Hard Dice</th></tr></thead><tbody><tr><td>DeconvNet</td><td>12,544,324</td><td>two_dim</td><td>Cross-entropy minus log(soft-dice)</td><td>0.46</td><td>0.48</td><td>0.43</td></tr><tr><td>U-Net</td><td>34,524,034</td><td>two_dim</td><td>Cross-entropy minus log(soft-dice)</td><td><strong>0.48</strong></td><td>0.48</td><td>0.43</td></tr><tr><td>PSPNet</td><td>28,280,773</td><td>two_dim</td><td>Cross-entropy minus log(soft-dice)</td><td>0.47</td><td>0.48</td><td>0.43</td></tr><tr><td>V-Net</td><td>8,232,274</td><td>uniform_patch3d</td><td>Cross-entropy minus log(soft-dice)</td><td>0.39</td><td>0.54</td><td>0.41</td></tr><tr><td>V-Net</td><td>8,232,274</td><td>three_dim</td><td>Cross-entropy</td><td>0.2</td><td>0.56</td><td>0.25</td></tr><tr><td>V-Net</td><td>8,232,274</td><td>three_dim</td><td>Cross-entropy minus log(soft-dice)</td><td><strong>0.48</strong></td><td>0.51</td><td>0.46</td></tr><tr><td>V-Net dropout 0.1</td><td>8,232,274</td><td>three_dim</td><td>Cross-entropy minus log(soft-dice)</td><td>0.47</td><td><strong>0.66</strong></td><td><strong>0.51</strong></td></tr><tr><td>DeepMedic</td><td>1,301,478</td><td>center_patch3d</td><td>Cross-entropy minus log(soft-dice)</td><td>0.36</td><td>0.43</td><td>0.35</td></tr><tr><td>DeepMedic</td><td>1,301,478</td><td>center_patch3d</td><td>Cross-entropy</td><td>0.37</td><td>0.43</td><td>0.33</td></tr></tbody></table>

A
DICE

<figure><img src="image_93.png" /></figure>

B
SENSITIVITY

<figure><img src="image_94.png" /><figcaption>**Figure 1.** *Cont.*</figcaption></figure>
C
PRECISION

<figure><img src="image_95.png" /><figcaption>**Figure 1.** Bar chart results of (A) dice score; (B) sensitivity; and (C) precision of deep learning-based segmentation versus ground truth on different lesion types in the NTUH dataset.</figcaption></figure>

As shown in Figure 2, lesions with smaller target volumes introduce lower average dice performance for each deep-learning model. V-Net, the best-performing model in the current study, obtained a fairly satisfactory dice score when lesion size exceeded the median size of all targets.

<figure><img src="image_96.png" /><figcaption>**Figure 2.** The performance of each model on lesions within different volume ranges.</figcaption></figure>

To compare the performances of different models trained with one-channel input on the segmentation of brain lesions, we performed another experiment in which models were trained with just T1+C input with tumor core labels of the BraTS dataset. The evaluation was based on prediction of the tumor cores. As shown in Figure 3, V-Net had the highest Dice score when trained with 4-channel input with 5-class labels. Interestingly, all models performed better in this circumstance than when trained with only T1+C images. Of note, V-Net and PSPNet could not yield comparable results when trained with only T1+C images, implying that they are more sensitive to the change from multimodal to single modality inputs. While the models trained with one-channel inputs yielded lower performances in segmentation, they still performed better than their counterparts trained with the NTUH dataset.
<figure><img src="image_97.png" /><figcaption>**Figure 3.** The performance of each model trained with four-channel or one-channel inputs.</figcaption></figure>

Because of the nature of PSPNet and DeepMedic, they took a significantly longer time for inference, as shown in Table 6. V-Net had the least number of parameters and the shortest inference time. We also found that adding dropouts in V-Net further improved its performance, which we have noted in the table with 0.1 being the dropout rate.

<table><caption>**Table 6.** Inference time on our hardware and parameters of different networks.</caption><thead><tr><th></th><th>V-Net<br>Dropout 0.1</th><th>DeconvNet</th><th>U-Net</th><th>PSPNet</th><th>DeepMedic</th></tr></thead><tbody><tr><td>Inference time<br>(minutes:seconds)</td><td>02:51</td><td>04:00</td><td>04:01</td><td>14:43</td><td>17:73</td></tr><tr><td>Number of<br>parameters</td><td>8.23M</td><td>12.5M</td><td>34.5M</td><td>28.3M</td><td>13M</td></tr></tbody></table>

# 4. Discussion

## 4.1. Segmentation Performance: NTUH vs. BraTS Dataset

The performance on our radiosurgery dataset was inferior to that on BraTS. Many factors might lead to such a result. First of all, the tumor volumes in the NTUH dataset are typically smaller than those in BraTS 2015. On average, the tumor occupied 1.23% of the whole image volume in the BraTS dataset, but only 0.145% in ours. It should also be noted that a significant portion of our dataset contained multiple targets, which is much less likely for glioma patients (BraTS). The lesions in NTUH dataset are thus more difficult to detect.

Moreover, there is significant heterogeneity in our dataset. To evaluate whether our model could achieve similar segmentation performance under a more realistic scenario, we used the dataset containing cranial lesions of various pathology, which is different from the BraTS dataset with only glioma cases. In a strict sense, we also have some images of non-neoplastic diseases such as AVM. Additionally, some of the tumors are extra-axial (outside the brain parenchyma) and may even extend extracranially, so we cannot perform skull stripping like BraTS. Due to the heterogeneity of tumor types and sites, we may need a much larger dataset to reach similar performance.

Our results indicated that better performance was correlated with more training samples (as in metastases, meningioma, and schwannoma, Figure 2) and larger lesion dimensions (Figure 3). We also report the effect of input channels (of BraTS) in this revision.

Another reason is that we only used one image set (T1+C) to predict instead of four sequences used in the BraTS dataset. Less information might lead to deteriorated performance.
It is also worth mentioning that our dataset is quite imbalanced disease-wise. From the performance of the models we trained, we could observe that this imbalance resulted in serious bias issues for minority patients. We found it quite difficult to train a model by the traditional soft-dice loss or cross-entropy loss. Using the weighted cross-entropy loss gave us a 0.25 dice score, while our modification of subtracting a log-soft-dice term improved the dice score to 0.40. Such difference may result from tumor size since tumors in our dataset were of fewer voxels on average. In addition to the data variety, the weighted cross-entropy function could be very unstable and thus harmful to the optimization. Empirically, we found that the model will most likely fail in 10 epochs and predict nothing but the background for all inputs. By adding another term with the dice score, the new loss function provides better guidance to the model, and we could empirically observe the significant improvements.

We added images of trigeminal neuralgia in the training set as negative samples, in which there was no real space-occupying lesion. We did not expect the machine to learn how to identify trigeminal neuralgia. Instead, it can be considered that images of trigeminal neuralgia are examples of the heterogeneity of real clinical datasets. This artificial impurity was meant to mimic the systematic bias that could occur in a larger and unpurified dataset to infer the availability of deep learning models.

Although the targets in our dataset were defined and contoured by experienced clinicians, it should be noted that they were the targets we wanted to treat. Therefore, in very few cases, not every lesion detected by human experts was labeled. For example, a patient with brain metastases may also have a small meningioma, which may be stable and will not be labeled and treated by radiosurgery. If an algorithm detects that meningioma is this rare, decreased precision and dice score can be expected. However, from the clinical experience of our expert neurosurgeons and radiation oncologists, the rate of intentionally ignored meningiomas and pituitary adenomas was estimated around 1%. This estimation was in parallel with the reported prevalence of intracranial incidentaloma. On the other hand, the estimated rate for ignored brain metastases was much higher (5%), because our clinical experts might decide not to treat small lesions (less than 5–10 mm or visible only on one axial slice) in patients with multiple brain metastases [41,42]. As a result, this should not impede the training due to its rarity, and most meningiomas were labeled.

## 4.2. Performance on Different Types of Tumor

We can see that these models performed better for brain metastases, meningiomas, and schwannomas, where there were more than 300 cases each. They performed best for schwannomas, probably because most of these are vestibular ones, whose locations are always around internal auditory meatus.

On the other hand, these models performed poorly for pituitary tumors, AVMs, and other tumor types. Besides the relatively small number of cases for training, pituitary tumors and AVMs are not always readily visible for humans using only the T1+C series. For example, dynamic contrast-enhanced MRI may be required to visualize pituitary tumors. AVMs are sometimes not visible even using time-of-flight (TOF) MRI, so computed tomography angiography and/or digital subtraction angiography may be required for target contouring.

## 4.3. Comparison between Deep Learning Models

With respect to the input format, there are two classes of model architectures. The 2D model predicts tumors in just one slice and completely discards the information along the z-axis, while the 3D model utilizes the full information on the MRI volume. This results in a trade-off between features and overfitting. When receiving more features, it is more likely to overfit the unrelated noise, especially with such a small dataset. Patch sizes in previous works range from 16 × 16 × 16 to 64 × 64 × 64 mm³ [18,43–55], of which Kamnitsas et al. outperformed the others. Thus, in the current proposed work, we restrained the receptive field and predicted on inputs patches with the size of 64 × 64 × 64 mm³. We examined
this trade-off in our benchmark experiment on the BraTS dataset. Surprisingly, when experimenting with V-Net on our dataset, small patch-wise prediction became detrimental, but receiving the full brain volume guaranteed the best performance.

Overall, the 3D models seem to be more appealing. The 3D models present the full potential of convolution networks, reducing the number of parameters and becoming far more efficient due to their convolution nature. Specifically, V-Net has approximately 1/30 of the parameters compared to U-Net, shortest inference time, and the best performance on dice metric. The only shortcoming of 3D models is the requirement of GPU RAM due to the large input. In our experiments, we solve this by using a smaller batch-size. Furthermore, replacing batch normalization with dropout is quite effective in preventing overfitting because of the small batch size.

We compared the performances of the models trained with one-channel inputs of the NTUH and BraTS datasets. When the models were trained with one-channel inputs, the segmentation performances were slightly better than when they were trained with four-channel inputs. It could be inferred that the models perform better on a dataset with less heterogeneity in lesion types as well as lesion sizes.

### 4.4. Comparison to Previous Studies Addressing Deep Learning-Based Segmentation in SRS Treatment Planning

Efforts to identify the targets and the OARs prior to SRS treatment are crucial for dosimetry planning to protect the organs other than the lesions themselves. Several studies have benefited from deep learning methods on the classification and nomenclature standardization of the OARs [56,57]. The above-mentioned studies could advance computer-assisted radiation therapy.

To evaluate the benchmark performed in this study on the segmentation of brain lesions, previous studies addressing the segmentation of brain tumors in the treatment planning process during SRS will be reviewed. Of all types of brain lesions, asymptomatic or unresectable metastases warrant SRS without maximal surgical resection. As SRS serves as the first-line treatment for oligometastatic lesions, which denotes metastases of lesser than five lesions, contouring the lesions is of important clinical significance. The models previously used included modified DeepMedic [11,27], an ensemble of DeepMedic and 3D U-Net [28], and CNN [29].

Tumor volume tremendously affects the performance of segmentation; higher variety in tumor sizes and smaller lesions usually imply adversity in segmentation. Smaller lesions, while not affecting dice scores much, are not easily detected in methods with lower sensitivity. Liu et al. (2017) [27] proposed a modification of DeepMedic and managed to reach a dice score of 0.67. In their study, the number of brain metastases per case varied from 1 to 93 (5.679 ± 8.917), and the mean tumor size was 672 ± 1994 mm³. Lu et al. (2019) [28] ensembled two neural networks, namely 3D U-Net and DeepMedic, yielding a good performance in segmentation with a median dice score of 0.74. The median size of the tumors in their dataset was 980 mm³, while the smallest tumor was 3 mm³. Fong et al. (2019) [29] trained the convoluted neural network with multiplanar slices, yielding a dice score of 0.77. Lu et al. (2021) [11] implemented an ensemble of 3D U-Net and DeepMedic and enhanced the prediction significantly, especially for non-experts. In their dataset, the median volume of the lesion was 890 mm³. In our dataset, the lesions possessed a median size of 656 mm³ and a mean of 2833 ± 6389 mm³, while the smallest lesion was 13.05 mm³. Generally speaking, with the highest dice score of 0.51, sensitivity of 0.66, and precision of 0.48, the lesions in our dataset had a higher size variety and smaller median size. The inconsistency in the lesion characteristics could cause difficulties for the deep learning models to extract features and hinder the prediction.
Ensemble models introduced higher segmentation performance than a single model in the previous studies [11,27–29]. Although in our study, V-Net with a dropout rate of 0.1 outperformed other methods in segmentation of brain lesions in the NTUH dataset, we did not perform a benchmark on the ensemble methods. It remains undetermined whether ensemble models yield better performance as well as which models ensembled could enhance segmentation the most.

As the difference in the imaging sequences used in the training process is a determinant of segmentation performance, the sequences used in previous works are discussed. Liu et al. (2017) used contrast-enhanced T1-weighted images [27] while Lu et al. (2019) used CT and T1-weighted MRI scans with contrast as the input [28]. Multiplanar slices of MPRAGE (magnetization-prepared rapid acquisition with gradient echo) images were taken as input in Fong et al. (2019) [29]. Lu et al. (2021) used contrast-enhanced CT and T1-weighted MR scans [11]. Out of the three studies, methods with MPRAGE as the input sequence yielded the highest dice score compared to the ground truth. Brain tumors on FLAIR, which is often used to contour the clinical target volume (CTV), mostly appeared as confluent hyperintense signals, introducing higher sensitivity and lower precision. On the other hand, brain tumors were mostly discrete on MPRAGE, an MRI modality taking advantage of gradient echo [58]. Despite the fact that higher precision could be achieved with MPRAGE, it is currently of lower significance in contouring before SRS. Of note, studies have shown that simultaneous use of different imaging modalities promised a better performance in segmentation compared to single modality use [38]. In our study, only contrast-enhanced T1-weighted MR images were used, and this could be a determinant of lower segmentation performance.

The required dataset size to yield high performances could not be confirmed, as we collected the data available to train the models and only draw conclusions from the current dataset. It is probably true that a larger dataset may generate better or different results, but such a dataset was not available to us.

### 4.5. *Limitation of This Study*

Compared to previous works investigating samples that underwent SRS, a relatively large dataset was implemented in the current study. However, the results suggest that the numbers of pituitary tumors, AVMs, and other tumors are probably insufficient for good results. Since the numbers of above lesions in a single institute may be insufficient, federated learning can be a potentially practical approach for better results.

Contrast-enhanced T1-weighted MR imaging was the only modality used as input in our study. Some tumors such as low-grade glioma or pituitary tumors are non-enhancing, introducing great difficulty in the detection and segmentation of these types of lesions. Simultaneous use of multiple imaging modalities could be the solution to this. Reviewing previous works, the sensitivity for detection of smaller brain lesions (<3 mm) with 3D U-Net, whether trained with black-blood or gradient echo modalities, decreased significantly compared to larger brain lesions (≥10 mm, 0.981, 3–10 mm 0.829, <3 mm 0.235) [59]. The same trend could be observed in studies performed with 2-stage MetNet (≥6 mm 0.99, 3–6 mm 0.87, ≤3 mm 0.25) [60] or GoogLeNet [61]. The 2-stage MetNet [60] and BMDS net [62] could achieve satisfactory segmentation prediction on tumors larger than 6 mm, with dice scores of 0.87 and 0.83, respectively. In our dataset counterpart, the diameters of 10.5% lesions were smaller than 6 mm, 45% lesions smaller than 10 mm, and 95.7% smaller than 3 cm. The small lesion sizes in our NTUH dataset contributed to the dice score lower than 0.6 predicted with V-Net.

The way dice score is derived could mask the effect of contouring small lesions. In our work, dice score was calculated per voxel, which favored larger lesions compared to dice score derived per lesion. Clinically, SRS is indicated and is of significant importance for patients with smaller brain lesions, whereas for patients who are surgical candidates with larger lesions, standard care remains surgery with adjuvant stereotactic radiation therapy or whole-brain radiation therapy. As for patients with diffuse lesions, whole-
brain radiation therapy is the standard treatment due to the lack of level 1 evidence to support the use of SRS in the patient population [63] (p. 865). Contouring deflection on the gross tumor volume (GTV) of such small lesions could introduce a huge impact on later target contouring, compromising organs at risk (OAR). Take brain metastases, for example, current guidelines for contouring for SRS generally indicate a 1.5 cm expansion from GTV to generate CTV. In our dataset, the smallest volume of brain lesion being 20 mm³ implies a 3.4 mm diameter, and the volume difference of CTV with GTV is about 3000 mm³. This expansion in target volume significantly differs if a small lesion was not correctly contoured. As a consequence, a dice score per lesion provides benefit in some circumstances.

Evidence derived from trials concerning treatment response to SRS based on either deep-learning segmentation or manual segmentation is still an unmet need. Several studies implemented multiple modalities (PET/MRI) in order to train machine learning models for tumor segmentation, which suggested that biological target volume (BTV) could be promising in helping CTV definition during SRS treatment and their ability to indicate dose escalation on biologically active targets [64,65]. Despite the effort in assisting CTV definition by taking advantage of the training set of multi-modalities, whether the addition in modalities to either of the learning methods improves clinical treatment response is yet undetermined.

## 5. Conclusions

We benchmarked five commonly used deep learning segmentation models on our SRS dataset. We confirmed that these approaches also work on a heterogeneous dataset, but with decreased performance. We discovered that the V-Net architecture worked best for this specific task. With the top dice scores, the smallest size of the model, and the shortest inference time, V-Net may be a good choice to improve upon. We also found that when training on the dataset with such heterogeneity and class imbalance, using weighted cross-entropy loss with log-soft-dice term significantly improved the performance.

**Supplementary Materials:** GitHub repository, https://github.com/raywu0123/Brain-Tumor-Segmentation, (accessed on 25 September 2021).

**Author Contributions:** Conceptualization, S.W., Y.W., H.C., and F.X.; Methodology, S.W., H.C., and Y.W.; Software, S.W. and H.C.; Validation, S.W., Y.W., and F.X.; Formal analysis, Y.W., C.L., and F.X.; Investigation, S.W., H.C., Y.W., and F.X.; Resources, F.X.; Data curation, S.W., H.C., Y.W., and F.X.; Writing—original draft preparation, Y.W., C.L., and F.X.; Writing—review and editing, Y.W., F.T.S., H.L., W.T., C.L., F.L., F.H., and F.X.; Visualization, C.L., Y.W., and F.X.; Supervision, F.X.; Project administration, F.X.; Funding acquisition, F.X. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Ministry of Science and Technology, Taiwan, ROC, grant numbers 107-2634-F-002-015, 110-2634-F-002-032, 110-2314-B-002-161.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by Research Ethics Committee of National Taiwan University Hospital (201708071RINC, 6 October 2017).

**Informed Consent Statement:** Informed consent was waived by the committee.

**Data Availability Statement:** The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.

**Conflicts of Interest:** The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
## Appendix A

<table><thead><tr><th>DICE</th><th>v_net_dropout0.1</th><th>Deconvnet</th><th>u_net</th><th>Pspnet</th><th>Deepmedic</th></tr></thead><tbody><tr><td>Metastasis</td><td>0.69</td><td>0.55</td><td>0.45</td><td>0.52</td><td>0.5</td></tr><tr><td>Meningioma</td><td>0.57</td><td>0.56</td><td>0.48</td><td>0.46</td><td>0.47</td></tr><tr><td>Schwannoma</td><td>0.73</td><td>0.66</td><td>0.72</td><td>0.55</td><td>0.63</td></tr><tr><td>Pituitary</td><td>0.27</td><td>0.38</td><td>0.07</td><td>0.24</td><td>0.29</td></tr><tr><td>AVM</td><td>0.27</td><td>0.31</td><td>0.22</td><td>0.24</td><td>0.2</td></tr><tr><td>Other tumors</td><td>0.34</td><td>0.25</td><td>0.26</td><td>0.28</td><td>0.29</td></tr><tr><td>Total</td><td>0.59</td><td>0.52</td><td>0.44</td><td>0.46</td><td>0.46</td></tr></tbody></table>

## Appendix B

<table><thead><tr><th>SENSITIVITY</th><th>v_net_dropout0.1</th><th>Deconvnet</th><th>u_net</th><th>Pspnet</th><th>Deepmedic</th></tr></thead><tbody><tr><td>Metastasis</td><td>0.82</td><td>0.55</td><td>0.37</td><td>0.56</td><td>0.5</td></tr><tr><td>Meningioma</td><td>0.71</td><td>0.58</td><td>0.41</td><td>0.54</td><td>0.57</td></tr><tr><td>Schwannoma</td><td>0.91</td><td>0.86</td><td>0.75</td><td>0.87</td><td>0.92</td></tr><tr><td>Pituitary</td><td>0.48</td><td>0.37</td><td>0.04</td><td>0.31</td><td>0.29</td></tr><tr><td>AVM</td><td>0.36</td><td>0.33</td><td>0.2</td><td>0.24</td><td>0.22</td></tr><tr><td>Other tumors</td><td>0.58</td><td>0.3</td><td>0.22</td><td>0.33</td><td>0.42</td></tr><tr><td>Total</td><td>0.74</td><td>0.56</td><td>0.39</td><td>0.55</td><td>0.54</td></tr></tbody></table>

## Appendix C

<table><thead><tr><th>PRECISION</th><th>v_net_dropout0.1</th><th>Deconvnet</th><th>u_net</th><th>Pspnet</th><th>Deepmedic</th></tr></thead><tbody><tr><td>Metastasis</td><td>0.63</td><td>0.62</td><td>0.78</td><td>0.6</td><td>0.64</td></tr><tr><td>Meningioma</td><td>0.51</td><td>0.6</td><td>0.67</td><td>0.54</td><td>0.53</td></tr><tr><td>Schwannoma</td><td>0.62</td><td>0.59</td><td>0.76</td><td>0.45</td><td>0.5</td></tr><tr><td>Pituitary</td><td>0.21</td><td>0.5</td><td>0.46</td><td>0.27</td><td>0.42</td></tr><tr><td>AVM</td><td>0.34</td><td>0.49</td><td>0.57</td><td>0.51</td><td>0.34</td></tr><tr><td>Other tumors</td><td>0.34</td><td>0.3</td><td>0.41</td><td>0.38</td><td>0.42</td></tr><tr><td>Total</td><td>0.53</td><td>0.56</td><td>0.68</td><td>0.52</td><td>0.54</td></tr></tbody></table>

## References

1. Adler, J.R., Jr.; Colombo, F.; Heilbrun, M.P.; Winston, K. Toward an expanded view of radiosurgery. *Neurosurgery* **2004**, *55*, 1374–1376. [CrossRef] [PubMed]

2. Chao, S.T.; Dad, L.K.; Dawson, L.A.; Desai, N.B.; Pacella, M.; Rengan, R.; Xiao, Y.; Yenice, K.M.; Rosenthal, S.A.; Hartford, A. ACR-ASTRO Practice Parameter for the Performance of Stereotactic Body Radiation Therapy. *Am. J. Clin. Oncol.* **2020**, *43*, 545–552. [CrossRef] [PubMed]

3. Schell, M.C.; Bova, F.J.; Larson, D.A.; Leavitt, D.D.; Latz, W.R.; Podgorsak, E.B.; Wu, A. *Stereotactic Radiosurgery*; AAPM Report NO. 54; American Association of Physicists in Medicine: Alexandria, VA, USA, 1995.

4. Seung, S.K.; Larson, D.A.; Galvin, J.M.; Mehta, M.P.; Potters, L.; Schultz, C.J.; Yajnik, S.V.; Hartford, A.C.; Rosenthal, S.A. American College of Radiology (ACR) and American Society for Radiation Oncology (ASTRO) Practice Guideline for the Performance of Stereotactic Radiosurgery (SRS). *Am. J. Clin. Oncol.* **2013**, *36*, 310–315. [CrossRef] [PubMed]

5. Shin, H.-C. Hybrid clustering and logistic regression for multi-modal brain tumor segmentation. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012.

6. Bauer, S.; Fejes, T.; Slotboom, J.; Wiest, R.; Nolte, L.-P.; Reyes, M. Segmentation of brain tumor images based on integrated hierarchical classification and regularization. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012.

7. Zhao, L.; Wu, W.; Corso, J.J. Brain tumor segmentation based on GMM and active contour method with a model-aware edge map. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012; pp. 19–23.

8. Xiao, Y.; Hu, J. Hierarchical random walker for multimodal brain tumor segmentation. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012.

9. Subbanna, N.; Arbel, T. Probabilistic gabor and markov random fields segmentation of brain tumours in mri volumes. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012; pp. 28–31.

10. Zikic, D.; Glocker, B.; Konukoglu, E.; Shotton, J.; Criminisi, A.; Ye, D.; Demiralp, C.; Thomas, O.M.; Das, T.; Jena, R. Context-sensitive classification forests for segmentation of brain tumor tissues. In Proceedings of the MICCAI-BRATS 2012, Nice, France, 1 October 2012; pp. 22–30.
11. Lu, S.-L.; Xiao, F.-R.; Cheng, J.C.-H.; Yang, W.-C.; Cheng, Y.-H.; Chang, Y.-C.; Lin, J.-Y.; Liang, C.-H.; Lu, J.-T.; Chen, Y.-F.; et al. Randomized multi-reader evaluation of automated detection and segmentation of brain tumors in stereotactic radiosurgery with deep neural networks. *Neuro-Oncology* **2021**, *23*, 1560–1568. [CrossRef]

12. Havaei, M.; Davy, A.; Warde-Farley, D.; Biard, A.; Courville, A.; Bengio, Y.; Pal, C.; Jodoin, P.-M.; Larochelle, H. Brain tumor segmentation with Deep Neural Networks. *Med. Image Anal.* **2017**, *35*, 18–31. [CrossRef]

13. Kamnitsas, K.; Ledig, C.; Newcombe, V.F.J.; Simpson, J.P.; Kane, A.D.; Menon, D.K.; Rueckert, D.; Glocker, B. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. *Med. Image Anal.* **2017**, *36*, 61–78. [CrossRef] [PubMed]

14. Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention*; Springer: Cham, Switzerland, 2015; pp. 234–241. [CrossRef]

15. Dong, H.; Yang, G.; Liu, F.; Mo, Y.; Guo, Y. Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks. In *Communications in Computer and Information Science*; Springer: Cham, Switzerland, 2017; pp. 506–517.

16. Livne, M.; Rieger, J.; Aydin, O.U.; Taha, A.A.; Akay, E.M.; Kossen, T.; Sobesky, J.; Kelleher, J.D.; Hildebrand, K.; Frey, D.; et al. A U-Net Deep Learning Framework for High Performance Vessel Segmentation in Patients With Cerebrovascular Disease. *Front. Neurosci.* **2019**, *13*, 97. [CrossRef]

17. Bakas, S.; Akbari, H.; Sotiras, A.; Bilello, M.; Rozycki, M.; Kirby, J.; Freymann, J.B.; Farahani, K.; Davatzikos, C. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. *Sci. Data* **2017**, *4*, 170117. [CrossRef]

18. Kamnitsas, K.; Bai, W.; Ferrante, E.; McDonagh, S.; Sinclair, M.; Pawlowski, N.; Rajchl, M.; Lee, M.; Kainz, B.; Rueckert, D.; et al. Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation. In *Machine Learning and Knowledge Discovery in Databases*; Springer: Cham, Switzerland, 2017; pp. 450–462.

19. Shelhamer, E.; Long, J.; Darrell, T. Fully convolutional models for semantic segmentation. *IEEE Trans. Pattern Anal. Mach. Intell.* **2016**, *39*, 640–651. [CrossRef]

20. Miletari, F.; Navab, N.; Ahmadi, S.-A. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In *Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV)*, Stanford, CA, USA, 25–28 October 2016; pp. 565–571. [CrossRef]

21. Militello, C.; Rundo, L.; Vitabile, S.; Russo, G.; Pisciotta, P.; Marletta, F.; Ippolito, M.; D'Arrigo, C.; Midiri, M.; Gilardi, M.C. Gamma Knife treatment planning: MR brain tumor segmentation and volume measurement based on unsupervised Fuzzy C-Means clustering. *Int. J. Imaging Syst. Technol.* **2015**, *25*, 213–225. [CrossRef]

22. Hamamci, A.; Kucuk, N.; Karaman, K.; Engin, K.; Unal, G. Tumor-Cut: Segmentation of Brain Tumors on Contrast Enhanced MR Images for Radiosurgery Applications. *IEEE Trans. Med. Imaging* **2011**, *31*, 790–804. [CrossRef]

23. Hu, M.; Zhong, Y.; Xie, S.; Lv, H.; Lv, Z. Fuzzy System Based Medical Image Processing for Brain Disease Prediction. *Front. Neurosci.* **2021**, *15*, 965. [CrossRef]

24. Rundo, L.; Militello, C.; Russo, G.; Vitabile, S.; Gilardi, M.C.; Mauri, G. GTV cut for neuro-radiosurgery treatment planning: An MRI brain cancer seeded image segmentation method based on a cellular automata model. *Nat. Comput.* **2018**, *17*, 521–536. [CrossRef]

25. Wu, X.; Bi, L.; Fulham, M.; Feng, D.D.; Zhou, L.; Kim, J. Unsupervised brain tumor segmentation using a symmetric-driven adversarial network. *Neurocomputing* **2021**, *455*, 242–254. [CrossRef]

26. Rundo, L.; Militello, C.; Tangherloni, A.; Russo, G.; Vitabile, S.; Gilardi, M.C.; Mauri, G. NeXt for neuro-radiosurgery: A fully automatic approach for necrosis extraction in brain tumor MRI using an unsupervised machine learning technique. *Int. J. Imaging Syst. Technol.* **2018**, *28*, 21–37. [CrossRef]

27. Liu, Y.; Stojadinovic, S.; Hrycushko, B.; Wardak, Z.; Lau, S.; Lu, W.; Yan, Y.; Jiang, S.B.; Zhen, X.; Timmerman, R.; et al. A deep convolutional neural network-based automatic delineation strategy for multiple brain metastases stereotactic radiosurgery. *PLoS ONE* **2017**, *12*, e0185844. [CrossRef]

28. Lu, S.; Hu, S.; Weng, W.; Chen, Y.; Lu, J.; Xiao, F.; Hsu, F. Automated Detection and Segmentation of Brain Metastases in Stereotactic Radiosurgery Using Three-Dimensional Deep Neural Networks. *Int. J. Radiat. Oncol.* **2019**, *105*, S69–S70. [CrossRef]

29. Fong, A.; Swift, C.; Wong, J.; McVicar, N.; Giambattista, J.; Kolbeck, C.; Nichol, A. Automatic Deep Learning-based Segmentation of Brain Metastasis on MPRAGE MR Images for Stereotactic Radiotherapy Planning. *Int. J. Radiat. Oncol.* **2019**, *105*, E134. [CrossRef]

30. Sachdeva, J.; Kumar, V.; Gupta, I.; Khandelwal, N.; Ahuja, C.K. Segmentation, Feature Extraction, and Multiclass Brain Tumor Classification. *J. Digit. Imaging* **2013**, *26*, 1141–1150. [CrossRef]

31. Gros, C.; Lemay, A.; Cohen-Adad, J. SoftSeg: Advantages of soft versus binary training for image segmentation. *Med. Image Anal.* **2021**, *71*, 102038. [CrossRef]

32. Wong, J.; Huang, V.; Wells, D.; Giambattista, J.; Giambattista, J.; Kolbeck, C.; Otto, K.; Saibishkumar, E.P.; Alexander, A. Implementation of deep learning-based auto-segmentation for radiotherapy planning structures: A workflow study at two cancer centers. *Radiat. Oncol.* **2021**, *16*, 101. [CrossRef] [PubMed]

33. Shattuck, D.W.; Leahy, R.M. BrainSuite: An automated cortical surface identification tool. *Med. Image Anal.* **2002**, *6*, 129–142. [CrossRef]
34. Wu, S.-R.; Wu, P.Y.; Chang, H.Y. Brain-Tumor-Segmentation/Models at Master • raywu0123/Brain-Tumor-Segmentation. Available online: https://github.com/raywu0123/Brain-Tumor-Segmentation/tree/master/models (accessed on 25 September 2021).

35. Noh, H.; Hong, S.; Han, B. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1520–1528.

36. Pennig, L.; Shahzad, R.; Caldeira, L.; Lennartz, S.; Thiele, F.; Goertz, L.; Zopfs, D.; Meißner, A.-K.; Fürtjes, G.; Perkuhn, M.; et al. Automated Detection and Segmentation of Brain Metastases in Malignant Melanoma: Evaluation of a Dedicated Deep Learning Model. *Am. J. Neuroradiol.* **2021**, *42*, 655–662. [CrossRef] [PubMed]

37. Jünger, S.T.; Hoyer, U.C.I.; Schaufler, D.; Laukamp, K.R.; Goertz, L.; Thiele, F.; Grunz, J.; Schlamann, M.; Perkuhn, M.; Kabbasch, C.; et al. Fully Automated MR Detection and Segmentation of Brain Metastases in Non-small Cell Lung Cancer Using Deep Learning. *J. Magn. Reson. Imaging* **2021**. [CrossRef]

38. Charron, O.; Lallement, A.; Jarnet, D.; Noblet, V.; Clavier, J.-B.; Meyer, P. Automatic detection and segmentation of brain metastases on multimodal MR images with a deep convolutional neural network. *Comput. Biol. Med.* **2018**, *95*, 43–54. [CrossRef] [PubMed]

39. Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid Scene Parsing Network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2881–2890.

40. Wu, S.R.; Wu, P.Y.; Chang, H.Y. Brain-Tumor-Segmentation/Models/Batch_Samplers at Master • raywu0123/Brain-Tumor-Segmentation. Available online: https://github.com/raywu0123/Brain-Tumor-Segmentation/tree/master/models/batch_s samplers (accessed on 25 September 2021).

41. Neugut, A.I.; Sackstein, P.; Hillyer, G.C.; Jacobson, J.S.; Bruce, J.; Lassman, A.B.; Stieg, P.A. Magnetic Resonance Imaging-Based Screening for Asymptomatic Brain Tumors: A Review. *Oncologist* **2019**, *24*, 375–384. [CrossRef]

42. Nakasu, S.; Notsu, A.; Nakasu, Y. Prevalence of incidental meningiomas and gliomas on MRI: A meta-analysis and meta-regression analysis. *Acta Neurochir.** *2021*, *1–15*. [CrossRef]

43. Andermatt, S.; Pezold, S.; Cattin, P. Multi-dimensional gated recurrent units for brain tumor segmentation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017; pp. 15–19.

44. Amorim, P.H.A.; Chagas, V.S.; Escudero, G.G.; Oliveira, D.D.C.; Pereira, S.M.; Santos, H.M.; Scussel, A.A. 3D u-nets for brain tumor segmentation in miccai 2017 brats challenge. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017.

45. Castillo, L.S.; Daza, L.A.; Rivera, L.C.; Arbeláez, P. Volumetric multimodality neural network for brain tumor segmentation. In Proceedings of the 13th International Conference on Medical Information Processing and Analysis, San Andres Island, Colombia, 5–7 October 2017; p. 105720E.

46. Feng, X.; Meyer, C. Patch-based 3d u-net for brain tumor segmentation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017.

47. Zhou, C.; Ding, C.; Lu, Z.; Zhang, T. Brain tumor segmentation with cascaded convolutional neural networks. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017; pp. 328–333.

48. Isensee, F.; Kickingereder, P.; Wick, W.; Bendszus, M.; Maier-Hein, K.H. Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge. In *International MICCAI Brainlesion Workshop*; Springer: Cham, Switzerland, 2018; pp. 287–297. [CrossRef]

49. Li, Y.; Shen, L. MnNet: Multi-view deep learning framework for multimodal brain tumor segmentation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017.

50. Pourreza, R.; Zhuge, Y.; Ning, H.; Miller, R. Brain Tumor Segmentation in MRI Scans Using Deeply-Supervised Neural Networks. In *International MICCAI Brainlesion Workshop*; Springer: Cham, Switzerland, 2018; pp. 320–331.

51. Zhou, F.; Li, T.; Li, H.; Yu, K.; Wang, Y.; Zhu, H. TP-CNN: A two-phase convolution neural network based model to do automatic brain tumor segmentation by using BRATS 2017 data. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017; pp. 334–341.

52. Zhu, J.; Wang, D.; Teng, Z.; Lio, P. A multi-pathway 3d dilated convolutional neural network for brain tumor segmentation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017; pp. 342–347.

53. Hu, Y.; Xia, Y. Automated brain tumor segmentation using a 3D deep detection-classification model. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017.

54. Chen, S.; Ding, C.; Zhou, C. Brain tumor segmentation with label distribution learning and multi-level feature representation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017.

55. Beers, A.; Chang, K.; Brown, J.; Sartor, E.; Mammen, C.; Gerstner, E.; Rosen, B.; Kalpathy-Cramer, J. Sequential 3d u-nets for brain tumor segmentation. In Proceedings of the International MICCAI BraTS Challenge 2017, Quebec City, QC, Canada, 14 September 2017; pp. 20–23.

56. Yang, Q.; Chao, H.; Nguyen, D.; Jiang, S. A Novel Deep Learning Framework for Standardizing the Label of OARs in CT. In *Workshop on Artificial Intelligence in Radiation Therapy*; Springer: Cham, Switzerland, 2019; pp. 52–60.

57. Yang, Q.; Chao, H.; Nguyen, D.; Jiang, S. Mining Domain Knowledge: Improved Framework Towards Automatically Standardizing Anatomical Structure Nomenclature in Radiotherapy. *IEEE Access* **2020**, *8*, 105286–105300. [CrossRef]

58. Brant-Zawadzki, M.; Gillan, G.D.; Nitz, W.R. MP RAGE: A three-dimensional, T1-weighted, gradient-echo sequence—Initial experience in the brain. *Radiology* **1992**, *182*, 769–775. [CrossRef]
59. Park, Y.W.; Jun, Y.; Lee, Y.; Han, K.; An, C.; Ahn, S.S.; Hwang, D.; Lee, S.-K. Robust performance of deep learning for automatic detection and segmentation of brain metastases using three-dimensional black-blood and three-dimensional gradient echo imaging. *Eur. Radiol.* **2021**, *31*, 6686–6695. [CrossRef]

60. Zhou, Z.; Sanders, J.W.; Johnson, J.M.; Gule-Monroe, M.; Chen, M.; Briere, T.M.; Wang, Y.; Son, J.B.; Pagel, M.D.; Ma, J.; et al. MetNet: Computer-aided segmentation of brain metastases in post-contrast T1-weighted magnetic resonance imaging. *Radiother. Oncol.* **2020**, *153*, 189–196. [CrossRef] [PubMed]

61. Grøvik, E.; Yi, D.; Iv, M.; Tong, E.; Rubin, D.; Zaharchuk, G. Deep learning enables automatic detection and segmentation of brain metastases on multisequence MRI. *J. Magn. Reson. Imaging* **2020**, *51*, 175–182. [CrossRef] [PubMed]

62. Xue, J.; Wang, B.; Ming, Y.; Liu, X.; Jiang, Z.; Wang, C.; Liu, X.; Chen, L.; Qu, J.; Xu, S.; et al. Deep learning-based detection and segmentation-assisted management of brain metastases. *Neuro-Oncology* **2020**, *22*, 505–514. [CrossRef]

63. Hansen, E.K.; Roach, M., III. *Handbook of Evidence-Based Radiation Oncology*; Springer: Berlin/Heidelberg, Germany, 2018.

64. Wang, X.; Cui, H.; Gong, G.; Fu, Z.; Zhou, J.; Gu, J.; Yin, Y.; Feng, D. Computational delineation and quantitative heterogeneity analysis of lung tumor on 18F-FDG PET for radiation dose-escalation. *Sci. Rep.* **2018**, *8*, 10649. [CrossRef]

65. Rundo, L.; Stefano, A.; Militello, C.; Russo, G.; Sabini, M.G.; D'Arrigo, C.; Marletta, F.; Ippolito, M.; Mauri, G.; Vitabile, S.; et al. A fully automatic approach for multimodal PET and MR image segmentation in gamma knife treatment planning. *Comput. Methods Programs Biomed.* **2017**, *144*, 77–96. [CrossRef] [PubMed]
<figure><img src="image_98.png" /></figure>

Article

# Multichannel Multiscale Two-Stage Convolutional Neural Network for the Detection and Localization of Myocardial Infarction Using Vectorcardiogram Signal

Jay Karhade <fnref n="1" />, Samit Kumar Ghosh <fnref n="1" />, Pranjali Gajbhiye <fnref n="1" />, Rajesh Kumar Tripathy <fnref n="1" /> and U. Rajendra Acharya <fnref n="2" />,<fnref n="3" />,<fnref n="4" />,*

**Abstract:** Myocardial infarction (MI) occurs due to the decrease in the blood flow into one part of the heart, and it further causes damage to the heart muscle. The 12-channel electrocardiogram (ECG) has been widely used to detect and localize MI pathology in clinical studies. The vectorcardiogram (VCG) is a 3-channel recording system used to measure the heart's electrical activity in sagittal, transverse, and frontal planes. The VCG signals have advantages over the 12-channel ECG to localize posterior MI pathology. Detection and localization of MI using VCG signals are vital in clinical practice. This paper proposes a multi-channel multi-scale two-stage deep-learning-based approach to detect and localize MI using VCG signals. In the first stage, the multivariate variational mode decomposition (MVMD) decomposes the three-channel-based VCG signal beat into five components along each channel. The multi-channel multi-scale VCG tensor is formulated using the modes of each channel of VCG data, and it is used as the input to the deep convolutional neural network (CNN) to classify MI and normal sinus rhythm (NSR) classes. In the second stage, the multi-class deep CNN is used for the categorization of anterior MI (AMI), anterior-lateral MI (ALMI), anterior-septal MI (ASMI), inferior MI (IMI), inferior-lateral MI (ILMI), inferior-posterior-lateral (IPLMI) classes using MI detected multi-channel multi-scale VCG instances from the first stage. The proposed approach is developed using the VCG data obtained from a public database. The results reveal that the approach has obtained the accuracy, sensitivity, and specificity values of 99.58%, 99.18%, and 99.87%, respectively, for MI detection. Moreover, for MI localization, we have obtained the overall accuracy value of 99.86% in the second stage for our proposed network. The proposed approach has demonstrated superior classification performance compared to the existing VCG signal-based MI detection and localization techniques.

**Keywords:** myocardial infarction; vectorcardiogram; multivariate VMD; deep CNN; accuracy

**Citation:** Karhade, J.; Ghosh, S.K.; Gajbhiye, P.; Tripathy, R.K.; Acharya, U.R. Multichannel Multiscale Two-Stage Convolutional Neural Network for the Detection and Localization of Myocardial Infarction Using Vectorcardiogram Signal. *Appl. Sci.* **2021**, *11*, 7920. https://doi.org/10.3390/app11177920

Academic Editors: Leonardo Rundo, Carmelo Militello and Andrea Tangherloni

Received: 26 July 2021

Accepted: 25 August 2021

Published: 27 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_99.png" /></figure>

Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

## 1. Introduction

The obstruction in one of the coronary arteries of the heart causes the myocardial infarction (MI) disease [1,2]. Typically, the MI is progressed in three phases [3]. These three phases are (a) ischemic phase, (b) acute phase, and (c) myocardial necrosis phase. The 12-lead ECG signal is used in the clinical study for the early detection and localization of MI pathology [4]. The ST-segment elevations, inverted T-waves, and pathological Q-waves are the morphological changes observed in the ECG signals of different leads in MI pathology [5]. The morphological changes in the ECG signals of the channels or leads, such as V1, V2, V3, and V4, are used to diagnose anterior MI (AMI) [6]. Similarly, inferior MI is diagnosed based on the variations in the morphologies of ECG signals for II, III,

<footnote n="1">¹ Department of Electrical and Electronics Engineering, BITS-Pilani, Hyderabad Campus, Hyderabad 500078, India; f20180852@hyderabad.bits-pilani.ac.in (J.K.); samitnitrkl@gmail.com (S.K.G.); gajbhiyepranjali@gmail.com (P.G.); rajeshiitg13@gmail.com (R.K.T.)</footnote>

<footnote n="2">² School of Engineering, Ngee Ann Polytechnic, Singapore 599489, Singapore</footnote>

<footnote n="3">³ Department of Bioinformatics and Medical Engineering, Asia University, Taichung 41354, Taiwan</footnote>

<footnote n="4">⁴ School of Management and Enterprise, University of Southern Queensland, Springfield, Ipswich, QLD 4300, Australia</footnote>

<footnote>* Correspondence: Rajendra_Udyavara_ACHARYA@np.edu.sg</footnote>
and aVF channels. Moreover, the morphological variations in the ECG signals of I, aVL, V5, and V6 channels are used to diagnose left lateral MI pathology [6]. In 12-lead ECG, no ECG lead capture the information about the diagnosis of posterior MI [6,7]. However, the reciprocal changes in the V1 and V2 channel ECG signals are used in the clinical study to diagnose posterior MI [8]. Vectorcardiogram (VCG) is an orthogonal three lead system which measures the heart's electrical activity along transverse, sagittal, and frontal planes, and it has been used for the detection of MI pathology [3,9]. The 12-lead ECG can be derived from the VCG signal using various transformation techniques [10]. In VCG, one of the orthogonal leads reveals the posterior view of the heart [11]. Hence, the method based on the analysis of VCG signal information is helpful to detect and localize MI pathology. The continuous recording and monitoring of VCG signal information for MI disease diagnosis is cumbersome, and hence automated approaches have been used for the accurate detection and localization of MI using VCG signals [3]. The development of novel approaches to detect and localize MI pathology using the VCG signals is challenging in clinical study.

In recent years, various approaches have been developed to detect MI using VCG signals [12–15]. The methods based on the evaluation of various VCG signal morphological features, such as difference in ST-T vector magnitude, area of ST-segment vector, and other T-wave features, have been used to detect MI disease [14,16–18]. Similarly, in [13], authors have applied independent component analysis (ICA) and principal component analysis (PCA) for projecting VCG signal feature vector into a lower-dimensional space. They have extracted various morphological features from the VCG signal to formulate the feature vector. The neural network-based classifier has been used for the detection of MI using reduced dimension feature vector of VCG signal [13]. In [14], authors have computed octant and vector-based features from VCG signals and used a decision tree model to detect MI pathology. These methods require the detection of P, Q, R, S, T-onset points manually in the VCG signal to compute the morphological features [3]. In literature, various wavelet-based techniques, such as multi-scale recurrent quantification analysis (MRQA) [15], and complex wavelet sub-band features [3] have been used to detect MI using VCG signals. In [15], each channel of the VCG signal is decomposed into sub-band signals using discrete wavelet transform (DWT). From each sub-band signal, the recurrent quantification analysis (RQA) based non-linear features have been extracted, and Gaussian discriminant analysis (GDA) classifier is used for the detection of MI [15]. Moreover, in [3], the dual-tree complex wavelet transform (DT-CWT) has been used to decompose the VCG signal into sub-band signals along each channel. The entropy and L1-norm features have been extracted from each sub-band signal. The relevance vector machine (RVM) classifier has been used to detect MI from these VCG signals features [3]. Along with cardiac signal processing, cardiac imaging today represents an important area of clinical research that has achieved excellent results in recent years, such as deep-learning approaches [19]; especially, this led to the development of computer-assisted tools capable of segmenting the whole heart [20,21], as well as identifying specific regions of interest [22]. In the wavelet-based approach, the pre-defined basis functions and the number of decomposition levels are used to compute sub-band signals from VCG signal [23]. Additionally, the mentioned VCG signal-based approaches have considered only for MI detection. The automated classification of various types of MI pathologies has not been considered using VCG signals. The existing VCG-based approaches have considered various feature extraction and machine learning methods to detect MI. In recent years, various deep learning-based approaches have been used to detect and localize MI using 12-lead ECG signals [2, 24, 25]. The deep learning-based MI detection and localization methods do not require extracting features from 12-lead ECG signals [2]. The deep learning-based methods have not been explored for the detection and localization of MI using VCG signals. Therefore, a deep learning-based approach can be developed to detect and localize MI using VCG signals.

The multivariate variational mode decomposition (MVMD) is a recently proposed signal processing technique to decompose the multi-channel signals into components or
modes [26]. This method is fully signal-driven and does not consider any basis functions and decomposition levels like DWT to obtain components of non-stationary signals. The univariate version of VMD has been used for the analysis of ECG signals for the detection of ventricular tachycardia and atrial fibrillation episodes [27,28]. The VCG is a multi-channel signal, and, therefore, the MVMD can be used to decompose the signal into modes. Moreover, deep learning-based methods have been used in the multi-scale or modal domain of ECG signals to detect cardiac ailments [29]. For VCG signal, the deep learning method has not been explored in the multi-scale domain to detect and localization of MI. The novelty of this work is to develop a multi-channel multi-scale deep learning-based framework to detect and localize MI using VCG signals. The important contributions of this work are given as follows:

1. The MVMD is introduced to decompose the VCG signals into sub-band signals or modes;

2. A multi-channel multi-scale two-stage deep convolutional neural network (CNN) framework is proposed for the detection and localization of MI;

3. The MI types, such as AMI, IMI, ILMI, ALMI, ASMI, and IPLMI, are classified in the second stage of the proposed multi-channel multi-scale deep CNN (MMD-CNN) model;

4. The multi-channel multi-scale two-stage deep CNN performance is evaluated using hold-out and 10-fold cross-validation (CV) schemes.

The remaining sections of this paper are written as follows. The explanation regarding the VCG signal database is written in Section 2. In Section 3, the proposed approach for MI detection and localization is described. Section 4 presents the results and discussion of the proposed approach. In Section 5, conclusions of this paper is summarized.

## 2. VCG Signal Database

In this work, the VCG signals from the PTB diagnostic database (https://www.physionet.org/content/ptbdb/1.0.0/) (accessed on 20 June 2021), were used to develop the proposed multi-channel multi-scale two-stage deep CNN approach [30,31]. The PTB database from Physionet comprises both 12-lead ECG and 3-lead VCG recordings of normal sinus rhythm (NSR) and various heart diseases, such as MI, hypertrophy, cardiomyopathy, bundle branch block, and dysrhythmia, respectively [30]. Each VCG signal has been sampled at 1000 samples per second in the PTB database, and the amplitude value of each lead VCG varied between -16.384 mV to 16.384 mV. In this study, we have used 73 VCG recordings from 52 healthy controls (HC) subjects of PTB diagnostic database. Similarly, 99 VCG recordings from 148 subjects with MI pathology are used. For MI localization, 13, 20, 11, 21, 21, and 13 VCG recordings from AMI, IMI, ALMI, ASMI, ILMI, and IPLMI classes, respectively, are considered. In the PTB diagnostic database [30,31], the number of VCG recordings for MI class is higher than the healthy class. A higher difference in the number of VCG instances between MI and healthy classes may cause the over-fitting problem during the training of the proposed MMDCNN model. Due to this reason, we have considered only 99 VCG recordings from the MI class in this work. Each VCG recording in the PTB diagnostic database contains three orthogonal leads (Vx, Vy, Vz), which represent the electrical activity of heart in three different planes [28].

## 3. Method

The proposed MI detection and localization stages are shown in a flow-chart form in Figure 1a,b, respectively. The MI detection stage comprises the filtering of VCG signal, segmentation of VCG recordings into beats, decomposition of VCG beat into multi-scale VCG tensors using MVMD, and deep CNN to detect MI pathology. Similarly, the localization stage consists of the classification of AMI, IMI, ALMI, ASMI, ILMI, and IPLMI beats using MI detected multi-scale VCG tensor data. The following section briefly discuss each part of the flow-chart, as shown in Figure 1.
<figure><img src="image_100.png" /><figcaption>**Figure 1.** (a) Stage1: MI detection block using VCG signals. (b) Stage 2: MI localization using MI detected multi-scale VCG tensor.</figcaption></figure>

### 3.1. Segmentation of VCG Data

In this work, we have performed the amplitude normalization for each lead of VCG recording [28]. The samples of raw VCG signal corresponding to each lead is divided by the maximum absolute value the signal to obtain the normalized VCG signal [6]. After normalization of each lead VCG recording, we have detected the R-peak in the $V_x$ lead of VCG signals. The beat by beat segmentation of each VCG recording is performed using a window of size 651 samples [2]. The 251 samples before each R-peak of the $V_x$ lead VCG signal and 400 samples after R-peak are considered for the beat by beat segmentation of VCG signal [2]. The number of MI and NSR VCG beats used for the proposed MI detection work are shown in Table 1. Similarly, the number of VCG beats evaluated for each type of MI are also shown in Table 1.

<table><caption>**Table 1.** Number of VCG beats used for MI detection and localization.</caption><thead><tr><th colspan="7">Stage 1: MI detection</th></tr></thead><tbody><tr><td>Class</td><td colspan="2">NSR</td><td colspan="4">MI</td></tr><tr><td>VCG beats</td><td colspan="2">9874</td><td colspan="4">13982</td></tr></tbody><thead><tr><th colspan="7">Stage 2: MI localization</th></tr></thead><tbody><tr><td>Class</td><td>AMI</td><td>ALMI</td><td>ASMI</td><td>IMI</td><td>ILMI</td><td>IPLMI</td></tr><tr><td>VCG beats</td><td>1664</td><td>1778</td><td>3057</td><td>2807</td><td>3049</td><td>1981</td></tr></tbody></table>

### 3.2. Multivariate VMD for VCG Signal Analysis

In this study, we have used MVMD to evaluate the modes of VCG beat along each orthogonal lead. The MVMD is the extension of VMD algorithm used to decompose multi-channel signals into modes [26]. The VCG beat is given as $v_m(n)$, with $n = 1, 2, ..., N$. N is the number of samples in the VCG beat. The parameter $m$ is denoted as the $m$th orthogonal lead of VCG beat. The VCG beat synthesized from its modes is given as follows:

$$v_m(n) = \sum_{i=1}^{k} u_m^i(n) \qquad (1)$$
where, $u_m^i(n)$ is the ith mode of mth lead VCG signal $u_m^i(n) = [u_1^i(n), u_2^i(n), u_3^i(n)]$ is also interpreted as the multivariate modulated oscillations of VCG signal with $i = 1, 2, \dots, k$, and $k$ is the total number of modes [26]. The vector analytic representation of ith mode of mth lead VCG is written as follows [26]:

$$ \tilde{u}_m^i(n) = u_m^i(n) + jH(u_m^i(n)) \quad (2) $$

where, $H(u_m^i(n))$ is the Hilbert transform of ith mode of mth lead VCG signal [26]. In MVMD, the objective is to evaluate the modes of VCG signal based on the criteria as (a) the sum of bandwidth of components or modes of VCG should be minimum and (b) sum of all modes should recover the VCG signal along each lead [26]. The optimization problem of MVMD for the decomposition of VCG signal is formulated as follows [26]:

$$ \begin{array}{l} \min_{u_m^i(n), w^i} \left\{ \sum_{i=1}^{k} \sum_{m=1}^{M} \left\| \frac{\partial}{\partial n} \left[ \tilde{u}_m^i(n) e^{-jw^i n} \right] \right\|_F^2 \right\} \\ \text{s.t.} \quad \sum_{i=1}^{k} u_m^i(n) = v_m(n), \quad m = 1, 2, \text{ and } 3 \end{array} \quad (3) $$

where, $\|\cdot\|_F$ is the representation of Frobenious norm [26]. The optimization problem in Equation (3) can be reformulated using augmented Lagrangian and it is given as follows:

$$ L\{u_m^i(n), w^i, \eta_m(n)\} = \beta \sum_{i=1}^{k} \sum_{m=1}^{M} \left\| \frac{\partial}{\partial n} \left[ u_m^i(n) e^{-jw^i n} \right] \right\|_F^2 + \sum_{m=1}^{M} \left\| v_m(n) - \sum_{i=1}^{k} u_m^i(n) \right\|_F^2 + \sum_{m=1}^{M} \left\langle \eta_m(n), v_m(n) - \sum_{i=1}^{k} u_m^i(n) \right\rangle \quad (4) $$

where, $\eta_m(n)$ is the Lagrangian multiplier for mth lead VCG beat, and $\beta$ is interpreted as the penalty factor for MVMD. The modes of VCG beat along each lead is iteratively evaluated based on the solution of Equation (4) using alternating direction method of multipliers (ADMM) [26]. The complete algorithm of MVMD for the extraction of modes from the non-stationary signals has been given in [26]. In this study, we have evaluated five modes from the VCG beat along each orthogonal lead. The multi-scale VCG tensor is formulated using the modes of VCG beat and the size of multi-scale VCG tensor is 651 × 3 × 5.

For NSR class, the $V_x$, $V_y$, and $V_z$ lead VCG beat are shown in Figure 2a,g,m, respectively. The modes of $V_x$, $V_y$, and $V_z$ lead VCG beats evaluated using MVMD are shown Figure 2b–f,h–l,n–r, respectively. Similarly, the $V_x$, $V_y$, and $V_z$ channel VCG beats for IPLMI class are shown in Figure 3a,g,m, respectively. For IPLMI class, the modes of $V_x$, $V_y$, and $V_z$ lead VCG beats are depicted in Figure 3b–f,h–l,n–r, respectively. It can be observed from these plots that the modes of each lead VCG beat have different shape and amplitude values for IPLMI and NSR classes. In VCG signal, the clinical parameters, such as QRS-complex shape, special QRS-T angle, T-wave shape are different for healthy and MI cases [32]. The study in [33] has reported the physiological parameters of VCG signal for MI class, such as QRS-loop maximum vector magnitude, QRS-area perimeter ratio, and ST-vector magnitude, have higher mean values than those of healthy class. Similarly, the VCG parameters, such as QRS-loop volume, QRS-loop planar area, maximum of the distance between QRS-centroid and QRS-loop, and QRS-perimeter have the lowest mean values for MI class as compared to healthy class [33]. For the AMI case, there is abnormal posterior deviation in the QRS-vector of VCG signal [34]. Similarly, for the posterior-lateral MI case, the pathological changes, such as oriented T-loop and maximal leftward deviation of frontal plane QRS-vector are observed [35]. The transverse plane QRS-vector maximum value greater than 1.5 mV is also used as the criteria for the detection of inferior and posterior MI using VCG signals [11]. These differences in the morphological parameters of VCG signal for NSR and various types of MI cases can be captured in the modes which are evaluated using MVMD. Therefore, the deep CNN model designed using the modes of the VCG beat can be used to detect and localize MI.
<figure><img src="image_101.png" /><figcaption>**Figure 2.** (a) $V_x$ lead VCG signal for NSR class. (b–f) mode 1 to 5 of $V_x$ lead VCG signal for NSR class. (g) $V_y$ lead VCG signal for NSR class. (h–l) mode 1 to 5 of $V_y$ lead VCG signal for NSR class. (m) $V_z$ lead VCG signal for NSR class. (n–r) mode 1 to 5 of $V_z$ lead VCG signal for NSR class.</figcaption></figure>

<figure><img src="image_102.png" /><figcaption>**Figure 3.** (a) $V_x$ lead VCG signal for IPLMI class. (b–f) mode 1 to 5 of $V_x$ lead VCG signal for IPLMI class. (g) $V_y$ lead VCG signal for IPLMI class. (h–l) mode 1 to 5 of $V_y$ lead VCG signal for IPLMI class. (m) $V_z$ lead VCG signal for IPLMI class. (n–r) mode 1 to 5 of $V_z$ lead VCG signal for IPLMI class.</figcaption></figure>
### 3.3. Multi-Channel Multi-Scale Deep Convolutional Neural Network

In this work, a novel MMDCNN model is proposed to detect and localize MI. The Python codes for the MMDCNN model is available at (https://github.com/JayKarbade/MI_VCG_DL (accessed on 20 August 2021)). The MMDCNN architecture shown in Figure 4 comprises 12 layers. The first and last layers are interpreted as input and output layers of MMDCNN model. The input layer contains the multi-scale VCG tensor. The output layer consists of two neurons for MI detection stage, one for NSR class and the other for MI class. Similarly, for the MI localization stage, the output layer contains six neurons corresponding to six types of MI classes as AMI, IMI, ALMI, ASMI, ILMI, and IPLMI, respectively. The MMDCNN contains four convolutions, two max-pooling, and four dense layers for both MI detection and localization stages. The mathematical expression to compute the tth feature map for first convolution layer is given as follows [29,36]:

$$ \mathbf{X}_{i}^{(l)}(\tilde{n}) = h\left(\sum_{n=1}^{N} \sum_{m=1}^{M} \sum_{i=1}^{l} \mathbf{X}(n, m, i) \mathbf{K}_{i}(\tilde{n} - n + \frac{N}{2}, m, i) + b_{i}\right) \quad (5) $$

where $\mathbf{X}(n, m, i)$ is the input to the MMDCNN and $i = 1, 2, \dots, I$ and $m = 1, 2, \dots, M$, respectively. The parameters $I$ and $M$ are total number of modes and channels, respectively. Similarly, the mathematical expression for the evaluation of feature maps in other convolution layers are evaluated as follows [29,36]:

$$ \mathbf{X}_{i}^{(l)}(\tilde{n}) = h\left(\sum_{n=1}^{N} \sum_{c=1}^{C} \mathbf{X}_{i}^{(l-1)}(n, c) \tilde{\mathbf{K}}_{i}(\tilde{n} - n + \frac{N}{2}, c) + \tilde{b}_{i}\right) \quad (6) $$

$\mathbf{X}_{i}^{(l-1)}(n, c)$ is the $\tilde{t}$th feature map at $(l-1)$th convolution layer. Similarly, the feature maps for second, third and fourth convolution layers are evaluated using Equation (6). The $\mathbf{X}_{i}^{(l)}$ is denoted as the $\tilde{t}$th feature map for $l$th convolution layer. Moreover, the mathematical expression to evaluate the pooling layer feature map is given as follows [29,36]:

$$ \mathbf{X}_{i}^{(l)}(\tilde{n}) = \max\text{-pooling}(\mathbf{X}_{i}^{(l-1)}(\tilde{n})) \quad (7) $$

For dense layers, the feature vector is evaluated as follows [37]:

$$ \mathbf{a}^{(l)} = h(\mathbf{a}^{(l-1)}\bar{\mathbf{W}}^{(l)} + \bar{\mathbf{b}}^{(l)}) \quad (8) $$

where $\mathbf{a}^{(l)}$ is the feature vector for $l$th dense layer. $\bar{\mathbf{W}}^{(l)}$ is the weight matrix between $(l-1)$th dense and $l$th dense layers [37]. $\bar{\mathbf{b}}^{(l)}$ is the bias for $l$th dense layer. The categorical cross-entropy-based cost function is used for MMDCNN for both detection and localization stages [38]. The hyper-parameters used for MMDCNN in detection and localization stages are shown in Table 2. In this study, for both MI detection and localization stages, the hold-out validation and 10-fold cross-validation (CV) methods [37] are used to select the training and test VCG beats. For hold-out validation 78.75%, 11.25%, and 10% VCG beats are used as training, validation, and testing, respectively, for MMDCNN model during detection and localization phases. We have used the performance measures such as accuracy, sensitivity, specificity, and Kappa scores for the MI detection using MMDCNN classifier [37,39]. Similarly, for MI localization, the overall accuracy (OA), individual accuracy (IA), and Kappa score are used to evaluate the performance in the second stage MMDCNN [6].
<figure><img src="image_103.png" /><figcaption>**Figure 4.** Proposed MMDCNN model to detect and localize MI using VCG beats.</figcaption></figure>

<table><caption>**Table 2.** Hyper-parameters used MI detection and localization using our proposed MMDCNN model.</caption><thead><tr><th rowspan="2">Parameters</th><th rowspan="2">Optimizer</th><th colspan="3">Hold-out (MI Detection)</th><th colspan="3">10-fold (MI Detection)</th></tr><tr><th>Batch size</th><th>Epochs</th><th>Learning rate</th><th>Batch size</th><th>Epochs</th><th>Learning rate</th></tr></thead><tbody><tr><td>Values</td><td>Adam</td><td>1024</td><td>15</td><td>0.0001</td><td>1024</td><td>15</td><td>0.0001</td></tr></tbody></table><table><thead><tr><th rowspan="2">Parameters</th><th rowspan="2">Optimizer</th><th colspan="3">Hold-out (MI Localization)</th><th colspan="3">10-fold (MI Localization)</th></tr><tr><th>Batch size</th><th>Epochs</th><th>Learning rate</th><th>Batch size</th><th>Epochs</th><th>Learning rate</th></tr></thead><tbody><tr><td>Values</td><td>Adam</td><td>1024</td><td>15</td><td>0.00004</td><td>256</td><td>15</td><td>0.0001</td></tr></tbody></table>

## 4. Results and Discussions

The results evaluated using the proposed MMDCNN for MI detection and localization using VCG signals are shown in this section. In Table 3, we have shown the accuracy, sensitivity, specificity, and kappa score values for our proposed MMDCNN model with hold-out CV. Similarly, for MI detection, the accuracy vs. epoch plots for training and validation VCG instances obtained using MMDCNN are illustrated in Figure 5. It is evident from this plot that both training and validation accuracy values are 100% after 10th epoch. Similarly, we have shown the confusion matrix obtained using the proposed MMDCNN for MI detection using VCG signals for one random hold-out trial in Table 4. The number of false-positive and false-negative values are 1 in the confusion matrix table. The accuracy, sensitivity, specificity, and kappa values for this random hold-out validation are 99.9%, 99.89%, 99.92%, and 0.998, respectively. The average values of accuracy, sensitivity, specificity, and kappa scores over five trial-based random validation are more than 99% (as seen from Table 3).