Artificial Intelligence Applied to Medical Imaging and
Computational Biology

Leonardo Rundo, Carmelo Militello, Andrea Tangherloni

► To cite this version:

Leonardo Rundo, Carmelo Militello, Andrea Tangherloni. Artificial Intelligence Applied to Medical Imaging and Computational Biology. MDPI, 2023, 978-3-0365-6488-3. 10.3390/books978-3-0365-6488-3. hal-04086036

HAL Id: hal-04086036

https://hal.science/hal-04086036

Submitted on 26 Jun 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
<figure><img src="image_1.png" /></figure>

applied sciences

Artificial Intelligence
Applied to Medical Imaging
and Computational Biology

Edited by

Leonardo Rundo, Carmelo Militello and Andrea Tangherloni

Printed Edition of the Special Issue Published in *Applied Sciences*

www.mdpi.com/journal/applsci

<figure><img src="image_2.png" /></figure>
Artificial Intelligence Applied to
Medical Imaging and Computational
Biology
<figure><img src="image_3.png" /></figure>
Artificial Intelligence Applied to
Medical Imaging and Computational
Biology

Editors

Leonardo Rundo

Carmelo Militello

Andrea Tangherloni

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

<figure><img src="image_4.png" /></figure>
Editors

Leonardo Rundo
University of Salerno
Italy

Carmelo Militello
National Research Council
(ICAR-CNR)
Italy

Andrea Tangherloni
University of Bergamo
Italy

Editorial Office

MDPI
St. Alban-Anlage 66
4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Applied Sciences* (ISSN 2076-3417) (available at: https://www.mdpi.com/journal/applsci/special-issues/AI_medical_imaging_biology).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

Last Name, A.A.; Last Name, B.B.; Last Name, C.C. Article Title. *Journal Name Year, Volume Number, Page Range*.

ISBN 978-3-0365-6487-6 (Hbk)

ISBN 978-3-0365-6488-3 (PDF)

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.
# Contents

<table>
  <tbody>
    <tr>
      <td>Leonardo Rundo, Andrea Tangherloni and Carmelo Militello<br>Artificial Intelligence Applied to Medical Imaging and Computational Biology<br>Reprinted from: Appl. Sci. 2022, 12, 9052, doi:10.3390/app12189052</td>
      <td>1</td>
    </tr>
    <tr>
      <td>Jianing Zhang, Yanhuan Huang, Fuqiang Ye, Bibo Yang, Zengyong Li and Xiaoling Hu<br>Evaluation of Post-Stroke Impairment in Fine Tactile Sensation by Electroencephalography (EEG)-Based Machine Learning<br>Reprinted from: Appl. Sci. 2022, 12, 4796, doi:10.3390/app12094796</td>
      <td>3</td>
    </tr>
    <tr>
      <td>Cleo-Aron Weis, Kian R. Weihrauch, Katharina Kriegsmann and Mark Kriegsmann<br>Unsupervised Segmentation in NSCLC: How to Map the Output of Unsupervised Segmentation to Meaningful Histological Labels by Linear Combination?<br>Reprinted from: Appl. Sci. 2022, 12, 3718, doi:10.3390/app12083718</td>
      <td>21</td>
    </tr>
    <tr>
      <td>Seungbin Park, Hannah Kim, Eungjune Shim, Bo-Yeon Hwang, Youngjun Kim, Jung-Woo Lee and Hyunseok Seo<br>Deep Learning-Based Automatic Segmentation of Mandible and Maxilla in Multi-Center CT Images<br>Reprinted from: Appl. Sci. 2022, 12, 1358, doi:10.3390/app12031358</td>
      <td>39</td>
    </tr>
    <tr>
      <td>Carmelo Militello, Andrea Ranieri, Leonardo Rundo, Ildebrando D'Angelo, Franco Marinozzi, Tommaso Vincenzo Bartolotta, et al.<br>On Unsupervised Methods for Medical Image Segmentation: Investigating Classic Approaches in Breast Cancer DCE-MRI<br>Reprinted from: Appl. Sci. 2022, 12, 162, doi:10.3390/app12010162</td>
      <td>55</td>
    </tr>
    <tr>
      <td>Yusuke Asami, Takaaki Yoshimura, Keisuke Manabe, Tomonari Yamada and Hiroyuki Sugimori<br>Development of Detection and Volumetric Methods for the Triceps of the Lower Leg Using Magnetic Resonance Images with Deep Learning<br>Reprinted from: Appl. Sci. 2021, 11, 12006, doi:10.3390/app112412006</td>
      <td>85</td>
    </tr>
    <tr>
      <td>Angie Fasoula, Luc Duchesne, Julio Daniel Gil Cano, Brian M. Moloney, Sami M. Abd Elwahab and Michael J. Kerin<br>Automated Breast Lesion Detection and Characterization with the Wavelia Microwave Breast Imaging System: Methodological Proof-of-Concept on First-in-Human Patient Data<br>Reprinted from: Appl. Sci. 2021, 11, 9998, doi:10.3390/app11219998</td>
      <td>99</td>
    </tr>
    <tr>
      <td>Hakim Baazaoui, Simon Hubertus, Máté E. Maros, Sherif A. Mohamed, Alex Förster, Lothar R. Schad and Holger Wenz<br>Artificial Neural Network-Derived Cerebral Metabolic Rate of Oxygen for Differentiating Glioblastoma and Brain Metastasis in MRI: A Feasibility Study<br>Reprinted from: Appl. Sci. 2021, 11, 9928, doi:10.3390/app11219928</td>
      <td>121</td>
    </tr>
    <tr>
      <td>Kawther Taibouni, Alexandra Miere, Abdourahmane Samake, Eric Souied, Eric Petit and Yasmina Chenoune<br>Choroidal Neovascularization Screening on OCT-Angiography Choriocapillaris Images by Convolutional Neural Networks<br>Reprinted from: Appl. Sci. 2021, 11, 9313, doi:10.3390/app11199313</td>
      <td>135</td>
    </tr>
  </tbody>
</table>
Siangruei Wu, Yihong Wu, Haoyun Chang, Florence T. Su, Hengchun Liao, Wanju Tseng, et al.
Deep Learning-Based Segmentation of Various Brain Lesions for Radiosurgery
Reprinted from: *Appl. Sci.* **2021**, *11*, 9180, doi:10.3390/app11199180
149

Jay Karhade, Samit Kumar Ghosh, Pranjali Gajbhiye, Rajesh Kumar Tripathy and U. Rajendra Acharya
Multichannel Multiscale Two-Stage Convolutional Neural Network for the Detection and Localization of Myocardial Infarction Using Vectorcardiogram Signal
Reprinted from: *Appl. Sci.* **2021**, *11*, 7920, doi:10.3390/app11177920
169

Mukta Sharma, Venkanagouda S. Goudar, Manohar Prasad Koduri, Fan Gang Tseng and Mahua Bhattacharya
Quantitative and Qualitative Image Analysis of *In Vitro* Co-Culture 3D Tumor Spheroid Model by Employing Image-Processing Techniques
Reprinted from: *Appl. Sci.* **2021**, *11*, 4636, doi:10.3390/app11104636
185

JiHwan Lee and Seok Won Chung
Deep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future
Reprinted from: *Appl. Sci.* **2022**, *12*, 681, doi:10.3390/app12020681
203
<figure><img src="image_5.png" /></figure>

Editorial

# Artificial Intelligence Applied to Medical Imaging and Computational Biology

Leonardo Rundo <fnref n="1" />, Andrea Tangherloni <fnref n="2" /> and Carmelo Militello <fnref n="3" />,*

The Special Issue "Artificial Intelligence Applied to Medical Imaging and Computational Biology" of the Applied Sciences Journal has been curated from February 2021 to May 2022, which covered the state-of-the-art and novel algorithms and applications of Artificial Intelligence methods for biomedical data analysis, ranging from classic Machine Learning to Deep Learning.

Medical imaging and computational biology continuously pose new fundamental medical and biological questions that often give rise to novel challenges in Artificial Intelligence. Moreover, the amount of biomedical data is constantly increasing due to the different image acquisition modalities and high-throughput technologies [1,2]. In these research fields, there is thus an increasing need for the application of cutting-edge computational approaches that generally involve Machine Learning or Computational Intelligence techniques, able to provide high-performance and specialized services in medical contexts [3]. Machine Learning and Computational Intelligence techniques can effectively perform image processing operations (such as segmentation [4–10], classification [11–14], and quantification [15–18]), in the fields of neuroimaging and oncological imaging. Although manual approaches often remain the golden standard in several tasks, Machine Learning can be exploited to automate and facilitate the work of researchers and clinicians. In addition, these fields often present new clustering and classification challenges, as well as combinatorial problems, which can be effectively addressed using novel strategies based on Machine Learning and Computational Intelligence techniques.

More recently, Deep Learning approaches [4,5,7,11,14,19] were shown to be very successful in computer vision and bioinformatics tasks owing to their ability to automatically extract hierarchical descriptive features from input images or gene expression data. They have also been used in the oncological, neuroimaging, and microscopy imaging domains for automatic disease diagnosis [12,13], tissue segmentation [16,20], and even synthetic image generation. However, the main issue remains the relative sample paucity of the typical datasets that leads to a poor generalization of the employed deep Artificial Neural Networks, considering the high number of required parameters. Consequently, parameter-efficient design paradigms specifically tailored to biomedical applications ought to be devised, also by exploiting Computational Intelligence based techniques (e.g., Evolutionary Computation, Swarm Intelligence, and neuroevolution).

In this context, advanced Machine Learning techniques were suitably exploited to combine heterogeneous sources of information, allowing for multiomics data integration [21,22]. Such kinds of analyses represent a significant step towards personalized medicine.

**Author Contributions:** Conceptualization, L.R., A.T. and C.M.; writing—original draft preparation, L.R., A.T. and C.M.; writing—review and editing, L.R., A.T. and C.M.; visualization, L.R. and A.T.; supervision, C.M. All authors have read and agreed to the published version of the manuscript.

**Citation:** Rundo, L.; Tangherloni, A.; Militello, C. Artificial Intelligence Applied to Medical Imaging and Computational Biology. *Appl. Sci.* **2022**, *12*, 9052. https://doi.org/10.3390/app12189052

Received: 5 September 2022

Accepted: 6 September 2022

Published: 8 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_6.png" /></figure>

Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

<footnote n="1">¹ Department of Information and Electrical Engineering and Applied Mathematics (DIEM), University of Salerno, 84084 Fisciano, Italy</footnote>

<footnote n="2">² Department of Human and Social Sciences, University of Bergamo, 24129 Bergamo, Italy</footnote>

<footnote n="3">³ Institute of Molecular Bioimaging and Physiology, Italian National Research Council (IBFM-CNR), 90015 Cefalù, Italy</footnote>

<footnote>* Correspondence: carmelo.militello@cnr.it</footnote>
**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Rundo, L.; Militello, C.; Vitabile, S.; Russo, G.; Sala, E.; Gilardi, M.C. A Survey on Nature-Inspired Medical Image Analysis: A Step Further in Biomedical Data Integration. *Fund. Inform.* **2019**, *171*, 345–365. [CrossRef]

2. Castiglioni, I.; Rundo, L.; Codari, M.; Di Leo, G.; Salvatore, C.; Interlenghi, M.; Gallivanone, F.; Cozzi, A.; D'Amico, N.C.; Sardanelli, F. AI Applications to Medical Images: From Machine Learning to Deep Learning. *Phys. Med.* **2021**, *83*, 9–24. [CrossRef] [PubMed]

3. Conti, V.; Militello, C.; Rundo, L.; Vitabile, S. A Novel Bio-Inspired Approach for High-Performance Management in Service-Oriented Networks. *IEEE Trans. Emerg. Top. Comput.* **2021**, *9*, 1709–1722. [CrossRef]

4. Weis, C.-A.; Weihrauch, K.R.; Kriegsmann, K.; Kriegsmann, M. Unsupervised Segmentation in NSCLC: How to Map the Output of Unsupervised Segmentation to Meaningful Histological Labels by Linear Combination? *Appl. Sci.* **2022**, *12*, 3718. [CrossRef]

5. Park, S.; Kim, H.; Shim, E.; Hwang, B.-Y.; Kim, Y.; Lee, J.-W.; Seo, H. Deep Learning-Based Automatic Segmentation of Mandible and Maxilla in Multi-Center CT Images. *Appl. Sci.* **2022**, *12*, 1358. [CrossRef]

6. Militello, C.; Ranieri, A.; Rundo, L.; D'Angelo, I.; Marinozzi, F.; Bartolotta, T.V.; Bini, F.; Russo, G. On Unsupervised Methods for Medical Image Segmentation: Investigating Classic Approaches in Breast Cancer DCE-MRI. *Appl. Sci.* **2021**, *12*, 162. [CrossRef]

7. Wu, S.; Wu, Y.; Chang, H.; Su, F.T.; Liao, H.; Tseng, W.; Liao, C.; Lai, F.; Hsu, F.; Xiao, F. Deep Learning-Based Segmentation of Various Brain Lesions for Radiosurgery. *Appl. Sci.* **2021**, *11*, 9180. [CrossRef]

8. Militello, C.; Rundo, L.; Dimarco, M.; Orlando, A.; Conti, V.; Woitek, R.; D'Angelo, I.; Bartolotta, T.V.; Russo, G. Semi-Automated and Interactive Segmentation of Contrast-Enhancing Masses on Breast DCE-MRI Using Spatial Fuzzy Clustering. *Biomed. Signal Process. Control.* **2022**, *71*, 103113. [CrossRef]

9. Militello, C.; Vitabile, S.; Rundo, L.; Russo, G.; Midiri, M.; Gilardi, M.C. A Fully Automatic 2D Segmentation Method for Uterine Fibroid in MRgFUS Treatment Evaluation. *Comput. Biol. Med.* **2015**, *62*, 277–292. [CrossRef]

10. Isensee, F.; Jaeger, P.F.; Kohl, S.A.A.; Petersen, J.; Maier-Hein, K.H. nnU-Net: A Self-Configuring Method for Deep Learning-Based Biomedical Image Segmentation. *Nat. Methods* **2021**, *18*, 203–211. [CrossRef]

11. Asami, Y.; Yoshimura, T.; Manabe, K.; Yamada, T.; Sugimori, H. Development of Detection and Volumetric Methods for the Triceps of the Lower Leg Using Magnetic Resonance Images with Deep Learning. *Appl. Sci.* **2021**, *11*, 12006. [CrossRef]

12. Baazaoui, H.; Hubertus, S.; Maros, M.E.; Mohamed, S.A.; Förster, A.; Schad, L.R.; Wenz, H. Artificial Neural Network-Derived Cerebral Metabolic Rate of Oxygen for Differentiating Glioblastoma and Brain Metastasis in MRI: A Feasibility Study. *Appl. Sci.* **2021**, *11*, 9928. [CrossRef]

13. Taibouni, K.; Miere, A.; Samake, A.; Souied, E.; Petit, E.; Chenoune, Y. Choroidal Neovascularization Screening on OCT-Angiography Choriocapillaris Images by Convolutional Neural Networks. *Appl. Sci.* **2021**, *11*, 9313. [CrossRef]

14. Karhade, J.; Ghosh, S.K.; Gajbhiye, P.; Tripathy, R.K.; Rajendra Acharya, U. Multichannel Multiscale Two-Stage Convolutional Neural Network for the Detection and Localization of Myocardial Infarction Using Vectorcardiogram Signal. *Appl. Sci.* **2021**, *11*, 7920. [CrossRef]

15. Zhang, J.; Huang, Y.; Ye, F.; Yang, B.; Li, Z.; Hu, X. Evaluation of Post-Stroke Impairment in Fine Tactile Sensation by Electroencephalography (EEG)-Based Machine Learning. *Appl. Sci.* **2022**, *12*, 4796. [CrossRef]

16. Sharma, M.; Goudar, V.S.; Koduri, M.P.; Tseng, F.G.; Bhattacharya, M. Quantitative and Qualitative Image Analysis of In Vitro Co-Culture 3D Tumor Spheroid Model by Employing Image-Processing Techniques. *Appl. Sci.* **2021**, *11*, 4636. [CrossRef]

17. Rundo, L.; Tangherloni, A.; Militello, C.; Gilardi, M.C.; Mauri, G. Multimodal Medical Image Registration Using Particle Swarm Optimization: A Review. In *Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence (SSCI)*, Athens, Greece, 6–9 December 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1–8.

18. Rundo, L.; Militello, C.; Vitabile, S.; Casarino, C.; Russo, G.; Midiri, M.; Gilardi, M.C. Combining Split-and-Merge and Multi-Seed Region Growing Algorithms for Uterine Fibroid Segmentation in MRgFUS Treatments. *Med. Biol. Eng. Comput.* **2016**, *54*, 1071–1084. [CrossRef]

19. Lee, J.; Chung, S.W. Deep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future. *Appl. Sci.* **2022**, *12*, 681. [CrossRef]

20. Fasoula, A.; Duchesne, L.; Cano, J.D.G.; Moloney, B.M.; Abd Elwahab, S.M.; Kerin, M.J. Automated Breast Lesion Detection and Characterization with the Wavelia Microwave Breast Imaging System: Methodological Proof-of-Concept on First-in-Human Patient Data. *Appl. Sci.* **2021**, *11*, 9998. [CrossRef]

21. Simidjievski, N.; Bodnar, C.; Tariq, I.; Scherer, P.; Andres Terre, H.; Shams, Z.; Jamnik, M.; Liò, P. Variational Autoencoders for Cancer Data Integration: Design Principles and Computational Practice. *Front. Genet.* **2019**, *10*, 1205. [CrossRef]

22. Tangherloni, A.; Ricciuti, F.; Besozzi, D.; Liò, P.; Cvejic, A. Analysis of Single-Cell RNA Sequencing Data Based on Autoencoders. *BMC Bioinform.* **2021**, *22*, 309. [CrossRef] [PubMed]
Article

# Evaluation of Post-Stroke Impairment in Fine Tactile Sensation by Electroencephalography (EEG)-Based Machine Learning

Jianing Zhang<fnref n="1" />, Yanhuan Huang<fnref n="1" />, Fuqiang Ye<fnref n="1" />, Bibo Yang<fnref n="1" />, Zengyong Li<fnref n="2" /> and Xiaoling Hu<fnref n="1" />,<fnref n="3" />,<fnref n="4" />,<fnref n="5" />,*

**Abstract:** Electroencephalography (EEG)-based measurements of fine tactile sensation produce large amounts of data, with high costs for manual evaluation. In this study, an EEG-based machine-learning (ML) model with support vector machine (SVM) was established to automatically evaluate post-stroke impairments in fine tactile sensation. Stroke survivors ($n = 12$, stroke group) and unimpaired participants ($n = 15$, control group) received stimulations with cotton, nylon, and wool fabrics to the different upper limbs of a stroke participant and the dominant side of the control. The average and maximal values of relative spectral power (RSP) of EEG in the stimulations were used as the inputs to the SVM-ML model, which was first optimized for classification accuracies for different limb sides through hyperparameter selection ($\gamma$, $C$) in radial basis function (RBF) kernel and cross-validation during cotton stimulation. Model generalization was investigated by comparing accuracies during stimulations with different fabrics to different limbs. The highest accuracies were achieved with ($\gamma = 2^1$, $C = 2^3$) for the RBF kernel (76.8%) and six-fold cross-validation (75.4%), respectively, in the gamma band for cotton stimulation; these were selected as optimal parameters for the SVM-ML model. In model generalization, significant differences in the post-stroke fabric stimulation accuracies were shifted to higher (beta/gamma) bands. The EEG-based SVM-ML model generated results similar to manual evaluation of cortical responses to fabric stimulations; this may aid automatic assessments of post-stroke fine tactile sensations.

**Keywords:** stroke; fine tactile sensation; electroencephalography; machine learning; evaluation

**Citation:** Zhang, J.; Huang, Y.; Ye, F.; Yang, B.; Li, Z.; Hu, X. Evaluation of Post-Stroke Impairment in Fine Tactile Sensation by Electroencephalography (EEG)-Based Machine Learning. *Appl. Sci.* **2022**, *12*, 4796. https://doi.org/10.3390/app12094796

Academic Editors: Leonardo Rundo, Carmelo Militello and Andrea Tangherloni

Received: 25 March 2022

Accepted: 6 May 2022

Published: 9 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_7.png" /></figure>

Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## 1. Introduction

Approximately 50% of stroke survivors have reported persistent sensory deficiencies for both somatosensation and proprioception [1,2]. For example, they often have difficulties in perceiving pain, temperature, pressure, posture, and light touch [3]. Sensory deficiencies have profound negative impacts on the functional ability and independency in daily living, which further affect motor recovery after stroke [4,5]. Fine tactile sensation is an elementary somatosensory function for obtaining external information through touch [6]. Previous studies have shown that fine tactile sensation also provide valid spatial references for body positions to reduce postural sway [7], and it may act as an indicator to enhance sensory feedback in position control [8,9]. However, rehabilitation for sensory functions has been overlooked in the traditional practices, when compared with efforts for motor restoration; this is attributed to the lack of effective evaluation measures for sensory impairments [10].

<footnote n="1">¹ Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong 999077, China; jayce.zhang@connect.polyu.hk (J.Z.); yanhuan.huang@connect.polyu.hk (Y.H.); 19073922r@connect.polyu.hk (F.Y.); bibo.yang@polyu.edu.hk (B.Y.)</footnote>

<footnote n="2">² National Research Centre for Rehabilitation Technical Aids Beijing, Beijing Key Laboratory of Rehabilitation Technical Aids for Old-Age Disability, Beijing 100176, China; lizengyong@nrcrta.cn</footnote>

<footnote n="3">³ University Research Facility in Behavioural and Systems Neuroscience (UBSN), The Hong Kong Polytechnic University, Hong Kong 999077, China</footnote>

<footnote n="4">⁴ Shenzhen Research Institute, The Hong Kong Polytechnic University, Shenzhen 518000, China</footnote>

<footnote n="5">⁵ Research Institute for Smart Ageing (RISA), The Hong Kong Polytechnic University, Hong Kong 999077, China</footnote>

<footnote>* Correspondence: xiaoling.hu@polyu.edu.hk; Tel.: +852-3400-3206</footnote>
Objective and efficient assessments of sensory impairments are important for long-
term post-stroke rehabilitation with repeated measurements during follow up [11]. How-
ever, subjective and manual measurements have been used traditionally for sensory im-
pairment assessments [12]. For example, the Fugl–Meyer assessment (FMA) [13] and
Semmes–Weinstein monofilament test [14] are commonly used in current evaluations of
fine tactile sensations because of the ease of interpretation of the assessment results. Addi-
tionally, the measurement process highly relies on the personal experiences of the assessor,
where achieving consistency in measurements is challenging when the stroke population
increases during long-term service [15].

Neuroimaging techniques have been introduced to provide objective data for sensory impairment assessments [16]. The common neuroimaging techniques include functional magnetic resonance imaging (fMRI), positron emission tomography (PET), neuromolecular imaging, and electroencephalography (EEG), among others [17–23]. These approaches characterize neural circuitry changes during post-stroke sensorimotor recovery; however, such medical equipment is expensive and the preparations before neuroimaging-based examinations are complicated compared to the traditional clinical assessments [24]. Among these techniques, owing to the advantage of high temporal resolution, EEG has been applied to detect transient sensory neural responses during fine tactile stimulations [25,26]. For example, Ahn et al. compared the effects of different tactile exploration tasks, i.e., passively or actively moving a tactile board, on post-stroke brain activation using EEG [27]. The sensory motor rhythm indicated by the EEG relative powers from the right prefrontal and parietal lobes during active tactile perceptions were significantly greater than those in the damaged left hemisphere during passive tactile perception [27]. In our previous work [28], post-stroke sensory impairment of fine tactile sensation was measured quantitatively via EEG during textile fabric stimulation, i.e., simulation of the common fabric-skin touch. We observed EEG relative spectral power (RSP) differences after stroke, i.e., RSP intensities in different frequency bands between unimpaired and stroke populations [28]. However, neuroimaging-based measurements usually generate large amounts of data, whose interpretations still heavily rely on human professionals, which is time consuming and labor demanding [29,30].

Neuroimaging data interpretation by machine-learning (ML) techniques has been a promising approach to reduce manpower workload in data interpretations [31]. ML is a technique that can help develop an automatic predictive model by learning the relationships between features and targets from a given set of historical data before application to repeated analyses on massive data [32]. Various ML algorithms, e.g., linear discriminant analysis (LDA), artificial neural network (ANN), and support vector machine (SVM), are being explored for the detection, classification, and characterization of neuroimaging data, e.g., EEG [31]. For instance, Jochumsen et al. classified single-trial movement intentions associated with different hand grasp types using the EEG spectra as input features to an LDA model [33]. Usama et al. distinguished correct/error feedbacks during hand and foot movements by feeding the EEG waveform features into an ANN model [34]. Limited classification accuracies were obtained in both studies: 41–86% [33,34]. This may be attributed to insufficiencies in feature mapping by simple linear transformation of the LDA, leading to inefficient construction of the optimal decision function (classification boundary) for multichannel EEG [35,36]. Although ANN-based models offer nonlinear feature mapping abilities during classifications, overfitting often occurs when there are several hyperparameters, e.g., numbers of hidden layers and nodes, to be determined during network optimization [36]. In contrast to the ANN, SVM-based models reduce the disadvantages of overfitting of the classification results with the help of kernel functions [37]. SVM with kernel functions effectively minimize model complexities via implicitly realizing nonlinear transformations of the feature spaces without explicit mathematical expressions, so that only specific hyperparameters related to the kernel functions of the SVM need to be optimized during model development [38]. In the SVM-ML models, several kernel functions are commonly used, namely linear, polynomial, and radial basis function (RBF)
kernels. For example, Liu et al. extracted the spectrum features from subject-related EEG frequency bands and channels, and the SVM with linear kernel was applied to each subject's EEG-based motor imagery classification [39]. Ghumman et al. investigated the classification performance of SVM with a polynomial kernel in multiclass motor imagery EEG [40]. Bousseta et al. used SVM with RBF kernel to classify the EEGs of imagined hand movements [41]. These studies reported a classification range of 67–92.8% [39–41]. Among the practical applications of the kernel functions mentioned above, the RBF kernel is a common choice in SVM-ML models because of its better performance on the nonlinearities in feature mapping capabilities with less hyperparameters compared to the other two types of kernel functions [42,43].

Automatic evaluations of neuroimaging data by SVM-ML techniques have not been fully explored in literature, e.g., fine tactile sensation. Kim et al. extracted the powers of the alpha and gamma bands as features representing EEG during touch with different objects, i.e., fabric, glass, and paper [44]. However, they only evaluated the tactile perception of unimpaired persons and obtained limited classification performance (68.1%) with the LDA model [44]. The purpose of this study was to automatically evaluate and assess post-stroke impairments in fine tactile sensation using a new EEG-based SVM-ML model.

## 2. Methodology

In this study, an SVM-ML model was established based on EEG measurements of cortical responses to fine tactile stimulations to the upper limbs in persons who have experienced stroke and in unimpaired participants via stimulations with different types of fabrics (cotton, nylon, and wool). The SVM-ML model was first developed and optimized using EEG RSP features with cotton fabric stimulation as the baseline input for classifying the responses from multiple upper limb groups, i.e., stimulation to the (1) affected sides of persons after stroke (SA), (2) unaffected sides of persons after stroke (SU), and (3) dominant sides of unimpaired participants (UD). Then, the generalization performance of the model was evaluated using the EEG RSP features during stimulations with different fabrics with and without considering arm differences.

### 2.1. EEG Acquisitions during Fabric Stimulations

After obtaining ethical approval from the Human Subjects Ethics Sub-committee (HSESC) of the Hong Kong Polytechnic University, twelve survivors of chronic stroke were recruited as the “stroke group”, and fifteen unimpaired participants were recruited as the “control group”, whose demographic details are listed in Table 1. The inclusion criteria of stroke group were: (1) individuals must be at least six months after the singular and unilateral brain lesion due to stroke; (2) the lesions occurring due to stroke were experienced in the subcortical area, to ensure the detectable EEG from the cortical area. All unimpaired participants were right-handed. No significant difference was found in age between the stroke and control groups ($p > 0.05$) by the independent t-test after verifying normality evaluations via the Shapiro-Wilk test [45].

<table><caption>Table 1. Demographic characteristics and clinical scores of the stroke and control groups [28].</caption><thead><tr><th>Measure</th><th>Stroke Group (n = 12)</th><th>Control Group (n = 15)</th></tr></thead><tbody><tr><td>Age in years</td><td>55.1 ± 16.0</td><td>46.4 ± 17.4</td></tr><tr><td>Gender (male/female)</td><td>11/1</td><td>5/10</td></tr><tr><td>Stroke type (ischemic/hemorrhagic)</td><td>10/2</td><td>-</td></tr><tr><td>Affected side (right/left)</td><td>6/6</td><td>-</td></tr><tr><td>Years since stroke</td><td>14.9 ± 5.8</td><td>-</td></tr><tr><td>FMA (upper extremity)</td><td>42.5 ± 15.2</td><td>-</td></tr><tr><td>FMA (light touch on forearm)</td><td>1 ± 0</td><td>-</td></tr><tr><td>MAS (elbow)</td><td>1.1 ± 0.7</td><td>-</td></tr></tbody></table>

Note: Data are given as mean ± standard deviation. MAS: Modified Ashworth scale; FMA: Fugl-Meyer assessment [28].
The experimental setup and protocol for the fabric fine tactile stimulation are shown in Figure 1. The three types of fabrics, i.e., cotton, nylon, and wool of the same size and different textural properties, were alternatively placed on the skin surface of the ventral forearm of the upper limb, i.e., a single stimulation trial (Figure 1c). Each trial consisted of a 30 s baseline measurement, i.e., no fabric stimulation to the skin, followed by alternative stimuli with the three different fabrics in a random sequence for 13 s stimulation with each fabric and 60 s gaps in between. The stimulation trial was repeated thrice for each target forearm. The whole brain EEG with 64 channels (BP-01830, Brain Products Inc., Gilching, Germany) based on the 10–20 system [46] was captured during the stimulation trials at a sampling frequency of 1000 Hz. Each subject was required to stay awake and calm during the EEG measurements while wearing ear plugs and an eye mask, whose purpose was to minimize visual and audio disturbances from the environment. The detailed experimental procedure is described in [28].

<figure><img src="image_8.png" /><figcaption>**Figure 1.** Electroencephalography (EEG) experimental setup and protocol. (a) Participant wearing the EEG cap, ear plugs, and eye mask, is seated in a chair and tested with their hands comfortably placed on the table. The areas of each fabric stimulation are the ventral forearms. The three fabric samples, namely cotton, nylon, and wool, are placed on the table. (b) Illustration of fabric stimulation. The nylon fabric is placed on the ventral forearm. (c) Protocol for fabric stimulation.</figcaption></figure>

## 2.2. EEG-Specific Feature Extraction for the SVM-ML Model

In this work, the RSP<sub>mean</sub> and RSP<sub>max</sub> values in different frequency bands were selected as the EEG features for input to the SVM-ML model as they represent the average and maximal cortical changes, respectively, during fabric stimulations based on manual recognition from the previous study [28]. During the real-time EEG recording, the sampling frequency was 1000 Hz. In the preparation of the EEG RSP features, a Butterworth bandpass filter from 0.1 to 100 Hz was first applied to the EEG to eliminate irrelevant high-frequency components. Then, an additional Butterworth notch filter from 49 Hz to 51 Hz was applied to eliminate the 50 Hz noise from the environment. Following this, the filtered EEG was segmented into different epochs, i.e., 30 s pre-stimulus baseline and three 13 s stimuli with different fabrics. The numbers of EEG samples after segmentation were 108 from the SA group (12 participants × 3 trials × 3 fabric stimuli), 108 for the SU group (12 participants × 3 trials × 3 fabric stimuli), and 135 for the UD group (15 participants × 3 trials × 3 fabric stimuli). Next, the EEG samples were transformed into their power spectra by Pwelch estimation [47], and the entire frequency band (0.1–100 Hz) of each segmented EEG epoch was decomposed into five frequency bands, i.e., delta (0.5–4 Hz), theta (4–8 Hz), alpha
(8–12 Hz), beta (12–30 Hz), and gamma (30–100 Hz) [48]. Finally, the RSP [49] of each frequency band for each fabric stimulus was calculated using the following equations:

$$P(f_1, f_2) = \int_{f_1}^{f_2} p(f) df \qquad (1)$$

$$RSP(f_1, f_2) = \frac{P(f_1, f_2)}{P(0.1, 100)} - \frac{P_{\text{baseline}}(f_1, f_2)}{P(0.1, 100)}, \qquad (2)$$

where p(f) is the power spectral density; f₁ and f₂ are the low and high cutoff frequencies of a given EEG frequency band, respectively; P(f₁, f₂) is the power spectrum from f₁ to f₂; and P<sub>baseline</sub> is the power spectrum of the EEG segments during the baseline test in each trial. The above spectral analysis of the raw EEG signals was implemented offline with the EEGLAB v12 toolbox in MATLAB (The MathWorks Inc., Natick, MA, USA).

After obtaining the RSP value from each EEG channel, the RSP<sub>mean</sub> and RSP<sub>max</sub> values were acquired to represent the RSP features of the multichannel EEG, where RSP<sub>mean</sub> is the average value of the RSPs of all the channels in a given frequency band of a signal epoch, and RSP<sub>max</sub> is the highest value among all the EEG channels. Then, the RSP<sub>mean</sub> and RSP<sub>max</sub> of the 62-channel EEG (ground and reference channels were neglected), which covered the entire cortical area, were calculated for each frequency band. To minimize the diversity of the ranges for the RSP<sub>mean</sub> and RSP<sub>max</sub>, the original RSP<sub>mean</sub> and RSP<sub>max</sub> were further normalized as in the following equation according to z-score normalization, which scales all the RSP<sub>mean</sub> and RSP<sub>max</sub> values in varying ranges with a zero mean and unit standard deviation [50]:

$$RSP'_{i} = \frac{RSP_{i} - \mu_{RSP}}{\sigma_{RSP}}, \qquad (3)$$

where RSP<sub>i</sub> is the original spectral feature, i.e., RSP<sub>mean</sub> or RSP<sub>max</sub>; μ<sub>RSP</sub> is the mean of RSP<sub>i</sub>; σ<sub>RSP</sub> is the standard deviation of RSP<sub>i</sub>; and RSP<sub>i</sub>' is the normalized spectral feature. The normalized features were then used as the inputs to the SVM-ML model.

## 2.3. SVM-ML Model Configuration

Figure 2 shows the configuration of the SVM-ML model, including optimization of the SVM RBF kernel function and k-fold cross-validation (CV) strategy. The normalized EEG features (i.e., RSP<sub>mean</sub> and RSP<sub>max</sub>) during stimulation with cotton fabric were adopted as the baseline inputs for model establishment. This is because cotton is the most widely used fabric that is in intimate contact with skin in daily living and provides minimum stimulation intensity with a comfortable feeling compared to other fabrics [51]. In addition, compared to nylon and wool, the textile physical properties of cotton fabric as quantitatively measured by the fabric touch tester (FTT) [52] were neutral with equivalent distances in the aspects of smoothness, thickness, etc. [28]. Therefore, the EEG RSP features evoked by the cotton fabric were used as the baseline inputs to configure the SVM-ML model.

The RBF kernel function of the SVM-ML model was determined by optimizing the classification boundaries that achieved the best accuracy on the RSP features related to cotton stimulation. For an RBF kernel, two hyperparameters, namely the kernel scaling parameter γ and regularization parameter C [53], are optimized in the SVM-ML model development to classify the different upper-limb groups. The search for optimal (γ, C) was conducted by a “grid search” approach [54]. The candidate values of (γ, C) were first defined as exponentially increasing sequences (γ = 2<sup>-15</sup>, 2<sup>-13</sup>, . . . , 2<sup>9</sup>; C = 2<sup>-5</sup>, 2<sup>-3</sup>, . . . , 2<sup>15</sup>), which were the ranges adopted by most EEG-based SVM-ML studies to identify the optimal (γ, C) values [42, 55, 56]. Following this, different pairs of γ and C (13 × 11 = 143 pairs) values were generated, and each pair was used to construct the RBF kernel of the SVM. The classification accuracies with the different hyperparameter pairs were evaluated by three-fold CV according to the greatest common divisor of the number of stroke patients (i.e., n = 12) and unimpaired controls (i.e., n = 15); this is a common pilot estimation approach used in previous studies [57]. The value pair that achieved the best classification
accuracy was then adopted as the optimal hyperparameters for model configuration. The above SVM algorithm was implemented using the Scikit-learn toolbox, an open-source ML toolbox in Python [58].

<figure><img src="image_9.png" /><figcaption>**Figure 2.** Configuration flowchart for the SVM-ML model. RSP: relative spectral power; SVM: support vector machine; RBF: radial basis function; Acc: accuracy; LOO: leave-one-out; CV: cross validation.</figcaption></figure>

The grid search results of γ and C are displayed in Figure 3, where Figure 3a–e present the accuracies of different (γ, C) pairs for distinguishing the UD, SA, and SU groups with the RSP<sub>mean</sub> and RSP<sub>max</sub> of the 62-channel EEG in the delta, theta, alpha, beta, and gamma bands, respectively, as input features. The coordinates and values of the highest accuracy for each frequency band are indicated by the red dots in Figure 3a–e. Among all the accuracies, the model with the highest accuracy of 67.4% (γ = 2<sup>3</sup>, C = 2<sup>9</sup>) was achieved in the gamma band.

As the sensorimotor cortex is the main response area for sensory stimulations [59], the classification performance achieved by including only the EEG channels covering the sensorimotor cortex was evaluated in the SVM-ML model configuration. The RSP<sub>mean</sub> and RSP<sub>max</sub> of the corresponding 21-channel EEG (i.e., FC1–FC6, FCZ, C1–C6, CZ, CP1–CP6, CPZ), which cover the sensorimotor area [60], were used as the inputs to the model. Figure 4 shows the accuracies with the RSP features for the 21-channel EEG, and the highest accuracy of 76.8% (γ = 2<sup>1</sup>, C = 2<sup>3</sup>) was obtained for the gamma band as well.

The accuracies of the SVM-ML model for classifying the UD, SA, and SU groups with the RBF kernel hyperparameter pairs in the different bands are summarized in Table 2. Compared to other frequency bands, the gamma band has the best average accuracy performance for both channel set selections. The average and peak accuracies of the gamma band of the 21-channel EEG were better than those of the 62-channel EEG. Therefore, the hyperparameter pair (γ = 2<sup>1</sup>, C = 2<sup>3</sup>) from the 21-channel EEG was selected as the optimal RBF kernel hyperparameters.

After the RBF kernel function was determined, the k-fold CV was also configured using the RSP features of the 21-channel EEG as inputs to improve generalization of the SVM-ML model. Compared to the simple train/test split, the k-fold CV ensures that each sample from the original dataset has the chance of appearing in the training and testing set, which results in less biased evaluations [61]. Since the partition of k folds is random, the k-fold CV was performed 10 times to calculate the mean estimate to decrease the variance of accuracy estimations of the one-shot k-fold CV [62–64]. Typically, the configuration of k is 5 or 10, as these values have been shown to be the bias-variance trade-off for model evaluation [61,65]. In our experiment, different selections of k from 2 to 10 were employed to compare the influence of k on model performance. In addition, the leave-one-out CV, where k is the number of samples in the dataset, was used as a complementary comparison to different k-fold CV. Although the leave-one-out CV is more computationally expensive compared to the above strategies, i.e., five-fold and ten-fold CV, it offers an unbiased
evaluation of the model performance as each sample is given the opportunity to represent
the entirety of the test dataset [61].

<figure><img src="image_10.png" /><figcaption>**Figure 3.** Grid search results of γ and C in the SVM-ML model with RBF kernel using the 62-channel EEG RSP features of the delta, theta, alpha, beta, and gamma bands. Acc: accuracy. Peak Acc: highest classification accuracy of the SVM-ML model in the predefined range of (γ, C); Peak Loc: location (γ, C) corresponding to the highest classification accuracy of the SVM-ML model.</figcaption></figure>

The accuracies of the SVM-ML model for distinguishing between the UD, SA, and
SU groups with different k-fold CV strategies in the different frequency bands are shown
in Table 3. The model achieved the highest accuracy of 75.4% in the gamma band by
six-fold CV. For the leave-one-out CV, the model obtained the highest classification accu-
racy of 74.4% in the gamma band as well. Therefore, the six-fold CV was selected as the
optimal evaluation strategy for the model when using the RSP features of the 21-channel
EEG as inputs.
<figure><img src="image_11.png" /><figcaption>**Figure 4.** Grid search results of γ and C in the SVM-ML model with RBF kernel using the 21-channel EEG RSP features of the delta, theta, alpha, beta, and gamma bands. Acc: accuracy. Peak Acc: highest classification accuracy of the SVM-ML model in the predefined range of (γ, C); Peak Loc: location (γ, C) corresponding to the highest classification accuracy of the SVM-ML model.</figcaption></figure>

## 2.4. Generalization of the SVM-ML Model

Using the RSP features during stimulation with cotton fabric as the baseline inputs, the SVM-ML model was established. Then, we first investigated the generalization performance of the model for classifying the upper-limb groups with the inputs of different fabrics, i.e., nylon, wool, and cotton. The measured RSP features in the respective stimulations were then input to the developed model, and the achieved accuracies are summarized in Table 4. The classification accuracies of the different fabric stimulations were not normally distributed (*p* < 0.5, Shapiro-Wilk test) in each frequency band. Significant intergroup differences in the accuracies (*p* < 0.001, Kruskal-Wallis test) with respect to fabric stimulation were observed in the delta, theta, alpha, beta, and gamma bands. The model achieved the highest classification accuracies of 75.4%, 83.5%, and 84.3% for the cotton, nylon, and wool stimulations, respectively, in the gamma band.
<table><caption>**Table 2.** Accuracies of the SVM-ML model for classifying the three upper-limb groups with the RBF kernel hyperparameter pairs in different frequency bands.</caption><thead><tr><th>Number of EEG Channels</th><th></th><th>Delta</th><th>Theta</th><th>Alpha</th><th>Beta</th><th>Gamma</th></tr></thead><tbody><tr><td rowspan="3">62</td><td>Average Acc</td><td>33.5% ± 0.05</td><td>37.8% ± 0.05</td><td>35.8% ± 0.03</td><td>38.2% ± 0.04</td><td>44.7% ± 0.11</td></tr><tr><td>Peak Acc</td><td>38.5%</td><td>51.8%</td><td>40.0%</td><td>47.6%</td><td>67.4%</td></tr><tr><td>Peak Loc (γ, C)</td><td>(2<sup>-3</sup>, 2<sup>5</sup>)</td><td>(2<sup>3</sup>, 2<sup>1</sup>)</td><td>(2<sup>-5</sup>, 2<sup>1</sup>)</td><td>(2<sup>-7</sup>, 2<sup>7</sup>)</td><td>(2<sup>3</sup>, 2<sup>9</sup>)</td></tr><tr><td rowspan="3">21</td><td>Average Acc</td><td>39.1% ± 0.06</td><td>35.6% ± 0.04</td><td>33.2% ± 0.06</td><td>41.3% ± 0.07</td><td>49.2% ± 0.16</td></tr><tr><td>Peak Acc</td><td>57.3%</td><td>49.3%</td><td>38.5%</td><td>57.8%</td><td>76.8%</td></tr><tr><td>Peak Loc (γ, C)</td><td>(2<sup>-3</sup>, 2<sup>11</sup>)</td><td>(2<sup>3</sup>, 2<sup>13</sup>)</td><td>(2<sup>-13</sup>, 2<sup>10</sup>)</td><td>(2<sup>-5</sup>, 2<sup>13</sup>)</td><td>(2<sup>1</sup>, 2<sup>3</sup>)</td></tr></tbody></table>

Note: Average Acc: average classification accuracy of the SVM-ML model with all the RBF kernel hyperparameter pairs; Peak Acc: highest classification accuracy of the SVM-ML model in the predefined range of (γ, C); Peak Loc: location (γ, C) corresponding to the highest classification accuracy of the SVM-ML model.

<table><caption>**Table 3.** Accuracies of the SVM-ML model for classifying the three upper-limb groups with different k-fold CV strategies in the different frequency bands.</caption><thead><tr><th rowspan="2">CV</th><th colspan="5">Accuracy</th></tr><tr><th>Delta</th><th>Theta</th><th>Alpha</th><th>Beta</th><th>Gamma</th></tr></thead><tbody><tr><td>2-fold</td><td>49.6% ± 0.07</td><td>38.1% ± 0.06</td><td>27.6% ± 0.06</td><td>50.7% ± 0.07</td><td>73.8% ± 0.05</td></tr><tr><td>3-fold</td><td>49.1% ± 0.06</td><td>33.3% ± 0.07</td><td>26.0% ± 0.05</td><td>51.0% ± 0.05</td><td>74.5% ± 0.04</td></tr><tr><td>4-fold</td><td>49.7% ± 0.06</td><td>34.7% ± 0.06</td><td>23.9% ± 0.05</td><td>50.1% ± 0.05</td><td>74.8% ± 0.04</td></tr><tr><td>5-fold</td><td>49.7% ± 0.06</td><td>32.4% ± 0.05</td><td>26.1% ± 0.05</td><td>51.2% ± 0.04</td><td>75.0% ± 0.03</td></tr><tr><td>6-fold</td><td>53.2% ± 0.05</td><td>35.0% ± 0.06</td><td>22.2% ± 0.05</td><td>50.8% ± 0.05</td><td>75.4% ± 0.04</td></tr><tr><td>7-fold</td><td>46.7% ± 0.06</td><td>32.6% ± 0.05</td><td>32.9% ± 0.05</td><td>48.2% ± 0.06</td><td>72.6% ± 0.04</td></tr><tr><td>8-fold</td><td>47.0% ± 0.07</td><td>31.3% ± 0.06</td><td>28.9% ± 0.06</td><td>47.2% ± 0.06</td><td>74.8% ± 0.04</td></tr><tr><td>9-fold</td><td>49.5% ± 0.07</td><td>31.4% ± 0.06</td><td>27.8% ± 0.06</td><td>49.1% ± 0.06</td><td>74.7% ± 0.05</td></tr><tr><td>10-fold</td><td>51.7% ± 0.07</td><td>33.0% ± 0.06</td><td>25.9% ± 0.06</td><td>50.0% ± 0.05</td><td>74.8% ± 0.05</td></tr><tr><td>LOO</td><td>51.3%</td><td>28.2%</td><td>12.8%</td><td>53.8%</td><td>74.4%</td></tr></tbody></table>

Note: Data are given as mean ± SD. CV: cross validation; LOO: leave-one-out.

<table><caption>**Table 4.** Overall accuracies of the SVM-ML model for classifying different fabric stimulations.</caption><thead><tr><th rowspan="2">Fabric Stimulation</th><th colspan="5">Accuracy</th></tr><tr><th>Delta</th><th>Theta</th><th>Alpha</th><th>Beta</th><th>Gamma</th></tr></thead><tbody><tr><td>Cotton</td><td>53.2% ± 0.05</td><td>35.0% ± 0.06</td><td>22.3% ± 0.05</td><td>50.8% ± 0.05</td><td>75.4% ± 0.04</td></tr><tr><td>Nylon</td><td>21.0% ± 0.04</td><td>40.6% ± 0.05</td><td>51.4% ± 0.04</td><td>63.2% ± 0.03</td><td>83.5% ± 0.02</td></tr><tr><td>Wool</td><td>30.3% ± 0.04</td><td>25.6% ± 0.06</td><td>43.0% ± 0.05</td><td>69.2% ± 0.04</td><td>84.3% ± 0.03</td></tr><tr><td>Significance (p-value)</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td></tr></tbody></table>

Note: Data are given as mean ± SD. The significant differences are indicated by ‘***’ ($p < 0.001$, Kruskal-Wallis test).

The comparison of the overall accuracies of the SVM-ML model with respect to fabric stimulation in each band are shown in Figure 5. Significant differences in the accuracies were observed in the delta, theta, alpha, beta, and gamma bands for pairwise comparisons among the three different fabric stimulations ($p < 0.001$, Kruskal-Wallis with Bonferroni post-hoc test), except for the difference between nylon and wool in the gamma band ($p > 0.05$, Kruskal-Wallis with Bonferroni post-hoc test). The models with nylon and wool achieved significantly higher accuracies in the beta and gamma bands than those with cotton ($p < 0.001$, Kruskal-Wallis with Bonferroni post-hoc test).
<figure><img src="image_12.png" /><figcaption>**Figure 5.** Comparisons of the overall classification accuracies of the SVM-ML model with respect to fabric stimulations in the delta, theta, alpha, beta, and gamma bands. The significant intergroup differences are indicated by ‘***’ (*p* < 0.001, Kruskal–Wallis with Bonferroni post-hoc test).</figcaption></figure>

The generalized performance of the model was also evaluated by considering the arm differences during stimulations with different fabrics (Table 5). The classification accuracies of each upper-limb group during stimulations with different fabrics are not normally distributed (*p* < 0.5, Shapiro–Wilk test). Significant differences in the accuracies (*p* < 0.001, Kruskal–Wallis test) with respect to fabric stimulations were observed in each band, except for the SU group in the gamma band (*p* > 0.05, Kruskal–Wallis test). The highest classification accuracy for each upper-limb group was achieved in the gamma band.

<table><caption>**Table 5.** Accuracies of the SVM-ML model for classifying the three upper-limb groups with different fabric stimulations.</caption><thead><tr><th rowspan="2">Fabric<br>Stimulation</th><th rowspan="2"></th><th colspan="5">Accuracy</th></tr><tr><th>Delta</th><th>Theta</th><th>Alpha</th><th>Beta</th><th>Gamma</th></tr></thead><tbody><tr><td rowspan="3">SA</td><td>Cotton</td><td>47.9% ± 0.09</td><td>28.8% ± 0.11</td><td>28.5% ± 0.10</td><td>48.9% ± 0.10</td><td>59.7% ± 0.08</td></tr><tr><td>Nylon</td><td>26.8% ± 0.09</td><td>31.9% ± 0.08</td><td>54.1% ± 0.11</td><td>64.3% ± 0.10</td><td>76.2% ± 0.06</td></tr><tr><td>Wool</td><td>22.9% ± 0.09</td><td>21.7% ± 0.08</td><td>51.1% ± 0.11</td><td>53.6% ± 0.08</td><td>78.9% ± 0.04</td></tr><tr><td rowspan="3">SU</td><td>p-value</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td></tr><tr><td>Cotton</td><td>48.3% ± 0.08</td><td>40.9% ± 0.13</td><td>27.0% ± 0.09</td><td>69.3% ± 0.05</td><td>91.0% ± 0.04</td></tr><tr><td>Nylon</td><td>15.5% ± 0.07</td><td>40.5% ± 0.13</td><td>61.7% ± 0.04</td><td>74.9% ± 0.01</td><td>91.2% ± 0.03</td></tr><tr><td rowspan="3">UD</td><td>p-value</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&gt;0.05</td></tr><tr><td>Cotton</td><td>51.4% ± 0.14</td><td>31.6% ± 0.10</td><td>22.0% ± 0.12</td><td>35.3% ± 0.12</td><td>78.0% ± 0.10</td></tr><tr><td>Nylon</td><td>24.0% ± 0.09</td><td>51.9% ± 0.07</td><td>39.2% ± 0.10</td><td>53.7% ± 0.05</td><td>83.4% ± 0.01</td></tr><tr><td rowspan="3">UD</td><td>p-value</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td><td>&lt;0.001 ***</td></tr></tbody></table>

Note: Data are given as mean ± SD. The significant intergroup differences are indicated by ‘***’ (*p* < 0.001, Kruskal–Wallis test).

Based on the results in Table 4 and Figure 5, the comparisons of the accuracies of the SVM-ML model with respect to fabric stimulations when considering arm differences are presented in Figure 6. In the SA group (Figure 6a), significant differences in accuracies with respect to the fabric stimulations were obtained in the higher frequency bands, i.e., beta (*p* < 0.001, Kruskal–Wallis with Bonferroni post-hoc test) and gamma (*p* < 0.05, Kruskal–Wallis with Bonferroni post-hoc test) bands. No significant differences were found between nylon and wool in the delta, theta, and alpha bands (*p* > 0.05, Kruskal–Wallis with Bonferroni post-hoc test). In the SU group (Figure 6b), significant differences in accuracies with
respect to fabric stimulations were found in the delta ($p < 0.001$, Kruskal–Wallis with Bonferroni post-hoc test), alpha ($p < 0.001$, Kruskal–Wallis with Bonferroni post-hoc test), and beta ($p < 0.05$, Kruskal–Wallis with Bonferroni post-hoc test) bands. No significant difference was observed between cotton and nylon in the theta band ($p > 0.05$, Kruskal–Wallis with Bonferroni post-hoc test). In the UD group (Figure 6c), significant differences in accuracies with respect to fabric stimulations were found in almost all frequency bands ($p < 0.001$, Kruskal–Wallis with Bonferroni post-hoc test), except for the difference between nylon and wool in the gamma band ($p > 0.05$, Kruskal–Wallis with Bonferroni post-hoc test).

<figure><img src="image_13.png" /><figcaption>**Figure 6.** Comparisons of the classification accuracies of the SVM-ML model with respect to fabric stimulations when considering arm differences, i.e., (a) SA, (b) SU, and (c) UD. The significant differences are indicated by '*' for $p < 0.05$ and '****' for $p < 0.001$ (Kruskal–Wallis with Bonferroni post-hoc test).</figcaption></figure>

### 3. Discussion

In this study, the EEG-based SVM-ML model was built using the RSP features during stimulation with cotton fabric as the baseline inputs. The model's generalization performance was investigated by comparing the classification accuracies during stimulations with different fabrics.

#### 3.1. SVM-ML Configuration

##### 3.1.1. RBF Kernel Determination

The grid search results of the hyperparameter pair ($\gamma$, $C$) for the RBF kernel (Figures 3 and 4) showed that the highest classification accuracies were obtained in the predefined range of the hyperparameter pair. Similar boundaries of the hyperparameter pair ($\gamma$, $C$) were also applied in other SVM-based studies, e.g., Chang et al. used the boundaries of ($e^{-8} \le \gamma \le e^8$, $e^{-8} \le C \le e^8$) [66], and Hsu et al. selected the boundaries of ($2^{-15} \le \gamma \le 2^3$, $2^{-5} \le C \le 2^{15}$) [42]. This showed that the selected optimal hyperparameter pair ($\gamma$, $C$) was in the traditional search space, and the SVM-ML model with the general search space was feasible for classifying the RSP features extracted from EEG during sensory assessments. The kernel scaling parameter $\gamma$ determines the complexity of the classification decision function of the model [67]. For smaller values of $\gamma$, the decision function is nearly linear, and for larger values of $\gamma$, the function becomes more curved [67]. The optimal value of $\gamma$ ($2^{1}$) chosen by the model was close to the upper boundary of the preset $\gamma$ range, which
suggested that there was a relatively strong nonlinearity among the EEGs of multiple upper-limb groups in the original feature space, and the model obtained the “curved” decision function by mapping the raw EEG to a higher dimensional space. The regularization parameter C defines the penalty degree of the model for the percentage of deviation from the misclassified trained data [67]. As the value of C increases, its penalty degree for the model becomes larger, and the percentage of deviation of the misclassified data is smaller during the training phase. The optimal value of C (2³) selected by the model was relatively lower compared to the predefined range of C. This suggested that the model tolerated a greater percentage of misclassified training data when searching for the optimal decision function, indicating that there was an overlap among the different groups of EEG data points near the decision function. Meanwhile, the model with the optimal hyperparameter pair (γ, C) achieved an accuracy of 76.8%, which was comparable to those reported in studies on multiclass classification of EEG using SVM-ML models with accuracies exceeding 71.0% [68,69].

### 3.1.2. EEG Channel Selection

It was observed that the overall accuracies of the model with the 21-channel EEG were better than those of the 62-channel EEG when not considering arm differences (Table 2). The 21-channel EEG covers the sensorimotor cortex, which is the main response area to sensory stimulations [70,71]. The accuracies based on the 21-channel EEG suggested that direct cortical processing from the sensorimotor cortex was sufficient to capture the sensory differences generated by different fabric samples through the SVM-ML model. Previous studies have demonstrated that significant RSP variations for different EEG bands during sensory stimulations were mainly captured in the sensorimotor cortex for both unimpaired and stroke populations [28,72,73]. On the contrary, involuntary attention activities beyond the sensorimotor cortex were involved in passive fabric stimulation experiments [28]. This could be a hurdle to the recognition of cortical responses to fabric stimulations using the SVM-ML model. Meanwhile, voluntary cognitive activities were also found to disturb measurement of cortical responses to sensory stimulations [73,74]. For example, in post-stroke sensory evaluation by a subjective questionnaire, stroke persons with sensory impairments could distinguish different fabric stimulations because of the compensation of cognitive processing, e.g., individual experiences, to the residual sensory pathways [28]. In this work, the voluntary cognitive activities were minimized by asking the participants to stay awake but mentally inactive during the fabric stimulations. Therefore, the EEG RSP features of the sensorimotor cortex detected by the 21-channel EEG were sufficient for representing the differences in direct cortical responses to fine tactile sensations.

Table 2 also shows that when identifying different fabric stimulations without considering arm differences, the model achieved better overall accuracies in higher bands, i.e., the beta and gamma bands. This was consistent with the results of previous neurophysiologic research into how the human brain reacts to tactile sensations induced by fabrics [74,75]. The cortical responses of the brain to tactile sensations are elicited by skin–fabric interactions, which are characterized by the EEG beta and gamma band activations [76]. Beta oscillations have been shown to be involved in the phasic locking process between the primary and secondary somatosensory cortex in response to tactile sensation [77]. Meanwhile, it was observed that the neuronal assemblies of the sensorimotor cortex were joined in large-scale networks oscillating in the beta band during maintenance of a sustained hand lever press activity [78]. This phenomenon suggested that the primary somatosensory and primary motor cortex were bound together in a beta-synchronized cortical network [78]. Furthermore, Greco et al. found that beta oscillations in the sensorimotor cortex served as an informative feature characterizing affective tactile stimulation by interactions with different fabrics [79]. Singh et al. demonstrated that pleasant and unpleasant tactile sensations present different beta-oscillation patterns [74]. The gamma oscillations of the sensorimotor cortex were also observed in response to tactile sensation. This may reflect the timing code and temporal organization for higher-order somatosensory processing, which is important
for sensory binding [80,81]. In addition, the study by Aya et al. showed that gamma oscillations were simultaneously evoked in the primary and secondary somatosensory cortex during sensory stimulations, thus suggesting that it is critical for forming functional cortico-cortical connections and for conveying somatosensory information from the primary and secondary somatosensory cortex [82]. Bauer et al. found that tactile-stimuli-induced gamma oscillations in the somatosensory cortex were enhanced and prolonged by spatial tactile attention [83]. This indicated that gamma-band synchronization was instrumental in the somatosensory system for processing behaviorally relevant stimulations [83]. Therefore, the RSP variations in the beta and gamma bands were sensitive input features for the SVM-ML model for both unimpaired and stroke persons.

## 3.2. K-Fold Cross Validation

During determination of k in CV, the model attained the highest accuracy with six-fold CV in the gamma band (Table 3). This was also observed in terms of approximately identical accuracies of the different numbers of k in the gamma band, which indicated that the SVM-ML model achieved stable classification performance with good generalization capacity for different combinations of training and testing datasets owing to the different k-fold CV strategies [84]. Furthermore, the model with the leave-one-out CV achieved an accuracy similar to the k-fold CV technique in the gamma band. This demonstrated the model's unbiased evaluation capability as a special case of k-fold CV, in which each sample has a chance to represent the entire test dataset [85]. However, the computational cost of the leave-one-out CV was greater than those of other configurations of k in the k-fold CV when evaluating the SVM-ML model performance. This was in line with previous studies [85,86] that investigated the computational efficiencies of k-fold and leave-one-out CV. Thus, it was preferable to use the optimal k-fold CV, i.e., six-fold CV during model evaluations.

# 4. Generalization of the SVM-ML Model

## 4.1. Different Fabric Stimulations

In the evaluation of model generalization, the wool and nylon fabrics in the gamma band achieved significantly higher accuracies than that of cotton fabric (Table 4 and Figure 5). This was attributable to the differences in their stimulation intensities on the skin. According to the study by Chen et al., neural oscillations in higher frequency bands, e.g., gamma band, were lower when executing an easy task; however, they increased to higher levels to obtain more information from the sensory environment when the task was difficult [87]. Cotton is the most familiar fabric that is in direct contact with skin in daily life, and it provides the lowest stimulation intensity during passive involuntary touch [28]. However, wool and nylon offer more stimulating sensory experiences because of their textile physical properties, which may require additional neural effort and cortical resources to evoke cortical responses to stimulations [28]. This was further supported by the study by Jiao et al., who found that wool elicited a relatively intense tactile stimulation in the form of scratching, resulting in the sensation of discomfort [88]. They also discovered that touching wool fabric elicited higher EEG RSP responses than cotton and nylon fabrics [88]. Hoefer et al. also observed that nylon induced significant higher event-related potential (ERP) signals than cotton, implying that there was less distraction and better cortical resources during tactile sensation [76]. As a result, the model achieved relatively higher accuracies with the RSP features of nylon and wool compared to cotton.

## 4.2. Different Upper-Limb Groups

When considering arm differences, the model yields various patterns for the comparison of accuracies between stimulations with different fabrics (Table 5 and Figure 6). It was observed that the significant differences in the accuracies for classifying post-stroke stimulations with different fabrics shifted to higher frequency bands, i.e., the beta and gamma bands, compared to the UD group. This pattern difference in the classification of stimulations with different fabrics among the different upper-limb groups was similar to
that for manual investigations comparing the EEG RSP differences between stroke and unimpaired persons. In the manual evaluations, the post-stroke representative power spectra to fine touch stimulation shifted to higher frequency bands, i.e., the beta and gamma bands [28]. The sensitivity of the EEG-based RSP features and their capability for feature mapping by the RBF kernel allowed the SVM-ML model to detect similar pattern as manual evaluations. As the input features of the model, the average and maximal values of the RSPs represent the significant differences in RSPs among multiple upper-limb groups. The differences in the EEG RSP patterns in response to fabric stimulations have been discovered in previous manual investigations and were mainly related to neuroplastic changes after stroke [89]. For example, when the damage to the brain neurons result in post-stroke sensorimotor function deficiencies, the cortex composed of various neural subsets could be rewired [90,91]. Neural compensation to lesional functions can lead to redistributed patterns of the cortical responses to external stimulations [92]. Meanwhile, the SVM with RBF kernel can find the optimal decision boundary among multiple upper-limb groups owing to its sufficient feature mapping capability. It implicitly transforms the original RSP features to a high-dimensional feature space with fewer hyperparameters to be determined, which guarantees the generalization capability of the model when supplying new input data [37]. Previous studies have also demonstrated that the SVM with RBF kernel achieves minimal classification error rates in different clinical scenarios while controlling the complexity of the model [36,93,94]. Therefore, based on the sensitivity of the RSP features and the feature mapping capability of the RBF kernel, the proposed model was expected to achieve similar performance as in manual inspection for distinguishing between unimpaired and post-stroke persons.

## 5. Conclusions

In this study, an EEG-based SVM-ML model was established using the RSP features of the EEG signals, i.e., RSP<sub>mean</sub> and RSP<sub>max</sub>, during stimulation with cotton fabric as the baseline input. The observations demonstrated that the RSP<sub>mean</sub> and RSP<sub>max</sub> were sensitive to fabric stimulations and could be used as representative input features to the model. The generalization performance of the model was investigated by comparing the classification accuracies during stimulations with different fabrics while considering arm differences. The model determined that the significant differences in the accuracies of fabric stimulations after stroke were shifted toward higher bands, i.e., beta and gamma bands, similar to the differences in RSP patterns between post-stroke persons and unimpaired participants as in manual investigations, thereby implying that the model could imitate manual evaluations of cortical responses to fabric stimulations; this ability is expected to aid in automatic assessments of post-stroke fine tactile sensation.

**Author Contributions:** Conceptualization, J.Z. and X.H.; methodology, J.Z. and X.H.; software, J.Z.; validation, Z.L. and J.Z.; data curation, J.Z. and Y.H.; writing—original draft preparation, J.Z.; writing—review and editing, J.Z., Y.H., F.Y., B.Y. and X.H.; supervision, X.H.; funding acquisition, X.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, China (NSFC 81771959), the University Grants Committee Research Grants Council, Hong Kong (GRF 15207120), and the Science and Technology Innovation Committee of Shenzhen, China (2021Szuup142).

**Institutional Review Board Statement:** The human experiments were conducted after we obtained the ethical approval from the Human Subjects Ethics Sub-Committee of the Hong Kong Polytechnic University.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data are available upon request.

**Acknowledgments:** The authors would like to thank the participants who participated in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.
References

1. Kessner, S.S.; Bingel, U.; Thomalla, G. Somatosensory deficits after stroke: A scoping review. *Top. Stroke Rehabil.* **2016**, *23*, 136–146.
   [CrossRef] [PubMed]

2. Sherwood, L. *Human Physiology: From Cells to Systems*; Cengage Learning: Boston, MA, USA, 2015.

3. Carey, L.M.; Matyas, T.A.; Baum, C. Effects of somatosensory impairment on participation after stroke. *Am. J. Occup. Ther.* **2018**, *72*, 7203205100p1–7203205100p10. [CrossRef] [PubMed]

4. Tyson, S.F.; Crow, J.L.; Connell, L.; Winward, C.; Hillier, S. Sensory impairments of the lower limb after stroke: A pooled analysis of individual patient data. *Top. Stroke Rehabil.* **2013**, *20*, 441–449. [CrossRef] [PubMed]

5. Carey, L.M.; Matyas, T.A.; Oke, L.E. Sensory loss in stroke patients: Effective training of tactile and proprioceptive discrimination. *Arch. Phys. Med. Rehabil.* **1993**, *74*, 602–611. [CrossRef]

6. Matsuda, K.; Satoh, M.; Tabei, K.-i.; Ueda, Y.; Taniguchi, A.; Matsuura, K.; Asahi, M.; Ii, Y.; Niwa, A.; Tomimoto, H. Impairment of intermediate somatosensory function in corticobasal syndrome. *Sci. Rep.* **2020**, *10*, 11155. [CrossRef]

7. Cunha, B.P.; Alouche, S.R.; Araujo, I.M.G.; Freitas, S.M.S.F. Individuals with post-stroke hemiparesis are able to use additional sensory information to reduce postural sway. *Neurosci. Lett.* **2012**, *513*, 6–11. [CrossRef]

8. Boonsinsukh, R.; Panichareon, L.; Phansuwan-Pujito, P. Light touch cue through a cane improves pelvic stability during walking in stroke. *Arch. Phys. Med. Rehabil.* **2009**, *90*, 919–926. [CrossRef]

9. Johannsen, L.; Wing, A.M.; Hatzitaki, V. Effects of maintaining touch contact on predictive and reactive balance. *J. Neurophysiol.* **2007**, *97*, 2686–2695. [CrossRef]

10. Campfens, S.F.; Zandvliet, S.B.; Meskers, C.G.; Schouten, A.C.; van Putten, M.J.; van der Kooij, H. Poor motor function is associated with reduced sensory processing after stroke. *Exp. Brain Res.* **2015**, *233*, 1339–1349. [CrossRef]

11. Doyle, S.D.; Bennett, S.; Dudgeon, B. Upper limb post-stroke sensory impairments: The survivor's experience. *Disabil. Rehabil. Ther.* **2014**, *36*, 993–1000. [CrossRef]

12. Pandyan, A.D.; Johnson, G.R.; Price, C.I.M.; Curless, R.H.; Barnes, M.P.; Rodgers, H. A review of the properties and limitations of the Ashworth and modified Ashworth Scales as measures of spasticity. *Clin. Rehabil.* **1999**, *13*, 373–383. [CrossRef] [PubMed]

13. Fugl-Meyer, A.R.; Jääskö, L.; Leyman, I.; Olsson, S.; Steglind, S. The post-stroke hemiplegic patient. 1. a method for evaluation of physical performance. *Scand. J. Rehabil. Med.* **1975**, *7*, 13–31. [PubMed]

14. Winward, C.E.; Halligan, P.W.; Wade, D.T. The Rivermead Assessment of Somatosensory Performance (RASP): Standardization and reliability data. *Clin. Rehabil.* **2002**, *16*, 523–533. [CrossRef] [PubMed]

15. Pan, N. Quantification and evaluation of human tactile sense towards fabrics. *Int. J. Des. Nat. Ecodyn.* **2006**, *1*, 48–60.

16. Auriat, A.M.; Neva, J.L.; Peters, S.; Ferris, J.K.; Boyd, L.A. A Review of Transcranial Magnetic Stimulation and Multimodal Neuroimaging to Characterize Post-Stroke Neuroplasticity. *Front. Neurol.* **2015**, *6*, 226. [CrossRef]

17. Lundgren, J.; Flodström, K.; Sjögren, K.; Liljequist, B.; Fugl-Meyer, A.R. Site of brain lesion and functional capacity in rehabilitated hemiplegics. *Scand. J. Rehabil. Med.* **1982**, *14*, 141–143.

18. Chollet, F.; DiPiero, V.; Wise, R.J.; Brooks, D.J.; Dolan, R.J.; Frackowiak, R.S. The functional anatomy of motor recovery after stroke in humans: A study with positron emission tomography. *Ann. Neurol.* **1991**, *29*, 63–71. [CrossRef]

19. Giaquinto, S.; Cobianchi, A.; Macera, F.; Nolfe, G. EEG recordings in the course of recovery from stroke. *Stroke* **1994**, *25*, 2204–2209.
   [CrossRef]

20. Broderick, P.A.; Kolodny, E.H. Biosensors for brain trauma and dual laser doppler flowmetry: Enoxaparin simultaneously reduces stroke-induced dopamine and blood flow while enhancing serotonin and blood flow in motor neurons of brain, in vivo. *Sensors* **2010**, *11*, 138–161. [CrossRef]

21. Radaelli, A.; Mancia, G.; Ferrarese, C.; Beretta, S. *New Concepts in Stroke Diagnosis and Therapy*; Bentham Science Publishers: Milan, Italy, 2017; Volume 1.

22. Chen, Q.; Xia, T.; Zhang, M.; Xia, N.; Liu, J.; Yang, Y. Radiomics in stroke neuroimaging: Techniques, applications, and challenges. *Aging Dis.* **2021**, *12*, 143. [CrossRef]

23. Militello, C.; Rundo, L.; Dimarco, M.; Orlando, A.; Woitek, R.; D'Angelo, I.; Russo, G.; Bartolotta, T.V. 3D DCE-MRI Radiomic Analysis for Malignant Lesion Prediction in Breast Cancer Patients. *Acad. Radiol.* **2021**, *29*, 830–840. [CrossRef] [PubMed]

24. Sharaev, M.; Andreev, A.; Artemov, A.; Burnaev, E.; Kondratyeva, E.; Sushchinskaya, S.; Samotaeva, I.; Gaskin, V.; Bernstein, A. *Pattern Recognition Pipeline for Neuroimaging Data*; Springer: Berlin/Heidelberg, Germany, 2018.

25. Caliandro, P.; Vecchio, F.; Miraglia, F.; Reale, G.; Della Marca, G.; La Torre, G.; Lacidogna, G.; Iacovelli, C.; Padua, L.; Bramanti, P. Small-world characteristics of cortical connectivity changes in acute stroke. *Neurorehabil. Neural Repair* **2017**, *31*, 81–94. [CrossRef] [PubMed]

26. Bentes, C.; Peralta, A.R.; Viana, P.; Martins, H.; Morgado, C.; Casimiro, C.; Franco, A.C.; Fonseca, A.C.; Geraldes, R.; Canhão, P. Quantitative EEG and functional outcome following acute ischemic stroke. *Clin. Neurophysiol.* **2018**, *129*, 1680–1687. [CrossRef] [PubMed]

27. Ahn, S.-N.; Lee, J.-W.; Hwang, S. Tactile Perception for Stroke Induce Changes in Electroencephalography. *Hong Kong J. Occup. Ther.* **2016**, *28*, 1–6. [CrossRef] [PubMed]

28. Huang, Y.; Jiao, J.; Hu, J.; Hsing, C.; Lai, Z.; Yang, Y.; Hu, X.J. Measurement of sensory deficiency in fine touch after stroke during textile fabric stimulation by electroencephalography (EEG). *J. Neural Eng.* **2020**, *17*, 045007. [CrossRef]
29. Roy, S.; Kiral-Kornek, I.; Harrer, S. ChronoNet: A deep recurrent neural network for abnormal EEG identification. In *Conference on Artificial Intelligence in Medicine in Europe*; Springer: Berlin/Heidelberg, Germany, 2019.

30. Golmohammadi, M.; Ziyabari, S.; Shah, V.; de Diego, S.L.; Obeid, I.; Picone, J. Deep architectures for automated seizure detection in scalp EEGs. *arXiv* **2017**, arXiv:1712,09776.

31. Sirsat, M.S.; Fermé, E.; Câmara, J. Machine Learning for Brain Stroke: A Review. *J. Stroke Cerebrovasc. Dis.* **2020**, *29*, 105162. [CrossRef]

32. Hosseini, M.-P.; Hemingway, C.; Madamba, J.; McKee, A.; Ploof, N.; Schuman, J.; Voss, E. Review of Machine Learning Algorithms for Brain Stroke Diagnosis and Prognosis by EEG Analysis. *arXiv* **2020**, arXiv:2008.08118.

33. Jochumsen, M.; Rovsing, C.; Rovsing, H.; Niazi, I.K.; Dremstrup, K.; Kamavuako, E.N. Classification of Hand Grasp Kinetics and Types Using Movement-Related Cortical Potentials and EEG Rhythms. *Comput. Intell. Neurosci.* **2017**, *2017*, 7470864. [CrossRef]

34. Usama, N.; Niazi, I.K.; Dremstrup, K.; Jochumsen, M. Detection of Error-Related Potentials in Stroke Patients from EEG Using an Artificial Neural Network. *Sensors* **2021**, *21*, 6274. [CrossRef]

35. Iáñez, E.; Azorín, J.M.; Úbeda, A.; Fernández, E.; Sirvent, J.L. LDA-based classifiers for a mental tasks-based brain-computer interface. In Proceedings of the 2010 IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey, 10–13 October 2010.

36. Garrett, D.; Peterson, D.A.; Anderson, C.W.; Thaut, M.H. Comparison of linear, nonlinear, and feature selection methods for EEG signal classification. *IEEE Trans. Neural Syst. Rehabil. Eng.* **2003**, *11*, 141–144. [CrossRef]

37. Cao, J.; Fang, Z.; Qu, G.; Sun, H.; Zhang, D. An accurate traffic classification model based on support vector machines. *Int. J. Netw. Manag.* **2017**, *27*, e1962. [CrossRef]

38. Vapnik, V. *The Nature of Statistical Learning Theory*; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2013.

39. Liu, Y.; Zhang, H.; Chen, M.; Zhang, L. A boosting-based spatial-spectral model for stroke patients’ EEG analysis in rehabilitation training. *IEEE Trans. Neural Syst. Rehabil. Eng.* **2015**, *24*, 169–179. [CrossRef]

40. Ghumman, M.K.; Singh, S.; Singh, N.; Jindal, B. Optimization of parameters for improving the performance of EEG-based BCI system. *J. Reliab. Intell. Environ.* **2021**, *7*, 145–156. [CrossRef]

41. Bousseta, R.; Tayeb, S.; El Ouakouak, I.; Gharbi, M.; Regragui, F.; Himmi, M.M. EEG efficient classification of imagined hand movement using RBF kernel SVM. In Proceedings of the 2016 11th International Conference on Intelligent Systems: Theories and Applications (SITA), Mohammadia, Morocco, 19–20 October 2016.

42. Hsu, C.-W.; Chang, C.-C.; Lin, C.-J. *A Practical Guide to Support Vector Classification*; National Taiwan University: Taipei, Taiwan, 2003.

43. Farid, N.; Elbagoury, B.; Roushdy, M.; Salem, A.-B.M. A comparative analysis for support vector machines for stroke patients. *Recent Adv. Inf. Sci.* **2013**, *41*, 71–76.

44. Kim, M.-K.; Cho, J.-H.; Jeong, J.-H. Classification of Tactile Perception and Attention on Natural Textures from EEG Signals. In Proceedings of the 2021 9th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Korea, 22–24 February 2021.

45. Lilliefors, H.W. On the Kolmogorov-Smirnov test for the exponential distribution with mean unknown. *J. Am. Stat. Assoc.* **1969**, *64*, 387–389. [CrossRef]

46. Homan, R.W.; Herman, J.; Purdy, P. Cerebral location of international 10–20 system electrode placement. *Electroencephalogr. Clin. Neurophysiol.* **1987**, *66*, 376–382. [CrossRef]

47. Welch, P. The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms. *IEEE Trans. Audio Electroacoust.* **1967**, *15*, 70–73. [CrossRef]

48. Teplan, M. Fundamentals of EEG measurement. *Meas. Sci. Rev.* **2002**, *2*, 1–11.

49. Bronzino, J.D. *Biomedical Engineering Handbook 2*; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2000; Volume 2.

50. Patro, S.; Sahu, K.K. Normalization: A preprocessing stage. *arXiv* **2015**, arXiv:1503,06462. [CrossRef]

51. Liao, X.; Li, Y.; Hu, J.; Li, Q.; Wu, X. Psychophysical Relations between Interacted Fabric Thermal-Tactile Properties and Psychological Touch Perceptions. *J. Sens. Stud.* **2016**, *31*, 181–192. [CrossRef]

52. Hu, J.Y.; Hes, L.; Li, Y.; Yeung, K.W.; Yao, B.G. Fabric Touch Tester: Integrated evaluation of thermal–mechanical sensory properties of polymeric materials. *Polym. Test.* **2006**, *25*, 1081–1090. [CrossRef]

53. Bishop, C.M. *Pattern Recognition and Machine Learning*; Springer: Berlin/Heidelberg, Germany, 2006.

54. Syarif, I.; Prugel-Bennett, A.; Wills, G. SVM parameter optimization using grid search and genetic algorithm to improve classification performance. *Telkomnika* **2016**, *14*, 1502. [CrossRef]

55. Zhang, Y.; Ji, X.; Liu, B.; Huang, D.; Xie, F.; Zhang, Y. Combined feature extraction method for classification of EEG signals. *Neural Comput. Appl.* **2017**, *28*, 3153–3161. [CrossRef]

56. Guler, I.; Ubeyli, E.D. Multiclass support vector machines for EEG-signals classification. *IEEE Trans. Inf. Technol. Biomed.* **2007**, *11*, 117–126. [CrossRef]

57. Avelino, J.; Paulino, T.; Cardoso, C.; Moreno, P.; Bernardino, A. Human-aware natural handshaking using tactile sensors for Vizzy, a social robot. In Proceedings of the Workshop on Behavior Adaptation, Interaction and Learning for Assistive Robotics at RO-MAN, Lisbon, Portugal, 28 August–1 September 2017.

58. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V. Scikit-learn: Machine learning in Python. *J. Mach. Learn. Res.* **2011**, *12*, 2825–2830.
59. Kattenstroth, J.-C.; Kalisch, T.; Peters, S.; Tegenthoff, M.; Dinse, H. Long-term sensory stimulation therapy improves hand function and restores cortical responsiveness in patients with chronic cerebral lesions. Three single case studies. *Front. Hum. Neurosci.* **2012**, *6*, 244. [CrossRef]

60. Pfurtscheller, G.; Neuper, C. Motor imagery activates primary sensorimotor area in humans. *Neurosci. Lett.* **1997**, *239*, 65–68. [CrossRef]

61. James, G.; Witten, D.; Hastie, T.; Tibshirani, R. *An Introduction to Statistical Learning*; Springer: Berlin/Heidelberg, Germany, 2013; Volume 112.

62. Breiman, L. Heuristics of instability and stabilization in model selection. *Ann. Stat.* **1996**, *24*, 2350–2383. [CrossRef]

63. Witten, I.H.; Frank, E. Data mining: Practical machine learning tools and techniques with Java implementations. *Acm Sigmod Rec.* **2002**, *31*, 76–77. [CrossRef]

64. Bouckaert, R.R. Choosing between two learning algorithms based on calibrated tests. *ICML* **2003**, *3*, 51–58.

65. Kuhn, M.; Johnson, K. *Applied Predictive Modeling*; Springer: Berlin/Heidelberg, Germany, 2013; Volume 26.

66. Chang, M.-W.; Lin, C.-J. Leave-one-out bounds for support vector regression model selection. *Neural Comput.* **2005**, *17*, 1188–1222. [CrossRef]

67. Burges, C.J.C. A tutorial on support vector machines for pattern recognition. *Data Min. Knowl. Discov.* **1998**, *2*, 121–167. [CrossRef]

68. Vivaldi, N.; Caiola, M.; Solarana, K.; Ye, M. Evaluating performance of eeg data-driven machine learning for traumatic brain injury classification. *IEEE Trans. Biomed. Eng.* **2021**, *68*, 3205–3216. [CrossRef]

69. Gao, L.; Cheng, W.; Zhang, J.; Wang, J. EEG classification for motor imagery and resting state in BCI applications using multi-class Adaboost extreme learning machine. *Rev. Sci. Instrum.* **2016**, *87*, 085110. [CrossRef] [PubMed]

70. Kira, K.; Rendell, L.A. A practical approach to feature selection. In *Machine Learning Proceedings*; Elsevier: Amsterdam, The Netherlands, 1992; pp. 249–256.

71. Kitada, R.; Hashimoto, T.; Kochiyama, T.; Kito, T.; Okada, T.; Matsumura, M.; Lederman, S.J.; Sadato, N. Tactile estimation of the roughness of gratings yields a graded response in the human brain: An fMRI study. *Neuroimage* **2005**, *25*, 90–100. [CrossRef]

72. Wu, J.; Srinivasan, R.; Quinlan, E.B.; Solodkin, A.; Small, S.L.; Cramer, T.C. Utility of EEG measures of brain function in patients with acute stroke. *J. Neurophysiol.* **2016**, *115*, 2399–2405. [CrossRef]

73. Singh, H.; Bauer, M.; Chowanski, W.; Sui, Y.; Atkinson, D.; Baurley, S.; Fry, M.; Evans, J.; Bianchi-Berthouze, N. The brain's response to pleasant touch: An EEG investigation of tactile caressing. *Front. Hum. Neurosci.* **2014**, *8*, 893. [CrossRef]

74. Merabet, L.B.; Pascual-Leone, A. Neural reorganization following sensory loss: The opportunity of change. *Nat. Rev. Neurosci.* **2010**, *11*, 44–52. [CrossRef]

75. Ackerley, R.; Carlsson, I.; Wester, H.; Olausson, H.; Backlund Wasling, H. Touch perceptions across skin sites: Differences between sensitivity, direction discrimination and pleasantness. *Front. Behav. Neurosci.* **2014**, *8*, 54. [CrossRef]

76. Hoefer, D.; Handel, M.; Müller, K.M.; Hammer, T.R. Electroencephalographic study showing that tactile stimulation by fabrics of different qualities elicit graded event-related potentials. *Ski. Res. Technol.* **2016**, *22*, 470–478. [CrossRef] [PubMed]

77. Simões, C.; Jensen, O.; Parkkonen, L.; Hari, R. Phase locking between human primary and secondary somatosensory cortices. *Proc. Natl. Acad. Sci. USA* **2003**, *100*, 2691–2694. [CrossRef] [PubMed]

78. Brovelli, A.; Ding, M.; Ledberg, A.; Chen, Y.; Nakamura, R.; Bressler, S.L. Beta oscillations in a large-scale sensorimotor cortical network: Directional influences revealed by Granger causality. *Proc. Natl. Acad. Sci. USA* **2004**, *101*, 9849–9854. [CrossRef] [PubMed]

79. Greco, A.; Guidi, A.; Bianchi, M.; Lanata, A.; Valenza, G.; Scilingo, E.P. Brain dynamics induced by pleasant/unpleasant tactile stimuli conveyed by different fabrics. *IEEE J. Biomed. Health Inform.* **2019**, *23*, 2417–2427. [CrossRef] [PubMed]

80. Hasenstaub, A.; Shu, Y.; Haider, B.; Kraushaar, U.; Duque, A.; McCormick, D.A. Inhibitory postsynaptic potentials carry synchronized frequency information in active cortical networks. *Neuron* **2005**, *47*, 423–435. [CrossRef]

81. Engel, A.K.; Singer, W. Temporal binding and the neural correlates of sensory awareness. *Trends Cogn. Sci.* **2001**, *5*, 16–25. [CrossRef]

82. Ihara, A.; Hirata, M.; Yanagihara, K.; Ninomiya, H.; Imai, K.; Ishii, R.; Osaki, Y.; Sakihara, K.; Izumi, H.; Imaoka, H.; et al. Neuromagnetic gamma-band activity in the primary and secondary somatosensory areas. *NeuroReport* **2003**, *14*, 273–277. [CrossRef] [PubMed]

83. Bauer, M.; Oostenveld, R.; Peeters, M.; Fries, P. Tactile spatial attention enhances gamma-band activity in somatosensory cortex and reduces low-frequency activity in parieto-occipital areas. *J. Neurosci.* **2006**, *26*, 490–501. [CrossRef]

84. Xiong, Z.; Cui, Y.; Liu, Z.; Zhao, Y.; Hu, M.; Hu, J. Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation. *Comput. Mater. Sci.* **2020**, *171*, 109203. [CrossRef]

85. Refaeilzadeh, P.; Tang, L.; Liu, H. Cross-validation. *Encycl. Database Syst.* **2009**, *5*, 532–538.

86. Jung, Y. Multiple predicting K-fold cross-validation for model selection. *J. Nonparametr. Stat.* **2018**, *30*, 197–215. [CrossRef]

87. Chen, A.; Wang, A.; Wang, T.; Tang, X.; Zhang, M. Behavioral oscillations in visual attention modulated by task difficulty. *Front. Psychol.* **2017**, *8*, 1630. [CrossRef] [PubMed]

88. Jiao, J.; Hu, X.; Huang, Y.; Hu, J.; Hsing, C.; Lai, Z.; Wong, C.; Xin, J.H. Neuro-perceptive discrimination on fabric tactile stimulation by Electroencephalographic (EEG) spectra. *PLoS ONE* **2020**, *15*, e0241378. [CrossRef] [PubMed]

89. Snyder, D.B.; Schmit, B.D.; Hyngstrom, A.S.; Beardsley, S.A. Electroencephalography resting-state networks in people with Stroke. *Brain Behav.* **2021**, *11*, e02097. [CrossRef]
90. Dąbrowski, J.; Czajka, A.; Zielińska-Turek, J.; Jaroszyński, J.; Furtak-Niczyporuk, M.; Mela, A.; Poniatowski, Ł.A.; Drop, B.; Dorobek, M.; Barcikowska-Kotowicz, M. Brain functional reserve in the context of neuroplasticity after stroke. *Neural Plast.* **2019**, *9708905*. [CrossRef]

91. Voss, P.; Thomas, M.E.; Cisneros-Franco, J.M.; de Villers-Sidani, É. Dynamic brains and the changing rules of neuroplasticity: Implications for learning and recovery. *Front. Psychol.* **2017**, *8*, 1657. [CrossRef]

92. Lin, M.P.; Liebeskind, D.S. Imaging of ischemic stroke. *Contin. Lifelong Learn. Neurol.* **2016**, *22*, 1399. [CrossRef]

93. Sun, S.; Zhang, C. Adaptive feature extraction for EEG signal classification. *Med. Biol. Eng. Comput.* **2006**, *44*, 931–935. [CrossRef]

94. Al-Qazzaz, N.K.; Ali, S.H.B.M.; Ahmad, S.A.; Islam, M.S.; Escudero, J. Discrimination of stroke-related mild cognitive impairment and vascular dementia using EEG signal analysis. *Med. Biol. Eng. Comput.* **2018**, *56*, 137–157. [CrossRef]
<figure><img src="image_14.png" /></figure>

Article

# Unsupervised Segmentation in NSCLC: How to Map the Output of Unsupervised Segmentation to Meaningful Histological Labels by Linear Combination?

Cleo-Aron Weis ¹,*,†, Kian R. Weihrauch ¹,†, Katharina Kriegsmann ¹,²,‡ and Mark Kriegsmann ³,‡

¹ Institute of Pathology, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University, 68167 Mannheim, Germany; kianw@umich.edu (K.R.W.); katharina.kriegsmann@med.uni-heidelberg.de (K.K.)

² Department of Hematology, Oncology and Rheumatology, University Hospital Heidelberg, 69120 Heidelberg, Germany

³ Institute of Pathology, University Medical Hospital Heidelberg, Heidelberg University, 69120 Heidelberg, Germany; mark.kriegsmann@med.uni-heidelberg.de

* Correspondence: cleo-aron.weis@medma.uni-heidelberg.de; Tel.: +49-621-383-4072

† Current address: Institute of Pathology, Medical Faculty Mannheim, Heidelberg University, 68167 Mannheim, Germany.

‡ These authors contributed equally to this work.

**Abstract:** Background: Segmentation is, in many Pathomics projects, an initial step. Usually, in supervised settings, well-annotated and large datasets are required. Regarding the rarity of such datasets, unsupervised learning concepts appear to be a potential solution. Against this background, we tested for a small dataset on lung cancer tissue microarrays (TMA) if a model (i) first can be in a previously published unsupervised setting and (ii) secondly can be modified and retrained to produce meaningful labels, and (iii) we finally compared this approach to standard segmentation models. Methods: (ad i) First, a convolutional neuronal network (CNN) segmentation model is trained in an unsupervised fashion, as recently described by Kanezaki et al. (ad ii) Second, the model is modified by adding a remapping block and is retrained on an annotated dataset in a supervised setting. (ad iii) Third, the segmentation results are compared to standard segmentation models trained on the same dataset. Results: (ad i–ii) By adding an additional mapping-block layer and by retraining, models previously trained in an unsupervised manner can produce meaningful labels. (ad iii) The segmentation quality is inferior to standard segmentation models trained on the same dataset. Conclusions: Unsupervised training in combination with subsequent supervised training offers for histological images here no benefit.

**Keywords:** histopathology; lung cancer; supervised segmentation; unsupervised segmentation

**Citation:** Weis, C.-A.; Weihrauch, K.R.; Kriegsmann, K.; Kriegsmann, M. Unsupervised Segmentation in NSCLC: How to Map the Output of Unsupervised Segmentation to Meaningful Histological Labels by Linear Combination? *Appl. Sci.* **2022**, *12*, 3718. https://doi.org/10.3390/app12083718

Academic Editors: Leonardo Rundo and Andrea Prati

Received: 9 February 2022

Accepted: 29 March 2022

Published: 7 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_15.png" /></figure>

Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

## 1. Introduction

After the emergence of immunohistochemistry in the 1980s, molecular pathology in the 2000s, and next-generation sequencing in the 2010s, the implementation of image analysis tools into the methodical arsenal of pathology appears to be the next level of development. Digital Pathology, Computational Pathology, and Pathomics are several names for this new branch of expertise, and each term represents a slightly different focus [1–3]. Pathomics, for example, focuses on the extraction of image features that can act as biomarkers in the context of, e.g., neoplastic diseases. In this context, image segmentation is one of the early but essential steps. On the basis of segmented images, image features are extracted and used in further analysis [1]. With machine learning-based image segmentation techniques such as convolutional neuronal networks (CNNs), high-quality and reliable image segmentation is possible. These CNN-based segmentation
approaches typically comprise four development phases: Phase 1 is the creation of a well-labelled dataset; phase 2 is the choice of the model architecture; phase 3 is the design or choice of an appropriate loss function; and phase 4 is choosing or defining an appropriate optimiser [4]. For phase 1, in a usual supervised setting, to avoid overfitting, typically, large annotated datasets are necessary. Creating a representative, large training database tends to be tedious, especially the segmentation tasks; therefore, good datasets are scarce [3,5–7]. To overcome this limitation, several publicly available databases are available online—for example, the Atlas of Digital Pathology [8]. Unfortunately, such databases do not help with more specific questions than segmenting different, non-neoplastic tissues. In addition, rare entities cannot be covered. Many technically different methods have been implemented to overcome the dependency on laboriously generated huge databases. These methods either reduce the number of annotated data needed or are completely independent of labelled data. In addition to approaches based on generated features, of particular interest in this study are machine learning methods that learn the features independently [5,9,10]. On one hand, some methods apply machine learning on small datasets, such as few-shot learning [11,12] or zero-shot learning [13,14]. On the other hand, there are completely unsupervised learning methods for classification or segmentation tasks [5].

Against this background, recent publications by Kanezaki et al. on unsupervised image segmentation are of substantial interest. They describe a framework to train CNN segmentation models that is completely unsupervised [15,16].

In this work, (i) we tested this approach for the segmentation of non-small cell lung carcinoma in tissue microarrays as an example. Furthermore, (ii) we addressed the problem that unsupervised segmentation leads to undefined labels. To map the labels by the unsupervised training to known, meaningful labels (e.g., adenocarcinoma), we tested a second training step with a small human-labelled dataset.

## 2. Materials and Methods

### 2.1. Data Collection and Management

Whole-slide tissue specimens of formalin-fixed paraffin-embedded tumour tissue and tissue microarrays (TMAs) were retrieved (Institute of Pathology, Medical Faculty Heidelberg, Heidelberg University) and used in a completely anonymous manner. No patient information—for example, age or sex—was included. Only the histological diagnoses (e.g., normal lung tissue, adenocarcinoma of the lung, squamous cell carcinoma of the lung) were used. This study was approved by the local ethics committee (#S-207/2005 and #S315/2020).

### 2.2. Whole-Slide Image and TMA Image Preparation

The whole tissue sections (haematoxylin–eosin(HE)-stained) and TMAs (HE-stained and stained by immunohistochemistry (IHC) for panCK) were scanned by a Leica whole-slide scanner or by a PreciPoint M8-scanner. The resulting whole-slide images were saved in the .svs format. For model training and validation, the TMA cores were automatically cropped and saved to 2600 × 2600 pixel-sized images by using QuPath implemented functions [17]. The whole-slide images were automatically cropped into tiles of the same size by a QuPath script published by Peter Bankhead.

### 2.3. Training and Validation Dataset (Dataset #1)

Dataset #1 was used for model training and validation. It is based on IHC-HE-TMA core pairs. In this case, every core has a clinical label (normal tissue, adenocarcinoma, squamous cell carcinoma). This dataset was created in a multi-step approach: Step #1: From the included TMA paraffin block, two subsequent sections were produced: the first was HE-stained and the second was IHC-stained (panCK). Both slides were scanned, and the TMA cores were extracted as described above. Step #2: On the basis of their location on the TMA grid, the HE- and the IHC-stained cores can be assigned to each other (e.g., TMA grid position A-1 in HE stain corresponds to TMA grid position A-1 in panCK stain). Next,
these images, containing a single TMA core each, were registered, resulting in the IHC-HE-
TMA core pair. For registration, the airlab tool published by Sandkuehler et al. was used
(https://github.com/airlab-unibas/airlab accessed on 1 February 2022). Step #3: The IHC-
positive area of each image (containing epithelium) was extracted by using a combination
of colour deconvolution [18] and thresholding, resulting in a map for background tissue
and IHC-positive tissue. Based on the clinical annotation (every TMA grid position is
assigned to one case), the IHC-positive areas are assigned to the defined labels: 1 non-
tumourous tissue (NT), 2 adenocarcinoma (ADC), and 3 squamous cell carcinoma (SqCC).
IHC-negative areas are assigned based on thresholding to the labels background (0) and
non-tumourous tissue (1). Notably, based on this assignment approach, IHC-positive
epithelium in normal tissue is labelled 1 together with the IHC-negative tissue in the same
cases. The label ratio between the area per label is approximately 19.2 (background (BG))
to 4.7 (normal tissue or non-tumourous tissue (NT)) to 1.5 (adenocarcinoma (ADC)) to 1.0
(sqqmous cell carcinoma (SqCC)). Because of the image pair production in steps #1 and
#2, the labels produced in step #3 can be used for the HE-stained images. By doing so, the
advantage here is that segmentation data are produced without the need for human experts
to have laboriously drawn each class per TMA core. However, this advantage is at the cost
of errors due to, for example, poor registration or false thresholding between IHC-negative
and -positive areas. The multi-stage process only produces a rough visual inspection of
the results. In summary, dataset #1 contained n = 247 images (n = 108 for NT, n = 84 for
ADC, n = 55 for SqCC). Nine examples are shown in Figure 1. This dataset was used for
the training and validation of the modified and retrained unsupervised models (hereafter
Kanezaki models) and the supervised model (a UNet-Variant). Dataset #1 is available at
HeiData: https://heidata.uni-heidelberg.de/privateurl.xhtml?token=0129f05c-b1a7-492
7-a841-2440eb0b3cc4.

<figure><img src="image_16.png" /><figcaption>**Figure 1.** Example for training and validation dataset (dataset #1). Based on registered pairs of HE- and IHC-stained sections from TMA cores, tumour segmentation or rather tumour mask generation is performed by a combination of colour deconvolution and thresholding. (A) Composite image of nine HE-stained TMA cores. For each of the three classes (NT, ADC, SqCC), there are three images. (B) Corresponding IHC(panCK)-stained images were registered on the HE-stained cores. Based on the IHC-positive area and the diagnosis per core, the according image and image regions were labelled: 0 background, 1 normal tissue or non-tumourous (NT), 2 adenocarcinoma (ADC), and 3 squamous cell carcinoma (SqCC)</figcaption></figure>

2.4. *Testing Dataset (Dataset #2)*

Dataset #2 was used for model testing only and was based on manual segmentation (examples shown in Figure 2). Therefore, TMA images were manually annotated, segmented, and further prepared in QuPath [17]. For this manual segmentation, the following labels were defined (in accordance with the definitions for dataset #1): 0 background, 1 non-tumourous tissue (NT), 2 adenocarcinoma (ADC), and 3 squamous cell carcinoma (SqCC). The ratio between the area per label was approximately 3.7 (background) to 1.2 (NT)
to 1.1 (ADC) to 1.0 (SqCC). Dataset #2 contained n = 40 images (n = 3 for NT, n = 18 for ADC, n = 19 for SqCC) and is available at HeiData: https://heidata.uni-heidelberg.de/privateurl.xhtml?token=0129f05c-b1a7-4927-a841-2440eb0b3cc4.

<figure><img src="image_17.png" /><figcaption>**Figure 2.** Example for testing dataset (dataset #2). Example of human-labelled ground truth images used for testing the models. (A) Composite image of nine TMA cores. For each of the three classes (NT, ADC, Sqcc), there were three images. (B) Composite image of corresponding labelled images with 0 for background, 1 for normal tissue or non-tumourous tissue (NT), 2 for adenocarcinoma (ADC), and 3 for squamous cell carcinoma (SqCC).</figcaption></figure>

## 2.5. Model Training

Machine learning was performed in Python with PyTorch [19]. For supervised learning, the Segmentation Models toolbox from Yakubovskiy et al. [20] was used (https://github.com/qubvel/segmentation_models.pytorch, accessed on 1 February 2022). For unsupervised training, the models and scripts from Kanezaki et al. [15] were adapted (https://github.com/kanezaki/pytorch-unsupervised-segmentation, accessed on 1 February 2022) and used.

## 2.6. Loss Functions

Against the background of unbalanced labels and heterogeneously shaped objects, different loss functions selected from the plethora of published functions were used. These loss functions are differently well suited for imbalanced datasets. Furthermore, the loss functions are differently well suited for different models [21,22] (Table 1).

<table><caption>**Table 1.** List of different loss functions here tested. A set of loss functions is tested against the background of imbalanced labels and heterogeneous objects.</caption><thead><tr><td>Loss Function</td><td>Source</td></tr></thead><tbody><tr><td>Cross-Entropy Loss</td><td>Pytorch implementation [19]</td></tr><tr><td>Dice Loss</td><td>Pytorch implementation [19]</td></tr><tr><td>Focal Loss</td><td>pytorch-toolbelt implementation [23]</td></tr><tr><td>Tversky Loss</td><td>pywick implementation [24]</td></tr><tr><td>Boundary loss</td><td>proposed by Bokhovkin et al. [25]</td></tr><tr><td>Surface Loss with Cross-Entropy Loss</td><td>proposed by Kervadec et al. [26]</td></tr><tr><td>Surface Loss with Focal Loss</td><td>proposed by Kervadec et al. [26]</td></tr></tbody></table>

## 2.7. Segmentation Quality Assessment

The segmentation quality per image was evaluated by calculating the accuracy and the F1 score, each in its scikit-learn implementation [27]. As ground truth for the calculations, the validation set (see Section 2.3) and the test set (see Section 2.4) were used.
## 3. Results

### 3.1. How Can Labels from a Model Trained in an Unsupervised Manner Be Converted to Meaningful Labels?

For unsupervised image segmentation, Kanezaki et al. (Figure 3) published a training approach in 2018 based on similarity [15] and another in 2020 based on differentiable feature clustering [28]. Here, we use the first approach and the framework described for it, which we refer to as the Kanezaki framework. It starts with a high number of classes and minimises the label classes in every training epoch, until a predefined number of classes is reached. The label classes are merged based on the similarity in the segmented image region. Finally, there are a given number of label classes segmented per image. However, these labels do not correspond to meaningful labels. Which label class (e.g., 1) belongs to which histological structure (e.g., alveolar epithelium) is unclear. We mapped such a model to defined classes by a two-step process.

First, the Kanezaki training approach was reproduced using the histological images available. The excised HE-stained cores were used in random order for training. As we have described, the approach of Kanezaki et al. starts with a predefined number of labels— in our case, 100. Next, at each epoch, the number of labels was reduced or the labels were merged. The training process ended when the previously specified expected number of labels was reached or undercut. Because, in our setting, the classes 'background', 'tumour stroma', 'squamous cell carcinoma', and 'adenocarcinoma' were expected, the number of expected labels was set to 10.

<figure><img src="image_18.png" /><figcaption>**Figure 3. Schematic of the remapping approach.** Kanezaki et al. described an approach for unsupervised segmentation [15]. This leads to meaningless morphological labelling. For example, the epithelial and stromal structures are segmented. In this manner, homogeneous labels (in terms of texture, for example) are created. Depending on the resolution, this can lead to tumour formation and splitting of the tumour stroma into different partial labels. The main hypothesis of the underlying work is that labels such as 'adenocarcinoma' are composed of a distinct set of morphological labels produced by unsupervised training. The approach described herein is divided into two main parts. (A) First, a CNN model (e.g., consisting of several convolutional and batch normalization blocks) is trained in an unsupervised manner, as described by Kanezaki et al. This training was performed on the image batches to ensure that all classes were represented. (B) An additional block was added to the model to map the classes of the model to patho-histological labels. The mapping was trained in a supervised manner. There were four components for the label frequency vectors per TMA core.</figcaption></figure>

For the CNN model, a simple model composed of a linear combination of convolution and batch normalisation blocks was used, as described by Kanezaki et al. [15]. This simple
CNN model is henceforth referred to as concise_CNN (visualised in Figure A1A). Other more complex segmentation models, such as UNets or FCNs, have also been trained in this framework. However, only concise models with a few layers (called concise_UNET and concise_FCN; visualised in Figure A1B,C) converged. Standard UNet-variants such as those published in Yakubovskiy et al. (henceforth called Standard_UNET) cannot be trained [20].

Second, to map the labels to meaningful labels, a fully connected layer is added to the model previously trained in an unsupervised setting. This layer is supposed to map the learned labels to given, defined labels, such as ‘stroma’. The extended model is then retrained again in a supervised setting on a small, labelled dataset. This two-step approach is performed for three image sizes (256 × 256, 512 × 512, and 1024 × 1024 pixels), to test whether the image or object size affects the segmentation performance.

### 3.2. *Do Different Loss Functions Affect Retraining?*

The used lung cancer datasets (dataset #1 based on IHC annotations (Section 2.3) and dataset #2 based on manual segmentation (Section 2.4)) are highly heterogeneous. For example, the background area (label 0) was three-times more frequent than the other three labels (1–3) in the overall dataset. In a single image, the ratio of, e.g., adenocarcinoma to stroma can easily exceed 1 to 10. Furthermore, the shape and histological characteristics of tumour formations of one entity (e.g., SqCC) can be diverse. To compensate for the imbalanced dataset with regard to the area per label, we compared different error functions and metrics.

The loss functions are as follows: (1) the PyTorch-implemented weighted cross-entropy loss function [19]; (2) the dice loss and (3) the focal loss function (with the pytorch-toolbelt implementation [23]); (4) the Tversky loss function (with its pywick implementation [24]); (5) the boundary loss function proposed by Bokhovkin et al. [25]; and (6) the surface loss function proposed by Kervadec et al. [26].

As a readout, the segmentation quality was measured by calculating the accuracy and the F1 score. These parameters were assessed for the validation dataset (being 0.25 for dataset #1) and the test dataset (dataset #2). Notably, there is a morphological or quality difference between IHC-based and manual segmentation. Thus, for the models, it is a certain transfer task, because the training, validation, or test data differ.

Independent of the image size, the segmentation quality reaches its highest value when the cross-entropy loss function is used alone or in combination (for both validation and testing). Notably, the segmentation quality was only moderate even for the best models. For example, for the validation dataset (see Table 2), we observed unbalanced cross-entropy with an accuracy of 0.88 ± 0.11 and an F1 score of 0.63 ± 0.19; for balanced cross-entropy, we observed an accuracy of 0.82 ± 0.13 and an F1 score of 0.47 ± 0.16; and for surface loss (in combination with balanced cross-entropy), as described by Kervadec et al. [26], we observed an accuracy of 0.86 ± 0.22 and an F1 score of 0.39 ± 0.12).

### 3.3. *In Comparison, What Are the Results of an Often-Used Segmentation Model Trained in a Supervised Manner?*

For comparing the segmentation quality, a standard UNet-variant [20] was trained. This model was trained and validated on dataset #1 (Section 2.3) and tested on dataset #2 (Section 2.4) under the same conditions as described above for the Kanezaki models. Different image sizes and error functions were used to visualise their effects on the segmentation quality (measured with accuracy and the F1 score).
<table><caption>**Table 2.** Retrained unsupervised segmentation results. An adapted Kanezaki model was retrained on different image sizes (256 × 256 and 512 × 512 pixels) with five different loss functions: (1) cross-entropy, (2) dice loss, (3) focal loss, (4) Tversky loss, and (5) boundary loss function. The validation set corresponds to 0.25 from dataset #1 (compare Section 2.3) and the test set corresponds to the entire dataset #2 (compare Section 2.4). As a metric for the segmentation quality, the accuracy and F1 score are calculated.</caption><thead><tr><th>Image Size</th><th>Loss Function</th><th>Validation Set (Accuracy/F1)</th><th>Test Set (Accuracy/F1)</th></tr></thead><tbody><tr><td rowspan="8">256</td><td>Cross-Entropy (balanced)</td><td>0.82 ± 0.13</td><td>0.47 ± 0.16</td><td>0.63 ± 0.21</td><td>0.37 ± 0.09</td></tr><tr><td>Cross-Entropy (unbalanced)</td><td>0.88 ± 0.11</td><td>0.63 ± 0.19</td><td>0.59 ± 0.21</td><td>0.43 ± 0.16</td></tr><tr><td>Surface Loss (with Cross-Entropy)</td><td>0.86 ± 0.22</td><td>0.39 ± 0.12</td><td>0.51 ± 0.17</td><td>0.33 ± 0.10</td></tr><tr><td>Surface Loss (with Dice Loss)</td><td>0.32 ± 0.12</td><td>0.28 ± 0.11</td><td>0.44 ± 0.23</td><td>0.30 ± 0.11</td></tr><tr><td>Focal Loss</td><td>0.88 ± 0.12</td><td>0.63 ± 0.21</td><td>0.59 ± 0.20</td><td>0.43 ± 0.18</td></tr><tr><td>Tversky Loss</td><td>0.87 ± 0.12</td><td>0.62 ± 0.21</td><td>0.57 ± 0.19</td><td>0.39 ± 0.16</td></tr><tr><td>Dice Loss</td><td>0.86 ± 0.12</td><td>0.60 ± 0.21</td><td>0.61 ± 0.16</td><td>0.37 ± 0.21</td></tr><tr><td>Boundary Loss</td><td>0.13 ± 0.07</td><td>0.15 ± 0.06</td><td>0.27 ± 0.17</td><td>0.17 ± 0.06</td></tr><tr><td rowspan="8">512</td><td>Cross-Entropy (balanced)</td><td>0.86 ± 0.12</td><td>0.53 ± 0.17</td><td>0.60 ± 0.11</td><td>0.40 ± 0.08</td></tr><tr><td>Cross-Entropy (unbalanced)</td><td>0.87 ± 0.12</td><td>0.56 ± 0.18</td><td>0.60 ± 0.16</td><td>0.37 ± 0.10</td></tr><tr><td>Surface Loss (with Cross-Entropy)</td><td>0.86 ± 0.11</td><td>0.56 ± 0.20</td><td>0.66 ± 0.16</td><td>0.41 ± 0.10</td></tr><tr><td>Surface Loss (with Dice Loss)</td><td>0.28 ± 0.09</td><td>0.26 ± 0.08</td><td>0.33 ± 0.10</td><td>0.25 ± 0.07</td></tr><tr><td>Focal Loss</td><td>0.88 ± 0.12</td><td>0.58 ± 0.21</td><td>0.64 ± 0.18</td><td>0.36 ± 0.11</td></tr><tr><td>Tversky Loss</td><td>0.88 ± 0.11</td><td>0.56 ± 0.20</td><td>0.60 ± 0.17</td><td>0.36 ± 0.10</td></tr><tr><td>Dice Loss</td><td>0.88 ± 0.11</td><td>0.54 ± 0.20</td><td>0.60 ± 0.15</td><td>0.37 ± 0.10</td></tr><tr><td>Boundary Loss</td><td>0.13 ± 0.08</td><td>0.14 ± 0.06</td><td>0.20 ± 0.11</td><td>0.15 ± 0.07</td></tr></tbody></table>

### 3.4. How Does the Training Dataset Size Affect the Segmentation Quality of the Models Trained under Unsupervised and Supervised Conditions?

To test whether the segmentation performance of the models tested depends on the size of the training data as expected, the modified concise_CNN and the UNet-model were pretrained with eight subsets of different size from the previous datasets (see x-axis in Figure 4: The first subset, named selection, was a manual image selection with three images per diagnosis (NT, ADC, and SqCC) from dataset #2 (see Section 2.4). The other eight subsets were named with 1.0, 0.75, 0.5, 0.25, 0.1, 0.05, and 0.01, respectively, after the fractions from dataset #1 (see Section 2.3). In addition, to test the effect of the pretraining of the UNet-model, two UNet-models differing in the means of pretraining were tested: one naive for histological images and henceforth referred to as UNet_naive (orange bars in Figure 4), and the other with a ResNet-model [20] pretrained on an image tile classification task with the classes normal tissue, adenocarcinoma, and squamous cell carcinoma, hereafter referred to as UNet_histo (green bars in Figure 4).

All models were trained with the surface loss function (with the combination of boundary loss and cross-entropy loss) [26] for 50 epochs. All image tiles used for training and testing had a size of 256 × 256 pixels.

Segmentation quality was assessed based on the testing images (dataset #2), as in the prior section.
<figure><img src="image_19.png" /><figcaption>**Figure 4.** Effect of the number of images used for training. Three models were trained on eight different datasets. The three models are: a small CNN variant as described by Kanezaki et al. [15] (called concise_CNN), trained in an unsupervised approach and then modified and retrained as described in this work to produce histologically meaningful labels; two UNet models [20] with a ResNet backbone pretrained on a tissue classification task (called UNet_histogram) and with a ResNet backbone without pretraining (called UNet_histogram). The eight datasets are a manual selection (with nine images, three per diagnosis; compare Figure 1) from dataset #1 (called selection), and seven increasingly larger fractions from dataset #2 (called 0.01, 0.05, 0.1, 0.25, 0.5, 0.75, and 1.0). Subsequently, the different models were tested using dataset #2. As a segmentation metric, the F1 score is plotted.</figcaption></figure>

The models trained on dataset #2 had equal segmentation quality, as in the prior section. For the Concise_CNN, the accuracy was 0.49 ± 0.22 and the F1 score was 0.31 ± 0.10. For the UNet_naive, the accuracy and the F1 score were 0.53 ± 0.20 and 0.35 ± 0.12. Finally, for the UNet_histogram, the accuracy was 0.63 ± 0.18 and F1 score was 0.40 ± 0.10 (right end of the box plot in Figure 4). Notably, no statistically significant differences were observed in the accuracy and F1 score for the models in the range 0.1 to 1.0: for 0.1 of dataset #1, the F1 score was 0.30 ± 0.11; for the UNet_naive, it was 0.33 ± 0.09; and for the UNet_histogram, it was 0.31 ± 0.07. Only for the small fractions (0.01 and the nine images from the selection) were the models' performance reduced (left end of the box plot in Figure 4). For the selection with nine images, there was a trend for better performance of the Kanezaki model, since its accuracy (0.43 ± 0.23) and F1 score (0.29±0.10) were not reduced as much as those for the UNet models (for UNet_naive, 0.26 ± 0.06 and 0.20 ± 0.04, respectively, and for UNet_histogram, 0.13 ± 0.19 and 0.20 ± 0.07, respectively).

### 3.5. Does the Model Architecture Trained in an Unsupervised Manner Influence the Segmentation Quality?

Kanezaki et al. [15] demonstrated that with their approach, different CNN models can be trained. As we have described, complex models such as the UNet variant by Yakubovskiy et al. [20] do not converge. However, simple model variants for FCN and UNet can be trained and do converge. To compare three different architectures, we first used the unsupervised Kanezaki approach to train the aforementioned models, which consist of a linear combination of convolution blocks (called concise_CNN; sketched in Appendix A Figure A1A), a relatively simple FCN variant (called concise_FCN; Appendix A Figure A1B), and a relatively simple UNet variant (called concise_UNet; Appendix A Figure A1C).
Next, these three models were retrained in a supervised setting (as described in Section 4.1). To test if there is an advantage of such pretrained models for smaller datasets, we performed retraining by using two training datasets: (i) a manual selection of nine images (three per NT, ADC, and SqCC; called selection) and (ii) the entire dataset #1.

Regarding segmentation quality measurement, these models were again tested on dataset #2 (the testing dataset).

For very small retraining datasets (selection; n = 9 images), the Concise_CNN shows better results if only the last layers are retrained (accuracy 0.53 ± 0.21 and F1 score 0.34 ± 0.11). For the more complex concise_FCN (accuracy 0.50 ± 0.24 and F1 score 0.30 ± 0.10) and concise_UNet (accuracy 0.43 ± 0.11 and F1 score 0.24 ± 0.09), the models only show moderate segmentation quality, if the entire models are retrained (Figure A2A).

For larger retraining datasets (dataset #1; n = 247 images (with 0.8 for training and 0.2 for validation)), there was no significant difference for all three models (Figure A2B). The best model was the simple_FCN. The accuracy was 0.63 ± 0.21 and the F1 score was 0.38 ± 0.10 when only the last layers were retrained, and the accuracy was 0.61 ± 0.15 and the F1 score was 0.387 ± 0.07 when all layers were retrained. The worst model was the simple_CNN. For retraining only the last layers, the accuracy was 0.57 ± 0.17 and the F1 score was 0.36 ± 0.10. For retraining all layers, the accuracy was 0.57 ± 0.19 and the F1 score was 0.37 ± 0.10.

Notably, segmentation results were more than 0.1 worse than the results for the complex UNet model variants, such as the Standard_UNET (compare Table 3) by Yakubovskiy et al. [20].

<table><caption>**Table 3.** Supervised segmentation results. A UNet model was trained on different image sizes (256x256 and 512x512 pixels) with five different loss functions: (1) cross-entropy, (2) dice loss, (3) focal loss, (4) Tversky loss, and (5) boundary loss function. The validation set corresponds to 0.25 from dataset #1 (compare Section 2.3) and the test set corresponds to the entire dataset #2 (compare Section 2.4). As a metric for the segmentation quality, the accuracy and the F1 score are calculated.</caption><thead><tr><th>Image Size</th><th>Loss Function</th><th>Validation Set (Accuracy/F1)</th><th>Test Set (Accuracy/F1)</th></tr></thead><tbody><tr><td rowspan="8">256</td><td>Cross-Entropy (balanced)</td><td>0.86 ± 0.13</td><td>0.52 ± 0.20</td><td>0.71 ± 0.17</td><td>0.43 ± 0.08</td></tr><tr><td>Cross-Entropy (unbalanced)</td><td>0.89 ± 0.11</td><td>0.64 ± 0.23</td><td>0.64 ± 0.19</td><td>0.45 ± 0.16</td></tr><tr><td>Surface Loss (with Cross-Entropy)</td><td>0.87 ± 0.12</td><td>0.57 ± 0.17</td><td>0.86 ± 0.17</td><td>0.41 ± 0.08</td></tr><tr><td>Surface Loss (with Dice Loss)</td><td>0.78 ± 0.18</td><td>0.41 ± 0.08</td><td>0.57 ± 0.16</td><td>0.34 ± 0.07</td></tr><tr><td>Focal Loss</td><td>0.88 ± 0.11</td><td>0.63 ± 0.23</td><td>0.70 ± 0.17</td><td>0.42 ± 0.08</td></tr><tr><td>Tversky Loss</td><td>0.88 ± 0.12</td><td>0.64 ± 0.23</td><td>0.70 ± 0.19</td><td>0.42 ± 0.09</td></tr><tr><td>Dice Loss</td><td>0.88 ± 0.12</td><td>0.54 ± 0.65</td><td>0.71 ± 0.22</td><td>0.51 ± 0.21</td></tr><tr><td>Boundary Loss</td><td>0.13 ± 0.14</td><td>0.06 ± 0.16</td><td>0.24 ± 0.04</td><td>0.19 ± 0.03</td></tr><tr><td rowspan="10">512</td><td>Cross-Entropy (balanced)</td><td>0.86 ± 0.12</td><td>0.47 ± 0.09</td><td>0.70 ± 0.14</td><td>0.44 ± 0.08</td></tr><tr><td>Cross-Entropy (unbalanced)</td><td>0.90 ± 0.10</td><td>0.66 ± 0.22</td><td>0.66 ± 0.15</td><td>0.42 ± 0.10</td></tr><tr><td>Surface Loss (with Cross-Entropy)</td><td>0.90 ± 0.12</td><td>0.59 ± 0.19</td><td>0.68 ± 0.12</td><td>0.42 ± 0.08</td></tr><tr><td>Surface Loss (with Dice Loss)</td><td>0.90 ± 0.09</td><td>0.54 ± 0.14</td><td>0.66 ± 0.14</td><td>0.40 ± 0.17</td></tr><tr><td>Focal Loss</td><td>0.91 ± 0.10</td><td>0.61 ± 0.19</td><td>0.66 ± 0.15</td><td>0.40 ± 0.08</td></tr><tr><td>Tversky Loss</td><td>0.90 ± 0.11</td><td>0.65 ± 0.21</td><td>0.66 ± 0.19</td><td>0.43 ± 0.13</td></tr><tr><td>Dice Loss</td><td>0.90 ± 0.10</td><td>0.64 ± 0.21</td><td>0.64 ± 0.16</td><td>0.39 ± 0.08</td></tr><tr><td>Boundary Loss</td><td>0.11 ± 0.06</td><td>0.12 ± 0.05</td><td>0.23 ± 0.11</td><td>0.15 ± 0.05</td></tr></tbody></table>
### 3.6. Are the Labels Learned in an Unsupervised Fashion Already Meaningful?

As we have described, the models trained without supervision produce a predefined number of labels that are not directly connected to labels defined by humans. These labels are based on texture or morphological similarity. To test whether these labels alone correlate to the known structures (e.g., tumour glands) or more to the diagnoses of non-tumourous tissue (NT), adenocarcinoma (ADC), or squamous cell carcinoma (SqCC), in this context, the frequency of these labels per diagnosis was examined.

The basic idea was that each TMA core has its own label composition or frequency that correlates with the diagnosis. For example, a TMA core from a case with ADC should contain (only) the labels NT and/or ADC. To test this assumption in principle, for human-generated labels, we plotted the label composition (frequency of labels per TMA core) against the known diagnosis per TMA kernel (see A1-2 in Figure 5). Plotting the label frequency per diagnosis (A1 in Figure 5) or running a principal component analysis (PCA) to compare the frequency vectors per TMA core (A2 in Figure 5) verifies this assumption. As expected, cases with, for example, a diagnosis of ADC differ in that only in these cases does the label ADC occur alongside the image background (BG) and non-tumourous tissue (NT).

<figure><img src="image_20.png" /></figure>

**Figure 5.** Segmentation results for different models trained. Kanezaki et al. [15] described a method for training CNN models in an unsupervised fashion, resulting in some labels that are not assigned to certain structures by a human. For testing whether these labels or a combination are already meaningful, the label frequency per TMA core was analysed with respect to the known diagnosis per core (normal tissue (NT), adenocarcinoma (ADC), and squamous cell carcinoma (SqCC)). (A1,A2) For testing whether the label frequency per TMA core could correlate with the diagnosis, the label frequency of the manually annotated TMA cores was examined. (A1) shows the label frequency (background (BG), normal tissue (NT), adenocarcinoma (ADC), and squamous cell carcinoma (SqCC)) plotted against the diagnosis. (A2) shows a PCA (two components) for the label frequency vector per TMA core. (B1,B2) For the labels produced by a simple CNN model (previously called concise_CNN) trained in an unsupervised manner, the label frequency was also analysed in regard to the known diagnosis per TMA core. (B1) shows the frequency distribution per label and diagnosis. (B2) shows a PCA (two components) for the label frequency vectors per TMA core.
In the next step, we examined the label frequency for the labels generated by the model after unsupervised training. Here, it can be seen that neither plotting the label frequency per diagnosis (B1 in Figure 5)) nor a PCA analysis (B2 in Figure 5) show a reliable correlation between label composition and diagnosis. A distinction between ADC and SqCC is not possible on this basis. Based on these plots, only TMA cores with and without tumour infiltration can be distinguished. A comparable analysis for the standard UNet [20] trained in a supervised setting also showed no sharp separation of diagnosis based on label frequency (compare Appendix A Figure A3). This fits well with the overall moderate segmentation quality of the models trained and validated. In the unsupervised trained or retrained models, the small network size might be causative. In the standard UNet, the training database may not be sufficiently large.

## 4. Discussion

Digital and complex medical data are available from various medical specialities. For patients with tumours, for example, there are molecular data, and radiological and pathological image data [10,29]. The analysis of these vast data from one field—or, better, combined—leads to an opportunity to find next-generation, data-driven biomarkers [29]. In pathology, Pathomics is the subdiscipline dedicated to mapping image data to clinical information such as nodal status. In other words, new image-based biomarkers are being sought in Pathomics. In this context, the segmentation of histological images (e.g., in tumour and stroma) is an early and major step in many projects [1,4]. For supervised segmentation approaches, the scarcity of large, properly annotated datasets is a common obstacle [3,7,30]. Not only is annotation tedious per se, but in many cases, the number of available images is limited. For example, managing thymoma, a rare disease, significantly limits the number of cases to be included [31]. Data augmentation techniques alone have usually not solved the problem of small numbers [7,10]. In addition to supervised training approaches, there are weakly supervised and unsupervised approaches that can help to overcome this constraint. However, mapping the results produced by unsupervised approaches to meaningful labels is a non-trivial task. Against this background, we tested, in the complex setting of the distinction between pulmonary (solid-growing) ADC and (non-keratinising) SqCC [32], whether a segmentation model could (i) be trained in an unsupervised approach and (ii) modified and retrained in a supervised setting to produce meaningful histological labels such as ‘ADC’ or ‘SqCC’. In the best case, these labels should handle the aforementioned non-trivial distinction between solid-growing ADC and non-keratinising SqCC. (iii) We compared the segmentation results to standard segmentation models for the same datasets.

(Ad i), we show that unsupervised image segmentation techniques or training frameworks as described by Kanezaki et al. can be used for the unsupervised segmentation of histological images (Figure 3A) [15,16]. Here, only aspects such as the ratio of filter size to object size in the image need to be considered.

(Ad ii), we demonstrate that these models can be extended by another block, which can, after a second supervised training, remap (by a linear combination) the produced labels to meaningful labels such as ‘ADC’ or ‘SqCC’ (Figure 3B). Our new contribution is that a simple linear combination of the different labels previously recognised based on unsupervised training is applied to predict difficult labels such as ‘ADC’.

(Ad iii), finally, we compare the results to conventional training approaches and demonstrate that our approach of remapping the labels is not superior to conventional supervised learning. It is indeed inferior and there are only limited settings where it can be useful.

### 4.1. Unsupervised Segmentation in Pathology and the Problem of Obtaining Meaningful Labels (Ad i)

Image segmentation is, for many projects in the realm of Digital Pathology or Pathomics, an important early step. There is a legion of different approaches that, based on
the training setting, can be broadly categorised as fully supervised, weakly supervised, and unsupervised approaches. For the supervised training approaches, the necessary annotations are time-consuming and tedious to produce. Indeed, the shortage of such annotated datasets is a well-known obstacle [3,7,30]. Needing less annotated data for weakly supervised or no annotated data at all for unsupervised methods sounds, in this context, very promising. In addition, by using unsupervised approaches, the need to tailor to every project a well-annotated training set for machine learning models will be reduced to gathering a fitting image collection [10].

Regarding weakly supervised approaches in pathology, generative adversarial networks can be used (after training in a weakly supervised setting) to generate synthetic data based on a small dataset [5,33]. However, this would mean adding a training cost-intensive step before the actual segmentation model training.

Regarding unsupervised image segmentation, in pathology, there are several published approaches [5,9]. These approaches cover a vast methodological spectrum with, for example, the combination of feature extraction and subsequent clustering [9,34] or the application of auto-encoders for classification or staining adaption [5,35–37].

In a nutshell, there are various working, published, easily adaptable, unsupervised approaches for segmenting (histological) images into different morphological regions. This would then overcome the problem of data scarcity. Unfortunately, this advantage brings a new problem. The labels generated based on morphological aspects (e.g., 'blue granular area') cannot in every case be simply mapped to (histologically) meaningful labels such as 'carcinoma'. One solution to this is to assign names to the labels by human experts. For example, we could allow the expert to define blue areas with many small cells as lymphoid infiltration. However, this expert approach only works if exactly one label is generated per annotation. A multiphase process such as a tumour consisting of tumour cells, stroma, and inflammatory infiltrate, etc., will not be nameable in this manner. For such multiphase entities, the true annotation can be considered as a combination of the morphological labels. However, this linear combination is too simplistic for many areas, as the context is then missing. For example, a homogeneous, blue area can be part of the sky or a blue car. In this regard, there are works that use graphs to include the neighbourhood relationships of the individual labels. For example, Pourian et al. used graphs of regions to combine the visual and spatial characteristics of different image parts to meaningful image-part groupings [38]. Alternatively, Wigness et al. used local graphs to combine labels in image regions [39].

Our approach, by contrast, is a simple linear combination of the different labels generated by unsupervised learning based on morphological similarity (by adding a fully connected layer to a CNN model; see B in Figure 3). This linear combination is in analogy to the pathological thinking of tissue or organs as a combination of different structures such as epithelium, stroma, blood vessels, etc. [40,41]. However, this approach ignores neighbourhood relations or local aspects.

## 4.2. CNN Models Previously Trained in an Unsupervised Manner Can Be Adapted to Produce Meaningful Histological Labels (Ad ii)

As we have discussed, unsupervised training approaches can be used for histological images, but they produce distinct regions or labels based on morphology (e.g., reddish area with little texture) without histologically meaningful labels (such as, e.g., 'fibrosis'). We have successfully trained several CNN model variations (a combination of convolutional blocks (called concise CNN), a shallow UNET variant (called concise_UNET), and a shallow FCN variant (called concise_FCN)) in an unsupervised approach, as described by Kanezaki et al. [15,16]. The produced image regions, or rather labels, however, are not mapped to the conventional histological structures. For instance, gland structures are composed of an epithelial layer (one label) and the luminal space (another label). Pathologists would usually annotate these structures together as a gland, in analogy to the typical thinking of tissues and organs as combinations of a limited number of substrata [40,41].
For remapping the labels produced by the model trained in an unsupervised manner, we added a block (a fully connected layer) and then trained it on mapping the labels to human-produced annotations (compare B in Figure 3). However, this again necessitates the presence of (a small amount of) annotated data. Notably, this approach is therefore no longer an unsupervised but a weakly supervised approach. By adding another block and retraining, we can show that a model can produce meaningful annotations. However, compared to other segmentation models (such as the UNet implementation by Yakubovskiy et al. [20]) trained on the same dataset, the approach proposed here leads to inferior results.

### 4.3. The Combination of Unsupervised and Subsequent Supervised Label Mapping Is Not Better than Conventional CNN-Based Segmentation (Ad iii)

The inferior segmentation results in combination with again the need of a labelled dataset argue against the herein proposed remapping of labels by adding an additional block and by retraining with a small annotated dataset (compare Figure 3).

There are several potential explanations for the rather moderate segmentation results of the proposed remapping approach:

(1) The model complexity is maybe not fitting with the task. Of note, the used framework for unsupervised training described by Kanezaki et al. [15,16] only works with shallow CNN models. Large models such as the UNet model implemented by Yakubovskiy et al. [20] do not converge.

(2) Another idea would be that the ratio of the CNN filter size to the object size in the images is either too small or too large. Therefore, we tested different images sizes (256 x 256, 512 x 512, and 1024 x 1024 pixels) and found no significant difference. Likewise, it would be possible in principle that the CNN models trained in this way were too shallow. However, this is contradicted by the fact that the tumour sub-type differentiation also did not work well in the UNet models used by other groups [20], which are frequently used and perform well.

(3) The task itself is non-trivial since neither models adapted as described nor standard segmentation models can make the distinction between different tumour types, particularly between (non-keratinising) SqCc and (solid) ADC. In the used dataset, on which a work on tumour classification has been published recently [32], the models can only distinguish background from tissue and normal tissue from tumour parts. Of course, the models trained supervised perform better (see Tables 2 and 3); however, their segmentation results are also only moderate with assigning mixed labels (such as ADC and SqCC) per tumour infiltration. This could be due to several reasons. For example, the task of distinguishing between a non-glandular growing ADC and a non-keratinising SqCC is non-trivial, even for an experienced pathologist, on the basis of HE-stained images alone. Moreover, unlike the previously published work on this dataset for classification [32], where each image must be assigned to one class, now, each pixel must be assigned the correct label.

(4) Finally, maybe the labels or morphological clusters segmented by the models after unsupervised training are found within non-neoplastic and neoplastic structures. For example, glandular structures are found in both. To test this, we looked at the distribution of morphological labels generated by such a model compared to the diagnoses per TMA core (see Figure 5). Based on plotting for every TMA core the frequency of the labels background (BG), normal tissue (NT), adenocarcinoma (ADC), and squamous cell carcinoma (SqCC) in a two-dimensional PCA, we were able to classify the TMA cores into the three classes of normal tissue, adenocarcinoma, and squamous cell carcinoma (see Figure 5A1,A2). Interestingly, for the models initially trained unsupervised and then retrained and for the UNets trained supervised, based on the label distribution, such a classification is not possible (see Figure 5B1,B2 and Appendix A Figure A3A,B). This is an argument for the assumption that solely morphological labels are not enough for the herein analysed task, in line with the discussion in the section above.
The limitations described above raise the question of whether the method can be improved. Increasing the complexity of the models pretrained in an unsupervised manner alone seems not promising, since even complex models such as UNets are not able to adequately solve the task. The most promising approaches, in view of the work of Pourian et al. [38] or Wigness et al. [39], seem to involve the neighbourhood when combining the individual labels into meaningful histological annotations. This should certainly be followed up in future work. Moreover, the approach described here should certainly be tested on a simple histological task as a proof of principle.

### 4.4. Are There Arguments for Using the Herein Proposed Remapping Approach?

Regarding the only moderate segmentation quality on one hand and the greater effort of retraining on the other hand, one could ask if there are arguments for using such an approach. Having a well-annotated dataset at hand, there are no arguments against using a standard supervised training setting. Of course, large and good datasets are rather scarce for histology [3,5–7]. The approach proposed herein, which, in combination, is more akin to weakly supervised approaches, might provide an initial advantage if there is only a dataset of limited size. Moreover, in a scenario in which there is only a small dataset, one could also consider using methods such as generative adversarial networks to produce synthetic data, on which then the segmentation model is trained [5,33].

**Author Contributions:** Conceptualisation, C.-A.W., K.K. and M.K.; methodology and formal analysis, C.-A.W.; data generation and curation, K.R.W., C.-A.W., K.K. and M.K.; manuscript writing and preparation, C.-A.W., K.K. and M.K. All authors have read and agreed to the published version of the manuscript.

**Institutional Review Board Statement:** The study was approved by the ethics committee of the Medical Faculty Heidelberg, Heidelberg University (#S-207/2005 and #S315/2020).

**Data Availability Statement:** Image data (for training, validation, and testing) along with ground truth are available at HeiData: https://heidata.uni-heidelberg.de/privateurl.xhtml?token=0129f05cb1a7-4927-a841-2440eb0b3cc4. The code for the work is available at GitHub: GitHub-Link/The repo is created when the publication title is clear so that it can be generated under the same name.

**Acknowledgments:** The authors gratefully acknowledge the data storage service SDS@hd supported by the Ministry of Science, Research and the Arts Baden-Württemberg (MWK) and the German Research Foundation (DFG) through grant INST 35/1314-1 FUGG and INST 35/1503-1 FUGG. Furthermore, the authors also thank the IT department staff of the Medical Faculty Mannheim and especially Bohne-Lang for supervising the computer administration and infrastructure.

**Conflicts of Interest:** The authors declare no conflicts of interest.

## Abbreviations

The following abbreviations are used in this manuscript:

MDPI Multidisciplinary Digital Publishing Institute

CNN Convolutional Neuronal Network

PCA Principal Component Analysis

GAN Generative Adversarial Network
Appendix A

<figure><img src="image_21.png" /><figcaption>**Figure A1.** Schematic representation of the models used. The models (A–C) were first trained in an unsupervised fashion and subsequently trained on mapping the labels to meaningful labels. In contrast, the model in (D) was trained solely in a supervised fashion. (A) Concise CNN: Schematic plot of the simple CNN model used by Kanezaki et al. [15], which is composed of a linear combination of convolutional, ReLu, and batch normalization layers. (B,C) Concise_UNET and Concise_FCN: Schematic representation of concise UNET and FCN model variants. (D) Standard_UNet: Schematic representation of the UNet model designed by Yakubovskiy et al. [20], here used for supervised segmentation.</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>**Figure A2. Segmentation results for different models trained.** Kanezaki et al. [15] described a method for training CNN models in an unsupervised fashion. In this approach, different CNN models can be trained. Here, a simple CNN model composed of several convolution blocks (simple_CNN), a simple FCN variant (simple_FCN), and a simple UNet variant (simple_UNet) are trained and retrained on two datasets (A,B). Furthermore, in the retraining, only the last layers (last) or the entire model (all) are retrained. (A) The models are retrained on a selection of nine images (three images per diagnosis: NT, ADC, and SqCC). (B) The models are retrained on the entire dataset #2.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Figure A3. Segmentation results for a UNet model.** The labels produced per image by a UNet model [20] trained in a supervised fashion are plotted. (A) shows the label frequency (background (BG), normal tissue (NT), adenocarcinoma (ADC), and squamous cell carcinoma (SqCC)) plotted against the diagnosis. (B) shows a PCA (two components) for the label frequency vector per TMA core.</figcaption></figure>

## References

1. Gupta, R.; Kurc, T.; Sharma, A.; Almeida, J.S.; Saltz, J. The Emergence of Pathomics. *Curr. Pathobiol. Rep.* **2019**, *7*, 73–84. [CrossRef]

2. Bui, M.M.; Asa, S.L.; Pantanowitz, L.; Parwani, A.; van der Laak, J.; Ung, C.; Balis, U.; Isaacs, M.; Glassy, E.; Manning, L. *Digital and Computational Pathology: Bring the Future into Focus*; Wolters Kluwer–Medknow Publications: Mumbai, India, 2019; Volume 10.

3. Abels, E.; Pantanowitz, L.; Aeffner, F.; Zarella, M.D.; van der Laak, J.; Bui, M.M.; Vemuri, V.N.; Parwani, A.V.; Gibbs, J.; Agosto-Arroyo, E. *Computational Pathology Definitions, Best Practices, and Recommendations for Regulatory Guidance: A White Paper from the Digital Pathology Association*; Wiley Online Library: Hoboken, NJ, USA, 2019; Volume 249, pp. 286–294. ISBN 0022-3417.

4. Ma, B.; Guo, Y.; Hu, W.; Yuan, F.; Zhu, Z.; Yu, Y.; Zou, H. Artificial Intelligence-Based Multiclass Classification of Benign or Malignant Mucosal Lesions of the Stomach. *Front. Pharmacol.* **2020**, *11*, 1542. [CrossRef] [PubMed]

5. McAlpine, E.D.; Michelow, P.; Celik, T. *The Utility of Unsupervised Machine Learning in Anatomic Pathology*; Oxford University Press: New York, NY, USA, 2022; Volume 157, pp. 5–14.

6. Hou, L.; Agarwal, A.; Samaras, D.; Kurc, T.M.; Gupta, R.R.; Saltz, J.H. Robust histopathology image analysis: To label or to synthesize? In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019*; pp. 8533–8542.
7. Ravi, D.; Wong, C.; Deligianni, F.; Berthelot, M.; Andreu-Perez, J.; Lo, B.; Yang, G.Z. Deep learning for health informatics. *IEEE J. Biomed. Health Inf.* **2016**, *21*, 4–21. [CrossRef] [PubMed]

8. Hosseini, M.S.; Chan, L.; Tse, G.; Tang, M.; Deng, J.; Norouzi, S.; Rowsell, C.; Plataniotis, K.N.; Damaskinos, S. Atlas of digital pathology: A generalized hierarchical histological tissue type-annotated database for deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–17 June 2019; pp. 11747–11756.

9. Roohi, A.; Faust, K.; Djuric, U.; Diamandis, P. Unsupervised machine learning in pathology. *Surg. Pathol. Clin.* **2020**, *13*, 349–358. [CrossRef] [PubMed]

10. Rundo, L.; Militello, C.; Vitabile, S.; Russo, G.; Sala, E.; Gilardi, M.C. *A Survey on Nature-Inspired Medical Image Analysis: A Step Further in Biomedical Data Integration*; IOS Press: Amsterdam, The Netherlands, 2020; Volume 171, pp. 345–365.

11. Medela, A.; Picon, A.; Saratxaga, C.L.; Belar, O.; Cabezón, V.; Cicchi, R.; Bilbao, R.; Glover, B. Few shot learning in histopathological images: Reducing the need of labeled data on biological datasets. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy, 8–11 April 2019; pp. 1860–1864.

12. Deuschel, J.; Firnbach, D.; Geppert, C.I.; Eckstein, M.; Hartmann, A.; Bruns, V.; Kuritcyn, P.; Dexl, J.; Hartmann, D.; Perrin, D.; et al. Multi-Prototype Few-shot Learning in Histopathology. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 620–628.

13. Mahapatra, D.; Bozorgtabar, B.; Ge, Z. Medical Image Classification Using Generalized Zero Shot Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 3344–3353.

14. Mahapatra, D.; Bozorgtabar, B.; Kuanar, S.; Ge, Z. Self-supervised multimodal generalized zero shot learning for gleason grading. In *Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health*; Springer: Berlin/Heidelberg, Germany, 2021; pp. 46–56.

15. Kanezaki, A. Unsupervised Image Segmentation by Backpropagation. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 1543–1547. [CrossRef]

16. Kim, W.; Kanezaki, A.; Tanaka, M. Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clustering. *IEEE Trans. Image Process.* **2020**, *29*, 8055–8068. [CrossRef]

17. Bankhead, P.; Loughrey, M.B.; Fernández, J.A.; Dombrowski, Y.; McArt, D.G.; Dunne, P.D.; McQuaid, S.; Gray, R.T.; Murray, L.J.; Coleman, H.G.; et al. QuPath: Open source software for digital pathology image analysis. *Sci. Rep.* **2017**, *7*, 16878. [CrossRef] [PubMed]

18. Ruifrok, A.C.; Johnston, D.A. Quantification of histochemical staining by color deconvolution. *Anal. Quant. Cytol. Histol.* **2001**, *23*, 291–299. [PubMed]

19. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In *Advances in Neural Information Processing Systems*; Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F.D., Fox, E., Garnett, R., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2019; Volume 32.

20. Yakubovskiy, P. *Segmentation Models Pytorch*; GitHub Repository: Online, 2020.

21. Ma, J.; Chen, J.; Ng, M.; Huang, R.; Li, Y.; Li, C.; Yang, X.; Martel, A.L. Loss odyssey in medical image segmentation. *Med. Image Anal.* **2021**, *71*, 102035. [CrossRef] [PubMed]

22. Yeung, M.; Sala, E.; Schönlieb, C.B.; Rundo, L. Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. *Comput. Med. Imaging Graph.* **2022**, *95*, 102026. [CrossRef] [PubMed]

23. Khvedchenya, E. Pytorch Toolbelt. Available online: https://github.com/BloodAxe/pytorch-toolbelt (accessed on 28 April 2022)

24. Salehi, S.S.M.; Erdogmus, D.; Gholipour, A. Tversky loss function for image segmentation using 3D fully convolutional deep networks. In *Machine Learning in Medical Imaging, Proceedings of the 8th International Workshop, MLMI 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, 10 September 2017*; Springer: Cham, Switzerland, 2017. [CrossRef]

25. Bokhovkin, A.; Burnaev, E. Boundary Loss for Remote Sensing Imagery Semantic Segmentation. In *International Symposium on Neural Networks*; Springer: Cham, Switzerland, 2019; pp. 388–401.

26. Kervadec, H.; Bouchtiba, J.; Desrosiers, C.; Granger, E.; Dolz, J.; Ben Ayed, I. Boundary loss for highly unbalanced segmentation. In Proceedings of the 2nd International Conference on Medical Imaging with Deep Learning, London, UK, 8–10 July 2019; Volume 67, p. 101851. [CrossRef]

27. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. *J. Mach. Learn. Res.* **2011**, *12*, 2825–2830.

28. Kim, H.; Ganslandt, T.; Miethke, T.; Neumaier, M.; Kittel, M. Deep Learning Frameworks for Rapid Gram Stain Image Data Interpretation: Protocol for a Retrospective Data Analysis. *JMIR Res. Protoc.* **2020**, *9*, e16843. [CrossRef] [PubMed]

29. Boehm, K.M.; Khosravi, P.; Vanguri, R.; Gao, J.; Shah, S.P. *Harnessing Multimodal Data Integration to Advance Precision Oncology*; Nature Publishing Group: Berlin, Germany, 2021; pp. 1–13.

30. Tizhoosh, H.R.; Pantanowitz, L. *Artificial Intelligence and Digital Pathology: Challenges and Opportunities*; Wolters Kluwer–Medknow Publications: Mumbai, India, 2018; Volume 9

31. World Health Organization. *WHO Classification of Tumours of the Lung, Pleura, Thymus and Heart*; World Health Organization: Geneva, Switzerland, 2015; Volume 7.
32. Kriegsmann, M.; Haag, C.; Weis, C.A.; Steinbuss, G.; Warth, A.; Zgorzelski, C.; Muley, T.; Winter, H.; Eichhorn, M.E.; Eichhorn, F.; et al. Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer. *Cancers* **2020**, *12*, 1604. [CrossRef] [PubMed]

33. Tschuchnig, M.E.; Oostingh, G.J.; Gadermayr, M. Generative adversarial networks in digital pathology: A survey on trends and future potential. *Patterns* **2020**, *1*, 100089. [CrossRef] [PubMed]

34. Peikari, M.; Salama, S.; Nofech-Mozes, S.; Martel, A.L. *A Cluster-Then-Label Semi-Supervised Learning Approach for Pathology Image Classification*; Nature Publishing Group: Berlin, Germany, 2018; Volume 8, pp. 1–13.

35. Janowczyk, A.; Basavanhally, A.; Madabhushi, A. Stain normalization using sparse autoencoders (StaNoSA): Application to digital pathology. *Comput. Med. Imaging Graph.* **2017**, *57*, 50–61. [CrossRef]

36. Song, T.H.; Sanchez, V.; Daly, H.E.; Rajpoot, N.M. Simultaneous cell detection and classification in bone marrow histology images. *IEEE J. Biomed. Health Inf.* **2018**, *23*, 1469–1476. [CrossRef]

37. Xu, J.; Xiang, L.; Liu, Q.; Gilmore, H.; Wu, J.; Tang, J.; Madabhushi, A. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. *IEEE Trans. Med. Imaging* **2015**, *35*, 119–130. [CrossRef]

38. Pourian, N.; Karthikeyan, S.; Manjunath, B.S. Weakly supervised graph based semantic segmentation by learning communities of image-parts. In Proceedings of the IEEE International Conference on Computer Vision, Washington, DC, USA, 7–13 December 2015; pp. 1359–1367.

39. Wigness, M.; Rogers, J.G. Unsupervised semantic scene labeling for streaming data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4612–4621.

40. Mills, S. *Histology for Pathologists*; LWW Medical Book Collection; Lippincott Williams & Wilkins: Philadelphia, PA, USA, 2007.

41. Fletcher, C. *Diagnostic Histopathology of Tumors*; Elsevier Health Sciences: Amsterdam, The Netherlands, 2013.
<figure><img src="image_24.png" /></figure>

Article

# Deep Learning-Based Automatic Segmentation of Mandible and Maxilla in Multi-Center CT Images

Seungbin Park <fnref n="1" />, Hannah Kim <fnref n="2" />, Eungjune Shim <fnref n="2" />, Bo-Yeon Hwang <fnref n="3" />, Youngjun Kim <fnref n="1" />,<fnref n="2" />, Jung-Woo Lee <fnref n="3" />,* and Hyunseok Seo <fnref n="1" />,*

**Abstract:** Sophisticated segmentation of the craniomaxillofacial bones (the mandible and maxilla) in computed tomography (CT) is essential for diagnosis and treatment planning for craniomaxillofacial surgeries. Conventional manual segmentation is time-consuming and challenging due to intrinsic properties of craniomaxillofacial bones and head CT such as the variance in the anatomical structures, low contrast of soft tissue, and artifacts caused by metal implants. However, data-driven segmentation methods, including deep learning, require a large consistent dataset, which creates a bottleneck in their clinical applications due to limited datasets. In this study, we propose a deep learning approach for the automatic segmentation of the mandible and maxilla in CT images and enhanced the compatibility for multi-center datasets. Four multi-center datasets acquired by various conditions were applied to create a scenario where the model was trained with one dataset and evaluated with the other datasets. For the neural network, we designed a hierarchical, parallel and multi-scale residual block to the U-Net (HPMR-U-Net). To evaluate the performance, segmentation with in-house dataset and with external datasets from multi-center were conducted in comparison to three other neural networks: U-Net, Res-U-Net and mU-Net. The results suggest that the segmentation performance of HPMR-U-Net is comparable to that of other models, with superior data compatibility.

**Keywords:** segmentation; mandible; craniomaxillofacial bone; deep learning; neural network; multi-center

## 1. Introduction

Segmentation of the craniomaxillofacial bones, such as the mandible and maxilla, in computed tomography (CT) images is one of the crucial steps for generating three-dimensional (3D) models that are required for the diagnosis and treatment planning of craniomaxillofacial deformities, craniofacial tumor resection, or free flap reconstruction of the mandible [1,2]. Additionally, 3D segmentation of organs at risk (OARs) in head and neck (H&N) CT including the mandible is a critical step in radiotherapy planning for H&N cancer treatment [3].

The conventional segmentation task is performed manually using professional software, which is labor-intensive and time-consuming in clinical practice [4,5]. Additionally, manual segmentation has limitations such as low reproducibility and operator variability. Moreover, accurate segmentation of head CT is challenging owing to the complexity of the anatomical structures, the low contrast of soft tissue, artifacts caused by metal implants, and variations between individual patients [6]. In specific, weak and false edges of condyles appearing in CT images adversely affect the accurate segmentation of the mandible [7]. Figure 1 shows examples of the difficulties in segmenting the mandible and maxilla.

**Citation:** Park, S.; Kim, H.; Shim, E.; Hwang, B.-Y.; Kim, Y.; Lee, J.-W.; Seo, H. Deep Learning-Based Automatic Segmentation of Mandible and Maxilla in Multi-Center CT Images. *Appl. Sci.* **2022**, *12*, 1358. https://doi.org/10.3390/app12031358

Academic Editor: Carmelo Militello

Received: 10 December 2021

Accepted: 21 January 2022

Published: 27 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_25.png" /></figure>

Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ Center for Bionics, Korea Institute of Science and Technology, Seoul 02792, Korea; seungbin201803@gmail.com (S.P.); ceo@imagoworks.ai (Y.K.)</footnote>

<footnote n="2">² Imagoworks, Inc., Seoul 06611, Korea; hannah.kim@imagoworks.ai (H.K.); ejshim@imagoworks.ai (E.S.)</footnote>

<footnote n="3">³ Department of Oral and Maxillofacial Surgery, School of Dentistry, Kyung Hee University, Seoul 02447, Korea; bo0426@hanmail.net</footnote>

<footnote>* Correspondence: omsace@khu.ac.kr (J.-W.L.); seo@kist.re.kr (H.S.)</footnote>
<figure><img src="image_26.png" /><figcaption>**Figure 1.** Difficulties in mandible and maxilla segmentation. (a) Metal artifacts caused by dental implants (b) Difficulty in distinguishing mandibular and maxillary teeth, or mandible and midface (c) Low intensity and thin edges in condyle (d) Inter-patient anatomical variance.</figcaption></figure>

Automatic segmentation can improve efficiency and reliability, reducing segmentation time and clinician workload [7]. Numerous studies exist on automatic or semi-automatic segmentation of the mandible from CT scans, including OARs. In the Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2015 Head and Neck Auto Segmentation Challenge [8], various approaches were proposed for the segmentation of OARs including the mandible. The use of public datasets, such as the Public Domain Database for Computational Anatomy (PDDCA) version 1.4.1, which was provided for the challenge, and how to evaluate the model performance have been a standard in head CT segmentation research. Most of these approaches utilize atlas-based methods [9] or model-based methods [10].

Atlas-based methods perform segmentation on novel data by image registration using the prior knowledge from the structures of interest [11]. Although atlas-based methods are popular and widely used for anatomy segmentation, they are sensitive to anatomical variations as they use a fixed set of atlases [12]. Moreover, they are computationally expansive and require many minutes to complete one registration task [13].

Statistical model-based methods utilize a statistical appearance model [14]. The models that best represent the shape or appearance variations in the structure of interest, which are obtained from training with a set of images and segmentations, are selected for a new patient image [15]. However, the shape or appearance described by the statistical model is limited to specific shapes, which gives it less flexibility unless large training sets are employed.

In some studies, atlas-based and statistical model-based methods have been combined with each other or with another method, leading to various other approaches for automatic mandible segmentation. Albrecht et al. [16] used a multi-atlas to obtain an initial segmentation of the OAR and an active shape model to refine the initial segmentation. Aghdasi et al. [17] employed anatomic landmarks and prior knowledge for segmentation. Chuang et al. [18] proposed a registration-based semi-automatic mandible segmentation pipeline that uses a nonlinear diffeomorphic method to register preprocessed test CT scans on the reference templates.

Recently, as convolutional neural networks (CNNs) have become more effective in computer vision, research on deep learning for medical image segmentation has increased exponentially [19]. The first deep learning-based algorithm utilizing a CNN for the segmentation of OARs in H&N CT was proposed by Ibragimov et al. [20], who employed a network
with three convolution layers. Tong et al. [21] then incorporated a CNN with the pretrained
shape representation model. Beyond simple CNNs, U-Net [22] has been one of the most
popular CNNs for medical image segmentation. Compared with other CNNs, U-Net,
with a simple and flexible structure, shows an outstanding performance in segmentation
extracting image features by multi-scale recognition and fusion [23]. Several approaches
have been developed by applying the U-Net structure as a baseline for mandible segmenta-
tion. Qiu et al. [1] used three U-Nets for orthogonal planes with dice loss to segment the
mandible. AnatomyNet [13] was proposed to segment OARs from H&N CT, which was
built on a 3D U-net architecture. A two-stage segmentation framework for OAR in CT was
also proposed, which employs two 3D U-Nets for localization and segmentation [24].

Several studies have utilized U-Net with other structures together as well. Both a faster
regional CNN and attention U-Net for localization and segmentation have been introduced
by Lei et al. [24]. A recurrent segmentation CNN was proposed that embeds the CNN into
a recurrent neural network for segmentation of the mandible from CT [7]. An attention
mechanism, which has been advanced with deep learning models in computer vision.
has been incorporated to U-Net for segmentation. Squeeze-and-excitation blocks were
incorporated into U-Net for prostate zonal segmentation of multi-institutional MRI datasets,
enhancing both intra- and cross-dataset generalization [25]. An attention gate model that
can be integrated into CNN models was proposed to automatically learn to focus on target
structures [26]. Focus U-Net with attention gate for spatial and channel-based attention
was proposed for fast and accurate polyp segmentation [27].

However, there is an inevitable and considerable pitfall in data-driven methods includ-
ing deep learning, which is the lack of data compatibility; that is, the method may fail to
accurately segment images with varying properties, such as those acquired using different
CT scanners and imaging protocols [28]. The compatibility of dataset in the models refers
to the ability of models to inference the input images that have different distributions in
the latent space from the multi-center training dataset [28]. In general, datasets are limited
so that they cannot fully represent the general patient population in the clinic [29]. As a
result, models trained on the specific center domain do not perform well on a different
center domains with disparate data distribution [30]. This drawback is more significant
when applying deep learning clinically on images from other institutions. For example, it
is known that the Hounsfield units measurement varies between scanners [31]. The results
of models targeted to CT can vary depending on the imaging parameters, the scanner type,
calibration, or the scan date [29,32,33]. That is, multicenter data tend to have different
data distributions, making trained neural network impractical. With consideration for this
variability, it has been recently been required to test the artificial intelligence model with an
external dataset [32]. From these limitations in clinical applications, data compatibility in
deep learning for medical images has been an essential challenge to be addressed.

To solve this problem, research has been conducted to utilize multicenter data in
neural network training [33,34]. Another potential solution to this problem is transfer
learning [35-37], which trains with more easily obtained datasets from different domains
to enhance performance [38]. However, these approaches have limitations for clinical use,
as available medical data are scarce compared to natural images and are not sufficient for
deep learning. Furthermore, labeling is more challenging with medical data.

In this study, we propose a framework for automated 3D segmentation of the mandible
and maxilla using deep learning. We aim not only to accurately delineate the mandible
and maxilla from CT, but also to improve the compatibility of multicenter data so that the
model performs well on new domain data. To this end, we employed four multi-center
datasets acquired by various conditions, with one used to train the models, and three
used to evaluate the performance of the segmentation and the data compatibility. For the
neural network, we applied residual connections [39] to U-Net, as it has been empirically
and theoretically determined that the generalization is improved in residual networks
compared with non-residual networks [40,41].
## 2. Materials and Methods

### 2.1. Data

We utilized four datasets: two of them from different centers (CenterA and CenterB) including mandible and maxilla segmentations and two public datasets (PDDCA and TCIA) for OAR segmentation in H&N CT. The CenterA dataset was randomly divided into training, validation, and test datasets, consisting of 146, 10, and 15 sets at the patient level, respectively. The training dataset was used to train the models, whereas the validation dataset was used to tune the hyperparameters of the models and check the validity of the training process. The test dataset from CenterA and other datasets were completely separated from the training and validation datasets, and were used for evaluating the performance of the models. Specifically, the PDDCA, TCIA, and CenterB datasets are external datasets that were used to evaluate the models for dataset compatibility. Detailed data characteristics of all datasets, including the number and size of slices, pixel spacing, and slice thicknesses, are presented in Table 1.

<table><caption>**Table 1.** Properties of the datasets.</caption><thead><tr><th>Dataset</th><th>CenterA</th><th>PDDCA †</th><th>TCIA †</th><th>CenterB</th></tr></thead><tbody><tr><td>No. of sets</td><td>171<br>(Train: 146, Validation: 10, Test: 15)</td><td>15</td><td>28</td><td>15</td></tr><tr><td>Acquisition type</td><td>MDCT</td><td>MDCT</td><td>MDCT</td><td>CBCT</td></tr><tr><td>Target structure</td><td>Mandible & Maxilla</td><td>OARs</td><td>OARs</td><td>Mandible & Maxilla</td></tr><tr><td>No. of slices</td><td>166–450,<br>208 ± 32</td><td>109–263,<br>154 ± 36</td><td>61–110,<br>93 ± 12</td><td>432</td></tr><tr><td>Slice size [pixel]</td><td>512</td><td>576</td><td>512</td><td>512</td></tr><tr><td>Pixel spacing [mm]</td><td>0.36–0.49,<br>0.44 ± 0.03</td><td>0.98–1.27, 1.11 ± 0.10</td><td>0.94–1.27, 1.04 ± 0.10</td><td>0.40</td></tr><tr><td>Slice thickness [mm]</td><td>0.50–1.04,<br>0.99 ± 0.07</td><td>2.0–3.0, 2.73 ± 0.31</td><td>2.50</td><td>0.40</td></tr></tbody></table>

CenterA and CenterB datasets include CT images and the corresponding segmentation of the mandible and maxilla provided by the clinical experts of oral and maxillofacial surgery department and orthodontic department, respectively. Targets in CenterA datasets were delineated manually by an expert surgeon (B.Y.H.) from Kyung Hee University Hospital, Seoul, Korea. Ethical approval was received from the institutional review board (IRB) (approval number KH-DT19033) for CenterA dataset. CenterB dataset was built with 15 sets of dental CBCT (i-CAT 17-19TM, Imaging Science International) from Chungang University Hospital, Seoul, Korea (approval number 1922-007-362). Those CT images were segmented by two well-trained biomedical engineers supervised by a clinical expert.

The PDDCA dataset is a public dataset for OAR segmentation in the H&N region of CT images released at the 2015 MICCAI H&N radiotherapy OAR segmentation challenge [8] provided and maintained by Dr. Sharp at Harvard Medical School. The CT scans in the dataset are available via the Cancer Imaging Archive (TCIA) and are originally from the radiation therapy oncology group (RTOG) 0522 study, which includes multi-institutional clinical studies from patients with stage III or IV H&N carcinoma [42]. The dataset consists of 48 H&N CT images with nine OAR structures manually re-segmented by experts for uniform quality and consistency. In the challenge, the dataset was divided into 25 training sets, 10 off-site test sets, and 5 on-site test sets. In this study, we employed 15 test sets with mandible annotation.

The TCIA dataset [43] contains 31 CT scans from TCIA [44] and segmentations for 21 OARs, in which we only used mandible segmentation. They were delineated by an experienced radiographer, with additional peer arbitration by another radiographer and a radiation oncologist. Both the PDDCA and TCIA datasets include a selected part of the Head–Neck Cetuximab open source dataset [45]; owing to different selection criteria and

<footnote>† denotes the public dataset. ‘No. of slices’, ‘Pixel spacing’, and ‘Slice thickness’ are indicated as the range, the average, and the standard deviation across the cases or the exact value if they are all same.</footnote>
different train/validation/test set division, there are five scans present in both PDDCA and TCIA test sets.

Examples of all datasets are illustrated in Figure 2. CenterA dataset is different from the PDDCA and TCIA datasets in terms of pixel spacing, slice thickness, and the scan range of the CT images. Comparatively, the PDDCA and TCIA datasets include a wider range of bodies that target OARs. CenterB uses cone beam CT (CBCT), which is fundamentally different from multidetector CT (MDCT) datasets, meaning the performance of a model trained with MDCT may be hindered when inferencing CBCT. Generally, segmentation of CBCT is more laborious and time-consuming than MDCT as the edge of the image is more blurred and noisy. Additionally, CenterB dataset includes many cases with orthognathic surgery or orthodontics, which makes segmentation more difficult owing to the noise caused by surgery plates or orthodontic appliances (Figure 3). By externally testing using datasets, including PDDCA, TCIA, and CenterB datasets, with various characteristics, it was possible to evaluate the compatibility of the models.

<figure><img src="image_27.png" /><figcaption>**Figure 2.** Example cases of the datasets.</figcaption></figure>

All datasets were preprocessed using the same procedure. A threshold of −1000 and 2500 HU was employed for each scan and normalized between zero and one. Both CT scans and segmentation slices were cropped to fit the skull. All CT and segmentation volumes were resampled to be isotropic (512 × 512 × 512). For a fair evaluation, the predicted segmentations were conversely uncropped and resampled into the original spacing and thickness before the evaluation metrics were calculated. For the PDDCA and TCIA datasets, we only used the range of the mandible for the training dataset, while using the entire range of slices for the validation and testing.
<figure><img src="image_28.png" /><figcaption>**Figure 3.** Examples of noises caused by surgical plates, orthodontic device, and dental implants in CenterB dataset, which make it more challenging to delineate the CT images.</figcaption></figure>

Datasets were input to the models as 2.5D [46], in which the input was a volume of images consisting of the target slice and its adjacent slices, and the segmentation map corresponding to the center target slice was produced as an output of the model. This method enables the use of adjacent context information in 3D, whilst lowering the computational power required relative to 3D inputs. The proposed approach is applicable to mandible and maxilla segmentation as the adjacent upper and lower spatial information is important for distinguishing the mandible and maxilla in a slice that appears similar. In this study, the 11 slices, composed of one target slice and five upper and lower slices, were input for one slice of the segmentation map.

## 2.2. *Framework and Network Architectures*

The overall framework and detailed architectures of neural networks are displayed in Figure 4. Preprocessed CT scans are input to the neural network as 2.5D, which outputs one segmentation mask map for each target slice. This process was repeated for all slices in each patient scan. Afterwards, the segmented volume for each patient was post-processed.

For the neural network, we applied a hierarchical, parallel, and multi-scale residual (HPMR) block [47] to U-Net to enhance the data compatibility of the CNN model. This block was first designed to enhance the performance of a CNN for landmark localization with limited computational resources. The starting point of the architecture is a residual bottleneck block [39] that enables the stable optimization of a deeper model by assisting the propagation of information both forward and backward, improving performance. The other basis for the architecture is the inception block [48], which concatenates features from parallel paths with different receptive field sizes. Compared to the inception residual block, the HPMR block has a smaller number of parameters with the advantage of a parallel path. Compared to the existing research, we combined HPMR block to U-Net and showed its performance on the segmentation task. We used HPMR block for efficient learning to utilize advances of residual bottleneck block and parallel path with the lower number of parameters compared to using inception blocks.

We compared U-Net with HPMR blocks (HPMR-U-Net) to its base component architecture, U-Net, and U-Net with residual blocks (Res-U-Net) to verify the effects of HPMR blocks. Additionally, modified U-Net (mU-Net) [49] was selected as another state-of-the-art segmentation CNN model for comparison because it requires minimum increase of network parameters. Its residual block is composed of deconvolution and activation operations to pass features to the skip connection of the U-Net adaptively with the object size. mU-Net is designed not only to extract high-level features of large object edges, but also high-level global features of small objects. We hypothesized that the increase in the complexity of model, i.e., the increased number of parameters in the neural network, would hinder the data compatibility of the model. It is well known that overfitting, which impedes the data compatibility of a model, occurs when the number of parameters increases [50].
Therefore, we chose a simpler neural network with lower number of parameters than other state-of-the-art neural networks for comparison.

<figure><img src="image_29.png" /><figcaption>**Figure 4.** Overall framework and block architectures of neural networks. (a) Overall framework. The numbers above the boxes refer to the channel number of the feature maps. The orange boxes for each neural network are represented in (b–e), with (b) block architecture of U-Net, (c) block architecture of Res-U-Net, (d) block architecture of mU-Net, and (e) block architecture of HPMR-U-Net.</figcaption></figure>

All networks were trained using PyTorch framework in Python under the same conditions for comparison. They were trained with a batch size of 10 for 30 epochs. We employed cross entropy loss as a loss function and Adam optimization with a learning rate of $10^{-5}$. Training and evaluation were performed on the computer hardware resources of a Nvidia GeForce RTX 3090 with 24 GB memory and 16 of DIMM DDR4 Synchronous 2666 MHz with 32 GiB in a Linux environment.

## 2.3. Performance Evaluation

To evaluate the regular segmentation performance of the models, an in-house test was conducted with the separated test portion of CenterA dataset, with the ground truths and output segmentations from the models compared. Additionally, an external test was performed to evaluate the data compatibility in the models. Output segmentations for CT scans in external datasets (PDDCA, TCIA, and CenterB) were obtained and compared with the ground truths. The external test characterizes how the model can be utilized generally in varied data, which is common in clinical settings. In the absence of maxilla segmentations in the PDDCA and TCIA datasets, only mandible segmentations were considered. To quantitatively evaluate the segmentation performance of the models, we used the Dice
coefficient (DC), 95% Hausdorff distance (95HD) and average surface distance (ASD) as evaluation metrics. Additionally, we qualitatively evaluated the segmentation results of the models by visualizing them in 3D.

The DC measures the degree of volumetric overlap between two volumes. It is defined as

$$DC = \frac{2|GT \cap OUT|}{|GT| + |OUT|'} \quad (1)$$

where GT and OUT are the labeled voxel sets of the manual segmentation ground truth and output segmentation from the model, respectively.

The 95HD and ASD are distance-related metrics, with 95HD being the 95th percentile of the Hausdorff distance (HD) between the GT and OUT points. HD measures the distance of a point in the GT to the nearest point in the OUT. It is defined as

$$\max_{gt \in GT} \min_{out \in OUT} \|gt - out\|. \quad (2)$$

The 95th percentile is used to eliminate the impact of outliers from a small subset of inaccurate points when evaluating the overall segmentation performance. ASD measures the average distance between the GT and the OUT, defined as:

$$ASD = \frac{1}{2} \left\{ \frac{\sum_{out \in OUT} d(out, GT)}{|OUT|} + \frac{\sum_{gt \in GT} d(gt, OUT)}{|GT|} \right\}, \quad (3)$$

where d(out, GT) is the minimum distance of a voxel on OUT to the voxels on GT, and d(gt, OUT) is the minimum distance of voxel gt on GT to the voxels on OUT.

### 3. Results

Tables 2 and 3 display the calculated evaluation metrics between the ground truths and the model outputs for the in-house and external tests. In the in-house test with the CenterA dataset, although the scores of HPMR-U-Net were not the best among the models, the score differences were lower compared to those for the other datasets. From the result, it can be inferred that the performance of HPMR-U-Net for the CenterA dataset was comparable to that of the other models. In the external tests, the scores of HPMR-U-Net ranked first for all external datasets. The results indicate that HPMR-U-Net has the highest performance among the models in this study for the external datasets. Comparing results among external datasets, the differences in scores were the largest in the CenterB dataset, where CenterB dataset may have the largest characteristic difference in the image obtained by CBCT as compared to CenterA dataset acquired by MDCT.

<table><caption>Table 2. Results of in-house and external tests for mandible segmentation. The best case is bolded.</caption><thead><tr><th rowspan="3"></th><th colspan="6">In-House Test</th><th colspan="6">External Test</th></tr><tr><th colspan="3">CenterA</th><th colspan="3">PDDCA</th><th colspan="3">TCIA</th><th colspan="3">CenterB</th></tr><tr><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th></tr></thead><tbody><tr><td>U-Net</td><td>98.3 ± 0.4</td><td>0.4 ± 0.1</td><td><strong>0.0</strong> ± 0.0</td><td>63.4 ± 20.2</td><td>7.3 ± 5.8</td><td>1.8 ± 3.5</td><td>62.8 ± 26.3</td><td>9.6 ± 11.8</td><td>3.2 ± 7.9</td><td>61.2 ± 17.9</td><td>33.7 ± 26.1</td><td>4.1 ± 4.0</td></tr><tr><td>Res-U-Net</td><td>98.2 ± 0.4</td><td>0.4 ± 0.1</td><td>0.1 ± 0.0</td><td>51.3 ± 20.1</td><td>13.5 ± 12.3</td><td>2.0 ± 1.8</td><td>46.3 ± 25.5</td><td>18.0 ± 19.8</td><td>6.5 ± 13.5</td><td>48.5 ± 13.1</td><td>28.8 ± 20.6</td><td>4.1 ± 3.3</td></tr><tr><td>mU-Net</td><td><strong>98.4</strong> ± 0.3</td><td><strong>0.4</strong> ± 0.0</td><td><strong>0.0</strong> ± 0.0</td><td>72.3 ± 21.6</td><td>5.6 ± 6.4</td><td>1.5 ± 3.5</td><td>71.4 ± 27.8</td><td>8.4 ± 12.9</td><td>2.5 ± 5.0</td><td>63.6 ± 14.7</td><td>22.5 ± 18.9</td><td>2.6 ± 2.2</td></tr><tr><td>HPMR-U-Net</td><td>97.4 ± 0.4</td><td>0.4 ± 0.1</td><td>0.1 ± 0.0</td><td><strong>86.5</strong> ± 3.9</td><td><strong>1.8</strong> ± 1.3</td><td><strong>0.2</strong> ± 0.1</td><td><strong>86.4</strong> ± 6.2</td><td><strong>2.8</strong> ± 7.7</td><td><strong>0.3</strong> ± 0.7</td><td><strong>77.7</strong> ± 4.1</td><td><strong>3.4</strong> ± 0.6</td><td><strong>0.7</strong> ± 0.2</td></tr></tbody></table>
<table><caption>**Table 3.** Results of in-house and external tests for maxilla segmentation. The best case is bolded.</caption><thead><tr><th rowspan="3"></th><th colspan="3">In-House Test</th><th colspan="3">External Test</th></tr><tr><th colspan="3">CenterA</th><th colspan="3">CenterB</th></tr><tr><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th><th>DC [%]</th><th>95HD [mm]</th><th>ASD [mm]</th></tr></thead><tbody><tr><td>U-Net</td><td><strong>96.5</strong><br>± 0.8</td><td>0.4<br>± 0.1</td><td>0.1<br>± 0.0</td><td>75.0<br>± 5.7</td><td>9.0<br>± 8.8</td><td>1.1<br>± 0.8</td></tr><tr><td>Res-U-Net</td><td>96.2<br>± 0.8</td><td>0.4<br>± 0.1</td><td>0.1<br>± 0.0</td><td>67.6<br>± 12.3</td><td>17.1<br>± 18.1</td><td>2.5<br>± 3.4</td></tr><tr><td>mU-Net</td><td>96.5<br>± 0.7</td><td><strong>0.4</strong><br>± 0.0</td><td>0.1<br>± 0.0</td><td>75.9<br>± 5.1</td><td>8.6<br>± 7.9</td><td>1.0<br>± 0.7</td></tr><tr><td>HPMR-U-Net</td><td>90.2<br>± 19.5</td><td>0.5<br>± 0.1</td><td>0.1<br>± 0.0</td><td><strong>82.8</strong><br>± 3.2</td><td><strong>2.7</strong><br>± 1.6</td><td><strong>0.4</strong><br>± 0.2</td></tr></tbody></table>

Figures 5–8 show 3D rendered ground truths and the highest DC cases of the output segmentations converted to isosurfaces from volumes for each dataset. Corresponding to the results of the quantitative tests, the ground truth and the outputs for the CenterA dataset are similar for all models, as shown in Figure 5. By contrast, for the external datasets, there are visually noticeable differences in the output segmentations of HPMR-U-Net and other models.

<figure><img src="image_30.png" /><figcaption>**Figure 5.** Sample case in the CenterA dataset. White and blue indicate the mandible and maxilla, respectively.</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>**Figure 6.** Sample case in the PDDCA dataset. White and blue indicate the mandible and maxilla, respectively. PDDCA has no maxilla ground truth.</figcaption></figure>
There were prominent decreases in quality of segmentations from other models for the external datasets. For the PDDCA dataset in Figure 6, the DC scores for the mandible of U-Net and mU-Net were 0.0%, as the model could not find the mandible at all, that is, they were unable to distinguish between the mandible and the maxilla. There were also many losses in the segmentations of the mandible and maxilla in the outputs of Res-U-Net. By contrast, the outputs of HPMR-U-Net were more intact and closer to the ground truth. As the teeth were included in the CenterA dataset segmentations that were used in training, the teeth were also segmented, despite not being in the ground truth. The results for the TCIA dataset in Figure 7 are also similar to those of the PDDCA dataset. U-Net and Res-U-Net failed to segment the mandible, which resulted in a 0.0% DC. Additionally, mU-Net included many portions of the maxilla in the mandible output and lost a large portion of the segmentations. However, HPMR-U-Net exhibited high performance with a DC of 91.7%. As displayed in Figure 8 for the CenterB dataset, HPMR-U-Net also showed the highest performance among the models, with many lost sections in the other models. Furthermore, the other models were more unable to accurately separate the mandible and maxilla compared to HPMR-U-Net.

<figure><img src="image_32.png" /><figcaption>**Figure 7.** Sample case in the TCIA dataset. White and blue indicate mandible and maxilla, respectively. TCIA has no maxilla ground truth.</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>**Figure 8.** Sample case in CenterB dataset. White and blue indicate the mandible and maxilla, respectively.</figcaption></figure>
Figure 9 shows rendered color maps in 3D for the distance from the ground truths to the output segmentations of the best DC case for the mandible in CenterA dataset to thoroughly examine the differences among the model outputs for this dataset. There were no significant differences, but the distances in the mandibular foramen were slightly different. This part is challenging to segment accurately owing to its small size, and the distance was less in the outputs of HPMR-U-Net than the other models.

<figure><img src="image_34.png" /><figcaption>**Figure 9.** Color maps of surface distance from the ground truths to the output segmentations of the best dice coefficient case in the CenterA dataset for the mandible. The pieces of images on the right side are enlargement of mandibular foramen.</figcaption></figure>

## 4. Discussion & Conclusions

The four neural networks compared in this research exhibited similar performance in the CenterA dataset, which was the domain used for training. Among other neural networks, U-Net and Res-U-Net were considered for the comparison because U-Net is a basic component of HPMR-U-Net, in which showing a difference would represent that HPMR block is effective compared to other basic architectures. Additionally, mU-Net was selected for the comparison as a state-of-the-art neural network for segmentation. We chose comparably simple neural networks because we hypothesized that the data compatibility of the more complex model with larger number of parameters would be worse because of overfitting. CenterA dataset was set as the train dataset because it was MDCT datasets in which easier to make ground truths than CBCT. With training with a dataset easier to constitute, we aimed to show the performance for other institutional MDCTs and CBCTs.

PDDCA and TCIA datasets were used for examples of MDCT and CenterB dataset for CBCT. For the external datasets of PDDCA, TCIA, and CenterB, HPMR-U-Net displayed significantly higher performance compared to the U-Net, Res-U-Net, and mU-Net models in both quantitative and qualitative evaluations.

All networks produced comparable results for data from the same cohort of the training dataset; however, they exhibited different results for data from out of the training dataset cohort. While the performances of other networks were degraded in the external datasets, HPMR-U-Net produced segmentation of the mandible and maxilla similar to
the ground truths. From these results, HPMR-U-Net infers a high data compatibility for mandible and maxilla features in CT images.

The assumed differences in the data cohorts were reflected in the results. For the PDDCA and TCIA datasets, the performance degraded significantly, and the mandible and maxilla were not classified accurately. This is due to their slice thickness being different from that of CenterA dataset, even though they are MDCT. The inter-slice information is important to classify a pixel in a slice as the mandible or maxilla. The results for CenterB dataset were the worst among the external datasets for all models. The segmentation of CenterB dataset is more challenging as it is CBCT, which is not only different from the in-house dataset, but also contains more noise. Additionally, CenterB dataset contains variances in anatomical structure caused by surgeries and noise from surgical plates, orthodontic device, and dental implants (Figure 3). It is remarkable that the score difference between HPMR-U-Net and other models is significant for CenterB. For CBCT, which is a different image protocol than MDCT that was used to train, there was a significant degradation of performance in other models, but minimal degradation in HPMR-U-Net. This demonstrates that HPMR-U-Net is more robust than other models to various data domains that may be different from the training data.

We assume that one of the reasons for the better performance of HPMR-U-Net compared to Res-U-Net is the number of parameters. The higher the complexity of the hypothesis space of the deep neural network, the worse is the generalizability, according to the principle of Occam's razor [51]. The number of parameters in HPMR-U-Net is 12,042,179, which is smaller than Res-U-Net with 17,118,019, U-Net with 28,959,299, and mU-Net with 35,230,019. The HPMR block could efficiently decrease the overall number of parameters, which as a result could enhance the generalizability of the model while maintaining its segmentation performance.

In future work, an attempt will be made to improve the performance in external datasets for actual clinical applications when the neural network is trained with only one data domain. Additionally, the structure of the neural network with residual connections and HPMR block can be analyzed theoretically to establish the reason for the greater generalizability, which may lead to the design of a stronger neural network for generalization.

In this study, we applied deep learning to accurately segment the mandible and maxilla from CT and improve the compatibility in the segmentation model. To achieve this, we utilized HPMR-U-Net and compared its results with those of U-Net, Res-U-Net, and mU-Net with in-house and external tests. The results show that the segmentation performance of HPMR-U-Net in the in-house test dataset was comparable to that of the other models. In particular, the data compatibility of HPMR-U-Net was superior to other models in the external datasets of PDDCA, TCIA, and CenterB, which have varying properties such as image protocol, pixel spacing, slice thickness, and target range.

**Author Contributions:** Conceptualization, S.P., E.S., Y.K., J.-W.L. and H.S.; methodology, S.P., H.K. and H.S.; software, S.P., H.K. and E.S.; validation, S.P.; formal analysis, S.P.; investigation, S.P.; resources, Y.K. and H.S.; data curation, B.-Y.H. and J.-W.L.; writing—original draft preparation, S.P.; writing—review and editing, S.P., H.K., Y.K., J.-W.L. and H.S.; visualization, S.P. and H.K.; supervision, Y.K., J.-W.L. and H.S.; project administration, Y.K., J.-W.L. and H.S.; funding acquisition, Y.K., J.-W.L. and H.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by KIST Institutional Program (grant number: 2E31158) and the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 9991006675, 202011A02, KMDF_PR_20200901_0002). In addition, this research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI18C1224).
**Institutional Review Board Statement:** The study was approved by the Institutional Review Board of Chungang University Hospital, Seoul, Korea (1922-007-362) and Kyung Hee University Hospital, Seoul, Korea (KH-DT19033).

**Informed Consent Statement:** Patient consent was waived because it was stated that the data can be used retrospectively without DICOM tags in IRB approval.

**Data Availability Statement:** Data sharing is not applicable to this article.

**Acknowledgments:** The authors wish to express their thanks for the support of Uilyong Lee (Department Oral and maxillofacial Surgery, Chung-Ang University) in the collection of datasets.

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Qiu, B.; Guo, J.; Kraeima, J.; Glas, H.H.; Borra, R.J.; Witjes, M.J.; van Ooijen, P.M. Automatic Segmentation of the Mandible from Computed Tomography Scans for 3D Virtual Surgical Planning Using the Convolutional Neural Network. *Phys. Med. Biol.* **2019**, *64*, 175020. [CrossRef] [PubMed]

2. Wang, L.; Chen, K.C.; Gao, Y.; Shi, F.; Liao, S.; Li, G.; Shen, S.G.; Yan, J.; Lee, P.K.; Chow, B. Automated Bone Segmentation from Dental CBCT Images Using Patch-based Sparse Representation and Convex Optimization. *Med. Phys.* **2014**, *41*, 043503. [CrossRef] [PubMed]

3. Kodym, O.; Španěl, M.; Herout, A. Segmentation of Head and Neck Organs at Risk Using Cnn with Batch Dice Loss. In Proceedings of the German Conference on Pattern Recognition; Springer: Stuttgart, Germany, 2018; pp. 105–114.

4. Byrne, N.; Velasco Forte, M.; Tandon, A.; Valverde, I.; Hussain, T. A Systematic Review of Image Segmentation Methodology, Used in the Additive Manufacture of Patient-Specific 3D Printed Models of the Cardiovascular System. *JRSM Cardiovasc. Dis.* **2016**, *5*, 2048004016645467. [CrossRef] [PubMed]

5. Huff, T.J.; Ludwig, P.E.; Zuniga, J.M. The Potential for Machine Learning Algorithms to Improve and Reduce the Cost of 3-Dimensional Printing for Surgical Planning. *Expert Rev. Med. Devices* **2018**, *15*, 349–356. [CrossRef]

6. Wang, Z.; Wei, L.; Wang, L.; Gao, Y.; Chen, W.; Shen, D. Hierarchical Vertex Regression-Based Segmentation of Head and Neck CT Images for Radiotherapy Planning. *IEEE Trans. Image Process.* **2017**, *27*, 923–937. [CrossRef]

7. Qiu, B.; Guo, J.; Kraeima, J.; Glas, H.H.; Borra, R.J.; Witjes, M.J.; Ooijen, P.M.V. Recurrent Convolutional Neural Networks for Mandible Segmentation from Computed Tomography. *arXiv* **2020**, arXiv:2003.06486.

8. Raudaschl, P.F.; Zaffino, P.; Sharp, G.C.; Spadea, M.F.; Chen, A.; Dawant, B.M.; Albrecht, T.; Gass, T.; Langguth, C.; Lüthi, M. Evaluation of Segmentation Methods on Head and Neck CT: Auto-segmentation Challenge 2015. *Med. Phys.* **2017**, *44*, 2020–2036. [CrossRef]

9. Chen, A.; Dawant, B. A Multi-Atlas Approach for the Automatic Segmentation of Multiple Structures in Head and Neck CT Images. *MIDAS J.* **2015**. [CrossRef]

10. Mannion-Haworth, R.; Bowes, M.; Ashman, A.; Guillard, G.; Brett, A.; Vincent, G. Fully Automatic Segmentation of Head and Neck Organs Using Active Appearance Models. *MIDAS J.* **2015**. [CrossRef]

11. Han, X.; Hoogeman, M.S.; Levendag, P.C.; Hibbard, L.S.; Teguh, D.N.; Voet, P.; Cowen, A.C.; Wolf, T.K. *Atlas-Based Auto-Segmentation of Head and Neck CT Images*; Springer: Berlin/Heidelberg, Germany, 2008; pp. 434–441.

12. Linares, O.C.; Bianchi, J.; Raveli, D.; Neto, J.B.; Hamann, B. Mandible and Skull Segmentation in Cone Beam Computed Tomography Using Super-Voxels and Graph Clustering. *Vis. Comput.* **2019**, *35*, 1461–1474.

13. Zhu, W.; Huang, Y.; Zeng, L.; Chen, X.; Liu, Y.; Qian, Z.; Du, N.; Fan, W.; Xie, X. AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy. *Med. Phys.* **2019**, *46*, 576–589. [CrossRef] [PubMed]

14. Cootes, T.F.; Edwards, G.J.; Taylor, C.J. Active Appearance Models. *IEEE Trans. Pattern Anal. Mach. Intell.* **2001**, *23*, 681–685. [CrossRef]

15. Fritscher, K.D.; Peroni, M.; Zaffino, P.; Spadea, M.F.; Schubert, R.; Sharp, G. Automatic Segmentation of Head and Neck CT Images for Radiotherapy Treatment Planning Using Multiple Atlases, Statistical Appearance Models, and Geodesic Active Contours. *Med. Phys.* **2014**, *41*, 051910. [CrossRef] [PubMed]

16. Albrecht, T.; Gass, T.; Langguth, C.; Lüthi, M. Multi Atlas Segmentation with Active Shape Model Refinement for Multi-Organ Segmentation in Head and Neck Cancer Radiotherapy Planning. *MIDAS J.* **2015**. [CrossRef]

17. Aghdasi, N.; Li, Y.; Berens, A.; Moe, K.; Hannaford, B. Automatic Mandible Segmentation on CT Images Using Prior Anatomical Knowledge. *MIDAS J.* **2016**. [CrossRef]

18. Chuang, Y.J.; Doherty, B.M.; Adluru, N.; Chung, M.K.; Vorperian, H.K. A Novel Registration-Based Semi-Automatic Mandible Segmentation Pipeline Using Computed Tomography Images to Study Mandibular Development. *J. Comput. Assist. Tomogr.* **2018**, *42*, 306. [CrossRef]

19. Shen, D.; Wu, G.; Suk, H.-I. Deep Learning in Medical Image Analysis. *Annu. Rev. Biomed. Eng.* **2017**, *19*, 221–248. [CrossRef]

20. Ibragimov, B.; Xing, L. Segmentation of Organs-at-risks in Head and Neck CT Images Using Convolutional Neural Networks. *Med. Phys.* **2017**, *44*, 547–557. [CrossRef]