# Intelligent Data Analytics for Diagnosing Melanoma Skin Lesions via Deep Learning in IoT System

Authors: Shixiang Zhang; Shuaiqi Huang; Hongkai Wu; Zicong Yang; Yinda Chen
Journal: Mobile Information Systems
 (2021)

---

## Abstract

Melanoma is considered to be one of the most dangerous human malignancy, which is diagnosed visually or by dermoscopic analysis and histopathological examination. However, as these traditional methods are based on human experience and implemented manually, there have been great limitations for general usability in current clinical practice. In this paper, a novel hybrid machine learning approach is proposed to identify melanoma for skin healthcare in various cases. The proposed approach consists of classic machine learning methods, including convolutional neural networks (CNNs), EfficientNet, and XGBoost supervised machine learning. In the proposed approach, a deep learning model is trained directly from raw pixels and image labels for classification of skin lesions. Then, solely based on modeling of various features from patients, an XGBoost model is adopted to predict skin cancer. Following that, a diagnostic system which composed of the deep learning model and XGBoost model is developed to further improve the prediction efficiency and accuracy. Different from experience-based methods and solely image-based machine learning methods, the proposed approach is developed based on the theory of deep learning and feature engineering. Experiments show that the hybrid model outperforms single model like the traditional deep learning model or XGBoost model. Moreover, the data-driven-based characteristics can help the proposed approach develop a guideline for image analysis in other medical applications.


---

## Body

## 1. Introduction

Melanoma is considered to be one of the most dangerous and malignant skin cancers due to its capability of deep invasion via the lymphatic vessels and blood vessels. It was estimated that there was a 53% increase annually in new melanoma case diagnosis over the last decade [1–5], and the rates of melanoma occurrence are expected to increase worldwide over the next decade [6–11]. Despite melanoma causes a majority of deaths related to skin cancer, the survival rates are promising if it can be diagnosed and treated correctly at the earliest stage [12–17]. Otherwise, the estimated 5-year survival rate of a patient will be greatly reduced [18–25]. As melanoma is visible on the skin at its initial phase, early detection of melanoma is a crucial issue to reduce mortality and morbidity.To determine if the lesion is a benign or malignant tumor, besides clinical tests, clinicians traditionally diagnose by inspecting the details such as shape, color, and texture visually. However, as shown in Figure1, the low precision of the visual details may result in low clinical diagnostic accuracy and unnecessary treatments [26–31]. On the contrary, as dermoscopy is a noninvasive technology that could capture images of the skin with high resolution, the dermatologists can inspect details of the skin which are invisible to the naked eyes [32–34]. Therefore, the dermoscopy images have the most potential for disruption of the traditional diagnose procedure, and an increasing effort has been put in image-based inspection for early detection of melanoma by the research community [35, 36]. However, as the image-based diagnosis requires great deal of experience and is highly dependent on subjective judgement [33–35], the diagnostic accuracy may be greatly reduced under complex cases, especially for unskilled dermatologists [36–38]. It is reported that experts can achieve 90% sensitivity in skin cancer detection, while around 60% for unskilled clinicians [33, 39]. Despite several scoring systems and rule-based systems have been proposed to learn expert knowledge and improve the diagnostic performance of unskilled clinicians [40–44], the diagnosis process is time consuming, and the clinical diagnostic accuracy remains suboptimal for melanoma detection. Therefore, automatic analysis of digitized images with high diagnostic accuracy to assist dermatologists in differentiating early melanoma from benign skin lesions is in high demand and very crucial for public health [45–49].Figure 1
Examples of (a) clinical, (b) dermoscopic, and (c) histopathological images of melanoma.
(a)(b)(c)Due to the rapid development of computer technology in the areas of image processing and artificial intelligence, various image-based machine learning methods have been applied as diagnostic aids to distinguish malignant melanoma from many benign tumors without biopsy, which will reduce a vast number of needless biopsy procedures. Different from traditional methods which are explicitly developed for a set of static cases, the machine learning-based diagnostic systems are trained based on the prior dataset and can produce faster and more reliable diagnoses in novel scenarios. As the key step of a machine learning-based dermatology diagnostic system is to classify the detected skin lesions as melanoma or benign, various supervised techniques have been applied for skin cancer recognition. The artificial neural network (ANN) is the most common machine learning method for the classification of skin cancer. By using different combinations of skin features (such as color, lesion texture, and visual) as inputs, an ANN model was developed by Ruiz et al. [50] and Giotis et al. [51] for medical decision making. Besides ANN, variants of the decision trees such as the random forest classifier and decision forest classifier were preferred for diagnosis of melanoma [52–55]. Additionally, k-means clustering and support vector machine (SVM) were also adopted to classify skin lesions in many research studies [56]. Previous work in dermatological computer-aided diagnosis systems has significant potential for screening and early detection of malignant melanoma. However, due to the emphasis on the standardized tasks such as histological and dermoscopy image classification, many previous methods require extensive image preprocessing procedures to remove extraneous elements and extract important features before fed to a classifier for melanoma detection, which will result in a loss of accuracy as well as an increase in computational time. Therefore, the efficiency of image classification in dermatology diagnostic systems will be greatly reduced.In recent years, due to the exponential growth of computation power and sufficient dataset, deep learning methods have achieved great development in various computer vision problems. As these powerful algorithms can learn abstract features of raw images without any expert knowledge and preprocessing procedures, they have been widely adopted for image-based melanoma detection. Dorj et al. [57] adopted a combination of pretrained AlexNet convolutional neural networks (CNNs) and ECOC SVM model for rapid classification of four types of skin cancer patterns, and the experiment results demonstrate the efficiency and accuracy of the proposed method. Esteva et al. [58] demonstrated the automated classification of skin lesions by using a single deep CNN, which was trained from raw pixels and image labels directly. The developed model can be applicable to both dermoscopic and nonstandardized images. Despite the deep learning models could improve the detection performance by extracting useful features from raw images directly, the developed models are computational heavy and time consuming due to a fixed resource budget [59, 60]. Therefore, it is necessary to fully exploit the computation resources to enhance the modeling efficiency and effectiveness. In recent years, some deep learning-based applications are applied in intelligent analytics of Internet of things (IoT) [61, 62]. Moreover, as these deep learning models are focused on extracting features from images, there is a need to consider patient-level contextual information [63–65], which could better support dermatological clinic work and improve the diagnostic accuracy.Under such a circumstance, a novel hybrid machine learning approach is proposed to identify melanoma for skin healthcare in various cases. The proposed approach consists of classic machine learning methods, including convolutional neural networks (CNNs), EfficientNet, and XGBoost supervised machine learning. In the proposed approach, a deep learning model is trained directly from raw pixels and image labels for classification of skin lesions. Then, solely based on modeling of various features from patients, an XGBoost model is adopted to predict skin cancer. Following that, a diagnostic system which is composed of the deep learning model and XGBoost model is developed to further improve the prediction efficiency and accuracy. Different from experience-based methods and solely image-based machine learning methods, the proposed approach is developed based on the theory of deep learning and feature engineering. And it can learn the linear mapping from neural network and nonlinear mapping from XGBoost. Therefore, the modeling process is in high efficiency and accuracy. Moreover, the data-driven-based characteristics can help the proposed approach develop a guideline for image analysis in other medical applications. The rest of the paper is organized as follows. Section2 describes the basic theories and introduces the proposed hybrid machine learning approach. Case study and discussion are presented in Section 3. Finally, conclusions and future research directions are presented in Section 4.


## 2. The Proposed Hybrid Approach for Melanoma Diagnosing

In this section, the basic theories of the adopted approaches are described, and the proposed hybrid machine learning approach is presented in the next.


### 2.1. Convolutional Neural Network

A convolutional neural network (CNN) is the fundamental architecture of deep learning algorithms [66, 67], which has been widely adopted in most computer vision applications [68, 69]. Different from a traditional artificial neural network which has the structure of multilayer perceptron, a CNN architecture adopts the operations of “convolution’ and pooling” to extract useful features from input images for pattern recognition and classification [70]. As shown in Figure 2, the basic architecture of CNN consists of four main blocks: convolution layer, activation layer, pooling layer, and fully connected layer. The convolution layer serves as a “filter” because it will convert the observed pixel values of input image into one value based on the operation of convolution. Therefore, the original images will be reduced into a matrix with smaller size after passing over the convolution layer. Then, the activation layer will be introduced to train the filtered images through backpropagation. As the pooling layer has the function of downsampling and reducing the size of the matrix, the training speed will be further enhanced. Following that, the fully connected layer (a traditional multilayer perceptron) will output the classification results based on the overall training process.Figure 2
Basic architecture of a CNN.


### 2.2. EfficientNet

To capture more complex and fine-grained features, width, depth, and resolution are considered as three crucial dimensions of CNN architecture [71–73]. Moreover, as these scaling dimensions are not independent, it is important to balance them during scaling process to achieve better modeling efficiency and accuracy. However, as CNNs are normally trained based on a fixed resource budget, the developed neural architecture may be suboptimal according to the empirical study. Despite the dimensions of network can be scaled arbitrarily to coordinate the neural architecture, it will be inefficient due to the manual tuning and still induce suboptimal accuracy. In this case, an EfficientNet scaling method is adopted in this research to effectively balance these three crucial dimensions and optimize the network structure. Different from conventional single-dimension scaling, EfficientNet scaling can scale these three dimensions uniformly to obtain a family of compound models, which is more efficient and accurate than previous methods. Figure 3 demonstrates the difference between EfficientNets and traditional methods. In this research, due to the diversity of input images, EfficientNet is applied to boost the modeling efficiency and accuracy.Figure 3
The comparison between traditional scaling approaches and EfficientNet scaling: (a) baseline network; (b) resolution scaling; (c) width scaling; (d) depth scaling; (d) EfficientNet scaling.
(a)(b)(c)(d)(e)


### 2.3. XGBoost for Contextual Feature-Based Classification

Conventional supervised machine learning methods normally make predictions by using a single model which is simply developed from training dataset directly. However, it may be insufficient to guarantee reliable results due to the limitations of the adopted method. Despite a standard ensemble learning method can make predictions by combining the advantages of different multiple learners, each model may repeat the same mistakes due to the isolated training process. On the contrary, XGBoost is an iterative ensemble learning method which is based on decision trees. As shown in Figure4, rather than training all the models separately, XGBoost will train new models iteratively to correct the residuals made by previous models, and all the trained models will be added sequentially when no further improvements can be achieved. To fully exploit the space of patient-related contextual features and ensure the modeling performance, XGBoost is adopted in this research for feature-based classification.Figure 4
The basic modeling principles of the XGBoost method.


### 2.4. Proposed Hybrid Approach for Melanoma Diagnosing

The main objective of this research is to detect melanoma skin lesions using the obtained images and patient-related features. As shown in Figure5, the proposed approach consists of three phases. In Phase I, the input datasets including images and contextual features are preprocessed before training, respectively. For images, they are first processed by gridmask (transform and rotate) for data augmentation, and then a downsampling approach is adopted to balance bias dataset. In Phase II, the preprocessed datasets are used for model training. To boost modeling efficiency, the EfficientNet and grid search is adopted for CNN and XGBoost, respectively. Moreover, the K-fold method is adopted in the training process to balance bias datasets and reduce overfitting. Following that, in the last phase, the developed deep learning model and XGBoost model are combined as a hybrid model for melanoma diagnosing, and the weights of different models can be determined by user preference.Figure 5
The proposed hybrid approach for melanoma diagnosing.


## 2.1. Convolutional Neural Network

A convolutional neural network (CNN) is the fundamental architecture of deep learning algorithms [66, 67], which has been widely adopted in most computer vision applications [68, 69]. Different from a traditional artificial neural network which has the structure of multilayer perceptron, a CNN architecture adopts the operations of “convolution’ and pooling” to extract useful features from input images for pattern recognition and classification [70]. As shown in Figure 2, the basic architecture of CNN consists of four main blocks: convolution layer, activation layer, pooling layer, and fully connected layer. The convolution layer serves as a “filter” because it will convert the observed pixel values of input image into one value based on the operation of convolution. Therefore, the original images will be reduced into a matrix with smaller size after passing over the convolution layer. Then, the activation layer will be introduced to train the filtered images through backpropagation. As the pooling layer has the function of downsampling and reducing the size of the matrix, the training speed will be further enhanced. Following that, the fully connected layer (a traditional multilayer perceptron) will output the classification results based on the overall training process.Figure 2
Basic architecture of a CNN.


## 2.2. EfficientNet

To capture more complex and fine-grained features, width, depth, and resolution are considered as three crucial dimensions of CNN architecture [71–73]. Moreover, as these scaling dimensions are not independent, it is important to balance them during scaling process to achieve better modeling efficiency and accuracy. However, as CNNs are normally trained based on a fixed resource budget, the developed neural architecture may be suboptimal according to the empirical study. Despite the dimensions of network can be scaled arbitrarily to coordinate the neural architecture, it will be inefficient due to the manual tuning and still induce suboptimal accuracy. In this case, an EfficientNet scaling method is adopted in this research to effectively balance these three crucial dimensions and optimize the network structure. Different from conventional single-dimension scaling, EfficientNet scaling can scale these three dimensions uniformly to obtain a family of compound models, which is more efficient and accurate than previous methods. Figure 3 demonstrates the difference between EfficientNets and traditional methods. In this research, due to the diversity of input images, EfficientNet is applied to boost the modeling efficiency and accuracy.Figure 3
The comparison between traditional scaling approaches and EfficientNet scaling: (a) baseline network; (b) resolution scaling; (c) width scaling; (d) depth scaling; (d) EfficientNet scaling.
(a)(b)(c)(d)(e)


## 2.3. XGBoost for Contextual Feature-Based Classification

Conventional supervised machine learning methods normally make predictions by using a single model which is simply developed from training dataset directly. However, it may be insufficient to guarantee reliable results due to the limitations of the adopted method. Despite a standard ensemble learning method can make predictions by combining the advantages of different multiple learners, each model may repeat the same mistakes due to the isolated training process. On the contrary, XGBoost is an iterative ensemble learning method which is based on decision trees. As shown in Figure4, rather than training all the models separately, XGBoost will train new models iteratively to correct the residuals made by previous models, and all the trained models will be added sequentially when no further improvements can be achieved. To fully exploit the space of patient-related contextual features and ensure the modeling performance, XGBoost is adopted in this research for feature-based classification.Figure 4
The basic modeling principles of the XGBoost method.


## 2.4. Proposed Hybrid Approach for Melanoma Diagnosing

The main objective of this research is to detect melanoma skin lesions using the obtained images and patient-related features. As shown in Figure5, the proposed approach consists of three phases. In Phase I, the input datasets including images and contextual features are preprocessed before training, respectively. For images, they are first processed by gridmask (transform and rotate) for data augmentation, and then a downsampling approach is adopted to balance bias dataset. In Phase II, the preprocessed datasets are used for model training. To boost modeling efficiency, the EfficientNet and grid search is adopted for CNN and XGBoost, respectively. Moreover, the K-fold method is adopted in the training process to balance bias datasets and reduce overfitting. Following that, in the last phase, the developed deep learning model and XGBoost model are combined as a hybrid model for melanoma diagnosing, and the weights of different models can be determined by user preference.Figure 5
The proposed hybrid approach for melanoma diagnosing.


## 3. Case Study and Discussion

In this section, the experimental materials are described, and a case study is conducted based on the proposed hybrid machine learning approach in the next.


### 3.1. Experimental Materials

#### 3.1.1. Image Datasets

The datasets photographed from a consecutive sample of lesions were originated from International Skin Imaging Collaboration (ISIC) between 2018 and 2019. These images were acquired through dermatoscope and digital cameras at different resolutions in polarizing or nonpolarizing mode. The cases with missing or equivocal histopathologic reports or with extremely low image quality were excluded in the dataset. Following that, the remaining images were reviewed manually and labelled as melanoma or benign skin lesion for classification, and Figure6 demonstrates some examples of melanoma images for classification.Figure 6
Samples of melanoma images for classification.


#### 3.1.2. Patient-Related Contextual Information

Currently, deep learning and image processing methods have achieved expert-level performance in melanoma detection. However, as the existing image-based artificial intelligence classification algorithms mainly focus on extracting features from images, the diagnosis process is insufficient and may induce the discrepancy between prior challenges and clinical practice. In practice, the patient-related contextual information are normally considered as external attributes, which will be beneficial to clinicians for differentiating melanoma from benign skin lesions. Therefore, there is a need to extract meaningful features from patient-related contextual information, which could better support dermatological clinic work and improve the diagnostic accuracy. In this research, the patient-related contextual information which are extracted and encoded as external features are shown in Table1. Then, based on the encoded features, a supervised machine learning model will be developed to predict melanoma.Table 1
Features derived from patient-related contextual information.
Extracted featuresRange of encoded features (integer)Anatomic positions (APs)[0, 6]Diagnosis results (DRs)[0, 1]Sex[0, 1]Number of images per patient (NIP)[0, 9]Image size (IS)[0, 10]Mean color (MC)[0, 9]Age[0, 9]Minimum age (MIA)[0, 17]Maximum age (MAA)[0, 17]


### 3.2. Results Analysis

#### 3.2.1. Development of Deep Learning Model

For deep learning, data augmentation is usually preferred to increase the size of training dataset virtually, which will make the developed model more robust to the input dataset. In this research, some classical operations such as gridmask and image rotation are adopted for data augmentation, and Figure7 illustrates an example of input image after data augmentation.Figure 7
An example of data augmentation for input image: (a) original image; (b) gridmask; (c) rotation.
(a)(b)(c)After data augmentation, TensorFlow is adopted for the development of deep learning models. TensorFlow is an open-source framework which is developed by Google for machine learning and deep learning functions. As TensorFlow provides users the flexibility of focusing on the structure of models rather than mathematical details, it has been widely adopted in various areas, from academia and research to industrial fields. During the training process, different configurations of TensorFlow structure will affect the modeling performance. For example, Figure8 demonstrates the influence of epoch on learning rate. It is necessary to balance the modeling efficiency and accuracy by adjusting the model configurations. The initial configurations of TensorFlow adopted in this research for deep learning implementation are illustrated in Table 2.Figure 8
The influence of epoch on learning rate during model training.
(a)(b)Table 2
Initial configurations of TensorFlow for deep learning implementation.
ParametersSettingsNumber of folders6Number of fully connected layers4Image size384 pixelsBalance ratio0.08Batch size64Epochs25Type of EfficientNetEfficientNet_6Loss typeBinary CrossEntropyLabel smoothing0.09Receiver Operating Characteristics (ROCs) describe the rate of true positives against false positives over threshold values. As the area of ROC curve can assess the performance of uncalibrated decision functions, even without the prior distribution of the classes, the Area under Curve (AUC) has been widely adopted for model evaluation. A 6-fold approach is adopted during the training process, and the AUC value is adopted to evaluate the modeling performance. In this research, the testing performance with respect to loss function and AUC is shown in Figure9, and the results demonstrate the effectiveness of the developed deep learning model.Figure 9
The testing performance with respect to (a) loss function value and (b) AUC value.
(a)(b)Moreover, it should be noted that the Test Time Augmentation (TTA) approach can also be applied to improve the performance of the developed model. The reason of that is TTA will randomly modify the test data several times, the final output will be the average prediction of each augmented image, and the overall performance can be improved by reducing single abnormal prediction error. In this research, the influence of TTA on AUC is shown in Figure10.Figure 10
The influence of TTA on AUC.


#### 3.2.2. Development of XGBoost Model

As most of the patient-related contextual information are unstructured data, it is necessary to encode the extracted features into a standardized format before training, which will remove redundancies from original dataset and boost the training process. In this research, the extracted features are encoded into different integers according to Table1, and Figure 11 shows the examples of some encoded features.Figure 11
Examples of extracted features after encoding.After features encoding, the input dataset is quantified into discrete integers within a specified range, and then the properties of the input data and the correlations between different features can be investigated statistically. As shown in Figure12, the quantified probability distribution demonstrates the statistical correlations between the encoded features and melanoma.Figure 12
Probability of melanoma with respect to the encoded features.Moreover, the covariance matrix as shown in Table3 demonstrates the statistical relationship between different encoded features. As some of the input features are highly correlated according to the large covariance in Table 3, there is a need to further reduce input dimensions before model training. Therefore, feature selection is adopted in this stage, and the results are shown in Figure 13. The ranking results demonstrate that image size and mean color are highly correlated with melanoma while sex has less correlation with melanoma.Table 3
Covariance matrix of encoded features.
FeaturesDRSexAPAgeNIPISMCMIAMAADR1.0000.0280.0060.0860.0760.0780.0980.0830.095Sex0.0281.0000.0100.1070.0370.0250.0780.0870.119AP0.0060.0101.0000.0370.0380.0000.0880.0270.043Age0.0860.1070.0371.0000.1440.1080.0730.9570.960NIP0.0760.0370.0380.1441.0000.3380.2860.1250.162IS0.0780.0250.0000.1080.3381.0000.4840.2400.023MC0.0980.0780.0880.0730.2860.4841.0000.0540.145MIA0.0830.0870.0270.9570.1250.2400.0541.0000.910MAA0.0950.1190.0430.9600.1620.0230.1450.9101.000Figure 13
The ranking results of extracted features based on feature selection.Based on the ranking results, the five more important features (five high score features) are selected as inputs for model training, and a 5-fold approach is used to balance bias dataset. Moreover, to avoid local optima, the grid search approach is adopted to identify optimal parameters during the training process, and the achieved AUC value of the developed XGBoost model is 0.855, which demonstrates the effectiveness of the developed model.After training, the developed deep learning model and XGBoost model are combined as a hybrid model for melanoma diagnosing, and the weights of different models are set as 0.8 and 0.2, respectively. Then, a confusion matrix which consists of specificity, accuracy, and sensitivity is adopted to fully evaluate the performance of the proposed model; the three specifications are defined as follows:(1)specificity=TNTN+FP,accuracy=TP+TNTP+TN+FP+FN,sensitivity=TPTP+FN,where True Positives (TP) define the correctly classified positive cases, True Negative (TN) define the correctly classified negative cases, False Positives (FP) are incorrectly classified negative cases, and False Negative (FN) are incorrectly classified positive cases. Because the developed deep learning model is in conjunction with an external feature-based XGBoost model, the proposed hybrid model can improve the performance of melanoma classification compared with base models, which demonstrates effectiveness of the proposed approach and will be beneficial to clinical diagnose for public health. Table 4 shows the comparison of testing performance between different models.Table 4
Comparison of testing performance between different models.
Melanoma classificationDeep learning modelXGBoost modelHybrid modelSpecificity (%)918796Accuracy (%)928595Sensitivity (%)918993


## 3.1. Experimental Materials

### 3.1.1. Image Datasets

The datasets photographed from a consecutive sample of lesions were originated from International Skin Imaging Collaboration (ISIC) between 2018 and 2019. These images were acquired through dermatoscope and digital cameras at different resolutions in polarizing or nonpolarizing mode. The cases with missing or equivocal histopathologic reports or with extremely low image quality were excluded in the dataset. Following that, the remaining images were reviewed manually and labelled as melanoma or benign skin lesion for classification, and Figure6 demonstrates some examples of melanoma images for classification.Figure 6
Samples of melanoma images for classification.


### 3.1.2. Patient-Related Contextual Information

Currently, deep learning and image processing methods have achieved expert-level performance in melanoma detection. However, as the existing image-based artificial intelligence classification algorithms mainly focus on extracting features from images, the diagnosis process is insufficient and may induce the discrepancy between prior challenges and clinical practice. In practice, the patient-related contextual information are normally considered as external attributes, which will be beneficial to clinicians for differentiating melanoma from benign skin lesions. Therefore, there is a need to extract meaningful features from patient-related contextual information, which could better support dermatological clinic work and improve the diagnostic accuracy. In this research, the patient-related contextual information which are extracted and encoded as external features are shown in Table1. Then, based on the encoded features, a supervised machine learning model will be developed to predict melanoma.Table 1
Features derived from patient-related contextual information.
Extracted featuresRange of encoded features (integer)Anatomic positions (APs)[0, 6]Diagnosis results (DRs)[0, 1]Sex[0, 1]Number of images per patient (NIP)[0, 9]Image size (IS)[0, 10]Mean color (MC)[0, 9]Age[0, 9]Minimum age (MIA)[0, 17]Maximum age (MAA)[0, 17]


## 3.1.1. Image Datasets

The datasets photographed from a consecutive sample of lesions were originated from International Skin Imaging Collaboration (ISIC) between 2018 and 2019. These images were acquired through dermatoscope and digital cameras at different resolutions in polarizing or nonpolarizing mode. The cases with missing or equivocal histopathologic reports or with extremely low image quality were excluded in the dataset. Following that, the remaining images were reviewed manually and labelled as melanoma or benign skin lesion for classification, and Figure6 demonstrates some examples of melanoma images for classification.Figure 6
Samples of melanoma images for classification.


## 3.1.2. Patient-Related Contextual Information

Currently, deep learning and image processing methods have achieved expert-level performance in melanoma detection. However, as the existing image-based artificial intelligence classification algorithms mainly focus on extracting features from images, the diagnosis process is insufficient and may induce the discrepancy between prior challenges and clinical practice. In practice, the patient-related contextual information are normally considered as external attributes, which will be beneficial to clinicians for differentiating melanoma from benign skin lesions. Therefore, there is a need to extract meaningful features from patient-related contextual information, which could better support dermatological clinic work and improve the diagnostic accuracy. In this research, the patient-related contextual information which are extracted and encoded as external features are shown in Table1. Then, based on the encoded features, a supervised machine learning model will be developed to predict melanoma.Table 1
Features derived from patient-related contextual information.
Extracted featuresRange of encoded features (integer)Anatomic positions (APs)[0, 6]Diagnosis results (DRs)[0, 1]Sex[0, 1]Number of images per patient (NIP)[0, 9]Image size (IS)[0, 10]Mean color (MC)[0, 9]Age[0, 9]Minimum age (MIA)[0, 17]Maximum age (MAA)[0, 17]


## 3.2. Results Analysis

### 3.2.1. Development of Deep Learning Model

For deep learning, data augmentation is usually preferred to increase the size of training dataset virtually, which will make the developed model more robust to the input dataset. In this research, some classical operations such as gridmask and image rotation are adopted for data augmentation, and Figure7 illustrates an example of input image after data augmentation.Figure 7
An example of data augmentation for input image: (a) original image; (b) gridmask; (c) rotation.
(a)(b)(c)After data augmentation, TensorFlow is adopted for the development of deep learning models. TensorFlow is an open-source framework which is developed by Google for machine learning and deep learning functions. As TensorFlow provides users the flexibility of focusing on the structure of models rather than mathematical details, it has been widely adopted in various areas, from academia and research to industrial fields. During the training process, different configurations of TensorFlow structure will affect the modeling performance. For example, Figure8 demonstrates the influence of epoch on learning rate. It is necessary to balance the modeling efficiency and accuracy by adjusting the model configurations. The initial configurations of TensorFlow adopted in this research for deep learning implementation are illustrated in Table 2.Figure 8
The influence of epoch on learning rate during model training.
(a)(b)Table 2
Initial configurations of TensorFlow for deep learning implementation.
ParametersSettingsNumber of folders6Number of fully connected layers4Image size384 pixelsBalance ratio0.08Batch size64Epochs25Type of EfficientNetEfficientNet_6Loss typeBinary CrossEntropyLabel smoothing0.09Receiver Operating Characteristics (ROCs) describe the rate of true positives against false positives over threshold values. As the area of ROC curve can assess the performance of uncalibrated decision functions, even without the prior distribution of the classes, the Area under Curve (AUC) has been widely adopted for model evaluation. A 6-fold approach is adopted during the training process, and the AUC value is adopted to evaluate the modeling performance. In this research, the testing performance with respect to loss function and AUC is shown in Figure9, and the results demonstrate the effectiveness of the developed deep learning model.Figure 9
The testing performance with respect to (a) loss function value and (b) AUC value.
(a)(b)Moreover, it should be noted that the Test Time Augmentation (TTA) approach can also be applied to improve the performance of the developed model. The reason of that is TTA will randomly modify the test data several times, the final output will be the average prediction of each augmented image, and the overall performance can be improved by reducing single abnormal prediction error. In this research, the influence of TTA on AUC is shown in Figure10.Figure 10
The influence of TTA on AUC.


### 3.2.2. Development of XGBoost Model

As most of the patient-related contextual information are unstructured data, it is necessary to encode the extracted features into a standardized format before training, which will remove redundancies from original dataset and boost the training process. In this research, the extracted features are encoded into different integers according to Table1, and Figure 11 shows the examples of some encoded features.Figure 11
Examples of extracted features after encoding.After features encoding, the input dataset is quantified into discrete integers within a specified range, and then the properties of the input data and the correlations between different features can be investigated statistically. As shown in Figure12, the quantified probability distribution demonstrates the statistical correlations between the encoded features and melanoma.Figure 12
Probability of melanoma with respect to the encoded features.Moreover, the covariance matrix as shown in Table3 demonstrates the statistical relationship between different encoded features. As some of the input features are highly correlated according to the large covariance in Table 3, there is a need to further reduce input dimensions before model training. Therefore, feature selection is adopted in this stage, and the results are shown in Figure 13. The ranking results demonstrate that image size and mean color are highly correlated with melanoma while sex has less correlation with melanoma.Table 3
Covariance matrix of encoded features.
FeaturesDRSexAPAgeNIPISMCMIAMAADR1.0000.0280.0060.0860.0760.0780.0980.0830.095Sex0.0281.0000.0100.1070.0370.0250.0780.0870.119AP0.0060.0101.0000.0370.0380.0000.0880.0270.043Age0.0860.1070.0371.0000.1440.1080.0730.9570.960NIP0.0760.0370.0380.1441.0000.3380.2860.1250.162IS0.0780.0250.0000.1080.3381.0000.4840.2400.023MC0.0980.0780.0880.0730.2860.4841.0000.0540.145MIA0.0830.0870.0270.9570.1250.2400.0541.0000.910MAA0.0950.1190.0430.9600.1620.0230.1450.9101.000Figure 13
The ranking results of extracted features based on feature selection.Based on the ranking results, the five more important features (five high score features) are selected as inputs for model training, and a 5-fold approach is used to balance bias dataset. Moreover, to avoid local optima, the grid search approach is adopted to identify optimal parameters during the training process, and the achieved AUC value of the developed XGBoost model is 0.855, which demonstrates the effectiveness of the developed model.After training, the developed deep learning model and XGBoost model are combined as a hybrid model for melanoma diagnosing, and the weights of different models are set as 0.8 and 0.2, respectively. Then, a confusion matrix which consists of specificity, accuracy, and sensitivity is adopted to fully evaluate the performance of the proposed model; the three specifications are defined as follows:(1)specificity=TNTN+FP,accuracy=TP+TNTP+TN+FP+FN,sensitivity=TPTP+FN,where True Positives (TP) define the correctly classified positive cases, True Negative (TN) define the correctly classified negative cases, False Positives (FP) are incorrectly classified negative cases, and False Negative (FN) are incorrectly classified positive cases. Because the developed deep learning model is in conjunction with an external feature-based XGBoost model, the proposed hybrid model can improve the performance of melanoma classification compared with base models, which demonstrates effectiveness of the proposed approach and will be beneficial to clinical diagnose for public health. Table 4 shows the comparison of testing performance between different models.Table 4
Comparison of testing performance between different models.
Melanoma classificationDeep learning modelXGBoost modelHybrid modelSpecificity (%)918796Accuracy (%)928595Sensitivity (%)918993


## 3.2.1. Development of Deep Learning Model

For deep learning, data augmentation is usually preferred to increase the size of training dataset virtually, which will make the developed model more robust to the input dataset. In this research, some classical operations such as gridmask and image rotation are adopted for data augmentation, and Figure7 illustrates an example of input image after data augmentation.Figure 7
An example of data augmentation for input image: (a) original image; (b) gridmask; (c) rotation.
(a)(b)(c)After data augmentation, TensorFlow is adopted for the development of deep learning models. TensorFlow is an open-source framework which is developed by Google for machine learning and deep learning functions. As TensorFlow provides users the flexibility of focusing on the structure of models rather than mathematical details, it has been widely adopted in various areas, from academia and research to industrial fields. During the training process, different configurations of TensorFlow structure will affect the modeling performance. For example, Figure8 demonstrates the influence of epoch on learning rate. It is necessary to balance the modeling efficiency and accuracy by adjusting the model configurations. The initial configurations of TensorFlow adopted in this research for deep learning implementation are illustrated in Table 2.Figure 8
The influence of epoch on learning rate during model training.
(a)(b)Table 2
Initial configurations of TensorFlow for deep learning implementation.
ParametersSettingsNumber of folders6Number of fully connected layers4Image size384 pixelsBalance ratio0.08Batch size64Epochs25Type of EfficientNetEfficientNet_6Loss typeBinary CrossEntropyLabel smoothing0.09Receiver Operating Characteristics (ROCs) describe the rate of true positives against false positives over threshold values. As the area of ROC curve can assess the performance of uncalibrated decision functions, even without the prior distribution of the classes, the Area under Curve (AUC) has been widely adopted for model evaluation. A 6-fold approach is adopted during the training process, and the AUC value is adopted to evaluate the modeling performance. In this research, the testing performance with respect to loss function and AUC is shown in Figure9, and the results demonstrate the effectiveness of the developed deep learning model.Figure 9
The testing performance with respect to (a) loss function value and (b) AUC value.
(a)(b)Moreover, it should be noted that the Test Time Augmentation (TTA) approach can also be applied to improve the performance of the developed model. The reason of that is TTA will randomly modify the test data several times, the final output will be the average prediction of each augmented image, and the overall performance can be improved by reducing single abnormal prediction error. In this research, the influence of TTA on AUC is shown in Figure10.Figure 10
The influence of TTA on AUC.


## 3.2.2. Development of XGBoost Model

As most of the patient-related contextual information are unstructured data, it is necessary to encode the extracted features into a standardized format before training, which will remove redundancies from original dataset and boost the training process. In this research, the extracted features are encoded into different integers according to Table1, and Figure 11 shows the examples of some encoded features.Figure 11
Examples of extracted features after encoding.After features encoding, the input dataset is quantified into discrete integers within a specified range, and then the properties of the input data and the correlations between different features can be investigated statistically. As shown in Figure12, the quantified probability distribution demonstrates the statistical correlations between the encoded features and melanoma.Figure 12
Probability of melanoma with respect to the encoded features.Moreover, the covariance matrix as shown in Table3 demonstrates the statistical relationship between different encoded features. As some of the input features are highly correlated according to the large covariance in Table 3, there is a need to further reduce input dimensions before model training. Therefore, feature selection is adopted in this stage, and the results are shown in Figure 13. The ranking results demonstrate that image size and mean color are highly correlated with melanoma while sex has less correlation with melanoma.Table 3
Covariance matrix of encoded features.
FeaturesDRSexAPAgeNIPISMCMIAMAADR1.0000.0280.0060.0860.0760.0780.0980.0830.095Sex0.0281.0000.0100.1070.0370.0250.0780.0870.119AP0.0060.0101.0000.0370.0380.0000.0880.0270.043Age0.0860.1070.0371.0000.1440.1080.0730.9570.960NIP0.0760.0370.0380.1441.0000.3380.2860.1250.162IS0.0780.0250.0000.1080.3381.0000.4840.2400.023MC0.0980.0780.0880.0730.2860.4841.0000.0540.145MIA0.0830.0870.0270.9570.1250.2400.0541.0000.910MAA0.0950.1190.0430.9600.1620.0230.1450.9101.000Figure 13
The ranking results of extracted features based on feature selection.Based on the ranking results, the five more important features (five high score features) are selected as inputs for model training, and a 5-fold approach is used to balance bias dataset. Moreover, to avoid local optima, the grid search approach is adopted to identify optimal parameters during the training process, and the achieved AUC value of the developed XGBoost model is 0.855, which demonstrates the effectiveness of the developed model.After training, the developed deep learning model and XGBoost model are combined as a hybrid model for melanoma diagnosing, and the weights of different models are set as 0.8 and 0.2, respectively. Then, a confusion matrix which consists of specificity, accuracy, and sensitivity is adopted to fully evaluate the performance of the proposed model; the three specifications are defined as follows:(1)specificity=TNTN+FP,accuracy=TP+TNTP+TN+FP+FN,sensitivity=TPTP+FN,where True Positives (TP) define the correctly classified positive cases, True Negative (TN) define the correctly classified negative cases, False Positives (FP) are incorrectly classified negative cases, and False Negative (FN) are incorrectly classified positive cases. Because the developed deep learning model is in conjunction with an external feature-based XGBoost model, the proposed hybrid model can improve the performance of melanoma classification compared with base models, which demonstrates effectiveness of the proposed approach and will be beneficial to clinical diagnose for public health. Table 4 shows the comparison of testing performance between different models.Table 4
Comparison of testing performance between different models.
Melanoma classificationDeep learning modelXGBoost modelHybrid modelSpecificity (%)918796Accuracy (%)928595Sensitivity (%)918993


## 4. Discussion

In this section, the basic theories of the adopted approaches were first described, and then the effectiveness of the proposed hybrid machine learning model for melanoma detection was analyzed by case study. The case study demonstrated that the proposed hybrid approach can improve the classification accuracy of melanoma compared with two base models. However, as the weights of the hybrid model were determined by user preference, which is subjective and may be suboptimal, it is necessary to optimize the weights of the hybrid model to further improve overall performance for melanoma detection.


## 5. Conclusions

In this paper, a novel hybrid machine learning approach is proposed to identify melanoma for skin healthcare in various cases. The proposed approach consists of classic machine learning methods, including CNNs, EfficientNet, and XGBoost supervised machine learning. In the proposed approach, a deep learning model is trained directly from raw pixels and image labels for classification of skin lesions. Then, solely based on modeling of various features from patients, an XGBoost model is adopted to predict skin cancer. Following that, a diagnostic system which is composed of the deep learning model and XGBoost model is developed to further improve the prediction efficiency and accuracy.Different from experience-based methods and solely image-based machine learning methods, the proposed approach is developed based on the theory of deep learning and feature engineering. Therefore, the modeling process is in high efficiency and accuracy. Moreover, the data-driven-based characteristics can help the proposed approach develop a guideline for image analysis in other medical applications. In the future, it is necessary to investigate and optimize the weights of the proposed hybrid model, which will be helpful to further improve the model effectiveness and performance for melanoma detection.



---

*Source: 8700506-2021-11-17.xml*