# A Novel Region-Extreme Convolutional Neural Network for Melanoma Malignancy Recognition

Authors: Nudrat Nida; Aun Irtaza; Muhammad Haroon Yousaf
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Melanoma malignancy recognition is a challenging task due to the existence of intraclass similarity, natural or clinical artefacts, skin contrast variation, and higher visual similarity among the normal or melanoma-affected skin. To overcome these problems, we propose a novel solution by leveraging “region-extreme convolutional neural network” for melanoma malignancy recognition as malignant or benign. Recent works on melanoma malignancy recognition employed the traditional machine learning techniques based on various handcrafted features or the recently introduced CNN network. However, the efficient training of these models is possible, if they localize the melanoma affected region and learn high-level feature representation from melanoma lesion to predict melanoma malignancy. In this paper, we incorporate this observation and propose a novel “region-extreme convolutional neural network” for melanoma malignancy recognition. Our proposed region-extreme convolutional neural network refines dermoscopy images to eliminate natural or clinical artefacts, localizes melanoma affected region, and defines precise boundary around the melanoma lesion. The defined melanoma lesion is used to generate deep feature maps for model learning using the extreme learning machine (ELM) classifier. The proposed model is evaluated on two challenge datasets (ISIC-2016 and ISIC-2017) and performs better than ISIC challenge winners. Our region-extreme convolutional neural network recognizes the melanoma malignancy 85% on ISIC-2016 and 93% on ISIC-2017 datasets. Our region-extreme convolutional neural network precisely segments the melanoma lesion with an average Jaccard index of 0.93 and Dice score of 0.94. The region-extreme convolutional neural network has several advantages: it eliminates the clinical and natural artefacts from dermoscopic images, precisely localizes and segments the melanoma lesion, and improves the melanoma malignancy recognition through feedforward model learning. The region-extreme convolutional neural network achieves significant performance improvement over existing methods that makes it adaptable for solving complex medical image analysis problems.


---

## Body

## 1. Introduction

Melanoma is a dangerous form of skin cancer and it is difficult to identify it at an earlier stage due to visual similarity with normal skin. The morality rate of melanoma is more than 75% every year [1]. In the year 2020, 76,380 new cases of melanoma are suspected to be diagnosed and around 10,130 deaths are reported in the USA [2]. Fortunately, melanoma is curable if it is diagnosed and medicated at an earlier stage. Therefore, earlier diagnosis is desirable to improve the patient’s survival rate [3–5]. Dermoscopy images are used for the diagnosis of melanoma and it is reliable than a visual melanoma inspection procedure through the naked eye. However, the manual melanoma diagnosis through the naked eye is error-prone and time-consuming, and different dermatologists’ diagnosis predictions often vary that adversely affect the earlier treatment of disease [6]. Therefore, the automatic diagnosis of melanoma facilitates the dermatologists and supports them to validate their predictions even at earlier stages.Melanoma recognition is a challenging domain due to visual similarity among the types of melanoma such as color, size, texture, location, and shape. The boundaries are also not obscure, and variation makes the task of recognition more complicated at an earlier stage. Moreover, some artefacts such as hair, blood vessels, gel bubble, and clinical ruler marks also degrade the recognition performance [1]. To support the earlier diagnosis and to solve this challenging problem, computer vision-based computer-aided diagnosis (CAD) tools are capable of detecting and recognizing melanoma to assist the dermatologists [7]. Recently, much effort has been invested in the recognition of melanoma automatically [8–14]. Earlier studies employ the low-level handcrafted features to classify normal and melanoma skin regions, including texture [15, 16], shape [17], and color [17, 18] attributes. Some traditional investigations suggest combining handcrafted features along with selection of features to improve the recognition of melanoma [19, 20]. However, the handcrafted features hold low-level representation of melanoma which is not effective to inculcate higher visual similarity among normal and melanoma lesions. Moreover, the artefacts within the dermoscopic images lead to a higher misclassification rate. The researchers also used the segmentation of melanoma region and extracted features from the segmented region to recognize melanoma [16, 19–21]. The feature representation from segmented melanoma region results in better recognition due to a well-defined area of interest. Still, these techniques employ low-level handcrafted feature representation for segmentation and classification with limited recognition capabilities.Recently, the deep learning techniques have revolutionized the biomedical image analysis domain, including segmentation [22, 23], detection [24–27], and classification [28–30]. The multilayer high-dimensional CNN feature representations capture discriminative attributes for the accurate recognition of melanoma. Codella et al. [31] suggested deep features representation to classify melanoma using support vector machine (SVM). Kawahara et al. [32] presented the benefit of transfer learning of pretrained model including AlexNet [31] and tuned the model for melanoma representation. Yu et al. employed the deep residual network to classify benign and malignant melanoma with a deeper model of CNN [33]. Zhen et al. introduced the features extraction from pretrained ResNet50 and local descriptor fisher vector encoding scheme to train the SVM model for melanoma malignancy recognition [34]. Melanoma malignancy recognition from deep learning techniques indicated good performance as compared to handcrafted features due to high-level feature representation of melanoma.This research work aims to propose a novel approach for melanoma malignancy recognition and to overcome all the challenges present in the dermoscopic images (Figure1). The proposed “region-extreme convolutional neural network” differs from the existing melanoma recognition methods in the following ways. First, while existing CNN approaches for melanoma recognition make deterministic diagnosis decision based on features extracted from the entire dermoscopy image, the proposed “region-extreme convolutional neural network” improves the recognition performance by introducing the localization block that allows precise melanoma segmentation and removes artefacts. Then, decision boundaries are established based on these segmented lesions to predict the melanoma malignancy. The malignancy model generated from melanoma lesion plays an important role in accurate prediction as it improves the melanoma malignancy recognition performance through considering only melanoma affected lesion unlike entire dermoscopy image. Therefore, the model obtained from the proposed method improves the recognition performance as compared to existing methods. The proposed approach is evaluated on the ISIC-2016 and ISIC-2017 challenge datasets and experimental finding portrays good recognition performance.Figure 1
(a–e) Sample of ISIC-2016, and ISIC-2017 dataset illustrating the clinical artefacts and natural artefacts. These artefacts consist of (a) color and illumination variation, (b) variation in contrast, (c) black frame and clinical rule marks, (d) natural hair, and (e) clinical color swatches.
(a)(b)(c)(d)(e)Our contributions in this paper are summarized as follows:(1)
We propose region-extreme convolutional neural network for localization and recognition of melanoma malignancy at an earlier stage(2)
Our experimental finding reveals that deep features extracted from the penultimate layers are more powerful than the features extracted from higher-order layers of CNN because the receptive fields of penultimate layers hold higher semantics and contextual information of melanoma malignancy(3)
Rigorous experimentation against the state-of-the-art methods is performed to establish the effectiveness of the proposed approach


## 2. The Proposed Region-Extreme Convolutional Neural Network for Melanoma Malignancy Recognition

Dermoscopic images used for the recognition of melanoma are occluded with clinical and natural artefacts. Moreover, a huge variation and intraclass visual similarity also exist among the melanoma and non-melanoma regions that adversely degrade the recognition performance [1]. To overcome these challenges, we propose a novel scheme to mitigate these challenges and recognize the melanoma malignancy with higher recognition rate, as shown in Figure 2. In the first step, we refine the images and localize the melanoma affected region. Then, the localized region is used to segment the melanoma region. The localization and segmentation block use the training pair of images including RGB dermoscopic image and corresponding ground truth mask images to generate melanoma localization model during training phase. The location coordinates of melanoma are obtained through binarization of the ground truth image. After binarization, we apply the region split and merged segmentation on binary ground truth image to acquire the location information x,y,w,h of melanoma by extracting the largest connected component. These coordinates’ information from ground truth image IG is used to map the location of melanoma in the RGB dermoscopy image I. These coordinates x,y,w,h along with RGB dermoscopic images are presented to RCNN to generate a model for the melanoma localization. The RCNN employs selective search to produce region proposal, and deep convolutional features are extracted from each proposal. The softmax layer classifies the test sample proposal into the melanoma affected or normal skin region, while the classification softmax layer recognizes the chosen proposals as melanoma-affected region, and the suspected proposals are passed through the regression layer of RCNN [35]. The regression layer computes the intersection over union score IoU using greedy suppression algorithm to detect the part of skin as melanoma or normal region. In our case, if the region’s regression score is higher than 0.5, then the proposal is localized as melanoma lesion to reduce the false positive localization. Therefore, only true positive proposals with regression IoU scores greater than 0.5 are considered as melanoma area; otherwise, it is considered as normal skin area. In the third step, the segmented melanoma regions are used to extract deep features for the recognition purpose. The architecture of proposed region-extreme convolutional neural network is shown in Figure 2 and elaborated in detail in the following sections.Figure 2
The architecture diagram of the proposed region-extreme convolutional neural network for malignancy recognition melanoma.


### 2.1. Mathematical Formulation of Melanoma Recognition

Given an input dermoscopy imageDi from the i training samples, the skin refinement block of region-extreme convolutional neural network enhances the perceptual quality of dermoscopic image and processed image I is processed further for the melanoma localization.Training of the melanoma localization block requires predicted melanoma regionIP and corresponding ground truth location of melanoma region IG. IG is presented as pairs IPi,IGi where i=1,2,3,…,N are the training samples and N represents the total number of training samples. Here, IG and IP are the rectangular regions as x,y,w,h with pixel location x,y, weight as w, and height as h. To localize the melanoma lesion precisely, optimal transformation functions are established by the region-extreme convolutional neural network to reduce the localization error and exactly map the predicted bounding box IP location x,y, width w, and height h of melanoma region over ground truth bounding box IG. The four transformation functions txIP, tyIP, twIP, and thIP are mathematically represented as(1)IGx=IPwtxIP+IPx.Here,IGx is the centroid x pixel location of actual melanoma lesion, IPx, IPw are horizontal and width-wise predicted melanoma lesions, and tw is a logarithmic representation of targeted horizontal coordinate of melanoma lesion.(2)IGy=IPhtyIP+IPy,where IGy is the centroid y pixel location of actual melanoma lesion, IPy and IPh are vertical and height-wise predicted melanoma lesions, and ty is a logarithmic representation of targeted vertical coordinate of melanoma lesion.(3)IGw=IPwetwIP.Similarly,IGw is the width of the actual melanoma lesion, IPw and IP represent the width-wise predicted melanoma lesion and overall predicted melanoma lesion, and tw is a logarithmic representation of targeted width of melanoma lesion.(4)IGh=IPhethIP.IGh represents height, IPh and IP represent the height-wise predicted melanoma lesion and overall predicted melanoma lesion, and th is a logarithmic representation of targeted height of melanoma lesion.(5)t∗Ip=ω∗Tϕ3Ip.Here,t∗IpɛtxIP,tyIP,twIP,thIP is the target location of melanoma lesion to map predicted melanoma lesion IP on actual melanoma lesion IG. txIP, tyIP, twIP, and thIP are transformation functions to linearly predict the region proposal IP using deep features of pooling layer 3 ϕ3 where ϕ is 2D compressed receptive field. For precise prediction of melanoma lesion, ω∗ are the weights adapted during training phase through gradient decent algorithm and t∗ is respective mapping function for location x,y, width w, and height h, where ∗ɛx,y,w,h and ω∗ is 1D weight matrix. txIP and tyIP specify mapping of IP centroid keeping the characteristic of scale in-variance. twIP and thIP are a logarithmic representation of width and height of IP.The regression targetsR∗ɛRx,Ry,Rw,Rh for accurate localization of melanoma region are listed numerically as follows:(6)Rx=IGx−IPxIPw.Here,Rx is the regression target function at horizontal axis. IGx is horizontal location of actual melanoma lesion. IPx, IPw represent the horizontal location and width-wise predicted melanoma lesions.(7)Ry=IGy−IPyIPh,Ry is the regression target function at vertical axis. IGy is vertical location of actual melanoma lesion. IPy, IPh represent the vertical location and height-wise predicted melanoma lesions.(8)Rw=logIGxIPw,Rw is the regression target function for width mapping. IGx is horizontal location of actual melanoma lesion. IPw represents the width-wise predicted melanoma lesion.(9)Rh=logIGyIPh,Rh is the regression target function for height mapping. IGy is vertical location of actual melanoma lesion. IPh represents the height-wise predicted melanoma lesion. For simplicity, we represent the regression target functions Rx, Ry, Rw, Rh as Rx,y,w,h.Region-extreme convolutional neural network performs melanoma lesion localization through greedy overlapping criteria of ground truth bounding boxes and predicted boxes, known as intersection-over-union (IoU). The information extracted from the proposalIP improves learning, if IP is close to one of the ground truth boxes IG [35]. The IoU threshold chosen for melanoma detection is greater than 0.5 and score lower than 0.5 is rejected by the localization block to improve localization. The acceptable range of melanoma lesion localization is opted within the IoU of 0.5–1, where 0.3–0 is chosen for background normal skin region.The melanoma localization block defines a bounding box across the melanoma lesion and then crops the infected region. The cropped region is segmented into melanoma defined region and normal skin region. The melanoma defined lesion is passed to deep feature extraction block for discriminative feature maps generation and melanoma malignancy recognition.The problem to recognize benign and malignant melanoma from training dermoscopy melanoma lesionRx,y,w,h can be written as forecasting class probability problem:(10)ℜEzj,Rx,y,w,h=Pclass|Rx,y,w,h,where ℜE is the predicted class recognized by the proposed region-extreme convolutional neural network and Pclass|Rx,y,w,h represents the probabilities of benign b and malignant m melanoma class, where classɛb,m, b stands for benign and m stands for malignant melanoma class. The test sample zj is fed to region-extreme convolutional neural network ℜE to predict the class probability of benign b and malignant m melanoma. Region-extreme convolutional neural network ℜE recognizes the benign and malignant melanoma classes as shown in (10).


### 2.2. Skin Refinement Block

Dermoscopy imaging techniques are used for diagnosis of melanoma because these images examine the skin lesion at deeper level and ensure maximum perceptual ability for melanoma diagnosis [3]. Although dermoscopic images enhance the visual clarity, still there is a need to improve the manual prediction through advanced deep learning techniques.The artefacts within the dermoscopy imagesD are responsible for degrading the melanoma segmentation performance. Therefore, the hair, clinical rule marks, and veins are removed through morphological closing operation applied twice across the image using equation (11). The two-line structuring elements S1 of 10 pixels in 90∘ and 180∘ are used to eliminate the hair and clinical rule marks in horizontal and vertical direction. We have selected these structuring elements to remove the line shape objects including the hair and clinical rule marks from the RGB dermoscopy images, as these artefacts degrade the segmentation quality of melanoma.(11)J=D⊕S1⊖S1.In equation (11), J represents the enhanced dermoscopic image through morphological closing operation, where D is input dermoscopic image and S1 is line structuring element. Then, the resultant image J is processed further to improve the image edges and entropy through unsharp filtering operation.(12)Js=J×funsharp.The unsharp filterfunsharp is convolved with the image J to smooth the image and generate a blur image Js.(13)funsharp=−iπσ41−a2+b22σ2ea2+b2/2σ2.Here,funsharp represents the unsharp kernel to transform dermoscopic image J into enhanced image Js. a and b represent the distance from centroid pixel in horizontal and vertical direction, and σ represents the standard deviation of image pixel’s probability distribution.(14)I=J−Jsx,y.Then, the smooth imageJs is subtracted from original image J, to obtain the sharped image I as shown in equation (14). After refining the dermoscopy images, I are forwarded to localization and segmentation block for localization of melanoma lesion.


### 2.3. Localization and Segmentation Block

Melanoma localization is a significant step in the proposed region-extreme convolutional neural network as the entire dermoscopic image consists of normal skin and clinical or natural artefacts along with melanoma lesion. The architecture of CNN network is used as backbone model in learning of region-extreme convolutional neural network model which is shown in Table1.Table 1
The region-extreme convolutional neural network architecture used for localization of melanoma.
LayersFunctionalityFiltersSizeStridePaddingInput layerPreprocesses the input dermoscopic image32 × 32 × 3Convolutional layerConvolves input data with kernels35 × 50Max poolingPerforms maximum pooling operation on receptive fields3 × 32RELUApplies RELU activation function on receptive fieldsConvolutional layerConvolves receptive fields with kernels325 × 512RELUApplies RELU activation function on receptive fieldsAverage poolingPerforms average pooling on receptive fields3 × 320Convolutional layerConvolves receptive fields with kernels645 × 512RELUApplies RELU activation function on receptive fieldsAvg poolingPerforms average pooling on receptive fields3 × 320Fully connectedFlattens the receptive fields from 2D to 1D64RELUApplies RELU activation function on receptive fieldsFully connectedTransforms high level melanoma representation to nonlinear transfer function2SoftmaxAssigns class probabilities to region proposalsRegressionPerforms non-maximum suppression and estimates the location of melanomaRecent deep learning approaches have achieved good results in automatic malignancy recognition of melanoma but still there is a need to improve the recognition performance as difference exists between the expert prediction and the prediction obtained from automated method due to challenges mentioned above. Therefore, proposing the discriminative melanoma malignancy representation still demands more analysis with limited training data available. Recent findings employ very large deep networks for the malignancy analysis and prediction. However, it is very difficult to explore very deep networks and identify optimal solution as compared to shallower networks. Moreover, the problem of vanishing gradient becomes more crucial during training of very deep convolutional neural network and thus, it is more challenging to identify reasonable hyperparameters of large networks. Like other biomedical image analysis tasks, the availability of skin lesion training data is limited which makes the malignancy recognition problem even more challenging. We exploit the significance of melanoma lesion segmentation and investigate the benefits of generating learning model based on melanoma affected skin pixels only, despite the entire dermoscopy image. In this way, inference model becomes more decisive and performs accurate melanoma malignancy predictions.In the proposed scheme, region based convolutional neural network (RCNN) is used for melanoma affected lesion localization that transforms the melanoma detection task as regression problem to predict the possible location of melanoma lesion [1]. We have selected RCNN because RCNN is able to learn the melanoma representation from the deep features and localize accurate region of melanoma [1]. Therefore, RCNN is applied to the image for localization of melanoma region. During training, region-extreme convolutional neural network considers the melanoma lesion and the rest of the region is considered as background, including the clinical artefacts, e.g., black frame, clinical swatches, clinical scale marks, and healthy skin.We opt for localization of melanoma lesionRx,y,w,h to reduce the search space for precise segmentation of melanoma affected region that is free from normal skin and artefacts. RCNN localizes a rectangular region Rx,y,w,h around the melanoma infected lesion. For more preciseness, melanoma lesion Rx,y,w,h is further processed to remove normal skin pixels and define precise boundaries around the melanoma affected region. Therefore, fuzzy c means (FCM) clustering algorithm is applied to segment out melanoma region. The FCM clusters the image pixels and the FCM algorithm requires a predefined number of clusters as an input parameter. In our approach, as localized region consists of melanoma region and normal skin pixels, therefore FCM divides the image into two groups resulting in definite melanoma boundary representation. The FCM clustering refines the segmentation clusters caɛ1,2 through evolving the objective function Od under defined initial conditions.(15)Ssa=1∑k=124Rx,y,w,h−ca2/d−1/Rx,y,w,h−ck.The variabled describes the total number of pixels existing within the localized rectangular melanoma region Rx,y,w,h and Ssa describes the degree of membership of pixels with cluster a, having cluster center ca.(16)ca=∑s=1NSsaRx,y,w,h∑s=1NSsad.In order to define precise pixel’s membership ofRx,y,w,h with cluster center ca, the local minima are estimated through computing the objective function Od:(17)Od=∑S=1d∑a=12SsadRx,y,w,h−ca2.When the centroids of FCM converge, the segmentation ofRx,y,w,h generates melanoma affected region with defined boundaries which is separated from the normal skin region.


### 2.4. Deep Feature Extraction Block

After segmenting the melanoma affected lesionRx,y,w,h, deep feature representation FN of benign and malignant melanoma is computed through CNN model. In our algorithm, we have used AlexNet [36], VGG16 [37], VGG19 [37], ResNet18 [38], ResNet101 [38], GoogleNet [39], and DenseNet201 [40], and the best performing model is chosen from these CNN models to capture a deep visual representation of benign and malignant melanoma. We have explored the recognition capabilities of the deep features at different network depths and the impact of suitable CNN network for melanoma malignancy prediction from the dermoscopic images. In the proposed technique, deep features are extracted from either pooling or fully connected layers and softmax layer is eliminated. The deep feature representation FN of benign and malignant melanoma is normalized through min-max normalization algorithm and then passed to PCA to optimize the features equations (18) and (19). The PCA reduces the variance among the features of similar class and improves the learning process.(18)FN′ν=τν.The mathematical expression (equation (18)) represents the principal component symmetric matrix FN′, which is also known as co-variance matrix, where τ represents the diagonal matrix and ν represents the orthogonal projections matrix which is used for PCA computation.(19)FN′=ν11FN…ν1nFN⋮⋱⋮νnnFN…νnnFN.The idea of using PCA on top of the deep features is to identify significant pattern that is helpful in recognition and improving the learning performance. Recently, PCA was applied in hierarchical layers for feature extraction that validates the performance improvement hypothesis [41]. PCA features were used to train autoencoder, and significant performance improvement was reported [42]. In our work, we have extracted the deep features from CNN network, normalized them, and then used PCA to optimize the deep features without reducing the dimension. Despite [41, 42], we have used PCA after deep features extraction from CNN to get discriminative features. We have observed that PCA significantly improves the recognition of benign and malignant melanoma using the deep convolutional features. The PCA detects the correlation ν among CNN features of the same class and reduces the correlated features within the different class which is significant to discriminate the benign and malignant melanoma. The experiments and results are discussed in Section 3.4.2.Deep featuresFN′ are normalized within minimum feature FNmin′ and maximum feature FNmax′ range to optimize the decision boundaries of recognition model and reduce the search space with benefits of lower computational time. The extracted deep principle component FN′ strengthens the decision boundaries of model and supports in precise prediction of melanoma benign and malignant class.


### 2.5. Feed-Forward Model Learning Block

The extreme learning machine (ELM) performs fast learning using single-layer feed-forward neural network with defined number of hidden nodes. Our previous work indicated that ELM holds better generalization performance as compared to back-propagation algorithms [43]. The weights and biases of ELM’s first layer are randomly assigned and remain constant, while the weights and biases of hidden layer are opted to minimize the least square error between actual and predicted class, thus establishing stronger decision boundaries [44, 45].The feed-forward model learning block constitutes ELM classifier to learn model for benign and malignant melanoma recognition using deep principle components featuresFN′. ELM consists of training parameters and these parameters are responsible for generating decision boundaries among benign and malignant melanoma. The selection of number of training parameters is a significant process and reduces the classification loss. A single output unit is used to predict benign and malignant melanoma through ELM output function yLFN′ described as(20)yLFN′=∑j=1LωihiFN′,where ωi=ω1,ω2,ω3,…,ωL describes the output weights and hiFN′=h1FN′,h2FN′,…,hLFN′ describes the weight existing between output vector and L hidden nodes. The classification decision function of ELM is(21)yLFN′=signhiωi.The objective function of ELM optimizes the decision boundaries to generalize the prediction performance with minimum classification loss. The mathematical representation of objective function is shown in the following expression:(22)f°=minimizeHωi−T2,ωi.Here,H† represents the matrix of hidden layer’s output as(23)H=h1FN1′…hLFN1′⋮⋱⋮h1FNN′…hLFNN′.The ELM decision boundaries are established through maximizing the margins among the benign and malignant melanoma within feature representation2/ωi and minimal least square method is used to minimize the norm of output weights ωi.(24)ωi=H†T.The parameterH† is Moore–Penrose inverse matrix which is computed from orthogonal projection along with single value decomposition method. At test time, the ELM performs the prediction of class labels using decision function and predicts the melanoma malignancy through estimating the highest value of output node. The detailed explanation about ELM can be found in [44].


## 2.1. Mathematical Formulation of Melanoma Recognition

Given an input dermoscopy imageDi from the i training samples, the skin refinement block of region-extreme convolutional neural network enhances the perceptual quality of dermoscopic image and processed image I is processed further for the melanoma localization.Training of the melanoma localization block requires predicted melanoma regionIP and corresponding ground truth location of melanoma region IG. IG is presented as pairs IPi,IGi where i=1,2,3,…,N are the training samples and N represents the total number of training samples. Here, IG and IP are the rectangular regions as x,y,w,h with pixel location x,y, weight as w, and height as h. To localize the melanoma lesion precisely, optimal transformation functions are established by the region-extreme convolutional neural network to reduce the localization error and exactly map the predicted bounding box IP location x,y, width w, and height h of melanoma region over ground truth bounding box IG. The four transformation functions txIP, tyIP, twIP, and thIP are mathematically represented as(1)IGx=IPwtxIP+IPx.Here,IGx is the centroid x pixel location of actual melanoma lesion, IPx, IPw are horizontal and width-wise predicted melanoma lesions, and tw is a logarithmic representation of targeted horizontal coordinate of melanoma lesion.(2)IGy=IPhtyIP+IPy,where IGy is the centroid y pixel location of actual melanoma lesion, IPy and IPh are vertical and height-wise predicted melanoma lesions, and ty is a logarithmic representation of targeted vertical coordinate of melanoma lesion.(3)IGw=IPwetwIP.Similarly,IGw is the width of the actual melanoma lesion, IPw and IP represent the width-wise predicted melanoma lesion and overall predicted melanoma lesion, and tw is a logarithmic representation of targeted width of melanoma lesion.(4)IGh=IPhethIP.IGh represents height, IPh and IP represent the height-wise predicted melanoma lesion and overall predicted melanoma lesion, and th is a logarithmic representation of targeted height of melanoma lesion.(5)t∗Ip=ω∗Tϕ3Ip.Here,t∗IpɛtxIP,tyIP,twIP,thIP is the target location of melanoma lesion to map predicted melanoma lesion IP on actual melanoma lesion IG. txIP, tyIP, twIP, and thIP are transformation functions to linearly predict the region proposal IP using deep features of pooling layer 3 ϕ3 where ϕ is 2D compressed receptive field. For precise prediction of melanoma lesion, ω∗ are the weights adapted during training phase through gradient decent algorithm and t∗ is respective mapping function for location x,y, width w, and height h, where ∗ɛx,y,w,h and ω∗ is 1D weight matrix. txIP and tyIP specify mapping of IP centroid keeping the characteristic of scale in-variance. twIP and thIP are a logarithmic representation of width and height of IP.The regression targetsR∗ɛRx,Ry,Rw,Rh for accurate localization of melanoma region are listed numerically as follows:(6)Rx=IGx−IPxIPw.Here,Rx is the regression target function at horizontal axis. IGx is horizontal location of actual melanoma lesion. IPx, IPw represent the horizontal location and width-wise predicted melanoma lesions.(7)Ry=IGy−IPyIPh,Ry is the regression target function at vertical axis. IGy is vertical location of actual melanoma lesion. IPy, IPh represent the vertical location and height-wise predicted melanoma lesions.(8)Rw=logIGxIPw,Rw is the regression target function for width mapping. IGx is horizontal location of actual melanoma lesion. IPw represents the width-wise predicted melanoma lesion.(9)Rh=logIGyIPh,Rh is the regression target function for height mapping. IGy is vertical location of actual melanoma lesion. IPh represents the height-wise predicted melanoma lesion. For simplicity, we represent the regression target functions Rx, Ry, Rw, Rh as Rx,y,w,h.Region-extreme convolutional neural network performs melanoma lesion localization through greedy overlapping criteria of ground truth bounding boxes and predicted boxes, known as intersection-over-union (IoU). The information extracted from the proposalIP improves learning, if IP is close to one of the ground truth boxes IG [35]. The IoU threshold chosen for melanoma detection is greater than 0.5 and score lower than 0.5 is rejected by the localization block to improve localization. The acceptable range of melanoma lesion localization is opted within the IoU of 0.5–1, where 0.3–0 is chosen for background normal skin region.The melanoma localization block defines a bounding box across the melanoma lesion and then crops the infected region. The cropped region is segmented into melanoma defined region and normal skin region. The melanoma defined lesion is passed to deep feature extraction block for discriminative feature maps generation and melanoma malignancy recognition.The problem to recognize benign and malignant melanoma from training dermoscopy melanoma lesionRx,y,w,h can be written as forecasting class probability problem:(10)ℜEzj,Rx,y,w,h=Pclass|Rx,y,w,h,where ℜE is the predicted class recognized by the proposed region-extreme convolutional neural network and Pclass|Rx,y,w,h represents the probabilities of benign b and malignant m melanoma class, where classɛb,m, b stands for benign and m stands for malignant melanoma class. The test sample zj is fed to region-extreme convolutional neural network ℜE to predict the class probability of benign b and malignant m melanoma. Region-extreme convolutional neural network ℜE recognizes the benign and malignant melanoma classes as shown in (10).


## 2.2. Skin Refinement Block

Dermoscopy imaging techniques are used for diagnosis of melanoma because these images examine the skin lesion at deeper level and ensure maximum perceptual ability for melanoma diagnosis [3]. Although dermoscopic images enhance the visual clarity, still there is a need to improve the manual prediction through advanced deep learning techniques.The artefacts within the dermoscopy imagesD are responsible for degrading the melanoma segmentation performance. Therefore, the hair, clinical rule marks, and veins are removed through morphological closing operation applied twice across the image using equation (11). The two-line structuring elements S1 of 10 pixels in 90∘ and 180∘ are used to eliminate the hair and clinical rule marks in horizontal and vertical direction. We have selected these structuring elements to remove the line shape objects including the hair and clinical rule marks from the RGB dermoscopy images, as these artefacts degrade the segmentation quality of melanoma.(11)J=D⊕S1⊖S1.In equation (11), J represents the enhanced dermoscopic image through morphological closing operation, where D is input dermoscopic image and S1 is line structuring element. Then, the resultant image J is processed further to improve the image edges and entropy through unsharp filtering operation.(12)Js=J×funsharp.The unsharp filterfunsharp is convolved with the image J to smooth the image and generate a blur image Js.(13)funsharp=−iπσ41−a2+b22σ2ea2+b2/2σ2.Here,funsharp represents the unsharp kernel to transform dermoscopic image J into enhanced image Js. a and b represent the distance from centroid pixel in horizontal and vertical direction, and σ represents the standard deviation of image pixel’s probability distribution.(14)I=J−Jsx,y.Then, the smooth imageJs is subtracted from original image J, to obtain the sharped image I as shown in equation (14). After refining the dermoscopy images, I are forwarded to localization and segmentation block for localization of melanoma lesion.


## 2.3. Localization and Segmentation Block

Melanoma localization is a significant step in the proposed region-extreme convolutional neural network as the entire dermoscopic image consists of normal skin and clinical or natural artefacts along with melanoma lesion. The architecture of CNN network is used as backbone model in learning of region-extreme convolutional neural network model which is shown in Table1.Table 1
The region-extreme convolutional neural network architecture used for localization of melanoma.
LayersFunctionalityFiltersSizeStridePaddingInput layerPreprocesses the input dermoscopic image32 × 32 × 3Convolutional layerConvolves input data with kernels35 × 50Max poolingPerforms maximum pooling operation on receptive fields3 × 32RELUApplies RELU activation function on receptive fieldsConvolutional layerConvolves receptive fields with kernels325 × 512RELUApplies RELU activation function on receptive fieldsAverage poolingPerforms average pooling on receptive fields3 × 320Convolutional layerConvolves receptive fields with kernels645 × 512RELUApplies RELU activation function on receptive fieldsAvg poolingPerforms average pooling on receptive fields3 × 320Fully connectedFlattens the receptive fields from 2D to 1D64RELUApplies RELU activation function on receptive fieldsFully connectedTransforms high level melanoma representation to nonlinear transfer function2SoftmaxAssigns class probabilities to region proposalsRegressionPerforms non-maximum suppression and estimates the location of melanomaRecent deep learning approaches have achieved good results in automatic malignancy recognition of melanoma but still there is a need to improve the recognition performance as difference exists between the expert prediction and the prediction obtained from automated method due to challenges mentioned above. Therefore, proposing the discriminative melanoma malignancy representation still demands more analysis with limited training data available. Recent findings employ very large deep networks for the malignancy analysis and prediction. However, it is very difficult to explore very deep networks and identify optimal solution as compared to shallower networks. Moreover, the problem of vanishing gradient becomes more crucial during training of very deep convolutional neural network and thus, it is more challenging to identify reasonable hyperparameters of large networks. Like other biomedical image analysis tasks, the availability of skin lesion training data is limited which makes the malignancy recognition problem even more challenging. We exploit the significance of melanoma lesion segmentation and investigate the benefits of generating learning model based on melanoma affected skin pixels only, despite the entire dermoscopy image. In this way, inference model becomes more decisive and performs accurate melanoma malignancy predictions.In the proposed scheme, region based convolutional neural network (RCNN) is used for melanoma affected lesion localization that transforms the melanoma detection task as regression problem to predict the possible location of melanoma lesion [1]. We have selected RCNN because RCNN is able to learn the melanoma representation from the deep features and localize accurate region of melanoma [1]. Therefore, RCNN is applied to the image for localization of melanoma region. During training, region-extreme convolutional neural network considers the melanoma lesion and the rest of the region is considered as background, including the clinical artefacts, e.g., black frame, clinical swatches, clinical scale marks, and healthy skin.We opt for localization of melanoma lesionRx,y,w,h to reduce the search space for precise segmentation of melanoma affected region that is free from normal skin and artefacts. RCNN localizes a rectangular region Rx,y,w,h around the melanoma infected lesion. For more preciseness, melanoma lesion Rx,y,w,h is further processed to remove normal skin pixels and define precise boundaries around the melanoma affected region. Therefore, fuzzy c means (FCM) clustering algorithm is applied to segment out melanoma region. The FCM clusters the image pixels and the FCM algorithm requires a predefined number of clusters as an input parameter. In our approach, as localized region consists of melanoma region and normal skin pixels, therefore FCM divides the image into two groups resulting in definite melanoma boundary representation. The FCM clustering refines the segmentation clusters caɛ1,2 through evolving the objective function Od under defined initial conditions.(15)Ssa=1∑k=124Rx,y,w,h−ca2/d−1/Rx,y,w,h−ck.The variabled describes the total number of pixels existing within the localized rectangular melanoma region Rx,y,w,h and Ssa describes the degree of membership of pixels with cluster a, having cluster center ca.(16)ca=∑s=1NSsaRx,y,w,h∑s=1NSsad.In order to define precise pixel’s membership ofRx,y,w,h with cluster center ca, the local minima are estimated through computing the objective function Od:(17)Od=∑S=1d∑a=12SsadRx,y,w,h−ca2.When the centroids of FCM converge, the segmentation ofRx,y,w,h generates melanoma affected region with defined boundaries which is separated from the normal skin region.


## 2.4. Deep Feature Extraction Block

After segmenting the melanoma affected lesionRx,y,w,h, deep feature representation FN of benign and malignant melanoma is computed through CNN model. In our algorithm, we have used AlexNet [36], VGG16 [37], VGG19 [37], ResNet18 [38], ResNet101 [38], GoogleNet [39], and DenseNet201 [40], and the best performing model is chosen from these CNN models to capture a deep visual representation of benign and malignant melanoma. We have explored the recognition capabilities of the deep features at different network depths and the impact of suitable CNN network for melanoma malignancy prediction from the dermoscopic images. In the proposed technique, deep features are extracted from either pooling or fully connected layers and softmax layer is eliminated. The deep feature representation FN of benign and malignant melanoma is normalized through min-max normalization algorithm and then passed to PCA to optimize the features equations (18) and (19). The PCA reduces the variance among the features of similar class and improves the learning process.(18)FN′ν=τν.The mathematical expression (equation (18)) represents the principal component symmetric matrix FN′, which is also known as co-variance matrix, where τ represents the diagonal matrix and ν represents the orthogonal projections matrix which is used for PCA computation.(19)FN′=ν11FN…ν1nFN⋮⋱⋮νnnFN…νnnFN.The idea of using PCA on top of the deep features is to identify significant pattern that is helpful in recognition and improving the learning performance. Recently, PCA was applied in hierarchical layers for feature extraction that validates the performance improvement hypothesis [41]. PCA features were used to train autoencoder, and significant performance improvement was reported [42]. In our work, we have extracted the deep features from CNN network, normalized them, and then used PCA to optimize the deep features without reducing the dimension. Despite [41, 42], we have used PCA after deep features extraction from CNN to get discriminative features. We have observed that PCA significantly improves the recognition of benign and malignant melanoma using the deep convolutional features. The PCA detects the correlation ν among CNN features of the same class and reduces the correlated features within the different class which is significant to discriminate the benign and malignant melanoma. The experiments and results are discussed in Section 3.4.2.Deep featuresFN′ are normalized within minimum feature FNmin′ and maximum feature FNmax′ range to optimize the decision boundaries of recognition model and reduce the search space with benefits of lower computational time. The extracted deep principle component FN′ strengthens the decision boundaries of model and supports in precise prediction of melanoma benign and malignant class.


## 2.5. Feed-Forward Model Learning Block

The extreme learning machine (ELM) performs fast learning using single-layer feed-forward neural network with defined number of hidden nodes. Our previous work indicated that ELM holds better generalization performance as compared to back-propagation algorithms [43]. The weights and biases of ELM’s first layer are randomly assigned and remain constant, while the weights and biases of hidden layer are opted to minimize the least square error between actual and predicted class, thus establishing stronger decision boundaries [44, 45].The feed-forward model learning block constitutes ELM classifier to learn model for benign and malignant melanoma recognition using deep principle components featuresFN′. ELM consists of training parameters and these parameters are responsible for generating decision boundaries among benign and malignant melanoma. The selection of number of training parameters is a significant process and reduces the classification loss. A single output unit is used to predict benign and malignant melanoma through ELM output function yLFN′ described as(20)yLFN′=∑j=1LωihiFN′,where ωi=ω1,ω2,ω3,…,ωL describes the output weights and hiFN′=h1FN′,h2FN′,…,hLFN′ describes the weight existing between output vector and L hidden nodes. The classification decision function of ELM is(21)yLFN′=signhiωi.The objective function of ELM optimizes the decision boundaries to generalize the prediction performance with minimum classification loss. The mathematical representation of objective function is shown in the following expression:(22)f°=minimizeHωi−T2,ωi.Here,H† represents the matrix of hidden layer’s output as(23)H=h1FN1′…hLFN1′⋮⋱⋮h1FNN′…hLFNN′.The ELM decision boundaries are established through maximizing the margins among the benign and malignant melanoma within feature representation2/ωi and minimal least square method is used to minimize the norm of output weights ωi.(24)ωi=H†T.The parameterH† is Moore–Penrose inverse matrix which is computed from orthogonal projection along with single value decomposition method. At test time, the ELM performs the prediction of class labels using decision function and predicts the melanoma malignancy through estimating the highest value of output node. The detailed explanation about ELM can be found in [44].


## 3. Experimental Results

### 3.1. Datasets, Hardware Specification, and Evaluation Parameters

The proposed technique is evaluated using ISIC-2016 and ISIC-2017 challenge dataset [46]. The ISIC-2016 consists of 900 training and 350 testing samples. The ISIC-2016 dataset constitutes training data from 727 non-melanoma patients and 173 melanoma patients, while test sample contains data of 304 melanoma patients and 75 melanoma patients. The ISIC-2017 dataset consists of 2000 training samples, 150 validation, and 600 test samples of dermoscopy images. The training set constitutes data of 374 melanoma patients, 254 seborrheic keratosis patients, and 1372 benign nevi patients samples, while the test set contains data of 117 melanoma patients, 97 seborrheic patients, and 393 benign nevi patients.The experiments are conducted on system with 2.59 GHz Core i7 CPU, 12 GB RAM, and NVIDIA GeForce GTX-950 Graphic Processing Unit.Our proposed technique constitutes skin refinement step, melanoma localization, segmentation, and recognition of melanoma benign and malignant class. Each part of the proposed technique is evaluated with different evaluation measures. As the skin refinement phase is an enhancement process which was applied to improve the visual representation of image, therefore we have used peak signal to noise ratioPSNR, root mean square error RMSE, and universal image quality index UIQI for the evaluation of skin refinement step.For the evaluation of melanoma localization, mean average precisionmAP is used to estimate the precise localization. The intersection over union IoU measure is chosen within the range of 0.5–1 for melanoma detection. Jaccard index Ja, dice score Di, sensitivity SE, specificity SP, and accuracy Ac are used for evaluation of segmentation step. At ISIC-2016 and ISIC-2017 challenges, the contestants were ranked based on Ja index for the segmentation task. In the case of recognition step, four parameters are used to evaluate the classification task including, accuracy AC, average precision AP, sensitivity SE, specificity SP, and F1−score, respectively. Ac, SP, and SE are a similar metric as in segmentation. However, in the case of classification, they are recorded at the image level. In classification problems, F1-score is interpreted as a better choice to examine the performance of classifier using weighted average of sensitivity and specificity information. In the dataset, the testing set constitutes imbalance samples around 75 melanoma and 304 non-melanoma samples. Therefore, the false positive rate is smaller and true negative rate is larger comparatively. The organizers of the challenge employ average precision AP to rank the classification algorithms of participants [46].


### 3.2. Performance Evaluation of Skin Refinement

In the skin refinement step, the noisy artefacts are removed and visual information of dermoscopic images is refined to enhance the perceptual quality of the images. The skin refinement step is applied to all the dermoscopy images of ISIC-2016 and ISIC-2017 datasets. The resultant output samples are represented in Figure3, and it can be observed that morphological operations and noise removal step resulted in dermoscopic images free from the clinical artefacts, e.g., rule marks and natural artefacts like the hair and blood vessels.Figure 3
The resultant output sample of skin refinement phase. (a) The clinical and natural artefacts including thick and thin hair, black frame, clinical rule marks, gel bubbles, and color swatches. (b) The resultant intermediate output of morphological operation. (c) The resultant output of morphological closing operation and smooth operation. (d) The sharp image obtained after unsharped filter’s convolution.
(a)(b)(c)(d)To further examine the quality of resultant image obtained from the skin refinement step, the image quality was estimated usingRMSE, PSNR, and UIQI, and results are represented in Table 2. The PSNR measure is higher than 37.8 dB that signifies the higher ratio of information within the resultant refined image and minimum RMSE error. Moreover, the UIQI observation portrays the lower information loss in terms of contrast, illumination, and structural information on application of skin refinement step. As UIQI value is within the range of 0.4–0.7, and this measure represents the good perceptual quality of the preprocessed image. Therefore, these evaluation measures indicate that skin refinement step removed the natural and clinical artefacts without degrading the perceptual information of dermoscopic images.Table 2
The performance of skin refinement step is estimated through computingPSNR, UIQI, and RMSE, considering each type of artefact.
DatasetISIC-2016ISIC-2017ArtefactsPSNRUIQIRMSEPSNRUIQIRMSEBlack frame33.000.619.9837.000.819.0128.520.4116.5638.520.777.8634.000.609.2432.400.688.4230.300.5413.5033.530.748.50Thick hair30.6706012.9234.7407010.8231.750.5011.4134.950.6809.4130.020.5013.9336.220.7808.1335.700.517.2538.700.674.98Thin hair34.500.438.3138.900.639.2132.570.6410.3933.670.709.7833.700.589.1537.970.788.8537.780.565.6544.780.873.85Rule marks18.900.4050.1228.980.5645.2229.710.5714.4332.340.6723.9832.260.5510.7734.660.5418.3732.350.5010.6536.790.6714.85Swatches35.310.557.5733.220.557.5732.350.5310.6634.750.659.7633.050.679.8238.150.7710.2430.250.7013.7237.670.8712.62


### 3.3. Segmentation Phase

The next step after refining the dermoscopic images is localization and segmentation of melanoma lesion and the resulting segmented lesion is considered as region of interest (ROI) for the learning of melanoma malignancy model. In this experiment, localization and segmentation performance of region-extreme convolutional neural network is evaluated. For generation of localization model, we trained the localization block using training images of datasets and evaluated the performance of trained localization model using test samples.The internal representation of the feature map to discriminate the melanoma region through localization and segmentation block is shown in Figure4. The CNN softmax layer was responsible for predicting the probabilities at pixel level for categorizing each pixel into normal or melanoma affected pixel. The localization block used these labeled pixels and applied selective search to select region proposal and computed the intersection over union (IoU) threshold for estimation of melanoma lesion. We have opted 0.5 IoU threshold for the detection of melanoma lesion; the region is considered as normal skin region below this value as shown in Figure 5. The RCNN localized the melanoma lesion with 0.94 mAP for ISIC-2016 and 0.95 mAP for ISIC-2017 dataset, as illustrated in Table 3.Figure 4
The deep feature representation of melanoma region and normal skin region. These deep features are used to establish the model for melanoma malignancy recognition. The green overlay represents the normal skin attributes and purple overlay represents the melanoma affected lesion through localization and segmentation block.Figure 5
The resultant ISIC samples obtained after melanoma localization step.Table 3
The achieved mean average precision (mAP) and processing time in milliseconds (ms) for melanoma localization through the region-extreme convolutional neural network using ISIC-2016 and ISIC-2017 dataset.
ISIC-2016 artefactsmAPTime (ms)ISIC-2017 artefactsmAPTime (ms)Thick hair0.9740.1Thick hair0.9540.2Think hair0.9642.1Think hair0.9843.4Rule marks0.9043.3Rule marks0.9254.6Color charts0.9139.4Color charts0.8540.8Black frame0.9641.6Black frame0.8852.4For the comparison with other methods, the melanoma localized regions are further processed to obtain fine boundaries of melanoma region. The FCM clustered the pixels into melanoma lesion and normal skin. The resulting performance portrayed the good melanoma segmentation results as illustrated in Figure6. The average values of SP as 0.9417, SE as 0.9782, Ac as 0.948, Di as 0.94, and Ja as 0.93 were recorded.Figure 6
(a) Input images with artefacts. (b) Ground truth melanoma lesion images. (c) Localization of melanoma lesion. (d) Blue overlay of fuzzy segmentation of melanoma boundary and red overlay of ground truth melanoma lesion.
(a)(b)(c)(d)Our results are better than the state-of-the-art methods due to precise localization of melanoma lesion through the proposed method. In the few cases, localization block was unable to detect the melanoma region due to visual similarity with normal skin regions, as shown in Figure7 [1].Figure 7
The localization and segmentation block fails to identify the melanoma benign cases, due to visual similarity with normal skin attributes.


### 3.4. Performance Evaluation of Recognition Task

To build the recognition model, we trained the recognition block using training images of ISIC-2016 and ISIC-2017 datasets and evaluated the performance of trained model using test samples. The sigmoid activation function of ELM was used for feed-forward model learning with 150000 number of hidden nodes. The backbone CNN models opted for to extract deep discriminative representation of melanoma were AlexNet [36], VGG16 [37], VGG19 [37], ResNet50 [38], ResNet101 [38], GoogleNet [39], and DenseNet201 [40], and their performance was observed based on segmented melanoma regions as listed in Table 4.Table 4
Comparison of different network layers on the classification of benign and malignant melanoma.
NetworkLayerFeaturesISIC-2016ISIC-2017ACAPACAPAlexNetpool592160.840.810.780.77Fc640960.850.830.930.91Fc740960.840.800.880.87Fc810000.840.770.870.86VGG16Fc640960.820.730.860.88FC740960.820.740.820.81FC810000.820.730.820.81VGG19Fc640960.840.830.880.87Fc740960.810.720.850.84Fc810000.810.700.800.80GoogleNetPred10000.810.560.850.84DenseNet201Fc100010000.820.770.870.86ResNet50Fc100010000.860.830.920.90ResNet101Fc100010000.810.680.860.88Bold values indicate the best results.


#### 3.4.1. Impact of the Features Extracted from Different Network Depths

In this experiment, we explored the recognition capability of region-extreme convolutional neural network using different deep features extracted from the various CNN networks from segmented melanoma lesion. The best network was chosen as a feature extractor of the region-extreme convolutional neural network’s recognition stage. The deep features extracted from segmented melanoma lesion were used to feed the ELM classifier for model learning. The generated ELM model was able to discriminate benign and malignant melanoma. The performance of proposed recognition model is reported in Table4 which illustrates the optimal CNN model and layer for the deep features extraction.The receptive fields generated from the deep features at different layers of convolutional neural networks represent contextual and semantic variations of dermoscopy images. Therefore, receptive fields at different level of convolutional network perform differently. On inspecting the behavior of melanoma malignancy model at different CNN layer, a common pattern is observed among all kinds of CNN models. The receptive field obtained from initial layers of CNN models was not decisive and is inferior as lower layers of CNN models collect low-level description and these patterns lack in-variance attribute. A similar characteristic was observed from the CNN middle layers, while the receptive fields obtained from the penultimate layers of CNN networks perform better than other layers because the penultimate layers receptive fields hold high level semantics and contextual information of melanoma malignancy.It is notable that AlexNet fully connected layerFc6 achieved the highest average precision, because Fc6 holds the higher dimensional (4096) deep features to represent the benign and malignant melanoma for ISIC-2016 and ISIC-2017 dataset, whereas the AlexNet pool5 layer captured 9216 features but pool5 feature’s recognition capability is lower than fully connected layer Fc6, because higher-dimensional features of pool5 become repeatable in both benign and malignant classes. This experiment supported us to identify the optimal backbone CNN model with penultimate layer to extract the deep features for generation of melanoma malignancy model. The performance of deep features extracted from the ResNet50 was comparable with AlexNet for ISIC-2016. However, the performance of deep features extracted from the ResNet50 was observed to be lower than AlexNet for ISIC-2017. The observed prediction accuracy and precision using AlexNet and ResNet50 features for ISIC-2016 are 0.86 and 0.83, while AlexNet performed better in prediction accuracy (0.93) and precision (0.91) to recognize benign and malignant melanoma.Another observation that can be derived from Table4 is that the penultimate layers of shallower sequential CNN model (e.g., AlexNet) performed better than the other deeper sequential and deeper directed acyclic CNN models, e.g., Resent50. Therefore, we have utilized AlexNet as the backbone model for features extraction and fed these extracted deep features from AlexNet to ELM for feed-forward model generation. Here, it is important to highlight the essence of feed-forward ELM model that learned the model using deep features without back-propagation and weight optimization like CNN networks. This attribute improves the learning and reduces the computational time as compared to the state-of-the-art CNN models.This experiment concluded that sequential shallower CNN network (e.g. AlexNet) penultimate layers deep features established the stronger ELM decision boundaries to predict the melanoma malignancy. On the basis of this experiment, we opted for AlexNet as feature extraction model for recognition block and examined the rest of our experiments using AlexNet as feature extractor for recognition block.


#### 3.4.2. Impact of PCA on Classification Task

To further improve the recognition model, we examined the dimensionality reduction of receptive fields obtained from Alexnet fc6 layer and applied PCA on it. As PCA reduces the covariant features and discovers compact, there is meaningful representation of deep features to recognize intraclass melanoma test samples. It can be noticed in Table5 that the deep features generated a good model to discriminate benign and malignant melanoma, when represented by PCA. Therefore, using PCA at the top of deep features is beneficial for classification task.Table 5
Results of classification with and without PCA.
DatasetPMWith PCAWithout PCAAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.830.820.840.85AP0.830.740.830.830.780.730.790.82ISIC-2017AC0.930.860.880.910.920.850.860.89AP0.910.850.860.890.890.840.850.88


#### 3.4.3. Classification with and without Melanoma Segmentation Phase

To evaluate the performance of our region-extreme convolutional network, we evaluated the performance of classification with and without the segmentation of melanoma region. All the parameters and network architectures remained the same in both experiments. All the recorded experimental results are listed in Table6. The segmented melanoma ROIs produced a better model in discriminating benign and malignant melanoma as compared to complete dermoscopic images because segmented lesion contains only benign and malignant feature representation despite having feature representation from normal skin or the artefacts. The AP of segmented image was relatively 5% higher than non-segmented dermoscopic images. Defining the melanoma area as a region of interest and the extraction of deep features to generate model is a smart way to train ELM, and to classify precisely without being distracted from the clinical artefacts or healthy skin region. We conclude from this experiment that the segmented melanoma lesion when fed to recognition block for extraction of receptive fields is more decisive in building accurate melanoma malignancy recognition model.Table 6
Performance of classification task with and without segmentation.
DatasetPMSegmentedWithout segmentationAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.840.810.830.84AP0.830.740.830.830.770.700.730.80ISIC-2017AC0.930.860.880.920.910.840.850.88AP0.910.850.860.900.890.830.830.85


#### 3.4.4. Impact of Classification Model in Melanoma Subclass Recognition

The performance of recognition block is further investigated through the impact of ELM as a classifier against traditional classifiers including support vector machine (SVM) and K-nearest neighbor (KNN). To observe the impact, the entire recognition block remained the same while the classifier layer was changed with ELM, support vector machine (SVM), and K-nearest neighbor (KNN). The region-extreme convolutional neural network segmented the melanoma lesion and extracted the deep features from backbone models such as Alexnet, VGG16, and VGG19 and trained different types of classifiers including ELM, SVM, and KNN as a separate model. It is evident from Table7 that the ELM classifier outperformed all the classifiers, where K-nearest neighbor performed better that SVM classifier.Table 7
Comparison of different classifiers for recognition of benign and malignant melanoma class.
DatasetPMELMKNNSVMAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.800.770.780.780.750.760.750.73AP0.830.740.830.830.580.560.580.570.620.630.600.59ISIC-2017AC0.930.860.880.880.830.830.840.800.860.800.810.85AP0.910.880.870.860.810.810.810.800.840.800.800.83The ELM classifier trains the learning model in a two-step process; first it randomizes the deep feature mapping and then it identifies the solution for linear parameters. In the first step, the ELM hidden layer randomly initializes to map the deep features on ELM feature space using nonlinear piece-wise function. The second step of ELM separates it from traditional classifiers including SVM which utilizes the kernel function for feature projection, while the deep learning models require autoencoder or restricted Boltzmann machines to map input features and form the similar deep representation.Essentially, the ELM generates hidden nodes randomly without explicitly being trained that makes ELM learning generalized and invariant from input training data. The ELM’s hidden nodes are assigned using continuous probability distribution. Therefore, ELM models are stronger than traditional classifiers (SVM, KNN) or back-propagation neural networks. Moreover, the regularization function within the ELM also supports it to avoid the problems of overfitting and underfitting without any computational overhead.We found that the ELM enhances the performance of the system to predict intraclass label of melanoma, with 83% average precision for ISIC-2016 and 91% for ISIC-2017 dataset. The performance of different classifiers is listed in Table7. All the performance metrics indicate the effectiveness of ELM within the recognition block of region-extreme convolutional neural network. Therefore, this experiment concludes that ELM is the optimal choice to learn the feed-forward melanoma malignancy model using dermoscopy images.


## 3.1. Datasets, Hardware Specification, and Evaluation Parameters

The proposed technique is evaluated using ISIC-2016 and ISIC-2017 challenge dataset [46]. The ISIC-2016 consists of 900 training and 350 testing samples. The ISIC-2016 dataset constitutes training data from 727 non-melanoma patients and 173 melanoma patients, while test sample contains data of 304 melanoma patients and 75 melanoma patients. The ISIC-2017 dataset consists of 2000 training samples, 150 validation, and 600 test samples of dermoscopy images. The training set constitutes data of 374 melanoma patients, 254 seborrheic keratosis patients, and 1372 benign nevi patients samples, while the test set contains data of 117 melanoma patients, 97 seborrheic patients, and 393 benign nevi patients.The experiments are conducted on system with 2.59 GHz Core i7 CPU, 12 GB RAM, and NVIDIA GeForce GTX-950 Graphic Processing Unit.Our proposed technique constitutes skin refinement step, melanoma localization, segmentation, and recognition of melanoma benign and malignant class. Each part of the proposed technique is evaluated with different evaluation measures. As the skin refinement phase is an enhancement process which was applied to improve the visual representation of image, therefore we have used peak signal to noise ratioPSNR, root mean square error RMSE, and universal image quality index UIQI for the evaluation of skin refinement step.For the evaluation of melanoma localization, mean average precisionmAP is used to estimate the precise localization. The intersection over union IoU measure is chosen within the range of 0.5–1 for melanoma detection. Jaccard index Ja, dice score Di, sensitivity SE, specificity SP, and accuracy Ac are used for evaluation of segmentation step. At ISIC-2016 and ISIC-2017 challenges, the contestants were ranked based on Ja index for the segmentation task. In the case of recognition step, four parameters are used to evaluate the classification task including, accuracy AC, average precision AP, sensitivity SE, specificity SP, and F1−score, respectively. Ac, SP, and SE are a similar metric as in segmentation. However, in the case of classification, they are recorded at the image level. In classification problems, F1-score is interpreted as a better choice to examine the performance of classifier using weighted average of sensitivity and specificity information. In the dataset, the testing set constitutes imbalance samples around 75 melanoma and 304 non-melanoma samples. Therefore, the false positive rate is smaller and true negative rate is larger comparatively. The organizers of the challenge employ average precision AP to rank the classification algorithms of participants [46].


## 3.2. Performance Evaluation of Skin Refinement

In the skin refinement step, the noisy artefacts are removed and visual information of dermoscopic images is refined to enhance the perceptual quality of the images. The skin refinement step is applied to all the dermoscopy images of ISIC-2016 and ISIC-2017 datasets. The resultant output samples are represented in Figure3, and it can be observed that morphological operations and noise removal step resulted in dermoscopic images free from the clinical artefacts, e.g., rule marks and natural artefacts like the hair and blood vessels.Figure 3
The resultant output sample of skin refinement phase. (a) The clinical and natural artefacts including thick and thin hair, black frame, clinical rule marks, gel bubbles, and color swatches. (b) The resultant intermediate output of morphological operation. (c) The resultant output of morphological closing operation and smooth operation. (d) The sharp image obtained after unsharped filter’s convolution.
(a)(b)(c)(d)To further examine the quality of resultant image obtained from the skin refinement step, the image quality was estimated usingRMSE, PSNR, and UIQI, and results are represented in Table 2. The PSNR measure is higher than 37.8 dB that signifies the higher ratio of information within the resultant refined image and minimum RMSE error. Moreover, the UIQI observation portrays the lower information loss in terms of contrast, illumination, and structural information on application of skin refinement step. As UIQI value is within the range of 0.4–0.7, and this measure represents the good perceptual quality of the preprocessed image. Therefore, these evaluation measures indicate that skin refinement step removed the natural and clinical artefacts without degrading the perceptual information of dermoscopic images.Table 2
The performance of skin refinement step is estimated through computingPSNR, UIQI, and RMSE, considering each type of artefact.
DatasetISIC-2016ISIC-2017ArtefactsPSNRUIQIRMSEPSNRUIQIRMSEBlack frame33.000.619.9837.000.819.0128.520.4116.5638.520.777.8634.000.609.2432.400.688.4230.300.5413.5033.530.748.50Thick hair30.6706012.9234.7407010.8231.750.5011.4134.950.6809.4130.020.5013.9336.220.7808.1335.700.517.2538.700.674.98Thin hair34.500.438.3138.900.639.2132.570.6410.3933.670.709.7833.700.589.1537.970.788.8537.780.565.6544.780.873.85Rule marks18.900.4050.1228.980.5645.2229.710.5714.4332.340.6723.9832.260.5510.7734.660.5418.3732.350.5010.6536.790.6714.85Swatches35.310.557.5733.220.557.5732.350.5310.6634.750.659.7633.050.679.8238.150.7710.2430.250.7013.7237.670.8712.62


## 3.3. Segmentation Phase

The next step after refining the dermoscopic images is localization and segmentation of melanoma lesion and the resulting segmented lesion is considered as region of interest (ROI) for the learning of melanoma malignancy model. In this experiment, localization and segmentation performance of region-extreme convolutional neural network is evaluated. For generation of localization model, we trained the localization block using training images of datasets and evaluated the performance of trained localization model using test samples.The internal representation of the feature map to discriminate the melanoma region through localization and segmentation block is shown in Figure4. The CNN softmax layer was responsible for predicting the probabilities at pixel level for categorizing each pixel into normal or melanoma affected pixel. The localization block used these labeled pixels and applied selective search to select region proposal and computed the intersection over union (IoU) threshold for estimation of melanoma lesion. We have opted 0.5 IoU threshold for the detection of melanoma lesion; the region is considered as normal skin region below this value as shown in Figure 5. The RCNN localized the melanoma lesion with 0.94 mAP for ISIC-2016 and 0.95 mAP for ISIC-2017 dataset, as illustrated in Table 3.Figure 4
The deep feature representation of melanoma region and normal skin region. These deep features are used to establish the model for melanoma malignancy recognition. The green overlay represents the normal skin attributes and purple overlay represents the melanoma affected lesion through localization and segmentation block.Figure 5
The resultant ISIC samples obtained after melanoma localization step.Table 3
The achieved mean average precision (mAP) and processing time in milliseconds (ms) for melanoma localization through the region-extreme convolutional neural network using ISIC-2016 and ISIC-2017 dataset.
ISIC-2016 artefactsmAPTime (ms)ISIC-2017 artefactsmAPTime (ms)Thick hair0.9740.1Thick hair0.9540.2Think hair0.9642.1Think hair0.9843.4Rule marks0.9043.3Rule marks0.9254.6Color charts0.9139.4Color charts0.8540.8Black frame0.9641.6Black frame0.8852.4For the comparison with other methods, the melanoma localized regions are further processed to obtain fine boundaries of melanoma region. The FCM clustered the pixels into melanoma lesion and normal skin. The resulting performance portrayed the good melanoma segmentation results as illustrated in Figure6. The average values of SP as 0.9417, SE as 0.9782, Ac as 0.948, Di as 0.94, and Ja as 0.93 were recorded.Figure 6
(a) Input images with artefacts. (b) Ground truth melanoma lesion images. (c) Localization of melanoma lesion. (d) Blue overlay of fuzzy segmentation of melanoma boundary and red overlay of ground truth melanoma lesion.
(a)(b)(c)(d)Our results are better than the state-of-the-art methods due to precise localization of melanoma lesion through the proposed method. In the few cases, localization block was unable to detect the melanoma region due to visual similarity with normal skin regions, as shown in Figure7 [1].Figure 7
The localization and segmentation block fails to identify the melanoma benign cases, due to visual similarity with normal skin attributes.


## 3.4. Performance Evaluation of Recognition Task

To build the recognition model, we trained the recognition block using training images of ISIC-2016 and ISIC-2017 datasets and evaluated the performance of trained model using test samples. The sigmoid activation function of ELM was used for feed-forward model learning with 150000 number of hidden nodes. The backbone CNN models opted for to extract deep discriminative representation of melanoma were AlexNet [36], VGG16 [37], VGG19 [37], ResNet50 [38], ResNet101 [38], GoogleNet [39], and DenseNet201 [40], and their performance was observed based on segmented melanoma regions as listed in Table 4.Table 4
Comparison of different network layers on the classification of benign and malignant melanoma.
NetworkLayerFeaturesISIC-2016ISIC-2017ACAPACAPAlexNetpool592160.840.810.780.77Fc640960.850.830.930.91Fc740960.840.800.880.87Fc810000.840.770.870.86VGG16Fc640960.820.730.860.88FC740960.820.740.820.81FC810000.820.730.820.81VGG19Fc640960.840.830.880.87Fc740960.810.720.850.84Fc810000.810.700.800.80GoogleNetPred10000.810.560.850.84DenseNet201Fc100010000.820.770.870.86ResNet50Fc100010000.860.830.920.90ResNet101Fc100010000.810.680.860.88Bold values indicate the best results.


### 3.4.1. Impact of the Features Extracted from Different Network Depths

In this experiment, we explored the recognition capability of region-extreme convolutional neural network using different deep features extracted from the various CNN networks from segmented melanoma lesion. The best network was chosen as a feature extractor of the region-extreme convolutional neural network’s recognition stage. The deep features extracted from segmented melanoma lesion were used to feed the ELM classifier for model learning. The generated ELM model was able to discriminate benign and malignant melanoma. The performance of proposed recognition model is reported in Table4 which illustrates the optimal CNN model and layer for the deep features extraction.The receptive fields generated from the deep features at different layers of convolutional neural networks represent contextual and semantic variations of dermoscopy images. Therefore, receptive fields at different level of convolutional network perform differently. On inspecting the behavior of melanoma malignancy model at different CNN layer, a common pattern is observed among all kinds of CNN models. The receptive field obtained from initial layers of CNN models was not decisive and is inferior as lower layers of CNN models collect low-level description and these patterns lack in-variance attribute. A similar characteristic was observed from the CNN middle layers, while the receptive fields obtained from the penultimate layers of CNN networks perform better than other layers because the penultimate layers receptive fields hold high level semantics and contextual information of melanoma malignancy.It is notable that AlexNet fully connected layerFc6 achieved the highest average precision, because Fc6 holds the higher dimensional (4096) deep features to represent the benign and malignant melanoma for ISIC-2016 and ISIC-2017 dataset, whereas the AlexNet pool5 layer captured 9216 features but pool5 feature’s recognition capability is lower than fully connected layer Fc6, because higher-dimensional features of pool5 become repeatable in both benign and malignant classes. This experiment supported us to identify the optimal backbone CNN model with penultimate layer to extract the deep features for generation of melanoma malignancy model. The performance of deep features extracted from the ResNet50 was comparable with AlexNet for ISIC-2016. However, the performance of deep features extracted from the ResNet50 was observed to be lower than AlexNet for ISIC-2017. The observed prediction accuracy and precision using AlexNet and ResNet50 features for ISIC-2016 are 0.86 and 0.83, while AlexNet performed better in prediction accuracy (0.93) and precision (0.91) to recognize benign and malignant melanoma.Another observation that can be derived from Table4 is that the penultimate layers of shallower sequential CNN model (e.g., AlexNet) performed better than the other deeper sequential and deeper directed acyclic CNN models, e.g., Resent50. Therefore, we have utilized AlexNet as the backbone model for features extraction and fed these extracted deep features from AlexNet to ELM for feed-forward model generation. Here, it is important to highlight the essence of feed-forward ELM model that learned the model using deep features without back-propagation and weight optimization like CNN networks. This attribute improves the learning and reduces the computational time as compared to the state-of-the-art CNN models.This experiment concluded that sequential shallower CNN network (e.g. AlexNet) penultimate layers deep features established the stronger ELM decision boundaries to predict the melanoma malignancy. On the basis of this experiment, we opted for AlexNet as feature extraction model for recognition block and examined the rest of our experiments using AlexNet as feature extractor for recognition block.


### 3.4.2. Impact of PCA on Classification Task

To further improve the recognition model, we examined the dimensionality reduction of receptive fields obtained from Alexnet fc6 layer and applied PCA on it. As PCA reduces the covariant features and discovers compact, there is meaningful representation of deep features to recognize intraclass melanoma test samples. It can be noticed in Table5 that the deep features generated a good model to discriminate benign and malignant melanoma, when represented by PCA. Therefore, using PCA at the top of deep features is beneficial for classification task.Table 5
Results of classification with and without PCA.
DatasetPMWith PCAWithout PCAAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.830.820.840.85AP0.830.740.830.830.780.730.790.82ISIC-2017AC0.930.860.880.910.920.850.860.89AP0.910.850.860.890.890.840.850.88


### 3.4.3. Classification with and without Melanoma Segmentation Phase

To evaluate the performance of our region-extreme convolutional network, we evaluated the performance of classification with and without the segmentation of melanoma region. All the parameters and network architectures remained the same in both experiments. All the recorded experimental results are listed in Table6. The segmented melanoma ROIs produced a better model in discriminating benign and malignant melanoma as compared to complete dermoscopic images because segmented lesion contains only benign and malignant feature representation despite having feature representation from normal skin or the artefacts. The AP of segmented image was relatively 5% higher than non-segmented dermoscopic images. Defining the melanoma area as a region of interest and the extraction of deep features to generate model is a smart way to train ELM, and to classify precisely without being distracted from the clinical artefacts or healthy skin region. We conclude from this experiment that the segmented melanoma lesion when fed to recognition block for extraction of receptive fields is more decisive in building accurate melanoma malignancy recognition model.Table 6
Performance of classification task with and without segmentation.
DatasetPMSegmentedWithout segmentationAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.840.810.830.84AP0.830.740.830.830.770.700.730.80ISIC-2017AC0.930.860.880.920.910.840.850.88AP0.910.850.860.900.890.830.830.85


### 3.4.4. Impact of Classification Model in Melanoma Subclass Recognition

The performance of recognition block is further investigated through the impact of ELM as a classifier against traditional classifiers including support vector machine (SVM) and K-nearest neighbor (KNN). To observe the impact, the entire recognition block remained the same while the classifier layer was changed with ELM, support vector machine (SVM), and K-nearest neighbor (KNN). The region-extreme convolutional neural network segmented the melanoma lesion and extracted the deep features from backbone models such as Alexnet, VGG16, and VGG19 and trained different types of classifiers including ELM, SVM, and KNN as a separate model. It is evident from Table7 that the ELM classifier outperformed all the classifiers, where K-nearest neighbor performed better that SVM classifier.Table 7
Comparison of different classifiers for recognition of benign and malignant melanoma class.
DatasetPMELMKNNSVMAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.800.770.780.780.750.760.750.73AP0.830.740.830.830.580.560.580.570.620.630.600.59ISIC-2017AC0.930.860.880.880.830.830.840.800.860.800.810.85AP0.910.880.870.860.810.810.810.800.840.800.800.83The ELM classifier trains the learning model in a two-step process; first it randomizes the deep feature mapping and then it identifies the solution for linear parameters. In the first step, the ELM hidden layer randomly initializes to map the deep features on ELM feature space using nonlinear piece-wise function. The second step of ELM separates it from traditional classifiers including SVM which utilizes the kernel function for feature projection, while the deep learning models require autoencoder or restricted Boltzmann machines to map input features and form the similar deep representation.Essentially, the ELM generates hidden nodes randomly without explicitly being trained that makes ELM learning generalized and invariant from input training data. The ELM’s hidden nodes are assigned using continuous probability distribution. Therefore, ELM models are stronger than traditional classifiers (SVM, KNN) or back-propagation neural networks. Moreover, the regularization function within the ELM also supports it to avoid the problems of overfitting and underfitting without any computational overhead.We found that the ELM enhances the performance of the system to predict intraclass label of melanoma, with 83% average precision for ISIC-2016 and 91% for ISIC-2017 dataset. The performance of different classifiers is listed in Table7. All the performance metrics indicate the effectiveness of ELM within the recognition block of region-extreme convolutional neural network. Therefore, this experiment concludes that ELM is the optimal choice to learn the feed-forward melanoma malignancy model using dermoscopy images.


## 3.4.1. Impact of the Features Extracted from Different Network Depths

In this experiment, we explored the recognition capability of region-extreme convolutional neural network using different deep features extracted from the various CNN networks from segmented melanoma lesion. The best network was chosen as a feature extractor of the region-extreme convolutional neural network’s recognition stage. The deep features extracted from segmented melanoma lesion were used to feed the ELM classifier for model learning. The generated ELM model was able to discriminate benign and malignant melanoma. The performance of proposed recognition model is reported in Table4 which illustrates the optimal CNN model and layer for the deep features extraction.The receptive fields generated from the deep features at different layers of convolutional neural networks represent contextual and semantic variations of dermoscopy images. Therefore, receptive fields at different level of convolutional network perform differently. On inspecting the behavior of melanoma malignancy model at different CNN layer, a common pattern is observed among all kinds of CNN models. The receptive field obtained from initial layers of CNN models was not decisive and is inferior as lower layers of CNN models collect low-level description and these patterns lack in-variance attribute. A similar characteristic was observed from the CNN middle layers, while the receptive fields obtained from the penultimate layers of CNN networks perform better than other layers because the penultimate layers receptive fields hold high level semantics and contextual information of melanoma malignancy.It is notable that AlexNet fully connected layerFc6 achieved the highest average precision, because Fc6 holds the higher dimensional (4096) deep features to represent the benign and malignant melanoma for ISIC-2016 and ISIC-2017 dataset, whereas the AlexNet pool5 layer captured 9216 features but pool5 feature’s recognition capability is lower than fully connected layer Fc6, because higher-dimensional features of pool5 become repeatable in both benign and malignant classes. This experiment supported us to identify the optimal backbone CNN model with penultimate layer to extract the deep features for generation of melanoma malignancy model. The performance of deep features extracted from the ResNet50 was comparable with AlexNet for ISIC-2016. However, the performance of deep features extracted from the ResNet50 was observed to be lower than AlexNet for ISIC-2017. The observed prediction accuracy and precision using AlexNet and ResNet50 features for ISIC-2016 are 0.86 and 0.83, while AlexNet performed better in prediction accuracy (0.93) and precision (0.91) to recognize benign and malignant melanoma.Another observation that can be derived from Table4 is that the penultimate layers of shallower sequential CNN model (e.g., AlexNet) performed better than the other deeper sequential and deeper directed acyclic CNN models, e.g., Resent50. Therefore, we have utilized AlexNet as the backbone model for features extraction and fed these extracted deep features from AlexNet to ELM for feed-forward model generation. Here, it is important to highlight the essence of feed-forward ELM model that learned the model using deep features without back-propagation and weight optimization like CNN networks. This attribute improves the learning and reduces the computational time as compared to the state-of-the-art CNN models.This experiment concluded that sequential shallower CNN network (e.g. AlexNet) penultimate layers deep features established the stronger ELM decision boundaries to predict the melanoma malignancy. On the basis of this experiment, we opted for AlexNet as feature extraction model for recognition block and examined the rest of our experiments using AlexNet as feature extractor for recognition block.


## 3.4.2. Impact of PCA on Classification Task

To further improve the recognition model, we examined the dimensionality reduction of receptive fields obtained from Alexnet fc6 layer and applied PCA on it. As PCA reduces the covariant features and discovers compact, there is meaningful representation of deep features to recognize intraclass melanoma test samples. It can be noticed in Table5 that the deep features generated a good model to discriminate benign and malignant melanoma, when represented by PCA. Therefore, using PCA at the top of deep features is beneficial for classification task.Table 5
Results of classification with and without PCA.
DatasetPMWith PCAWithout PCAAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.830.820.840.85AP0.830.740.830.830.780.730.790.82ISIC-2017AC0.930.860.880.910.920.850.860.89AP0.910.850.860.890.890.840.850.88


## 3.4.3. Classification with and without Melanoma Segmentation Phase

To evaluate the performance of our region-extreme convolutional network, we evaluated the performance of classification with and without the segmentation of melanoma region. All the parameters and network architectures remained the same in both experiments. All the recorded experimental results are listed in Table6. The segmented melanoma ROIs produced a better model in discriminating benign and malignant melanoma as compared to complete dermoscopic images because segmented lesion contains only benign and malignant feature representation despite having feature representation from normal skin or the artefacts. The AP of segmented image was relatively 5% higher than non-segmented dermoscopic images. Defining the melanoma area as a region of interest and the extraction of deep features to generate model is a smart way to train ELM, and to classify precisely without being distracted from the clinical artefacts or healthy skin region. We conclude from this experiment that the segmented melanoma lesion when fed to recognition block for extraction of receptive fields is more decisive in building accurate melanoma malignancy recognition model.Table 6
Performance of classification task with and without segmentation.
DatasetPMSegmentedWithout segmentationAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.840.810.830.84AP0.830.740.830.830.770.700.730.80ISIC-2017AC0.930.860.880.920.910.840.850.88AP0.910.850.860.900.890.830.830.85


## 3.4.4. Impact of Classification Model in Melanoma Subclass Recognition

The performance of recognition block is further investigated through the impact of ELM as a classifier against traditional classifiers including support vector machine (SVM) and K-nearest neighbor (KNN). To observe the impact, the entire recognition block remained the same while the classifier layer was changed with ELM, support vector machine (SVM), and K-nearest neighbor (KNN). The region-extreme convolutional neural network segmented the melanoma lesion and extracted the deep features from backbone models such as Alexnet, VGG16, and VGG19 and trained different types of classifiers including ELM, SVM, and KNN as a separate model. It is evident from Table7 that the ELM classifier outperformed all the classifiers, where K-nearest neighbor performed better that SVM classifier.Table 7
Comparison of different classifiers for recognition of benign and malignant melanoma class.
DatasetPMELMKNNSVMAlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50AlexNetVGG16VGG19ResNet50ISIC-2016AC0.860.820.840.860.800.770.780.780.750.760.750.73AP0.830.740.830.830.580.560.580.570.620.630.600.59ISIC-2017AC0.930.860.880.880.830.830.840.800.860.800.810.85AP0.910.880.870.860.810.810.810.800.840.800.800.83The ELM classifier trains the learning model in a two-step process; first it randomizes the deep feature mapping and then it identifies the solution for linear parameters. In the first step, the ELM hidden layer randomly initializes to map the deep features on ELM feature space using nonlinear piece-wise function. The second step of ELM separates it from traditional classifiers including SVM which utilizes the kernel function for feature projection, while the deep learning models require autoencoder or restricted Boltzmann machines to map input features and form the similar deep representation.Essentially, the ELM generates hidden nodes randomly without explicitly being trained that makes ELM learning generalized and invariant from input training data. The ELM’s hidden nodes are assigned using continuous probability distribution. Therefore, ELM models are stronger than traditional classifiers (SVM, KNN) or back-propagation neural networks. Moreover, the regularization function within the ELM also supports it to avoid the problems of overfitting and underfitting without any computational overhead.We found that the ELM enhances the performance of the system to predict intraclass label of melanoma, with 83% average precision for ISIC-2016 and 91% for ISIC-2017 dataset. The performance of different classifiers is listed in Table7. All the performance metrics indicate the effectiveness of ELM within the recognition block of region-extreme convolutional neural network. Therefore, this experiment concludes that ELM is the optimal choice to learn the feed-forward melanoma malignancy model using dermoscopy images.


## 4. Discussion and Comparison with State-of-the-Art Methods

### 4.1. Melanoma Segmentation

At ISIC-2016 challenge of skin lesion segmentation task, 28 contestants participated and the results of top scorers are presented in Figure8. The final positions were marked based on highest Ja index. Almost all the top scorers utilized the benefit of deep learning methods to segment the melanoma region due to the performance gain of deep learning algorithms. The majority of researchers explored Alexnet, VGG-16, or deep residual network for the segmentation task and performed better than traditional handcrafted features-based segmentation techniques, whereas we have used RCNN [1] with shallower network for the same task and our results validated the good performance of our method as compared to other state-of-the-art traditional handcrafted feature segmentation and deep learning methods. From Figure 8, it can be observed that Ja of our method improved by 0.11 points as compared to ISIC-2016 segmentation task top-scorer ExB. The reason behind this significant improvement of our method is due to precise localization step before segmentation. Region-extreme convolutional neural network precisely reduced the search space for melanoma lesion and produced the effective melanoma segmentation through considering the artefacts and normal skin as background region.Figure 8
Comparison of the proposed segmentation technique with ISIC-2016 challenge participants results of skin lesion segmentation task.To further evaluate the performance of our proposed approach for melanoma segmentation, we compared our method with state-of-the art segmentation techniques, as represented in Table8. Traditional approaches for melanoma segmentation, e.g., active contour, bootstrapping [9], contextual hypergraph [10], clustering [54], region split and merge, region growing [55], sparse coding [48], and thresholding techniques [11] produce lower value of average Ja score than our method. This is due to the fact that these approaches segmented the melanoma lesion based on spatial context information in unsupervised fashion, while our proposed approach initially builds a model based on the high-level color, texture, and spatial representation of melanoma. Therefore, the trained model localized melanoma lesion with high mAP at test time and only the detected region was considered for segmentation.Table 8
Performance comparison of region-extreme convolutional neural network’s melanoma segmentation with state-of-the-art methods.
TechniqueAcDiJaTime (ms)Adaptive thresholding [11]0.720.560.452Bootstrap learning [9]0.780.720.57—Contextual hypergraph [10]0.830.750.60.3ISO [47]0.820.680.56—Level set [12]0.70.580.460.84Sparse coding [48]0.910.80.670.1Statistical region growing [13]0.730.550.430.4Yen’s thresholding [49]0.810.670.58—FCN [14]0.820.820.860.05Segnet [50]0.910.920.860.06Region-extreme convolutional neural network0.940.940.930.05Bold values indicate the best results.Besides the traditional techniques for melanoma segmentation, deep learning-based approaches including FCN [14] and Segnet [50] performed melanoma segmentation with Ja score of 0.86. The FCN [14] and Segnet [50] were designed as a deeper network for segmentation of melanoma lesion, therefore training the model requires tuning of thousands of hyperparameters which makes the melanoma segmentation task computationally expensive for real-time application. It is evident from Table 8 that our method exhibits lower computational time with precise melanoma segmentation as compared to state-of-the-art methods. Our Jaccard score is 13% higher than Segnet [50], due to the training of region-extreme convolutional neural network considering the clinical and natural artefacts as background region and only detecting the melanoma lesions.


### 4.2. Melanoma Malignancy Recognition

In ISIC-2016 challenge, 25 teams participated at melanoma malignancy recognition task and submitted their results. The results of classification task were evaluated based on average precisionAP. In Table 9, top ten results of ten participants are listed. It can be observed that our system’s AP value 0.831 is higher than all the participants, validating the effectiveness of the proposed scheme for the classification of melanoma into benign and malignant.Table 9
Performance comparison of region-extreme convolutional neural network with ISIC-2016 challenge results for skin lesion classification task.
MethodACAPCUMED0.8550.637GTDL0.8130.619BF-TB0.8340.598ThrunLab0.7860.563Jordan Yap0.8440.559Haebeom Lee0.8210.555GT-DL10.8150.552GT-DL20.6810.545Sebastien PARIS0.7310.542USYD-BMIT0.5990.537Region-extreme convolutional neural network (our)0.860.831Bold values indicate the best results.Moreover, our region-extreme convolutional neural network also performed better that existing state-of-the-art melanoma malignancy models, as illustrated in Table10. To compare the melanoma malignancy recognition performance of region-extreme convolutional neural network against other CNN based methods, we have included a comparative study as illustrated in Table 10. Our region-extreme convolutional neural network first identified the melanoma lesion and segmented it to extract the deep features from penultimate layers of backbone model and further used these deep features to generate feed-forward extreme learning model for melanoma malignancy recognition. The method proposed by [34, 51] also used CNN models to extract the deep features and used fisher encoding scheme to train the SVM for melanoma malignancy recognition. The performance of [34] is comparable to our method, while our method performed 3% higher than [51] due to the difference in mapping function of classifier. The mapping function of the SVM [51] cannot define the decision boundaries accurately among the benign and malignant melanoma. Our region-extreme convolutional neural network improved the recognition performance without adapting transfer learning across domains and effectively generalized the recognition model through the ELM.Table 10
Performance comparison of region-extreme convolutional neural network’s melanoma malignancy recognition with state-of-the-art CNN methods.
MethodNetworkISIC-2016ISIC-2017Time (ms)CNN SVM [51]Alexnet0.830.800.1VGG-160.840.79ResNet500.840.82CNNaug SVM [51]Alexnet0.830.810.1VGGNet0.830.82ResNet500.830.81Fisher GL [52]Alexnet0.820.810.13Transfer learning CNN [38]ResNet500.840.830.1CUMED [33]ResNet500.850.870.2Codella [53]HybridNet0.800.910.09DCNN-FV [34]ResNet500.860.890.05Region-extreme convolutional neural network (Ours)Alexnet0.860.930.05Region-extreme convolutional neural network (Ours)ResNet500.860.920.06Bold values indicate the best results.Comparing the recognition performance with [33, 52, 53] our region-extreme convolutional neural network outperformed these methods by 1%, 4%, and 1% on ISIC-2016 dataset and 2%, 10%, and 6% on ISIC-2017 dataset, because [33, 52, 53] built the melanoma malignancy model using entire dermoscopic image which also includes the artefacts and more than 50% of normal skin pixels. However, our region-extreme convolutional neural network only considered the melanoma lesion to establish the melanoma malignancy model.


### 4.3. Computational Time Complexity

The average computational time to recognize the melanoma malignancy using single dermoscopic image is reported in milliseconds (ms) in Table10. The state-of-the-art methods have higher computational cost as [33] utilized very deep residual convolutional neural network that requires fine-tuning of millions of hyperparameters, while [53] aggregated the handcrafted features, combined three CNN pretrained networks, and used sparse coding scheme to ensemble fully convolutional U-Net for melanoma malignancy recognition. Our proposed method requires 0.05 milliseconds to recognize the malignancy of melanoma. This is due to the feed-forward model learning of ELM that generated melanoma malignancy model without back-propagation of the loss for weight optimization. Therefore, our region-extreme convolutional neural network is more efficient and can be easily employed to other applications as well.


## 4.1. Melanoma Segmentation

At ISIC-2016 challenge of skin lesion segmentation task, 28 contestants participated and the results of top scorers are presented in Figure8. The final positions were marked based on highest Ja index. Almost all the top scorers utilized the benefit of deep learning methods to segment the melanoma region due to the performance gain of deep learning algorithms. The majority of researchers explored Alexnet, VGG-16, or deep residual network for the segmentation task and performed better than traditional handcrafted features-based segmentation techniques, whereas we have used RCNN [1] with shallower network for the same task and our results validated the good performance of our method as compared to other state-of-the-art traditional handcrafted feature segmentation and deep learning methods. From Figure 8, it can be observed that Ja of our method improved by 0.11 points as compared to ISIC-2016 segmentation task top-scorer ExB. The reason behind this significant improvement of our method is due to precise localization step before segmentation. Region-extreme convolutional neural network precisely reduced the search space for melanoma lesion and produced the effective melanoma segmentation through considering the artefacts and normal skin as background region.Figure 8
Comparison of the proposed segmentation technique with ISIC-2016 challenge participants results of skin lesion segmentation task.To further evaluate the performance of our proposed approach for melanoma segmentation, we compared our method with state-of-the art segmentation techniques, as represented in Table8. Traditional approaches for melanoma segmentation, e.g., active contour, bootstrapping [9], contextual hypergraph [10], clustering [54], region split and merge, region growing [55], sparse coding [48], and thresholding techniques [11] produce lower value of average Ja score than our method. This is due to the fact that these approaches segmented the melanoma lesion based on spatial context information in unsupervised fashion, while our proposed approach initially builds a model based on the high-level color, texture, and spatial representation of melanoma. Therefore, the trained model localized melanoma lesion with high mAP at test time and only the detected region was considered for segmentation.Table 8
Performance comparison of region-extreme convolutional neural network’s melanoma segmentation with state-of-the-art methods.
TechniqueAcDiJaTime (ms)Adaptive thresholding [11]0.720.560.452Bootstrap learning [9]0.780.720.57—Contextual hypergraph [10]0.830.750.60.3ISO [47]0.820.680.56—Level set [12]0.70.580.460.84Sparse coding [48]0.910.80.670.1Statistical region growing [13]0.730.550.430.4Yen’s thresholding [49]0.810.670.58—FCN [14]0.820.820.860.05Segnet [50]0.910.920.860.06Region-extreme convolutional neural network0.940.940.930.05Bold values indicate the best results.Besides the traditional techniques for melanoma segmentation, deep learning-based approaches including FCN [14] and Segnet [50] performed melanoma segmentation with Ja score of 0.86. The FCN [14] and Segnet [50] were designed as a deeper network for segmentation of melanoma lesion, therefore training the model requires tuning of thousands of hyperparameters which makes the melanoma segmentation task computationally expensive for real-time application. It is evident from Table 8 that our method exhibits lower computational time with precise melanoma segmentation as compared to state-of-the-art methods. Our Jaccard score is 13% higher than Segnet [50], due to the training of region-extreme convolutional neural network considering the clinical and natural artefacts as background region and only detecting the melanoma lesions.


## 4.2. Melanoma Malignancy Recognition

In ISIC-2016 challenge, 25 teams participated at melanoma malignancy recognition task and submitted their results. The results of classification task were evaluated based on average precisionAP. In Table 9, top ten results of ten participants are listed. It can be observed that our system’s AP value 0.831 is higher than all the participants, validating the effectiveness of the proposed scheme for the classification of melanoma into benign and malignant.Table 9
Performance comparison of region-extreme convolutional neural network with ISIC-2016 challenge results for skin lesion classification task.
MethodACAPCUMED0.8550.637GTDL0.8130.619BF-TB0.8340.598ThrunLab0.7860.563Jordan Yap0.8440.559Haebeom Lee0.8210.555GT-DL10.8150.552GT-DL20.6810.545Sebastien PARIS0.7310.542USYD-BMIT0.5990.537Region-extreme convolutional neural network (our)0.860.831Bold values indicate the best results.Moreover, our region-extreme convolutional neural network also performed better that existing state-of-the-art melanoma malignancy models, as illustrated in Table10. To compare the melanoma malignancy recognition performance of region-extreme convolutional neural network against other CNN based methods, we have included a comparative study as illustrated in Table 10. Our region-extreme convolutional neural network first identified the melanoma lesion and segmented it to extract the deep features from penultimate layers of backbone model and further used these deep features to generate feed-forward extreme learning model for melanoma malignancy recognition. The method proposed by [34, 51] also used CNN models to extract the deep features and used fisher encoding scheme to train the SVM for melanoma malignancy recognition. The performance of [34] is comparable to our method, while our method performed 3% higher than [51] due to the difference in mapping function of classifier. The mapping function of the SVM [51] cannot define the decision boundaries accurately among the benign and malignant melanoma. Our region-extreme convolutional neural network improved the recognition performance without adapting transfer learning across domains and effectively generalized the recognition model through the ELM.Table 10
Performance comparison of region-extreme convolutional neural network’s melanoma malignancy recognition with state-of-the-art CNN methods.
MethodNetworkISIC-2016ISIC-2017Time (ms)CNN SVM [51]Alexnet0.830.800.1VGG-160.840.79ResNet500.840.82CNNaug SVM [51]Alexnet0.830.810.1VGGNet0.830.82ResNet500.830.81Fisher GL [52]Alexnet0.820.810.13Transfer learning CNN [38]ResNet500.840.830.1CUMED [33]ResNet500.850.870.2Codella [53]HybridNet0.800.910.09DCNN-FV [34]ResNet500.860.890.05Region-extreme convolutional neural network (Ours)Alexnet0.860.930.05Region-extreme convolutional neural network (Ours)ResNet500.860.920.06Bold values indicate the best results.Comparing the recognition performance with [33, 52, 53] our region-extreme convolutional neural network outperformed these methods by 1%, 4%, and 1% on ISIC-2016 dataset and 2%, 10%, and 6% on ISIC-2017 dataset, because [33, 52, 53] built the melanoma malignancy model using entire dermoscopic image which also includes the artefacts and more than 50% of normal skin pixels. However, our region-extreme convolutional neural network only considered the melanoma lesion to establish the melanoma malignancy model.


## 4.3. Computational Time Complexity

The average computational time to recognize the melanoma malignancy using single dermoscopic image is reported in milliseconds (ms) in Table10. The state-of-the-art methods have higher computational cost as [33] utilized very deep residual convolutional neural network that requires fine-tuning of millions of hyperparameters, while [53] aggregated the handcrafted features, combined three CNN pretrained networks, and used sparse coding scheme to ensemble fully convolutional U-Net for melanoma malignancy recognition. Our proposed method requires 0.05 milliseconds to recognize the malignancy of melanoma. This is due to the feed-forward model learning of ELM that generated melanoma malignancy model without back-propagation of the loss for weight optimization. Therefore, our region-extreme convolutional neural network is more efficient and can be easily employed to other applications as well.


## 5. Conclusion

In this paper, we presented a novel region-extreme convolutional neural network model to predict the malignancy of melanoma from dermoscopic images. The novel region-extreme convolutional neural network is capable of melanoma localization with precise melanoma boundary segmentation along with malignancy recognition of benign and malignant melanoma. Our demonstrated results on ISIC-2016 and ISIC-2017 showed that the localization of melanoma lesion is fruitful to learn accurate model for melanoma recognition as compared to using an entire dermoscopic image. We achieved higher performance gain as compared to state-of-the-art methods for melanoma segmentation and recognition task. We also revealed that the deep features extracted from inner layers of CNN model hold high-level representation of melanoma malignancy information and established stronger model as compared to the features extracted from the last layer of CNN model. Furthermore, our region-extreme convolutional neural network also validates that the deep features with ELM classifier generate accurate model for melanoma recognition. Our region-extreme convolutional neural network highlighted the effective procedure for training and can be utilized in other biomedical image analysis applications. In the future, we will investigate the comprehensive clinical decision support system for medical images and explore our system on other applications.



---

*Source: 6671498-2021-07-13.xml*