# On the Reusability of ISIC Data for Training DL Classifiers Applied on Clinical Skin Images

Konstantinos Moutselos, Ilias Maglogiannis

► To cite this version:

Konstantinos Moutselos, Ilias Maglogiannis. On the Reusability of ISIC Data for Training DL Classifiers Applied on Clinical Skin Images. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.199-212, 10.1007/978-3-031-08341-9_17. hal-05335628

HAL Id: hal-05335628

https://inria.hal.science/hal-05335628v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# On the Reusability of ISIC Data for Training DL Classifiers Applied on Clinical Skin Images

Konstantinos Moutselos and Ilias Maglogiannis

Department of Digital Systems, University of Piraeus, Athens, Greece
{kmouts,imaglo}@unipi.gr

**Abstract.** The ISIC archive is an open dermoscopy dataset containing thousands of images so that new Deep Learning skin classifiers can be trained. ISIC Challenges attract many participants to build a model that will bring the best performance to the ISIC test dataset. The question is whether such a model has consistent behavior in different datasets and other clinical images. In this work, we build and study the performance of a classifier trained in the ISIC 2019 dataset in three different cases: the performance during the cross-validation training process, the performance in the separate ISIC 2019 test dataset, and dermoscopy images taken from the SYGGROS skin disease hospital. The results show a stable performance compared to the metric F1 score for the categories in which there are more than 3000 images in the training dataset. In addition, we identify the factors that make it difficult to transfer and use classifiers from a competitive to a clinical setting.

**Keywords:** ISIC · Skin lesions · Deep learning classifiers

## 1 Introduction

Cutaneous melanoma is one of the most aggressive forms of cancer and in 2021 cases are expected to increase by about 5%, with a corresponding increase in mortality due to this disease [1]. Early diagnosis plays an important role in the progression of the disease. If diagnosed early, the 5-year survival rate for melanoma is 99% [1] while the normal statistic in the United States is 91.6% [2]. Therefore, great effort has been made to develop smart computing tools that use dermatoscopic images and can assist clinicians in their diagnosis [3]. The ever-better results of skincare diagnostic applications increase the acceptance of their use by the medical community and encourage dermatologists to provide more anonymous clinical data that further helps to develop better supporting applications. In this direction, the International Skin Imaging Collaboration (ISIC) has developed the ISIC Archive [4], an international repository of dermoscopic images. ISIC supports research toward automated algorithmic analysis by hosting the ISIC Challenges. The annual ISIC competitions [5], providing a common data set as a starting point, encourage competition in computing technologies to achieve better-performing algorithms in the classification of a skin lesion. In the context of these competitions, there is an opportunity to develop and test new techniques and architectures of deep learning that in recent years have recorded impressive results.
An important question, however, is whether a prognostic model trained in these competitions can respond similarly and consistently to new clinical data, obtained with different technical equipment and under different clinical conditions. To this question, there is a scarcity of publications that study the effectiveness of models trained in a specific dataset, upon different clinical data.

Valle et al. [6] conducted a thorough multifactorial analysis having as a starting point the ISIC2017 Challenge which involved considerably smaller image datasets (compared to ISIC2019) and three classes of skin lesions. They reported that the correlation of the results of ensembles on different datasets was far from perfect. E.g., the Spearman correlation between the ISIC2017/test and EDRA/dermo [7] datasets was .65 on melanoma AUC. As was reported, this was another example of the pitfalls of hyper-optimizing on specific datasets. To deal with vulnerabilities of hyper-optimizing and assess models on their generalization capabilities Maron et al. [8] proposed recently a benchmark skin dataset with unknown ground-truth labels, where the evaluation is provided by the authors themselves.

Nahata et al. [9] combined data from ISIC 2018 & 2019 to create a larger dataset, where the evaluation of the tested models, however, is done in the same dataset. This - as discussed in Discussion - carries an even greater risk of over-optimistic reports. In a different perspective, Lucius et al. [3] use a random split of the HAM10000 dataset [10] via the ISIC archive to train and evaluate the performance of eight different DL architectures with low/high image resolution modes and with/without clinical meta-data. The split is reported that resulted in completely disjoint train/test datasets. However, no action is described against the class imbalances of the initial dataset, and the two outcomes after the split. Moreover, most of the results are evaluated using the metric accuracy (and the complement: error rate) which, contestably, the authors report as measures suitable to evaluate performance on imbalanced datasets. Next, they compare the least performant classifier against the accuracy of diagnoses from non-dermatologist general practitioners and assess the use of the same classifier as a decision support tool for the physicians.

As an example of high performant classifiers over reduced confounding factors, Zhu et al. [11] study the performance of an enhanced DL skin-disease model in a clinical environment using, however, a controlled clinical dataset as well. Their images -spanning 14 skin conditions - are collected in the same Department, by the same technician, using the same dermoscopy system. Furthermore, all the patients were having the same skin type. Regarding the skin type, however, we should note that the HAM10000 dataset [10] and ISIC itself are also biased, consisting of samples from fair-skinned Caucasian population.

In this work, we use the data of the ISIC 2019 competition [12] which consists of about 25 thousand dermoscopy images, for the training of a DL classifier, without the use of other metadata. Next, we compare the effectiveness of the classifier in three cases: during the cross-validation training process, the result in the comparative test dataset of ISIC 2019 consisting of about 8 thousand images with unknown ground-truth labels and class distribution, and the performance in a new clinical dataset (SYGGROS) consisting of only 311 images. We chose the ISIC 2019 dataset due to the number of freely available images that include 8 categories of skin diseases, as well as the external evaluation
and comparison procedures of the models that the 2019 ISIC challenge includes. A requirement for the participants is the submission of a report describing the construction of the submitted model. In the Discussion section, we compare our results with the best model of the competition that was trained in the same ISIC 2019 dataset.

Regarding the structure of the paper, after the Introduction follows the Materials & Methods where we describe the datasets of the work, the DL Architecture together with the methods of image pre-processing. Follows the reporting of the selected DL parameters as well as the performance metrics used. In the Results, the classifier is evaluated in the Cross-Validation training procedure, in the ISIC 2019 Test Dataset, and the new SYGGROS Dataset. Following is the Discussion where the results are compared, and last the Conclusions of the work.

## 2 Materials and Methods

Figure 1 shows the workflow for the comparison of the performance of a DL Classifier trained with the ISIC 2019 Training Dataset. On the left side of the figure with the thickest arrow, the training process of the model is represented. The other two arrows represent the prediction process of the trained model, for the SYGGROS and the ISIC 2019 Test Dataset. All three datasets contribute to an independent assessment of the classifier’s performance in the 8 classes of skin conditions.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Workflow for the comparison of the DL Model performance.</figcaption></figure>

The dataflows of Fig. 1 represented by the arrows from their beginning up to the DL Model include the loading and preprocessing procedure of the images. Due to the TF2 efficient tf.data [13] pipeline, the data is consumed and processed in parallel which reduces the execution time of each training step. The ‘online’ image augmentation process is also handled inside the pipeline during the training stage.

The preprocessing and augmentation steps as well as the features of the DL architecture are described below. But first follows the description of the three image datasets along with their special features, on which the image pipeline applies.
## 2.1 Skin Datasets

**ISIC 2019 Training Dataset.** The ISIC 2019 dataset [10, 14, 15] consists of 25,331 dermoscopy images divided into 8 classes: Melanoma (MEL), Melanocytic nevus (NV), Basal cell carcinoma (BCC), Actinic keratosis (AK), Benign keratosis (solar lentigo/seborrheic keratosis/lichen planus-like keratosis) (BKL), Dermatofibroma (DF), Vascular lesion (VASC), Squamous cell carcinoma (SCC). The distribution of classes is given in Table 1.

<table><caption>**Table 1.** Class distribution in the ISIC 2019 training dataset</caption><tr><td>AK 867</td><td>BCC 3323</td></tr><tr><td>BKL 2624</td><td>DF 239</td></tr><tr><td>MEL 4522</td><td>NV 12875</td></tr><tr><td>SCC 628</td><td>VASC 253</td></tr></table>

The ISIC 2019 training dataset show great variation in terms of dimensions, the presence or not of the circle from the dermatoscope, the coloring, as well as additional objects (hairs, rulers, markings). Also, the dataset is accompanied by a metadata file, which provides meta-information about the available images, in terms of age, sex, body area of the nevus, as well as the nevus number (lesion_id). We do not use this information in this work.

An important parameter of the dataset is that there are cases where many images come from the same spot (e.g., with a different angle, different resolution, etc.) or there is an ID conflict between different spots. In this case, the same lesion_id corresponds to multiple rows of the file, with a different image file name (column Image, e.g., ISIC_0000002, etc.). In the analysis we performed, these cases were identified, so that images from the same mole are not separated during the cross-validation process. Notably, there are 5050 lesions having duplicates. As an example, lesion_id BCN_0001728 has 31 duplicate images in the dataset.

**Stratified Group k-fold Validation.** In the training we used 5-fold Cross-Validation with stratification of classes, so that all 5 folds contain -as much as possible- the same ratio of categories with skin diseases. As mentioned earlier, additional logic has been added so that multiple images of the same ID always belong to the same fold [16].

**ISIC 2019 Test Dataset.** The evaluation of the classifier in the ISIC 2019 challenge is done with the test dataset which consists of 8,238 images with an unknown distribution of classes. In addition to the 8 classes of the training dataset, images of moles belonging to other categories are also included (Unknown - UNK). The predicted image classes are submitted to the ISIC 2019 challenge server and the classifier is evaluated with the metric BAC: normalized multi-class accuracy metric (balanced across categories), and statistics are generated with ROC curves and other performance metrics.
**SYGGROS Dataset.** The dataset of dermatoscopic images used for the evaluation of the DL skin classifier is provided by Andreas SYGGROS Hospital and includes 311 dermatoscopic photographs with the corresponding labeling by dermatologists of the hospital. For automated classification, the images have been divided into 8 classes, as follows (Table 2). Regarding the technical description of the dataset, the digital dermatoscopy in the Oncology Department of Andreas SYGGROS Hospital was performed with the Digital Epiluminescence Microscopy system (MoleMaxHD 1.6) until 2017. From the beginning of 2018, the clinical and dermatological images of the suspected pigmented lesions are obtained with the digital imaging system FotoFinder Medicam 1000. The technical characteristics of the Medicam 1000 camera are: Full HD brilliance (CrystalView technology), continuous visual live-zoom with autofocus, suitable for dermatoscopy of skin, scalp and nail lesions, magnification up to 140X, control via a panel on the back of the camera.

<table><caption>**Table 2.** Class distribution in the SYGGROS dataset</caption><tr><td>AK 8</td><td>BCC 37</td></tr><tr><td>BKL 23</td><td>DF 1</td></tr><tr><td>MEL 33</td><td>NV 193</td></tr><tr><td>SCC 9</td><td>VASC 7</td></tr></table>

Additionally, a small number of shots have been taken with a special Nikon D300 dermatoscopic camera. Images from both systems are utilized within the project. All the dermatoscopic images are related to the histological features as they result from the biopsies of the patients that have been performed at Andreas SYGGROS Hospital and have been processed and evaluated by the Pathology/Anatomic Department.

## 2.2 DL Architecture

EfficientNets [17] were used as the basic Deep Learning architecture in our study. EfficientNets achieve top performances while utilizing lighter structures that reduce the complexity of the model. They also greatly facilitate the selection of hyperparameters, as the initial tests can be done in the B0 model, and then, depending on the availability of the GPU hardware, the B1-B7 models can be selected. As the code number of the model increases, so does the depth of the hidden levels, but also the resolution of the images as input. Their effectiveness on the ISIC challenge has been showcased by top performing submissions [18–20].

## 2.3 Image Preprocessing Functions

Image preprocessing is a fundamental step before the training and prediction stage of a classifier, and various techniques have been proposed in the literature [21] towards this scope. In this work, a set of pre-processing techniques frequently found in the ISIC challenges are applied. Image preprocessing functions include:
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Application of dermoscopy circle, nice crop and color constancy.</figcaption></figure>

* Resizing and cropping, where we change the smallest dimension of the image to reach the desired size, while maintaining the ratio, and a cut is made to obtain a square table.

* Nice crop. Dermoscopy images from ISIC 2019 may contain the characteristic dermoscope cycle with numerous variations. This technique is applied so that the circle is exactly cut off, removing unnecessary black areas around the perimeter.

* Random erasing. This technique [22] randomly erases a small area of the image, to reduce overfitting in the classifier.

* Color constancy. This technique [23] is based on normalizing the color of dermoscopic images [24] and attempts to remove differences in images resulting from changes in lighting and cameras.

* Dermoscopy cycle. This technique adds the dermoscope circle to all images so that there is no variability in the images regarding this parameter. The technique, if applied during the model training, should also be applied to the images that are then inserted to predict the class. In Fig. 2 in the 1st column 2 images are displayed in their original form, while in the 2nd column the result after the application of nice_cropping, color constancy and dermoscopy circle. The result is square images of the same resolution ready for input for the training of the model. The application of this technique has the additional advantage that it removes the distortion of angles due to random rotations of the image.
## 2.4 Selected DL Parameters

All training images are converted to resolution 400 × 400 and caching is applied to make the process of importing the training fold faster in each epoch. The final resolution with which the training is done is 300 × 300 as the augmentation random cropping technique is applied as well. The parameter “DROP_CONNECT_RATE” corresponds to the drop-out during the construction of the layers and was set to 0.4, which is also a method to fight overfitting.

BATCH_SIZE is set to 16 images. This variable depends on the selected resolution of the training images as well as the available GPU memory. The training of the model was carried out, in our case, with the use of an NVIDIA 2080 Ti GPU card. We chose CV = 5 to be the number of folds in the cross-validation. Finally, each model is trained, for a maximum of 40 epochs.

The EfficientNet B3 architecture is introduced in our code headless (without the last layers), but by carrying the weights of the trained model in the ImageNet images. The Transfer Learning technique is used [25, 26], and it is set that the weights of the headless model will not be frozen during the training. Next, we add to the sequential model the GlobalMaxPooling layer and a Dense layer with 9 neurons (which are our data classes). The Softmax activation function is selected. Rectified Adam [27] is chosen as the optimizer, with a lookahead option, after many trials for better performance. The categorical cross-entropy is set as the loss function.

Also, the callback early_stop was set during the training, so that if the validation loss metric does not improve for 8 seasons (patience duration), the training process will stop and the weights with which the best value of the metric was observed will be kept.

As previously described, during the CV process, the dataset is divided into training and validation sets. An example of the distribution of the classes in the two sets for one of the 5 folds (which contains in the training set 20.262 images, and in the validation set 5.069 images), is shown in Table 3.

To deal with the class imbalanced dataset, a weight is assigned to each class, which is the inverse of the probability of occurrence in the dataset. As an example, the weights for the 1st fold are MEL: 0.700, NV: 0.246, BCC: 0.953, AK: 3.655, BKL: 1.207, DF: 13.260, VASC: 12.538, SCC: 5.045.
<table><caption>**Table 3.** Class distribution for a training fold</caption><thead><tr><td></td><td>Training</td><td>Validation</td></tr></thead><tbody><tr><td>AK</td><td>693</td><td>174</td></tr><tr><td>BKL</td><td>2099</td><td>525</td></tr><tr><td>MEL</td><td>3617</td><td>905</td></tr><tr><td>SCC</td><td>502</td><td>126</td></tr><tr><td>BCC</td><td>2658</td><td>665</td></tr><tr><td>DF</td><td>191</td><td>48</td></tr><tr><td>NV</td><td>10300</td><td>2575</td></tr><tr><td>VASC</td><td>202</td><td>51</td></tr></tbody></table>

## 2.5 Performance Metrics

The `sklearn.metrics.classification_report` function was used to calculate the classifier performance in the case of CV training in the ISIC 2019 training dataset, as well as in the SYGGROS dataset [28]. The result tables report the metrics: precision, recall and f1-score for each class. If TP, FP, FN are the number of true positives, false positives, false negatives respectively, then:

* Precision = TP / (TP + FP)

* Recall = TP / (TP + FN)

* F1 = the weighted harmonic mean of the precision and recall

Average values are calculated in the following ways: micro, macro and weighted [29]. In the first way, the metrics are calculated by counting the total number of TP, FN, FP from all classes. In the second way, the metric is calculated for each class separately and thus the class imbalance is not considered. Finally, in the third way, the metric for each class is calculated but their participation in the average is weighted by the number of samples of each class.

# 3 Results

## 3.1 Classifier Evaluation by the CV Training Process

Table 4 shows the performance evaluation per class, and overall.
<table><caption>**Table 4.** CV classifiers performance</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>f1-score</th><th>Support</th></tr></thead><tbody><tr><td>MEL</td><td>0.69</td><td>0.59</td><td>0.64</td><td>4522</td></tr><tr><td>NV</td><td>0.86</td><td>0.90</td><td>0.88</td><td>12875</td></tr><tr><td>BCC</td><td>0.72</td><td>0.84</td><td>0.77</td><td>3323</td></tr><tr><td>AK</td><td>0.43</td><td>0.43</td><td>0.43</td><td>867</td></tr><tr><td>BKL</td><td>0.67</td><td>0.58</td><td>0.62</td><td>2624</td></tr><tr><td>DF</td><td>0.51</td><td>0.50</td><td>0.51</td><td>239</td></tr><tr><td>VASC</td><td>0.67</td><td>0.77</td><td>0.72</td><td>253</td></tr><tr><td>SCC</td><td>0.40</td><td>0.30</td><td>0.34</td><td>628</td></tr><tr><td>UNK</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>Micro avg</td><td>0.77</td><td>0.77</td><td>0.77</td><td>25331</td></tr><tr><td>Macro avg</td><td>0.55</td><td>0.55</td><td>0.55</td><td>25331</td></tr><tr><td>Weighted avg</td><td>0.76</td><td>0.77</td><td>0.76</td><td>25331</td></tr></tbody></table>

## 3.2 Classifier Performance in the ISIC 2019 Test Dataset

The performance of the model in the ISIC 2019 Test was Balanced Multiclass Accuracy (BAC): 0.524.

Figure 3 shows the table of metrics per class, as provided by the ISIC 2019 Challenge server [30].

<table><caption>**Fig. 3.** Summary evaluation table in the ISIC 2019 test dataset</caption><thead><tr><th rowspan="2">Category Metrics</th><th rowspan="2">Mean Value ●</th><th colspan="8">Diagnosis Category</th></tr><tr><th>MEL ●</th><th>NV ●</th><th>BCC ●</th><th>AK ●</th><th>BKL ●</th><th>DF ●</th><th>VASC ●</th><th>SCC ●</th><th>UNK ●</th></tr></thead><tbody><tr><td>Integral Metrics<br>AUC ●</td><td>0.891</td><td>0.915</td><td>0.947</td><td>0.934</td><td>0.899</td><td>0.874</td><td>0.967</td><td>0.918</td><td>0.916</td><td>0.649</td></tr><tr><td>AUC, Sens > 80% ●</td><td>0.785</td><td>0.833</td><td>0.906</td><td>0.868</td><td>0.798</td><td>0.747</td><td>0.928</td><td>0.890</td><td>0.826</td><td>0.270</td></tr><tr><td>Average Precision ●</td><td>0.525</td><td>0.747</td><td>0.906</td><td>0.698</td><td>0.330</td><td>0.482</td><td>0.453</td><td>0.512</td><td>0.283</td><td>0.314</td></tr><tr><td>Accuracy</td><td>0.922</td><td>0.895</td><td>0.891</td><td>0.887</td><td>0.944</td><td>0.918</td><td>0.989</td><td>0.988</td><td>0.977</td><td>0.807</td></tr><tr><td>Threshold Metrics ●<br>Sensitivity ●</td><td>0.430</td><td>0.575</td><td>0.838</td><td>0.769</td><td>0.219</td><td>0.340</td><td>0.367</td><td>0.485</td><td>0.268</td><td>0.0120</td></tr><tr><td>Specificity</td><td>0.969</td><td>0.960</td><td>0.917</td><td>0.904</td><td>0.983</td><td>0.973</td><td>0.997</td><td>0.995</td><td>0.992</td><td>0.996</td></tr><tr><td>Dice Coefficient ●</td><td>0.461</td><td>0.648</td><td>0.833</td><td>0.639</td><td>0.285</td><td>0.419</td><td>0.452</td><td>0.521</td><td>0.329</td><td>0.0235</td></tr><tr><td>PPV ●</td><td>0.567</td><td>0.744</td><td>0.829</td><td>0.547</td><td>0.408</td><td>0.547</td><td>0.589</td><td>0.563</td><td>0.429</td><td>0.447</td></tr><tr><td>NPV ●</td><td>0.942</td><td>0.918</td><td>0.922</td><td>0.963</td><td>0.959</td><td>0.940</td><td>0.992</td><td>0.993</td><td>0.984</td><td>0.809</td></tr></tbody></table>
## 3.3 Classifier Performance in SYGGROS Dataset

The model that gave the best results from the 5-CV training at ISIC 2019 was then used to classify the SYGGROS images. Table 5 shows the classifier performance metrics in the SYGGROS Dataset.

<table><caption>**Table 5.** Classifier performance on the SYGGROS Dataset</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>f1-score</th><th>Support</th></tr></thead><tbody><tr><td>MEL</td><td>0.59</td><td>0.73</td><td>0.65</td><td>33</td></tr><tr><td>NV</td><td>0.91</td><td>0.75</td><td>0.82</td><td>193</td></tr><tr><td>BCC</td><td>0.73</td><td>0.65</td><td>0.69</td><td>37</td></tr><tr><td>AK</td><td>0.43</td><td>0.38</td><td>0.40</td><td>8</td></tr><tr><td>BKL</td><td>0.21</td><td>0.57</td><td>0.30</td><td>23</td></tr><tr><td>DF</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1</td></tr><tr><td>VASC</td><td>0.75</td><td>0.43</td><td>0.55</td><td>7</td></tr><tr><td>SCC</td><td>1.00</td><td>0.22</td><td>0.36</td><td>9</td></tr><tr><td>UNK</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0</td></tr><tr><td>Micro avg</td><td>0.68</td><td>0.68</td><td>0.68</td><td>311</td></tr><tr><td>Macro avg</td><td>0.51</td><td>0.41</td><td>0.42</td><td>311</td></tr><tr><td>Weighted avg</td><td>0.78</td><td>0.68</td><td>0.71</td><td>311</td></tr></tbody></table>

Compared to Table 4, in the F1-score column, we observe a correlation of the results, where the performances in the SYGGROS Dataset are slightly lower. In the analysis per class in the metric F1, we observe a performance deviation in the BKL and DF classes and a smaller one in VASC. The DF and VASC classes, however, have only 1 and 7 images in the SYGGROS Dataset respectively.

## 4 Discussion

In this work, we used a single architecture: EfficientNets, without ensemble models. The reason was to keep the complexity of the process relatively simple and focus on the transferability of the results by keeping a common baseline. Ensemble modeling is a common technique for increasing the classification performance and the scalability of EfficientNets makes them suitable candidates towards this end [31]. As was noted in [6], ensembles also alleviate the instability of performance across different datasets. In this work, we wanted to study the behavior of such instabilities and so we opted out from ensembles. A recent review on the types of DL architectures and cases with combinations of multiple techniques can be found in [32].

As previously reported, we study the performance of the model trained in the ISIC 2019 training dataset with the CV training process (F1 CV), the ISIC 2019 test (F1
ISIC) and the SYGGROS Dataset (F1 SYGGROS). Because both the ISIC 2019 training dataset and the SYGGROS dataset are severely class imbalanced, and the largest number of images are in the MEL, NV and BCC classes, Table 6 compares the performance in the metric F1 with only these three classes.

<table><caption>**Table 6.** Model performance comparison F1 metric</caption><thead><tr><th></th><th>CV</th><th>ISIC</th><th>SYGGROS</th><th>Best ISIC</th></tr></thead><tbody><tr><td>MEL</td><td>0.64</td><td>0.64</td><td>0.65</td><td>0.68</td></tr><tr><td>NV</td><td>0.88</td><td>0.83</td><td>0.82</td><td>0.84</td></tr><tr><td>BCC</td><td>0.77</td><td>0.64</td><td>0.69</td><td>0.66</td></tr></tbody></table>

In addition, to be able to compare our model in the ISIC 2019 test dataset with the best performing model of the ISIC 2019 competition, for which the training was done without any additional external data [20], we include in the last column of Table 6 the results also on metric F1 (F1 Best ISIC).

Regarding the model with the best performance at ISIC 2019, its authors [20] report the largest improvement in the metric Normalized Multi-class Accuracy (BAC) from the 5-fold CV training process, using ensemble, by 0.014 points. The best performance in single architecture - se_resnext101 has a value of 0.739 (BAC) and very close is the performance of an efficientnet-B3. However, the authors also cite tests with ensembles that are inferior to the best single models. In the final ISIC 2019 test, they submitted three attempts, which had in the test results a difference of only 0.003 points from each other in the metric BAC, with the best value: 0.607.

From the first three columns of Table 6, we observe - for the metric F1 score - consistent performance of the model for the three categories that had the most dermoscopy images. The most stable behavior is in the melanoma category, while in the other two NV, BCC there is an overestimation in the CV process 6 and 9 mm of F1 respectively. For the other categories in which the available images (either in the ISIC 2019 dataset or in the SYGGROS dataset) are much smaller, the performances, as expected, show greater variations. We also observe an analogous performance, in these three categories (MEL, NV, BCC) with the performance in the ISIC 2019 Test dataset of the best performing model. The highest value F1 is achieved in class NV, while they are similar for MEL and BCC. The NV class is also the most numerous in the ISIC 2019 training dataset, having almost three times more images in this category compared to the immediately smaller MEL. We notice that although during the training of our model, weights were used - apart from online augmentation - to compensate for the class imbalance, still the most populous class gives the best results.

# 5 Conclusions

ISIC Challenges attract many participants and offer an environment where new techniques can be tested based on a common dataset for objective model comparison.
There are, however, many factors that make it difficult to generalize and use these
models in a different/clinical setting. In a competitive environment, multiple techniques
are used in combination to improve even a few millimeters of the final index, without
considering the computational complexity and prediction time. In the ISIC 2019 compe-
tition, the submission of multiple attempts was allowed, so results from different models
are submitted, in the hope that one of the attempts will succeed even slightly better in
the unknown test dataset. This is like shooting in the dark.

Another factor that makes it difficult to directly compare the developed models
is the specific conditions of each competition. In the ISIC 2019 challenge, while the
training dataset had 8 categories, the test dataset contained -as previously mentioned in
Methodology- an additional category - Unknown. Participants tried various ingenious
methods for outlier detection, but the most common is to apply a threshold to the output
of the classifier so that low-value predictions are assigned to the unknown class. This
is the method used in the model of this work, as well as in the model that had the best
estimate in the ISIC 2019 challenge (without the use of external data)) [20]. The use
of this threshold, however, affects the performance metrics in the other 8 classes. In
addition, as the distribution of the ISIC 2019 test dataset classes is unknown, it is no
longer possible to compare models on equal terms. A classifier that performs best in a
large class of the test dataset will perform better than another that is better in other
classes with less representation in the test dataset.

However, apart from the characteristics of the terms of competition and the unknown
distribution of classes in the test dataset, the peculiarities of the provided training dataset
are also important. To ISIC 2019 training dataset is characterized by class imbalance
and duplicate/ID conflict images as mentioned in the Methodology. Both characteristics
affect the performance metrics we receive from the cross-validation process. Although
the performance estimation with the CV method is considered robust, the fold construc-
tion process must consider the class distribution, as we saw in the Methodology, to be
stratified. The existence of multiple images corresponding to the same mole (same ID),
introduces another important consideration in the CV process. If not considered when
creating the folds, it leads to an overly optimistic estimate of performance. A typical
example is the paper [9], where the authors report a Normalized Multi-class Accuracy
(BAC) and an F1 score of 0.91.

To summarize, even when there is a common set of dermoscopy images, factors such
as the characteristics of the specific training dataset, the terms of the competition (e.g.,
the way of comparison/metric, the unknown class distribution in the test dataset, the
existence of a requirement for outlier detection) add also complexity in comparing the
performance of the models. In addition, such competition characteristics make it difficult
to transfer these models to a clinical setting, either due to complexity or performance
instability. In this work, we observed consistent performance of a classifier trained in
the ISIC 2019 training dataset in three different and independent cases, for the skin
categories in which there were over 3,000 images for each class. To avoid over-optimistic
performance during the CV process, we paid special attention to dealing with class
imbalance in the training dataset, and the existence of duplicate/ID collision cases.

**Acknowledgments.** We would like to thank Alexios Zarras MD and Professor Alexander J. Stratigos MD, at the Department of Dermatology-Venereology, University of Athens Medical
School, Andreas Sygros Hospital (Athens, Greece) for providing the SYGGROS Dataset used in this paper, as well as information on the technical specifications and collection procedures of the dataset.

**Funding.** This work was supported by the National Project TRANSITION – Translating the diagnostic complexity of melanoma into rational therapeutic stratification – Hellenic General Secretariat of Research and Technology, [T1EΔΚ-01385] co-funded by the European Union.

## References

1. Skin Cancer Facts & Statistics, https://www.skincancer.org/skin-cancer-information/skin-can-cer-facts. Accessed 08 Mar 2019

2. USCS Data Visualizations. https://gis.cdc.gov/Cancer/USCS/#/AtAGlance/. Accessed 09 Oct 2021

3. Lucius, M., et al.: Deep neural frameworks improve the accuracy of general practitioners in the classification of pigmented skin lesions. Diagnostics **10**, 969 (2020). https://doi.org/10.3390/diagnostics10110969

4. ISIC Archive, https://www.isic-archive.com/#!/topWithHeader/tightContentTop/about/abou tIsicOverview. Accessed 08 Oct 2021

5. ISIC Challenge History, https://www.isic-archive.com/#!/topWithHeader/tightContentTop/ about/isicChallengesHistory. Accessed 09 Oct 2021

6. Valle, E., et al.: Data, depth, and design: learning reliable models for skin lesion analysis. Neurocomputing **383**, 303–313 (2020). https://doi.org/10.1016/j.neucom.2019.12.003

7. Lio, P.A., Nghiem, P.: Interactive Atlas of Dermoscopy: Giuseppe Argenziano, H. Peter Soyer, Vincenzo De Giorgio, Domenico Piccolo, Paolo Carli, Mario Delfino, Angela Ferrari, Rainer Hofmann-Wellenhof, Daniela Massi, Giampiero Mazzocchetti, Massimiliano Scalvenzi, and Ingrid H. Wolfpages. J. Am. Acad. Dermatol. **50**, 807–808 (2004). https://doi.org/10.1016/j. jaad.2003.07.029.ISBN 88–86457–30–8

8. Maron, R.C., et al.: A benchmark for neural network robustness in skin cancer classification. Eur. J. Cancer. **155**, 191–199 (2021). https://doi.org/10.1016/j.ejca.2021.06.047

9. Nahata, H., Singh, S.P.: Deep learning solutions for skin cancer detection and diagnosis. In: Jain, V., Chatterjee, J.M. (eds.) Machine Learning with Health Care Perspective. LAIS, vol. 13, pp. 159–182. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-40850-3_8

10. Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data. **5**, 180161 (2018). https://doi.org/10.1038/sdata.2018.161

11. Zhu, C.-Y., et al.: A deep learning based framework for diagnosing multiple skin diseases in a clinical environment. Front. Med. **8**, 626369 (2021). https://doi.org/10.3389/fmed.2021.626369

12. ISIC 2019, https://challenge2019.isic-archive.com/. Accessed 08 Oct 2021

13. Module: tf.data | TensorFlow Core v2.7.0, https://www.tensorflow.org/api_docs/python/tf/data. Accessed 03 Feb 2022

14. Codella, N.C.F., et al.: Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC). ArXiv171005006 Cs. (2017). https://arxiv.org/abs/1710.05006v3

15. Combalia, M., et al.: BCN20000: Dermoscopic Lesions in the Wild. ArXiv E-Prints. 1908, arXiv:1908.02288 (2019)
16. Wasikowski, J.: Stratified Group k-Fold Cross-Validation. https://www.kaggle.com/jakubw asikowski/stratified-group-k-fold-cross-validation

17. Tan, M., Le, Q.V.: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ArXiv190511946 Cs Stat. (2020). http://arxiv.org/abs/1905.11946

18. Ha, Q., Liu, B., Liu, F.: Identifying Melanoma Images using EfficientNet Ensemble: Winning Solution to the SIIM-ISIC Melanoma Classification Challenge. ArXiv201005351 Cs. (2020). http://arxiv.org/abs/2010.05351

19. Gessert, N., Nielsen, M., Shaikh, M., Werner, R., Schlaefer, A.: Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7, 100864 (2020). https://doi.org/10.1016/j.mex.2020.100864

20. Zhou, S., Zhuang, Y., Meng, R.: Multi-Category Skin Lesion Diagnosis Using Dermoscopy Images and Deep CNN Ensembles (2019). https://challenge.isic-archive.com/leaderboards/ 2019/

21. Kontogianni, G., Maglogiannis, I.: A review on state-of-the-art computer-based approaches for the early recognition of malignant melanoma. In: Maglogiannis, I., Brahnam, S., Jain, L.C. (eds.) Advanced Computational Intelligence in Healthcare-7. SCI, vol. 891, pp. 81–101. Springer, Heidelberg (2020). https://doi.org/10.1007/978-3-662-61114-2_6

22. Random-Erasing-tensorflow, https://github.com/uranusx86/Random-Erasing-tensorflow/blob/master/random_erasing.py

23. Shawn, N.: Shades of Grey

24. Barata, C., Celebi, M.E., Marques, J.S.: Improving dermoscopy image classification using color constancy. IEEE J. Biomed. Health Inform. 19, 1146–1152 (2015). https://doi.org/10.1109/JBHI.2014.2336473

25. Delibasis, K., Georgakopoulos, S.V., Tasoulis, S.K., Maglogiannis, I., Plagianakos, V.P.: On image prefiltering for skin lesion characterization utilizing deep transfer learning. In: Iliadis, L., Angelov, P.P., Jayne, C., Pimenidis, E. (eds.) EANN 2020. PINNS, vol. 2, pp. 377–388. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-48791-1_29

26. Georgakopoulos, S.V., Kottari, K., Delibasis, K., Plagianakos, V.P., Maglogiannis, I.: Detection of malignant melanomas in dermoscopic images using convolutional neural network with transfer learning. In: Boracchi, G., Iliadis, L., Jayne, C., Likas, A. (eds.) EANN 2017. CCIS, vol. 744, pp. 404–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65172-9_34

27. tfa.optimizersRectifiedAdamTensorFlow Addons, https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam. Accessed 08 Oct 2021

28. sklearn.metrics.classification_report, https://scikit-learn/stable/modules/generated/sklearn.metrics.classification_report.html. Accessed 08 Oct 2021

29. sklearn.metrics.precision_recallovscore_support. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recallovscore_support.html. Accessed 08 Oct 2021

30. ISIC Challenge 2019 Leaderboard - Lesion Diagnosis - Images Only, https://challenge.isic-archive.com/leaderboards/2019/. Accessed 12 Oct 2021

31. Tziomaka, M., Maglogiannis, I.: Ensembles of deep convolutional neural networks for detecting melanoma in dermoscopy images. In: Nguyen, N.T., Iliadis, L., Maglogiannis, I., Trawiński, B. (eds.) ICCCI 2021. LNCS (LNAI), vol. 12876, pp. 523–535. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88081-1_39

32. Popescu, D., El-Khatib, M., El-Khatib, H., Ichim, L.: New trends in melanoma detection using neural networks: a systematic review. Sensors 22, 496 (2022). https://doi.org/10.3390/s22020496