Skin Malignancy Classification Using Patients’ Skin Images and Meta-data: Multimodal Fusion for Improving Fairness

Authors: ke  wang
ningyuan  shan
henry  gouk
iris szu-szu  ho
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 19:43:26

Abstract: Skin cancer image classification across skin tones is a challenging problem due to the fact that skin cancer can present differently on different skin tones. This study evaluates the performance of image only models and fusion models in skin malignancy classification. The fusion models we consider are able to take in additional patient data, such as an indicator of their skin tone, and merge this information with the features provided by the image-only model. Results from the experiment show that fusion models perform substantially better than image-only models. In particular, we find that a form of multiplicative fusion results in the best performing models. This finding suggests that skin tones add predictive value in skin malignancy prediction problems. We further demonstrate that feature fusion methods reduce, but do not entirely eliminate, the disparity in performance of the model on patients with different skin tones.

Main paper: 
 introduction: 
 Skin cancer is one of the top five most common cancers in the United Kingdom (UK), with around 17,845 new cases in 2019 (World Cancer Research Fund, 2019) . Of all of the skin cancer types, around 5% of skin cancers are melanoma skin cancer and the other 95% include the Basal Cell Carcinoma (BCC) and Squamous Cell Carcinoma (SCC) and others (Diepgen and Mahler, 2002) . Early diagnosis of a skin cancer improves patient outcomes and is correlated with over 90% survival (Cancer Research UK, 2015) . However, once disease progresses beyond the skin, survival is poor, falling to 32% when it metastasizes to distant organs (American Cancer Society, 2023) . This highlights the importance of early skin cancer screenings and detection.Most of the computer aided techniques for skin cancer diagnosis are focused on analyzing images of skin lesions (Xu et al., 1999; Jain et al., 2015) . Convolutional neural network (CNN), for example, has been applied successfully to various tasks in skin cancer diagnosis using Dermoscopy or clinical images (Refianti et al., 2019; Zhang et al., 2020; Dorj et al., 2018; Mijwil, 2021) . Other trained models that have been used for skin cancer image classification include Artificial Neural Networks (ANNs) (Jaleel et al., 2012; Kanimozhi and Murthi, 2016) and Long Short Term Memory (LSTM) (Mahum and Aladhadh, 2022; Yunandar and Irianto, 2022) .Unlike unimodal learning, multimodal learning is a relatively new field in skin cancer diagnostics. Multimodal learning involves the integration of multiple modalities, such as image data and medical records, to create more accurate predictions and diagnoses (Poria et al., 2017) . Prior study has shown that multimodal fusion can facilitate fair and unbiased medical diagnostics by leveraging multiple sources of information (Zhou et al., 2021) . Fusion is especially important in skin malignancy classification because skin cancer affects people of all skin types and colors, but certain groups, such as people with darker skin tones or manifestations of non-melanoma skin cancer, are often underdiagnosed or diagnosed later in the disease progression (Gloster Jr and Neal, 2006) . To ensure fairness and improve model performance, it is critical to incorporate a diverse dataset that accurately reflects the population being studied and also with consideration for privacy concerns (Yan et al., 2023) . Therefore, the research questions of this study aim to address are:1. whether patient skin tone can add predictive value to a model that could, otherwise, only accommodate images?2. which models are effective in predicting skin maglignancy equally between the three skin tone groups?3. which models minimize the gap in performance across the three skin tone groups? 
related work: 
 In the field of dermatological research, image-only models have been at the forefront of addressing biases associated with skin tones. a prior study by Correa-Medero et al. (2023) exemplified this approach by relying on image data, employing adversarial training to mitigate bias in skin color recognition-a method that significantly contributes to the ongoing efforts to refine bias mitigation in image-based skin cancer diagnosis models. Similarly, the works by Oguguo et al. (2023) ; Du et al. (2022) show the importance of visual data in fostering fairness, specifically targeting skin tone disparities. These efforts are complemented by Kinyanjui et al. (2020) , who, while focusing on skin tone classification in the context of skin diseases, highlighted the critical role of skin color in dermatological studies.Building upon these image-centric methodologies, the exploration of fusion techniques represents a promising avenue for enhancing model performance and fairness. To our knowledge, little attention has been paid to fuse patients' skin tones into a skin malignancy classification model. Unlike skin tones, age and sex were commonly used as input features, whereas previous research established that fusing age and sex into a model did not perform as good as an image-only model (Höhn et al., 2021) . The existence of bias in skin cancer classification remains for people with different skin tones (Yuan et al., 2022) . The use of fusion methods has been reported to be effective in addressing fairness in classification tasks (Zhou et al., 2021) . In skin cancer research, late-fusion techniques are commonly adopted, which fuse the separately extracted data at the last stage of the network (Ge et al., 2017; Yap et al., 2018; Kawahara et al., 2018; Singh et al., 2022) . The late-fusion approaches tend to ignore the potential correlation in mixed feature space. To overcome the drawback of late fusion, early fusion can address some of the limitations of late fusion. In early fusion, the raw inputs from different modalities are combined to learn a joint representation of the different input modalities, which could improve the performance of a model by capturing complementary information from different sources (Heiliger et al., 2022; Vielzeuf et al., 2018) . This can be exemplified by Chen et al. (2022) work where they proposed the MDFNet that used a combination of feature-level and decision-level fusion techniques to integrate clinical data into image data, and the fusion model achieved 80% accuracy (Chen et al., 2022) . Building on prior work, the use of early fusion methods in this study demonstrates an approach to addressing fairness in classification models, and highlights the importance of considering the impact of different input features and modalities on the performance of the models. 
methodology: 
 There were three models involved in this study: the Image-only Model (Figure 1A ), the Feature Fusion Model (Figure 1B ), and the Learning Feature Fusion Model (Figure 1C ). These models were built on pre-trained architectures and utilized a common set of hyperparameters during training to ensure a fair performance assessment (Table A2 ). In each of the three models, there are two main components: feature extraction and classification.• Feature Extraction: This component involved extracting relevant features from the input data, which in this case are skin lesion images. In all three models, feature extraction was performed by a pre-trained model. The selection of pre-trained models are discussed in Model Training Process.• Classification: Once the features were extracted, the next step was classification, where the models made predictions about skin malignancy. The sigmoid classifier learned patterns in the features extracted by the pre-trained model and predicted the class label (malignant vs benign) for each input.In our image-only model (Figure 1A ), the preprocessed images were used as input for feature extraction using a pre-trained model, which was then converted into a one-dimensional vector for classification. The initial and subsequent linear layers served to reduce the dimensionality of the feature space, first to 1024, then to 512. This reduction in dimensionality helped improve computational efficiency and aided in capturing essential information while discarding potentially less relevant features. To capture complex data relationships, the architecture incorporated non-linearity by applying Rectified Linear Unit (ReLU) activation functions after each linear layer (Agarap, 2018) . Additionally, to address overfitting, dropout layers with a dropout rate of 0.25 were added after each ReLU activation function. By randomly dropping out units, it helped prevent the model from relying too heavily on specific features or relationships, thus promoting more robust and generalized learning (Garbin et al., 2020) . In the final stage of the architecture, the classifier consolidated the 512-dimensional feature space into a single output node. This simplified the classification The preprocessed image is represented as a three-dimensional tensor with dimensions NxNx3. The value of N depends on the specific pre-trained model and is determined based on the architecture and design choices of the model. The exact values of N can be found in Table A2 . The extracted features are then transformed into a 1-dimensional vector of length M, with M being contingent on the inherent parameters of the applied pre-trained model.The architectures of the image-only, feature fusion, and learned feature fusion models task to a binary decision: malignancy or non-malignancy. The output node represented the probability of the predicted malignancy classification.Two fusion methods were used for this study: Feature fusion (Figure 1B ) and Learned feature fusion (Figure 1C ). In feature fusion, the features from image modality were extracted using pre-trained models. The image feature vector was combined with patients characteristics using concatenation. The combined features were then passed through the rest of the model for classification. In contrast, in learned feature fusion, a neural network was trained to extract features from image data and patients' metadata separately, and then combined them into a fused representation that captured the most important information from each source (Huang et al., 2020; Holste et al., 2021) . One advantage of learned feature fusion is that it allowed the model to capture complex relationships between different sources of data (Huang et al., 2020) . To understand how different fusion operations impacted predictive performance, we experimented with three fusion operations-concatenation, elemenwise addition and elementwise multiplication in the learned feature models, more details could be seen in Appendix B. 
dataset and baseline: 
 Despite the importance of fairness in medical diagnostic, there remains a paucity of evidence on incorporating patient skin tones into malignancy classification. This motivated us to explore its potential impact. We used the Diverse Dermatology Images (DDI) dataset created by Daneshjou et al. (2022) (Daneshjou et al., 2022) for our experiments -a publicly available and pathologically confirmed image dataset with diverse skin tones. More detailed informaiton about the dataset can be seen in Appendix C.In the work by Daneshjou et al.(2022) (Daneshjou et al., 2022) , they used the DDI dataset to train an image-only model with a pre-trained ResNet model 1 2 . It's interesting to note that their model did not incorporate skin tone information yet achieved AUCs (Area Under the ROC Curve) ranging from 50% to 72%. The model by Daneshjou et al.( 2022) (Daneshjou et al., 2022) was chosen as the baseline in our study, and our objective is to improve predictive performance and enhance fairness by considering different skin tones.For a comprehensive comparison of our results with other state-of-the-art methods using the DDI dataset, please refer to the summary table in the appendix (Table A1 ).The data were split into two sets: train and test. The train set consists of 590 images, while the test set contains 66 images. Ten-fold cross-validation was performed on the training set to evaluate the performance of the model. We used a 1:1:1 ratio of the three skin tone types to ensure that each skin tone type was represented equally in the test set (22 images per skin tone type). This can help to reduce the risk of bias and ensure fairness in the evaluation (Ricci Lara et al., 2022) .Before training, we resized all images to a consistent input size and normalized them using ImageNet's mean and standard deviation (Gouda et al., 2022; Pal and Sudeep, 2016) .We applied data augmentation techniques, such as horizontal flip, random contrast adjustment, elastic transform, random scaling, and rotation to the cross-validation sets (Bloice et al., 2017) . In the testing phase, we employed Test-Time Augmentation (TTA) with similar methods-horizontal flips, random contrast adjustments, and rotations-to mirror the training conditions. This approach not only ensured model evaluation consistency but also helped to improve adaptability to new, unseen image variations (Shanmugam et al., 2021; Gonzalo-Martín et al., 2021) . As for the clinical data, the two categorical variables (skin tone and cancer type) were one hot encoded to convert them into numerical values that would be used for fusion.Since there were no pre-trained models trained specifically based on skin datasets, researchers often used popular pre-trained models to extract features from skin images. In this study, we experimented with a variety of pre-trained feature extractors, including Inception-V3 (Szegedy et al., 2016) , ResNet-50/101/152 (He et al., 2016) , DenseNet-121/161/169/201 (Huang et al., 2017) , MobileNetV3-large/small (Howard et al., 2019) , and EfficientNet-B5/B6/B7 (Melas-Kyriazi, 2023). Of image-only models, we trained the pre-trained models on the image data (1) without pre-trained weights, (2) with pre-trained weights, and (3) with frozen weights. Pre-trained models with frozen weights were found to perform better than those with pre-trained weights (unfrozen) and without pre-trained weights. Hence, in the following experiments, we focus on the models with frozen weights, particularly within 2 ). These image features were then combined with patient characteristics and fed into three neural network architectures for further classification.During model training, we used label smoothing and early stopping to address class imbalance and prevent overfitting (Lukasik et al., 2020) . Label smoothing was used in the experiments to deal with imbalanced class (malignant: 171 cases vs benign: 485 cases). In label smoothing, instead of setting the target value for the correct class to 1 and all others to 0, a small positive value, usually around 0.1, was assigned to the negative class, and the target value for the true class was slightly reduced by the same amount. This encouraged the model to consider all classes equally and prevent overfitting to the majority class (Müller et al., 2019) . Early stopping was used in the experiments to prevent overfitting and improve the generalization performance of the model (Prechelt, 2012) , setting an upper training limit of 1000 epochs. However, the actual stopping varied, ranging from 40 to 100 epochs. For implementation details, see Table A2 .To assess the overall discriminatory power of the models, the Area Under the Receiver Operating Characteristic (ROC) Curve, or AUC, was computed on a bootstrapped test set with 5000 iterations (Robin et al., 2011) . The metric used to assess fairness of a model in this study is performance gap, which was calculated by taking the difference between the maximum and minimum AUC among the three skin tone datasets for each model (Ricci Lara et al., 2022) . This metric quantifies the degree of bias and illuminates potential areas where model performance can be enhanced. In addition, the two aforesaid non-parametric statistical tests (the Wilcoxon and Friedman tests) were conducted to determine whether the differences in AUC values across skin tone groups were statistically significant or occurred by chance. By combining the measurement of performance gaps with a statistical test, it helped to gain insights into both the magnitude of the disparities and their statistical significance. 
performance of image-only and fusion models: 
 Of all image-only models, the models with EfficientNet-B5 and Inception-V3 had the best performance, with AUC 80.7% and 78.3%, respectively (Figure 2 ). The p-value of 0.119 was obtained for the model with EfficientNet-B5 in the Friedman test, and p-value of 0.202 obtained for the models with Inception-V3 and DenseNet-121. The image only model with the best performance (EfficientNet-B5) had the smallest performance gap between the three skin tones datasets (8.3%). This finding indicates that the EfficientNet-B5 image only model is less biased towards people with certain skin tones than Inception-V3 and DenseNet-121 models. By contrast, the largest performance gap (18.5%) was found in the model with Inception-V3. In this study, fusion models (78.7%-92.0%) predicted skin malignancy better than image-only models (73.5%-80.7%) (Figure 2 ). This indicates that fusion models can potentially capture complex relationships between different features that may be missed by image only models alone. However, in the Wilcoxon test, we found that for DenseNet-121 there was no statistical difference between the image only model and fusion models (p-value > 0.05).On the other hand, for EfficientNet-B5, learned feature fusion models significantly outperformed image only models (p-value < 0.05), whereas there was no difference between feature fusion and image only models. For Inception-V3, all fusion models significantly performed better than image only models except for the learned feature fusion model with concatenation operation.Of the fusion models, it was found that learned feature fusion had better predictive performance compared to feature fusion models. This could be explained by the fact that features from the two modalities were used to learn a weight for each feature before combining into a weighted feature vector. This would allow the model to learn the optimal way to combine the features (Huang et al., 2020) . Of all of the models, the best predictive performance was found in learned feature fusion model with EfficientNet-B5 (92.0%). Of the learned feature fusion models with three different fusion operations, the models using elementwise multiplication fusion had better predictive performance than the models using concatenation and elementwise addition. Multiplication fusion operation is particularly useful when the modalities are expected to have a strong interaction or correlation (Liang et al., 2021; Fu et al., 2020; Liu et al., 2018) . Hence, its effectiveness could imply that the image data and patients' meta-data were relevant and complementary.It was noted through experiments that models employing learned feature fusion, particularly multiplication fusion, generally exhibited a smaller gap compared to image-only models and feature fusion models (Figure 3 ). The smallest gap was observed in the EfficientNet-B5 model, with a disparity of merely 2.5%. By contrast, the models with DenseNet-121 and Inception-V3 did not yield similar predictive performance between the three skin tones (p-value < 0.05) (Figure 3 ). This reflects the fusion approach used was effective in combining information from the different subgroups without introducing any systematic bias or disproportionately weighting any particular subgroup. 
discussion and conclusion: 
 Our research used the unique DDI skin cancer database that includes skin tone information. We found that incorporating skin tone significantly enhanced model performance, showing skin color's vital role in predicting malignancies. While the fusion models improved performance, they did not eliminate disparities across skin tones, pointing to the need for further research for equitable skin cancer classification. In addition, employing "frozen weights" in our models led to quicker and more precise learning, particularly with a limited dataset. This is consistent with prior findings that fixed pre-trained weights are beneficial under data scarcity (Hinterstoisser et al., 2018) . For fair model comparison, we used a uniform set of hyperparameters, offering clear insights into fusion strategies' impact on fairness and performance, though this uniformity might not represent each model's optimal settings.In our study, we have laid the groundwork for the integration of skin tone information in skin malignancy prediction models. Future research should explore attention-based models with diverse, large datasets to balance performance and fairness, a challenge not pursued here. Implementing interpretability techniques, like feature importance analysis, could improve transparency and equity by revealing biases. Moreover, future experiments could expand the analysis to include various age groups (Rivers et al., 1989) , gender diversity (Dao and Kazin, 2007) , in addition to various skin tones, given our dataset's limited metadata, leading to a more comprehensive understanding of skin malignancies.To summarize, this study represents an advancement in the field of skin malignancy prediction by being the first to combine skin color information with image data through a multimodal fusion approach. While the approach has proven effective in mitigating disparities and reducing bias in this study, future research is needed to delve deeper into the interpretability of the model (Kaur et al., 2020) , and further explore the interaction between different data modalities. Clinical images and metadata were extracted from medical records. Skin lesions were taken using clinically-issued cameras. In the dataset, each patient has one image along with metadata (skin tones, clinical diagnosis, malignancy, and type of skin cancers). Malignancy is the binary target variable (malignant: 171 cases vs benign: 485 cases). Clinical diagnosis was excluded from the model training due to its strong association with the target variable 'malignancy'. As discussed earlier in the introduction, skin tones and type of skin cancers could potentially confound the classification of malignancy, and hence these patient features were included in our analyses. The skin tone variable has three categories (light: 208 cases, light brown: 241 cases and dark: 207 cases). Type of skin cancer variable consists of two categories: melanoma and non-melanoma. 
 
 Appendix:  


Decision: The paper explores how adding skin tone affects skin cancer classification. The reviewers agreed on the clarity of the paper, and useful insights being presented. Several reviewers mentioned lack of methodological innovation. Some concerns to address were generalizability of the findings, missing baseline experiments to provide insights, and engaging with related literature. After the revision period, some of these concerns were addressed, leading to improved scores for some of the reviewers. 

I believe the paper could still be stronger by using more datasets and using relevant baselines. Beyond what the reviewers have discussed, I would also recommend the authors to look into the problem of “shortcuts” which could be influencing the results.  Overall I think the research is still worth discussing at the conference so I would cautiously recommend acceptance for the version present on OpenReview*. 

*The authors wanted to do a further update of the paper after the deadline had passed and upload was no longer possible on OpenReview.

Reviews: review: paper_summary: paper_summary: The authors have studied the impact of multimodal fusion on model fairness on a public skin cancer dataset. The authors conducted comprehensive experiment to address the research questions on feature fusion strategies and model fairness.
main_review: main_review: Many thanks to the authors for addressing most of my concerns. Despite limited methodological innovation, this paper could provide valuable insight into feature fusion methods in the context of skin malignancy classification.
strength_weakness: strength_weakness: - Limited methodological innovation.
- The authors did not consider attention mechanisms when evaluating representative feature fusion methods, making the work less thorough. 
-  It remains a question whether the findings of this work is generalizable. The findings about the effect of skin tone is only evaluated on one skin cancer dataset. The insights about feature fusion is only assessed in the context of skin cancer.
questions: questions: see above.
paper_summary: paper_summary: The authors focus on the classification of skin malignancy in the context of the samples depicting skin of different tones. The authors evaluate different methods of fusing the information on skin tone to reduce the heterogeneity of the performances in the skin-tone subgroups. Several architectures are evaluated and the AUC scores per subgroup are reported. The significance test is performed in all the experiments. The reported results show the homogeneity of the results across sub-groups illustrating the interest of the method.
main_review: main_review: I thank the authors for providing a feedback on the review. 

While there are still some concerns with regard to the contribution of different labels during the training, I believe the paper is a fine conference material.
strength_weakness: strength_weakness: There are two major weaknesses. First, the authors propose a method of fusing images with both the skin type and cancer type labels. This raises the question of the contribution of each of the label. That is, it is not clear, whether the skin tone alone allows to achieve the same level of performance if no cancer type is provided and vice versa.

Second, the literature related to skin-tone unfairness is worth more in-depth discussion. As this remains the main target of the paper, it would be useful if other methods and practices could be presented.
questions: questions: I would like the authors to address the major weaknesses described above.
paper_summary: paper_summary: The authors present a study on the effect of using additional patient data, such as an indicator of their skin tone, to enhance the performance of the image-only models trained on dermatology images. Their findings suggest that, image only models conditioned using patient information (particularly skin color information) through a multiplication fusion strategy performs substantially better than image only models.
main_review: main_review: The paper presents a technically simple yet useful approach to show how skin tone information can be incorporated to the diagnostic classification problem. 
It is necessary to see a few more experiments to understand the effectiveness of the proposed
strength_weakness: strength_weakness: - Technical novelty is limited as all these fusion models are known to be effective strategies to boost the performance of classification models. 
- Design of the cross validation sets can be presented in more detail.
- A key experiment is missing.
questions: questions: - Comparisons against 3 independent models for each the 3 FST groups in DDI.

Comments: title: Follow-up discussion
comment: Dear authors,

The deadline for revising for papers has passed now so I don't think you can update the paper anymore, furthermore the other reviewers will not be able to review your changes.
Dear chairs at MIDL,
 
We greatly appreciate all reviewers' valuable feedback. In response, we have incorporated additional content as suggested. Unfortunately, this resulted in exceeding the 8-page limit. To comply, we've moved some detailed information to the appendix. Our latest version now meets the 8-page requirement. However, currently we are unable to upload our document. Can we ask if it is possible to submit the latest paper to the system again, please? We are very sorry. 
 
 
Kind regards,
Authors
Many thanks to the authors for addressing most of my concerns.
Thank you for clearing this up, I am able to see your comments now and I think the reviewers should be able to also. I will update my post.
Dear chairs at MIDL,

Sorry for any confusion. I did provide our rebuttal.
I believe we need to make the comments visible to everyone? We have changed the setting to make our comments visible to everyone.

Yours sincerely,
Iris
Dear reviewers,

The authors prepared a rebuttal based on your comments (but possibly, it was not visible to you until now). We invite you to reply to their responses and change your ratings if necessary. Please do so before March 27th.
Thank you for your thoughtful review and feedback on our paper. We appreciate the opportunity to address the concerns raised. Please see our comments to the reviewer's feedback as follows: 

Technical Novelty:
We acknowledge the reviewer's observation regarding methodological innovation. Our aim was to apply known effective strategies within a novel context— the use of skin tone information for improving malignancy prediction. We have clarified this aim in the revised manuscript and emphasised the practical significance of our findings in the context of skin cancer detection. 

Cross-validation Design Details and Concerns: 
Our dataset was split into training and test sets, with the training set comprising 590 images and the test set 66 images. To further ensure the integrity of our evaluation process, we employed ten-fold cross-validation within the training set. The training set was randomly divided into 10 smaller sets. The process repeated 10 times, with each of the 10 folds serving as the validation set once, and the remaining 9 folds forming the training set. This methodological choice was particularly helpful when we had a small dataset and it allowed for assessment of the model's performance across multiple subsets of the data, enhancing the stability of our findings. For the composition of the test set, we adopted a deliberate and systematic approach to maintain a balanced representation of the three skin tone types, allocating a 1:1:1 ratio (22 images per skin tone type). This strategic allocation was designed to mitigate the risk of bias in model evaluation by ensuring that each skin tone type was equally represented, thereby promoting fairness in the assessment of model performance across the three skin tones. 

Experiments on each of the three skin tones: 
Given the constraints of our dataset's size, with 208 cases for light, 241 for light brown, and 207 for dark skin tones, training separate models for each group posed a risk of overfitting, particularly due to the architecture of our model (that consists of both a feature extraction part and a classification part with three layers). To mitigate the risk of overfitting, we explored strategies including training without pre-trained weights, with pre-trained weights, and with frozen weights for the feature extraction component. Training separate models on the relatively small subsets (specific to each skin tone) would not allow the models to generalize effectively, primarily due to the limited diversity and number of samples in each skin tone category.
Thank you for your thoughtful review and feedback on our paper. We appreciate the opportunity to address the concerns raised. Please see our comments to the reviewer's feedback as follows: 

Contribution of the Skin Tone and Cancer Type Labels: 
Thank you for your insightful comments. In the development of our algorithm, our objective was to explore the synergistic potential of combining both skin tone and cancer type labels to improve malignancy classification. However, we acknowledge the importance of understanding how the algorithm performs with independent labels of skin tone and cancer type. Due to the scope of our initial study and the constraints we faced, this comprehensive analysis was beyond our immediate capacity. We plan to undertake this suggested exploration as part of our future work. For the current study, we have included all the detailed results of our experiments in the appendix. 

Skin Tone Unfairness in Discussion: 
In the related work section, we have now included the paper by Medero et al. (2023) to contrast their image-only approach with our multi-modal fusion model. While Medero et al. relied solely on image data and employed adversarial training to mitigate bias in skin color recognition, our approach combined metadata with images to provide additional context crucial for accurate classification. Although the work by Kinyanjui et al. (2020) primarily centers on classifying skin tones in the context of skin diseases, we briefly acknowledge their contribution to recognizing the significance of skin tones in dermatological studies. In addition, we have incorporated a table in the appendix summarising the outcomes of our research alongside other studies employing the same DDI dataset, using state-of-the-art methods.  

Data Details: 
We have now added a detailed description of the dataset composition, including the count of images used and the distribution of cancer types across skin tone subgroups.  

Choice of ten-fold cross-validation: 
In our study, we chose ten-fold cross-validation to ensure the reliability of our findings. This decision was based on its effectiveness in reducing estimate variance, providing a more stable evaluation of the model's predictive performance. We recognise the concerns regarding class imbalance and its potential impact on cross-validation results. However, our dataset exhibits only a moderate level of class imbalance, and thus the risk of skewed performance metrics should be mitigated. 

Early Stopping and Implementation Details: 
For a detailed overview of our implementation, please refer to Appendix Table A1. To ensure optimal training for all models, we set an upper limit of 1000 epochs for training. Upon review, we found that the actual stopping point varied across models, ranging from 40 to 100 epochs. this information has now been added in the paper.  

Proof-reading: 
We have proofread the paper to correct the typos and redundancies in citations as advised.
Thank you for your thoughtful review and feedback on our paper. We appreciate the opportunity to address the concerns raised. Please see our comments to the reviewer's feedback as follows: 

Methodology Innovation: 
We acknowledge the reviewer's observation regarding methodological innovation. Our primary focus was on exploring the impact of multimodal fusion on model fairness rather than introducing novel methodologies. We have clarified this aim in the revised manuscript and emphasised the practical significance of our findings in the context of skin malignancy detection. 

Attention Mechanism: 
Attention mechanisms, while powerful, introduce additional layers of complexity and parameters to the model. Our initial experiments suggested that the incorporation of attention mechanisms would necessitate a significantly broader exploration of model architectures and hyperparameters to optimize performance and fairness effectively. We felt this would exceed the scope of our current research and could potentially obscure the primary focus on basic fusion strategies and their direct impacts on fairness. We understand the importance of attention mechanisms in enhancing model performance. Therefore, we have recommended this area as a future research direction, particularly for studies with access to larger and more diverse datasets where the benefits of attention mechanisms can be more thoroughly evaluated. 

Generalizability of Findings: 
We understand the concern regarding the generalizability of our findings. To address this, we have included in the discussion of the revised manuscript that elaborates on the potential applicability of our insights to other datasets and domains, while acknowledging the need for further research to validate these findings across diverse contexts 

Previous Research Comparison (Hohn et al., 2021) :
Thank you for highlighting the importance of comparing our work with Hohn et al. (2021), specifically regarding the integration of age and sex as predictive features. Our study's reliance on a publicly available skin cancer dataset presented a unique constraint: the dataset did not include age and sex information among its shared metadata. This omission limited our ability to directly assess the impact of these demographic features on model fairness and performance in our experiments. Recognising its potential value, we recommend future research in this area, in the discussion section, to use datasets that include a broader range of metadata, including age and sex.  

Table A1: 
We apologise for the oversight and have corrected the missing Table A1 in the supplementary materials. 

Common set of hyper-parameters: 
Using a uniform set of hyperparameters across models allowed us to establish a consistent baseline for comparison. This approach minimises the variability that could arise from model specific optimisations. thereby providing a clearer view of the intrinsic effects of the fusion strategies on model fairness and performance. However, we recognise that different networks may have different optimal hyperparameters, and have included this as a limitation in our study.  

Table for Comparison of Literature: 
We have now incorporated a table in the appendix summarising the outcomes of our research alongside other studies employing the same DDI dataset, using state-of-the-art methods.  

Test-Time Augmentation (TTA): 
Using TTA mirrors the data augmentation techniques used during the training phase, ensuring that the model is evaluated under conditions similar to how it was trained. This consistency helps to maintain the model's performance stability by exposing it to a variety of transformations that it has learned to generalise from during training. Hence, it enabled the model to better adapt to and recognise new and unseen variations of image data.  

Frozen Weights:
By frozen weights, we refer to the technique where the parameters of certain layers in the model are kept constant. This approach does not imply that the entire model's weights are frozen, but rather it applied to the layers associated with feature extraction, which have been pre-trained on a large dataset.  

AUCs: 
Upon further check, it has been confirmed that the AUC percentages detailed in the text are consistent with those depicted in Figure 2. For a comprehensive view of all AUC scores, please see the tables provided in the appendix Table A5. 

Results: 
To address the confusion, we confirm that all results presented in our study, including those mentioned in the text, tables and figures, were derived from the test set. We have now added more detailed information about the dataset in the paper. 

Citation:
We apologise for the oversight regarding the missing citation. The reference has now been added to the paper: "On Pre-Trained Image Features and Synthetic Images for Deep Learning."