How to organize the dataset for multi-label, multi-model classification

username: Manny_Parkinson
cooked: <p>Hello All,<br>
I am working with the <a href="https://stanfordaimi.azurewebsites.net/datasets/35866158-8196-48d8-87bf-50dca81df965" rel="noopener nofollow ugc">Stanford DDI Dataset</a>. Basically, the data contains images of dermatology related diseases classified as either Benign or Malignant. Furthermore, the data is diversified across various skin tones (FST 12 - fair skin, FST 56- dark skin, FST 34 -In Between). Without the skin tone information, the problem is fairly binary classification (Either benign or malignant). However, I want to build a model which consists of two pre-trained models (One trained on FST 12, One trained on FST 56) based on this <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">idea</a> and then have them output the label benign/malignant. I am not sure how to structure the data in folders though. This is what I want to build.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0c48dadf7314b649df08714a487e4a1ac975d8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0c48dadf7314b649df08714a487e4a1ac975d8b" title="Screenshot 2022-12-03 at 5.20.37 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0c48dadf7314b649df08714a487e4a1ac975d8b_2_690x324.png" alt="Screenshot 2022-12-03 at 5.20.37 PM" data-base62-sha1="mWdvyQDfuuEeFsh1KBFv2HZfj5h" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0c48dadf7314b649df08714a487e4a1ac975d8b_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0c48dadf7314b649df08714a487e4a1ac975d8b_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0c48dadf7314b649df08714a487e4a1ac975d8b_2_1380x648.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-12-03 at 5.20.37 PM</span><span class="informations">1530×720 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Any help is appreciated.<br>
Thanks!</p>

username: tiramisuNcustard
cooked: <p>Please take a look at this: <a href="https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html</a></p>
<p>If you end up using ImageFolder then you can create an ensemble of models (similar to your proposed solution) where each model will train/predict using different dataset located in different folders i.e.</p>
<p>FST_12/train/benign_class_images (for training dataset)</p>
<p>FST_12/train/malignant_class_images (for training dataset)<br>
FST_12/test_images</p>
<p>FST_56/train/benign_class_images (for training dataset)<br>
FST_56/train/malignant_class_images (for training dataset)<br>
FST_56/test_images</p>
<p>Depending on how the images are currently named, you can automate the move operations using a python/shell script.</p>

username: Manny_Parkinson
cooked: <p>Hey,<br>
Thanks for the suggestion. I have used the ImageFolder approach and trained the models separately (also while following a similar folder structure as you specified).<br>
I have also saved the state_dicts for these models and I am able to load them in to the ensemble model. Now, I was wondering how I could inject the skin_type info to the outputs of both these models and pass them through some layers before making a final classification?<br>
Thank you</p>

username: tiramisuNcustard
cooked: <p><a class="mention" href="/u/manny_parkinson">@Manny_Parkinson</a>, Sounds like you want to separate the images based on skin_type before predicting benign/malignant. Maybe you can train a model to perform binary classification on skin_type then separate the images into the folder structures mentioned previously and finally use the existing ensemble (the one you already trained) to predict benign/malignant.</p>

username: Manny_Parkinson
cooked: <p><a class="mention" href="/u/tiramisuncustard">@tiramisuNcustard</a>, I don’t necessarily want to separate the images based on skin type. I just want to provide an auxiliary input, (the skin tone) to have some sort of weightage when making a vote. I apologize if I cannot articulate it better. For now, I was able to train two models <strong>individually</strong> based on this directory structure.<br>
12/train/Benign<br>
12/train/Malignant<br>
12/val/Benign<br>
12/val/Malignant<br>
The same for 56 skin tone as well.<br>
What I am struggling at is training both the models <strong>together</strong> while also providing an additional input about the skin tone. The problem arises when thinking about how to structure the data for combined training and how to provide the auxiliary input label (skin tone) as the model trains.<br>
I really really appreciate your patience.</p>

username: tiramisuNcustard
cooked: <p><a class="mention" href="/u/manny_parkinson">@Manny_Parkinson</a>, Sounds like you can differentiate between the skin tone based on the images alone, so I am assuming the images are color images. Models are capable of extracting features from all 3 channels (red, green and blue) of color images. Skin tones will lead to different signatures (i.e. characteristic variation in one or more of these channels) in these images. This means that the information about the skin tone is already present in all the images. I would only create a binary classifier (i.e. benign or malignant) and disregard the skin tone classification as the presence/absence of the disease is the constant to be determined across all skin tones.</p>
<p>If you must, add text annotations on all the images manually before training. Regards.</p>