You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Thumbs up Gesture & Facial concept training for Stable diffusion

In this excercise, the Stable diffusion 1.5 model was fine-tuned using DreamBooth + LoRA technique. Code development was done through jupyter notebooks running on a desktop computer running RTX 3070. Working model demo was created on Google colab.

Overall approach

Images data corresponding to Thumbs Up gesture, Facial concept & Regularization were retrieved using clip retrival from LAION5B-H-14 index and also Google image search.
Images were pre-processed and filtered for dimension (> 512px * 512px) and visual content (face visibility, hand gestures and hand-elbow consistency). Four pre-trained tflite models from MediaPipe were used for the purpose
Two separate LoRA models corresponding to Facial concept & Regularization were trained using Kohya's GUI for Stable diffusion training
The two LoRA models were initially evaluated separately. Script based evaluation was done for Thumbs up gesture concept and Facial concept learning was assessed based on visual inspection.
The prompts used to generate images were man thumbs up and tcr for gesture and facial concepts resepectively.
Script based evaluation computes the following metrics on a dataset consisting of 100 images generated randomly by the model :
- % of images with face detected
- % of images with wrists matching body
- % of images without excess hands
- % of images passing all three above checks
The two LoRA models and base model were fused to create final for ease of deployment. While a better approach was to deploy LoRA models and base models individually and modify prompts, this could not be attempted due to time constraints. Gesture model was given 2% weight & Facial concept model was given 98% weight by fusing the two together.
The weights were determined through visual examination. After model was fused, 100 randomly generated images for the prompt tcr thumbs up were assessed for image metrics.
The models were pushed to HuggingFace Hub and project demo created using Colab notebook

Notes:

To reduce size of the repo, only the evaluation images have been retained in the repo
'images/eval_images/'

Downloads last month: -