Package: facechain
FaceChainIntroduction如果您熟悉中文，可以阅读中文版本的README。FaceChain is a deep-learning toolchain for generating your Digital-Twin. With a minimum of 1 portrait-photo, you can create a Digital-Twin of your own and start generating personal portraits in different settings (multiple styles now supported!). You may train your Digital-Twin model and generate photos via FaceChain's Python scripts, or via the familiar Gradio interface. You can also experience FaceChain directly with ourModelScope Studio.FaceChain is powered byModelScope.NewsSupport a series of new style models in a plug-and-play fashion. Refer to:Features(August 16th, 2023 UTC)Support customizable prompts. Refer to:Features(August 16th, 2023 UTC)Colab notebook is available now! You can experience FaceChain directly with ourColab Notebook.   (August 15th, 2023 UTC)To-Do ListSupport existing style models (such as those on Civitai) in a plug-an-play fashion.  --on-goingSupport customizable prompts (try on different outfits etc.)  --on-goingSupport customizable poses, with controlnet or composerSupport more beauty-retouch effectsSupport latest foundation models such as SDXLProvide Colab compatibility   --doneProvide WebUI compatibilityFeaturesSupport a series of new style models in a plug-and-play fashionDescriptionAllow users to select different style models for training distinct types of Digital-Twins.InstallationRefer toInstallation GuideExecutioncdfacechain/advanced-stylepython3app.pyExampled outcomesReferencexiapei lora modelFor more LoRA styles, refer toCivitaiSupport customizable promptsDescriptionAllow users to achieve various portrait styles with customized prompts.InstallationRefer toInstallation GuideExecutioncdfacechain/advanced-promptpython3app.pyExampled outcomes (prompt: wearing an elegant evening gown)InstallationCompatibility VerificationThe following are the environment dependencies that have been verified:python: py3.8, py3.10pytorch: torch2.0.0, torch2.0.1tensorflow: 2.8.0, tensorflow-cpuCUDA: 11.7CUDNN: 8+OS: Ubuntu 20.04, CentOS 7.9GPU: Nvidia-A10 24GResource UsageGPU: About 19GDisk: About 50GBInstallation GuideThe following installation methods are supported:1. ModelScope notebook【recommended】The ModelScope notebook has a free tier that allows you to run the FaceChain application, refer toModelScope NotebookIn addition to ModelScope notebook and ECS, I would suggest that we add that user may also start DSW instance with the option of ModelScope (GPU) image, to create a ready-to-use environment.# Step1: 我的notebook -> PAI-DSW -> GPU环境# Step2: Open the Terminal，clone FaceChain from github:GIT_LFS_SKIP_SMUDGE=1gitclonehttps://github.com/modelscope/facechain.git--depth1# Step3: Entry the Notebook cell:importos
os.chdir('/mnt/workspace/facechain')print(os.getcwd())!pip3installgradio
!python3app.py# Step4: click "public URL" or "local URL", upload your images to# train your own model and then generate your digital twin.2. DockerIf you are familiar with using docker, we recommend to use this way:# Step1: Prepare the environment with GPU on local or cloud, we recommend to use Alibaba Cloud ECS, refer to: https://www.aliyun.com/product/ecs# Step2: Download the docker image (for installing docker engine, refer to https://docs.docker.com/engine/install/）dockerpullregistry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.7.1-py38-torch2.0.1-tf1.15.5-1.8.0# Step3: run the docker containerdockerrun-it--namefacechain-p7860:7860--gpusallregistry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.7.1-py38-torch2.0.1-tf1.15.5-1.8.0/bin/bash(Note:youmayneedtoinstallthenvidia-container-runtime,refertohttps://github.com/NVIDIA/nvidia-container-runtime)# Step4: Install the gradio in the docker container:pip3installgradio# Step5 clone facechain from githubGIT_LFS_SKIP_SMUDGE=1gitclonehttps://github.com/modelscope/facechain.git--depth1cdfacechain
python3app.py# Step6Runtheappserver:click"public URL"-->intheformof:https://xxx.gradio.live3. conda Virtual EnvironmentUse the conda virtual environment, and refer toAnacondato manage your dependencies. After installation, execute the following commands:
(Note: mmcv has strict environment requirements and might not be compatible in some cases. It's recommended to use Docker.)condacreate-nfacechainpython=3.8# Verified environments: 3.8 and 3.10condaactivatefacechainGIT_LFS_SKIP_SMUDGE=1gitclonehttps://github.com/modelscope/facechain.git--depth1cdfacechain

pip3install-rrequirements.txt
pip3install-Uopenmimmiminstallmmcv-full==1.7.0# Navigate to the facechain directory and run:python3app.py# Finally, click on the URL generated in the log to access the web page.Note: After the app service is successfully launched, go to the URL in the log, enter the "Image Customization" tab, click "Select Image to Upload", and choose at least one image with a face. Then, click "Start Training" to begin model training. After the training is completed, there will be corresponding displays in the log. Afterwards, switch to the "Image Experience" tab and click "Start Inference" to generate your own digital image.4. colab notebookPlease refer toColab Notebookfor details.Script ExecutionFaceChain supports direct training and inference in the python environment. Run the following command in the cloned folder to start training:PYTHONPATH=.shtrain_lora.sh"ly261666/cv_portrait_model""v2.0""film/film""./imgs""./processed""./output"Parameter meaning:ly261666/cv_portrait_model: The stable diffusion base model of the ModelScope model hub, which will be used for training, no need to be changed.
v2.0: The version number of this base model, no need to be changed
film/film: This base model may contains multiple subdirectories of different styles, currently we use film/film, no need to be changed
./imgs: This parameter needs to be replaced with the actual value. It means a local file directory that contains the original photos used for training and generation
./processed: The folder of the processed images after preprocessing, this parameter needs to be passed the same value in inference, no need to be changed
./output: The folder where the model weights stored after training, no need to be changedWait for 5-20 minutes to complete the training. Users can also adjust other training hyperparameters. The hyperparameters supported by training can be viewed in the file oftrain_lora.sh, or the complete hyperparameter list infacechain/train_text_to_image_lora.py.When inferring, please edit the code in run_inference.py:# Fill in the folder of the images after preprocessing above, it should be the same as during trainingprocessed_dir='./processed'# The number of images to generate in inferencenum_generate=5# The stable diffusion base model used in training, no need to be changedbase_model='ly261666/cv_portrait_model'# The version number of this base model, no need to be changedrevision='v2.0'# This base model may contains multiple subdirectories of different styles, currently we use film/film, no need to be changedbase_model_sub_dir='film/film'# The folder where the model weights stored after training, it must be the same as during trainingtrain_output_dir='./output'# Specify a folder to save the generated images, this parameter can be modified as neededoutput_dir='./generated'Then execute:pythonrun_inference.pyYou can find the generated personal digital image photos in theoutput_dir.Algorithm IntroductionArchitectural OverviewThe ability of the personal portrait generatation evolves around the text-to-image capability of Stable Diffusion model. We consider the main factors that affect the generation effect of personal portraits: portrait style information and user character information. For this, we use the style LoRA model trained offline and the face LoRA model trained online to learn the above information. LoRA is a fine-tuning model with fewer trainable parameters. In Stable Diffusion, the information of the input image can be injected into the LoRA model by the way of text generation image training with a small amount of input image. Therefore, the ability of the personal portrait model is divided into training and inference stages. The training stage generates image and text label data for fine-tuning the Stable Diffusion model, and obtains the face LoRA model. The inference stage generates personal portrait images based on the face LoRA model and style LoRA model.TrainingInput: User-uploaded images that contain clear face areasOutput: Face LoRA modelDescription: First, we process the user-uploaded images using an image rotation model based on orientation judgment and a face refinement rotation method based on face detection and keypoint models, and obtain images containing forward faces. Next, we use a human body parsing model and a human portrait beautification model to obtain high-quality face training images. Afterwards, we use a face attribute model and a text annotation model, combined with tag post-processing methods, to generate fine-grained labels for training images. Finally, we use the above images and label data to fine-tune the Stable Diffusion model to obtain the face LoRA model.InferenceInput: User-uploaded images in the training phase, preset input prompt words for generating personal portraitsOutput: Personal portrait imageDescription: First, we fuse the weights of the face LoRA model and style LoRA model into the Stable Diffusion model. Next, we use the text generation image function of the Stable Diffusion model to preliminarily generate personal portrait images based on the preset input prompt words. Then we further improve the face details of the above portrait image using the face fusion model. The template face used for fusion is selected from the training images through the face quality evaluation model. Finally, we use the face recognition model to calculate the similarity between the generated portrait image and the template face, and use this to sort the portrait images, and output the personal portrait image that ranks first as the final output result.Model ListThe models used in FaceChain:[1]  Face detection model DamoFD：https://modelscope.cn/models/damo/cv_ddsar_face-detection_iclr23-damofd[2]  Image rotating model, offered in the ModelScope studio[3]  Human parsing model M2FP：https://modelscope.cn/models/damo/cv_resnet101_image-multiple-human-parsing[4]  Skin retouching model ABPN：https://modelscope.cn/models/damo/cv_unet_skin-retouching[5]  Face attribute recognition model FairFace：https://modelscope.cn/models/damo/cv_resnet34_face-attribute-recognition_fairface[6]  DeepDanbooru model：https://github.com/KichangKim/DeepDanbooru[7]  Face quality assessment FQA：https://modelscope.cn/models/damo/cv_manual_face-quality-assessment_fqa[8]  Face fusion model：https://modelscope.cn/models/damo/cv_unet-image-face-fusion_damo[9]  Face recognition model RTS：https://modelscope.cn/models/damo/cv_ir_face-recognition-ood_rtsMore InformationModelScope library​		ModelScope Library provides the foundation for building the model-ecosystem of ModelScope, including the interface and implementation to integrate various models into ModelScope.Contribute models to ModelScopeLicenseThis project is licensed under theApache License (Version 2.0).