Package: photomakerv
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding[Paper]   [Project Page]   [Model Card][🤗 Demo (Realistic)]   [🤗 Demo (Stylization)][Replicate Demo (Realistic)]   [Replicate Demo (Stylization)]If the ID fidelity is not enough for you, please try ourstylization application, you may be pleasantly surprised.Official implementation ofPhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding.🌠Key Features:Rapid customizationwithin seconds, with no additional LoRA training.Ensures impressive ID fidelity, offering diversity, promising text controllability, and high-quality generation.Can serve as anAdapterto collaborate with other Base Models alongside LoRA modules in community.❗❗ Note: If there are any PhotoMaker based resources and applications, please leave them in thediscussionand we will list them in theRelated Resourcessection in README file.
Now we know the implementation ofReplicate,Windows,ComfyUI, andWebUI. Thank you all!🚩New Features/Updates✅ Jan. 20, 2024. Animportantnote: For those GPUs that do not support bfloat16, please changethis linetotorch_dtype = torch.float16, the speed will begreatly improved(1min/img (before) vs. 14s/img (after) on V100). The minimum GPU memory requirement for PhotoMaker is15G.✅ Jan. 15, 2024. We release PhotoMaker.🔥ExamplesRealistic generationPhotoMaker notebook demoStylization generationNote: only change the base model and add the LoRA modules for better stylizationPhotoMaker-Style notebook demo🔧 Dependencies and InstallationPython >= 3.8 (Recommend to useAnacondaorMiniconda)PyTorch >= 2.0.0condacreate--namephotomakerpython=3.10
condaactivatephotomaker
pipinstall-Upip# Install requirementspipinstall-rrequirements.txt# Install photomakerpipinstallgit+https://github.com/TencentARC/PhotoMaker.gitThen you can run the following command to use itfromphotomakerimportPhotoMakerStableDiffusionXLPipeline⏬ Download ModelsThe model will be automatically downloaded through the following two lines:fromhuggingface_hubimporthf_hub_downloadphotomaker_path=hf_hub_download(repo_id="TencentARC/PhotoMaker",filename="photomaker-v1.bin",repo_type="model")You can also choose to download manually from thisurl.💻 How to TestUse likediffusersDependencyimporttorchimportosfromdiffusers.utilsimportload_imagefromdiffusersimportEulerDiscreteSchedulerfromphotomakerimportPhotoMakerStableDiffusionXLPipeline### Load base modelpipe=PhotoMakerStableDiffusionXLPipeline.from_pretrained(base_model_path,# can change to any base model based on SDXLtorch_dtype=torch.bfloat16,use_safetensors=True,variant="fp16").to(device)### Load PhotoMaker checkpointpipe.load_photomaker_adapter(os.path.dirname(photomaker_path),subfolder="",weight_name=os.path.basename(photomaker_path),trigger_word="img"# define the trigger word)pipe.scheduler=EulerDiscreteScheduler.from_config(pipe.scheduler.config)### Also can cooperate with other LoRA modules# pipe.load_lora_weights(os.path.dirname(lora_path), weight_name=lora_model_name, adapter_name="xl_more_art-full")# pipe.set_adapters(["photomaker", "xl_more_art-full"], adapter_weights=[1.0, 0.5])pipe.fuse_lora()Input ID Images### define the input ID imagesinput_folder_name='./examples/newton_man'image_basename_list=os.listdir(input_folder_name)image_path_list=sorted([os.path.join(input_folder_name,basename)forbasenameinimage_basename_list])input_id_images=[]forimage_pathinimage_path_list:input_id_images.append(load_image(image_path))Generation# Note that the trigger word `img` must follow the class word for personalizationprompt="a half-body portrait of a man img wearing the sunglasses in Iron man suit, best quality"negative_prompt="(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth, grayscale"generator=torch.Generator(device=device).manual_seed(42)images=pipe(prompt=prompt,input_id_images=input_id_images,negative_prompt=negative_prompt,num_images_per_prompt=1,num_inference_steps=num_steps,start_merge_step=10,generator=generator,).images[0]gen_images.save('out_photomaker.png')Start a local gradio demoRun the following command:pythongradio_demo/app.pyYou could customize this script inthis file.If you want to run it on MAC, you should followthis Instructionand then run the app.py.Usage Tips:Upload more photos of the person to be customized to improve ID fidelity. If the input is Asian face(s), maybe consider adding 'Asian' before the class word, e.g.,Asian woman imgWhen stylizing, does the generated face look too realistic? Adjust the Style strength to 30-50, the larger the number, the less ID fidelity, but the stylization ability will be better. You could also try out other base models or LoRAs with good stylization effects.Reduce the number of generated images and sampling steps for faster speed. However, please keep in mind that reducing the sampling steps may compromise the ID fidelity.Related ResourcesReplicate demo of PhotoMaker:Demo link, run PhotoMaker on replicate.Demo link (style version).Windows version of PhotoMaker:bmaltais/PhotoMakerby@bmaltais, easy to deploy PhotoMaker on Windows. The description can be found inthis link.sdbds/PhotoMaker-for-windowsby@sdbds.ComfyUI:https://github.com/ZHO-ZHO-ZHO/ComfyUI-PhotoMakerhttps://github.com/StartHua/Comfyui-Mine-PhotoMakerhttps://github.com/shiimizu/ComfyUI-PhotoMakerGraido demo in 45 linesProvided by@Gradio🤗 AcknowledgementsPhotoMaker is co-hosted by Tencent ARC Lab and Nankai UniversityMCG-NKU.Inspired from many excellent demos and repos, includingIP-Adapter,multimodalart/Ip-Adapter-FaceID,FastComposer, andT2I-Adapter. Thanks for their great work!Thanks to the Venus team in Tencent PCG for their feedback and suggestions.Thanks to the HuggingFace team for their generous support!DisclaimerThis project strives to impact the domain of AI-driven image generation positively. Users are granted the freedom to create images using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.BibTeXIf you find PhotoMaker useful for your research and applications, please cite using this BibTeX:@article{li2023photomaker,title={PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding},author={Li, Zhen and Cao, Mingdeng and Wang, Xintao and Qi, Zhongang and Cheng, Ming-Ming and Shan, Ying},booktitle={arXiv preprint arxiv:2312.04461},year={2023}}