--- license: apache-2.0 --- This repository contains a pruned and partially reorganized version of [AniPortrait](https://fudan-generative-vision.github.io/champ/#/). ``` @misc{wei2024aniportrait, title={AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animations}, author={Huawei Wei and Zejun Yang and Zhisheng Wang}, year={2024}, eprint={2403.17694}, archivePrefix={arXiv}, primaryClass={cs.CV} } ``` # Usage ## Installation First, install the AniPortrait package into your python environment. If you're creating a new environment for AniPortrait, be sure you also specify the version of torch you want with CUDA support, or else this will try to run only on CPU. ```sh pip install git+https://github.com/painebenjamin/aniportrait.git ``` Now, you can create the pipeline, automatically pulling the weights from this repository, either as individual models: ```py from aniportrait import AniPortraitPipeline pipeline = AniPortraitPipeline.from_pretrained( "benjamin-paine/aniportrait", torch_dtype=torch.float16, variant="fp16", device="cuda" ).to("cuda", dtype=torch.float16) ``` Or, as a single file: ```py from aniportrait import AniPortraitPipeline pipeline = AniPortraitPipeline.from_single_file( "benjamin-paine/aniportrait", torch_dtype=torch.float16, variant="fp16", device="cuda" ).to("cuda", dtype=torch.float16) ``` The `AniPortraitPipeline` is a mega pipeline, capable of instantiating and executing other pipelines. It provides the following functions: ## Workflows ### img2img ```py pipeline.img2img( reference_image: PIL.Image.Image, pose_reference_image: PIL.Image.Image, num_inference_steps: int, guidance_scale: float, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using a reference image (for structure) and a pose reference image (for pose), render an image of the former in the pose of the latter. - The pose reference image here is an unprocessed image, from which the face pose will be extracted. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. ### vid2vid ```py pipeline.vid2vid( reference_image: PIL.Image.Image, pose_reference_images: List[PIL.Image.Image], num_inference_steps: int, guidance_scale: float, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, video_length: Optional[int]=None, context_schedule: str="uniform", context_frames: int=16, context_overlap: int=4, context_batch_size: int=1, interpolation_factor: int=1, use_long_video: bool=True, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using a reference image (for structure) and a sequence of pose reference images (for pose), render a video of the former in the poses of the latter, using context windowing for long-video generation when the poses are longer than 16 frames. - Optionally pass `use_long_video = false` to disable using the long video pipeline. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. - Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose reference images. ### audio2vid ```py pipeline.audio2vid( audio: str, reference_image: PIL.Image.Image, num_inference_steps: int, guidance_scale: float, fps: int=30, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, pose_reference_images: Optional[List[PIL.Image.Image]]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, video_length: Optional[int]=None, context_schedule: str="uniform", context_frames: int=16, context_overlap: int=4, context_batch_size: int=1, interpolation_factor: int=1, use_long_video: bool=True, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using an audio file, draw `fps` face pose images per second for the duration of the audio. Then, using those face pose images, render a video. - Optionally include a list of images to extract the poses from prior to merging with audio-generated poses (in essence, pass a video here to control non-speech motion). The default is a moderately active loop of head movement. - Optionally pass width/height to modify the size. Defaults to reference image size. - Optionally pass `use_long_video = false` to disable using the long video pipeline. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. - Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose reference images. ## Internals/Helpers ### img2pose ```py pipeline.img2pose( reference_image: PIL.Image.Image, width: Optional[int]=None, height: Optional[int]=None ) -> PIL.Image.Image ``` Detects face landmarks in an image and draws a face pose image. - Optionally modify the original width and height. ### vid2pose ```py pipeline.vid2pose( reference_image: PIL.Image.Image, retarget_image: Optional[PIL.Image.Image], width: Optional[int]=None, height: Optional[int]=None ) -> List[PIL.Image.Image] ``` Detects face landmarks in a series of images and draws pose images. - Optionally modify the original width and height. - Optionally retarget to a different face position, useful for video-to-video tasks. ### audio2pose ```py pipeline.audio2pose( audio_path: str, fps: int=30, reference_image: Optional[PIL.Image.Image]=None, pose_reference_images: Optional[List[PIL.Image.Image]]=None, width: Optional[int]=None, height: Optional[int]=None ) -> List[PIL.Image.Image] ``` Using an audio file, draw `fps` face pose images per second for the duration of the audio. - Optionally include a reference image to extract the face shape and initial position from. Default has a generic androgynous face shape. - Optionally include a list of images to extract the poses from prior to merging with audio-generated poses (in essence, pass a video here to control non-speech motion). The default is a moderately active loop of head movement. - Optionally pass width/height to modify the size. Defaults to reference image size, then pose image sizes, then 256. ### pose2img ```py pipeline.pose2img( reference_image: PIL.Image.Image, pose_image: PIL.Image.Image, num_inference_steps: int, guidance_scale: float, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using a reference image (for structure) and a pose image (for pose), render an image of the former in the pose of the latter. - The pose image here is a processed face pose. To pass a non-processed face pose, see `img2img`. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. ### pose2vid ```py pipeline.pose2vid( reference_image: PIL.Image.Image, pose_images: List[PIL.Image.Image], num_inference_steps: int, guidance_scale: float, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, video_length: Optional[int]=None, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using a reference image (for structure) and pose images (for pose), render a video of the former in the poses of the latter. - The pose images here are a processed face poses. To non-processed face poses, see `vid2vid`. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. - Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose images. ### pose2vid_long ```py pipeline.pose2vid_long( reference_image: PIL.Image.Image, pose_images: List[PIL.Image.Image], num_inference_steps: int, guidance_scale: float, eta: float=0.0, reference_pose_image: Optional[Image.Image]=None, generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None, output_type: Optional[str]="pil", return_dict: bool=True, callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None, callback_steps: Optional[int]=None, width: Optional[int]=None, height: Optional[int]=None, video_length: Optional[int]=None, context_schedule: str="uniform", context_frames: int=16, context_overlap: int=4, context_batch_size: int=1, interpolation_factor: int=1, **kwargs: Any ) -> Pose2VideoPipelineOutput ``` Using a reference image (for structure) and pose images (for pose), render a video of the former in the poses of the latter, using context windowing for long-video generation. - The pose images here are a processed face poses. To non-processed face poses, see `vid2vid`. - Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected. - Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose images.