Update README.md

README.md

@@ -1,3 +1,283 @@
 ---
 license: apache-2.0
 ---
+---
+license: apache-2.0
+---
+This repository contains a pruned and partially reorganized version of [AniPortrait](https://fudan-generative-vision.github.io/champ/#/).
+
+```
+@misc{wei2024aniportrait,
+      title={AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animations}, 
+      author={Huawei Wei and Zejun Yang and Zhisheng Wang},
+      year={2024},
+      eprint={2403.17694},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+# Usage
+
+## Installation
+
+First, install the AniPortrait package into your python environment. If you're creating a new environment for AniPortrait, be sure you also specify the version of torch you want with CUDA support, or else this will try to run only on CPU.
+
+```sh
+pip install git+https://github.com/painebenjamin/aniportrait.git
+```
+
+Now, you can create the pipeline, automatically pulling the weights from this repository, either as individual models:
+
+```py
+from aniportrait import AniPortraitPipeline
+pipeline = AniPortraitPipeline.from_pretrained(
+  "benjamin-paine/aniportrait",
+  torch_dtype=torch.float16,
+  variant="fp16",
+  device="cuda"
+).to("cuda", dtype=torch.float16)
+```
+
+Or, as a single file:
+
+```py
+from aniportrait import AniPortraitPipeline
+pipeline = AniPortraitPipeline.from_single_file(
+  "benjamin-paine/aniportrait",
+  torch_dtype=torch.float16,
+  variant="fp16",
+  device="cuda"
+).to("cuda", dtype=torch.float16)
+```
+
+The `AniPortraitPipeline` is a mega pipeline, capable of instantiating and executing other pipelines. It provides the following functions:
+
+## Workflows
+
+### img2img
+
+```py
+pipeline.img2img(
+    reference_image: PIL.Image.Image,
+    pose_reference_image: PIL.Image.Image,
+    num_inference_steps: int,
+    guidance_scale: float,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using a reference image (for structure) and a pose reference image (for pose), render an image of the former in the pose of the latter.
+- The pose reference image here is an unprocessed image, from which the face pose will be extracted.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+
+### vid2vid
+
+```py
+pipeline.vid2vid(
+    reference_image: PIL.Image.Image,
+    pose_reference_images: List[PIL.Image.Image],
+    num_inference_steps: int,
+    guidance_scale: float,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    video_length: Optional[int]=None,
+    context_schedule: str="uniform",
+    context_frames: int=16,
+    context_overlap: int=4,
+    context_batch_size: int=1,
+    interpolation_factor: int=1,
+    use_long_video: bool=True,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using a reference image (for structure) and a sequence of pose reference images (for pose), render a video of the former in the poses of the latter, using context windowing for long-video generation when the poses are longer than 16 frames.
+- Optionally pass `use_long_video = false` to disable using the long video pipeline.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+- Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose reference images.
+
+### audio2vid
+
+```py
+pipeline.audio2vid(
+    audio: str,
+    reference_image: PIL.Image.Image,
+    num_inference_steps: int,
+    guidance_scale: float,
+    fps: int=30,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    pose_reference_images: Optional[List[PIL.Image.Image]]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    video_length: Optional[int]=None,
+    context_schedule: str="uniform",
+    context_frames: int=16,
+    context_overlap: int=4,
+    context_batch_size: int=1,
+    interpolation_factor: int=1,
+    use_long_video: bool=True,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using an audio file, draw `fps` face pose images per second for the duration of the audio. Then, using those face pose images, render a video.
+- Optionally include a list of images to extract the poses from prior to merging with audio-generated poses (in essence, pass a video here to control non-speech motion). The default is a moderately active loop of head movement.
+- Optionally pass width/height to modify the size. Defaults to reference image size.
+- Optionally pass `use_long_video = false` to disable using the long video pipeline.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+- Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose reference images.
+
+## Internals/Helpers
+
+### img2pose
+
+```py
+pipeline.img2pose(
+	reference_image: PIL.Image.Image,
+	width: Optional[int]=None,
+	height: Optional[int]=None
+) -> PIL.Image.Image
+```
+
+Detects face landmarks in an image and draws a face pose image.
+- Optionally modify the original width and height.
+
+### vid2pose
+
+```py
+pipeline.vid2pose(
+	reference_image: PIL.Image.Image,
+    retarget_image: Optional[PIL.Image.Image],
+	width: Optional[int]=None,
+	height: Optional[int]=None
+) -> List[PIL.Image.Image]
+```
+
+Detects face landmarks in a series of images and draws pose images.
+- Optionally modify the original width and height.
+- Optionally retarget to a different face position, useful for video-to-video tasks.
+
+### audio2pose
+
+```py
+pipeline.audio2pose(
+    audio_path: str,
+    fps: int=30,
+    reference_image: Optional[PIL.Image.Image]=None,
+    pose_reference_images: Optional[List[PIL.Image.Image]]=None,
+    width: Optional[int]=None,
+	height: Optional[int]=None
+) -> List[PIL.Image.Image]
+```
+
+Using an audio file, draw `fps` face pose images per second for the duration of the audio.
+- Optionally include a reference image to extract the face shape and initial position from. Default has a generic androgynous face shape.
+- Optionally include a list of images to extract the poses from prior to merging with audio-generated poses (in essence, pass a video here to control non-speech motion). The default is a moderately active loop of head movement.
+- Optionally pass width/height to modify the size. Defaults to reference image size, then pose image sizes, then 256.
+
+### pose2img
+
+```py
+pipeline.pose2img(
+    reference_image: PIL.Image.Image,
+    pose_image: PIL.Image.Image,
+    num_inference_steps: int,
+    guidance_scale: float,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using a reference image (for structure) and a pose image (for pose), render an image of the former in the pose of the latter.
+- The pose image here is a processed face pose. To pass a non-processed face pose, see `img2img`.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+
+### pose2vid
+
+```py
+pipeline.pose2vid(
+    reference_image: PIL.Image.Image,
+    pose_images: List[PIL.Image.Image],
+    num_inference_steps: int,
+    guidance_scale: float,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    video_length: Optional[int]=None,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using a reference image (for structure) and pose images (for pose), render a video of the former in the poses of the latter.
+- The pose images here are a processed face poses. To non-processed face poses, see `vid2vid`.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+- Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose images.
+
+### pose2vid_long
+
+```py
+pipeline.pose2vid_long(
+    reference_image: PIL.Image.Image,
+    pose_images: List[PIL.Image.Image],
+    num_inference_steps: int,
+    guidance_scale: float,
+    eta: float=0.0,
+    reference_pose_image: Optional[Image.Image]=None,
+    generation: Optional[Union[torch.Generator, List[torch.Generator]]]=None,
+    output_type: Optional[str]="pil",
+    return_dict: bool=True,
+    callback: Optional[Callable[[int, int, torch.FloatTensor], None]]=None,
+    callback_steps: Optional[int]=None,
+    width: Optional[int]=None,
+    height: Optional[int]=None,
+    video_length: Optional[int]=None,
+    context_schedule: str="uniform",
+    context_frames: int=16,
+    context_overlap: int=4,
+    context_batch_size: int=1,
+    interpolation_factor: int=1,
+    **kwargs: Any
+) -> Pose2VideoPipelineOutput
+```
+
+Using a reference image (for structure) and pose images (for pose), render a video of the former in the poses of the latter, using context windowing for long-video generation.
+- The pose images here are a processed face poses. To non-processed face poses, see `vid2vid`.
+- Optionally pass `reference_pose_image` to designate the pose of `reference_image`. When not passed, the pose of `reference_image` is automatically detected.
+- Optionally pass `video_length` to use this many frames. Default is the same as the length of the pose images.