akshan-main/tiny-wan22-vace-modular-pipe

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: Wan22VaceBlocks

Description: Modular pipeline for controllable video generation using Wan2.2 VACE.

This pipeline uses a 5-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. text_encoder (WanTextEncoderStep)
   • Text Encoder step that generate text_embeddings to guide the video generation
2. vace_encoder (WanVaceEncoderStep)
   • Vace Encoder step that preprocesses the control video, mask and reference images and encodes them into the conditioning latents used by the VACE control branch of the transformer
3. denoise (Wan22VaceCoreDenoiseStep)
   • denoise block that takes encoded text and vace conditioning latents and runs the denoising process.
4. trim_latents (WanVaceTrimReferenceLatentsStep)
   • Step that removes the prepended reference image frames from the denoised latents before decoding
5. decode (WanVaeDecoderStep)
   • Step that decodes the denoised latents into images

Model Components

1. text_encoder (UMT5EncoderModel)
2. tokenizer (AutoTokenizer)
3. guider (ClassifierFreeGuidance)
4. transformer (WanVACETransformer3DModel)
5. vae (AutoencoderKLWan)
6. video_processor (VideoProcessor)
7. scheduler (UniPCMultistepScheduler)
8. guider_2 (ClassifierFreeGuidance)
9. transformer_2 (WanVACETransformer3DModel)

Configuration Parameters

boundary_ratio (default: 0.875): The boundary ratio to divide the denoising loop into high noise and low noise stages.

Input/Output Specification

Inputs:

• prompt (None, optional): No description provided
• negative_prompt (None, optional): No description provided
• max_sequence_length (None, optional, defaults to 512): No description provided
• video (list, optional): The control video to condition the generation on. If not provided, an empty video is used.
• mask (list, optional): The mask that defines which video regions to condition on (black) and which to generate (white). Can only be passed if video is passed as well.
• reference_images (Image | list, optional): One or more reference images as extra conditioning for the generation.
• conditioning_scale (float | list | Tensor, optional, defaults to 1.0): The conditioning scale applied in each control layer of the model. If a float, it is applied uniformly to all layers; a list or tensor must have the same length as the number of control layers.
• height (None, optional): No description provided
• width (None, optional): No description provided
• num_frames (int, optional, defaults to 81): No description provided
• generator (None, optional): No description provided
• num_videos_per_prompt (None, optional, defaults to 1): No description provided
• num_inference_steps (None, optional, defaults to 50): No description provided
• timesteps (None, optional): No description provided
• sigmas (None, optional): No description provided
• latents (Tensor | NoneType, optional): No description provided
• attention_kwargs (None, optional): No description provided
• output_type (str, optional, defaults to np): The output type of the decoded videos

Outputs:

• videos (list): The generated videos.