DiffSynth-Studio/MiniMax-H3-Text-Embeddings

MiniMax-H3 Templates: Text Embeddings

This repository provides a collection of video effect templates for the MiniMax-H3 model. These templates encapsulate specific model capabilities as text embeddings, implemented via Diffusion Templates.

Showcase

Effects Gallery

art_is_explosion	storm_magic
dark_magic	kiss_camera
bullet_time	truman_show
fire_breath	blooming_flowers
four_seasons	spiral_ascent

Effect Combinations

Combining effects: art_is_explosion + storm_magic

How It Works

Each template consists of a single tensor. This tensor can either replace or be combined with the output of the Text Encoder, a mechanism very similar to Textual Inversion.

While MiniMax-H3 boasts powerful base capabilities, its massive parameter count makes LoRA training challenging. Text Embeddings offer a lightweight alternative to LoRA with the following advantages:

• Modular Capabilities: Text Embeddings serve as carriers of model functionality. Similar to agent skills, they enable specific capabilities to be saved, reused, and distributed as atomic units through model platforms.
• Flexible Initialization: Embeddings can be initialized from text prompts, images, or videos. Simply pass the input data through the MiniMax-H3 Text Encoder to quickly generate a usable Text Embedding.
• Trainable: Like LoRA, Text Embeddings support end-to-end training on video datasets, enabling targeted enhancement of specific generative abilities.
• Composable: Multiple templates can be combined to jointly influence generation, allowing you to create complex and visually striking effects.

Inference and Training

Installation

Install DiffSynth-Studio:

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e ".[all]"

Initialize a Text Embedding via Text Encoder

The following code has low VRAM requirements and can run with as little as 6GB of VRAM:

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from safetensors.torch import save_file

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="MiniMax/MiniMax-H3",
        origin_file_pattern="FL2VA/text_encoder/model*.safetensors",
        offload_dtype="disk",
        offload_device="disk",
        onload_dtype="disk",
        onload_device="disk",
        preparing_dtype=torch.bfloat16,
        preparing_device="cuda",
        computation_dtype=torch.bfloat16,
        computation_device="cuda",
    )],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=0,
)
prompt = "xxx"
text_embedding = pipe.export_text_embedding(prompt)
save_file({"weight": text_embedding}, "model.safetensors")

Inference with Text Embeddings via Diffusion Templates

You can load text embeddings through Diffusion Templates for inference. Optionally, load an acceleration LoRA to speed up inference (note: the text encoder is not required during this stage):

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/",
    )],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    template_inputs=[{}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)

Inference with Multiple Text Embeddings

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

```python
vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/"),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/storm_magic/"),
    ],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    template_inputs=[{"model_id": 0}, {"model_id": 1}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)

Inference Using the Native Pipeline

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from diffsynth import load_state_dict
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
text_embedding_config = ModelConfig(
    model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings",
    origin_file_pattern="models/art_is_explosion/model.safetensors",
)
text_embedding_config.download_if_necessary()
text_embedding = load_state_dict(text_embedding_config.path)["weight"]

snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = pipe(
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    text_embedding=text_embedding,
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)

Training Text Embeddings

modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-Text-Embeddings/*" --local_dir ./data/diffsynth_example_dataset

accelerate launch examples/minimax_h3/model_training/train.py \
  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings \
  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings/metadata.json \
  --data_file_keys "video,input_audio" \
  --extra_inputs "input_audio,input_image,template_inputs" \
  --height 832 \
  --width 480 \
  --num_frames 124 \
  --dataset_repeat 100 \
  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-pruned-nf4.safetensors" \
  --template_model_id_or_path "DiffSynth-Studio/MiniMax-H3-Text-Embeddings:models/art_is_explosion/" \
  --learning_rate 1e-4 \
  --num_epochs 2 \
  --remove_prefix_in_ckpt "pipe.template_model." \
  --output_path "./models/train/MiniMax-H3-Text-Embeddings-full" \
  --trainable_models "template_model" \
  --use_gradient_checkpointing