MiniMax-H3-Fun-Controlnet-Union-2.0

What's new in 2.0

	MiniMax-H3-Fun-Controlnet-Union (v1)	MiniMax-H3-Fun-Controlnet-Union-2.0 (this model)
Control conditions	5 — Canny, Depth, HED, MLSD, Pose	8 — + Scribble, Layout, Gray
Control branch depth	5 control blocks (layers 0, 10, 20, 30, 40)	10 control blocks (layers 0, 5, 10, …, 45) — skips injected every 5 of the 50 transformer blocks
Inpaint masked-pixel recipe	pre_norm (holes ≈ −2 in VAE input space, extreme dark)	post_norm (holes at 0, mid-gray, following Wan 2.1) — cleaner inpaint blending
Checkpoint contents	control_proj_in + 5 control_blocks (~6.8 GB)	control_proj_in + 10 control_blocks (~13.5 GB)
Required config	minimax_h3_control.yaml	minimax_h3_control_inpaint_post_norm.yaml

Everything else is carried over from v1: control_in_dim = 49 (latent + masked latent + mask, so the same branch does control and inpaint), control_apply_audio = false, guidance-distilled (guidance_scale = 1.0), and the same zero-gated skip-add into the main branch.

Loading a v1 config against this checkpoint is a silent failure. With minimax_h3_control.yaml (5 blocks) the model builds only half the control branch; load_state_dict(strict=False) drops control_blocks.5~9 as unexpected keys and misplaces the rest, producing wrong outputs. Always use minimax_h3_control_inpaint_post_norm.yaml.

Model Card

Name	Description
MiniMax-H3-Fun-Controlnet-Union-2.0.safetensors	ControlNet-Union-2.0 branch weights for MiniMax-H3. Holds only the control branch (control_proj_in plus 10 control_blocks, about 13.5 GB) and is loaded on top of the base MiniMax-H3 transformer. One checkpoint supports 8 control conditions (Canny, Depth, HED, MLSD, Pose, Scribble, Layout, Gray) and video inpainting.

Model Features

• Union control over 8 conditions: one checkpoint handles Canny, Depth, HED, MLSD, Pose, Scribble, Layout and Gray control videos for video-to-video generation — no per-condition checkpoint switching.
• Denser control injection: the control branch attaches to 10 of the 50 transformer blocks (layers 0, 5, 10, 15, 20, 25, 30, 35, 40, 45); every control skip is added to the main branch through a zero-gated projection. This is roughly 2× the injection points of v1 and gives tighter structural adherence.
• Guidance-distilled: run with guidance_scale = 1.0, one forward pass per step, no classifier-free guidance needed.
• Inpainting is supported, with the post_norm recipe: the control input is widened to control_in_dim = 49 (latent + masked latent + mask channels). Unlike v1, the masked pixels are zeroed after the ImageNet normalization (holes sit at 0 / mid-gray) rather than before it (holes landed near −2 / extreme dark), which improves how filled regions blend with kept regions. Use examples/minimax_h3_fun/predict_v2v_control_inpaint.py.
• control_context_scale scales every control skip before it is added to the main branch: 1.0 gives the strongest control (used for all results below), values below 1.0 weaken the guidance of the control video, 0.0 switches the control branch off.
• The generation follows the control video: the frame count snaps down to the largest 17 * n + 5 the video VAE can decode (duration capped at 15 seconds), the canvas keeps the control video's own aspect ratio at the height * width pixel budget (both multiples of 32), at a fixed 24 fps.
• Detailed prompts give better stability; we recommend describing the scene, the subject and the camera in the prompt.

Supported control conditions

Condition	Control signal	New in 2.0?
Canny	Canny edge map
Depth	Monocular depth map
HED	HED edge detection
MLSD	Line-segment detection
Pose	DWPose skeleton
Scribble	Free-hand / sketch lines	✅
Layout	Bounding-box layout	✅
Gray	Grayscale (luminance) video	✅

The Layout control videos follow the layout generation recipe of Wan2.1-VACE: per-subject bounding boxes (detected/tracked or given directly) are rendered as color-coded boxes on a white background, producing an ordinary RGB video that conditions the model. You can reuse the VACE-Annotators preprocessing tools (e.g. vace_preproccess.py --task layout_track ...) to produce layout videos from a reference video or a pair of bboxes.

Results

All samples below are generated with num_inference_steps = 40, guidance_scale = 1.0, control_context_scale = 1.00, seed 43, canvas mode control at a 704×1280 pixel budget, 24 fps. In each pair the top row is the control video, the bottom row is the output.

Canny	Depth	HED	MLSD
Pose	Scribble ✨	Layout ✨	Gray ✨

Inpainting (post_norm)

A masked region of the source video is re-drawn from the prompt while the rest of the frame is preserved. The mask video is white where the content should be re-generated and black where it should be kept.

Source video	Mask	Inpaint output

Inference

Go to the VideoX-Fun repository for more details.

Please clone the VideoX-Fun repository and create the required directories:

# Clone the code
git clone https://github.com/aigc-apps/VideoX-Fun.git

# Enter VideoX-Fun's directory
cd VideoX-Fun

# Create model directories
mkdir -p models/Diffusion_Transformer

Then download the base MiniMax-H3 model and this checkpoint into models/Diffusion_Transformer.

📦 models/
├──  Diffusion_Transformer/
│   ├── 📂 MiniMax-H3/
│   └──  MiniMax-H3-Fun-Controlnet-Union-2.0/
│       └──  MiniMax-H3-Fun-Controlnet-Union-2.0.safetensors

Then edit the settings at the top of examples/minimax_h3_fun/predict_v2v_control.py (or predict_v2v_control_inpaint.py for inpainting) and run it.

model_name          = "models/Diffusion_Transformer/MiniMax-H3"
config_path         = "config/minimax_h3/minimax_h3_control_inpaint_post_norm.yaml"
transformer_path    = "models/Diffusion_Transformer/MiniMax-H3-Fun-Controlnet-Union-2.0/MiniMax-H3-Fun-Controlnet-Union-2.0.safetensors"
control_video       = "your_control_video.mp4"
prompt              = "your prompt"

python examples/minimax_h3_fun/predict_v2v_control.py

Notes:

• config_path must be config/minimax_h3/minimax_h3_control_inpaint_post_norm.yaml. It builds the control branch exactly as the checkpoint expects (control_blocks_places: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45], control_in_dim: 49, control_apply_audio: false, inpaint_masked_pixel_mode: post_norm); the v1 minimax_h3_control.yaml (5 blocks) will silently drop half the control weights.
• For pure control (no inpaint input) the pipeline zero-pads the mask channels, so this inpaint checkpoint still runs plain Canny/Depth/… control correctly.
• The checkpoint is guidance-distilled: keep guidance_scale = 1.0; a value above 1 applies guidance twice and degrades the output.
• The control checkpoint carries only the control branch; the base MiniMax-H3 weights must be present in model_name.
• For the Layout condition, generate the control video with the Wan2.1-VACE layout pipeline (see Supported control conditions); other control-video formats are unchanged from v1.
• Memory: the transformer (about 62 GB) plus the Qwen3-VL text encoder (about 62 GB) do not fit one 80 GB GPU fully loaded; use model_group_offload (fastest) or model_cpu_offload_and_qfloat8 on a single 80 GB GPU.

License

This model is a derivative of MiniMax-H3 and is released under the MiniMax H3 Community License Agreement. Please read the license carefully, especially the territorial restrictions and the Acceptable Use Policy, before use.