🎬 CineDub — Video Dubbing with Coherent Sound Effects

ACM MM 2026 · CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects

Model Description

CineDub is a unified diffusion framework for video dubbing that jointly generates temporally aligned speech and coherent ambient sound effects from raw video. A single checkpoint handles three tasks without task-specific heads or diarization pre-processing:

• V2A (Video → Sound Effects) — generate a scene-aligned soundscape with no speech
• V2S (Video → Dubbed Speech) — support both single-speaker and multi-speaker dialogue, with optional zero-shot voice cloning
• V2SA (Video → Speech + Ambient) — unified joint generation of dubbed speech and background sound

Multi-speaker dubbing is a first-class capability: pass ref_wav as a list of reference audio and the model automatically routes to two-speaker synthesis — no diarization or face crops required.

Architecture

CineDub is built on a standard Diffusion Transformer (DiT) backbone with a decoupled two-branch conditioning design:

• Semantic Branch — Fuses transcript embeddings from a Gemma-T5 conditioner and visual features from CLIP via cross-attention. Two learned sentinel embeddings ("none_speech" and "clean speech") let a single model disable either branch on demand, unifying V2A / V2S / V2SA in one forward pass.
• Temporal Branch — Uses uncropped Synchformer features injected via Adaptive Layer Normalization (AdaLN), providing frame-accurate audio-visual synchronization.

Training follows a three-stage progressive schedule — ambient pre-training → joint multi-task fine-tuning → audio-prompt-conditioned speaker cloning — implemented via the Implicit Cross-Head Conditioning (ICHC) paradigm and Ambient-to-Linguistic Curriculum (ALC), which resolve the sub-task collapse and cross-prompt interference commonly seen in joint speech+SFX generators.

Key Features

• Unified dubbing model: V2A, V2S, V2SA share a single checkpoint
• Multi-speaker dialogue dubbing without diarization: pass a list of reference wavs; the model handles turn allocation
• Zero-shot voice cloning from a 3-second reference
• Coherent joint synthesis: speech and ambient are generated in the same latent forward, avoiding post-hoc mixing artifacts
• Sentinel-token task routing: no explicit --task argument in the model — mode is inferred from the values of trans_cap and text_prompt fields

Model Files

cinedub/
├── model_config.json                          # DiT + conditioner architecture
└── checkpoint/
    └── step=300000.ckpt                       # DiT weights + VAE (5.9 GB)

This is the ap_m4 variant at step 300k — the audio-prompt-tuned checkpoint used for all V2S / V2SA demos in the paper. The VAE (diffusion.pretransform.*) is embedded inside the ckpt (365 keys, ~610 MB, injected from the shared base checkpoint). EMA weights and Adam optimizer states are stripped to keep the download to ~6 GB; if you need to resume training you must retrieve the full 22 GB checkpoint separately.

Additional Dependencies

The following auxiliary weights are auto-downloaded on first inference:

Model	Source	Notes
DFN5B CLIP ViT-H/14-384	apple/DFN5B-CLIP-ViT-H-14-384	Apple ML Research License (research only)
Gemma-T5 (transcript encoder)	google/t5gemma-ml-ml-ul2	Gated — accept license first
Synchformer	hkchengrex/MMAudio ext_weights/synchformer_state_dict.pth	CC BY-NC 4.0 (research only)

Quick Start

git clone https://github.com/dalision/cinedub_infer.git
cd cinedub_infer

# 1. Environment (CUDA 12.8)
conda env create -f environment.yml && conda activate cinedub

# 2. Download this checkpoint
huggingface-cli login   # accept the CC BY-NC 4.0 license
huggingface-cli download Dalision/cinedub --local-dir weights/cinedub

# 3. Run a demo (three tasks share the same script)
bash scripts/demo_infer.sh v2a     # video → sound effects
bash scripts/demo_infer.sh v2s     # video → dubbed speech (single or multi-speaker)
bash scripts/demo_infer.sh v2sa    # video → speech + ambient

# Or drive the Python entrypoint directly on your own clip
python inference.py --input my_clip.mp4 --task v2s --ref_wav speaker.wav --output ./out

See the GitHub repo for the full JSONL schema, meta-token cheatsheet, and the Gemini prompt used to auto-generate trans_cap fields.

Directory Layout Requirement

The inference code resolves model_config.json via dirname(dirname(ckpt))/model_config.json — please keep the downloaded layout intact:

weights/cinedub/
├── model_config.json                    ← MUST be here
└── checkpoint/
    └── step=300000.ckpt                 ← ckpt is 2 levels below

Links

• Paper: arXiv:2608.15734 (ACM MM 2026)
• Inference code: github.com/dalision/cinedub_infer
• Benchmark: Dalision/cinedub_benchmark
• Evaluation harness: github.com/dalision/cinedub_eval

Citation

@inproceedings{cinedub2026,
  title     = {CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects},
  author    = {Dai, Yusheng and Wang, Kangdi and Gao, Baolong and Jiang, Yuxuan
               and Wang, Weiqiang and Ke, Qiuhong and Cai, Jianfei},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
  year      = {2026},
  publisher = {ACM},
  url       = {https://arxiv.org/abs/2608.15734}
}

License

Model weights are released under CC BY-NC 4.0 — research and non-commercial use only. See the auxiliary dependency table above for third-party weight licenses; the strictest license among all constituent weights governs your use of the assembled system.

The inference source code at github.com/dalision/cinedub_infer is separately released under the MIT License.