Bbkblo/vdub-hindi-dubbing

🎬 vdub — Complete Video Dubbing Pipeline (Hindi)

INPUT = video LINK ya FILE → OUTPUT = Hindi dubbed video.

Chinese (ya kisi bhi) video lo, aur complete dubbing pao: transcript → speaker detection → emotion → translation → voice-cloning TTS (Hindi) → timing/pattern fit → BGM separation → final mux. Saare models isi repo me hain — kahin aur jaane ki zaroorat nahi.

📦 Is repo me SAB KUCH hai

Cheez	Kahan	Size
Pipeline code	vdub_full.py (12 stages)	39 KB
Colab/Kaggle runner	notebooks/vdub_full_runner.ipynb	—
Chatterbox Hindi TTS (voice clone + emotion)	models/chatterbox_hi/	3.2 GB
Chatterbox 23-lang pack (optional A/B)	models/mtl23/	2.1 GB
Kim_Vocal_2 separation (light, default)	models/separation/	67 MB
MelBandRoformer separation (best quality)	models/separation/	913 MB
NLLB-200-600M translation (zh→hi)	models/nllb-200-distilled-600M/	2.5 GB
Pattern transfer (DSP)	scripts/pattern_transfer.py	—
System docs	docs/SYSTEM_TREE.md	—
Model refill script	fetch_models.sh	—

Total repo: ~8.8 GB — git clone karte hi sab kuch aa jata hai (LFS).

⚡ Quickstart

A) Notebook (sabse aasaan — Colab/Kaggle)

1. notebooks/vdub_full_runner.ipynb kholo (Colab: File→Upload notebook; Kaggle: upload/import)
2. GPU (T4) + Internet ON karo
3. Cell 2 me INPUT_URL ya INPUT_FILE do
4. Run All → vdub_out/dubbed_video.mp4

B) CLI (local GPU / cloud)

git clone https://huggingface.co/Bbkblo/vdub-hindi-dubbing   # code + saare models
cd vdub-hindi-dubbing
pip install -e "git+https://github.com/resemble-ai/chatterbox.git" --no-deps
pip install librosa safetensors s3tokenizer transformers huggingface_hub soundfile scipy conformer omegaconf pyloudnorm diffusers ml-collections
pip install "git+https://github.com/resemble-ai/Perth.git@master"
# vdub_full.py ke upar CONFIG me INPUT_URL do, phir:
python3 vdub_full.py

(models agar clone se na mile to: bash fetch_models.sh)

🔄 Pipeline (12 stages, resume-safe)

VIDEO (link/file)
  → 1. fetch (yt-dlp ya file)
  → 2. transcript (SRT ya SenseVoice ASR)
  → 3. clip cutting
  → 4. speaker diarization (campplus clustering)
  → 5. emotion (SenseVoice tags)
  → 6. translation → Hindi (NLLB-200 offline / manual json)
  → 7. TTS (Chatterbox: speaker clone + emotion exaggeration)
  → 8. pattern transfer (original ke pauses/speed/loudness pe fit)
  → 9. separation (Kim_Vocal_2 / MelBandRoformer)
  → 10. mux (SRT timing + instrumental + keep-original ranges + 2 audio tracks)
  → 11. QC (voice similarity report)
  → 12. dubbed_video.mp4 + vdub_output.zip (+ Drive upload optional)

Har stage ka marker banta hai — VM/process mare to wahi se resume.

🎛️ Key config (vdub_full.py ke upar)

Param	Default	Kya karta hai
INPUT_URL / INPUT_FILE	—	dubbing ka source
SRT_FILE	""	apna subtitle (warna ASR)
TRANSLATE_MODE	nllb	nllb (offline) / manual (json)
SPEAKER_MAP	""	{"Speaker 1": "Hero"} names
SONG_RANGES	""	[[s,e],...] — yahan original audio rahega
SEP_MODE	kim	kim (67MB) / melband (913MB)
DRIVE_UPLOAD	False	output Drive me bhi bhejo

TTS knobs: exaggeration (emotion: ANGRY 1.4 / HAPPY 1.1 / SAD 0.4 / NEUTRAL 0.5), cfg_weight 0.3 (cross-lingual accent kam), temperature 0.8.

🤖 Models — license table

Model	License	Use
Chatterbox t3_hi + S3Gen + ve (ResembleAI)	MIT	TTS core
Chatterbox t3_mtl23ls (23-lang)	MIT	optional A/B
Kim_Vocal_2 (UVR5)	MIT	separation default
MelBandRoformer (FunAudioLLM)	Apache-2.0	separation best
NLLB-200-distilled-600M (Meta)	CC-BY-NC-4.0	translation (non-commercial)
SenseVoice/campplus (funasr)	auto-download	ASR/emotion/speaker QC

⚖️ Honest notes

• Chatterbox output me PerTh watermark embed hota hai
• Cloned voice use karne se pehle awaaz wale insaan ki permission lein
• NLLB translation CC-BY-NC hai — commercial use ke liye apna translation API lagao
• funasr ke chhote models (SenseVoice/campplus) pehli baar auto-download hote hain

👥 Credits

• Voice cloning + Hindi TTS: Resemble AI (Chatterbox, MIT)
• Separation (best): FunAudioLLM (Fun-CineForge, Apache-2.0)
• Translation: Meta (NLLB-200)
• Pipeline + pattern transfer: vdub project (Azim Khorajiya)