Bbkblo/vdub-hindi-dubbing-lite

🎬 vdub-LITE — Lightweight Video Dubbing Pipeline (Hindi, CPU-first)

Bbkblo/vdub-hindi-dubbing (8.8 GB) ka LIGHTWEIGHT version — same pipeline, ~80% chhote models, bina GPU ke.

INPUT = video LINK ya FILE → OUTPUT = Hindi dubbed video (2 audio tracks).

Chinese drama → Hindi dubbing ka pura system: transcript → speaker detection → emotion → translation → voice-cloning TTS (Hindi, INT8) → timing/pattern fit → BGM separation → final mux. CPU (Colab free / laptop / 4 GB RAM) pe chal jata hai.

⚡ Kya naya hai (vs full version)

Cheez	Full (vdub-hindi-dubbing)	LITE (ye repo)
TTS core (t3_hi)	2.14 GB fp32	0.65 GB INT8 (Q8_0) ✅
TTS RAM (CPU)	~3.2 GB	~1.9 GB ✅
TTS speed (CPU)	baseline	~3-4x (packed INT8 GEMM) ✅
Translation	NLLB fp32 (2.5 GB)	NLLB ONNX q8 (~0.9 GB) ✅
Separation	Kim 67 MB / MelBand 913 MB	Kim 67 MB (CPU) / MelBand optional
S3Gen vocoder	1.06 GB (upstream se)	1.06 GB (upstream se, auto-fetch)
Repo total	8.8 GB	~0.7 GB + runtime fetch ~1.1 GB ✅
Cinematic (Fun-CineForge-style)	pattern transfer	+ loudnorm -23 LUFS, pitch (f0) transfer*, lip-sync*

* experimental stages — neeche "Cinematic notes" dekho.

🔢 INT8 quantization — kya aur kaise

• t3_hi.safetensors (Chatterbox Hindi T3, 30-layer Llama-520M) ka 218 Linear layers GGUF Q8_0 scheme se quantize hue hain (per-32-block symmetric INT8 — llama.cpp wala standard).
• Quality: near-lossless — mean weight SNR 45.4 dB, mean relative RMS error 0.54% (max 0.74%). Poore 30-layer model ka forward + generation CPU pe verify kiya gaya (tests/ me results).
• Embeddings/norms/biases FP32 hi rahe.
• Loader (lite_loader.py):
  • model meta-device pe banta hai → fp32 kabhi RAM me nahi aata (2 GB RAM OK)
  • CPU pe oneDNN packed INT8 GEMM (3.7x fast) ya VDUB_W8_PACKED=0 se exact dequant-matmul (quality mode)
  • GPU ho to automatically fallback (W8Linear)

⚡ Quickstart

A) Notebook (sabse aasaan — Colab/Kaggle CPU ya GPU)

1. notebooks/vdub_lite_runner.ipynb kholo
2. INPUT_URL ya INPUT_FILE do
3. Run All → vdub_out/dubbed_video.mp4

B) CLI (local / cloud)

git clone https://huggingface.co/Bbkblo/vdub-hindi-dubbing-lite
cd vdub-hindi-dubbing-lite
bash fetch_models_lite.sh            # ~1.8 GB models
pip install -e "git+https://github.com/resemble-ai/chatterbox.git" --no-deps
pip install librosa safetensors s3tokenizer transformers huggingface_hub soundfile \
    scipy conformer omegaconf pyloudnorm diffusers ml-collections optimum onnxruntime
pip install "git+https://github.com/resemble-ai/Perth.git@master"
# vdub_lite.py ke upar CONFIG me INPUT_URL/INPUT_FILE do, phir:
python3 vdub_lite.py

C) Sirf TTS (voice clone, bina video)

from lite_loader import build_t3_lite
# ya seedha vdub_lite.py ka load_tts_lite() — full MTL TTS wrapper ready milta hai

🔄 Pipeline (12 stages, resume-safe)

VIDEO (link/file)
  → 1. fetch (yt-dlp ya file)
  → 2. transcript (SRT ya SenseVoice ASR)
  → 3. clip cutting
  → 4. speaker diarization (campplus clustering)
  → 5. emotion (SenseVoice tags)
  → 6. translation → Hindi (NLLB ONNX q8 / nllb / manual json)
  → 7. TTS (Chatterbox HI INT8: speaker clone + emotion exaggeration)
  → 8. pattern transfer (+ loudnorm -23 LUFS, pitch transfer optional)
  → 9. separation (Kim_Vocal_2 CPU / MelBandRoformer)
  → 10. mux (SRT timing + instrumental + keep-original ranges + 2 audio tracks)
  → 11. QC (voice similarity report)
  → 12. dubbed_video.mp4 + vdub_output.zip (+ Drive upload optional)

🎛️ Key config (vdub_lite.py ke upar)

Param	Default	Kya karta hai
INPUT_URL / INPUT_FILE	—	dubbing ka source
SRT_FILE	""	apna subtitle (warna ASR)
TRANSLATE_MODE	onnx	onnx (lite) / nllb / manual / auto
QUANT_MODE	int8	int8 (Q8_0) / fp32 fallback
SPEAKER_MAP	""	{"Speaker 1": "Hero"} names
SONG_RANGES	""	[[s,e],...] — yahan original audio rahega
SEP_MODE	kim	kim (67MB CPU) / melband (913MB best)
CINEMATIC	True	pattern transfer + loudnorm + emotion
LOUDNORM	True	broadcast -23 LUFS
PITCH_TRANSFER	False	f0 (pitch) original se match — experimental
LIP_SYNC	False	Wav2Lip — experimental, CPU slow
DRIVE_UPLOAD	False	output Drive me bhi bhejo

TTS knobs: exaggeration (ANGRY 1.4 / HAPPY 1.1 / SAD 0.4 / NEUTRAL 0.5), cfg_weight 0.3, temperature 0.8, MAX_SPEED 2.0 (pattern transfer).

⚠️ t3_hi EOS note (22-Aug-2026 live test): mtl_tts.generate ke default params (top_p=1.0, temp 0.8) se model kabhi-kabhi EOS nahi marta (loop). vdub_lite.py me ab tts_generate_lite() hai — duration-cap + repetition- tail trim + pattern-transfer fit. Ye Colab/CPU dono jagah use hota hai.

🎬 Cinematic notes (Fun-CineForge wala sawal)

Fun-CineForge (FunAudioLLM) ek zh/en-only cinematic dubbing model hai — Hindi TTS isse nahi hota. Is liye LITE me "Fun-CineForge-style" cinematic feel inhi chizo se aata hai:

1. Emotion exaggeration — SenseVoice emotion tag → Chatterbox ka emotion control
2. Pattern transfer — original actor ke pauses/speed/loudness pe exact fit (aapka idea — v2/v3 sessions me live test ho chuka hai ✅)
3. Loudness normalize — broadcast standard (-23 LUFS)
4. Pitch transfer (experimental) — original f0_hz se voice pitch match
5. Lip-sync (experimental) — Wav2Lip; honest note: CPU pe slow aur quality basic hai. Asli Fun-CineForge-quality Hindi lip-sync ke liye alag bada project lagega (ya Fun-CineForge team ka multilingual release ka wait).

Separation ke liye Fun-CineForge ka MelBandRoformer (913 MB) optional best quality hai (SEP_MODE=melband).

🤖 Models — license table

Model	License	Use
Chatterbox t3_hi (INT8 Q8_0)	MIT (ResembleAI)	TTS core
Chatterbox S3Gen + ve + tokenizer	MIT	vocoder / voice encoder
Kim_Vocal_2 (UVR5)	MIT	separation default
MelBandRoformer (Fun-CineForge)	Apache-2.0	separation best (optional)
NLLB-200-600M ONNX q8	CC-BY-NC-4.0	translation (non-commercial)
SenseVoice/campplus (funasr)	auto-download	ASR/emotion/speaker QC

⚖️ Honest notes

• Chatterbox output me PerTh watermark embed hota hai (upstream behavior)
• Cloned voice use karne se pehle awaaz wale insaan ki permission lein
• NLLB translation CC-BY-NC hai — commercial use ke liye apna translation API lagao
• INT8 quality near-lossless hai (Q8_0), par 100% fp32 jaisi guarantee nahi — koi line ajeeb lage to QUANT_MODE=fp32 fallback hai
• Sandbox/CPU RAM < 2 GB ho to TTS Colab pe chalao (notebook bana hua hai)

👥 Credits

• Voice cloning + Hindi TTS: Resemble AI (Chatterbox, MIT)
• Q8_0 quantization scheme: GGUF/llama.cpp standard
• Separation (best): FunAudioLLM (Fun-CineForge, Apache-2.0)
• Translation: Meta (NLLB-200, Xenova ONNX conversion)
• Pipeline + pattern transfer + lite conversion: vdub project (Azim Khorajiya)