actorcritic/voxcpm2-finetuned-100hr

VoxCPM2 Fine-Tuned (Bengali - 100 Hours)

This is a fine-tuned version of the VoxCPM2 model, optimized for Bengali text-to-speech generation. It was trained on a 100-hour Bengali speech dataset.

Fine-tuning Details & Optimizations

Training a massive 3-billion parameter multimodal architecture normally requires multiple A100s. To squeeze the entire fine-tuning pipeline onto a single 48GB GPU (NVIDIA RTX A6000), the following advanced optimizations were utilized:

• bfloat16 Native Execution: Model weights and gradients were cast natively to bfloat16, bypassing standard PyTorch 32-bit overhead and cutting memory usage in half.
• 8-bit AdamW (BitsAndBytes): The optimizer states were heavily quantized, reducing the optimizer VRAM footprint from ~24 GB down to ~6 GB without sacrificing convergence stability.
• Expandable Segments: PyTorch's PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True was enabled, which prevented debilitating VRAM fragmentation crashes during dynamic context length generation and gradient accumulation.
• Memory-Optimized DataLoader: Arrow-format datasets were pre-cached, bypassing redundant mapping/casting during initialization and radically improving dataset streaming speed.
• Strict Memory Clamping: batch_size: 1 paired with grad_accum_steps: 32 (effective batch size 32) and capped max_batch_tokens: 3000 to prevent unpredictable sequence lengths from OOMing the 48GB limit during backward passes.

Final Training Metrics:

• Final Training Diff Loss: 0.9512
• Final Validation Diff Loss: 0.9539

(Note: The checkpoint with the lowest validation loss was actually step 500 (val loss: 0.8210), but this repository contains the final converged weights at step 1122).

Inference Ready Code

You can use this model instantly by loading it with the official voxcpm repository:

import sys
import torch
import soundfile as sf
import os

# Ensure you have cloned the VoxCPM repo and added it to your python path
sys.path.insert(0, "/path/to/VoxCPM/src")
import voxcpm

# 1. Provide the path to these downloaded HF weights
model_path = "./voxcpm2-finetuned-100hr"

# 2. Initialize the model
model = voxcpm.VoxCPM(voxcpm_model_path=model_path)

# 3. Enter your target text
text = "ধান ফুরাল, পান ফুরাল, খাজনার উপায় কী? আর কটা দিন সবুর করো, রসুন বুনেছি।" 

print(f"Generating audio for text: '{text}'")

# 4. Generate
wav = model.generate(
    text=text,
    cfg_value=2.0,
    inference_timesteps=10,
    normalize=True,
    denoise=False
)

# 5. Save the output
output_path = "output.wav"
sample_rate = model.tts_model.sample_rate
sf.write(output_path, wav, sample_rate)
print(f"✅ Success! Audio saved to {output_path}")