Add VibeVoice Model Support (Config, Modeling, Tokenizer, Processor, Scheduler, Auto-Registration + Weight Converter)
Author: anaslari23
Pull request: yes
Labels: name: Code agent slop

PR Description:

This PR adds support for Microsoft’s VibeVoice text-to-speech models (microsoft/VibeVoice-1.5B and microsoft/VibeVoice-Realtime-0.5B) to the Hugging Face Transformers library. It introduces a new architecture named “vibevoice” with configuration, modeling, tokenizer, processor, scheduler, and weight conversion utilities. After this integration, VibeVoice models can be loaded and used natively in Transformers without requiring trust_remote_code=True.
	1.	New model package: src/transformers/models/vibevoice/

Files added:
	•	configuration_vibevoice.py
Defines VibeVoiceConfig and sub-configs:
semantic_tokenizer_config, acoustic_tokenizer_config (VAE), decoder_config (Qwen2), diffusion_head_config.
	•	modeling_vibevoice.py
Implements:
VibeVoiceModel (Qwen2 LM + TTS LM layers, semantic tokenizer, acoustic tokenizer, diffusion head),
VibeVoiceForConditionalGeneration (full inference pipeline),
VibeVoiceStreamingModel (streaming generation),
VibeVoiceDiffusionHead (diffusion noise predictor).
	•	tokenization_vibevoice.py
Defines VibeVoiceTokenizer extending Qwen2Tokenizer with required audio/semantic/acoustic tokens.
	•	processing_vibevoice.py
Defines VibeVoiceProcessor to unify tokenizer and audio preprocessing.
	•	dpm_solver.py
Implements DPMSolver for diffusion-based speech generation.
	•	convert_vibevoice_weights_to_hf.py
Converts original VibeVoice .pt checkpoints to Hugging Face format and splits Qwen2 layers appropriately.

	2.	Integration with Transformers core

	•	Updated src/transformers/models/init.py to export VibeVoice modules.
	•	Updated src/transformers/models/auto/configuration_auto.py:
CONFIG_MAPPING_NAMES[“vibevoice”] = “VibeVoiceConfig”
	•	Updated src/transformers/models/auto/modeling_auto.py:
MODEL_MAPPING_NAMES[“vibevoice”] = “VibeVoiceModel”
MODEL_FOR_TEXT_TO_WAVEFORM_MAPPING_NAMES[“vibevoice”] = “VibeVoiceForConditionalGeneration”

These changes enable:
AutoConfig.from_pretrained(“microsoft/VibeVoice-1.5B”)
AutoModel.from_pretrained(“microsoft/VibeVoice-1.5B”)
AutoModelForTextToWaveform.from_pretrained(“microsoft/VibeVoice-1.5B”)
	3.	Verification

Manual testing:
	•	Successfully loaded converted VibeVoice checkpoints using from_pretrained
	•	Verified that model weights match expected architecture layout
	•	Ran complete semantic → acoustic → diffusion → waveform generation pipeline
	•	Produced correct audio outputs from simple text prompts

Automated testing:
	•	Added tests/models/vibevoice/test_vibevoice_integration.py
	•	Tests include: config load/save, forward pass, tokenizer/processor integration, minimal diffusion test

All tests passed locally.

Notes:
This is a foundational implementation. Architecture details may require small adjustments depending on any future official release by Microsoft. Additional features such as training utilities or alternate schedulers may be added in future PRs.

Related issue:
https://github.com/huggingface/transformers/issues/42758

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audio_spectrogram_transformer, auto