AnthonyDi/VoiceChatExperiment

Qwen3.6 NVIDIA VoiceChat Stage A adapters

These are experimental Stage A adapters for replacing the semantic backbone of nvidia/NVIDIA-NemotronLabs-VoiceChat-11B with Qwen/Qwen3.6-35B-A3B while preserving NVIDIA's FastConformer, AddFusion, output/timing heads, RNNT, DuplexEARTTS talker, and codec.

This checkpoint is not a working speech-to-speech model. It passes acoustic-dependence controls at its parent content-margin stage, but the latest unseen free-running diagnostic fails semantic coherence and repeats meta-text. It is published for reproducibility and continued research, not deployment.

Files

• perception_projection.safetensors: trained NVIDIA perception projection adapter.
• qwen_lora.safetensors: rank-8 LoRA weights for the upper Qwen layers.
• nvidia_voicechat_qwen36_120ms.json: exact composite graph configuration.
• checkpoint_metadata.json: dependencies, hashes, training objective, and gate result.
• failed_preview/: the held-out input and generated text proving the current failure.

The repository intentionally does not redistribute either required base checkpoint.

Required bases

• nvidia/NVIDIA-NemotronLabs-VoiceChat-11B, revision 5631f538c74d1b4a8adfbc0b3a2c4aed6eba4d56
• Qwen/Qwen3.6-35B-A3B, BF16 revision 995ad96eacd98c81ed38be0c5b274b04031597b0
• NVIDIA Speech branch nemotron-labs-voicechat, commit 097dfe9e2f55baf653b83035868bdc89849f1b47

Base-model licenses and access terms apply. The NVIDIA checkpoint uses OpenMDW 1.1; consult the original model cards before use.

Training state

The adapter was trained on real SpokenWOZ conversational pairs with the SALM-Duplex paper clock/channel curriculum, offline Qwen top-64 text-path distillation, ground-truth response CE, compact user-end representation alignment, text replay, scheduled sampling, and a duration-aligned wrong-audio margin. Training and inference use direct Hugging Face Transformers/PyTorch; vLLM is not used.

The operational fallback frame length is 120 ms because the strict 80 ms direct-HF serving gate did not pass on the test GH200.

Gate result

FAIL_M2A_FREE_RUNNING_GATE

For the unseen user audio “Hello, is this Customer Service Center?”, this checkpoint emitted repeated “I will provide a response in English.” No talker/codec output was accepted. Continue with encoder co-adaptation or scheduled continuous-modality mixing before attempting later duplex-policy stages.