QuantumDesk-AI/ultravox-qwen3.6-27b-base

Ultravox-Qwen3.6-27B (Base)

A speech-language model built on the Ultravox architecture, using Qwen 3.6-27B as the language backbone and Whisper-large-v3-turbo as the audio encoder. Pretrained by Quantum Desk LTD for real-time voice agent applications.

Unlike a cascaded ASR → LLM pipeline, this model consumes raw audio embeddings directly into the LLM's token stream — eliminating the transcription hop and cutting 200-400 ms of latency per turn.

Model Details

Base LLM	Qwen/Qwen3.6-27B (27B params, hybrid GDN/DeltaNet attention)
Audio encoder	openai/whisper-large-v3-turbo
Projection adapter	Swiglu MLP, stack_factor=8, hidden_size=5120
Trainable params	~100 M (projector + Whisper LoRA r=8)
Frozen params	27 B (LLM never gradient-updated)
Training precision	bfloat16
Recommended serve precision	Base LLM in FP8, projector + LoRA in bf16
Language	English (base checkpoint)
Context length	131 072 tokens

Training

• Loss: KL divergence to teacher (temperature 2.0)
• Steps: 10 000 (loss converged near step ~500, remaining steps refined the projection)
• Effective batch: 16 (8 × 2 gradient accumulation)
• LR: 5e-4 cosine → 5e-5, 500-step warmup
• Optimizer: AdamW
• Hardware: 1 × NVIDIA B200 (180 GB HBM3e), single-GPU, no FSDP
• Wall-clock: ~6 h 30 min
• Whisper adaptation: LoRA rank 8 on Whisper k_proj, q_proj, linear_k, linear_q

Training data

English audio-transcription and audio-continuation mix (~10 datasets):

• fixie-ai/librispeech_asr (clean + other, continuation + transcription)
• fixie-ai/peoplespeech-clean
• fixie-ai/gigaspeech-xl
• fixie-ai/commonvoice-en

This is an English-only base checkpoint. Multilingual and phone-quality audio training were deliberately excluded to accelerate iteration and reduce cost — Phase 1B (broader dataset mix) may be released later.

Evaluation

Training-time metrics on fixie-ai/librispeech-clean-transcription (validation, 256 samples):

Step	Eval loss
500	0.442
5 000	0.383
9 000	0.381
10 000 (final)	0.372

WER benchmark is being computed against librispeech-clean-test and will be added to this card as soon as it lands. Fixie's ultravox-v0_6-qwen-3-32b achieves 2.88 WER on librispeech (multilingual training, 32B base). Our English-only + 27B target is 3.5-5.0.

Usage

With vLLM (recommended for serving)

vllm serve QuantumDesk-AI/ultravox-qwen3.6-27b-base \
  --served-model-name ultravox \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --port 8000

With transformers (for research)

from transformers import AutoModel, AutoProcessor
import torchaudio

processor = AutoProcessor.from_pretrained(
    "QuantumDesk-AI/ultravox-qwen3.6-27b-base",
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    "QuantumDesk-AI/ultravox-qwen3.6-27b-base",
    trust_remote_code=True,
    torch_dtype="bfloat16",
).to("cuda")

audio, sr = torchaudio.load("hello.wav")
messages = [
    {"role": "user", "content": [
        {"type": "audio", "audio": audio[0].numpy(), "sampling_rate": sr},
        {"type": "text", "text": "What did I say?"},
    ]},
]
inputs = processor.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(outputs[0], skip_special_tokens=True))

Serving hardware

FP8 base + bf16 adapter fits on any Blackwell (RTX PRO 6000, B200) or Hopper (H100 80/94GB) GPU:

Card	VRAM	Concurrent voice sessions*
RTX PRO 6000 Blackwell	96 GB	~30-40
H100 94 GB	94 GB	~30-40
B200 180 GB	180 GB	~80-100

*Compute-bound (KV cache is not the limit). Assumes ~2000 tokens context per session and voice-typical 5-6 tok/s per session sustained rate.

Limitations

• English only — do not use for other languages; expect degraded quality even for accented English.
• Clean audio only — trained without phone-codec, noise, or music-background data. Real-world telephony will underperform relative to studio audio.
• No chat / instruction tuning — this is a base checkpoint. For interactive voice agents, fine-tune on your domain (see [Phase 2 discussion in repo]).
• Reasoning under audio input — some capacity is spent decoding audio; expect small (2-5 %) degradation vs pure-text prompting on reasoning-heavy tasks. Use text cascade for complex queries.

License

Apache 2.0. Model weights are Quantum Desk LTD's contribution. See individual base-model licenses for Qwen 3.6-27B (Apache 2.0) and Whisper-large-v3-turbo (MIT).

Acknowledgements

Built on:

• Fixie AI's Ultravox — architecture and training code
• Qwen team — 3.6-27B base
• OpenAI — Whisper encoder

Citation

@misc{quantumdesk_ultravox_qwen36_2026,
  author = {Quantum Desk LTD},
  title = {Ultravox-Qwen3.6-27B-Base},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/QuantumDesk-AI/ultravox-qwen3.6-27b-base},
}