Underdog Poodle 4B DFlash
This is a speculative-decoding draft for Underdog Poodle 4B. It is not a standalone language model. Serve it next to Poodle and it proposes up to 15 tokens at a time, which Poodle checks in a single forward pass. Poodle accepts or rejects every drafted token, so its outputs keep the same distribution they have without the draft. At one request on an H100, it makes Poodle 1.8x to 4.2x faster depending on the workload.
Speed
Single request on one H100 80GB with vLLM 0.31. Thinking on, temperature 0.6, top_p 0.95, top_k 20, up to 2,048 output tokens, 12 prompts per workload. Numbers are output tokens per second.
| Workload | No draft | Built-in MTP head (3 tokens) | Qwen3.5-4B-DFlash (15 tokens) | This draft (15 tokens) |
|---|---|---|---|---|
| Multi-turn agent episodes with tools | 210 | 386 | 350 | 462 (2.2x) |
| BFCL tool calls | 213 | 455 | 456 | 621 (2.9x) |
| IFEval | 222 | 404 | 376 | 451 (2.0x) |
| GSM8K | 222 | 460 | 655 | 677 (3.0x) |
| HumanEval | 222 | 499 | 950 | 925 (4.2x) |
| MT-Bench | 222 | 380 | 401 | 395 (1.8x) |
The gain is largest where Poodle spends most of its time: tool calls and agent turns. On those, Poodle with this draft runs 32% to 36% faster than with the Qwen3.5-4B-DFlash draft it shares an architecture with.
With 32 requests in flight, the same comparison shows up as tokens accepted per verification step. This draft averages 4.20 on agent episodes, 6.01 on BFCL and 3.98 on IFEval, against 3.33, 4.22 and 3.84 for Qwen3.5-4B-DFlash. At that load, drafting 7 tokens per step gives more total throughput than 15.
Architecture
Six Qwen3-style decoder layers (hidden size 2560, five with a 4,096-token sliding window and one with full attention), about 634M parameters in bf16. The draft has no embedding table or output head of its own; it uses Poodle's. It reads Poodle's hidden states from layers 1, 5, 9, 13, 17, 21, 25 and 29, and fills a block of masked positions in one pass (block size 16).
Use
With vLLM 0.31 or newer:
vllm serve ConwayResearch/Underdog-Poodle-4B \
--speculative-config '{"method": "dflash", "model": "ConwayResearch/Underdog-Poodle-4B-DFlash", "num_speculative_tokens": 15}' \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 --enable-auto-tool-choice
Use "num_speculative_tokens": 15 for single-user latency and 7 for many concurrent requests.
License
Apache-2.0, inherited from Qwen3.5-4B and z-lab/Qwen3.5-4B-DFlash.
- Downloads last month
- 26