Lottolabs/Qwen3.5-9B-TT-BFP4-P150

Qwen3.5-9B — TT BFP4 / BF16 for Tenstorrent P150

An experimental, hardware-specific quantized weight-cache package and custom inference runtime derived from Qwen/Qwen3.5-9B, tested on one Tenstorrent Blackhole P150 with 32 GB device memory.

The published configuration combines packed quantized projections, fused GDN and SwiGLU kernels, sharded normalization, and one-token MTP speculative decoding. A fresh canonical LocalMaxxing test measured 76.15 output tokens/s; the separate three-workload clean-download benchmark measured 70.57–71.43 tokens/s.

Compatibility: These are native TTNN .tensorbin files, not a standalone Transformers checkpoint or an AWQ, GPTQ, GGUF, NF4, or CUDA-compatible quant. Use the supplied custom runtime. The original pinned Qwen checkpoint is still required by the loader.

Experimental behavior: Sharded normalization changes numerical results relative to the earlier interleaved-norm configuration. Repeated long-prompt requests produced different completions even at temperature zero. Deterministic output and quality equivalence are not established. A verified speed measurement is not a correctness or quality certification.

What is included

Artifact	Contents
quantized/	460 runtime-used TTNN tensor files, 9,699,607,184 bytes (9.70 GB decimal)
runtime/	52 changed/new runtime source files, licenses, provenance, and a Dockerfile pinned to a public base-image digest
model-spec.json	Tested P150 configuration: MTP-1, concurrency 1, maximum context 8,192 tokens
prepare.py	Downloads the pinned upstream checkpoint, verifies published tensor checksums, and installs the TT caches
serve.py	Docker-based OpenAI-compatible API launcher
benchmark.py	Sequential streaming speed benchmark with timing, output hashes, and speculation counters
manifest.json	File hashes, dependencies, configuration, limitations, and reproduction metadata
benchmarks/	Clean-download measurements and verification evidence

Quantization and storage

Stored dtype	Files	Serialized bytes
TTNN BFLOAT4_B	298	7,597,040,400
BFLOAT16	162	2,102,566,784
Total	460	9,699,607,184

Weights are converted with TTNN and model-specific packed worker-bank layouts. The cache contains mixed precisions and multiple runtime representations for operations such as prefill and decode. Its size is not a uniform four-bit parameter count, and BFP4 should not be interpreted as another framework's four-bit format.

Some tensors are reconstructed from the original checkpoint or derived at initialization, including GDN convolution/frontend tensors and derived projection/state/position structures. Uploading these caches therefore does not eliminate the upstream checkpoint download or its host-memory requirements.

Requirements

• One Tenstorrent P150, with device access at /dev/tenstorrent.
• Linux x86-64 host and Docker.
• Python 3.11 or newer for the host installation scripts, plus huggingface_hub.
• Working Tenstorrent driver, firmware, permissions, and hugepage configuration. Verification used KMD 2.11.0 and firmware bundle 19.13.2.
• The launcher mounts /dev/hugepages-1G and /dev/hugepages; configure these for your host before launch.
• Sufficient host RAM and disk for the original checkpoint, published cache, Docker image/build artifacts, and working copies. The cache size alone is not the total installation footprint.

Only text generation on a single P150 was tested. Vision, other accelerators, multi-device operation, and higher-concurrency serving are not validated by this release.

Quick start

These commands download the revision used for the clean-download benchmark. Later documentation and benchmark-result commits do not change those tested tensors or runtime sources.

python3 -m pip install huggingface_hub

hf download Lottolabs/Qwen3.5-9B-TT-BFP4-P150 \
  --revision 57a271fb1073166cb8746ba684c3aba5fc81ca3c \
  --local-dir ./qwen-tt

# Choose a disk-backed directory with sufficient free space.
CACHE_ROOT="$HOME/qwen-tt-cache"
python3 qwen-tt/prepare.py --cache-root "$CACHE_ROOT"

docker build --platform linux/amd64 \
  --build-arg BUILD_JOBS=16 \
  -t qwen35-tt-bfp4:hf \
  -f qwen-tt/runtime/Dockerfile qwen-tt/runtime

python3 qwen-tt/serve.py --cache-root "$CACHE_ROOT" --port 8000

The served API model name remains Qwen/Qwen3.5-9B:

curl http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3.5-9B","prompt":"The capital of France is","max_tokens":128,"temperature":0}'

Network security: The supplied launcher uses host networking and --no-auth. Do not expose it directly to an untrusted network; use network restrictions or an authenticated proxy.

Cold startup includes model loading and device-kernel compilation. Wait until the API is ready before benchmarking.

Performance

All results below use one P150, the custom vLLM / TT-Metal runtime, MTP-1, and one active request. They measure the complete configured system, not the isolated benefit of quantization. Acceptance rate, prompt distribution, output length, and host/runtime conditions affect throughput.

LocalMaxxing canonical speed test

Public approved, Verified run

Metric	Result
Canonical prompt	reasoning-v1
Prompt tokens	254
Output tokens	512
Decode throughput	76.15 tok/s
Time to first token	209.07 ms
Draft tokens proposed / accepted	279 / 232
MTP acceptance rate	83.15%

Protocol: temperature zero, plain /v1/completions streaming, two warmups and five measured requests, with prefix caching disabled. The median-throughput iteration was submitted with its own matching timing and acceptance evidence, rather than the fastest sample. The Verified badge concerns the site's speed-test evidence checks; it does not establish model quality or output determinism.

Clean-download benchmark

Workload	Prompt tokens	Median decode	Median TTFT	Distinct completions across five repeats
Short	5	71.43 tok/s	188.73 ms	1
Code	21	71.25 tok/s	178.43 ms	1
Longer prompt	481	70.57 tok/s	910.72 ms	3

Protocol: one warmup per workload, five measured sequential requests per workload, temperature zero, and 512 generated tokens in every measured request. Decode throughput is (completion_tokens - 1) / (last_text_time - first_text_time), excluding TTFT.

These workloads differ from the LocalMaxxing canonical prompt; their throughput figures are not interchangeable.

Run the included benchmark against a ready server:

python3 qwen-tt/benchmark.py \
  --url http://127.0.0.1:8000 \
  --output speed.json

Evidence: raw speed measurements and verification details.

Reproducibility and validation scope

• Freshly downloaded the published quantized package at revision 57a271fb1073166cb8746ba684c3aba5fc81ca3c.
• Verified 527 package files byte-for-byte against the staged release.
• Rebuilt the custom runtime from the downloaded source overlay and pinned public base image.
• Installed the downloaded tensors in an isolated cache; the old TT cache volume was not mounted.
• The original upstream checkpoint was copied into that isolated directory and verified against its Hub hashes, not redownloaded in full during this test.
• All 460 quantized tensor files remained unchanged after serving.
• All 15 corresponding pre-upload and post-download completions matched. This comparison used the same request sequence; it does not imply repeatability across identical requests. The longer-prompt test still produced three distinct completions.

Upstream checkpoint revision:

Qwen/Qwen3.5-9B
c202236235762e1c871ad0ccb60c8ee5ba337b9a

The Dockerfile pins its public base by digest. See runtime provenance, source manifest, and the release manifest for exact identities and checksums.

Intended use and limitations

Intended for experimentation, reproduction, and kernel/runtime performance research on the supported P150 configuration. It is not presented as a production-quality, hardware-independent quantized checkpoint.

• Original-checkpoint dependence and runtime-specific layouts limit portability.
• Sharded norms are not bit-exact to the earlier interleaved-norm path.
• MTP output variability at temperature zero is a known unresolved limitation.
• No quality-equivalence claim is made for this published MTP configuration.
• No peak device-memory usage or power measurement is claimed by the reported speed tests.
• An 8,192-token configured context does not mean the full context range was benchmarked; the measured prompts above are much shorter.

License and attribution

Derived from Qwen/Qwen3.5-9B. The upstream model license is Apache-2.0; see LICENSE. Runtime sources retain their upstream copyright notices and bundled licenses. This is a community experimental release under Lottolabs, not an official Qwen or Tenstorrent release.