Qwen3.8-27B — Runtime Optimizations Tested on a Single RTX 3090
Overview
M23AI releases a source-bound runtime optimization package for Qwen3.8-27B with DFlash2 speculative decoding on a single NVIDIA RTX 3090.
The package addresses MLP execution, GDN metadata preparation, and selected normalization and strided tensor operations in the vLLM execution path.
The verified configuration achieved 204.11 tokens/s aggregate decode throughput on a single NVIDIA RTX 3090 (1 GPU), with a 350 W power limit, across an eight-task coding benchmark. The result was reproduced after restarting the inference service with the published runtime modules. All throughput figures below refer to this single-GPU configuration, not a multi-GPU system.
Release classification: runtime integration package for developers. This release does not contain target or draft model weights, does not introduce a newly trained checkpoint, and does not provide a validated standalone installer. The reported result is attributable to the tested inference configuration.
Benchmark configuration
| Parameter | Configuration |
|---|---|
| Hardware | 1 × NVIDIA RTX 3090 |
| GPU power limit | 350 W |
| Target | Qwen3.8-27B W4A16 AutoRound, prepared fast variant |
| Draft | Qwen3.8-27B DFlash2 W4A16 |
| Speculative tokens | 7 |
| Workload | Eight coding prompts; sequential requests |
| Input length | 74–102 tokens per task |
| Output length | 384 tokens per task |
| Sampling | Temperature 0; thinking disabled |
| Warm-up | One request before the measured suite |
| Concurrent workload | None during measurement |
Decode throughput
| Task | Tokens/s |
|---|---|
| Python — incremental index | 256.59 |
| TypeScript — transactional text edits | 243.67 |
| SQL — append-only ledger | 230.64 |
| Go — configuration reloader | 212.29 |
| Zig — generation-counted memory pool | 202.56 |
| C++ — incremental frame parser | 182.72 |
| Rust — bounded MPMC queue | 182.69 |
| CUDA — exclusive prefix scan | 159.64 |
| Aggregate | 204.11 |
Decode time is measured from the first non-empty content chunk to the end of the stream. Per-request throughput is (completion_tokens - 1) / decode_seconds. Aggregate throughput uses the sum of these token counts divided by the sum of decode durations; it is not the arithmetic mean of task throughputs.
The table describes individual prompts, not language-wide averages. The benchmark measures generation speed, not the correctness of generated programs. It does not establish standalone prefill throughput, long-context performance, or a world-record ranking. Variability across a large number of repeated runs has not been characterized.
The prompts and measurement implementation are provided in coding_benchmark.py. Raw results are available in benchmark-after-restart.json; the earlier run is retained in benchmark-results.json.
Verification and reproducibility
The post-restart check established that:
- All 24 Python runtime modules matched the verified snapshot byte for byte. The same files are now published individually under
runtime/, without code changes. - No additional training-data capture hook was present.
- The optimized MLP kernels were captured for all 64 target layers.
- Output hashes for all eight benchmark tasks matched the earlier run.
The check reused the server's existing weights, installed dependencies, libraries, and caches. It therefore validates the published modules in the tested environment, not installation from scratch on a separate system.
The runtime retains source-version checks, historical build paths, and environment-specific shared libraries. The original AOT cache and internal startup controllers are not included. Integration requirements and measurement caveats are documented in INSTALL.md.
Model dependencies and compatibility
Model weights must be obtained separately from the upstream repositories:
| Dependency | Source |
|---|---|
| W4A16 target base | dbirks/Qwen3.8-27B-W4A16-AutoRound |
| Fast-variant companion files | syvai/qwen3.8-27b-3090-fast-variant |
| DFlash2 W4A16 draft | syvai/Qwen3.8-27B-DFlash2-W4A16 |
The draft weight file's SHA256 matches the file used in the measurement. The target requires upstream preparation; byte-identical reconstruction of the complete tested target from these downloads has not yet been verified. Fast-variant companion files are not a standalone checkpoint. See WEIGHTS.md for provenance, pinned draft downloads, and limitations.
Compatibility is established only for the tested configuration. Other checkpoints, fine-tunes, model sizes, quantization formats, GPUs, and runtime versions have not been validated. The dependencies remain upstream work and are not newly trained M23AI weights.
Release contents
| File | Purpose |
|---|---|
| runtime/ | 24 individually accessible Python runtime modules |
| libraries/ | Two compiled CUDA libraries |
| manifest.json, README.txt | Preserved snapshot integrity manifest and technical notes |
coding_benchmark.py |
Benchmark prompts, warm-up, and measurement implementation |
benchmark-after-restart.json |
Post-restart measurements with the published runtime modules |
benchmark-results.json |
Earlier optimized-run measurements |
INSTALL.md |
Benchmark usage and runtime integration limitations |
WEIGHTS.md |
External dependencies and weight provenance |
SHA256SUMS.json |
Release-file checksums |
THIRD_PARTY_NOTICES.md, LICENSE-APACHE-2.0.txt |
Attribution and license materials |
This is not a checkpoint for from_pretrained. Review the startup integration before adding the runtime directory to PYTHONPATH: sitecustomize.py can activate hooks at interpreter startup when the corresponding flags are enabled. Checksums establish file integrity, not execution safety.
Independent evaluation
Independent evaluation on additional prompts and benchmarks is encouraged. Please submit results in the repository's Community section or in the comments accompanying this release.
Reports should identify the GPU and power limit, weight revisions, runtime versions, input/output lengths, concurrency, generation settings, throughput definition, number of repetitions, and observed variability. Where permissible, include a reproducible prompt and observations about output correctness. Installation failures and performance regressions are also relevant findings.
Do not include credentials, personal information, or confidential code. Results on other workloads should be reported under their own test conditions rather than attributed to the benchmark above.