Shockem/Qwen3.8-27b-Terse-Coder-NVFP4

Qwen3.8-27B Terse-Coder — NVFP4

NVFP4 (modelopt W4A16) quantization of Shockem/Qwen3.8-27b-Terse-Coder, a fine-tune of Qwen/Qwen3.8-27B with ~1/10 the chain-of-thought reasoning tokens on coding tasks and correctness preserved. This is the tested deployment artifact — every number below was measured on this checkpoint.

Actively researched and improving. Expect updated quants on this page as the study continues.

Background

Qwen-3.8-27b is an excellent dense model. It's a breakthrough in locally hosted models on consumer-grade hardware. The biggest challenge it faces is that its reasoning can be, at times, overly verbose. This isn't necessarily an issue, as it can pull itself out of hallucinations, but with user hardware around this model size, it ends up causing long waits, oftentimes into hours, before any outputs or edits occur.

Recently, agentionai released Signal-3.8-27B. This opened the gates to the idea of lowering reasoning by finetuning the model, rather than handling it via templates or configuration. The outcome was a model that thought significantly less than the base model, with marginal differences in error. This sent me down a rabbit hole of testing its reasoning and how it affected the model's output. To my surprise, it was incredibly close to the base, even its traces were very close, just cleaner overall.

My experiment is to continue this research and push it further. So far, I've trained on traces from Signal, the base NVIDIA provided NVFP4 quant, and my own abliterated variant, against HumanEval, 600 questions per round. This is now round 7, and it has shown significant improvement. We are now at a 95% reduction in reasoning against the NVIDIA quant, and 52% against Signal. This was originally created as a LoRA adapter that is then merged into a custom recipe for Qwen-3.8-27b. I have provided both a merged model and a LoRA adapter. Thank you for testing and providing feedback!

Results

Held-out 40 coding problems (20 HumanEval + 20 MBPP-sanitized, disjoint from training), vLLM 0.28 on 2× RTX 5060 Ti 16 GB, MTP spec decode on, sampling temp 0.6 / top_k 20 / top_p 0.95 / rep-penalty 1.05, pass@1 by automated test execution:

Model (all NVFP4)	pass@1	Reasoning tokens / problem	Wall tok/s
nvidia/Qwen3.8-27B-NVFP4 (stock)	72.5%	~701	54.1
This model	67.5%	~38 (−95%)	54.5

Runs at stock-base wall speed with MTP acceptance 0.412 — the reasoning cut is free end-to-end. Independent benchmarks (NVFP4 quant, vLLM 0.28, thinking on, house sampling; reasoning = completion_tokens_details.reasoning_tokens):

Benchmark	Score	Reasoning tokens (mean / median)
GSM8K (n=200)	98.0%	84 / 72
GPQA-Diamond (full 198)	78.3%	1,485 / 969
CRUXEval-I (full 800, input prediction)	92.1%	197 / 83
CRUXEval-O (full 800, output prediction)	92.9%	146 / 96
HumanEval+ (164, official EvalPlus, greedy)	90.2% (93.9% base)	43 / 28
MBPP+ (378, official EvalPlus, greedy)	78.6% (92.9% base)	91 / 25

CRUXEval was run with the official Meta harness (direct prompts, official extraction, exec-based scoring, temp 0.2) — code understanding (input/output prediction), complementing the generation-side coding table above.

A note on GPQA-Diamond: this is where a terseness fine-tune is supposed to bleed — PhD-level science, far outside the coding training distribution, where long deliberation is the whole game. Holding 78.3% at ~1.5k mean reasoning tokens (thinking models typically burn 10–20k here) means the training cut the deliberation budget, not the capability — the model still scales effort up on hard problems (median 969 → max 16k) instead of answering blindly fast.

Internal agentic harness (30 tests across easy/medium/hard — instruction following, coding, reasoning, compaction handoff, tool/JSON contracts — ×10 runs each, this checkpoint served by vLLM): easy 100% (40/40), medium 100% (90/90), hard 100% (140/140), zero truncations, zero reasoning fallbacks. Prior best on the same harness was 100/100/98.7.

Quantization recipe

This is a v3-recipe house quant, built to preserve the adapter effect through 4-bit compression:

• modelopt 0.45 W4A16 NVFP4, per-tensor streaming PTQ (the same 400-tensor quantize set + ignore list as the published house Signal quants)
• FP8 attention (absmax — byte-matches NVIDIA's checkpoint at 97–99%)
• Local-Hessian-weighted calibration on MLP + lm_head (Hessian captured from 2048 house-traffic chunks; Hessian-weighted MSE scale solve with per-block e4m3 bracketing). This matters: an absmax-calibrated quant of the same weights attenuates the terse-reasoning effect to roughly half (−49.5% vs −92.4% cut measured). Geomean Hessian-weighted error ratio 0.805 vs the absmax baseline on the stock base.
• MTP draft stack included (1 MTP layer, BF16, vocab-truncated 40960-id draft head) so speculative decoding works out of the box.

Serving (vLLM, tested path)

vllm serve Shockem/Qwen3.8-27b-Terse-Coder-NVFP4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --kv-cache-dtype fp8

Turn MTP spec decode on — outputs are target-verified (lossless) and acceptance is 0.41. If you serve with spec decode, make sure the generation config has no min_p — vLLM 0.28 rejects min_p under spec decode.

Recommended sampling (mirrors testing): temp 0.6, top_k 20, top_p 0.95, repetition_penalty 1.05.

On 2×16 GB cards cap context at ~200k with a ~3.9 GiB FP8 KV pin; single-card 24 GB+ rigs are unaffected.

Notes

• Do not stack the Terse-Coder adapter on this checkpoint — the preference is already merged in; double application over-shortens reasoning (63% pass with no_code failures).
• The fp16 source weights are at Shockem/Qwen3.8-27b-Terse-Coder if you want to quantize differently or merge further.
• Behavioral edit, not a knowledge edit — targeted at coding with thinking enabled. Should work on other backends (SGLang, TabbyAPI/EXL3), but only vLLM has been measured; validate before relying on them.

Attributions & licenses

This checkpoint is a quantized derivative of Shockem/Qwen3.8-27b-Terse-Coder, itself a derivative of Qwen/Qwen3.8-27B, © Qwen Team, Alibaba Cloud, licensed Apache 2.0; this checkpoint remains Apache 2.0 and the original license and copyright notices are retained. Credits:

• Qwen Team (Alibaba Cloud) — the Qwen3.8-27B base model (Apache 2.0).
• NVIDIA — TensorRT Model Optimizer 0.45 (Apache 2.0) drove this NVFP4 quantization; NVIDIA's published Qwen3.8-27B-NVFP4 checkpoint informed the Hessian-calibrated recipe.
• agentionai and p-e-w (Heretic) — Signal and a heretic-ara variant were two of the three trace-generation policies in the upstream adapter's preference data.
• OpenAI (HumanEval, MIT) and Google (MBPP, CC-BY 4.0) — prompt sources for training and held-out evaluation.
• Hugging Face TRL (Apache 2.0) — DPO training; Datacurve — DeepSWE, independent evaluation only.

None of these parties endorse this model; all remaining errors are ours.