JEV-Qwen3.8-27B-NVFP4 Runtime
A Jev-style closed-choice runtime over unchanged upstream Qwen3.8-27B mixed NVFP4/FP8 weights. This release includes a compact original BF16 output head and code/configuration; the backbone weights are downloaded from upstream. Not affiliated with TypeSafe Jev.
The three upstream safetensors shards (21,921,697,280 bytes) were SHA256-verified against RadixArk/Qwen3.8-27B-NVFP4, revision 52d1adc5f38aa5ebf099c29ed7025ba34cfbb854. There is no LoRA, additional weight training, or fitted temperature.
Original BF16 linear update (2026-09-22)
The default runtime now removes the full vocabulary head and uses an 86 x 5120 BF16 linear head. Its weights are exact original BF16 rows, not NVFP4 weights converted back to BF16, from Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. The NVFP4/FP8 backbone is unchanged. The included small head is sufficient; download the backbone with python download_weights.py, then run MAX_JOBS=3 python serve.py. Set QWEN38_LINEAR_HEAD='' for the historical full-head path.
New public result: 194/231 (83.98%), p50 97.72 ms, p95 190.03 ms. The historical full-head result was 192/231; the preliminary dequantized head got 190/231 and is superseded. These are single runs, not an official full-suite JevBench Score. No training or calibration was added. The historical section below describing 192/231 refer to the historical full-head release.
Full comparison · Reproduction · Head provenance · Final results. Runtime tested using the SGLang versions recorded above.
Exact evaluated configuration
The local config differs from upstream in text_config: max_position_embeddings=1048576; rope_parameters.rope_type="yarn", factor=4.0, original_max_position_embeddings=262144. The complete evaluated config is runtime-config.json. Preserve it to reproduce the score. These are positional configuration changes, not a claim that this release was tested at 1M context. The benchmark service used context_length=4096, reserving one token for output.
The default runtime pools the final token hidden state and projects it through the compact 86-row original BF16 head, then normalizes over valid options. The full vocabulary head is removed. Setting QWEN38_LINEAR_HEAD='' restores the historical full-head one-token generation/logprob path. There is no multi-token reasoning or generated probability text.
Historical full-head public JevBench self-test
| Tier | Correct | Accuracy |
|---|---|---|
| Easy | 48/48 | 100.00% |
| Standard | 67/72 | 93.06% |
| Hard | 77/111 | 69.37% |
| Overall | 192/231 | 83.12% |
Aggregates, per-item results, manifest, and exact public requests/responses in evaluation/raw/. Benchmark commit 75e6224ed8103bbc3485ca74820a2eaf7ce8abe0. Public items only; not an official full-suite JevBench result or rank. No benchmark-specific tuning or calibration was performed in this run. Probabilities are not guaranteed calibrated. This model has not been submitted for official benchmark inclusion as part of this release.
Run
Requires a compatible NVIDIA GPU with native NVFP4 support (tested on RTX PRO 6000 Blackwell). No H100 compatibility claim. Tested environment: SGLang commit e161bd1265a0082478b7f1c09f224a52d315dc71, torch 2.13.0/CUDA 13.0, transformers 5.12.1, FlashInfer 0.6.17, compressed-tensors 0.18.0; FastAPI, Uvicorn, Pydantic 2, huggingface_hub also required. SGLang internal APIs may differ on other revisions.
Download this repository, then inside it:
python download_weights.py
MAX_JOBS=3 python serve.py
download_weights.py downloads the pinned upstream revision to ./weights and applies the evaluated config only to that downloaded directory. It does not modify the global HF cache. CHOICE_MODEL_PATH overrides the model directory; PORT defaults to 8237. JEV_GPU_MEMORY_FRACTION defaults to 0.65; choose a safe budget for your host. No server is launched by the download script.
curl http://127.0.0.1:8237/v1/choice -H 'Content-Type: application/json' \
-d '{"question":"请选择猫","options":["狗","猫"]}'
2–86 options. The scored input template is in jev/core.py. This packaged service retains the evaluated question/options path; the unrelated Snake-specific raw-prompt input path was removed. Local paths and port/memory settings were made configurable; weights, scoring and question rendering are unchanged. Packaging was checked without launching another GPU model.
Reproduce the benchmark
git clone https://github.com/fstandhartinger/jevbench.git
cd jevbench
git checkout 75e6224ed8103bbc3485ca74820a2eaf7ce8abe0
export JEVBENCH_REPO="$PWD"
cd ..
python run_benchmark.py --out fresh-run --delay 0.5
Run the last command from this runtime repository (or give its full path). --out must be new. The HTTP client uses no GPU, runs one request at a time, and pauses requests below its free-memory threshold. It never stops training. The original run used zero pacing delay and a 2048 MiB guard as recorded in the manifest; change these only as appropriate for your host. Cost is unknown, not zero. The standard pipeline/AutoModel loader alone does not implement this decision interface.
Attribution
Qwen model: Qwen team. Upstream quantization: RadixArk/NVIDIA Model Optimizer. Runtime adaptation: Apache-2.0. Benchmark and public authored task licensing remain JevBench's own terms. See LICENSE.