ViniciosLugli/jpt-4b-nvfp4

JPT-4B NVFP4

NVFP4 (W4A4) quantization of kirp/jpt-4b, made with NVIDIA ModelOpt. Same interface and temperature as the original; serve it with vLLM and llm2jev. Unofficial, not affiliated with the JPT-4B authors.

	JPT-4B (BF16)	NVFP4
Checkpoint	9.08 GB	3.95 GB
Weights in GPU memory (vLLM)	8.61 GiB	3.92 GiB
Latency, single request (p50)	54.0 ms	31.1 ms
Throughput, 16 concurrent requests	14.9 items/s	53.4 items/s
Macro accuracy, 16,136 items	84.43%	83.65%

Measured on an RTX 5060 Ti 16 GB with vLLM 0.30.0 and llm2jev 0.6.1.

Usage

Requires a Blackwell GPU (compute capability 10.x or 12.x). Tested with vLLM only.

1. Start vLLM:

vllm serve ViniciosLugli/jpt-4b-nvfp4 --port 8000 --max-model-len 32768 \
  --max-logprobs 256 --return-tokens-as-token-ids --enable-scale-out

2. In a second terminal, start llm2jev, which serves the decision API on port 8080:

llm2jev --model ViniciosLugli/jpt-4b-nvfp4 --backend vllm --url http://127.0.0.1:8000 --port 8080 --temperature 1.036

3. Ask a question:

import requests
r = requests.post("http://127.0.0.1:8080/v1/systemone", json={
    "state": "Refund policy: full refund within 30 days of purchase; 50% until day 60; none after.\n"
             "Order 1182 was bought on 3 March and returned on 20 April.",
    "questions": {
        "refund": {"type": "choice", "instructions": "What refund does order 1182 get?",
                   "criteria": {"full": "Full refund", "half": "50% refund", "none": "No refund"}},
        "late": {"type": "noul", "instructions": "Was the return made after day 30?",
                 "criteria": {"true": "Yes", "false": "No"}}}})
print(r.json()["answers"])

• 32,768 tokens is the maximum recommended context.
• vLLM reserves 92% of GPU memory by default. If other processes use the GPU (a desktop, for example), add --gpu-memory-utilization 0.85.
• llm2jev needs the last three vLLM flags; 1.036 is JPT-4B's calibrated temperature.
• The first vLLM start compiles FlashInfer's FP4 kernels. Set MAX_JOBS=2 if the host has little RAM.

Accuracy

Both models scored on the same held-out items with the same prompt and temperature. No benchmark item was used for calibration.

Suite	Items	BF16	NVFP4	Change (95% CI)
AG News	2,000	90.20	90.25	+0.05 (-0.45, +0.55)
Banking77	3,080	91.01	91.14	+0.13 (-0.42, +0.68)
Emotion	2,000	62.65	63.60	+0.95 (+0.05, +1.80)
IMDB	2,000	95.55	95.75	+0.20 (-0.15, +0.55)
Typed decisions	2,000	78.55	76.90	-1.65 (-2.90, -0.45)
ANLI r1-r3	1,000	80.30	78.90	-1.40 (-3.00, +0.30)
CLINC150 + OOS	1,000	93.00	93.30	+0.30 (-0.50, +1.10)
CondaQA	1,000	75.90	75.50	-0.40 (-2.30, +1.50)
JevBench v1.4 public	231	88.31	85.28	-3.03 (-5.63, -0.43)
BEV decision	1,120	73.75	73.66	-0.09 (-1.70, +1.52)
SRE triage*	252	93.65	93.25	-0.40 (-2.38, +1.59)
Agent guardrail*	252	86.11	81.35	-4.76 (-8.73, -1.19)
Ticket priority*	201	88.56	88.56	+0.00 (-3.48, +3.48)
Macro	16,136	84.43	83.65	-0.78 (-1.31, -0.26)

* Private hand-written suites. Test splits for the public datasets. CIs from a paired bootstrap over items.

• 95.2% of answers are identical to BF16. Nearly all changed answers are on items where BF16's top option is below 0.7 probability.
• The loss is concentrated in agent guardrail decisions and JevBench. Classification suites are unchanged.
• Calibration is preserved: ECE 0.016 (BF16 0.019).

Known issues

• Answers can depend on batch composition: 95.5% stay the same when batching changes (BF16: 99.5%). Use BF16 if a decision must not depend on concurrent traffic.
• Image input works (the vision tower is kept in BF16) but was only smoke-tested, not benchmarked.
• If vLLM fails at startup with 'NoneType' object has no attribute 'size' after you change --limit-mm-per-prompt, clear ~/.cache/vllm/torch_compile_cache.

Quantization

• ModelOpt 0.47 NVFP4 W4A4: FP4 weights and activations with an FP8 scale per 16 values. KV cache not quantized.
• All text-decoder linear layers are quantized. The vision tower, the tied embedding and output head, and the DeltaNet conv1d, in_proj_a and in_proj_b stay in BF16.
• Calibrated on 720 train-split prompts with nvfp4_act_headroom activation scales and local-Hessian weight scales.
• Then refined block by block: each decoder layer's weights were tuned through its quantizers to match the BF16 layer outputs, on 2,782 train-split prompts.

License

CC BY-NC 4.0, inherited from kirp/jpt-4b, which is built on Qwen3.5-4B (Apache-2.0). The text decoder weights were quantized and refined; everything else is unchanged. Non-commercial use only.