llm-jp-4.1-33b-thinking-FP8-Dynamic

English | 日本語

An FP8 Dynamic (W8A8) quantization of LLM-jp 4.1 33B. InfiniCloud quantized and validated it because we wanted a tested, pre-quantized checkpoint for running with vLLM. It needs no calibration data, and its weights are about half the size of BF16.

Serialized weights occupy 32.823 GiB; this is not the minimum inference VRAM requirement. KV cache and runtime workspace require additional memory.

Source and quantization

• Source: llm-jp/llm-jp-4.1-33b-thinking
• Pinned revision: de581ee4353568c480d47129948d1be99f02f4ff
• Scheme: FP8 Dynamic W8A8; E4M3 weights with one scale per output channel (RTN/minmax); activations are quantized to FP8 per token at run time. No calibration data is used.
• Attention and all MLP projections are quantized. Embeddings, lm_head and norms remain BF16.
• The tokenizer files (native Unigram tokenizer, Harmony code, chat template) are byte-identical to the source revision, and EOS [2,13] is preserved.

Quality

Japanese MGSM, 250 questions, 5-shot, low reasoning effort, temperature 0, seed 42, maximum 4096 generated tokens. Official flexible-extract scoring is applied only to complete final answers; incomplete answers score zero. FP8 was measured on 2026-10-03 and the other rows on 2026-09-29, with the same procedure and environment (vLLM 0.29.0, the same GPU).

Precision	MGSM correct / 250	Accuracy	Complete final	Smoke low / medium (3 each)	Weights GiB
BF16	203	81.2%	248	2/3・2/3	61.876
FP8-Dynamic	212	84.8%	250	2/3・2/3	32.823
GPTQ-INT4-G64	203	81.2%	249	2/3・3/3	19.191
NVFP4	203	81.2%	248	2/3・2/3	20.099

FP8 answered 9 more questions than BF16 (it fixed 10 BF16 errors and missed 1 BF16 successes). This comes from one fixed-condition run and does not establish an advantage over BF16; small numerical differences can change greedy generation paths. One arithmetic benchmark does not establish general quality equivalence or superiority. See methodology and limitations.

Performance: eager execution

RTX PRO 6000 Blackwell Max-Q 96GB; vLLM 0.29.0; BF16 KV cache 8 GiB; context 16K; max_num_seqs 4; CUDA Graphs and prefix caching disabled. Random inputs and 256 output tokens, ignoring EOS; medians of three runs of 32 requests per condition. Percentiles are also medians of per-run percentiles.

Input tokens	Concurrency	Output tok/s	TTFT p50 / p95 ms	TPOT p50 / p95 ms
512	1	35.6	105.0 / 106.4	26.89 / 31.06
512	4	135.8	360.6 / 365.3	27.61 / 31.10
4096	1	31.8	728.8 / 733.5	28.00 / 31.50
4096	4	94.4	2137.5 / 2811.5	35.27 / 39.45

Output tok/s of all four formats under the same eager settings (FP8 measured on 2026-10-03):

Input tokens	Concurrency	BF16 tok/s	FP8 tok/s	GPTQ tok/s	NVFP4 tok/s
512	1	19.8	35.6	41.1	15.5
512	4	78.0	135.8	149.4	60.8
4096	1	18.0	31.8	34.8	15.1
4096	4	57.7	94.4	92.1	56.8

These are not maximum-performance claims for Blackwell or FP8.

Performance: compiled CUDA Graph execution

Same GPU and inference packages; BF16 KV cache 40 GiB, max_num_seqs 16, context 16K, max_num_batched_tokens 4096, chunked prefill enabled, prefix caching disabled. Both torch.compile and FULL_AND_PIECEWISE CUDA Graphs are enabled. Each condition uses 32 requests x 3 runs, random inputs, 256 output tokens, seed 42, and at least max(4, concurrency) warmups per run. Values are medians; p50/p95 are medians of the per-run percentiles.

Input tokens	Concurrency	Output tok/s	TTFT p50 / p95 ms	TPOT p50 / p95 ms
512	1	38.5	107.2 / 109.0	25.71 / 25.76
512	4	146.9	307.9 / 403.9	26.19 / 26.89
512	16	489.9	968.5 / 1350.1	29.24 / 31.89
4096	1	34.3	730.2 / 736.3	26.45 / 26.48
4096	4	101.2	2123.3 / 2454.0	31.37 / 36.60
4096	16	194.0	2892.2 / 10974.1	70.64 / 75.65
8192	1	30.1	1567.0 / 1582.8	27.22 / 27.26
8192	4	72.7	3835.8 / 5283.1	39.90 / 46.30
8192	16	107.9	4040.3 / 23328.6	130.39 / 136.34

Output tok/s of all four formats with graph + compile. BF16 used a 24 GiB KV cache because 40 GiB does not fit next to its weights on one 96 GB GPU; no run had KV preemption, so the smaller pool does not change the computation (BF16 measured on 2026-10-04). — marks input 8192 x concurrency 16, which does not fit in BF16 (about 33 GiB of KV).

Input tokens	Concurrency	BF16 tok/s	FP8 tok/s	GPTQ tok/s	NVFP4 tok/s
512	1	20.0	38.5	60.6	59.3
512	4	78.8	146.9	216.7	229.3
512	16	264.3	489.9	556.0	762.9
4096	1	18.2	34.3	45.8	53.7
4096	4	58.0	101.2	101.0	159.0
4096	16	119.4	194.0	140.5	298.5
8192	1	16.4	30.1	35.4	46.7
8192	4	43.9	72.7	61.3	111.6
8192	16	—	107.9	72.9	160.4

FP8 is faster than BF16 in every measured condition and falls between GPTQ and NVFP4 for long inputs at high concurrency. At concurrency 1 it is the slowest of the three quantized formats: its weights are about 1.7x the 4-bit releases, which matters when generation is memory-bandwidth bound. This does not rank the formats for every input length or concurrency.

Matched eager controls use the same 40 GiB KV cache and max_num_seqs 16:

Input tokens	Concurrency	Eager tok/s	Graph + compile tok/s
512	1	35.5	38.5
512	4	132.4	146.9
4096	1	31.5	34.3
4096	4	95.4	101.2

Across both FP8 models, all 78 runs / 2,496 measured requests passed validation, with zero KV preemptions. Graph capture completion was confirmed for both FP8 graph servers; per-request replay traces were not collected. MGSM was not rerun with graph execution. See follow-up methodology, all model results, and audit. The comparison combines compilation and CUDA Graph effects and is not a claim of maximum performance or a separate measurement of either effect.

bash reproduce/serve-cudagraph.sh InfiniCloud/llm-jp-4.1-33b-thinking-FP8-Dynamic

Pass eager as the second argument to reproduce the matched eager server settings.

Usage and validation

Tested inference: vLLM 0.29.0, compressed-tensors 0.17.0, Transformers 5.14.1, PyTorch 2.13.0, CUDA 13.0, driver 580.178.04. Native CUTLASS FP8 (W8A8) linear kernels were verified on Blackwell. Other GPUs, ordinary Transformers model loading, 64K context, and application-specific JSON schema compliance were not validated.

bash reproduce/serve.sh InfiniCloud/llm-jp-4.1-33b-thinking-FP8-Dynamic

This command reproduces the eager settings above (KV 8 GiB, max_num_seqs 4). trust_remote_code is optional. Without it, Transformers and vLLM load the tokenizer from tokenizer.json with the same token IDs; with it, the supplied Llmjp4Tokenizer is used, so review its code first. reproduce/serve.sh keeps the flag used for the published measurements; vLLM 0.29.0 was also checked to start and answer without it. Applications must separate Harmony analysis/final channels and use the supplied tokenizer, not direct openai-harmony tokenization. See the upstream cookbook.

Full numerical verification checked finite tensors, positive scales, unchanged retained BF16 weights, the FP8 dequantization error (relative, mean 2.66%, max 2.68%) and tokenizer files identical to the source. See verification.json, release inventory, and quantization reproduction instructions.

Changelog

• 2026-10-04: Initial release.

License and attribution

Distributed under Apache License 2.0, as declared by the source model card. LLM-jp 4.1 was developed by the Research and Development Center for Large Language Models at Japan's National Institute of Informatics. See NOTICE for changes and attribution, and the source model card for training data, risks, acknowledgements and contact information.