ProCreations/Nanbeige4.3-33B-EXL3

Nanbeige4.3-33B EXL3

Calibrated EXL3 quantizations of Nanbeige4.3-33B, revision c1093c501cc6460427c0ac634db7223d0450c0ee, under Apache-2.0. The main branch contains this README; model files are on the eight bitrate branches. Labels describe the decoder target bitrate. Every rung has a six-bit output head, unquantized embeddings and its own sensitivity allocation.

Quality and speed at a glance

All eight rungs have complete official held-out evidence, 298 required core trials and native default checks 4/4. Quality warnings are retained rather than dropping requested compression levels. The initial speed measurements below are Modal RTX PRO 6000 Server Edition, 600 W, nativeQ8 KV. 4.05 and 5.05 were measured on a Vega workstation instead (RTX PRO 6000 Workstation Edition, 580 W), with and without the DFlash2 drafter; see DFlash2.

The token-use chart is an off-policy prediction from teacher-forced stop mass; measured generation length for 4.05 is reported under Generated length and matched agentic control. Exact values for every chart are in the tables.

Branch	Weights GiB	Assistant KLD	Agentic KLD	Chinese KLD	Decision KLD	Stop pooled / agentic	Core / quality notes	Modal single decode tok/s
2.05bpw	10.410	0.17679383	0.04669882	0.28587402	0.13189686	0.929178 / 0.938891	Warning: all stop mass 0.929, outside quality band; agentic stop mass 0.939, outside quality band	84.718
2.55bpw	12.199	0.09221438	0.02052682	0.14516585	0.04448751	0.972300 / 0.957319	Warning: agentic stop mass 0.957, outside quality band; core regression: agentic_coding_on	78.861
3.05bpw	13.987	0.04811491	0.01026654	0.07415080	0.02531030	0.981139 / 1.009003	Warning: core regression: agentic_coding_on	70.795
3.55bpw	15.774	0.02509275	0.00556376	0.03875037	0.01225279	0.990975 / 1.019516	Pass	65.095
4.05bpw	17.563	0.01419367	0.00330054	0.02194952	0.00589713	0.988804 / 0.976797	Warning: core regression: chinese	Vega: see DFlash2
4.55bpw	19.353	0.00758917	0.00213971	0.01205522	0.00420690	0.993334 / 0.991095	Pass	56.536
5.05bpw	21.140	0.00440525	0.00151053	0.00697160	0.00216656	1.004557 / 1.010142	Pass	Vega: see DFlash2
6.05bpw	24.717	0.00189574	0.00106824	0.00302480	0.00119133	0.992302 / 0.990592	Pass	46.565

Low-rung warnings: 2.05 has pooled/agentic thinking-stop mass 0.929/0.939, both 95% CIs below the 0.97 quality floor. 2.55 has agentic stop mass 0.957 and coding-on accuracy difference -8.89 pp [95% CI -18.89, -1.11]. 3.05 has coding-on difference -11.11pp [-21.11, -3.33]. These rungs remain available but are not recommended for agentic/coding work. A smaller thinking-stop mass suggests longer thinking under an inverse-hazard proxy; it is not a measured generation-length increase.

4.05 Chinese caveat: original 37/46 versus BF16 43/46; fresh selected/alternative/BF16 each 84/90. The combined paired contrast remains -4.72 pp [-9.72, -0.28], with a narrow single-template and multiple-testing scope. Full replication details follow.

Which rung should I use?

GPU VRAM	Rung	Total context assumed	Weights + FP16 KV	Headroom for runtime / workspace	Reason
24 GiB class	4.05	16K, batch1	17.56 +3.25 =20.81 GiB	3.19 GiB	Good quality with useful context; Chinese replicate disclosed
32 GiB class	5.05	32K, batch1	21.14 +6.50 =27.64 GiB	4.36 GiB	Lower KLD, stop/core pass
48 GiB or larger	6.05	64K, batch1	24.72 +13.00 =37.72 GiB	10.28 GiB on48GiB	Lowest measured KLD, stop/core pass
20 GiB class, memory-bound	3.55	4K, batch1	15.77 +0.81 =16.59 GiB	3.41 GiB	Quality compromise when4.05 cannot fit
Aggressive compression	2.05 /2.55 /3.05	Depends on available KV memory	See branch cards	Reserve runtime and KV capacity	Not recommended for agentic/coding work

These are capacity-planning recommendations from measured weight bytes and the model's exact KV geometry, rather than tests on each consumer GPU. FP16 KV costs 208 KiB/token: 16K is 3.25 GiB, 32K 6.5 GiB, 64K 13 GiB. Context means prompt plus generated tokens in the total cache pool, not just prompt length. The estimates assume one active sequence, chunked prefill and explicit FP16 KV; keep the listed remainder for CUDA/PyTorch, graphs and workspace. Reduce the pool if your environment needs more headroom. These estimates cover the target model; a drafter needs additional weights, cache and workspace.4.55 has better KLD than 4.05 but leaves only about 1.40 GiB at 16K on a 24 GiB GPU, so 4.05 is the safer useful-context recommendation. Q8 KV can extend capacity, but the FP16 quality evidence does not validate quantized-cache quality.

Calibration and held-out measurement

Converted with stock exllamav3 v1.6.4, commitda68d57de61781e5b74ad68698ac4caf36408d7a, native LDLQ/trellis, mul1 and automatic output scales. Each rung uses a native supported-rate tensor allocation optimized from 80% assistant thinking/answer/tool-call KLD plus 20% response-decision KLD, domain weighted. Nine complete measurement contexts contain 7,525 assistant positions, with 346 Hessian rows and one noise draw; this proxy is not a repeat-stability claim.

Calibration source: 1,213 complete Nanbeige BF16-weight/BF16-KV conversations, 4,054,730 tokens, rendered in its own template through actual EOS. Domain tokens: agentic coding/tool trajectories 34.96%, other code 14.98%, math 11.99%, chat 11.99%, Chinese 8.00%, multilingual 4.99%, structured 4.99%, long context 5.10%, STEM 3.00%. The measured-selected subset is 310 complete conversations / 1,044,043 tokens, max 28,366, seed 20261025. Manifest SHA256 4305272fa895081ac90038f656cf3eec2e40cedf17edc3110e459304abc12191; packed calibration SHA256 469eea04a01461763fcbe4710f352bae2793de6d692a685a4899ff46045adb56. Evaluation families are disjoint.

Headline KLD is KL(BF16||quant), nats, on 634,268 assistant positions including assistant history, with 1,523 assistant decisions. Eval-v1 has 272 complete conversations / 1,006,589 tokens; manifest SHA256 4fe31e47a0b7426e1e3f89fa4945343561681f5112ef125cd43958cac3556fe5. Quant scoring is native full-sequence uncached FP16 attention, no KV cache; reference is stored SGLang BF16-weight/BF16-KV EXTEND. The full padded 166,912 vocabulary is normalized. BF16 top 256 + aggregate tail is a coarsened-KLD lower bound; assistant-only full-vocabulary validation is retained per branch. Prompt/schema/user/tool-output positions are excluded from the headline. Generation uses native paged FP16 KV and is a distinct execution mode.

Rung	agentic	code	math	chat	chinese	multilingual	structured	longctx	stem
2.05	0.04669882	0.10374227	0.12284800	0.29729469	0.28587402	0.25287467	0.08894046	0.07615026	0.30049739
2.55	0.02052682	0.05339256	0.06368523	0.15580690	0.14516585	0.14386659	0.04286376	0.03352771	0.16471419
3.05	0.01026654	0.02753207	0.03342256	0.08121627	0.07415080	0.07674444	0.02250903	0.01802247	0.08924217
3.55	0.00556376	0.01459736	0.01736176	0.04202985	0.03875037	0.04040878	0.01182095	0.01010047	0.04537105
4.05	0.00330054	0.00802932	0.00984379	0.02349686	0.02194952	0.02363558	0.00655424	0.00566641	0.02615745
4.55	0.00213971	0.00429337	0.00510951	0.01226188	0.01205522	0.01296309	0.00346492	0.00310685	0.01350913
5.05	0.00151053	0.00265012	0.00298885	0.00694687	0.00697160	0.00697065	0.00211004	0.00211246	0.00779926
6.05	0.00106824	0.00124594	0.00132932	0.00276628	0.00302480	0.00259881	0.00100173	0.00116499	0.00296153

Stop fidelity

The recipe-selection quality band is [0.97, 1.03] for canonical pooled and agentic stop-mass point ratios. All requested rungs remain published with prominent deviations and core-regression warnings. 95% conversation-cluster CIs and the native-runtime BF16 control are reported alongside. Stop mass concerns current BF16 response positions; assistant history is excluded. Teacher-forced hazards do not establish generated length.

Rung	Canonical pooled [95% CI]	Canonical agentic [95% CI]	Native pooled [95% CI]	Native agentic [95% CI]
2.05	0.929178 [0.913978, 0.943770]	0.938891 [0.915048, 0.959237]	0.930091 [0.914949, 0.944400]	0.937762 [0.913747, 0.958363]
2.55	0.972300 [0.963441, 0.980896]	0.957319 [0.940144, 0.972926]	0.973255 [0.964805, 0.981503]	0.956168 [0.939187, 0.971718]
3.05	0.981139 [0.973846, 0.988279]	1.009003 [0.994083, 1.023641]	0.982103 [0.975012, 0.988772]	1.007790 [0.995169, 1.019833]
3.55	0.990975 [0.985766, 0.995982]	1.019516 [1.009426, 1.030100]	0.991949 [0.987132, 0.996689]	1.018290 [1.008922, 1.027797]
4.05	0.988804 [0.984841, 0.992812]	0.976797 [0.966610, 0.986314]	0.989775 [0.985909, 0.993455]	0.975623 [0.966475, 0.983963]
4.55	0.993334 [0.989595, 0.996903]	0.991095 [0.983654, 0.998290]	0.994310 [0.990894, 0.997482]	0.989903 [0.985306, 0.994417]
5.05	1.004557 [1.002075, 1.007277]	1.010142 [1.004224, 1.016815]	1.005543 [1.003696, 1.007643]	1.008927 [1.005359, 1.012892]
6.05	0.992302 [0.989654, 0.994999]	0.990592 [0.984326, 0.996767]	0.993276 [0.991617, 0.994932]	0.989401 [0.986047, 0.992506]

Generated length and matched agentic control

The selected4.05 expanded arm completed 500/500 trials with 0 infrastructure errors. At an8,192-token restriction, the pooled length ratio is partially identified in[0.978725, 1.045298], with95% bootstrap bound envelope[0.945861, 1.095004]. Cap hits were 79/500 versus BF16 84/500; risk ratio 0.940476 [0.847059, 1.037500]. There were 16 quant time-budget censors versus 4 BF16 early censors. The native quant mixed queue used concurrency32, whereas retained BF16 agentic controls used4. This walltime/load mismatch prevents attributing the difference to longer thinking. All500 original rows remain retained. Restricted mean concerns min(completion length,8192), not unrestricted completion length. These results do not establish ±3% length equivalence.

The separate matched agentic rerun is complete: 30 fixed coding-agent tasks ×2 seeds, 60 valid pairs, concurrency4 in both arms, 8,192 output tokens, 12 turns and 300 seconds per task. Existing BF16 controls were retained after fixture/parser/protocol verification. Native generation used paged FP16 KV. At 8,192 tokens, the restricted-mean length ratio has identification bounds [0.977195, 1.040305] and a 95% paired prompt-cluster bootstrap bound envelope [0.793257, 1.277307] (4,000 draws). Early censors are 1/60 quant versus 4/60 BF16. Output-cap hits are 2/60 versus 1/60; rate-difference 95% CI [-5.00, +10.00] percentage points. The cap-hit risk-ratio confidence interval has an unbounded upper endpoint. This replaces the concurrency-mismatched agentic comparison for interpretation, while preserving all original rows separately. It does not establish unrestricted-length or statistical equivalence.

Matched hidden-test accuracy is 58/60 for EXL3 and 58/60 for BF16. The paired difference is +0.00 percentage points [95% prompt-cluster CI -6.67, +6.67]. These are existing short30 hidden-test grades, summarized without new inference or rerunning tests; source/protocol SHA256 values are retained in EVAL_METRICS.json. Equal observed scores do not establish accuracy equivalence.

Chinese arithmetic replication, selected EXL34.05

This probe contains 30 numeric variants of one Chinese arithmetic template (20 with thinking on, 10 off), rather than broad Chinese language coverage. Sampling and prompts were identical across arms: T=0.6, top_p=0.95, 1,536-token cap. The fresh run used three new seeds (20261201/02/03), 90 trials per arm. All 360 fresh trials completed without infrastructure errors.

Arm	Original seeds: pass/trials	Fresh seeds: pass/trials
Canonical SGLang BF16	43/46	84/90
Selected corrected assistant-only EXL34.05	37/46	84/90
Alternative optimized1M EXL34.05	Not run	84/90
Native exllamav3 BF16-weight control	22/24	83/90

The selected recipe's original nine failures and canonical BF16's three failures were wrong numeric answers, with no language-switch, format/grader-parse, truncation or thinking-only failures. Selected failures covered seven prompts; one prompt failed all three original seeds. Detailed per-prompt counts and failed answers are retained in CHINESE_REPLICATE_AUDIT.md.

Accuracy differences below average the paired seed differences within each prompt, then weight the 30 prompts equally. The 95% confidence intervals use 4,000 paired prompt-cluster bootstrap draws; repeated seeds are not independent prompt samples.

Selected recipe comparison	Paired trials	Difference, percentage points [95% CI]
Original vs canonical BF16	46	-15.56 [-28.89, -4.44]
Fresh vs canonical BF16	90	0.00 [-6.67, +6.67]
Original+fresh vs canonical BF16	136	-4.72 [-9.72, -0.28]
Original+fresh vs native BF16 control	114	-1.11 [-6.11, +3.89]
Fresh vs alternative EXL3 recipe	90	0.00 [-4.44, +4.44]

The fresh run did not reproduce the original gap. The combined canonical comparison remains marginally negative, while the same-runtime control comparison includes zero. Neither establishes equivalence. Around 30 artifacts by eight suites were examined: these nominal confidence intervals are unadjusted and do not control family-wise error across exploratory comparisons. The fresh contrasts were fixed before observing their results.

After this replication the selected recipe was retained. Native exllamav3 generation uses its pinned FP16 loading/cache path and native RNG; canonical SGLang uses BF16 weights/cache. Exact source hashes, runtime configurations, all results and the dated ruling are preserved in the evaluation scoreboard and CHINESE_REPLICATE_AUDIT.json.

Combined raw selected/canonical totals are 121/136 versus 127/136. Prompt-weighted paired estimates differ from pooled trial proportions because prompts receive equal weight. These probes do not establish broad Chinese equivalence.

Stock runtime compatibility

Every rung loads and decodes on unpatched v1.6.4 and v1.6.5, with identical cold/warm greedy outputs within each version. Cross-version token identity is not guaranteed. Full-vocabulary cross-version KL is at most 1% of the same fixed-bank quant-vs-BF16 KL on every rung. The control uses 20 positions on three short prompts and an actual BF16 Transformers4.57.6/Torch2.10 SDPA/noKV reference, separate from the official SGLang held-out scores. Scoring is uncached/noKV; greedy is native paged FP16 KV. Emitted IDs match the captured argmax. The Chinese 4.05 difference is 8/14 ULPs, not a one-ULP tie, with cross-version KL about 8e-6, far below quantization error.

Rung	Stock1.6.4 /1.6.5	Cross-version KL	Cross/min quant-BF16 KL	First divergence margins,164/165
2.05	Pass / pass	4.3455834e-06	6.5667594e-06	after25 tokens: 1/5 FP16 ULPs
2.55	Pass / pass	2.1109733e-05	0.00018874932	Greedy identical
3.05	Pass / pass	1.8092454e-05	0.00047814972	after10 tokens: 6/4 FP16 ULPs
3.55	Pass / pass	0	0	Greedy identical
4.05	Pass / pass	7.6832958e-06	0.00032967561	after29 tokens: 8/14 FP16 ULPs
4.55	Pass / pass	5.2476548e-06	0.00036026626	Greedy identical
5.05	Pass / pass	0	0	Greedy identical
6.05	Pass / pass	3.5239809e-06	0.00083363689	Greedy identical

Install and generate

Both stock releases were tested: v1.6.4 (da68d57de61781e5b74ad68698ac4caf36408d7a) and v1.6.5 (cdc785db89e263e65e5849cb53e1ada156411414), with Python 3.12, PyTorch 2.10.0 and CUDA 12.8. Choose either matching wheel:

python3.12 -m venv exl3-venv
exl3-venv/bin/pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
EXL3_VERSION=1.6.4  # 1.6.5 is also tested
exl3-venv/bin/pip install "https://github.com/turboderp-org/exllamav3/releases/download/v${EXL3_VERSION}/exllamav3-${EXL3_VERSION}+cu128.torch2.10.0-cp312-cp312-linux_x86_64.whl" transformers==4.57.6 sentencepiece==0.2.1 protobuf==6.33.5 huggingface_hub

This batch-one example uses the recommended 4.05 branch with a 16K total FP16 KV pool. Prompt plus generated tokens must fit the pool.

from huggingface_hub import snapshot_download
from exllamav3 import Cache, CacheLayer_fp16, Config, Generator, Model, Tokenizer, TopPSampler
from transformers import PreTrainedTokenizerFast

directory = snapshot_download("ProCreations/Nanbeige4.3-33B-EXL3", revision="4.05bpw")
config = Config.from_directory(directory)
config.override_dynamic_seq_len(2048)  # Chunked prefill workspace.
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=16384, layer_type=CacheLayer_fp16)
model.load(device="cuda:0", max_batch_size=1)
tokenizer = Tokenizer.from_config(config)
hf = PreTrainedTokenizerFast.from_pretrained(directory, local_files_only=True)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer, max_batch_size=1)
prompt = hf.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that adds two integers."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
answer = generator.generate(
    prompt=prompt, max_new_tokens=512, stop_conditions=[166101],
    sampler=TopPSampler(top_p=0.95, temperature=0.6), seed=20261025,
    encode_special_tokens=True, add_bos=False, completion_only=True,
)
print(answer)

The tokenizer JSON is BPE despite the original LlamaTokenizer class declaration; explicit PreTrainedTokenizerFast avoids that ambiguity. Preserve EOS 166101, padded vocabulary 166912, the supplied chat template and RoPE theta 120000000. For agentic/tool trajectories, preserve thinking history, provide the tool schema and use the model's XML tool-call format. The source model recommends T=1.0/top_p=0.95 for agentic work and T=0.6/top_p=0.95 for chat/reasoning.

DFlash2

The Nanbeige4.3-33B-DFlash2 draft model has a native exllamav3 branch for these quants. It uses stock exllamav3 v1.6.4, and its card has a copy-paste quickstart that was run unmodified on the 4.05bpw branch. The draft shares this target's embedding and 6-bit head. exllamav3 samples the target and accepts matching draft tokens, so output follows the target's own sampling.

Single-stream decode on Vega, an RTX PRO 6000 Workstation Edition at 580 W: stock exllamav3 v1.6.4, Q8 target KV and FP16 draft KV, 25 prompts (five each from agentic, code, math, chat and Chinese), 256-token caps, plain and DFlash2 in the same process.

Branch	Temperature	Plain decode tok/s	DFlash2 decode tok/s	Speedup	Agentic	Slowest domain (Chinese)
4.05bpw	0	66.961	101.084	1.51×	2.16×	1.18×
4.05bpw	1, top_p 0.95	67.291	98.277	1.46×	2.00×	1.11×
5.05bpw	0	58.354	90.626	1.55×	2.25×	1.18×
5.05bpw	1, top_p 0.95	58.246	85.792	1.47×	2.04×	1.13×

Every domain was faster than plain decoding on both branches at both temperatures. Per-domain rates and acceptance lengths are on the exllamav3 branch card. These short-prompt Vega rates use a different machine and protocol from the Modal table below, so don't compare the two tables row for row. Other rungs should work with the same draft but were not measured.

Measured speed on Modal

Modal RTX PRO 6000 Server Edition, 600 W; raw rates, no Vega normalization. Stock exllamav3 v1.6.4 (da68d57), Torch2.10/cu128, native internal graphs with warmed batch shapes, no source patch. Cache is native Q8 K8/V8, stored as packed int32 with FP16 scales. These speed results are distinct from official FP16-KV quality checks and do not validate Q8-cache quality. Full artifact-file hash checks passed before loading. All rates below are tokens/s.

Rung	Single decode	32K decode	128K decode	250K decode	8K input / TTFT	Conc32 aggregate	Conc64 aggregate	Conc128 aggregate	Cases
2.05bpw	84.718	68.599	44.916	31.907	3876.481	663.301	632.158	Not completed	18/19
2.55bpw	78.861	64.372	43.526	31.092	3849.848	664.050	638.552	Not completed	18/19
3.05bpw	70.795	59.126	40.548	29.780	3881.334	645.456	623.570	Not completed	18/19
3.55bpw	65.095	55.420	39.095	28.952	3887.216	662.958	641.276	Not completed	18/19
4.05bpw	Not run on Modal								See DFlash2 (Vega)
4.55bpw	56.536	48.856	35.662	26.866	3900.273	639.370	627.466	729.172	19/19
5.05bpw	Not run on Modal								See DFlash2 (Vega)
6.05bpw	46.565	41.322	31.524	24.396	3793.415	603.110	597.053	700.933	19/19

Decode uses first-to-last token arrival; concurrency rates use total output / wall time including prefill. The 8K input/TTFT rate includes head, scheduling and host overhead and is not pure prefill-kernel throughput. The four lower-rung sweeps completed 18/19 cases, then reached the 780-second bound during concurrency128; that rate is missing and is not estimated. Their completed cases and native counts are verified. Rungs4.55/6.05 completed19/19. Exact raw samples, runtime/settings and manifest hashes remain with the serving exports. 4.05 and 5.05 were not run on Modal; their Vega measurements are in the DFlash2 section.

Files and integrity

Each model branch includes its own card, exact allocation, calibration/quality/compatibility evidence, LICENSE/NOTICE, FILELIST.json and MANIFEST.sha256. Weight, configuration and tokenizer hashes are unchanged from the tested artifacts. Internal run logs and workstation metadata are excluded from the public payload. The branch manifest lists every payload file and its SHA256; the manifest's own fingerprint is recorded by the release receipt.

Related releases: FP8, NVFP4-A16 and DFlash2. The base model's capabilities and limitations apply.

Frozen download sizes

Branch	Files	Total bytes	MANIFEST.sha256 fingerprint
2.05	23	11,200,207,921	a8ae4af5dc562b35d099110613471ae6e85d6d882f02d171957f669bcf11d0f2
2.55	24	13,120,542,538	7dc23d5d728a83a20b3cc590e85ed2b149811be7bd58805ab705927403eb8522
3.05	24	15,040,878,056	8920b9fcc8c80e73a823f63478645ad574f4825a7209ae0abced9c4c5f31e8da
3.55	25	16,959,376,550	e7ab653167dc39c82436c24e773966c2bbdc5993d68d2305ffc89fff2aab88a3
4.05	25	18,880,183,498	022877db3f0c0724d8853b9735a79ce03529c5df5e099ba25dcdd62c6e75cfda
4.55	25	20,801,883,280	c3367a357e485a183e84bad2db40554f6aff39e0e7378961edbd5cf931f5b8b8
5.05	26	22,720,838,831	c473d0d5b970cd4e5b2a4ee90c6a8ed5041f163d4343a70b032439696ac563ae
6.05	27	26,561,515,595	9e9fc513360bb6fc9307fedf5e3608709bdf813a534d439a17f46c1210672d5b