Tdamre/MiniCPM5-2B-LiteRT-LongContext

MiniCPM5-2B LiteRT — 16K / 32K / 64K context

Long-context LiteRT-LM (.litertlm) conversions of openbmb/MiniCPM5-2B. The upstream checkpoint supports up to 131,072 positions; this repository provides three separate fixed KV-cache builds at exactly 16,384, 32,768, and 65,536 tokens.

Quality summary (2026-09-10): direct / thinking-off quality is close to the original BF16 checkpoint (90/100 vs 91/100 on the first 100 GSM8K test questions), and four-needle retrieval passed at ~14K, ~28K, and ~56K accumulated context. Thinking-on is not at BF16 parity in INT4: a matched 2,048-token reasoning test failed to finish and produced the wrong extracted answer, while BF16 finished correctly. For these INT4 files, thinking-off is currently the recommended mode when correctness and predictable latency matter.

Files

Variant	KV-cache context	Bytes	SHA-256	File
16K	16,384	1,561,643,360	e2f12291af5239197b2a8801cf5acf6a9e029f8adcefb9804a7b295fcf378406	MiniCPM5-2B-LiteRT-INT4-16k.litertlm
32K	32,768	1,561,643,360	a45af218d1cb4a67f3d0eab08dbcefdfbd3904ae6691e0c80a29f6d779d916a8	MiniCPM5-2B-LiteRT-INT4-32k.litertlm
64K	65,536	1,561,643,360	27727f7ae1ae21a9c62fe6f7a505563c6b77e77e7e147e84e2b9a0716b8c77da	MiniCPM5-2B-LiteRT-INT4-64k.litertlm

All variants use the same prefill-signature ladder: 1024, 256, 64, 16, 4, 1.

Benchmark results

Benchmarks below compare the converted artifacts against the original openbmb/MiniCPM5-2B BF16 checkpoint. They are intended as conversion-parity checks, not claims of full benchmark coverage.

GSM8K — thinking off

Protocol: first 100 examples from the official GSM8K test split; zero-shot step-by-step suffix; the upstream MiniCPM5 chat template; enable_thinking=false; greedy decoding; maximum 2,048 new tokens; identical numeric answer extraction on both sides.

Configuration	Score	Delta vs BF16
Original MiniCPM5-2B BF16	91 / 100	—
LiteRT INT4 16K	90 / 100	-1 pp

Paired analysis across the same 100 examples:

• 91/100 predictions were exactly identical between BF16 and LiteRT INT4.
• BF16-correct → INT4-wrong occurred on only 2 examples: IDs 7 and 37 (160→120, 2→0).
• INT4 corrected one BF16 miss on ID 41 (400→200, gold 200).
• Eight examples were wrong for both: IDs 4, 12, 21, 62, 75, 87, 89, 93.
• No LiteRT response in this 100-example thinking-off run was unfinished.

The sensitive cases were replayed on the larger-cache builds. The 32K build reproduced the 16K decisions on all eight replayed controls; the 64K build reproduced the same three decisive outcomes on IDs 7, 37, and 41. This is evidence that the observed short-context differences come from the shared INT4 quantization, not from the 16K / 32K / 64K KV-cache compilation size.

Long-context retrieval

A synthetic four-needle retrieval test placed four independent access codes at approximately 8%, 35%, 65%, and 90% of the archive. The model was asked to return all four codes exactly. BF16 recovered 4/4 at every tested length.

Build	Tested prompt/history length	Result	Notes
Original BF16	13,976 tokens	4 / 4	single prompt
LiteRT 16K	13,976 tokens	4 / 4	single prompt
Original BF16	27,984 tokens	4 / 4	single prompt
LiteRT 32K	27,984 tokens	4 / 4	single prompt; 16 XNNPACK threads
Original BF16	56,000 tokens	4 / 4	single prompt
LiteRT 64K	~56K accumulated history	4 / 4	same persistent conversation, six staged chunks

The expected answer was recovered exactly at every scored LiteRT length:

A=K4M7Q2; B=V9R2L6; C=H3T8N5; D=P6W1X9

The 64K artifact was additionally tested with a single 56,000-token message, but LiteRT-LM 0.17's stock engine has a hard 10-minute per-message/session completion timeout and XNNPACK did not finish the prefill before that deadline, even with 16 CPU threads. To test the KV cache rather than the timeout, the same archive was supplied in six sequential chunks in one persistent conversation. The final ~56K accumulated-context retrieval was 4/4, demonstrating functional retention beyond 32K and near the intended 64K operating range.

Intermediate acknowledgement text in that staged 64K run became mildly noisy after later turns (CONTINUEgJC, etc.), although the final four-code retrieval remained exact. This should be treated as a long-history generation-quality caution rather than a retrieval failure.

Thinking / reasoning mode

INT4 is not currently at BF16 parity for long reasoning trajectories.

Matched test: GSM8K item 0, enable_thinking=true, greedy decoding, 2,048-token output budget.

Configuration	Gold	Extracted answer	Final answer emitted?	Thought text
Original BF16	18	18	Yes	5,306 chars
LiteRT INT4 16K	18	16	No	8,070 chars

BF16 completed the reasoning chain and emitted the correct final answer. The INT4 artifact used the entire budget inside a substantially longer reasoning trajectory, never emitted final answer text, and the best number extract from the unfinished thought was wrong.

A second thinking-on attempt with a 3,584-token output budget also exposed a practical runtime problem: the first LiteRT INT4 item did not finish before LiteRT-LM 0.17's hard 10-minute session deadline. The equivalent BF16 five-item run produced recoverable/correct answers on 5/5 (four with normal final-answer completion; one answer recoverable from a reasoning chain that itself reached the generation cap).

Recommendation: use enable_thinking=false for these INT4 artifacts when you need the most reliable behavior. A higher-precision long-context conversion (for example INT8 / mixed precision) is a better candidate for reasoning-heavy use.

Benchmark environment and interpretation

• BF16 reference: upstream openbmb/MiniCPM5-2B, PyTorch BF16 on NVIDIA RTX 4090 / CUDA.
• LiteRT artifact tests: LiteRT-LM 0.17.0 on WSL2 / XNNPACK CPU; long-context 28K/56K tests explicitly used 16 CPU threads.
• The WSL LiteRT gpu backend was not used for benchmark claims because the current WebGPU path selected Mesa llvmpipe (software) rather than the RTX 4090.
• Runtime timings across BF16 CUDA and LiteRT XNNPACK are therefore not comparable performance measurements. Accuracy, completion, and retrieval are the metrics reported here.

Conversion

• Format: LiteRT-LM .litertlm
• Quantization: BOCTAV4 — blockwise-32 INT4 + OCTAV clipping for linear weights, with INT8 embedding
• Embedder: externalized
• Prompting: upstream chat_template.jinja is preserved through the MiniCPM5 LiteRT Jinja path, and the bundle declares a thought channel. The format supports the upstream thinking switch, but the benchmark results above show that INT4 thinking quality is not at BF16 parity.
• Start/stop behavior: inherited from the upstream MiniCPM5 checkpoint (<s> start; EOS IDs 1 and 130073)
• Context cache lengths: exact powers of two (16384 / 32768 / 65536)

MiniCPM5 contains all-zero rows in layer 0 MLP weights. The INT4 block quantizer emits zero scales for those blocks, which XNNPACK refuses. Each bundle therefore applies the established in-place zero-scale repair: 1,664 zero fp16 scales are replaced by the smallest positive scale already present in the affected scale tensor. Exactly 3,328 bytes change; quantized block values are zero, so this does not change their dequantized weight values.

Toolchain

Built on 2026-09-10 in WSL2 using current toolchain revisions at conversion time:

• Python 3.13.13
• litert-torch 0.10.0 (6d4c622c9a3aade4d411a1e859c6bbe571d38ee0)
• LiteRT-LM 0.17.0 (0d43b55250dfd2a69d64b761e238f2fc4b12c959)
• litert-lm-builder 0.17.0
• litert-converter 0.4.0
• ai-edge-quantizer-nightly 0.10.0.dev20260910
• torch 2.14.0
• transformers 5.17.0
• huggingface-hub 1.31.0
• conversion/reproduction reference a72dd53df14f50e56b6115081406d4415e88ab38

See litertlm_manifest.json, validation_report.txt, and convert_minicpm5_longcontext.sh for conversion reproducibility. The benchmark results above were produced after the initial upload and supersede the original smoke-test-only quality statement in validation_report.txt.

Runtime notes / known limitations

• These files change the compiled KV-cache capacity; they do not fine-tune or extend the upstream model's learned context behavior.
• The 64K build remains below the upstream 131,072-position limit.
• Long-context memory and compute costs increase with KV-cache length. The 32K and especially 64K artifacts can be expensive on CPU.
• LiteRT-LM 0.17 currently uses a hard 10-minute engine/session completion timeout. On the tested WSL2 XNNPACK setup, a single 56K-token prefill exceeded that limit; staged persistent-context ingestion succeeded.
• After a separate thinking-on run hit the 10-minute timeout, LiteRT-LM also emitted an Abseil mutex-corruption fatal during cleanup. This occurred after the timed-out session and is treated as a runtime robustness issue, not evidence of corrupted model weights.
• Thinking-off is the recommended mode for this INT4 release. Reasoning-heavy use should prefer a higher-precision conversion when available.

License

Apache-2.0, following the upstream MiniCPM5-2B license. The weights are transformed/quantized from openbmb/MiniCPM5-2B; no fine-tuning was applied.