HyperScale/lumi-4b-GGUF

CEIA Lumi-4B — GGUF

GGUF conversions of ceia-br/lumi-4b (revision 1c8928e1), a 4B-parameter hybrid Mamba-2 / attention model (NemotronH, dense) for Portuguese. All credit for the model goes to CEIA. Licence: the original repository states none at the time of conversion; the terms of the original model apply.

file	quant	size	VRAM (8k ctx)	gen speed, RTX 3060 12 GB
lumi-4b-Q4_K_M.gguf	Q4_K_M	2.7 GB	3.1 GB	~99 tok/s

Usage (llama.cpp)

llama-server -m lumi-4b-Q4_K_M.gguf -ngl 99 -c 16384 --jinja --repeat-penalty 1.1

Thinking is on by default in the chat template; turn it off per request with "chat_template_kwargs": {"enable_thinking": false}.

Use a repetition penalty (1.1): see the measurements below.

Measured (RTX 3060 12 GB, llama.cpp b11985)

Global-MMLU PT

CohereLabs/Global-MMLU, config pt, the whole test split (14,042 questions), 0-shot, greedy, thinking off. One chat turn per question; the score is the first A–D letter the model writes (a reply with none counts wrong).

file	accuracy	unparsed	Business	Humanities	Medical	Other	STEM	Social Sciences
lumi-4b-Q4_K_M.gguf	57.88%	6	59.8%	50.3%	61.4%	66.1%	56.6%	66.3%

The original model card reports 59.36% on Global MMLU PT for the BF16 model. Its method is not stated (likely log-likelihood scoring), so the two numbers are close but not strictly comparable; the BF16 weights were not run with this generative protocol.

Before and after quantisation: a task-summary benchmark

15 Portuguese summaries of real agent outcomes (MRs opened vs merged, held gates, timeouts, unverified contracts), scored by deterministic checks: required facts present, no false status claim, no identifier the input lacks. Greedy, thinking off, the same prompt for every row.

variant	score	false claims	answers looping to the limit	latency	speed	VRAM
BF16 GGUF (before)	0.97	0/15	0	6.1 s	27 tok/s	7.6 GB
Q8_0	0.96	0/15	0	2.6 s	66 tok/s	4.3 GB
Q4_K_M	0.96	1/15	2	2.4 s	93 tok/s	3.1 GB
Q4_K_M, repeat_penalty 1.1	0.96	0/15	0	1.8 s	93 tok/s	3.1 GB
transformers NF4 (bitsandbytes), for reference	0.97	0/15	1	4.5 s	48 tok/s	~6 GB

Q4_K_M keeps the task score within 0.01 of BF16 at 3.4× the speed and 40% of the memory. Use the repetition penalty with it: without one, greedy decoding looped on 2 of 15 cases (one of them repeating an MR reference until the token limit, which also produced a reference the input never had).

How it was converted

llama.cpp convert_hf_to_gguf.py (build b11985) through convert_lumi.py, included here. Under transformers 5, NemotronHConfig fills mixture-of-experts defaults (num_experts_per_tok, n_routed_experts, n_group…) and replaces hybrid_override_pattern with layers_block_type; the stock converter then writes a nemotron_h_moe file with every MLP width at 0, which llama.cpp cannot load (tensor 'blk.1.ssm_in.weight' not found). The wrapper drops the MoE keys for a model with no expert layer and restores the pattern string from config.json. Quantised with llama-quantize.