Instructions to use NikiKrutan/VeriLoop-E2-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
# Run inference directly in the terminal:
./llama-cli -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Use Docker

docker model run hf.co/NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

LM Studio
Jan

vLLM

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NikiKrutan/VeriLoop-E2-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NikiKrutan/VeriLoop-E2-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Ollama
How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with Ollama:
```
ollama run hf.co/NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
```
Unsloth Desktop

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "NikiKrutan/VeriLoop-E2-MTP-GGUF:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/NikiKrutan/VeriLoop-E2-MTP-GGUF:F16
```

Lemonade

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Run and chat with the model

lemonade run user.VeriLoop-E2-MTP-GGUF-F16

List all available models

lemonade list

Hermes Agent

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use NikiKrutan/VeriLoop-E2-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/VeriLoop-E2-MTP-GGUF:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "NikiKrutan/VeriLoop-E2-MTP-GGUF:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Quality-first GGUF quants of VeriLoop-E2

Why these quants? First of all they are properly measured by meaningful metrics and compared to other popular quants (most other providers don't do this). You can see which quant is better and why.

My quants were produced by my quality-first experimental tool, niki-allocator. The main difference of my method is per-tensor type allocation (not by community lore). Each tensor gets what is estimated to be best for it. Estimation is tuned by KLD runs, not just theory. After that best candidate allocations are tested with KLD runs to choose the winners.

VeriLoop-E2 is a 27B post-trained model for code, mathematics and physics, built on Qwen3.8-27B by Tsinghua SIGS Robot Lab.

Source model

Quantization quality comparison

To prove my method I've performed comparison with quants from other providers:

tsinghua-sigs-robot-lab (official creator quants, shipped without MTP head inside the model), rodrigoramosrs

Lower is better

Main metrics

Tail metrics

p99.9 is extremely corpus-driven with usual test corpus sizes of ~300-400K tokens

Same top probability

Useful for temperature 0 only, otherwise RMS Δp should be considered.

Note that graphs show 1 - same top p, so lower is better.

Perplexity (PPL)

Very crude noisy metric, cheaper to compute though, that's why so popular

Notes

TL;DR difference of corpora for end user: niki-eval-4 - coding, agentic, reasoning, hard logic prompts, English, Russian; indep-v3 - multilingual, web texts, English science papers, no reasoning or agentic traces, almost no code.

tsinghua-sigs-robot-lab and rodrigoramosrs quants for comparison were downloaded on September 25, 2026 (revisions pinned in the run log: VeriLoop-E2-GGUF@adafaeea, veriloop-coder-e2-gguf@c363cea7).

Note that tsinghua-sigs-robot-lab quants keep some tensors at higher precision than vanilla llama.cpp defaults of the same type (1-4.5 GB extra, the lower the quant the more), so their file sizes are not comparable with vanilla quant sizes. They also ship the MTP head as separate mtp-*.gguf files instead of inside the model — hence "no MTP" in the legend.

See the tables below for exact numbers for all quants and metrics

Sizes

I've chosen 15 sizes to fill 8500-23000 MiB with geometric progression and rounded to 100 MiB

niki-allocator is able to produce allocation with ~1 MiB precision.

Which one to choose?

If you are after quality: the bigger - the better. So it is always a compromise of what you can afford considering context size and other constraints. Choose the biggest you can fit.

About speed

I haven't measured speed since it is highly hardware and configuration dependable. What is faster for me may be slower for you. If the speed is the only that matters and you have Blackwell - get some NVFP4 quant (but quality is considerably worse for the same quant size). Otherwise you should measure speed yourself on your hardware and your configuration to make real decision. By the way, on my setup ubatch is the parameter that has huge impact on speed. On 27B models 384 was found to be optimal in my case (2x5060Ti), 512 or 256 are much slower. So probably you should test this first.

Quant names

Hugging Face requires quant type in the filename to properly show it. But all my quants are mixed by design. I decided to include dominating quant type (by size, not by tensor count) to the name therefore. It may easily be less than 50% of the model though. So choose by size, ignore listed quant type. For my quants: bigger --> better quality. Always.

Methodology

niki-allocator

What is niki-allocator

It is quality-first per-tensor quantization allocator for llama.cpp llama-quantize.

Allocator decides, per tensor, which quant type to use so that the resulting GGUF hits an exact target file size with the best estimated quality.

It is not direct measurement of KLD-metrics, it is only estimation tuned by some KLD runs. But as you can see from real KLD evaluations estimation is good enough.

Raw results are already good. But I added some calibration levers on top to make it better (for Qwen3.5-0.8B I used uncalibrated estimation). These levers need series of KLD evaluations to move estimation closer to reality, it takes days of GPU time on my consumer setup.

Corpora

Corpora choice, differences, details

All corpora is chunk aligned by model's tokenizer. 1 chunk = 1 turn/document exactly. I've found that standard workflow with simply concatenated documents gives high tail metrics where context suddenly changes. Most calibration decisions are made to overcome these corpora concatenation artifacts instead of real hard parts of documents. So I believe that chunk alignment is a must for any calibration or evaluation corpora measured by llama-perplexity.

imatrix: comb-imatrix-4 corpus — concatenation (without separator) of niki-imatrix-4 and an independent-domain imatrix set, 128 chunks of 6144 tokens, ~790K tokens total:
- niki-imatrix-4 part — 64 chunks, ~390K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning) + multilingual block with top 32 languages. Deduped, cleaned, filtered by simhash similarity to include only really different turns.
- independent part (v3-imatrix) — 64 chunks, ~390K tokens: top-32 languages web text prose (HuggingFaceFW/fineweb-2) excluding English + English science papers (recursal/arXiv-CC0-v0.5).
Evaluation 1: niki-eval-4 corpus - 64 chunks of 6144 tokens. ~390K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns. Does not include any exact or similar (by simhash) agentic turns from niki-imatrix-4 or the calibration corpora. Never seen during calibration.
Evaluation 2: indep-v3 corpus - 64 chunks of 6144 tokens, ~390 K tokens. This corpus is random compilation of top-32 languages web text prose (HuggingFaceFW/fineweb-2) excluding English (it is in science papers), English science papers (recursal/arXiv-CC0-v0.5). Never seen during calibration.
Calibration of niki-allocator: separate different-seed "copies" of the evaluation corpora (checked by simhash not to contain similar turns/docs), 12 hardest chunks of each. Here chunk alignment is most critical. Without chunk alignment calibration is done for concatenation artifacts and not for natural hard content.

Notes:

indep-v3 corpus will be published here. Note that you can't redistribute it without proper attributions due to license of some included papers.
niki-* corpora are not published. There is nothing really special actually. They just contain my private information. Sorry for inconvenience.
Why not wikitext-2 test? While wiki may be considered as independent evaluation corpus for my quants. It easily may not be true for others quants (wikitext-2 is the most popular corpus somehow for such jobs). They may have seen it (or part of it) during imatrix creation or their methodology calibration. So for really independent evaluation there should be corpus that wasn't seen anyhow by any quants and even better if it wasn't seen during base model creation. Also wiki is mostly homogenous (not very easy, but there are very few hard tokens and chunks). My corpora is everything from "walk in the park" to "high mountain climbing". Tail metrics are much higher. It is actually more representative for the real model work (if crawling English wiki is not your main use case of course).

Metrics

All metrics were evaluated with llama-perplexity against Q8_0 standard quant (due to hardware constraints). Note that against BF_16 metrics would be little higher.

Quantization was performed with patched llama-quantize from BF_16 source. Fully compatible with original llama.cpp.

All my and foreign quants were evaluated against the same base.

llama.cpp commit 749f688fcaa4c472ec034b08cb8a907c45cfaa02.

The main metrics during calibration of niki-allocator were mean KLD and RMS Δp.

Better PPL doesn't always mean better other more important metrics. So it can't be used as a main metric (some people still use). It is just fast indicator (no need for logits base) to check for big problems. It can't be main quality metric. Don't make or accept comparisons done only by PPL. It is noise.

same top p (top-1) is only good for temperature = 0. Since most models are used with much higher temperatures (~0.6-1.2) it shows nothing really meaningful for real use cases. RMS Δp is what should be measured and compared instead.

p99 KLD, p99.9 KLD, max KLD. With such a small corpora these metrics are not very robust. While p99 KLD can give some plausible comparison for the allocation method as a whole, p99.9 is extremely corpus-driven (too little data), max KLD is just random at any corpus size (one hardest token of the whole dataset, saying nothing about other hard tokens). So I don't tune to these. And anybody who does with less than several M tokens corpus just over-fits to their exact corpus and concatenation artifacts. Please correct me if I am wrong.

Evaluation results

niki-eval-4 corpus

Warning! Huge table inside!

Lower is better except same top p

VeriLoop-E2 — niki-eval-4 corpus

Source	Label	Size, MiB*	Size w/o MTP, MiB	Mean KLD	RMS Δp	KLD p95	KLD p99	KLD p99.9	Same top p, %	PPL
NikiKrutan	NIKI-IQ2_S	8500	8325	0.10807 ± 0.00057	10.280 ± 0.039	0.3796	0.7436	2.2142	86.64 ± 0.08	3.175 ± 0.013
NikiKrutan	NIKI-IQ2_S	9100	8925	0.07463 ± 0.00031	8.609 ± 0.032	0.2700	0.5253	1.4427	88.63 ± 0.07	3.086 ± 0.012
NikiKrutan	NIKI-IQ2_S	9800	9622	0.05902 ± 0.00026	7.731 ± 0.030	0.2094	0.4065	1.1140	90.13 ± 0.07	3.047 ± 0.012
NikiKrutan	NIKI-IQ3_XS	10500	10322	0.04198 ± 0.00020	6.342 ± 0.027	0.1524	0.3111	0.8319	91.56 ± 0.06	3.000 ± 0.012
NikiKrutan	NIKI-IQ3_S	11300	11116	0.03484 ± 0.00015	5.780 ± 0.026	0.1270	0.2518	0.6775	92.25 ± 0.06	2.990 ± 0.012
rodrigoramosrs	LoopCoder-VeriLoop-E2-IQ3_XS	11843	11413	0.03877 ± 0.00017	6.132 ± 0.027	0.1388	0.2746	0.8169	92.03 ± 0.06	3.000 ± 0.012
NikiKrutan	NIKI-IQ4_XS	12100	11885	0.02561 ± 0.00012	5.001 ± 0.023	0.0936	0.1895	0.5227	93.20 ± 0.06	2.965 ± 0.012
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q3_K_M	12875	12685	0.03595 ± 0.00016	5.824 ± 0.025	0.1314	0.2634	0.7558	92.21 ± 0.06	2.985 ± 0.012
NikiKrutan	NIKI-IQ4_XS	13000	12785	0.01835 ± 0.00011	4.170 ± 0.024	0.0671	0.1418	0.4448	94.40 ± 0.05	2.950 ± 0.012
NikiKrutan	NIKI-IQ4_XS	14000	13785	0.01103 ± 0.00006	3.242 ± 0.019	0.0398	0.0823	0.2630	95.57 ± 0.05	2.935 ± 0.012
rodrigoramosrs	LoopCoder-VeriLoop-E2-IQ4_XS	14600	14384	0.01071 ± 0.00006	3.179 ± 0.019	0.0388	0.0805	0.2399	95.70 ± 0.05	2.938 ± 0.012
NikiKrutan	NIKI-IQ4_XS	15000	14785	0.00844 ± 0.00006	2.847 ± 0.021	0.0305	0.0628	0.1873	96.12 ± 0.04	2.931 ± 0.011
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q4_K_M	16032	15781	0.00962 ± 0.00007	3.026 ± 0.021	0.0349	0.0723	0.2157	95.91 ± 0.04	2.932 ± 0.012
NikiKrutan	NIKI-Q5_K_M	16100	15885	0.00619 ± 0.00005	2.420 ± 0.018	0.0226	0.0489	0.1529	96.71 ± 0.04	2.927 ± 0.011
tsinghua-sigs	VeriLoop-E2-Q3_K_M (no MTP)	17230	17230	0.00616 ± 0.00006	2.437 ± 0.023	0.0220	0.0472	0.1409	96.64 ± 0.04	2.927 ± 0.011
NikiKrutan	NIKI-Q5_K_M	17300	17085	0.00429 ± 0.00003	2.023 ± 0.018	0.0156	0.0346	0.1056	97.32 ± 0.04	2.923 ± 0.011
NikiKrutan	NIKI-Q5_K_M	18600	18385	0.00334 ± 0.00002	1.764 ± 0.013	0.0122	0.0269	0.0842	97.61 ± 0.03	2.922 ± 0.011
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q5_K_M	18631	18340	0.00447 ± 0.00005	2.069 ± 0.024	0.0158	0.0343	0.1104	97.19 ± 0.04	2.924 ± 0.011
tsinghua-sigs	VeriLoop-E2-Q4_K_M (no MTP)	18740	18740	0.00432 ± 0.00005	2.053 ± 0.019	0.0155	0.0326	0.1006	97.15 ± 0.04	2.925 ± 0.011
tsinghua-sigs	VeriLoop-E2-Q5_K_M (no MTP)	19420	19420	0.00313 ± 0.00005	1.734 ± 0.020	0.0111	0.0241	0.0718	97.62 ± 0.03	2.924 ± 0.011
NikiKrutan	NIKI-Q6_K	20000	19785	0.00189 ± 0.00001	1.338 ± 0.008	0.0068	0.0151	0.0506	98.11 ± 0.03	2.921 ± 0.011
tsinghua-sigs	VeriLoop-E2-Q6_K (no MTP)	21060	21060	0.00177 ± 0.00003	1.315 ± 0.022	0.0062	0.0132	0.0410	98.16 ± 0.03	2.921 ± 0.011
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q6_K	21392	21060	0.00178 ± 0.00003	1.323 ± 0.022	0.0062	0.0132	0.0409	98.15 ± 0.03	2.921 ± 0.011
NikiKrutan	NIKI-Q6_K	21400	21185	0.00128 ± 0.00001	1.088 ± 0.012	0.0046	0.0106	0.0362	98.48 ± 0.03	2.920 ± 0.011
NikiKrutan	NIKI-Q6_K	23000	22785	0.00090 ± 0.00001	0.927 ± 0.015	0.0032	0.0071	0.0235	98.70 ± 0.03	2.920 ± 0.011

indep-v3 corpus

Warning! Huge table inside!

Lower is better except same top p

VeriLoop-E2 — indep-v3 corpus

Source	Label	Size, MiB*	Size w/o MTP, MiB	Mean KLD	RMS Δp	KLD p95	KLD p99	KLD p99.9	Same top p, %	PPL
NikiKrutan	NIKI-IQ2_S	8500	8325	0.15363 ± 0.00073	11.417 ± 0.056	0.5347	1.3991	4.1438	83.29 ± 0.08	6.651 ± 0.037
NikiKrutan	NIKI-IQ2_S	9100	8925	0.11289 ± 0.00058	9.685 ± 0.052	0.3845	1.0497	3.4765	85.36 ± 0.08	6.440 ± 0.036
NikiKrutan	NIKI-IQ2_S	9800	9622	0.08609 ± 0.00046	8.585 ± 0.048	0.2903	0.7860	2.6110	87.57 ± 0.07	6.283 ± 0.034
NikiKrutan	NIKI-IQ3_XS	10500	10322	0.06165 ± 0.00036	7.166 ± 0.045	0.2122	0.5745	1.9888	89.35 ± 0.07	6.181 ± 0.034
NikiKrutan	NIKI-IQ3_S	11300	11116	0.04825 ± 0.00027	6.248 ± 0.040	0.1650	0.4267	1.4994	90.44 ± 0.07	6.131 ± 0.034
rodrigoramosrs	LoopCoder-VeriLoop-E2-IQ3_XS	11843	11413	0.05627 ± 0.00033	6.933 ± 0.044	0.1872	0.5005	1.8474	90.02 ± 0.07	6.177 ± 0.034
NikiKrutan	NIKI-IQ4_XS	12100	11885	0.03671 ± 0.00021	5.447 ± 0.036	0.1250	0.3305	1.1459	91.45 ± 0.06	6.081 ± 0.033
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q3_K_M	12875	12685	0.04652 ± 0.00028	6.143 ± 0.043	0.1572	0.4353	1.6216	90.84 ± 0.07	6.141 ± 0.034
NikiKrutan	NIKI-IQ4_XS	13000	12785	0.02632 ± 0.00018	4.587 ± 0.035	0.0900	0.2509	0.9244	93.00 ± 0.06	6.042 ± 0.033
NikiKrutan	NIKI-IQ4_XS	14000	13785	0.01683 ± 0.00012	3.718 ± 0.032	0.0562	0.1543	0.6120	94.35 ± 0.05	6.014 ± 0.033
rodrigoramosrs	LoopCoder-VeriLoop-E2-IQ4_XS	14600	14384	0.01559 ± 0.00012	3.529 ± 0.030	0.0525	0.1445	0.5063	94.54 ± 0.05	6.012 ± 0.033
NikiKrutan	NIKI-IQ4_XS	15000	14785	0.01156 ± 0.00008	3.004 ± 0.027	0.0386	0.1027	0.4108	95.20 ± 0.05	5.986 ± 0.033
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q4_K_M	16032	15781	0.01314 ± 0.00010	3.227 ± 0.029	0.0432	0.1197	0.4382	94.98 ± 0.05	5.990 ± 0.033
NikiKrutan	NIKI-Q5_K_M	16100	15885	0.00911 ± 0.00007	2.673 ± 0.026	0.0310	0.0846	0.3480	95.77 ± 0.04	5.972 ± 0.033
tsinghua-sigs	VeriLoop-E2-Q3_K_M (no MTP)	17230	17230	0.00884 ± 0.00007	2.631 ± 0.023	0.0288	0.0816	0.3349	95.80 ± 0.04	5.977 ± 0.033
NikiKrutan	NIKI-Q5_K_M	17300	17085	0.00643 ± 0.00005	2.245 ± 0.020	0.0213	0.0603	0.2467	96.41 ± 0.04	5.967 ± 0.033
NikiKrutan	NIKI-Q5_K_M	18600	18385	0.00458 ± 0.00004	1.889 ± 0.019	0.0153	0.0428	0.1710	96.98 ± 0.04	5.957 ± 0.033
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q5_K_M	18631	18340	0.00562 ± 0.00004	2.093 ± 0.019	0.0186	0.0493	0.1939	96.61 ± 0.04	5.961 ± 0.033
tsinghua-sigs	VeriLoop-E2-Q4_K_M (no MTP)	18740	18740	0.00590 ± 0.00005	2.147 ± 0.019	0.0194	0.0522	0.2064	96.47 ± 0.04	5.966 ± 0.033
tsinghua-sigs	VeriLoop-E2-Q5_K_M (no MTP)	19420	19420	0.00384 ± 0.00003	1.742 ± 0.017	0.0126	0.0330	0.1327	97.11 ± 0.04	5.955 ± 0.033
NikiKrutan	NIKI-Q6_K	20000	19785	0.00267 ± 0.00002	1.445 ± 0.014	0.0088	0.0242	0.0977	97.62 ± 0.03	5.952 ± 0.033
tsinghua-sigs	VeriLoop-E2-Q6_K (no MTP)	21060	21060	0.00213 ± 0.00001	1.302 ± 0.012	0.0068	0.0175	0.0644	97.81 ± 0.03	5.953 ± 0.033
rodrigoramosrs	LoopCoder-VeriLoop-E2-Q6_K	21392	21060	0.00212 ± 0.00001	1.293 ± 0.012	0.0068	0.0174	0.0629	97.80 ± 0.03	5.952 ± 0.033
NikiKrutan	NIKI-Q6_K	21400	21185	0.00179 ± 0.00001	1.199 ± 0.012	0.0060	0.0171	0.0697	98.13 ± 0.03	5.951 ± 0.033
NikiKrutan	NIKI-Q6_K	23000	22785	0.00112 ± 0.00001	0.936 ± 0.011	0.0037	0.0098	0.0394	98.49 ± 0.03	5.950 ± 0.033

MTP

I've tested MTP performance and discovered that there is no point to keep MTP head higher than IQ4_XS. Q3_K is near but slightly worse (used for lower quants). Exact allocation of IQ4_XS/Q3_K is chosen by allocator (not pinned). Lower than that there is considerable quality drop, so not used in my quants. Higher is just waste of model size and VRAM.

For numbers check my GRM-2.6-Plus-0628 quant's MTP section (same model architecture).

I've done fast tests on other models (same architecture). Same conclusions.

Also I've additionally checked fast if there is any degradation on IQ4_XS vs Q8_0 with longer context (tested up to 28K), none found. And temperature = 0.6, still no evidence of degradation. But I must say that doing this test properly and scientifically much time and effort is needed. Since not 0 temperature gives different results: many runs needed to stabilize. Reasonably long context (100K+ from my opinion) is very slow on my hardware. So there actually may be some degradation. But that should be proven and not taken by "it seems" or "everybody knows". If there is such comprehensive analysis already, please let me know.

Note that the official tsinghua-sigs-robot-lab quants ship the MTP head as separate mtp-*.gguf files instead of inside the model file. My quants include it inline (1-2% of the model size), so speculative decoding works out of the box with a supporting llama.cpp build.

Basically I use these parameters for MTP: --spec-draft-n-max 5 --spec-draft-p-min 0.8. Primary reason of poor results with MTP is ignoring --spec-draft-p-min. Don't put obvious crap in your drafts. But let drafter do more if it is sure enough. MTP performance speed-up is highly dependable on hardware, configuration and specific context. So to decide what is best for your case you should test yourself.

Also don't use other llama.cpp drafters like ngram-mod with MTP. MTP wastes time even if ngram-mod already did draft. That is how llama.cpp is programmed. Not obvious. I have created experimental fork to overcome this issue (and added much better ngram-mod-v2 + minor fixes): https://github.com/NikiKrutan/niki-llama.cpp. I use it myself on a daily basis, but it is more like crude draft than real working fork. It breaks some llama.cpp behavior. But it gives another ~1.5x speed-up on top of MTP for my use cases.

Chat template

Chat template is from the original model (included in the GGUF).

The model card marks the tokenizer and chat template as part of the model protocol (identity handling, tool-call syntax, reasoning behavior). Don't replace them with forked or community-modified variants — observed behavior can change even with unchanged weights.

Multimodality

The official release is language-model only. But I've tested it with standard Qwen3.8-27B mmproj and it works fine.

I include quantized mmproj following DAXZEIT's receipt for Qwen3.6 that I used in Qwen3.6 era. It is lighter with no noticeable degradation (but it wasn't anyhow measured, so it is just "take my word on this").

Acknowledgments

tsinghua-sigs-robot-lab - source model.
rodrigoramosrs - quants for comparison.
Qwen - great base model.
llama.cpp - quantization and evaluation framework.
FineWeb - data for evaluation dataset.
recursal - data for evaluation dataset.
DAXZEIT - mmproj receipt.
michaelw9999 - some ideas to start with.
Required attributions for exact science papers used for evaluation dataset are on dataset page.

Downloads last month: 2,289

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

Model tree for NikiKrutan/VeriLoop-E2-MTP-GGUF

Base model

Qwen/Qwen3.8-27B

Finetuned

tsinghua-sigs-robot-lab/VeriLoop-E2

Quantized

(5)

this model