Instructions to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ")
model = AutoModelForMultimodalLM.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ

SGLang

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ with Docker Model Runner:
```
docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ
```

KAT-Coder-V2.5-Dev REAP-50 NVFP4A16 — GPTQ rounding variant

Published for completeness and independent verification, not as a recommended alternative. If you're choosing a checkpoint, use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 — that is the primary release, actively maintained, and this card exists specifically to document why this variant is not a better choice.

What this is

Same REAP-50%-pruned source checkpoint as the primary release, quantized to the identical NVFP4A16 scheme (weight-only, 4-bit) — but with GPTQ (Hessian-based, error-correcting) rounding instead of the primary release's plain round-to-nearest (RTN). Same size, same architecture, same serving kernel. The only variable changed is the quantization algorithm.

12.4512 GiB — exact byte-for-byte size match to the primary release.

Why this was tried

GPTQ is documented in current literature to generally outperform RTN for NVFP4 weight recovery. It was tested here specifically to see whether that general finding held for this checkpoint and closed any of the gap to the competitive bar (Devstral Small 2512, 56.4% SWE-bench Verified under the same mini-swe-agent bash-only scaffold).

Result: no measurable difference from the primary release

Paired McNemar test (not just comparing point estimates) against the primary release's own accuracy suite, same evaluation methodology:

benchmark	primary (RTN)	this build (GPTQ)	discordant pairs	McNemar p
HumanEval+	90.85%	89.63%	6 / 164	0.68 (not significant)
MBPP+	89.95%	89.42%	18 / 378	0.81 (not significant)

Both comparisons are underpowered (well under the ~40-50 discordant pairs needed for adequate statistical power) — this is genuinely "no detectable difference," not confident evidence of true equivalence, and not evidence GPTQ is worse either. The run itself was clean: 5h23min, zero exceptions, zero fallback-to-RTN warnings across all 15,520 quantized modules.

Practical read: this specific checkpoint's RTN baseline appears to already be close to its own accuracy ceiling, leaving little room for a better rounding algorithm to show up. GPTQ's general advantage over RTN (well-supported in the literature for other models) doesn't disappear — it just isn't visible here at the precision this evaluation can resolve.

Full writeup, including the literature verification done before running this and the exact promotion criteria this didn't clear: ROADMAP.md in the primary release's GitHub repo.

Quantization details

Field	Value
Base	Same REAP-50%-pruned checkpoint as the primary release
Quantization method	`llm-compressor` `GPTQModifier` — `actorder="static"`, `block_size=128`, `dampening_frac=0.01`
Quantization scheme	NVFP4A16 — weight-only, identical to the primary release
Calibration	`evol-codealpaca` (same set, same seed as the primary release, isolating the algorithm as the only variable)

License

Apache 2.0, inherited from the base model Kwaipilot/KAT-Coder-V2.5-Dev.

Downloads last month: 11

Safetensors

Model size

19B params

Tensor type

BF16

Model tree for Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ

Base model

Kwaipilot/KAT-Coder-V2.5-Dev

Quantized

(66)

this model

Ttimms
/

KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ