Instructions to use Kwaipilot/KAT-Coder-V2.5-Dev with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Kwaipilot/KAT-Coder-V2.5-Dev with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Kwaipilot/KAT-Coder-V2.5-Dev")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Kwaipilot/KAT-Coder-V2.5-Dev")
model = AutoModelForMultimodalLM.from_pretrained("Kwaipilot/KAT-Coder-V2.5-Dev", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Kwaipilot/KAT-Coder-V2.5-Dev with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Kwaipilot/KAT-Coder-V2.5-Dev"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Kwaipilot/KAT-Coder-V2.5-Dev

SGLang

How to use Kwaipilot/KAT-Coder-V2.5-Dev with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Kwaipilot/KAT-Coder-V2.5-Dev" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Kwaipilot/KAT-Coder-V2.5-Dev" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Kwaipilot/KAT-Coder-V2.5-Dev with Docker Model Runner:
```
docker model run hf.co/Kwaipilot/KAT-Coder-V2.5-Dev
```

GGUF Release: Handcrafted APEX-I-MiniPlus V2.1 (Optimized for System RAM Streaming & Massive Context)

#32

by IsValorum - opened about 22 hours ago

Discussion

IsValorum

about 22 hours ago

•

edited about 22 hours ago

Hugging Face CLI

Hi everyone,

I have published a custom, handcrafted APEX-I-MiniPlus V2.1 GGUF release of KAT-Coder-V2.5-Dev:
IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Key Engineering Highlights:

Specially Engineered for Full or Partial System RAM Streaming: Tailored specifically for workstations and consumer setups running most or all of the model out of system RAM (DDR4 / DDR5). By utilizing linear SIMD-optimized Q3_K edge experts and upgrading shared foundation experts to Q5_K across all 40 layers, AVX2 CPU lookup stalls are completely eliminated (sustaining +24 to 28+ tok/s generation).
Near-VRAM Speed on Modern CPUs: Depending on your processor architecture (IPC / single-core performance) and memory bandwidth (dual-channel DDR4 or high-speed DDR5 6000+ MT/s), streaming throughput across system RAM can approach speeds remarkably close to having the entire model in VRAM.
Massive & Full Context Window Support (+160k to 256k): Designed to handle deep contexts without memory fragmentation or quality degradation.
Hybrid Offload Optimization: Engineered for repository-wide codebase intelligence, multi-file debugging, and autonomous developer agentic workflows. Keeps large repository KV caches responsive in GPU VRAM while the 35B MoE code engine streams seamlessly across system RAM.
Zero Routing Drift & Intact Syntax: Retains uncompressed F32 router gates (blk.*.ffn_gate_inp.weight), high-precision Q6_K token output head (output.weight), and Q8_0 attention gates across all recurrent hybrid layers.
Full 24GB VRAM Offload: If you have 24GB+ VRAM (-ngl 99), the model runs natively at full GPU tensor core speeds.

Quick Run with llama.cpp:

# High-speed hybrid RAM streaming with GPU offload (example with 16 layers or vision in VRAM):
llama-cli -m KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF.gguf -ngl 16 -c 32768 --temp 0.6 --threads 8

Direct link to the model repository: IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Feedback and community benchmarks are warmly welcome! Enjoy!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment