Instructions to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
# Run inference directly in the terminal:
llama cli -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
# Run inference directly in the terminal:
llama cli -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
# Run inference directly in the terminal:
./llama-cli -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Use Docker

docker model run hf.co/Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

LM Studio
Jan

vLLM

How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Ollama
How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with Ollama:
```
ollama run hf.co/Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
```
Unsloth Desktop
Docker Model Runner
How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with Docker Model Runner:
```
docker model run hf.co/Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF
```

Lemonade

How to use Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Run and chat with the model

lemonade run user.Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

Model Card for Model ID

This is a IQ4_KT quantization of Naphula/Slimaki-Tavern-24B-v1.3, created specifically to challenge mainline IQ4_XS quants by being smaller and not being dumber.

Quant Details

iq4_kt: attn_q, attn_v, attn_k, ffn_down, ffn_gate, ffn_up, attn_output (280)
iq4_k: token_embd (1)
iq6_k: output (1)
imatrix by mradermacher

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp
only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)

Comparison:

IQ4_XS by mradermacher:

winogrande: 74.3489 ± 1.2274 (942/1267)
hellaswag (n=400, s=42): 82.25%

with K q5_0, V q4_1:
pp512: 1457.63 ± 157.65
pp32768: 1108.26 ± 5.68
tg128: 31.27 ± 0.79
tg512: 31.54 ± 0.16
size: 12758946304 bytes

this IQ4_KT:

winogrande: 74.7435 ± 1.2211 (947/1267) (+0.53%, statistically insignificant)
hellaswag (n=400, s=42): 83.25% (+1.2%, statistically insignificant)

with K q5_0, V q4_1:
pp512: 1168.07 ± 30.28 (19.8% slower)
pp32768: 955.13 ± 4.38 (13.8% slower)
tg128: 32.39 ± 0.22 (3.58% faster)
tg512: 31.52 ± 0.26 (0.07% slower)
size: 12070818560 bytes (5.39% smaller) (-656 MB)

In conclusion

-Is it smarter?
Not really.

-Is it slower?
Yeah.

-Is it smaller?
Yeah. 656 MB is about 12k extra context at K(Q5_0) V(Q4_1).

-Is it worth it?
No idea. I made it for personal use to squeeze some extra context in without losing brainpower. I can take the PP speed hit. WYSIWYG.

-Possible 12G VRAM full fit with KVO?
Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. It's going to be slower. It's probably not going to be THAT much dumber.

-Why Q5_0/Q4_1?
Reasons

Cheers

MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
Naphula - you cooked a sick one.
Everyone whose finetunes were included in the merge!
mradermacher - for the imatrix + the myriad of quants we all benefit from.