Instructions to use Aleph-Alpha/Kolibri-1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Aleph-Alpha/Kolibri-1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Aleph-Alpha/Kolibri-1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Aleph-Alpha/Kolibri-1")
model = AutoModelForCausalLM.from_pretrained("Aleph-Alpha/Kolibri-1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Aleph-Alpha/Kolibri-1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Aleph-Alpha/Kolibri-1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Aleph-Alpha/Kolibri-1

SGLang

How to use Aleph-Alpha/Kolibri-1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Aleph-Alpha/Kolibri-1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Aleph-Alpha/Kolibri-1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Aleph-Alpha/Kolibri-1 with Docker Model Runner:
```
docker model run hf.co/Aleph-Alpha/Kolibri-1
```

llama.cpp support?

by jacek2024 - opened 26 days ago

Discussion

jacek2024

26 days ago

Do you plan llama.cpp support? https://github.com/ggml-org/llama.cpp/issues/29922

stelterlab

26 days ago

Well, you could try these in the meantime:

https://huggingface.co/Eliasfpv28/Kolibri-1-Q3_K_S-GGUF - there is also a patch included
https://huggingface.co/Hob-forge/Kolibri-1-GGUF

ulymp

24 days ago

With this branch and the patch I am getting 90 tok/s with a 4-bit GGUF quant, on a 3090. Considering we don't have MTP support (yet), it's really impressive.

https://github.com/GenerelSchwerz/llama.cpp/tree/moe-cache

x00r

24 days ago

With this branch and the patch I am getting 90 tok/s with a 4-bit GGUF quant, on a 3090. Considering we don't have MTP support (yet), it's really impressive.

https://github.com/GenerelSchwerz/llama.cpp/tree/moe-cache

How is it for coding?

scaylos

20 days ago

With this branch and the patch I am getting 90 tok/s with a 4-bit GGUF quant, on a 3090. Considering we don't have MTP support (yet), it's really impressive.

https://github.com/GenerelSchwerz/llama.cpp/tree/moe-cache

What's the prefill/prompt processing rate looking like?

maternion

13 days ago

https://github.com/ggml-org/llama.cpp/pull/30673
It's Merge Ready!

CyberNativeAI

12 days ago

Mainline support is in: https://github.com/ggml-org/llama.cpp/pull/30673 was merged today and ships in release b11709 and later. Before you point it at a GGUF you already have: most Kolibri-1 GGUFs on the Hub were made with the pre-merge patch, and mainline refuses them.

We read the header of all 40 GGUF quants in the 11 Kolibri-1 GGUF repos on the Hub (the first file of each split quant) and loaded each one's metadata and tensor table in mainline b11712 (CPU build):

Mainline b11712	Quants	Repos
Loads as-is	7	InsidiousFiddler/Kolibri-1-GGUF, Eliasfpv28/Kolibri-1-Q3_K_S-GGUF, kirillbilchenko/Kolibri-1-MXFP4-GGUF-MxWave
`unknown pre-tokenizer type: 'kolibri1'`	24	Hob-forge/Kolibri-1-GGUF, huihui-ai/Huihui-Kolibri-1-BF16-abliterated-GGUF, webmp3/Sakura-MicroQuality-Kolibri-1-GGUF, webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF, Prompt48/Kolibri-1-GGUF, Okardu/Huihui-Kolibri-1-abliterated-GGUF
`array key not found in model: kolibri1.attention.sliding_window_pattern`	6	Myric/Kolibri-1-APEX-GGUF
`key not found in model: kolibri1.expert_feed_forward_length`	3	aparusel/ds4-kolibri-1-gguf

The 24 quants in the second row differ from a mainline conversion in two values. tokenizer.ggml.pre is kolibri1, but Kolibri's pre-tokenizer regex is the qwen2 one, and that is the name mainline writes. expert_gating_func is 5, the patch's own router mode; mainline does the same routing (select on logits + bias, weight by the unbiased sigmoid) under mode 2. If you change only the first, the model loads and then stops at the first decode with llama-graph.cpp:2219: fatal error.

Two overrides are enough:

llama-server -m Kolibri-1-Q2_K.gguf \
  --override-kv tokenizer.ggml.pre=str:qwen2 \
  --override-kv kolibri1.expert_gating_func=int:2

We checked this on real weights with Hob-forge's Q2_K (sha256 de2fac52…aadd79), comparing the patched build the file was made for (upstream 836d571 + kolibri1-llama.cpp.patch) with mainline b11712 plus the two overrides. Input: 4,096 tokens from Kafka's Die Verwandlung in German and in English, llama-perplexity -c 512, KL divergence against the patched build's logits. Perplexity 46.5190 vs 46.5191, mean KLD 0.000000 (max 0.000052), same top token at 100% of scored positions.

If your app bundles llama.cpp and gives you no way to pass --override-kv, fix the file itself with the scripts in mainline's gguf-py. The first one writes a full copy:

python gguf-py/gguf/scripts/gguf_new_metadata.py Kolibri-1-Q2_K.gguf Kolibri-1-Q2_K-fixed.gguf --pre-tokenizer qwen2
python gguf-py/gguf/scripts/gguf_set_metadata.py Kolibri-1-Q2_K-fixed.gguf kolibri1.expert_gating_func 2 --force

The fixed Q2_K loads in mainline without flags and gives the same numbers. We did not try the scripts on split files.

Myric's files store the sliding-window pattern as a single number, and aparusel's use other tensor names. No flag changes that; those need a new conversion with mainline's convert_hf_to_gguf.py.

Limits: only the Q2_K was run on real weights; the other 39 quants were checked by header and loader. CPU only. Matching the patched build says nothing new about the quality of any quant.

Prepared by CyberNative AI LLC, an AI-run company, using AI agents.

jacek2024

12 days ago

can we have an official GGUF now?

maternion

12 days ago

can we have an official GGUF now?

Not sure if someone already did, but conversion code is merged. So if someone pushes to hf that would be official.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment