Instructions to use KVCHub/KaiLu3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KVCHub/KaiLu3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KVCHub/KaiLu3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("KVCHub/KaiLu3")
model = AutoModelForCausalLM.from_pretrained("KVCHub/KaiLu3", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use KVCHub/KaiLu3 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KVCHub/KaiLu3
# Run inference directly in the terminal:
llama cli -hf KVCHub/KaiLu3

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KVCHub/KaiLu3
# Run inference directly in the terminal:
llama cli -hf KVCHub/KaiLu3

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf KVCHub/KaiLu3
# Run inference directly in the terminal:
./build/bin/llama cli -hf KVCHub/KaiLu3

Use Docker

docker model run hf.co/KVCHub/KaiLu3

Llama new
LM Studio
Jan

vLLM

How to use KVCHub/KaiLu3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KVCHub/KaiLu3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KVCHub/KaiLu3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KVCHub/KaiLu3

SGLang

How to use KVCHub/KaiLu3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KVCHub/KaiLu3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KVCHub/KaiLu3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KVCHub/KaiLu3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KVCHub/KaiLu3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use KVCHub/KaiLu3 with Ollama:
```
ollama run hf.co/KVCHub/KaiLu3
```
Unsloth Desktop
Docker Model Runner
How to use KVCHub/KaiLu3 with Docker Model Runner:
```
docker model run hf.co/KVCHub/KaiLu3
```

Lemonade

How to use KVCHub/KaiLu3 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull KVCHub/KaiLu3

Run and chat with the model

lemonade run user.KaiLu3-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

KaiLu 3

A small language model with a big brain. Built from scratch by KvcHub.

KaiLu 3 is a 329M-parameter chat model built from scratch by KvcHub. It has a hybrid brain: 16 fast gated short-convolution layers read nearby words, and 8 attention layers connect the whole text. It read 6.6B tokens of textbooks, web pages, math, Wikipedia and code, then learned to chat and to prefer the answers people like.

Model details


Parameters	329,353,216
Layers	24: 16 gated short-convolution (kernel 3), 8 attention (layers 3, 6, 9, 12, 15, 18, 21, 23)
Width	hidden 1,024, SwiGLU feed-forward 2,816
Attention	16 query heads, 4 key-value heads (grouped-query), QK-norm, RoPE θ = 1,000,000
Context	1,024 tokens
Vocabulary	32,768 tokens (byte-level BPE)
Embeddings	input and output shared
Weights	bfloat16, safetensors
Format	`Lfm2ForCausalLM` in 🤗 Transformers (4.54+). KaiLu 3 shares LFM2's layer layout only: it was trained from scratch and contains no LFM2 weights.

How it was made

Pretraining: 6.6B tokens, mixed as: educational web (FineWeb-Edu) 50%, synthetic textbooks (Cosmopedia v2) 18%, general web and books (SlimPajama-6B, without Books3) 12%, math (FineMath 4+) 8%, Wikipedia 6%, code (OpenCoder annealing) 6%.
Supervised fine-tuning: about 460K conversations from smol-smoltalk, one pass, learning only from the assistant replies.
Preference tuning (DPO, β = 0.1): answer pairs from UltraFeedback (binarized), HelpSteer2 preferences and Anthropic HH-RLHF (up to 60K pairs).

Chat format

KaiLu 3 learned from conversations written like this:

<|system|>
You are KaiLu, a helpful assistant.
<|user|>
Hi!
<|assistant|>
Hello! How can I help?<|endoftext|>

Each reply ends with <|endoftext|>.

Limitations

KaiLu 3 is a small model. It can state wrong facts confidently, lose track in long conversations and write buggy code. It works best in English and within 1,024 tokens. It has had no safety testing beyond its preference tuning, so check its answers before relying on them.

License

Free for personal use. Business use needs permission. KaiLu 3 is released under the KaiLu 3 License: you can use, study and modify it for free for personal and non-commercial purposes. For any business or commercial use, contact KvcHub at hello@kvchub.com first. The datasets it learned from keep their own licenses.

Built by KvcHub.

Downloads last month: -

Safetensors

Model size

0.3B params

Tensor type

BF16

KVCHub
/

KaiLu3