# 04 — Inference

Hardware from official model card. KV intuition: [`derived/memory_notes.md`](../derived/memory_notes.md).

---

## Hardware

| Checkpoint | Aggregate VRAM | Example |
|---|---|---|
| BF16 | ≥ ~**2 TB** | 8× B300 · 16× H200 |
| NVFP4 | ≥ ~**600 GB** | 4× B300 W4A4 (SM100+) · 8× H200 W4A16 |
| GGUF | quant-dependent | Unsloth / llama.cpp |

---

## Day-0 engines

| Engine | Notes |
|---|---|
| transformers ≥~5.14 | `AutoModelForMultimodalLM`, `pipeline("any-to-any")` |
| SGLang | Fast open serve + Miles RL |
| vLLM | Production OpenAI API |
| TokenSpeed | Alt engine |
| llama.cpp | GGUF local |
| HF Inference Providers | Serverless |
| Tinker | Fine-tune + playground |

---

## Minimal transformers

```python
from transformers import AutoModelForMultimodalLM, AutoProcessor

mid = "thinkingmachines/Inkling"  # or Inkling-NVFP4
processor = AutoProcessor.from_pretrained(mid)
model = AutoModelForMultimodalLM.from_pretrained(mid, dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "What is 17 * 23?"}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    reasoning_effort="high",  # none|minimal|low|medium|high|max | float∈[0,0.99]
).to(model.device)

out = model.generate(**inputs, max_new_tokens=2000)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
```

Content parts: `{"type":"image","image":...}`, `{"type":"audio","audio":...}`.

---

## SGLang / vLLM (sketch)

```bash
python3 -m sglang.launch_server \
  --model-path thinkingmachines/Inkling --tp-size 8 \
  --served-model-name inkling --host 0.0.0.0 --port 30000
```

```bash
vllm serve thinkingmachines/Inkling --tensor-parallel-size 8 --served-model-name inkling
```

---

## HF router

```python
from openai import OpenAI
import os
client = OpenAI(base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"])
client.chat.completions.create(
    model="thinkingmachines/Inkling:auto",
    messages=[{"role": "user", "content": "Hello"}],
)
```

---

## Local GGUF

```bash
llama serve -hf unsloth/inkling-GGUF:UD-IQ1_S
```

---

## Protocol

Do not invent special tokens — use [derived/special_tokens.md](../derived/special_tokens.md) and [code-study/02-chat-protocol.md](../code-study/02-chat-protocol.md). Wrong packing breaks tools, effort, and multimodal scatter.

Next: [05-customize.md](05-customize.md)