Instructions to use KIEFERSA/Sophea-Titan-1-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KIEFERSA/Sophea-Titan-1-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KIEFERSA/Sophea-Titan-1-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("KIEFERSA/Sophea-Titan-1-FP8")
model = AutoModelForMultimodalLM.from_pretrained("KIEFERSA/Sophea-Titan-1-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KIEFERSA/Sophea-Titan-1-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KIEFERSA/Sophea-Titan-1-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KIEFERSA/Sophea-Titan-1-FP8

SGLang

How to use KIEFERSA/Sophea-Titan-1-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KIEFERSA/Sophea-Titan-1-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KIEFERSA/Sophea-Titan-1-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KIEFERSA/Sophea-Titan-1-FP8 with Docker Model Runner:
```
docker model run hf.co/KIEFERSA/Sophea-Titan-1-FP8
```

Sophea-Titan-1-FP8

FP8_DYNAMIC (W8A8) quantization of Sophea-Titan-1

FP8_DYNAMIC (W8A8) quantization of KIEFERSA/Sophea-Titan-1 — the 27B Greek fine-tuned chat model built on Qwen3.6-27B. Same behaviour as the bf16 release, ≈0.55× the disk / VRAM, and verified lossless (numbers below).

Base model: KIEFERSA/Sophea-Titan-1 (bf16)
Scheme: FP8_DYNAMIC — per-channel FP8 (E4M3) weights + dynamic per-token FP8 activations (W8A8), compressed-tensors float-quantized
Size: 29.4 GB (bf16 base ≈ 54 GB → 0.55×)
Creator: Kiefer SA
Decoding: non-thinking (enable_thinking=false) — see recommended sampling under Usage

What is / isn't quantized. Data-free one-shot PTQ via llm-compressor (FP8_DYNAMIC, no calibration). Quantized: the language-model Linear layers (attention q/k/v/o, MLP, GDN in/out projections). Kept bf16: lm_head. GDN conv1d/A_log/dt_bias/norms are non-Linear and untouched. This quantized build is text-only — the vision tower of the multimodal base (Sophea-Titan-1) is not included in this FP8 release.

Serve non-thinking. Set enable_thinking=false in the chat template. With thinking left on, Greek output quality degrades.

Verified lossless (this FP8 build)

Measured on this checkpoint through the vLLM decode path (greedy, non-thinking), vs the bf16 base:

Metric	bf16 (published)	FP8 (this build)	Δ
General Greek benchmarks macro (9, logprob)	0.7369	0.7373	+0.0004
English retention macro (5)	0.8783	0.8767	−0.0016
GreekMMLU (30-subj)	0.854	0.853	−0.001

Every axis is within measurement noise of bf16 → lossless. FP8 also loads in HF transformers.

Per-benchmark detail (measured on this FP8 build)

General Greek benchmarks (9, accuracy)

benchmark	FP8	bf16
arc_challenge	0.9486	0.950
arc_easy	0.9731	0.973
belebele	0.9511	0.950
greekmmlu	0.8531	0.854
hellaswag	0.6794	0.680
medical_mcqa	0.3912	0.387
truthfulqa	0.4125	0.415
winogrande	0.6290	0.626
mmlu_greek	0.7980	0.797
MACRO	0.7373	0.7369

English retention — 5 benchmarks (accuracy)

benchmark	FP8	bf16
arc_challenge	0.9770	0.979
arc_easy	0.9920	0.992
hellaswag	0.8030	0.805
mmlu	0.8554	0.858
winogrande	0.7561	0.758
MACRO	0.8767	0.8783

greekmmlu — per-subject (accuracy)

subject	FP8	subject	FP8
Accounting	0.859	Greek Traditions	0.886
Agriculture	0.875	Law	0.700
Art	0.753	Management	0.838
Biology	0.868	Maritime Safety & Rescue	0.669
Chemistry	0.802	Mathematics	0.917
Civil Engineering	0.822	Medicine	0.901
Clinical Knowledge	0.820	Modern Greek Language	0.920
Computer Networks & Security	0.762	Physics	0.849
Computer Science	0.880	Prehistory	1.000
Driving Rules	0.821	World History	0.950
Economics	0.922	World Religions	0.774
Education	0.867	Geography	0.964
Electrical Engineering	0.817	Government and Politics	0.932
General Knowledge	0.789	Greek History	0.874
Greek Literature	0.714	Greek Mythology	0.845
OVERALL	0.853

Usage

Serve with vLLM (compressed-tensors FP8 auto-detected; Cutlass FP8 kernels on Blackwell/Hopper):

vllm serve KIEFERSA/Sophea-Titan-1-FP8 --served-model-name sophea-titan-1-fp8 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Recommended sampling (instruct / non-thinking): temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-titan-1-fp8",
    messages=[{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)