Instructions to use KIEFERSA/Sophea-Titan-1-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KIEFERSA/Sophea-Titan-1-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KIEFERSA/Sophea-Titan-1-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("KIEFERSA/Sophea-Titan-1-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("KIEFERSA/Sophea-Titan-1-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KIEFERSA/Sophea-Titan-1-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KIEFERSA/Sophea-Titan-1-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KIEFERSA/Sophea-Titan-1-NVFP4

SGLang

How to use KIEFERSA/Sophea-Titan-1-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KIEFERSA/Sophea-Titan-1-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KIEFERSA/Sophea-Titan-1-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KIEFERSA/Sophea-Titan-1-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KIEFERSA/Sophea-Titan-1-NVFP4 with Docker Model Runner:
```
docker model run hf.co/KIEFERSA/Sophea-Titan-1-NVFP4
```

Sophea-Titan-1-NVFP4

NVFP4 4-bit quantization of Sophea-Titan-1

NVFP4A16 (weight-only 4-bit) quantization of KIEFERSA/Sophea-Titan-1 — the 27B Greek fine-tuned chat model built on Qwen3.6-27B. This is the smallest Sophea-Titan-1 variant — ≈0.35× the disk / VRAM of bf16 — at a modest, measured accuracy cost (see below). For a lossless drop-in, use the FP8 build KIEFERSA/Sophea-Titan-1-FP8 instead.

Base model: KIEFERSA/Sophea-Titan-1 (bf16)
Scheme: NVFP4A16 — weight-only 4-bit NVFP4 (E2M1) weights + FP8 block-scales, 16-bit activations, compressed-tensors nvfp4-pack-quantized
Size: 18.8 GB (bf16 base ≈ 54 GB → 0.35×)
Creator: Kiefer SA
Decoding: non-thinking (enable_thinking=false) — see recommended sampling under Usage

What is / isn't quantized. Data-free one-shot PTQ via llm-compressor (NVFP4A16, no calibration). Quantized to 4-bit: the language-model attention + MLP Linear layers. Kept bf16: lm_head and the two GDN gate projections linear_attn.in_proj_a / in_proj_b (out-dim 48 → fused 96; kept full-precision so vLLM's Marlin FP4 kernel — which requires output dims ÷64 — can serve the model). This quantized build is text-only — the vision tower of the multimodal base (Sophea-Titan-1) is not included in this NVFP4 release.

Serving. NVFP4 runs through vLLM (Marlin FP4 kernel on Blackwell/Hopper). Unlike FP8, the nvfp4-pack-quantized format does not load in HF transformers — use vLLM. Serve non-thinking (enable_thinking=false).

Accuracy vs bf16 (measured on this build)

Measured through the vLLM decode path (greedy, non-thinking), vs the bf16 base:

Metric	bf16 (published)	NVFP4 (this build)	Δ
General Greek benchmarks macro (9, logprob)	0.7369	0.7138	−2.3 pt
English retention macro (5)	0.8783	0.8688	−0.9 pt
GreekMMLU (30-subj)	0.854	0.832	−2.2 pt

The 4-bit weight quantization costs ≈2.3 pt on General Greek benchmarks (concentrated in Greek-knowledge benchmarks), while English retention is nearly intact (−0.9 pt) and generation stays degeneration-free. This is the size/quality trade for a 4-bit footprint; the FP8 build is lossless if you can afford ~30 GB.

Per-benchmark detail (measured on this NVFP4 build)

General Greek benchmarks (9, accuracy)

benchmark	NVFP4	bf16
arc_challenge	0.9349	0.950
arc_easy	0.9663	0.973
belebele	0.9422	0.950
greekmmlu	0.8322	0.854
hellaswag	0.6414	0.680
medical_mcqa	0.3380	0.387
truthfulqa	0.3819	0.415
winogrande	0.6219	0.626
mmlu_greek	0.7655	0.797
MACRO	0.7138	0.7369

English retention — 5 benchmarks (accuracy)

benchmark	NVFP4	bf16
arc_challenge	0.9718	0.979
arc_easy	0.9916	0.992
hellaswag	0.7980	0.805
mmlu	0.8433	0.858
winogrande	0.7395	0.758
MACRO	0.8688	0.8783

greekmmlu — per-subject (accuracy, 30 subjects)

subject	NVFP4
Accounting	0.848
Agriculture	0.846
Art	0.760
Biology	0.847
Chemistry	0.741
Civil Engineering	0.791
Clinical Knowledge	0.807
Computer Networks & Security	0.651
Computer Science	0.866
Driving Rules	0.803
Economics	0.897
Education	0.871
Electrical Engineering	0.801
General Knowledge	0.769
Geography	0.936
Government and Politics	0.935
Greek History	0.841
Greek Literature	0.500
Greek Mythology	0.840
Greek Traditions	0.846
Law	0.665
Management	0.812
Maritime Safety and Rescue Operations	0.662
Mathematics	0.901
Medicine	0.841
Modern Greek Language	0.901
Physics	0.835
Prehistory	0.968
World History	0.950
World Religions	0.761
OVERALL	0.832

Usage

Serve with vLLM (compressed-tensors NVFP4 → Marlin FP4 kernel):

vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Recommended sampling (instruct / non-thinking): temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-titan-1-nvfp4",
    messages=[{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)

License

Inherits the Qwen3.6-27B base-model license. Verify base-model terms before use.

Downloads last month: 122

Safetensors

Model size

17B params

Tensor type

F32

BF16

F8_E4M3

Model tree for KIEFERSA/Sophea-Titan-1-NVFP4

Base model

Qwen/Qwen3.6-27B

Adapter

KIEFERSA/Sophea-Titan-1

Quantized

(3)

this model

Collection including KIEFERSA/Sophea-Titan-1-NVFP4

Sophea-Titan-1 Dense

Collection

5 items • Updated 5 days ago • 3

Evaluation results

General Greek benchmarks on General Greek Benchmark Suite
self-reported

0.714
Accuracy on GreekMMLU
self-reported

0.832
English retention on English Retention Suite
self-reported

0.869