```markdown
# Step-5-Preview-GGUF-Q3_K_M-3bit
Step-5-Preview — GGUF (Q3_K_M, 3-bit)
🔥 Step-5-Preview Q3_K_M GGUF is now available!
This is the 3-bit k-quant GGUF build of Step-5-Preview, converted from the official BF16 safetensors checkpoint (
stepfun-ai/Step-5-Preview-BF16) and packaged for llama.cpp-based inference. It runs on CPU + GPU hybrid setups and is intended for local deployment, low-VRAM environments, and quick experimentation.A 4.1 GB vision
mmprojfile is included for multimodal (image + video) input, using the Step3-VL perception encoder (projector_type=step3vl).
📖 Table of Contents
- Introduction
- What's in this Repository
- Quantization Details
- Model Architecture
- Model Specifications
- Benchmark Reference
- Hardware Requirements
- Quickstart with llama.cpp
- Multimodal (mmproj) Usage
- Server Mode
- Convert It Yourself
- Known Limitations
- Performance Notes
- Citation
- License
- Contact
🚀 Introduction
Step-5-Preview is StepFun's flagship foundation model, designed for real-world agentic tasks — AI coding, software engineering, professional knowledge work, and financial analysis. It is a 600B-parameter sparse Mixture-of-Experts (MoE) model with 27B active parameters per token, a 1M-token context window, and native text + image + video input.
This repository contains the Q3_K_M (3-bit) GGUF conversion, produced with the official
convert_hf_to_gguf.py pipeline (see step5-llamacpp.patch) and llama-quantize.
💡 Why a 3-bit GGUF?
• ~5× smaller than the BF16 checkpoint — fits on consumer and prosumer hardware. • llama.cpp native — pure CPU, CPU+GPU hybrid, or full GPU offload. • Multimodal ready — paired
mmprojfor image and video input. • Agent-capable — tool calling and long-context reasoning preserved.
📦 What's in this Repository
| File | Size (approx.) | Description |
|---|---|---|
Step-5-Preview-Q3_K_M.gguf |
~220–260 GB | Text model, 3-bit k-quant |
Step-5-Preview-mmproj-BF16.gguf |
~4.1 GB | Vision projector (projector_type=step3vl) |
README.md |
— | This file |
Note: Exact file sizes depend on the final shard layout. If the model is split, load with
-m Step-5-Preview-Q3_K_M-00001-of-000NN.gguf— llama.cpp auto-loads the rest.
⚙️ Quantization Details
| Parameter | Value |
|---|---|
| Base checkpoint | stepfun-ai/Step-5-Preview-BF16 (bfloat16 safetensors) |
| Conversion tool | convert_hf_to_gguf.py (with step5-llamacpp.patch) |
| Quantization tool | llama-quantize |
| Quant type | Q3_K_M (k-quant, mixed 3-bit) |
| File type tag | Q3_K_M |
| Tokenizer pre-tokenizer | deepseek-v3 (identical BPE to DeepSeek-V3) |
| Architecture tag | STEP35 |
| Vision projector | step3vl (Step3-VL perception encoder) |
What Q3_K_M Means
Q3_K_M is llama.cpp's mixed 3-bit k-quant — most tensors are quantized to 3-bit, while sensitive tensors (attention output, some FFN) are kept at higher precision (4–6 bit). This gives a better quality/size trade-off than plain Q3_K_S but is larger.
| Quant | Bits (avg) | Relative Size | Relative Quality |
|---|---|---|---|
| BF16 | 16 | 100% | Reference |
| Q8_0 | 8 | ~50% | Near-lossless |
| Q5_K_M | ~5 | ~33% | Very high |
| Q4_K_M | ~4 | ~27% | High |
| Q3_K_M ← this repo | ~3 | ~20% | Moderate |
| Q2_K | ~2 | ~15% | Noticeable loss |
⚠️ At 3-bit, expect perplexity to increase and some degradation on coding, math, and fine-grained reasoning tasks compared to BF16 or Q4_K_M. See Known Limitations.
🏗️ Model Architecture
The GGUF graph reuses the STEP35 architecture path in llama.cpp, extended for Step-5.
Core Trunk
| Component | Value |
|---|---|
| Architecture | Sparse Mixture-of-Experts (MoE) |
| Total parameters | 600B |
| Active parameters | 27B per token (~4.5% sparsity) |
| Layers | 95 (narrow-deep Transformer) |
| Attention pattern | Hybrid — 23 full_attention + 72 sliding_attention |
| SWA type | LLAMA_SWA_TYPE_STANDARD |
| MoE | moe_num_experts + moe_top_k |
| Tokenizer | DeepSeek-V3 BPE (tokenizer.ggml.pre = deepseek-v3) |
RoPE Configuration
| Layer Type | rope_theta |
Rotary Fraction |
|---|---|---|
| full_attention | 1e7 |
1/3 of head_dim |
| sliding_attention | 1e4 |
1.0 (full) |
The GGUF explicitly writes
rope.dimension_count(head_dim/3) andrope.dimension_count_swa(head_dim) so llama.cpp honours Step-5's partial RoPE. Without these keys, llama.cpp would fall back to the Step3p5 default ofhead_dim/2.
Vision Encoder (mmproj)
| Parameter | Value |
|---|---|
| Resolution | 728 px |
| Patch size | 14 |
| Width | 1536 |
| Layers | 47 |
| Downsampler | stride-2 pair |
| Projector | vit_large_projector |
projector_type |
step3vl |
🔬 Sparse GQA indexer — what happens in GGUF
The BF16 checkpoint uses a Sparse GQA mechanism on the full_attention layers:
- CSA (block compression) indexer
- top-k selection over compressed KV blocks
This is not modelled in llama.cpp yet. During conversion, the affected tensors are dropped and those layers fall back to dense attention:
- Tensors dropped:
*.sparse_indexer*,*.ssmax_s - Result: correct output, but slower and with different long-context behaviour than the reference BF16 model.
To re-enable sparse attention in the future, the filter_tensors() hook in
Step5Model must be removed once attention_impl=sparse_gqa exists in the graph builder.
📋 Model Specifications
| Category | Specification |
|---|---|
| Model Name | Step-5-Preview |
| Repository | Step-5-Preview-GGUF-Q3_K_M-3bit |
| Base Checkpoint | stepfun-ai/Step-5-Preview-BF16 |
| Format | GGUF |
| Quantization | Q3_K_M (3-bit k-quant, mixed) |
| Developer | StepFun |
| Architecture Tag | STEP35 |
| Total Parameters | 600B |
| Active Parameters | 27B per token (~4.5% sparsity) |
| Layers | 95 (23 full + 72 sliding attention) |
| Context Window | Up to 1,000,000 tokens (KV cache limited) |
| Attention | Hybrid full + sliding; dense fallback in GGUF |
| Input Modalities | Text, Image, Video (via mmproj) |
| Output Modalities | Text |
| Video Formats | MP4, QuickTime, Matroska (≤128 MB, ≤5 min recommended) |
| Reasoning Effort | low / medium / high (xhigh) |
| Tool Calling | Parallel, strict JSON schema |
| Token Pre-tokenizer | deepseek-v3 |
| License | StepFun Community License |
📊 Benchmark Reference
This GGUF is a quantized derivative of the BF16 model. The scores below are for the BF16 reference model and are provided for context — Q3_K_M will score lower, especially on coding, math, and long-context tasks.
Reference scores (BF16, high reasoning effort)
| Benchmark | Score |
|---|---|
| GPQA Diamond | 93.5% |
| Humanity's Last Exam (HLE) | 46.5% |
| AA-LCR v1.1 | 88.3% |
| DeepSWE v1.1 | 67.7% |
| Terminal-Bench 4 | 33.3% |
| CyberGym | 84.7% |
| SciCode | 58.9% |
| ProgramBench | 80.5% |
| FrontierFinance | 66.4% |
| DRACO | 83.3% |
| MMMU-Pro | 76.0% |
| Output Speed | 99.8 tok/s (8× H100, BF16) |
📉 Expected degradation at Q3_K_M
Quantization at 3-bit affects tasks unevenly. Rough expectations (may vary):
| Task Type | Sensitivity to 3-bit | Notes |
|---|---|---|
| Conversational / general text | Low | Usually near-identical |
| Summarization / extraction | Low–Moderate | Minor fluency loss |
| Coding / SWE | High | Syntax errors, subtler logic bugs |
| Math / formal reasoning | High | Multi-step chains degrade first |
| Long-context (100K+) | High | Compounded by dense fallback |
| Tool calling / JSON | Moderate | Schema adherence usually OK |
| Vision (mmproj is BF16) | Low | Encoder precision is preserved |
If you need maximum quality, use the BF16 safetensors or a Q4_K_M / Q5_K_M GGUF.
🖥️ Hardware Requirements
📐 KV cache is the real constraint
The 3-bit weights are small, but the 1M-token context window is not. KV cache memory scales with context length and batch size. Realistic configurations below.
| Setup | RAM / VRAM | Notes |
|---|---|---|
| CPU-only (small context) | 256 GB RAM | Slow but works; ~4–8K context |
| CPU-only (recommended) | 384–512 GB RAM | 32–64K context feasible |
| CPU + GPU hybrid | 256 GB RAM + 24 GB VRAM | Offload some layers to GPU |
| CPU + GPU hybrid (pro) | 256 GB RAM + 48–96 GB VRAM | Best single-node balance |
| Full GPU offload | ~240 GB VRAM (aggregate) | e.g. 4× A100 80GB / 4× H100 80GB |
| 1M context | 512 GB RAM + paged KV | llama.cpp --flash-attn + --cache-type-k/v q8_0 |
Tip: Use
--n-gpu-layers Nto tune offload. Start with-ngl 99and reduce until it fits. For hybrid setups, offloading attention layers first usually helps most.
⚡ Quickstart with llama.cpp
1. Build llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # or -DGGML_METAL=ON, -DGGML_VULKAN=ON, etc.
cmake --build build --config Release -j
2. Download the model
# using huggingface-cli
huggingface-cli download SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit \
--local-dir ./Step-5-Preview-Q3_K_M
# or with git-lfs
git lfs install
git clone https://huggingface.co/SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit
3. Run text generation
./build/bin/llama-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
-p "Explain the significance of the Pareto Frontier in AI scaling." \
-n 512 \
-ngl 99 \
--temp 0.7 \
--top-p 0.95 \
--ctx-size 32768
4. Run with a chat template
./build/bin/llama-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
--conversation \
-ngl 99 \
--ctx-size 32768
5. Control reasoning effort
Step-5-Preview supports low / medium / high / xhigh. Pass it via the system prompt
or as a template argument, depending on the chat template exposed by the GGUF:
./build/bin/llama-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
-sys "You are a helpful assistant. Reasoning effort: high." \
-p "Write a Python function to merge two sorted lists." \
-ngl 99
👁️ Multimodal (mmproj) Usage
The vision encoder is shipped as a separate BF16 mmproj file. This is intentional — the projector stays high-precision while the language model is 3-bit.
Image input
./build/bin/llama-mtmd-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
--mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
--image ./screenshot.png \
-p "Describe this UI and suggest improvements." \
-ngl 99
Video input
./build/bin/llama-mtmd-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
--mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
--video ./demo.mp4 \
-p "Summarize what happens in this video." \
-ngl 99
Limits: MP4, QuickTime, Matroska. Recommended ≤128 MB and ≤5 minutes per clip.
🌐 Server Mode
llama-server exposes an OpenAI-compatible HTTP API.
Text-only
./build/bin/llama-server \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
-ngl 99 \
--ctx-size 32768 \
--host 0.0.0.0 \
--port 8080
With vision
./build/bin/llama-server \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
--mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
-ngl 99 \
--ctx-size 32768 \
--host 0.0.0.0 \
--port 8080
Query it
from openai import OpenAI
client = OpenAI(api_key="none", base_url="http://localhost:8080/v1")
response = client.chat.completions.create(
model="step-5-preview",
messages=[
{"role": "user", "content": "Write a Python function to merge two sorted lists."}
],
max_tokens=512,
temperature=0.7,
)
print(response.choices[0].message.content)
🔧 Convert It Yourself
The conversion requires the step5-llamacpp.patch applied on top of llama.cpp.
1. Apply the patch
cd llama.cpp
git checkout ce8caa6e60a03093351d6016a818720e0d46f0fb
git apply -p1 /path/to/step5-llamacpp.patch
# or: patch -p1 < /path/to/step5-llamacpp.patch
2. Convert BF16 safetensors → BF16 GGUF
python convert_hf_to_gguf.py /path/to/Step5_safetensors \
--outtype bf16 \
--outfile Step-5-Preview-BF16.gguf
3. Convert the vision projector
python convert_hf_to_gguf.py /path/to/Step5_safetensors \
--mmproj \
--outtype bf16 \
--outfile Step-5-Preview-mmproj-BF16.gguf
4. Quantize to Q3_K_M
./build/bin/llama-quantize \
Step-5-Preview-BF16.gguf \
Step-5-Preview-Q3_K_M.gguf \
Q3_K_M
🧩 What the patch changes
| File | Change |
|---|---|
conversion/__init__.py |
Routes Step4ForCausalLM and MMGPTStepRoboticsForCausalLM into the step3 converter (text + mmproj maps). |
conversion/base.py |
Maps the Step-5 tokenizer hash to the deepseek-v3 pre-tokenizer. |
conversion/step3.py |
Adds Step5Model (text) and Step5VisionModel (mmproj) on the STEP35 arch; per-layer rope_theta by layer_type; writes rope.dimension_count / rope.dimension_count_swa; drops sparse-GQA indexer tensors. |
src/models/step35.cpp |
Only halves n_rot_full when rope.dimension_count is absent. |
⚠️ Known Limitations
No sparse attention in GGUF. The CSA indexer + top-k selection is dropped during conversion. Affected layers run dense attention — correct but slower, and long-context quality differs from the BF16 reference.
23 of 95 layers remain full attention. Combined with the dense fallback, this means long-context performance is not equivalent to the reference model until the indexer is implemented in llama.cpp.
3-bit quantization loss. Q3_K_M introduces noticeable degradation on coding, math, and fine-grained reasoning. Expect lower scores than the BF16 benchmark table.
MTP / NextN tensors convert through but are only used if a draft model is configured.
KV cache dominates memory at long context. The weights are small; the 1M-token context window is not. See Hardware Requirements.
Vision is separate. The mmproj must be loaded explicitly; it is not bundled inside the text GGUF.
Language coverage. Primarily optimized for English and Chinese.
⚡ Performance Notes
| Factor | Impact |
|---|---|
| Quant type | Q3_K_M is ~20% of BF16 size; expect slower per-token speed on CPU than Q4_K_M due to less efficient kernels |
| GPU offload | -ngl 99 offloads everything if VRAM allows; reduce if not |
| Flash attention | Add --flash-attn for lower KV memory and faster long-context |
| KV quantization | --cache-type-k q8_0 --cache-type-v q8_0 roughly halves KV memory |
| Batch size | Larger batches improve throughput but increase KV memory |
| Context length | Larger --ctx-size allocates more KV cache upfront |
Recommended flags for long context:
./build/bin/llama-cli \
-m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
-ngl 99 \
--ctx-size 131072 \
--flash-attn \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-p "Summarize this codebase."
📚 Citation
If you use Step-5-Preview in your research, please cite:
@misc{stepfun2026step5preview,
title = {Step-5-Preview: A 600B Sparse MoE Foundation Model for Real-World Agentic Work},
author = {StepFun Team},
year = {2026},
howpublished = {\url{https://huggingface.co/SHSLab/Step-5-Preview-BF16}},
note = {Released September 20, 2026}
}
If you use this GGUF build, please also reference the base checkpoint and llama.cpp.
📜 License
Step-5-Preview is released under the StepFun Community License. See the LICENSE file for full terms.
⚠️ Usage Restrictions
• Commercial use is permitted under the StepFun Community License. • Redistribution must include the license and attribution. • See LICENSE for full details.
📬 Contact
- Hugging Face: SHSLab
- GitHub: github.com/stepfun-ai
- Discord: Join our Discord
- Email: opensource@stepfun.com
- Website: stepfun.com
Built with ❤️ by StepFun
- Downloads last month
- 6
3-bit
Model tree for SHSLab/Step-5-Preview-GGUF
Base model
stepfun-ai/Step-5-Preview-BF16