Instructions to use axteam/Motif-3-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/Motif-3-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/Motif-3-NVFP4A16", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("axteam/Motif-3-NVFP4A16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/Motif-3-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/Motif-3-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Motif-3-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/Motif-3-NVFP4A16

SGLang

How to use axteam/Motif-3-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/Motif-3-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Motif-3-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/Motif-3-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Motif-3-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/Motif-3-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/axteam/Motif-3-NVFP4A16
```

Motif-3 · NVFP4A16 (W4A16, FP4 weights)

4-bit quantization of Motif-Technologies/Motif-3 for 🤗 Transformers (trust_remote_code). Quantized by axteam.


Base model	`Motif-Technologies/Motif-3`
Method	NVFP4A16 (W4A16, FP4 weights)
Format	compressed-tensors `nvfp4-pack-quantized`
Weights / activations	W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations
Architecture	MoE (Motif arch · 384 experts + shared) · 53 layers · text-only
Size on disk	~180.8 GB
Calibration	None (data-free, name-based `model_free_ptq`)
License	mit

Quantization details

Scheme — W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — token embeddings, the LM head, all RMSNorms, the MoE routers (router.gate), the attention output gates (wq_b_gate), the PolyNorm / mHC scalars, and the whole MTP layer (mtp_layers.0) — about 1% of all parameters. Quantized: attention, routed/shared expert and mHC projections.
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.
Modified modeling_motif.py — the original stores routed experts as fused 3D tensors (gate_up_proj [E, 2I, H], down_proj [E, H, I]). Quantization splits them per expert, so this repo's MotifExperts holds one set of nn.Linear layers per expert (experts.<e>.{gate,up,down}_proj). gate_proj / up_proj are the first / second half of the original gate_up_proj[e]; the computation is unchanged. With the original modeling file the expert weights would not load.

Weight check (measured)

53-layer weight check (2026-11-01): every one of the 59,547 quantized modules was decompressed with compressed-tensors and compared with the BF16 original — median relative error 9.5%, no outliers; all 1,339 BF16-kept tensors are bit-identical to the original. This checks the weights, not end-to-end generation (no benchmark was run on Motif-3).

Requirements


Runtime	Transformers (checked here). Official vLLM (0.32) does not include Motif-3; Motif's own vLLM fork (MotifTechnologies/vllm, based on v0.26.0, docker `ghcr.io/motiftechnologies/vllm:v0.26.0-motif3`) serves the BF16 original — this quantized build is not yet tested on it.
Transformers	Transformers 5.x (`transformers>=5.7.0`, the version the config was saved with) + `compressed-tensors>=0.19` + `trust_remote_code=True` + `flash-attn` with `attn_implementation="flash_attention_2"` (CUDA GPU). Loading checked with 5.17.0 + 0.19.0 on the first layers; full 53-layer generation was not run.
Memory	about 590 GB at runtime: Transformers decompresses the weights to BF16. Spread over GPUs and CPU RAM with `device_map="auto"`.

Usage (Transformers)

Official vLLM (checked with 0.32) does not include Motif-3. Motif publishes its own vLLM fork with Motif-3 support (MotifTechnologies/vllm, based on vLLM v0.26.0; docker ghcr.io/motiftechnologies/vllm:v0.26.0-motif3), which the original card uses with the BF16 checkpoint — this quantized build has not been tested on that fork yet. The path checked here is Transformers with the bundled modeling code (trust_remote_code=True).

pip install -U transformers compressed-tensors accelerate flash-attn

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "axteam/Motif-3-NVFP4A16"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto", device_map="auto",
    attn_implementation="flash_attention_2")

messages = [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=1.0, top_p=0.95)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Motif-3's modeling code requires attn_implementation="flash_attention_2" (it raises an error otherwise). The checkpoint is large — device_map="auto" spreads it over all visible GPUs. Memory: Transformers (5.x) decompresses compressed-tensors weights to BF16 when the model is loaded / first run, so peak memory is about the BF16 model size (~590 GB for Motif-3) — use device_map="auto" across GPUs and CPU RAM. This build mainly reduces download and storage size; no inference engine runs Motif-3 with quantized kernels yet (vLLM 0.32 has no Motif architecture). Recommended sampling (from the original generation_config.json): temperature=1.0, top_p=0.95.

License

Quantized derivative of Motif-Technologies/Motif-3 (mit).

한국어 요약

Motif-Technologies/Motif-3의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	`Motif-Technologies/Motif-3`
양자화 방식	NVFP4A16 (W4A16, FP4 가중치)
저장 형식	compressed-tensors `nvfp4-pack-quantized`
모델 구조	MoE (Motif 구조, 전문가 384개 + 공유 전문가) · 53층 · 텍스트 전용
용량	약 180.8 GB
보정 데이터	없음 (가중치만으로 스케일 계산, 이름 기반 `model_free_ptq`)

양자화 상세

가중치: FP4(E2M1), 그룹 크기 16 + 그룹별 FP8 스케일
활성화: BF16 (양자화하지 않음)
양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 보정 데이터 없이 가중치만으로 스케일을 정해 반올림(RTN)
BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm, MoE 라우터(router.gate), 어텐션 출력 게이트(wq_b_gate), PolyNorm·mHC의 스칼라 파라미터, MTP 층 전체(mtp_layers.0) — 전체 파라미터의 약 1%
양자화 대상: 어텐션·라우팅 대상 전문가(routed expert)·공유 전문가·mHC의 선형층(projection)
원본 파일: 토크나이저·chat template은 원본과 동일
모델 코드: 이 리포의 modeling_motif.py 필수 — 원본은 전문가 가중치를 3D 텐서(gate_up_proj·down_proj)로 저장하는데, 이 빌드는 양자화하면서 전문가별로 분리했으므로 전문가마다 nn.Linear를 두도록 수정 (계산은 원본과 동일, 원본 모델 코드로는 전문가 가중치 로드 불가)

품질 (실측)

가중치 전수 대조 (2026-11-01): 53층의 양자화 모듈 59,547개를 compressed-tensors로 풀어 원본과 비교, 상대 오차 중앙값 9.5%·이상치 없음, BF16 유지 텐서 1,339개는 원본과 비트 단위로 일치 (가중치만 대조, 생성 품질·벤치마크는 미측정)

요구사항

실행 환경: Transformers (앞부분 층 로드까지 확인), 공식 vLLM 0.32는 Motif-3 미지원 — Motif의 vLLM fork(MotifTechnologies/vllm, v0.26.0 기반, docker ghcr.io/motiftechnologies/vllm:v0.26.0-motif3)는 원본 BF16용이며 이 양자화 빌드는 fork에서 실행 미확인
Transformers: transformers 5.x(5.7.0 이상) + compressed-tensors 0.19 이상 + trust_remote_code=True + flash-attn(attn_implementation="flash_attention_2", CUDA GPU) — 5.17.0 + 0.19.0으로 앞부분 층 로드까지 확인, 53층 전체 생성은 실행 미확인
메모리: 실행 시 약 590 GB (원본 BF16과 같음) — Transformers가 가중치를 BF16으로 풀어 쓰므로 주된 효과는 다운로드·저장 용량 절감, device_map="auto"로 여러 GPU와 CPU 메모리에 분산

사용법

설치: pip install -U transformers compressed-tensors accelerate flash-attn
로드: AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, attn_implementation="flash_attention_2", device_map="auto") → chat template 적용 후 generate (전체 예시는 위 영어 본문 'Usage' 절 참고)
권장 샘플링: temperature=1.0, top_p=0.95 (원본 generation_config.json 기준)

라이선스

라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
권리: 원본 모델의 권리는 Motif Technologies에게 있음, 이 리포는 양자화 파생물

Downloads last month: 341

Safetensors

Model size

315B params

Tensor type

BF16

Model tree for axteam/Motif-3-NVFP4A16

Base model

Motif-Technologies/Motif-3-Base

Finetuned

Motif-Technologies/Motif-3

Quantized

(15)

this model