Instructions to use Atomic-Germ/Grug-35B-A3B-NPU2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Atomic-Germ/Grug-35B-A3B-NPU2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Atomic-Germ/Grug-35B-A3B-NPU2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Atomic-Germ/Grug-35B-A3B-NPU2")
model = AutoModelForMultimodalLM.from_pretrained("Atomic-Germ/Grug-35B-A3B-NPU2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Atomic-Germ/Grug-35B-A3B-NPU2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Atomic-Germ/Grug-35B-A3B-NPU2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Atomic-Germ/Grug-35B-A3B-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Atomic-Germ/Grug-35B-A3B-NPU2

SGLang

How to use Atomic-Germ/Grug-35B-A3B-NPU2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Atomic-Germ/Grug-35B-A3B-NPU2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Atomic-Germ/Grug-35B-A3B-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Atomic-Germ/Grug-35B-A3B-NPU2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Atomic-Germ/Grug-35B-A3B-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Atomic-Germ/Grug-35B-A3B-NPU2 with Docker Model Runner:
```
docker model run hf.co/Atomic-Germ/Grug-35B-A3B-NPU2
```

IF YOU USE COMMUNITY QWEN MODELS DO NOT UPGRADE TO FLM v1.0.2+

Grug-35B-A3B - Q4NX for FastFlowLM (AMD Ryzen AI XDNA2)

Grug 35B A3B is a compact-reasoning fine-tune of Qwen/Qwen3.6-35B-A3B. It keeps the Qwen MoE/A3B architecture: 35B total parameters with roughly 3B activated per token, 40 text layers, 256 experts, and 8 routed experts per token plus the shared expert path. Converted to Q4NX for FastFlowLM. This variant ships vision Q4NX weights alongside the text weights.

What is Q4NX?

Q4NX is FastFlowLM's native packed-quantization format - a rearranged Q4_1 layout tuned for the NPU matrix engine's tile sizes and memory access patterns. It is not a GGUF file and it does not run on llama.cpp or Ollama; it is meant exclusively for the FastFlowLM engine on AMD Ryzen AI NPUs.

Requirements

FastFlowLM >= 0.9.45 (flm CLI)
AMD Ryzen AI processor with XDNA2 (NPU2) - Strix Point / Ryzen AI 300 series or later
Linux with the XRT NPU stack installed
~51 GB of unified system memory (Q4NX weights + activations + KV cache)

Files

File	Purpose
model.q4nx	Quantized Q4NX weights
config.json	FastFlowLM model configuration
tokenizer.json	Tokenizer
tokenizer_config.json	Special tokens and chat template
chat_template.jinja	Chat template (optional)
vision_weight.q4nx	Vision tower weights (multimodal input)
flm-add.py	Installer script - registers this model with FastFlowLM

Install and run

This repository works with flm-add, a small installer that copies the model into the FastFlowLM user directory and registers the tag. It never modifies the system FastFlowLM install.

pip install flm-add or uv tool install flm-add

uv tool install flm-add
flm-add Atomic-Germ/Grug-35B-A3B-NPU2 --tag grug-moe:35b-a3b --family qwen3.6-moe
FLM_CONFIG_PATH="$HOME/.config/flm/model_list.json" FLM_XCLBIN_PATH="$HOME/.config/flm" flm run grub-moe:35b-a3b

Kernels

FastFlowLM's NPU kernels (xclbins) are closed source and are not shipped in this repository. flm-add.py links the kernels of the official qwen3.6-moe:35b-a3b model (Qwen3.6-35B-A3B-NPU2), because this model shares the same engine family (qwen3.6-moe) and architecture.

Model

Registry tag: qwen3.6-a3b:35b
Engine family: qwen3.6-moe
Kernel source: Qwen3.6-35B-A3B-NPU2
Context length: 262,144 tokens (from config)
model.q4nx size: 23.24 GB
Base model: kai-os/Grug-35B-A3B
License: apache-2.0

Original model card

See the upstream model card for training details, benchmarks, and upstream usage. This repository only contains the Q4NX conversion for FastFlowLM.

Upstream card: kai-os/Grug-35B-A3B

Downloads last month: 628

Model tree for Atomic-Germ/Grug-35B-A3B-NPU2

Base model

Qwen/Qwen3.6-35B-A3B

Finetuned

kai-os/Grug-35B-A3B

Quantized

(4)

this model

Atomic-Germ
/

Grug-35B-A3B-NPU2