Instructions to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
# Run inference directly in the terminal:
llama cli -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
# Run inference directly in the terminal:
llama cli -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
# Run inference directly in the terminal:
./llama-cli -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Use Docker

docker model run hf.co/Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

LM Studio
Jan

vLLM

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Atomic-Germ/Qwen3.8-Distilled-4B-NPU2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Atomic-Germ/Qwen3.8-Distilled-4B-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Ollama
How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Ollama:
```
ollama run hf.co/Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
```

Unsloth Studio

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 to start chatting

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Docker Model Runner:
```
docker model run hf.co/Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1
```

Lemonade

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Run and chat with the model

lemonade run user.Qwen3.8-Distilled-4B-NPU2-Q4_1

List all available models

lemonade list

Hermes Agent

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Atomic-Germ/Qwen3.8-Distilled-4B-NPU2:Q4_1" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8 4B Distilled — Q4NX

FastFlowLM Q4NX conversion of Ma7ee7/Qwen3.8_4B_Distilled for AMD XDNA NPU inference.

This repository contains a quantized Q4NX port of the model, compiled for the FastFlowLM (FLM) runtime. It is not a GGUF file.

Qwen3.8 4B Distilled is a 4-billion-parameter reasoning model created by distilling outputs from Qwen3.8-Max into the smaller Qwen3-4B-Thinking-2507 student model.

The full model was fine-tuned on r0b0tlab/qwen3.8-max-distillation-50k, a dataset of responses and reasoning traces generated by qwen3.8-max-preview.

What is Q4NX?

Q4NX is FastFlowLM's native packed-quantization format - a rearranged Q4_1 layout tuned for the NPU matrix engine's tile sizes and memory access patterns. It is not a GGUF file and it does not run on llama.cpp or Ollama; it is meant exclusively for the FastFlowLM engine on AMD Ryzen AI NPUs.

Requirements

FastFlowLM >= 0.9.45 (flm CLI)
AMD Ryzen AI processor with XDNA2 (NPU2) - Strix Point / Ryzen AI 300 series or later
Linux with the XRT NPU stack installed
~16 GB of unified system memory (Q4NX weights + activations + KV cache)

FLM Bench

Tested on an AMD Ryzen AI 340 Framework 13 laptop.

Context Length	TTFT (s) (mean ± std)	Prefill Speed (tok/s) (mean ± std)	Decoding Speed (tok/s) (mean ± std)
1k	2.973 ± 0.077	355.72 ± 9.25	12.31 ± 0.25
2k	4.337 ± 0.050	466.84 ± 12.27	12.15 ± 0.26
4k	8.283 ± 0.014	478.02 ± 4.65	11.01 ± 0.12
8k	17.243 ± 0.065	454.02 ± 0.02	9.38 ± 0.01
16k	41.308 ± 0.071	376.84 ± 0.00	7.27 ± 0.07
32k	116.909 ± 0.294	265.53 ± 0.06	5.06 ± 0.00

Files

File	Purpose
model.q4nx	Quantized Q4NX weights
config.json	FastFlowLM model configuration
tokenizer.json	Tokenizer
tokenizer_config.json	Special tokens and chat template
chat_template.jinja	Chat template (optional)

Install and run

This repository works with flm-add, a small installer that copies the model into the FastFlowLM user directory and registers the tag. It never modifies the system FastFlowLM install.

pip install flm-add or uv tool install flm-add

uv tool install flm-add
flm-add Atomic-Germ/Qwen3.8-Distilled-4B-NPU2 --tag qwen3.8-distilled:4b --family qwen3-tk
FLM_CONFIG_PATH="$HOME/.config/flm/model_list.json" FLM_XCLBIN_PATH="$HOME/.config/flm" flm run qwen3.8-distilled:4b

Kernels

FastFlowLM's NPU kernels (xclbins) are closed source and are not shipped in this repository. flm-add links the kernels of the official qwen3-tk:4b model (Qwen3-4B-Thinking-2507-NPU2), because this model shares the same engine family (qwen3) and architecture.

Model Lineage and Naming

The name Qwen3.8 4B Distilled describes the model's distillation lineage:

Teacher model: qwen3.8-max-preview
Student/base model: Qwen/Qwen3-4B-Thinking-2507
Full-precision fine-tune: Ma7ee7/Qwen3.8_4B_Distilled
Resulting model size: Approximately 4 billion parameters
Distillation dataset: r0b0tlab/qwen3.8-max-distillation-50k
Distribution format: Q4NX

This is a Qwen3-architecture student model distilled from Qwen3.8-Max-generated outputs.

The repository does not claim that the underlying architecture or original weights are from Qwen3.8-Max. Qwen3.8-Max is the teacher whose generated responses and reasoning traces were used as training targets for the 4B student.

This repository contains quantized conversions of that independently fine-tuned student model. It is not an official Qwen or Alibaba release.

Model Details

Property	Value
Model type	Decoder-only causal language model
Architecture	Qwen3
Parameters	Approximately 4B
Student/base model	`Qwen/Qwen3-4B-Thinking-2507`
Teacher model	`qwen3.8-max-preview`
Full model	`Ma7ee7/Qwen3.8_4B_Distilled`
Training method	Sequence-level supervised distillation
File format	Q4NX
Primary task	Reasoning and conversational text generation
Primary language	English
Thinking mode	Enabled

What “Distilled” Means Here

This model uses sequence-level knowledge distillation.

The smaller student was trained on complete responses produced by the larger teacher. This transfers parts of the teacher's behavior, reasoning patterns, solution structure, and response style without copying the teacher's architecture or weights.

Therefore:

The architecture and original student weights come from Qwen3-4B-Thinking-2507.
The distillation targets come from Qwen3.8-Max-generated outputs.
The resulting checkpoint remains a 4B Qwen3 model.
These Q4NX file is a quantized versions of that 4B distilled model.
The model is not expected to reproduce the full capabilities of Qwen3.8-Max.

Thinking Output

The model inherits a thinking-oriented chat format from Qwen3-4B-Thinking-2507. Depending on the inference application and reasoning parser, visible reasoning may be displayed in a form similar to:

[FLM]  Model RAW Output: 
<think>
The user is asking a straightforward factual question: "What is the capital of France?"

Since no specific tools or functions are available for answering such questions, I'll simply provide the answer directly.
</think>

The capital of France is **Paris**.

Some applications may hide the thinking section, place it in a collapsible panel, or display it separately from the final answer.

Distillation Dataset

The full model was trained on:

r0b0tlab/qwen3.8-max-distillation-50k

The dataset contains teacher-generated examples across areas including:

Mathematics
Programming
General reasoning
Scientific reasoning
Instruction following
Limited tool use

Teacher responses were generated by qwen3.8-max-preview. Visible <think>...</think> reasoning traces were retained when present in the dataset.

Intended Uses

This model is intended for local experimentation with:

Mathematical reasoning
Programming and code generation
Logical reasoning
Scientific question answering
General instruction following
Long-form problem solving
Local conversational assistants
Research into teacher-to-student distillation
FastFlowLM inference

Limitations

This is a quantized 4B student model and does not contain the complete capabilities of Qwen3.8-Max.
Quantization may reduce quality relative to the full-precision model.
Lower-bit quantizations may degrade more noticeably on difficult reasoning or programming tasks.
Distillation transfers patterns from teacher-generated outputs; it does not copy the teacher's architecture or weights.
Teacher-generated answers may contain factual, mathematical, or programming errors.
Visible reasoning traces should not automatically be assumed to be correct.
The model may hallucinate information or produce confidently incorrect answers.
The training mixture is predominantly English.
The training dataset includes examples originating from common evaluation benchmarks.
Results on overlapping benchmarks may be contaminated and should not be treated as clean evaluations without additional controls.
Outputs should be reviewed before use in high-stakes, medical, financial, legal, or security-sensitive settings.

License and Training-Data Notice

This model repository is published under the Apache License 2.0.

This license does not override the licenses, attribution requirements, or usage terms associated with:

The full-precision fine-tuned model
The Qwen3 base model
The Qwen3.8-Max teacher provider
The distillation dataset
The original datasets from which prompts were sourced

Users are responsible for reviewing the relevant upstream model, dataset, and provider terms before redistribution or commercial use.

Acknowledgements

This model builds upon work from:

The Qwen team for Qwen3-4B-Thinking-2507
r0b0tlab for the Qwen3.8-Max Distillation 50K dataset
llama.cpp and the GGUF ecosystem
AMD's FastFlowLM team and NPU
Hugging Face

Citation

Distillation Dataset

@misc{r0b0tlab2026qwen38distillation50k,
  title        = {Qwen3.8-Max Distillation 50K},
  author       = {r0b0tlab},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/r0b0tlab/qwen3.8-max-distillation-50k}}
}

Disclaimer

Qwen3.8 4B Distilled GGUF is an independent community release by Ma7ee7.

It is not produced, endorsed, or officially released by the Qwen team, Alibaba, or Alibaba Cloud.

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}

Downloads last month: 2,269

GGUF

Model size

4B params

Architecture

qwen3

Hardware compatibility

4-bit

Model tree for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Base model

Qwen/Qwen3-4B-Thinking-2507

Finetuned

Ma7ee7/Qwen3.8_4B_Distilled

Quantized

(2)

this model

Dataset used to train Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Collections including Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

NPU2/Q4NX

Collection

These work with FastFlowLM • 21 items • Updated 2 days ago

GhostWriter App

Collection

Best models for the writing assistant app GhostWriter which doesn't try to write the book for you. Find at http://github.com/Atomic-Germ/GhostWriter • 9 items • Updated 4 days ago

Paper for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Qwen3 Technical Report

Paper • 2505.09388 • Published May 14, 2025 • 344

Atomic-Germ
/

Qwen3.8-Distilled-4B-NPU2

Qwen3.8 4B Distilled — Q4NX

What is Q4NX?

Requirements

FLM Bench

Files

Install and run

Kernels

Model Lineage and Naming

Links

Model Details

What “Distilled” Means Here

Thinking Output

Distillation Dataset

Intended Uses

Limitations

License and Training-Data Notice

Acknowledgements

Citation

Distillation Dataset

Disclaimer

Model tree for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Dataset used to train Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Collections including Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

NPU2/Q4NX

GhostWriter App

Paper for Atomic-Germ/Qwen3.8-Distilled-4B-NPU2

Qwen3 Technical Report