Instructions to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF",
	filename="Unsloth-Qwen3.6-27B-NVFP4-A.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
./llama-cli -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Use Docker

docker model run hf.co/CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

LM Studio
Jan

vLLM

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Ollama
How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Ollama:
```
ollama run hf.co/CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
```

Unsloth Studio

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF to start chatting

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4
```

Lemonade

How to use CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF:NVFP4

Run and chat with the model

lemonade run user.Unsloth-Qwen3.6-27B-NVFP4-GGUF-NVFP4

List all available models

lemonade list

Unsloth-Qwen3.6-27B-NVFP4 - GGUF

Quantized GGUF versions of unsloth/Qwen3.6-27B-NVFP4 from 2026-07-10. These were generated using llama.cpp (b9859).

Unsloth-Qwen3.6-27B-NVFP4-A.gguf: The NVFP4 group 1 layers are preserved. The FP8 group 0 linear-attention (attn_qkv, ssm_out) and tail-MLP (layers 56–63) are converted to NVFP4, while the remaining layers are kept at Q8. The MTP draft layer is preserved at BF16. This was generated from NVFP4-Q8 using a modified llama-quantize to allow re-quantizing Q8-quantized layers to NVFP4.
Unsloth-Qwen3.6-27B-NVFP4-Q8.gguf: The NVFP4 group 1 layers are preserved, while the FP8 group 0 layers are converted to Q8. This was generated using a modified convert_hf_to_gguf.py to support Unsloth's differing groups.
Unsloth-Qwen3.6-27B-NVFP4-BF16.gguf: The NVFP4 group 1 layers are preserved, while the FP8 group 0 layers are upcast to BF16. This version is not published because it is redundant with NVFP4-Q8.

Quantizations provided

File	Quantization	Size
Unsloth-Qwen3.6-27B-NVFP4-A.gguf	Mixed Q8/NVFP4 group 0, NVFP4 group 1	20.1 GB
Unsloth-Qwen3.6-27B-NVFP4-Q8.gguf	Q8 group 0, NVFP4 group 1	23.2 GB

Perplexity test

I tested perplexity using llama-perplexity and Salesforce's wikitext-2-raw-v1.

File	Ctx	PPL
Unsloth-Qwen3.6-27B-NVFP4-A.gguf	512	7.5789 ± 0.05260
Unsloth-Qwen3.6-27B-NVFP4-Q8.gguf	512	7.3792 ± 0.05057
Unsloth-Qwen3.6-27B-NVFP4-BF16.gguf	512	7.3702 ± 0.05052

Evaluation

The following models were evaluated for a fair comparison of capability, size and speed.

Model	Quantization	Size	Reason
CodeFault/Nvidia-Qwen3.6-27B-MTP-GGUF	NVFP4-A¹	17.9 GB	Alternative NVFP4 quant.
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL	20.4 GB	Closest non-NVFP4 in size to NVFP4-A.
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL	26 GB	Closest non-NVFP4 in size to NVFP4-Q8.
unsloth/Qwen3.6-27B-NVFP4	NVFP4²	25.4 GB	Previous NVFP4 checkpoint from 2026-05-31.

1: My quantization of nvidia/Qwen3.6-27B-MTP.

2: unsloth/Qwen3.6-27B-NVFP4 does not provide a GGUF. I used llama.cpp's conversion which passes through Unsloth's NVFP4 tensors. This is Unsloth's previous version from 2026-05-31.

	CodeFault Unsloth NVFP4-A	CodeFault Unsloth NVFP4-Q8	CodeFault Nvidia NVFP4-A	Unsloth NVFP4 (2026-05-31)	Unsloth UD-Q5_K_XL	Unsloth UD-Q6_K_XL
Coding
HumanEval	0.8354 ± 0.0290	0.8293 ± 0.0295	0.8293 ± 0.0295	0.8110 ± 0.0307	0.8598 ± 0.0272	0.8537 ± 0.0277
HumanEval+	0.7622 ± 0.0333	0.7622 ± 0.0333	0.7683 ± 0.0330	0.7744 ± 0.0327	0.7866 ± 0.0321	0.7805 ± 0.0324
MBPP	0.7540 ± 0.0193	0.7720 ± 0.0188	0.7280 ± 0.0199	0.7420 ± 0.0196	0.7540 ± 0.0193	0.7540 ± 0.0193
MBPP+	0.8836 ± 0.0165	0.8915 ± 0.0160	0.8783 ± 0.0168	0.8995 ± 0.0155	0.8968 ± 0.0157	0.8836 ± 0.0165
Instruction
IFEval	0.8540 ± 0.0152	0.8577 ± 0.0150	0.8558 ± 0.0151	0.8447 ± 0.0156	0.8558 ± 0.0151	0.8447 ± 0.0156
Knowledge
ARC-Challenge	0.9735 ± 0.0047	0.9718 ± 0.0048	0.9710 ± 0.0049	0.9710 ± 0.0049	0.9684 ± 0.0051	0.9710 ± 0.0049
MMLU-Pro	0.8291 ± 0.0034	0.8288 ± 0.0034	0.8250 ± 0.0034	0.8293 ± 0.0033		0.8319 ± 0.0033
STEM & Reasoning
BIG-Bench Hard	0.9201 ± 0.0032	0.9174 ± 0.0032	0.9149 ± 0.0032	0.9131 ± 0.0033	0.9220 ± 0.0031	0.9214 ± 0.0031
GPQA Diamond flexible	0.7929 ± 0.0289	0.7929 ± 0.0289	0.7929 ± 0.0289	0.8182 ± 0.0275	0.8131 ± 0.0278	0.7778 ± 0.0296
GSM8K	0.9151 ± 0.0077	0.9136 ± 0.0077	0.9196 ± 0.0075	0.9098 ± 0.0079	0.9128 ± 0.0078	0.9158 ± 0.0076
Hendrycks Math	0.3698 ± 0.0066	0.3624 ± 0.0066	0.3438 ± 0.0065	0.3662 ± 0.0066	0.3778 ± 0.0067	0.3876 ± 0.0067

NOTICE: These evals are actively running.

These evaluations were run using lm_eval. The models were run in instruct (non-thinking) mode with the following parameters in llama-server (b9775):

ctx-size = 32768
cache-type-k = q8_0
cache-type-v = q8_0
top-p = 0.8
top-k = 20
min-p = 0
presence-penalty = 1.5
spec_type = draft-mtp
spec_draft_n_max = 2
chat-template-kwargs = {"enable_thinking":false}

Benchmarks

Model	Quant	MTP n-max	Prompt Len	Output Len	Acceptance Rate	pp/s	tg/s
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A		267	6654		2205.4	70.6
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	1	267	7240	0.880	1880.7	91.8
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	2	267	5842	0.856, 0.712	1853.9	109.8
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	3	267	5711	0.852, 0.712, 0.596	1782.3	119.7
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	4	267	9709	0.841, 0.688, 0.556, 0.462	1738.4	122.7
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-Q8		267	6544		2077.4	63.0
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-Q8	1	267	6596	0.876	1742.8	84.3
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-Q8	2	267	7088	0.868, 0.724	1766.4	102.8
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-Q8	3	267	5903	0.831, 0.675, 0.543	1785.1	108.8
CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF	NVFP4-Q8	4	267	5785	0.837, 0.684, 0.563, 0.471	1771.2	116.3
CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A		267	8676		2225.1	76.6
CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	1	267	5958	0.879	1877.0	99.5
CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	2	267	6708	0.856, 0.701	1897.2	119.1
CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	3	267	6900	0.855, 0.699, 0.559	1950.4	128.7
CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF	NVFP4-A	4	267	5635	0.836, 0.681, 0.567, 0.483	1862.6	137.6
unsloth/Qwen3.6-27B-NVFP4	NVFP4³		267	7347		2056.5	57.6
unsloth/Qwen3.6-27B-NVFP4	NVFP4³	1	267	6826	0.843	1749.9	74.9
unsloth/Qwen3.6-27B-NVFP4	NVFP4³	2	267	8142	0.851, 0.685	1794.7	90.1
unsloth/Qwen3.6-27B-NVFP4	NVFP4³	3	267	7612	0.837, 0.671, 0.541	1787	96.4
unsloth/Qwen3.6-27B-NVFP4	NVFP4³	4	267	7621	0.817, 0.620, 0.485, 0.400	1772.4	96.5
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL		267	7866		1522.0	64.0
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL	1	267	6977	0.866	1296.4	99.0
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL	2	267	7418	0.850, 0.686	1383.8	115.0
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL	3	267	9204	0.850, 0.684, 0.565	1305.6	118.9
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q5_K_XL	4	267	7600	0.855, 0.677, 0.565, 0.472	1294.0	123.5
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL		267	7180		1257.3	53.7
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL	1	267	5868	0.876	1249.1	84.8
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL	2	267	6104	0.864, 0.701	1232.8	102.2
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL	3	267	5000	0.847, 0.688, 0.563	1228.7	109
unsloth/Qwen3.6-27B-MTP-GGUF	UD-Q6_K_XL	4	267	7060	0.852, 0.703, 0.577, 0.474	1052.5	116.8

3: This is Unsloth's previous version from 2026-05-31.

These benchmarks were run on an RTX 5090 (limited to 480 W) using llama-cli (b9775) with the CUDA driver and a prompt to generate an Ansible playbook.

Serving with llama.cpp

It has a max context size of 262,114. This can be served using:

llama-server \
-hf CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF \
-hff Unsloth-Qwen3.6-27B-NVFP4-A.gguf \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--repeat-penalty 1.1 \
--spec-type draft-mtp \
--spec-draft-n-max 3

Downloads last month: 3,485

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for CodeFault/Unsloth-Qwen3.6-27B-NVFP4-GGUF

Base model

Qwen/Qwen3.6-27B

Quantized

unsloth/Qwen3.6-27B-NVFP4

Quantized

(5)

this model