Instructions to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Terathox-Coder/Qwen3.8-27B-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Terathox-Coder/Qwen3.8-27B-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Ollama
How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Ollama:
```
ollama run hf.co/Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
```

Unsloth Studio

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Terathox-Coder/Qwen3.8-27B-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Terathox-Coder/Qwen3.8-27B-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Terathox-Coder/Qwen3.8-27B-MTP-GGUF to start chatting

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M
```

Lemonade

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-MTP-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Terathox-Coder/Qwen3.8-27B-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Terathox-Coder/Qwen3.8-27B-MTP-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

Qwen3.8-27B MTP GGUF — Q4_K_M

Community GGUF Q4_K_M quantization of Qwen/Qwen3.8-27B with the original MTP / NextN tensors preserved.

This repository contains a format conversion and quantization of the original Qwen3.8-27B checkpoint.

No fine-tuning or additional training has been performed.

Model Details

Property	Value
Base model	`Qwen/Qwen3.8-27B`
Parameters	27B
Quantization	`Q4_K_M`
Format	GGUF
MTP / NextN	Preserved
File size	~16.8 GB
Conversion	`llama.cpp`
License	Apache-2.0

Available File

Qwen3.8-27B-Q4_K_M-MTP.gguf

MTP / NextN

The original Qwen3.8-27B checkpoint contains Multi-Token Prediction / NextN tensors.

During conversion:

MTP enabled: 866 tensors
NextN disabled: 851 tensors
Difference: 15 MTP tensors

The 15 additional mtp.* tensors were intentionally preserved in this GGUF.

The current llama.cpp converter recognized the MTP export path and correctly mapped the NextN tensors into the additional model block.

llama.cpp Validation

The resulting Q4_K_M GGUF was successfully loaded and executed with llama.cpp using:

llama-cli \
  -m Qwen3.8-27B-Q4_K_M-MTP.gguf \
  --spec-type draft-mtp

The runtime successfully:

loaded the GGUF;
recognized the model architecture;
recognized the embedded MTP / NextN tensors;
constructed the main and MTP graphs;
processed the prompt;
generated output through the MTP-compatible runtime path.

Ollama

The GGUF has also been successfully imported and executed with Ollama 0.32.9.

Minimal Modelfile:

FROM ./Qwen3.8-27B-Q4_K_M-MTP.gguf

Create:

ollama create qwen3.8:27b-mtp-q4_K_M -f Modelfile

Run:

ollama run qwen3.8:27b-mtp-q4_K_M

MTP Configuration

For runtimes that expose MTP speculative decoding, the local TERATHOX configuration uses:

draft_num_predict = 4

Note that loading a GGUF containing MTP tensors does not by itself guarantee that a runtime is actively using speculative MTP decoding.

Users should verify MTP support and configuration for their specific runtime version.

Local TERATHOX Deployment

This quantization has been tested locally under the alias:

Terathox-Coder:Nova

Hardware

NVIDIA GeForce RTX 5080  16 GB
NVIDIA GeForce RTX 4070  12 GB
NVIDIA GeForce RTX 4070  12 GB

Three GPUs were used for the local validation.

Ollama Runtime Configuration

Context:                  204800
OLLAMA_FLASH_ATTENTION:   1
OLLAMA_VULKAN:            false
OLLAMA_KV_CACHE_TYPE:     q4_0
OLLAMA_SCHED_SPREAD:      false
OLLAMA_GPU_OVERHEAD:      0
OLLAMA_NUM_PARALLEL:      1
OLLAMA_MAX_LOADED_MODELS: 1
OLLAMA_KEEP_ALIVE:        -1
draft_num_predict:        4

Observed status:

NAME                   SIZE     PROCESSOR    CONTEXT
Terathox-Coder:Nova    24 GB    100% GPU     204800

Local Performance

Observed interactive generation performance:

Run	Eval rate
1	49.90 tok/s
2	49.52 tok/s
3	54.18 tok/s
4	49.32 tok/s

Typical observed generation range:

~49–54 tokens/s

Prompt evaluation varied depending on conversation state and cached context, reaching values from approximately:

51 tok/s → 263 tok/s

These are local hardware measurements and not standardized model benchmarks.

Performance depends on hardware, context size, GPU offload, KV-cache configuration, runtime version and MTP implementation.

Intended Use

This GGUF is intended for:

local text generation;
coding and software engineering;
agentic coding workflows;
technical reasoning;
long-context workloads;
experimentation with MTP / NextN speculative decoding;
local inference with llama.cpp or compatible GGUF runtimes.

Limitations

This is a quantized derivative of the original model.

Q4_K_M significantly reduces memory requirements but may introduce some quality degradation compared with the original BF16 checkpoint.

The base model may also produce inaccurate, biased or hallucinated information. Outputs should be independently verified for high-impact or safety-critical use cases.

Vision / Multimodal Support

The original Qwen3.8-27B model includes multimodal capabilities.

This repository currently provides the GGUF language-model artifact only.

No independently validated multimodal projector (mmproj) is currently included in this repository.

Therefore this release should currently be considered text-oriented unless an appropriate multimodal projector is added and validated.

Training

No training or fine-tuning was performed for this repository.

The original weights come from:

Qwen/Qwen3.8-27B

This repository only performs:

Original checkpoint
        ↓
GGUF BF16 with MTP preserved
        ↓
Q4_K_M quantization
        ↓
Qwen3.8-27B-Q4_K_M-MTP.gguf

Datasets

No additional dataset was used.

This repository does not contain a fine-tuned model.

Evaluation

No standardized quality benchmark was performed specifically on this quantization at the time of publication.

The performance results above measure local inference throughput only and should not be interpreted as accuracy or capability benchmarks.

For official capability benchmarks, refer to the original Qwen/Qwen3.8-27B model card.

Attribution

Original foundation model developed by the Qwen Team.

Base model:

Qwen/Qwen3.8-27B

GGUF conversion and Q4_K_M quantization:

Terathox-Coder

The original MTP / NextN tensors were preserved during conversion.

TERATHOX does not claim authorship or training of the original Qwen foundation model.

License

This repository follows the Apache License 2.0 of the base model.

Please review the original Qwen3.8-27B repository and license for additional information.

Disclaimer

This is a community conversion and is not an official Qwen release.

Compatibility, performance and MTP behavior may vary between versions of llama.cpp, Ollama and other GGUF runtimes.

Downloads last month: -

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for Terathox-Coder/Qwen3.8-27B-MTP-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(255)

this model