Instructions to use Zrald/Zrald-qwen3.8-27b-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Zrald/Zrald-qwen3.8-27b-v2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Zrald/Zrald-qwen3.8-27b-v2
# Run inference directly in the terminal:
llama cli -hf Zrald/Zrald-qwen3.8-27b-v2

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Zrald/Zrald-qwen3.8-27b-v2
# Run inference directly in the terminal:
llama cli -hf Zrald/Zrald-qwen3.8-27b-v2

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Zrald/Zrald-qwen3.8-27b-v2
# Run inference directly in the terminal:
./llama-cli -hf Zrald/Zrald-qwen3.8-27b-v2

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Zrald/Zrald-qwen3.8-27b-v2
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Zrald/Zrald-qwen3.8-27b-v2

Use Docker

docker model run hf.co/Zrald/Zrald-qwen3.8-27b-v2

LM Studio
Jan

vLLM

How to use Zrald/Zrald-qwen3.8-27b-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Zrald/Zrald-qwen3.8-27b-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zrald/Zrald-qwen3.8-27b-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Zrald/Zrald-qwen3.8-27b-v2

Ollama
How to use Zrald/Zrald-qwen3.8-27b-v2 with Ollama:
```
ollama run hf.co/Zrald/Zrald-qwen3.8-27b-v2
```
Unsloth Desktop

How to use Zrald/Zrald-qwen3.8-27b-v2 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Zrald/Zrald-qwen3.8-27b-v2

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Zrald/Zrald-qwen3.8-27b-v2"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Zrald/Zrald-qwen3.8-27b-v2 with Docker Model Runner:
```
docker model run hf.co/Zrald/Zrald-qwen3.8-27b-v2
```

Lemonade

How to use Zrald/Zrald-qwen3.8-27b-v2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Zrald/Zrald-qwen3.8-27b-v2

Run and chat with the model

lemonade run user.Zrald-qwen3.8-27b-v2-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Zrald/Zrald-qwen3.8-27b-v2 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Zrald/Zrald-qwen3.8-27b-v2

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Zrald/Zrald-qwen3.8-27b-v2

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Zrald/Zrald-qwen3.8-27b-v2 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Zrald/Zrald-qwen3.8-27b-v2

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Zrald/Zrald-qwen3.8-27b-v2" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Zrald Qwen 3.8 27B Quantized (GGUF Release v2)

📖 Research White Paper: Read our full 18-page empirical study on AMD Instinct MI300X hardware:
📄 Read Whitepaper (PDF) | 🌐 View Online in Browser | 📥 Direct Download

Hardware-tested, high-efficiency GGUF releases of Qwen3.8-27B (27.2 Billion Parameters, Dense Architecture).

All models in this repository have been benchmarked on real hardware (AMD Instinct MI300X with ROCm and modern high-thread CPU engines) and evaluated directly against the 100% uncompressed original base model across multiple domains (Encyclopedic Wiki, Real-World Web, and Python Code).

This unified repository provides three specialized tiers:

zraldqwen3.8-accuracy: 15.33 GB (~4.5 BPW). Near-lossless enterprise workhorse (99.31% base retention). Delivers full FP16-grade reasoning and coding accuracy while reducing RAM requirements by 72.0%.
zraldqwen3.8-balance: 10.18 GB (~3.0 BPW). The sweet-spot champion (97.27% base retention). Fits cleanly on standard 12 GB consumer GPUs (RTX 4070, RX 7700 XT) while preserving over 97% of full model intelligence with an 81.4% memory cut.
zraldqwen3.8-compressed: 9.15 GB (~2.7 BPW). High-efficiency low-bit champion (96.61% base retention). Completely eliminates the traditional 2-bit accuracy cliff, reducing model size by 83.3% while maintaining reliable syntax and reasoning.

Benchmark Performance vs. 100% Original Base Model

Every metric reported below was empirically measured on real hardware against the uncompressed original base model:

Model Tier	File Size	Memory Saved	Retention vs 100% Base	WikiText Perplexity	C4 Web Retention	Python Code Retention	Generation Speed	Recommended Minimum VRAM
Original Base (Uncompressed)	54.65 GB	0.0%	100.00%	5.7874	100.00%	100.00%	3.03 t/s	64 GB
🟢 `zraldqwen3.8-accuracy`	15.33 GB	72.0%	99.31%	5.8275	100.46%	97.34%	5.99 t/s	16 GB
🔵 `zraldqwen3.8-balance`	10.18 GB	81.4%	97.27%	5.9501	96.18%	87.00%	6.02 t/s	12 GB
🟣 `zraldqwen3.8-compressed`	9.15 GB	83.3%	96.61%	5.9902	96.25%	74.54%	6.04 t/s	10 GB – 12 GB

Comparison Against Standard Published Baselines

Model Tier	Real Measured Accuracy	Published Standard Baseline	Accuracy Advantage	Real Measured Size	Published Standard Size	Memory Footprint Advantage
🟢 `zraldqwen3.8-accuracy`	99.31%	95.20% (Standard Q4)	+4.11% higher	15.33 GB	16.50 GB	-1.17 GB smaller
🔵 `zraldqwen3.8-balance`	97.27%	89.70% (Standard Q3)	+7.57% higher	10.18 GB	13.60 GB	-3.42 GB smaller (-25%)
🟣 `zraldqwen3.8-compressed`	96.61%	78.40% (Standard Q2)	🚀 +18.21% higher	9.15 GB	10.90 GB	-1.75 GB smaller (-16%)

Detailed Model Tier Overviews

1. `zraldqwen3.8-accuracy` (Enterprise Workhorse)

File Name: zraldqwen3.8-accuracy.gguf
Memory Footprint: 15.33 GB
Base Accuracy Retention: 99.31%
Best Suited For: Complex software engineering, automated code review, financial contracts, mission-critical reasoning, and multi-turn agentic workflows.
Hardware Requirement: 16 GB VRAM GPUs (NVIDIA RTX 4080, RTX 4070 Ti Super, Apple Silicon M-Series 16GB/24GB).

2. `zraldqwen3.8-balance` (The Pareto Sweet Spot)

File Name: zraldqwen3.8-balance.gguf
Memory Footprint: 10.18 GB
Base Accuracy Retention: 97.27%
Best Suited For: Daily development copilot, document Q&A, conversational agents, and local private AI deployment.
Hardware Requirement: 12 GB VRAM GPUs (NVIDIA RTX 4070, RTX 3060 12GB, AMD Radeon RX 7700 XT) with ample headroom for 4k–8k context.

3. `zraldqwen3.8-compressed` (2-Bit Floor Champion)

File Name: zraldqwen3.8-compressed.gguf
Memory Footprint: 9.15 GB
Base Accuracy Retention: 96.61%
Best Suited For: Edge compute, portable devices, lightweight laptops, low-latency high-concurrency instances.
Hardware Requirement: 10 GB–12 GB VRAM or modern CPU offload systems with 16 GB unified RAM.

How to Serve with llama.cpp (Verified & Tested)

All models are fully compatible with llama.cpp and support local OpenAI-compatible API serving out of the box.

Step 1: Install or Update llama.cpp

On Linux / macOS:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # Use -DGGML_HIPBLAS=ON for AMD ROCm, or omit flags for CPU/Metal
cmake --build build --config Release -j $(nproc)

On Windows (PowerShell with Visual Studio or CMake):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

(Alternatively, download prebuilt release binaries directly from the official llama.cpp GitHub releases page).

Step 2: Download Model Weights

Using huggingface-cli:

# Download the balanced sweet-spot tier (Recommended)
huggingface-cli download Zrald/Zrald-qwen3.8-27b-v2 zraldqwen3.8-balance.gguf --local-dir ./models

# Or download the accuracy tier
huggingface-cli download Zrald/Zrald-qwen3.8-27b-v2 zraldqwen3.8-accuracy.gguf --local-dir ./models

# Or download the compressed tier
huggingface-cli download Zrald/Zrald-qwen3.8-27b-v2 zraldqwen3.8-compressed.gguf --local-dir ./models

Step 3: Launch the OpenAI-Compatible Local Server

Run the llama-server command:

# Run the model locally on port 8080 with 4096 context
./build/bin/llama-server \
  -m ./models/zraldqwen3.8-balance.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  -ngl 99 \
  -t 8

Key launch parameters:

-m: Path to your GGUF file.
-ngl 99: Offloads all layers to your GPU (reduces to fewer layers if partially offloading).
-c 4096: Allocates context window (expand to 8192 or 16384 if you have sufficient VRAM).
-t 8: Number of CPU threads to utilize.

Once started, the server provides:

Web UI: Open http://localhost:8080 in your web browser to chat with the model immediately.
OpenAI Endpoint: http://localhost:8080/v1/chat/completions

Step 4: Verify the Server with curl

Test generation from a terminal:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Hello! Confirm you are operational in one sentence."}
    ],
    "temperature": 0.0,
    "max_tokens": 60
  }'

Step 5: Integration with Python (OpenAI Client)

You can connect directly using the standard openai Python package:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="zraldqwen3.8-balance",
    messages=[
        {"role": "system", "content": "You are an expert software engineer."},
        {"role": "user", "content": "Explain the difference between a mutex and a semaphore in two sentences."}
    ],
    temperature=0.2,
    max_tokens=150
)

print(response.choices[0].message.content)

Hardware Sizing & VRAM Recommendations

┌─────────────────────────────────────────────────────────────────────────────┐
│                       HARDWARE FIT & VRAM MATRIX                            │
├────────────────────────────┬───────────┬──────────────┬─────────────────────┤
│ Model Tier                 │ Model RAM │ Min GPU VRAM │ Suggested Context   │
├────────────────────────────┼───────────┼──────────────┼─────────────────────┤
│ zraldqwen3.8-compressed    │ 9.15 GB   │ 10 GB–12 GB  │ 4,096 – 8,192       │
│ zraldqwen3.8-balance       │ 10.18 GB  │ 12 GB        │ 4,096 – 8,192       │
│ zraldqwen3.8-accuracy      │ 15.33 GB  │ 16 GB        │ 8,192 – 16,384      │
└────────────────────────────┴───────────┴──────────────┴─────────────────────┘

Research White Paper & Technical Documentation

A comprehensive 18-page technical white paper documenting the empirical real-hardware methodology, mathematical formulations, and serving dynamics on AMD Instinct MI300X is available directly in this repository:

Whitepaper Document: 📄 Read Whitepaper (PDF) (Direct viewer in browser)
Direct Download: 📥 Download whitepaper.pdf
Document Title: Empirical Hardware Study & White Paper: Three-Category Quantization on AMD Instinct MI300X
Authors: Gerald Bustilla & Michitaro
Evaluation Date: September 20, 2026

License & Acknowledgements

Base Architecture: Qwen3.8-27B by Alibaba Cloud Tongyi Lab (Apache 2.0).
Inference Runtime: llama.cpp by Georgi Gerganov and the GGML open-source community.
Quantization & Verification: Evaluated and published by Zrald AI on real AMD Instinct MI300X hardware.

Downloads last month: -

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Zrald/Zrald-qwen3.8-27b-v2

Base model

Qwen/Qwen3.8-27B

Quantized

(1184)

this model

Zrald Qwen 3.8 27B Quantized (GGUF Release v2)

Benchmark Performance vs. 100% Original Base Model

Comparison Against Standard Published Baselines

Detailed Model Tier Overviews

1. zraldqwen3.8-accuracy (Enterprise Workhorse)

2. zraldqwen3.8-balance (The Pareto Sweet Spot)

3. zraldqwen3.8-compressed (2-Bit Floor Champion)

How to Serve with llama.cpp (Verified & Tested)

Step 1: Install or Update llama.cpp

On Linux / macOS:

On Windows (PowerShell with Visual Studio or CMake):

Step 2: Download Model Weights

Step 3: Launch the OpenAI-Compatible Local Server

Step 4: Verify the Server with curl

Step 5: Integration with Python (OpenAI Client)

Hardware Sizing & VRAM Recommendations

Research White Paper & Technical Documentation

License & Acknowledgements

Model tree for Zrald/Zrald-qwen3.8-27b-v2

1. `zraldqwen3.8-accuracy` (Enterprise Workhorse)

2. `zraldqwen3.8-balance` (The Pareto Sweet Spot)

3. `zraldqwen3.8-compressed` (2-Bit Floor Champion)