Igriscodes/qwen3-4b-tool-gguf

Qwen3-4B-Agentic-MCP-RL - GGUF

This repository contains the GGUF quantization files for Igriscodes/qwen-tool, a fine-tuned Qwen/Qwen3-1.7B model optimized for multi-step tool use and structured payload delivery via the Model Context Protocol (MCP).

The base model was aligned using Proximal Policy Optimization (PPO) on strict JSON validation, execution tracking, and tool-error recovery loops. These GGUF files allow for low-latency, low-memory local inference on edge devices, CPU-only systems, and Apple Silicon.

Available Quantizations

• Q2_K: Maximum compression. Significant loss in logic, not recommended for complex tool-use but fits on ultra-low-memory devices.
• Q3_K_M: Balanced 3-bit compression. Better logic than Q2, suitable for highly constrained memory footprints.
• Q4_0: Standard legacy 4-bit quantization. Faster on certain older hardware architectures but slightly lower quality than K-quants.
• Q4_K_M: Recommended. Optimal balance of reasoning performance, generation speed, and VRAM savings.
• Q5_0: Standard legacy 5-bit quantization. Good middle ground, but outpaced by K-quants.
• Q5_K_M: High quality 5-bit compression. Retains nearly all unquantized capabilities while saving substantial VRAM.
• Q6_K: 6-bit quantization. Near-zero degradation from F16 while shaving off a decent chunk of file size.
• Q8_0: Maximum 8-bit fidelity. Extremely close to native F16 performance, ideal for strict syntax and reliable tool-calling.
• F16: Unquantized baseline. High fidelity, near-native performance for systems with more memory overhead.

Local Deployment Quickstart

Using Ollama

Ollama supports running models directly from Hugging Face via the hf.co registry prefix. You can pull and run your preferred precision instantly:

# Q2_K (Extreme compression)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q2_K

# Q3_K_M (Medium 3-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q3_K_M

# Q4_0 (Legacy 4-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q4_0

# Q4_K_M (Recommended balanced version)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q4_K_M

# Q5_0 (Legacy 5-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q5_0

# Q5_K_M (High-fidelity 5-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q5_K_M

# Q6_K (Deep 6-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q6_K

# Q8_0 (Near-lossless 8-bit)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q8_0

# F16 (High-fidelity unquantized float version)
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:F16

or

Ollama Setup Guide

To run this model locally with full tool-calling (function calling) and thinking capabilities, you can easily package it into an Ollama model using the provided template configuration.

1. Create the Modelfile

Save the configuration block below exactly as a file named Modelfile in the same directory where your downloaded GGUF file is located.

💡 Note: If you are using a different quantization format than the q4_k_m example below, make sure to update the FROM line to match your exact .gguf filename.

# Point to your quantized GGUF file
FROM ./qwen3-4b-tool-q4_k_m.gguf

# Custom template optimizing tool-use syntax and thought blocks
TEMPLATE """{{- $lastUserIdx := -1 -}}
{{- range $idx, $msg := .Messages -}}
{{- if eq $msg.Role "user" }}{{ $lastUserIdx = $idx }}{{ end -}}
{{- end }}
{{- if or .System .Tools }}<|im_start|>system
{{ if .System }}{{ .System }}

{{ end }}
{{- if .Tools }}# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{{- range .Tools }}
{"type": "function", "function": {{ .Function }}}
{{- end }}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
{{- end -}}
<|im_end|>
{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}
<think>{{ .Thinking }}</think>
{{ end -}}
{{ if .Content }}{{ .Content }}{{ end }}
{{- if .ToolCalls }}
{{- range .ToolCalls }}
<tool_call>
{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}
</tool_call>
{{- end }}
{{- end }}{{ if not $last }}<|im_end|>
{{ end }}
{{- else if eq .Role "tool" }}<|im_start|>user
<tool_response>
{{ .Content }}
</tool_response><|im_end|>
{{ end }}
{{- if and (ne .Role "assistant") $last }}<|im_start|>assistant
<think>
{{ end }}
{{- end }}"""

# Inference parameters optimized for structured reasoning
PARAMETER temperature 0.6
PARAMETER num_ctx 8192
PARAMETER num_gpu -1
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

2. Build and Run the Model

Open your terminal, navigate to the directory containing your Modelfile and your .gguf file, and execute the build command:

ollama create qwen3-4b-tool --file Modelfile

Once the build process completes, you can launch and interact with your new custom model natively via Ollama:

ollama run qwen3-4b-tool

Using Python (llama-cpp-python)

First, ensure you have the library installed:

pip install llama-cpp-python

Depending on your hardware constraints, you can load either the uncompressed precision or the quantized version using the snippets below:

Option 1: High Fidelity (F16 Precision)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Igriscodes/qwen3-4b-tool-gguf",
    filename="qwen3-4b-tool-f16.gguf",
    n_ctx=2048,
    n_gpu_layers=-1 # Use -1 to offload all layers to GPU (Metal/CUDA)
)

Option 2: Low Resource (Q4 Quantization)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Igriscodes/qwen3-4b-tool-gguf",
    filename="qwen3-4b-tool-q4.gguf",
    n_ctx=2048,
    n_gpu_layers=-1 # Optimized for CPU execution or limited VRAM
)