XMB480/Qwen3.6-fixed-tool-calling

Qwen 27B - Hermes Optimized JSON Agent (GGUF)

This repository hosts a pre-configured deployment setup for the Qwen 27B model, specifically optimized to run as a local autonomous agent within structured tool-calling frameworks like Hermes.

The Problem Solved

When deploying local agent frameworks with standard Qwen GGUF files, the agent frequently breaks down during tool execution. The agent framework expects the model to communicate tool calls in strict, structured JSON formats.

However, the default chat template baked into standard Qwen models often causes the text generation to drift. The model regular attempts to wrap tool calls in raw XML blocks, markdown code wrappers, or outputs casual conversational text alongside the payload. This breaks the framework's parser, leading to continuous loop errors, missed execution arguments, and failed automation workflows.

The Fix

This configuration fixes the structural drift by overriding the default chat translation layer.

We updated the environment's chat_template.jinja file to explicitly inject strict JSON tool constraints (_tool_format = 'json'). This forces the underlying model weights to process inputs and format outputs into predictable, rock-solid JSON payloads, ensuring clean tool execution loop completion without breaking the parser.

---

Repository Contents

• Qwen3.6-27B-Q4_K_M.gguf: The core text generation model quantized to 4-bit medium precision.
• mmproj-Qwen3.6-27B-BF16.gguf: The multimodal vision projector block, allowing the text model to interpret images and screenshots via local browser automation tools.
• chat_template.jinja: The custom Jinja template file that enforces the JSON formatting constraints.

---

How to Deploy and Use

Because GGUF engines default to using the template embedded inside the binary file, you must manually apply the custom Jinja template included in this repository to activate the fix.

Method 1: Linux / Arch Terminal (Recommended Engine)

For maximum efficiency and minimal resource overhead, run the model using raw llama.cpp. Pass the text weights, vision projector, and the custom template file directly via the terminal command:

llama-server -m Qwen3.6-27B-Q4_K_M.gguf --mmproj mmproj-Qwen3.6-27B-BF16.gguf --port 1234 -ctx 65000 -ngl 47 --chat-template-file chat_template.jinja