chat_template ignores enable_thinking=false, and reasoning leaks into content under vLLM
Thanks for Flint. One integration note from serving the NVFP4 GGUF with vLLM (PR #62938, glm47 reasoning parser).
Clients that turn thinking off per request send chat_template_kwargs: {"enable_thinking": false}. Voice front ends and several agent frameworks do this for latency. Flint's template ignores the flag, so the model still reasons, but with the flag set to false the glm47 parser returns no reasoning and the whole <think>...</think> block arrives in content. A voice client then reads the reasoning aloud.
We added a branch that pre-fills a one line think block when the flag is false. Measured on raw completions at temperature 1.0, 8 samples per prompt:
| Pre-fill | Result |
|---|---|
<think></think> |
Terse prompts echoed the user's question (14/24) |
<think>We need answer.</think> |
No echo, but with tools present the model answered instead of calling a tool (7/24 calls where one was needed) |
<think>We need respond, using a tool if the request needs one.</think>, when tools are present |
24/24 calls where needed, 0/16 where not |
With short prompts that was clean: 12/12 tool calls where needed, 0/6 for small talk, no leaks or echoes, and short replies in 135 to 220 ms instead of 0.8 to 1 s. With our real voice prompt (about 45k tokens of instructions) it isn't.
On 30 replays of one turn the model answered, then carried on reasoning in plain text after the answer ("We need answer. The user is ...") in 4 of 30, whatever the pre-fill said. Letting it think fixes that but costs about 3.3 s before the first spoken word. We now cut the spoken stream at the first line in that reasoning style, which took it from 10 in 280 replies to 2.
The branch is below in case it is useful upstream. If there is a trained way to switch thinking off, we'd rather use that.
{%- if add_generation_prompt -%}
<|assistant|>
{%- if enable_thinking is defined and enable_thinking is false -%}
{%- if tools -%}
<think>We need respond, using a tool if the request needs one.</think>
{%- else -%}
<think>We need answer.</think>
{%- endif -%}
{%- endif -%}
{%- endif -%}
Separately, we checked two of your published numbers on one RTX PRO 6000 with the NVFP4 GGUF: LiveCodeBench v6 82.2% on a 45 problem sample and SWE-bench Verified 80% on a 50 task sample with no network, both in line with your NVFP4 results. Method, results and every agent transcript: https://github.com/DataCraftsmanAU/flint-independent-eval
It was pretty nice for instruction following too, actually fun to talk to when the thinking blocks work properly.
Will daily drive this for a while and see how it goes long term. Can't wait to see what you guys make next.