Hskyto/lfm2.5-2.6b-toolcall-mlx-q4

LFM2.5-2.6B Tool-Call MLX (4-Bit · On-Device)

This repository contains the 4-bit quantized MLX version of Liquid AI's LFM2.5-2.6B, merged with a custom QLoRA adapter trained specifically for on-device tool and function calling on Apple Silicon and native iOS.

At ~1.5 GB it is built to run inside a real iOS app (via mlx-swift-lm) on modern iPhones, where the 16-bit and 8-bit builds exceed the per-app memory limit on 6 GB devices.

Model Details

• Base Model: LiquidAI/LFM2.5-2.6B
• Fine-Tuning: Custom QLoRA adapter trained for structured tool routing and argument normalization.
• Training Data: Salesforce/xlam-function-calling-60k
• Precision: 4-bit, group size 64 (affine MLX quantization)
• Framework: Apple MLX (mlx-lm / mlx-swift-lm)
• Siblings: 16-bit master · 8-bit

---

Capabilities & Strengths

• Argument Normalization: Transforms natural, casual phrasing (e.g. "call Mom at 6pm") into precise programmatic arguments (e.g. time='18:00').
• Multi-Tool Orchestration: Can emit parallel independent function calls in a single response.
• On-Device Footprint: 4-bit weights fit comfortably in memory on 6 GB-class iPhones, with a minor quality trade-off vs. the 8-bit build.

---

Custom Chat Template — reasoning toggle (enable_thinking)

LFM2 normally forces an opening <think> block on every response, so the model always produces a chain-of-thought before its tool call. On-device this reasoning is often discarded (only the tool call is used) yet still costs decode time.

This repo's chat_template.jinja adds an enable_thinking guard so reasoning can be turned off for faster tool calls:

{%- if add_generation_prompt -%}
    {%- if enable_thinking is defined and not enable_thinking -%}
        {{- "<|im_start|>assistant\n<think>\n\n</think>\n\n" -}}
    {%- else -%}
        {{- "<|im_start|>assistant\n<think>" -}}
    {%- endif -%}
{%- endif -%}

• Default (unchanged): if enable_thinking is not passed, the model reasons as usual — nothing breaks for existing callers.
• enable_thinking=false: the template emits a pre-closed empty think block, so the model skips reasoning and goes straight to the tool call — noticeably fewer decode tokens on-device.

---

How to Use (Python)

from mlx_lm import load, generate

model, tokenizer = load("Hskyto/lfm2.5-2.6b-toolcall-mlx-q4")

prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "You are an assistant with local iOS tools."},
        {"role": "user", "content": "Remind me to call Mom at 6pm"},
    ],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,   # skip chain-of-thought for a faster tool call
)

response = generate(model, tokenizer, prompt=prompt, verbose=True)
print(response)

Set enable_thinking=True (or omit it) to keep the reasoning trace.

How to Use (Swift · on-device)

With mlx-swift-lm, pass the flag through UserInput's additionalContext:

let userInput = UserInput(
    prompt: .text("Remind me to call Mom at 6pm"),
    tools: toolSchemas,
    additionalContext: ["enable_thinking": false]
)