Qwen2.5-1.5B-Instruct function-calling LoRA adapters (MLX)

Small LoRA adapters that teach Qwen2.5-1.5B-Instruct to do function (tool) calling: read a request, pick the right tool from a list, and fill in its arguments, or say that no tool fits. Trained with MLX on a MacBook Pro (Apple M3 Pro, 18 GB). Full code, notebook and logs: https://github.com/neerajprao/Fine-tuning-Qwen1.5B.

Where	Adapter	Base model to load it on	Size
repo root (`adapters.safetensors`)	Run B: LoRA on bf16, the best run	`Qwen/Qwen2.5-1.5B-Instruct`	21.1 MB
`qlora-4bit/`	Run A: QLoRA on the 4-bit model	`mlx-community/Qwen2.5-1.5B-Instruct-4bit`	21.1 MB

Both are rank-8 LoRA adapters. They are in MLX format (lora_a / lora_b keys), so they load with mlx_lm, not directly with 🤗 PEFT. Each adapter must be used on the base model it was trained on.

Training

Dataset: Salesforce/xlam-function-calling-60k (CC BY 4.0), converted to Qwen's OpenAI-style tools chat format. 10% of all tools were held out of training to build the "unseen tools" test, and extra "irrelevance" examples (no tool fits, the assistant politely declines) were added. Settings (copied from the GitHub README):

Setting	Value
Base models	`mlx-community/Qwen2.5-1.5B-Instruct-4bit`, `Qwen/Qwen2.5-1.5B-Instruct`
LoRA	rank 8, scale 20, dropout 0, last 16 of 28 layers, all 7 linear layers per layer
Training	learning rate 1e-4 (3% warm-up, cosine decay to 1e-5), Adam, batch 4 × accumulation 2, gradient checkpointing, prompt masking
Length	840 iterations per run (equal for A, B, C), max sequence length 1,216 tokens
Data	12,000 training (8% "no tool fits"), 300 validation, test sets of 400 seen / 400 unseen / 200 irrelevance
Evaluation	greedy decoding, max 275 new tokens, 1,000 bootstrap resamples

	Run A (QLoRA 4-bit)	Run B (LoRA bf16)
Training examples	12,000	12,000
Iterations / epochs	840 / 0.28	840 / 0.28
Validation loss, start → end	0.426 → 0.044	0.495 → 0.043
Peak memory	5.61 GB	7.75 GB
Training time	1.40 h	1.23 h

Run B is called "best" because it has the highest average exact match over seen + unseen tools; the difference to Run A is within the error bars (see below).

Results

Held-out test questions, greedy decoding. Exact match = every function and every argument right. Irrelevance = correctly making no tool call when no tool fits. Brackets are 95% bootstrap confidence intervals (1,000 resamples). n = 400 seen, 400 unseen, 200 irrelevance.

Model	Seen tools (exact match)	Unseen tools (exact match)	Irrelevance (correct "no call")
Base 4-bit, zero-shot	67.0% [62.2%, 71.5%]	67.2% [62.5%, 72.2%]	72.0% [65.0%, 78.0%]
Base 4-bit, 3-shot	64.0% [59.2%, 68.5%]	63.7% [58.5%, 68.8%]	94.5% [91.0%, 97.5%]
Base bf16, zero-shot	75.0% [70.5%, 79.0%]	74.8% [70.5%, 79.0%]	56.5% [49.5%, 63.0%]
Run A: QLoRA 4-bit, 12k (this repo, `qlora-4bit/`)	75.2% [71.2%, 79.2%]	77.5% [73.8%, 81.8%]	100.0% [100.0%, 100.0%]
Run B: LoRA bf16, 12k (this repo, root)	78.2% [74.2%, 82.0%]	76.5% [72.2%, 80.8%]	99.0% [97.5%, 100.0%]
Run C: QLoRA 4-bit, 2k	75.5% [71.0%, 79.8%]	77.2% [73.2%, 81.2%]	99.0% [97.5%, 100.0%]

All metrics for all six models

Model	Test set	Valid format	Function-name acc.	Argument acc.*	Exact match	Call-count acc.	No-call rate
Base 4-bit, zero-shot	seen (n=400)	97.5% [96.0%, 99.0%]	95.8% [93.8%, 97.5%]	86.4% [83.5%, 89.1%]	67.0% [62.2%, 71.5%]	94.8% [92.5%, 96.8%]	1.2% [0.2%, 2.5%]
Base 4-bit, zero-shot	unseen (n=400)	95.8% [93.8%, 97.5%]	95.2% [93.2%, 97.2%]	82.7% [79.8%, 85.7%]	67.2% [62.5%, 72.2%]	91.5% [88.8%, 94.2%]	3.8% [2.0%, 5.8%]
Base 4-bit, zero-shot	irrelevance (n=200)	97.0% [94.5%, 99.0%]	–	–	–	–	irrelevance acc. 72.0% [65.0%, 78.0%]
Base 4-bit, 3-shot	seen (n=400)	92.2% [89.5%, 94.8%]	89.8% [87.0%, 92.8%]	87.0% [84.3%, 89.5%]	64.0% [59.2%, 68.5%]	88.2% [85.0%, 91.5%]	7.5% [5.0%, 10.0%]
Base 4-bit, 3-shot	unseen (n=400)	91.2% [88.5%, 94.0%]	90.5% [87.5%, 93.3%]	83.7% [80.8%, 86.6%]	63.7% [58.5%, 68.8%]	87.2% [83.8%, 90.5%]	8.2% [5.7%, 11.0%]
Base 4-bit, 3-shot	irrelevance (n=200)	99.0% [97.5%, 100.0%]	–	–	–	–	irrelevance acc. 94.5% [91.0%, 97.5%]
Base bf16, zero-shot	seen (n=400)	98.8% [97.5%, 99.8%]	97.2% [95.5%, 98.8%]	88.9% [86.3%, 91.3%]	75.0% [70.5%, 79.0%]	97.2% [95.5%, 98.8%]	0.0% [0.0%, 0.0%]
Base bf16, zero-shot	unseen (n=400)	98.8% [97.5%, 99.8%]	98.5% [97.2%, 99.5%]	86.8% [84.0%, 89.6%]	74.8% [70.5%, 79.0%]	95.0% [92.8%, 97.0%]	1.0% [0.2%, 2.0%]
Base bf16, zero-shot	irrelevance (n=200)	94.0% [90.5%, 97.0%]	–	–	–	–	irrelevance acc. 56.5% [49.5%, 63.0%]
Run A: QLoRA 4-bit, 12k	seen (n=400)	99.0% [97.8%, 99.8%]	97.5% [96.0%, 99.0%]	89.6% [87.2%, 91.8%]	75.2% [71.2%, 79.2%]	98.2% [96.8%, 99.5%]	0.2% [0.0%, 0.8%]
Run A: QLoRA 4-bit, 12k	unseen (n=400)	99.2% [98.2%, 100.0%]	99.0% [98.0%, 99.8%]	88.1% [85.4%, 90.8%]	77.5% [73.8%, 81.8%]	98.2% [96.8%, 99.5%]	0.5% [0.0%, 1.2%]
Run A: QLoRA 4-bit, 12k	irrelevance (n=200)	100.0% [100.0%, 100.0%]	–	–	–	–	irrelevance acc. 100.0% [100.0%, 100.0%]
Run B: LoRA bf16, 12k	seen (n=400)	99.5% [98.8%, 100.0%]	98.0% [96.5%, 99.2%]	90.3% [87.7%, 92.6%]	78.2% [74.2%, 82.0%]	98.5% [97.2%, 99.5%]	0.0% [0.0%, 0.0%]
Run B: LoRA bf16, 12k	unseen (n=400)	99.5% [98.8%, 100.0%]	99.5% [98.8%, 100.0%]	88.2% [85.6%, 90.7%]	76.5% [72.2%, 80.8%]	98.0% [96.5%, 99.2%]	0.2% [0.0%, 0.8%]
Run B: LoRA bf16, 12k	irrelevance (n=200)	100.0% [100.0%, 100.0%]	–	–	–	–	irrelevance acc. 99.0% [97.5%, 100.0%]
Run C: QLoRA 4-bit, 2k	seen (n=400)	98.2% [97.0%, 99.5%]	97.5% [96.0%, 99.0%]	89.4% [87.1%, 91.6%]	75.5% [71.0%, 79.8%]	97.8% [96.2%, 99.0%]	0.8% [0.0%, 1.8%]
Run C: QLoRA 4-bit, 2k	unseen (n=400)	99.0% [98.0%, 99.8%]	98.8% [97.5%, 99.8%]	87.8% [85.4%, 90.4%]	77.2% [73.2%, 81.2%]	97.2% [95.5%, 98.8%]	1.0% [0.2%, 2.0%]
Run C: QLoRA 4-bit, 2k	irrelevance (n=200)	100.0% [100.0%, 100.0%]	–	–	–	–	irrelevance acc. 99.0% [97.5%, 100.0%]

* Argument accuracy is computed only over answers that got the function names right.

Numbers come straight from final_summary.json and metrics_summary.csv in the GitHub repo.

Caveats (please read)

Versus the 4-bit base, the gain is clear. QLoRA (Run A) moves exact match from 67.0% to 75.2% on seen tools (+8.2 pts) and from 67.2% to 77.5% on unseen tools (+10.3 pts). The unseen-tool intervals do not overlap; the seen-tool intervals overlap by 0.25 pts.
Versus the bf16 base, Run B's tool-calling gain is within the confidence intervals. Seen: 75.0% → 78.2% (+3.2 pts); unseen: 74.8% → 76.5% (+1.7 pts). The intervals overlap, so this part may be luck of the test sample. What clearly improves for bf16 is knowing when not to call a tool (irrelevance 56.5% [49.5%, 63.0%] → 99.0% [97.5%, 100.0%]).
General knowledge dropped. On 100 random MMLU questions, Run B's accuracy fell from 55% to 47% (exact McNemar p ≈ 0.10, 13 questions lost vs 5 gained). With only 100 questions this is not statistically certain, but it is a warning sign.
Most remaining errors are argument values. Of Run B's 183 failures across the three test sets, 164 (90%) have the right function(s) and number of calls but a wrong argument value or type. The notebook's error analysis (§12) counts 163 of 183 (89%) as plain "wrong argument value". Often the dataset expects an unstated naming convention (e.g. "avon-and-somerset" where the model wrote "Avon and Somerset Constabulary").
Short training (0.28 epoch over a 12,000-example pool), synthetic English-only data, a single seed, and a strict exact-match grader. The refusal sentence learned from the irrelevance examples can occasionally leak into odd places. No safety evaluation was done. Do not use this to call real tools without validating its arguments.

How to use (mlx-lm)

mlx_lm.load needs a local adapter folder, so download it first.

import json
from huggingface_hub import snapshot_download
from mlx_lm import load, generate

# Run B (root of this repo) on the bf16 base model
adapter_dir = snapshot_download("Neerajprao/qwen2.5-1.5b-function-calling-lora", allow_patterns=["adapter_config.json", "adapters.safetensors"])
model, tokenizer = load("Qwen/Qwen2.5-1.5B-Instruct", adapter_path=adapter_dir)

# Run A instead (QLoRA on the 4-bit base):
# root = snapshot_download("Neerajprao/qwen2.5-1.5b-function-calling-lora", allow_patterns=["qlora-4bit/*"])
# model, tokenizer = load("mlx-community/Qwen2.5-1.5B-Instruct-4bit", adapter_path=root + "/qlora-4bit")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
messages = [{"role": "user", "content": "What's the weather like in Paris right now?"}]

prompt = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=False)
out = generate(model, tokenizer, prompt=prompt, max_tokens=275)   # greedy by default
print(out)
# <tool_call>
# {"name": "get_weather", "arguments": {"city": "Paris"}}
# </tool_call>

The model answers with one <tool_call>{...}</tool_call> block per call, or with a plain-text refusal when no tool fits.

Ollama

The adapters are not Ollama-ready as they are. Fuse one into a full model first (mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct --adapter-path <adapter_dir> --save-path fused_model), then import that folder. An imported folder gets no tool-calling template in Ollama, so tools are ignored if you use /api/chat. The workaround used in the notebook (section 15) is to render Qwen's exact prompt yourself (tokenizer.apply_chat_template(..., tools=tools, tokenize=False, add_generation_prompt=True)) and send it to /api/generate with "raw": true, then parse the <tool_call> blocks from the reply.

License and attribution

These adapters: Apache-2.0. The base models (Qwen/Qwen2.5-1.5B-Instruct, mlx-community/Qwen2.5-1.5B-Instruct-4bit) are Apache-2.0, and the training data is CC BY 4.0, which allows reuse and derivatives with attribution and has no share-alike condition, so Apache-2.0 is compatible with both.
Training data: APIGen / xLAM function-calling 60k by Salesforce, CC BY 4.0. No dataset files are included in this repo. Please cite APIGen:

@article{liu2024apigen,
  title   = {APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author  = {Liu, Zuxin and Hoang, Thai and Zhang, Jianguo and others},
  journal = {arXiv preprint arXiv:2406.18518},
  year    = {2024}
}

Code, notebook, logs and all raw model outputs: https://github.com/neerajprao/Fine-tuning-Qwen1.5B.