Nayif7/opentune-phi3-mini-xlam-functioncalling

OpenTune — Phi-3-mini QLoRA adapter for function calling

A QLoRA adapter that teaches microsoft/Phi-3-mini-4k-instruct (3.8B) to emit valid, structured function calls instead of prose.

Results

Base model vs. adapter on a seeded 120-example held-out split, greedy decoding (temperature=0) so the numbers reproduce exactly.

Metric	Base	Fine-tuned	Change
Exact match (tool + all arguments)	0.0%	72.5%	+72.5 pts
Tool-selection accuracy	15.8%	98.3%	+82.5 pts
JSON validity	40.8%	98.3%	+57.5 pts
ROUGE-L (secondary)	0.230	0.954	+0.724

The base model produced zero exactly-correct tool calls across all 120 examples. The adapter — roughly 100 MB of LoRA weights over a frozen 4-bit base — produces a fully correct call, arguments included, on 72.5% of them.

Training and evaluation code: OpenTune on GitHub

What this is for

Small instruction-tuned models are poor at tool calling by default: asked to produce a tool call, they narrate, wrap JSON in markdown fences, hallucinate tool names that were never offered, or emit output that does not parse at all. This adapter addresses that specific failure mode.

Intended use — converting a natural-language request plus a list of available tool schemas into a structured JSON call, in agent loops, tool routers, and structured-extraction pipelines where a small local model is preferred over an API.

Out of scope — general chat, reasoning, or instruction following. The adapter is tuned narrowly for structured call generation and will be worse than the base model at open-ended tasks. It is not safety-tuned, and it performs no validation of the calls it produces: a syntactically valid call can still be semantically wrong, so validate arguments before execution.

Limitations

Tool-schema overlap between splits is high (~98.3%). Both splits are drawn from the same 1,200-example subset, so nearly all evaluation tools also appear in training. The defensible claim is therefore reliable calling of a known toolset on novel phrasing, not generalization to previously unseen tools. Treat the numbers accordingly.

• No frontier-model zero-shot baseline. These numbers establish what fine-tuning bought over this base model. They do not establish how a 3.8B fine-tune compares to a large model prompted zero-shot.
• Small training budget. One epoch over 1,080 examples — about 2% of the available 60k.
• 120 evaluation examples. Enough to make a 72.5-point gap unambiguous, but wide confidence intervals on any small difference.
• Adapter is not merged. Inference requires the 4-bit base plus the adapter. No merged-weight or GGUF export is provided.
• English only, inheriting the base model's and dataset's coverage.

Usage

The adapter was trained against a 4-bit NF4 quantized base. Load the base the same way at inference — attaching the adapter to a differently-quantized base is a genuine train/inference skew and will surface as key-path errors and degraded output.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE = "microsoft/Phi-3-mini-4k-instruct"
ADAPTER = "Nayif7/opentune-phi3-mini-xlam-functioncalling"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

prompt = (
    'Available tools:\n'
    '[{"name": "get_weather", "parameters": {"city": {"type": "string"}}}]\n\n'
    'User request: What is the weather in Hyderabad?'
)

messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

# apply_chat_template returns a BatchEncoding (transformers >= 4.46), not a
# plain tensor, so input_ids is read via dict-key access here rather than a
# .shape attribute on the return value itself.
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Prompt format. The adapter expects available tools and the user request in a single message, in the shape it was trained on:

Available tools:
{tool schemas as a JSON list}

User request: {natural-language query}

Requirements. A CUDA GPU. pip install transformers peft bitsandbytes accelerate torch

Reproducing the evaluation

colab_reevaluate_existing_adapter.py in the GitHub repository reconstructs the same 120 held-out examples from the seed, loads this adapter without retraining, and prints the full comparison table. Because both the subset selection and the train/eval split are seeded, the evaluation set is reproducible exactly without being persisted anywhere.

How the metrics were chosen

ROUGE-L is reported but is explicitly secondary. Token overlap is close to noise for structured output: a response can share most of its tokens with the reference while calling the wrong tool, and a correct call can score poorly on cosmetic formatting differences.

The evaluation is decomposed into three questions that actually matter — does it parse, did it pick the right tools, and is the whole call right. That decomposition is what makes the result interpretable: 98.3% tool-selection accuracy against 72.5% exact match localizes the remaining errors to argument construction rather than tool choice, which a single aggregate score would have hidden.

Implementation details behind the numbers:

• Exact match compares parsed and canonicalized JSON, not raw text. Key order within a call and ordering across independent calls are normalized, so equivalent calls are not counted as mismatches.
• Tool-selection and exact-match return None, not False, when either side fails to parse — keeping "called the wrong tool" distinct from "emitted nothing parseable." None still counts against the reported rate, with JSON validity published alongside so the failure can be decomposed.
• Both models are decoded greedily at temperature=0, so re-running the harness reproduces the table exactly.

Training

Parameter	Value
Base model	microsoft/Phi-3-mini-4k-instruct (3.8B, 4k context)
Method	QLoRA via PEFT LoraConfig
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	qkv_proj, o_proj, gate_up_proj, down_proj
Quantization	4-bit NF4, double quantization, float16 compute
Epochs	1
Per-device batch size	2
Learning rate	2e-4
Max sequence length	1024
Gradient checkpointing	Enabled
Seed	42
Hardware	Single free-tier Google Colab GPU

Target modules are architecture-specific rather than the conventional Llama-style list: Phi-3 fuses its projections into qkv_proj and gate_up_proj, so ["q_proj", "k_proj", "v_proj"] fails outright on this model.

Data

Salesforce/xlam-function-calling-60k — function-calling examples generated and verified through Salesforce's APIGen pipeline. Each row provides a natural-language query, the tools available for that query, and reference answers as structured calls. The dataset is gated; access must be requested on Hugging Face before use.

Training used a deterministic 1,200-example subset (shuffle(seed=42).select(range(1200))), split into 1,080 training and 120 held-out examples, reformatted into:

instruction: "Available tools:\n{tool schemas}\n\nUser request: {query}"
output:      "{reference function calls as JSON}"

Development notes

The pipeline is built as typed, testable modules — Pydantic v2 run configuration, a per-architecture LoRA target registry, and a pytest suite covering everything that does not require a GPU.

Several correctness issues were found and fixed during development, including a prompt-echo defect that was scoring the templated prompt as model output and invalidating every metric, and a train/inference quantization mismatch that broke adapter loading. All numbers above come from after those fixes. The repository's README documents each in detail.

Framework versions

• PEFT 0.19.1 (confirmed from adapter_config.json — the version this adapter was serialized with)
• Transformers >= 4.46 required (the usage snippet above depends on apply_chat_template(..., return_dict=True), which is only meaningful from this version onward; on Transformers 5.x, return_dict=True is the only supported path since the method no longer returns a plain tensor)
• bitsandbytes 0.43+
• PyTorch 2.4+

Verified working end-to-end in a fresh Google Colab T4 runtime, 13 Sep 2026.

Citation

@misc{opentune2026,
  author = {Nayifuddin Muhammed},
  title  = {OpenTune: QLoRA Fine-Tuning Pipeline for LLM Function Calling},
  year   = {2026},
  url    = {https://github.com/mohdnayif799/OpenTune-QLoRA-Fine-Tuning-Pipeline-for-LLM-Function-Calling}
}

Author

Nayifuddin Muhammed — B.E. Computer Science and Engineering (AI & ML), Neil Gogte Institute of Technology, Hyderabad.

Part of an applied ML portfolio focused on making small open-weight models reliable at structured, production-shaped tasks.