CuriousDragon/northline-functiongemma-270m-sft

Northline FunctionGemma-270M SFT

LoRA adapter that turns google/functiongemma-270m-it into the Northline Express customer-support assistant: it calls the site's 6 tools and writes grounded conversational replies.

• Base model: google/functiongemma-270m-it @ 39eccb091651513a5dfb56892d3714c1b5b8276c
• Init: fresh LoRA on the base model (base)
• LoRA: r=16, alpha=32, dropout=0.05
• Recipe: 3 epochs, lr=0.0002, effective batch 16, max_len 1408
• Data: 1328 records (train_v1_enriched.jsonl, sha256 d3673057244f)
• Run: fg270m_base_v6_r16_lr2e-04_seed42 on NVIDIA GeForce RTX 5060 Laptop GPU, 866.9s

Evaluation

Task success = action + tool + exact args (or missing fields) + grounded text.

split	n	task	tool_sel	args_EM	miss_EM	invalid	fabric	coverage
dev	263	87.5%	88.6%	78.5%	52.0%	0.0%	0.0%	93.5%
gold (held-out)	303	88.1%	89.5%	82.1%	48.0%	0.0%	0.0%	95.7%
rule baseline (dev)	257	67.7%	39.2%	27.9%	11.5%	0.0%	0.0%	97.2%
rule baseline (gold)	299	69.6%	47.5%	32.3%	8.3%	0.0%	0.0%	97.0%

Zero invalid tool calls and zero fabricated responses on both splits.

Intended use

Local / on-prem demo assistant for the Northline Express site. Serve through any OpenAI-compatible server that renders the FunctionGemma chat template with the 6 tool declarations (see the site's src/assistant/tools.ts). Train/serve parity is structural: the same adapter renders training targets and inference prompts, so tool syntax cannot drift.

Limitations

• The demo world is tiny (4 shipments, 6 locations); identities are split train/dev/gold so held-out families measure generalization, not memorization.
• Long runs of repeated digits in a waybill can occasionally be miscounted (a tokenizer artifact of the base model).
• Invalid-format tracking input may still call the tool instead of asking for a corrected number.
• English + Hinglish only.

Provenance

Generated by the northline-agent pipeline; run_meta.json and the evaluation reports are included alongside the weights.