ThoxNeedle-Micro

On-device function calling for THOX hardware, based on Cactus's public needle2, packaged in the .cact format the device runtime loads.

The weights shipped here (needle2.cact) are the upstream needle2 artifact, unmodified. This model is NOT THOX-trained.

A THOX fine-tune was trained and is published alongside, under training/artifacts/, together with its full evidence. It is not promoted to the shipped slot because the release gate did not pass: held-out exact-match did not improve over base. Everything below is reported as measured.

What this model is for

Local, on-device function calling for THOX hardware: given a set of THOX device tool schemas and a user request, emit the tool call that satisfies it — or emit an empty call list when no available tool can serve the request.

That second behaviour is a first-class objective, not an afterthought. On a device, a model that invents a plausible-looking call for an unservable request fires real actuators.

Training

base	Cactus-Compute/needle2 → weights/needle2.pkl (45,211,383 params, stored float16)
method	LoRA rank 16, alpha 32, on q_proj, k_proj, v_proj, gate_proj, out_proj across all 27 layers
corpus	216 rows of THOX device tool-calling (needle-tooluse.jsonl, sha256 668c0ddba1287629) — but only 46 distinct (query, answer) pairs, 78.7% duplicates
split	168 train / 48 held-out rows, group-aware and stratified by class, seed 3407 — every copy of an example stays on one side, so held-out leakage is zero
optimiser	AdamW, warmup-cosine, global-norm clip 1.0, float32 params and optimiser state
learning rate	0.0003 (selected by sweep)
epochs	3, batch 8
non-finite steps	0

Learning-rate sweep

learning rate	best held-out loss
3e-05	2.8545
0.0001	2.8022
0.0003	2.4067 ← selected

Loss curve

Held-out loss by epoch: 2.5688 → 2.1139 → 1.9188

step	train loss
1	2.9295
5	2.9573
9	2.7323
13	3.1168
17	3.0506
21	2.7201
25	2.5620
29	2.1005
33	2.1446
37	2.0924
41	2.4095
45	1.9863
49	2.2326
53	1.8365
57	2.4182
61	2.1349
63	2.7020

Evaluation

Scored on the 12 distinct examples held out from training (drawn from 48 rows — duplicates are scored once, not weighted by how often they repeat). Greedy decode, JAX float32. call_exact_match and refusal_accuracy are reported separately because averaging them would hide the failure mode that matters on a device.

metric	base (passthrough)	THOX fine-tune	delta
well-formed output	100.0% (12/12)	100.0% (12/12)	unchanged
correct tool selected	83.33% (10/12)	75.0% (9/12)	-8.33 (-1 ex.)
exact match (name + args)	58.33% (7/12)	58.33% (7/12)	unchanged
exact match, call examples only	50.0% (5/10)	50.0% (5/10)	unchanged
refusal accuracy (answers: [])	100.0% (2/2)	100.0% (2/2)	unchanged

Read these deltas with the sample size in mind. One example is 8.33 percentage points, so the only honest summary is that the fine-tune left exact-match unchanged — it neither improved nor meaningfully harmed the held-out score. The single-example movement in tool selection is not a trend this set can resolve.

Known limitation of this corpus

needle-tooluse.jsonl contains 216 rows but only 46 distinct (query, answer) pairs — 78.7% duplicates, with up to 12 copies of a single example.

Two consequences, both material to reading the numbers above:

1. A row-level train/test split of this file leaks. An earlier run of this pipeline split at row level and put a duplicate of 31 of its 32 held-out rows into training; those before/after numbers measured memorisation and have been discarded. The split is now group-aware and asserts zero leakage.
2. After holding out distinct examples for evaluation, only a few dozen distinct examples remain to train on. That is a hard ceiling on what any fine-tune of this corpus can demonstrate against a base model that already performs well on the task.

Deduplicating and expanding the corpus is the prerequisite for a meaningful improvement here — not more epochs, and not a different learning rate.

Did the fine-tune survive quantization?

The deployment scheme is mixed-precision at ~2.2 effective bits (embedding=4, mhc=4, default=2, group size 128). At that width a small LoRA delta can be quantized away entirely, which would be indistinguishable from training having done nothing. Both artifacts were dequantized and diffed:

• tensors changed: 135/404
• weight elements changed: 13,908,650 of 43,634,423 (31.88%)
• max relative L2 change in a tensor: 0.1079
• survived quantization: True

The exporter is bit-deterministic — two exports of identical parameters produce identical bytes — so the noise floor for this comparison is exactly zero and any non-zero delta is real.

Reproducing

Trainer, config, corpus, split, and eval are in Thox-ai/thoxneedle-trainer-src:

python -m thoxneedle.cli all --config configs/thoxneedle-micro.yaml

Lineage and licensing

Derived from Cactus-Compute/needle2, which declares Apache-2.0 for its weights; this repo carries that license forward.

Separately, and affecting the trainer rather than this artifact: the cactus-needle 2.0.0 PyPI wheel is internally inconsistent about its own license — METADATA declares Apache-2.0 while the bundled LICENSE file is MIT. THOX vendored parts of that wheel's code into its trainer and attributes against the MIT text there, since that is the document actually shipped with the code. That discrepancy does not govern the weights published here.

THOX did not use upstream's training code. Its finetune.py NaNs on the first optimiser step: the checkpoint is float16, init_lora inherits that dtype, zero-initialised B makes the step-1 gradient w.r.t. A exactly zero, and optax.adamw's default eps=1e-8 is below float16's smallest subnormal (5.96e-8) and rounds to zero — so Adam computes 0/(sqrt(0)+0) = NaN. THOX wrote its own trainer that keeps parameters and optimiser state in float32. Details: docs/UPSTREAM_NAN.md in the trainer repo.