Spidey

Author: Abhinay Ambati (@abhinayambati) · Version: 1.0 (2026-10-22)

Spidey is a small on-device intent model for an Android assistant. It reads one English sentence and returns tool calls for five phone actions. It is fine-tuned from Cactus Needle 3 (Cactus Compute, Apache-2.0) with a rank-16 LoRA, merged and exported as a 4-bit .cact at the full 20 layers.

Status: experimental. English only, five tools, no screen reading. Not measured on a phone yet.

Tools

Tool	Arguments
open_app	app (string)
send_message	contact (string), text (string)
call_contact	contact (string)
set_alarm	time (string, as spoken, e.g. "6 am")
set_timer	minutes (integer)

The exact schema is in tools.json. An app must pass these definitions unchanged: names, argument names and descriptions are what the model was trained on. Requests that no tool covers return an empty list.

Files

File	What
spidey.cact	merged model, 20 layers, 4-bit weights / 8-bit activations, 63.44 MB
adapter/adapter.safetensors	the LoRA adapter (rank 16, scale 2.0) on needle3.safetensors
tools.json	the five tool definitions
assets/spidey-icon.png	the Spidey app icon (used in this card)
eval/	held-out test set test_v4.jsonl, hand-written dev_natural.jsonl, per-row eval logs
training/	train_v4.jsonl (2,900 synthetic examples) and the scripts that built the data and ran the evals
SHA256SUMS	checksums. spidey.cact = 5d94343f3cdf9e9009926d9fe6ba688d7c1eaa9857720243b10316dc1ecc54ff

The Cactus engine (libneedle.a, needle.h) is not included. Fetch it from the Needle repository / Cactus-Compute/needle3 (android-arm64/).

Results (exact match of the full list of calls)

Test set	Rows	Correct
eval/test_v4.jsonl (held out: new names, apps, phrasings)	345	81.4% (281). Wrong actions 55, safe refusals 9
test_v4 with name/app checks against known lists	345	84.6% (292). Wrong actions 31
eval/dev_natural.jsonl (hand-written)	152	98.7% (150). Wrong actions 0, refusals 2

On test_v4: no-tool requests 94/105, single-step commands 168/208, multi-command sentences 19/32. For reference the untuned base scores 56.8% on an earlier test set and v3 (earlier fine-tune) scores 75.1% on test_v4.

Read these honestly:

• dev_natural.jsonl was written by the same author as the training templates, and 44 of its 97 names/apps also appear in training. Treat 98.7% as a best case, not an independent number.
• test_v4 is the harsh, genuinely unseen test, and the model missed the author's own pre-set bars on it (single-step positives 168/208 against 187; wrong actions 55 against 27).
• Weak spots: several commands in one sentence (59%), lowercase or noisy names (69%), idioms such as "open the curtains", and mixing up "call" and "text".
• Common errors are misspelled names/apps ("zoya" -> "ziaya"), an extra spurious call, and sometimes calling a tool for a sentence that only mentions one.
• Depth matters: the same adapter cut to 16 / 12 / 8 layers scores 87.5 / 75.0 / 51.3% on dev_natural.
• The local build drops the calibrated confidence head, so confidence is None.
• Speed was ~0.2 s per query on a GPU server. Phone speed has not been measured.

Use it safely

Validate every model output in the app before acting:

• match contact and app against the user's real contacts and installed apps, and reject values that are not in the user's request;
• ask for confirmation before every message or call. A wrong text or call is the real risk; a wrong alarm or timer is cheap to undo.

Training

• Base: Cactus-Compute/needle3 (20 layers, ~121M parameters).
• Method: needle finetune (LoRA rank 16, alpha 32, attention projections, seed 0), 8 epochs on training/data/train_v4.jsonl, then needle build --lora ....
• Data: 2,900 synthetic English examples (840 with no tool call), 63 negative families, lowercase/typo variants, 14 chain connectors. Names and phrasings in test_v4 are disjoint from training.
• Hardware: one NVIDIA A10G.
• Reproduce: see training/scripts/ (build_v4.py regenerates the data with seed 44; scripts expect to run from a folder with data/tools.json).

Limitations

English only. Five tools only. Durations in words ("half an hour") and spoken-number times ("seven thirty") were not trained. Message-body edges and unusual lead-ins are error sources. Not tested with real users' phrasings.

License and credits

Apache-2.0 (LICENSE, NOTICE). Modified from Cactus Needle 3 by Cactus Compute, Inc.: this model is a LoRA fine-tune of it for five tools in English. Please also cite the base model:

@misc{needle3_2026,
  title        = {Needle: Foundation Tool-Calling Model for Tiny Devices},
  author       = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and
                  Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
  year         = {2026},
  organization = {Cactus Compute, Inc.},
  howpublished = {\url{https://github.com/cactus-compute/needle}}
}

To cite Spidey:

@misc{spidey2026,
  title        = {Spidey: an on-device tool-calling model for Android phone commands},
  author       = {Ambati, Abhinay},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/abhinayambati/spidey}}
}