corbanyax/andy-select-v2-2b-GGUF

Andy-Select v2 2B (GGUF)

A 2B on-device action selector for Android agents: given the goal, the screen's accessibility tree and a numbered list of allowed actions, it picks the next action and says how sure it is, so the agent can act or hand the step to you.

Model (AndroidControl test, 2,003 steps)	Step accuracy	File size	Tested on a phone
Andy-Select v2 2B Q4_K_M (this repo)	0.655	1.27 GB	via v1 (same size), CPU only
Qwen3.5 4B Q4_K_M	0.501	2.74 GB	no
Gemma 4 E2B Q4_0	0.444	2.84 GB	no
Qwen3.5 2B Q4_K_M (the base model, not fine-tuned)	0.335	1.28 GB	yes, CPU only

• Calibrated confidence: expected calibration error 0.026 after temperature scaling. At the shipped threshold it hands 22.4% of steps to the user and is 0.750 accurate on the rest.
• Never malformed: scoring picks from the allowed list, and free generation produced 0 malformed replies.
• On a phone: on a OnePlus Nord CE5 (CPU only) a decision takes about 8.7 s when the prompt's saved state is restored, or about 27 s from scratch. We measured this with v1, which has the same architecture and file size.
• v2 vs v1: +0.020 accuracy on the same steps (McNemar p = 0.001), mostly on wait steps. v2's prompt adds a screen-change and loading line plus element positions.

Model family

Repo	What	Use with
andy-select-v2-2b-GGUF (recommended)	v2, Q4_K_M + BF16 GGUF	prompt 2
andy-select-2b-GGUF	v1, Q4_K_M + BF16 GGUF	prompt 1
andy-select-2b-lora	LoRA adapters for v1 and v2	merge into Qwen3.5-2B

Files here: andy-select-v2-2b-Q4_K_M.gguf (1,274,396,384 B, sha256 afb36dcb…, evaluated and calibrated), andy-select-v2-2b-BF16.gguf (3,775,708,576 B, sha256 6541218f…, not separately evaluated or calibrated), SHA256SUMS, and andy_select.py, a small helper that renders the exact prompt and scores the candidates.

Quick start

pip install llama-cpp-python huggingface_hub numpy

import importlib.util
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

REPO = "corbanyax/andy-select-v2-2b-GGUF"
gguf = hf_hub_download(REPO, "andy-select-v2-2b-Q4_K_M.gguf")
spec = importlib.util.spec_from_file_location("andy_select", hf_hub_download(REPO, "andy_select.py"))
A = importlib.util.module_from_spec(spec); spec.loader.exec_module(A)
cal = A.CALIBRATION["andy-select-v2-2b-Q4_K_M.gguf"]     # version 2, T = 0.907, tau80 = 0.4494

llm = Llama(model_path=gguf, n_ctx=4096, logits_all=True, verbose=False)

elements = [  # the accessibility tree: id, class, label, flags, depth, bounds in pixels
    {"id": 1, "cls": "FrameLayout", "res": "content", "depth": 0, "bounds": [0, 0, 1080, 2400]},
    {"id": 2, "cls": "View", "desc": "Our Latest Recipes", "depth": 1, "bounds": [42, 1780, 894, 1864]},
    {"id": 3, "cls": "Button", "desc": "See all", "flags": ["click"], "depth": 1, "bounds": [894, 1775, 1038, 1864]},
    {"id": 4, "cls": "ImageView", "desc": "Today, Tab 1 of 3", "flags": ["click", "selected"], "depth": 1, "bounds": [0, 2190, 360, 2337]},
    {"id": 5, "cls": "ImageView", "desc": "Search, Tab 2 of 3", "flags": ["click"], "depth": 1, "bounds": [360, 2190, 720, 2337]},
    {"id": 6, "cls": "ImageView", "desc": "Profile , Tab 3 of 3", "flags": ["click"], "depth": 1, "bounds": [720, 2190, 1080, 2337]},
]
candidates = ([{"op": "tap", "id": i} for i in (3, 4, 5, 6)] + [{"op": "scroll", "dir": d} for d in ("up", "down")]
              + [{"op": o} for o in ("back", "home", "wait", "done", "handoff")])

prompt = A.render_prompt(goal="Watch the video on how to make molten chocolate cake in the Kitchen Stories app.",
                         app="com.ajnsnewmedia.kitchenstories", history=["opened app Kitchen Stories"],
                         elements=elements, candidates=candidates, version=cal["version"],
                         prev_elements=None)   # pass the previous screen's elements when you have them

action, conf, probs = A.decide(llm, prompt, candidates, cal["T"], cal["tau80"])
print(action, f"{conf:.3f}")   # {'op': 'tap', 'id': 5} 0.505 -> at least tau80, so act (None = hand off)

How it decides: every candidate is scored by log-likelihood, softmax with temperature T gives probabilities, and the agent acts only if the top one is at least tau80.

Free generation with a grammar also works (no confidence; check that the reply is in the allowed list):

from llama_cpp import LlamaGrammar
out = llm.create_completion(prompt, max_tokens=48, temperature=0.0, stop=["<|im_end|>"],
                            grammar=LlamaGrammar.from_string(A.ACTION_GRAMMAR))
print(out["choices"][0]["text"])   # {"op":"tap","id":5}

Notes: use prompt 2 (version=2). T and tau80 are fitted for this exact file and prompt; refit them if you change either. llama-cpp-python's scores can differ slightly from llama.cpp's, so confidences right at tau80 may flip.

Results

These are single-step results on the AndroidControl test split. The 2,003 steps were drawn as whole episodes and frozen before any model was run. Each step uses the high-level goal, the accessibility tree as text and the gold action history. Accuracy is exact match with the official answer.

Intended use and limitations

Intended: research on small, calibrated on-device agents, and the action selector inside the Andy Android app. The caller builds the candidate list from the screen and the model picks one.

Not for: a general chatbot, multi-step planning, typing text (the type action only picks the field) or unattended operation. This is a research preview. In the Andy app the agent never sends messages, deletes content, buys anything, enters passwords or grants permissions, and uncertain steps go to the user. The app enforces these rules, not the model. Any other integration needs its own safeguards.

Limitations:

• The model reads text only (no pixels), so icon-only or unlabelled controls are hard for it.
• Single-step accuracy is not task success, because errors compound over a task.
• It was trained and tested on AndroidControl only and is English-centric.
• This is an early-stopped checkpoint (update 1,250 of 1,835 planned).

Training (short)

LoRA fine-tune of Qwen3.5-2B (r 16, one epoch) on 67,907 AndroidControl training steps rendered with prompt 2, distilled from a proprietary teacher's soft labels mixed with the official answers. Merged, exported to GGUF and quantized to Q4_K_M with an importance matrix.

License and data

• Weights: Apache-2.0. The base model Qwen/Qwen3.5-2B is Apache-2.0.
• Training data: AndroidControl (Google). Check the AndroidControl dataset terms before redistribution: its code is Apache-2.0, but we could not confirm a license for the data.
• Teacher labels: part of the training targets are soft labels from a proprietary model.

Citation

Paper in preparation. Code: repository link to be added.

@misc{deo2026andyselect,
  title  = {Know When to Hand Off: Calibrated Closed-Set Action Selection for 4 GB On-Device Mobile Agents},
  author = {Deo, Atharva},
  year   = {2026},
  note   = {Paper in preparation}
}