corbanyax/andy-select-2b-GGUF

Andy-Select 2B (GGUF, v1)

A 2B on-device action selector for Android agents: given the goal, the screen's accessibility tree and a numbered list of allowed actions, it picks the next action and says how sure it is, so the agent can act or hand the step to you.

A newer version, Andy-Select v2, is more accurate (0.655) and better calibrated. Use this v1 model only with prompt 1.

Model (AndroidControl test, 2,003 steps)	Step accuracy	File size	Tested on a phone
Andy-Select 2B Q4_K_M (this repo)	0.635	1.27 GB	yes, CPU only
Andy-Select 2B BF16 (this repo)	0.664	3.78 GB	no
Qwen3.5 4B Q4_K_M	0.501	2.74 GB	no
Gemma 4 E2B Q4_0	0.444	2.84 GB	no
Qwen3.5 2B Q4_K_M (the base model, not fine-tuned)	0.335	1.28 GB	yes, CPU only

• Calibrated confidence: expected calibration error 0.042 after temperature scaling. At the shipped threshold it hands 22.9% of steps to the user and is 0.744 accurate on the rest.
• Never malformed: scoring picks from the allowed list, and free generation produced 0 malformed replies.
• On a phone: on a OnePlus Nord CE5 (CPU only) the Q4_K_M file loads in about 6 s and peaks at 2.75 GB RSS. A decision takes about 8.7 s when the prompt's saved state is restored, or about 27 s from scratch. Its top-1 choice matched a desktop run on 0.973 of 300 steps.

Model family

Repo	What	Use with
andy-select-v2-2b-GGUF (recommended)	v2, Q4_K_M + BF16 GGUF	prompt 2
andy-select-2b-GGUF	v1, Q4_K_M + BF16 GGUF	prompt 1
andy-select-2b-lora	LoRA adapters for v1 and v2	merge into Qwen3.5-2B

Files here: andy-select-2b-Q4_K_M.gguf (1,274,396,384 B, sha256 f50dea1e…) and andy-select-2b-BF16.gguf (3,775,708,576 B, sha256 41b0bfee…), both evaluated and calibrated, plus andy_select.py, a small helper that renders the exact prompt and scores the candidates.

Quick start

pip install llama-cpp-python huggingface_hub numpy

import importlib.util
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

REPO = "corbanyax/andy-select-2b-GGUF"
gguf = hf_hub_download(REPO, "andy-select-2b-Q4_K_M.gguf")
spec = importlib.util.spec_from_file_location("andy_select", hf_hub_download(REPO, "andy_select.py"))
A = importlib.util.module_from_spec(spec); spec.loader.exec_module(A)
cal = A.CALIBRATION["andy-select-2b-Q4_K_M.gguf"]        # version 1, T = 0.907, tau80 = 0.4195

llm = Llama(model_path=gguf, n_ctx=4096, logits_all=True, verbose=False)

elements = [  # the accessibility tree: id, class, label, flags, depth, bounds in pixels
    {"id": 1, "cls": "FrameLayout", "res": "content", "depth": 0, "bounds": [0, 0, 1080, 2400]},
    {"id": 2, "cls": "View", "desc": "Our Latest Recipes", "depth": 1, "bounds": [42, 1780, 894, 1864]},
    {"id": 3, "cls": "Button", "desc": "See all", "flags": ["click"], "depth": 1, "bounds": [894, 1775, 1038, 1864]},
    {"id": 4, "cls": "ImageView", "desc": "Today, Tab 1 of 3", "flags": ["click", "selected"], "depth": 1, "bounds": [0, 2190, 360, 2337]},
    {"id": 5, "cls": "ImageView", "desc": "Search, Tab 2 of 3", "flags": ["click"], "depth": 1, "bounds": [360, 2190, 720, 2337]},
    {"id": 6, "cls": "ImageView", "desc": "Profile , Tab 3 of 3", "flags": ["click"], "depth": 1, "bounds": [720, 2190, 1080, 2337]},
]
candidates = ([{"op": "tap", "id": i} for i in (3, 4, 5, 6)] + [{"op": "scroll", "dir": d} for d in ("up", "down")]
              + [{"op": o} for o in ("back", "home", "wait", "done", "handoff")])

prompt = A.render_prompt(goal="Watch the video on how to make molten chocolate cake in the Kitchen Stories app.",
                         app="com.ajnsnewmedia.kitchenstories", history=["opened app Kitchen Stories"],
                         elements=elements, candidates=candidates, version=cal["version"])

action, conf, probs = A.decide(llm, prompt, candidates, cal["T"], cal["tau80"])
print(action, f"{conf:.3f}")   # {'op': 'tap', 'id': 5} 0.513 -> at least tau80, so act (None = hand off)

How it decides: every candidate is scored by log-likelihood, softmax with temperature T gives probabilities, and the agent acts only if the top one is at least tau80.

Free generation with a grammar also works (no confidence; check that the reply is in the allowed list):

from llama_cpp import LlamaGrammar
out = llm.create_completion(prompt, max_tokens=48, temperature=0.0, stop=["<|im_end|>"],
                            grammar=LlamaGrammar.from_string(A.ACTION_GRAMMAR))
print(out["choices"][0]["text"])   # {"op":"tap","id":5}

Notes: use prompt 1 (version=1). The BF16 file has its own calibration (T = 0.880, tau80 = 0.433). T and tau80 are fitted for this exact file and prompt; refit them if you change either. llama-cpp-python's scores can differ slightly from llama.cpp's, so confidences right at tau80 may flip.

Results

These are single-step results on the AndroidControl test split. The 2,003 steps were drawn as whole episodes and frozen before any model was run. Each step uses the high-level goal, the accessibility tree as text and the gold action history. Accuracy is exact match with the official answer.

Intended use and limitations

Intended: research on small, calibrated on-device agents, and the action selector inside the Andy Android app. The caller builds the candidate list from the screen and the model picks one.

Not for: a general chatbot, multi-step planning, typing text (the type action only picks the field) or unattended operation. This is a research preview. In the Andy app the agent never sends messages, deletes content, buys anything, enters passwords or grants permissions, and uncertain steps go to the user. The app enforces these rules, not the model. Any other integration needs its own safeguards.

Limitations:

• The model reads text only (no pixels), so icon-only or unlabelled controls are hard for it.
• Single-step accuracy is not task success, because errors compound over a task.
• It was trained and tested on AndroidControl only and is English-centric.
• 4-bit quantization costs this fine-tuned model about 3 points (BF16 0.664 vs Q4_K_M 0.635).

Training (short)

LoRA fine-tune of Qwen3.5-2B (r 16, one epoch) on 67,907 AndroidControl training steps rendered with prompt 1, distilled from a proprietary teacher's soft labels mixed with the official answers. Merged, exported to GGUF and quantized to Q4_K_M with an importance matrix.

License and data

• Weights: Apache-2.0. The base model Qwen/Qwen3.5-2B is Apache-2.0.
• Training data: AndroidControl (Google). Check the AndroidControl dataset terms before redistribution: its code is Apache-2.0, but we could not confirm a license for the data.
• Teacher labels: part of the training targets are soft labels from a proprietary model.

Citation

Paper in preparation. Code: repository link to be added.

@misc{deo2026andyselect,
  title  = {Know When to Hand Off: Calibrated Closed-Set Action Selection for 4 GB On-Device Mobile Agents},
  author = {Deo, Atharva},
  year   = {2026},
  note   = {Paper in preparation}
}