JivinDotL/pony-android-grounding

Pony-AndroidGrounding

A vision-language model fine-tuned for visual grounding on Android UI screenshots. Base model: Qwen3-VL-2B-Instruct. Trained with LoRA (r=16 on q/k/v/o_proj), then merged into full bf16 weights (merge_and_unload). Also carries a custom identity/behavior corpus (Identity + Behavior SFT + DPO).

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("JivinDotL/pony-android-grounding", dtype="bfloat16")
proc = AutoProcessor.from_pretrained("JivinDotL/pony-android-grounding")
# instruction + portrait phone screenshot -> <bbox>x1,y1,x2,y2</bbox> (normalized 0-1000), or 未找到 ("not found")

Protocol: for a target-finding instruction the model returns exactly one bbox; if the target is absent it must output 未找到 (refusal).

Metrics (frozen test sets: val 225 / test 115 = held-out app domain, frozen before training, reproducible; v10.0b, 2026-11)

split	iou50	iou75	click	correct refusal	valid JSON
val (225)	87.3	85.0	93.5	100	100
test (115)	80.4	72.5	87.5	100	100
dense enumeration holdout (52, multi-target)	100.0	98.1	—	—	100

Latency: ~0.5 s per request (bf16, single datacenter GPU). The test split contains a fully held-out app domain, so the numbers reflect cross-interface generalization. Split manifests (splits/v9.json, splits/dense_exam.json) ship with this repo. Tool-call protocol (<tool_call> JSON; locate_ui_element / tap) scores 6/6 under a strict parser; the model only outputs suggested coordinates and never executes actions.

Known issues (honest limits, measured 2026-11-04, v10.0b)

1. Persona probes 5/6. Direct and paraphrased "who are you / what can you do / when were you released" are answered truthfully (including "I cannot tap for you, I only output coordinates"). The single unstable probe Q6 (relationship to "Pony.ai") is confabulation-prone: depending on the numerical form of the same model (adapter vs merged weights were both measured), it either honestly denies any relationship or fabricates a fake "co-founded by" story. This axis is not fully fixed — treat its answers as unreliable chit-chat, not facts. No other probe shows this behavior.
2. Meta info such as the release date (2027-01-01) is a fictional persona value, not a real event.
3. Training domain is portrait 720x1600 phones; landscape/web UIs are out of distribution. Dense enumeration ("list all clickable elements") scores perfect within trained app families only.
4. Weak on small icons (<40 px) and near-identical icon groups.
5. The personal/lab names the model uses about itself are self-created fictional personas, unrelated to any real person or organization (author attribution in public texts follows the usual abbreviation; model behavior keeps the original names).

Data & Training

• Training screenshots were captured on real devices and are not redistributed with the model (third-party app UI copyright). Annotations are released separately as JSON (L1 scope, annotations only) — see DATASET_CARD.md.
• Training: two-stage SFT (LoRA) + DPO on a single GPU; recipes and protocols are documented in the project repository.

Citation / License

Model weights Apache-2.0 (aligned with the base model Qwen3-VL-2B-Instruct).