Skill: murongg/ui-element-ops (version 1.0.2)

--- name: ui-element-ops description: Parse UI screenshots into structured element JSON (type, OCR text, bbox) and operate desktop UI from parsed elements. Use when a user asks to detect/locate UI elements, return coordinates, find elements by text/type, wait for element appearance or disappearance, click/type/press keys/hotkeys, take screenshots, or calibrate coordinates for multi-display/DPI/window offsets. --- # UI Element Ops Parse one or more screenshots into a machine-readable JSON schema with: - `type` (normalized UI element type) - `bbox_px` and `bbox_norm` - `text` (OCR/caption content when available) - `clickable` flag - optional overlay image with labeled boxes - desktop actions via `scripts/operate_ui.py` (click/type/key/hotkey/screenshot) - element query and orchestration via `scripts/operate_ui.py` (`find`, `wait`) - coordinate calibration profile for multi-display/DPI/window offset (`calibrate`) ## Quick Start 1. Prepare runtime once per machine: ```bash skills/ui-element-ops/scripts/bootstrap_omniparser_env.sh "$PWD" ``` 2. Parse one screenshot: ```bash skills/ui-element-ops/scripts/run_parse_ui.sh /abs/path/to/1.jpeg ``` 3. Read outputs: - `<image>.elements.json` - `<image>.overlay.png` 4. One-step capture + parse with randomized names: ```bash skills/ui-element-ops/scripts/capture_and_parse.sh ``` ## Workflow 1. Confirm screenshot path and desired output path. 2. Run `scripts/bootstrap_omniparser_env.sh` when `.venv` or OmniParser weights are missing. 3. Run `scripts/run_parse_ui.sh` for standard parsing. 4. Report absolute output paths and summary counts: `total`, `clickable`, `by_type`. 5. Call out obvious quality risks for tiny text or dense icon layouts. 6. Execute desktop actions when requested: - list elements: `python3 skills/ui-element-ops/scripts/operate_ui.py list --elements <json>` - find elements: `python3 skills/ui-element-ops/scripts/operate_ui.py find --elements <json> --type button --text-contains login` - wait for appear/disappear: `python3 skills/ui-element-ops/scripts/operate_ui.py wait --elements <json> --state appear --text-contains continue` - click by id: `python3 skills/ui-element-ops/scripts/operate_ui.py click --elements <json> --id e_0001` - screenshot: `python3 skills/ui-element-ops/scripts/operate_ui.py screenshot` (defaults to user tmp dir) - calibrate coordinates: `python3 skills/ui-element-ops/scripts/operate_ui.py calibrate --parsed-size <w> <h> --actual-size <w> <h>` ## Tunables - Edit type mapping keywords in `references/type_rules.example.json`. - Use advanced parser args via `scripts/parse_ui.py --help`. - Use `--use-paddleocr` only when `paddleocr`/`paddlepaddle` are installed. ## Outputs - Main JSON output: - `schema_version`, `pipeline`, `image`, `counts`, `elements` - each element has `id`, `type`, `bbox_px`, `bbox_norm`, `text`, `clickable` - Overlay PNG output: - same screenshot with labeled detection boxes ## Failure Handling - Missing dependencies or weights: run bootstrap script again. - Permission/cache errors under `$HOME`: keep temporary caches under `/tmp` (handled by run script). - CPU-only machine: expect slower inference. - Performance note: parse/capture-and-parse commands are heavy; avoid very tight loops and reuse recent `elements.json` when possible. - Headless environment limitation: - usable without GUI: parse/list/find/wait/calibrate on existing files. - requires GUI session: click/click-xy/type/key/hotkey/screenshot/screen-info.

path: scripts/run_parse_ui.sh
content: #!/usr/bin/env bash
set -euo pipefail

# Usage:
#   run_parse_ui.sh <image> [output_json] [overlay_png]
#
# Env (optional):
#   VENV_PATH: Python venv path (default: $PWD/.venv)
#   OMNIPARSER_DIR: OmniParser repo path (default: /tmp/OmniParser)
#   TYPE_RULES: type rule json path

if [ $# -lt 1 ]; then
  echo "Usage: $0 <image> [output_json] [overlay_png]" >&2
  exit 1
fi

IMAGE="$1"
IMAGE_BASE="${IMAGE%.*}"
OUTPUT_JSON="${2:-${IMAGE_BASE}.elements.json}"
OVERLAY_PNG="${3:-${IMAGE_BASE}.overlay.png}"

VENV_PATH="${VENV_PATH:-$PWD/.venv}"
OMNIPARSER_DIR="${OMNIPARSER_DIR:-/tmp/OmniParser}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TYPE_RULES="${TYPE_RULES:-$SCRIPT_DIR/../references/type_rules.example.json}"

if [ ! -f "$IMAGE" ]; then
  echo "Input image not found: $IMAGE" >&2
  exit 1
fi

if [ ! -x "$VENV_PATH/bin/python" ]; then
  echo "Python runtime not found at $VENV_PATH/bin/python" >&2
  echo "Run bootstrap first: skills/ui-element-ops/scripts/bootstrap_omniparser_env.sh \"\$PWD\"" >&2
  exit 1
fi

mkdir -p /tmp/mpl /tmp/xdg /tmp/easyocr /tmp/hf /tmp/ultra

MPLCONFIGDIR=/tmp/mpl \
XDG_CACHE_HOME=/tmp/xdg \
EASYOCR_MODULE_PATH=/tmp/easyocr \
HF_HOME=/tmp/hf \
YOLO_CONFIG_DIR=/tmp/ultra \
"$VENV_PATH/bin/python" "$SCRIPT_DIR/parse_ui.py" \
  --image "$IMAGE" \
  --output "$OUTPUT_JSON" \
  --overlay "$OVERLAY_PNG" \
  --omniparser-dir "$OMNIPARSER_DIR" \
  --type-rules "$TYPE_RULES"


path: scripts/operate_ui.py
content: #!/usr/bin/env python3
"""Operate desktop UI using parsed element JSON from parse_ui.py.

Headless note:
  - Works without GUI: list, find, wait, calibrate (when using existing elements JSON).
  - Requires GUI session: click, click-xy, type, key, hotkey, screenshot, screen-info.

Examples:
  python3 operate_ui.py list --elements ./1.elements.json
  python3 operate_ui.py find --elements ./1.elements.json --type button --text-contains login
  python3 operate_ui.py find --elements ./1.elements.json --text-regex "sign\\s*in" --click
  python3 operate_ui.py wait --elements ./1.elements.json --state appear --type button --text-contains continue
  python3 operate_ui.py click --elements ./1.elements.json --id e_0007
  python3 operate_ui.py click-xy --x 800 --y 520
  python3 operate_ui.py click-xy --x 420 --y 300 --coord-space parsed --coord-profile ./coord_profile.json
  python3 operate_ui.py type --text "hello world"
  python3 operate_ui.py key --key enter
  python3 operate_ui.py hotkey --keys command c
  python3 operate_ui.py screenshot
  python3 operate_ui.py calibrate --parsed-size 1515 2880 --actual-size 982 1864 --output ./coord_profile.json
  python3 operate_ui.py calibrate --parsed-anchor 300 200 --actual-anchor 620 430 --parsed-anchor-2 1200 2400 --actual-anchor-2 1010 1690 --output ./coord_profile.json
"""

from __future__ import annotations

import argparse
import json
import re
import secrets
import subprocess
import tempfile
import time
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple


@dataclass
class CoordProfile:
    x_scale: float = 1.0
    y_scale: float = 1.0
    x_offset: float = 0.0
    y_offset: float = 0.0
    display_origin_x: float = 0.0
    display_origin_y: float = 0.0
    name: str = "default"
    created_at: str = ""

    @staticmethod
    def identity() -> "CoordProfile":
        return CoordProfile(created_at=utc_now())


def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()


def load_json(path: Path) -> Dict[str, Any]:
    if not path.exists():
        raise FileNotFoundError(f"elements json not found: {path}")
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)


def get_elements(data: Dict[str, Any]) -> List[Dict[str, Any]]:
    elements = data.get("elements", [])
    if not isinstance(elements, list):
        raise ValueError("Invalid JSON: 'elements' must be a list.")
    return elements


def find_element(elements: List[Dict[str, Any]], element_id: str) -> Dict[str, Any]:
    for elem in elements:
        if str(elem.get("id")) == element_id:
            return elem
    raise ValueError(f"Element id not found: {element_id}")


def parse_bool_str(value: Any) -> Optional[bool]:
    if isinstance(value, bool):
        return value
    if value is None:
        return None
    s = str(value).strip().lower()
    if s in {"true", "1", "yes", "y"}:
        return True
    if s in {"false", "0", "no", "n"}:
        return False
    return None


def element_text(elem: Dict[str, Any]) -> str:
    for key in ("text", "content"):
        if key in elem and elem[key] is not None:
            return str(elem[key])
    return ""


def element_bbox_px(elem: Dict[str, Any]) -> Optional[Tuple[int, int, int, int]]:
    bbox = elem.get("bbox_px")
    if isinstance(bbox, list) and len(bbox) == 4:
        x1, y1, x2, y2 = [int(v) for v in bbox]
        return x1, y1, x2, y2
    return None


def element_center(elem: Dict[str, Any]) -> Tuple[int, int]:
    center = elem.get("center_px")
    if isinstance(center, list) and len(center) == 2:
        return int(center[0]), int(center[1])

    bbox = element_bbox_px(elem)
    if bbox:
        x1, y1, x2, y2 = bbox
        return int(round((x1 + x2) / 2)), int(round((y1 + y2) / 2))

    raise ValueError(f"Element has no valid center/bbox: {elem.get('id')}")


def element_area(elem: Dict[str, Any]) -> int:
    bbox = element_bbox_px(elem)
    if not bbox:
        return 0
    x1, y1, x2, y2 = bbox
    return max(0, x2 - x1) * max(0, y2 - y1)


def require_pyautogui():
    try:
        import pyautogui  # type: ignore
    except ImportError as exc:
        raise RuntimeError(
            "pyautogui is required for UI operation. Install it with:\n"
            "  pip install pyautogui"
        ) from exc
    return pyautogui


def apply_delay(seconds: float) -> None:
    if seconds > 0:
        time.sleep(seconds)


def default_screenshot_path() -> Path:
    ts = datetime.now().strftime("%Y%m%d-%H%M%S")
    rand = secrets.token_hex(4)
    return Path(tempfile.gettempdir()) / f"ui-shot-{ts}-{rand}.png"


def default_coord_profile_path() -> Path:
    return Path.cwd() / "coord_profile.json"


def expand_csv(values: Optional[Sequence[str]]) -> List[str]:
    out: List[str] = []
    if not values:
        return out
    for v in values:
        if v is None:
            continue
        for piece in str(v).split(","):
            p = piece.strip()
            if p:
                out.append(p)
    return out


def load_coord_profile(path: Optional[Path]) -> CoordProfile:
    if path is None:
        return CoordProfile.identity()
    if not path.exists():
        raise FileNotFoundError(f"coord profile not found: {path}")

    with path.open("r", encoding="utf-8") as f:
        data = json.load(f)

    display_origin = data.get("display_origin", [0, 0])
    if not (isinstance(display_origin, list) and len(display_origin) == 2):
        display_origin = [data.get("display_origin_x", 0), data.get("display_origin_y", 0)]

    return CoordProfile(
        x_scale=float(data.get("x_scale", 1.0)),
        y_scale=float(data.get("y_scale", 1.0)),
        x_offset=float(data.get("x_offset", 0.0)),
        y_offset=float(data.get("y_offset", 0.0)),
        display_origin_x=float(display_origin[0]),
        display_origin_y=float(display_origin[1]),
        name=str(data.get("name", "default")),
        created_at=str(data.get("created_at", "")),
    )


def transform_parsed_point(x: float, y: float, profile: CoordProfile) -> Tuple[int, int]:
    tx = x * profile.x_scale + profile.x_offset + profile.display_origin_x
    ty = y * profile.y_scale + profile.y_offset + profile.display_origin_y
    return int(round(tx)), int(round(ty))


def describe_profile(profile: CoordProfile) -> str:
    return (
        f"name={profile.name} "
        f"scale=({profile.x_scale:.6f},{profile.y_scale:.6f}) "
        f"offset=({profile.x_offset:.3f},{profile.y_offset:.3f}) "
        f"display_origin=({profile.display_origin_x:.3f},{profile.display_origin_y:.3f})"
    )


def click_at(x: int, y: int, args: argparse.Namespace, source: str = "") -> None:
    apply_delay(args.delay)
    if args.dry_run:
        prefix = f"{source} " if source else ""
        print(f"[dry-run] {prefix}click at ({x}, {y})")
        return

    pyautogui = require_pyautogui()
    pyautogui.FAILSAFE = not args.disable_failsafe
    pyautogui.moveTo(x=x, y=y, duration=args.duration)
    pyautogui.click(
        x=x,
        y=y,
        clicks=args.clicks,
        interval=args.click_interval,
        button=args.button,
    )
    prefix = f"{source} " if source else ""
    print(f"{prefix}clicked at ({x}, {y})")


def matches_element(elem: Dict[str, Any], args: argparse.Namespace) -> bool:
    type_filters = {x.lower() for x in expand_csv(getattr(args, "types", None))}
    raw_type_filters = {x.lower() for x in expand_csv(getattr(args, "raw_types", None))}
    text_contains = [x.lower() for x in expand_csv(getattr(args, "text_contains", None))]

    elem_type = str(elem.get("type", "")).lower()
    raw_type = str(elem.get("raw_type", "")).lower()
    elem_text_value = element_text(elem)
    elem_text_lower = elem_text_value.lower()
    elem_id = str(elem.get("id", ""))

    if type_filters and elem_type not in type_filters:
        return False
    if raw_type_filters and raw_type not in raw_type_filters:
        return False

    clickable_filter = getattr(args, "clickable", "any")
    elem_clickable = bool(elem.get("clickable", False))
    if clickable_filter == "true" and not elem_clickable:
        return False
    if clickable_filter == "false" and elem_clickable:
        return False

    for token in text_contains:
        if token not in elem_text_lower:
            return False

    id_regex = getattr(args, "id_regex", None)
    if id_regex and re.search(id_regex, elem_id, flags=re.IGNORECASE) is None:
        return False

    text_regex = getattr(args, "text_regex", None)
    if text_regex and re.search(text_regex, elem_text_value, flags=re.IGNORECASE) is None:
        return False

    return True


def filter_elements(elements: List[Dict[str, Any]], args: argparse.Namespace) -> List[Dict[str, Any]]:
    matches = [elem for elem in elements if matches_element(elem, args)]
    sort_by = getattr(args, "sort", "yx")
    if sort_by == "id":
        matches.sort(key=lambda e: str(e.get("id", "")))
    elif sort_by == "area":
        matches.sort(key=lambda e: element_area(e), reverse=True)
    else:
        matches.sort(
            key=lambda e: (
                int(element_center(e)[1]) if "center_px" in e or "bbox_px" in e else 0,
                int(element_center(e)[0]) if "center_px" in e or "bbox_px" in e else 0,
                str(e.get("id", "")),
            )
        )
    return matches


def format_element_line(elem: Dict[str, Any]) -> str:
    elem_id = elem.get("id", "")
    elem_type = elem.get("type", "")
    raw_type = elem.get("raw_type", "")
    clickable = bool(elem.get("clickable", False))
    center = elem.get("center_px")
    text = element_text(elem).replace("\n", " ").strip()
    if len(text) > 80:
        text = text[:77] + "..."
    return (
        f"id={elem_id} type={elem_type} raw_type={raw_type} "
        f"clickable={clickable} center={center} text={text!r}"
    )


def resolve_click_point_from_element(
    elem: Dict[str, Any],
    coord_profile_path: Optional[Path],
    offset_x: int = 0,
    offset_y: int = 0,
) -> Tuple[int, int, CoordProfile]:
    px, py = element_center(elem)
    px += offset_x
    py += offset_y
    profile = load_coord_profile(coord_profile_path)
    tx, ty = transform_parsed_point(px, py, profile)
    return tx, ty, profile


def run_refresh_command(cmd: str, timeout: float, ignore_errors: bool) -> None:
    result = subprocess.run(
        cmd,
        shell=True,
        text=True,
        capture_output=True,
        timeout=timeout,
    )
    if result.returncode == 0:
        return

    msg = (
        f"refresh command failed (exit={result.returncode}): {cmd}\n"
        f"stdout:\n{result.stdout[-600:]}\n"
        f"stderr:\n{result.stderr[-600:]}"
    )
    if ignore_errors:
        print(f"warning: {msg}")
    else:
        raise RuntimeError(msg)


def cmd_list(args: argparse.Namespace) -> None:
    data = load_json(args.elements)
    elements = get_elements(data)
    limit = args.limit if args.limit > 0 else len(elements)

    print(f"total={len(elements)}")
    for elem in elements[:limit]:
        print(format_element_line(elem))


def cmd_find(args: argparse.Namespace) -> None:
    data = load_json(args.elements)
    elements = get_elements(data)
    matches = filter_elements(elements, args)

    if args.as_json:
        output = {
            "count": len(matches),
            "matches": matches[: args.limit if args.limit > 0 else len(matches)],
        }
        print(json.dumps(output, ensure_ascii=False, indent=2))
    else:
        print(f"matched={len(matches)}")
        limit = args.limit if args.limit > 0 else len(matches)
        for elem in matches[:limit]:
            print(format_element_line(elem))

    if args.click:
        if not matches:
            raise ValueError("No element matched the find criteria, cannot click.")
        idx = args.index
        if idx < 0 or idx >= len(matches):
            raise ValueError(f"--index out of range. got={idx}, available={len(matches)}")
        elem = matches[idx]
        x, y, profile = resolve_click_point_from_element(
            elem,
            args.coord_profile,
            offset_x=args.offset_x,
            offset_y=args.offset_y,
        )
        print(f"using profile: {describe_profile(profile)}")
        click_at(x, y, args, source=f"find id={elem.get('id')}")


def cmd_click(args: argparse.Namespace) -> None:
    data = load_json(args.elements)
    elements = get_elements(data)
    elem = find_element(elements, args.id)
    x, y, profile = resolve_click_point_from_element(
        elem,
        args.coord_profile,
        offset_x=args.offset_x,
        offset_y=args.offset_y,
    )
    print(f"using profile: {describe_profile(profile)}")
    click_at(x, y, args, source=f"id={args.id}")


def cmd_click_xy(args: argparse.Namespace) -> None:
    if args.coord_space == "parsed":
        profile = load_coord_profile(args.coord_profile)
        x, y = transform_parsed_point(args.x, args.y, profile)
        print(f"using profile: {describe_profile(profile)}")
    else:
        x, y = int(args.x), int(args.y)
    click_at(x, y, args)


def cmd_type(args: argparse.Namespace) -> None:
    apply_delay(args.delay)
    if args.dry_run:
        print(f"[dry-run] type text={args.text!r}")
        return

    pyautogui = require_pyautogui()
    pyautogui.FAILSAFE = not args.disable_failsafe
    pyautogui.write(args.text, interval=args.interval)
    print("typed text")


def cmd_key(args: argparse.Namespace) -> None:
    apply_delay(args.delay)
    if args.dry_run:
        print(f"[dry-run] press key={args.key}")
        return

    pyautogui = require_pyautogui()
    pyautogui.FAILSAFE = not args.disable_failsafe
    pyautogui.press(args.key)
    print(f"pressed key={args.key}")


def cmd_hotkey(args: argparse.Namespace) -> None:
    apply_delay(args.delay)
    if args.dry_run:
        print(f"[dry-run] hotkey keys={args.keys}")
        return

    pyautogui = require_pyautogui()
    pyautogui.FAILSAFE = not args.disable_failsafe
    pyautogui.hotkey(*args.keys)
    print(f"pressed hotkey keys={args.keys}")


def cmd_screenshot(args: argparse.Namespace) -> None:
    output = Path(args.output) if args.output else default_screenshot_path()
    output.parent.mkdir(parents=True, exist_ok=True)
    apply_delay(args.delay)

    if args.dry_run:
        if args.region:
            print(f"[dry-run] screenshot region={tuple(args.region)} -> {output}")
        else:
            print(f"[dry-run] screenshot full screen -> {output}")
        return

    pyautogui = require_pyautogui()
    pyautogui.FAILSAFE = not args.disable_failsafe
    if args.region:
        x, y, w, h = args.region
        image = pyautogui.screenshot(region=(x, y, w, h))
    else:
        image = pyautogui.screenshot()
    image.save(str(output))
    print(f"saved screenshot: {output}")


def cmd_wait(args: argparse.Namespace) -> None:
    deadline = time.time() + args.timeout
    attempts = 0

    if args.state == "disappear" and args.click:
        raise ValueError("--click is only valid when --state appear.")

    while True:
        attempts += 1

        if args.refresh_cmd:
            run_refresh_command(args.refresh_cmd, args.refresh_timeout, args.ignore_refresh_errors)

        matches: List[Dict[str, Any]] = []
        read_error: Optional[Exception] = None
        try:
            data = load_json(args.elements)
            elements = get_elements(data)
            matches = filter_elements(elements, args)
        except Exception as exc:  # noqa: BLE001
            read_error = exc
            if args.strict_read_errors:
                raise

        matched_count = len(matches)
        ok = matched_count > 0 if args.state == "appear" else matched_count == 0

        if args.verbose:
            elapsed = args.timeout - max(0.0, deadline - time.time())
            err_txt = f" read_error={read_error}" if read_error else ""
            print(
                f"attempt={attempts} elapsed={elapsed:.1f}s state={args.state} "
                f"matches={matched_count}{err_txt}"
            )

        if ok:
            print(
                f"wait success after {attempts} attempts: "
                f"state={args.state}, matches={matched_count}"
            )
            if args.state == "appear" and args.click:
                idx = args.index
                if idx < 0 or idx >= len(matches):
                    raise ValueError(
                        f"--index out of range for click. got={idx}, available={len(matches)}"
                    )
                elem = matches[idx]
                x, y, profile = resolve_click_point_from_element(
                    elem,
                    args.coord_profile,
                    offset_x=args.offset_x,
                    offset_y=args.offset_y,
                )
                print(f"using profile: {describe_profile(profile)}")
                click_at(x, y, args, source=f"wait id={elem.get('id')}")
            return

        if time.time() >= deadline:
            raise SystemExit(
                f"wait timeout after {args.timeout:.1f}s: state={args.state}, matches={matched_count}"
            )

        time.sleep(args.interval)


def cmd_screen_info(_args: argparse.Namespace) -> None:
    pyautogui = require_pyautogui()
    width, height = pyautogui.size()
    print(f"primary_screen_size=({width}, {height})")

    try:
        from screeninfo import get_monitors  # type: ignore
    except Exception:
        print("monitor_details=unavailable (install screeninfo for monitor list)")
        return

    monitors = get_monitors()
    if not monitors:
        print("monitor_details=none")
        return

    for i, m in enumerate(monitors):
        print(
            f"monitor[{i}]: x={m.x} y={m.y} width={m.width} "
            f"height={m.height} name={getattr(m, 'name', '')}"
        )


def validate_pair(name: str, value: Optional[Sequence[float]]) -> Tuple[float, float]:
    if value is None or len(value) != 2:
        raise ValueError(f"{name} requires exactly 2 numeric values")
    return float(value[0]), float(value[1])


def cmd_calibrate(args: argparse.Namespace) -> None:
    parsed_anchor = tuple(args.parsed_anchor) if args.parsed_anchor else None
    actual_anchor = tuple(args.actual_anchor) if args.actual_anchor else None
    parsed_anchor_2 = tuple(args.parsed_anchor_2) if args.parsed_anchor_2 else None
    actual_anchor_2 = tuple(args.actual_anchor_2) if args.actual_anchor_2 else None

    if bool(parsed_anchor_2) != bool(actual_anchor_2):
        raise ValueError("--parsed-anchor-2 and --actual-anchor-2 must be provided together")

    if parsed_anchor_2 and actual_anchor_2:
        if not parsed_anchor or not actual_anchor:
            raise ValueError(
                "Two-point calibration requires both --parsed-anchor and --actual-anchor"
            )

        px1, py1 = parsed_anchor
        ax1, ay1 = actual_anchor
        px2, py2 = parsed_anchor_2
        ax2, ay2 = actual_anchor_2

        if px2 == px1 or py2 == py1:
            raise ValueError("Anchor points must not share same x or y in parsed coordinates")

        x_scale = (ax2 - ax1) / (px2 - px1)
        y_scale = (ay2 - ay1) / (py2 - py1)

        x_offset = ax1 - px1 * x_scale
        y_offset = ay1 - py1 * y_scale
        mode = "two-point"
    else:
        if bool(args.parsed_size) != bool(args.actual_size):
            raise ValueError("--parsed-size and --actual-size must be provided together")
        if not args.parsed_size or not args.actual_size:
            raise ValueError(
                "Provide either two-point anchors or size-based calibration via "
                "--parsed-size/--actual-size"
            )

        parsed_w, parsed_h = validate_pair("--parsed-size", args.parsed_size)
        actual_w, actual_h = validate_pair("--actual-size", args.actual_size)
        if parsed_w <= 0 or parsed_h <= 0:
            raise ValueError("--parsed-size values must be > 0")
        if actual_w <= 0 or actual_h <= 0:
            raise ValueError("--actual-size values must be > 0")

        x_scale = actual_w / parsed_w
        y_scale = actual_h / parsed_h

        if parsed_anchor and actual_anchor:
            px1, py1 = parsed_anchor
            ax1, ay1 = actual_anchor
            x_offset = ax1 - px1 * x_scale
            y_offset = ay1 - py1 * y_scale
        else:
            x_offset = 0.0
            y_offset = 0.0
        mode = "size"

    win_off_x, win_off_y = validate_pair("--window-offset", args.window_offset)
    disp_x, disp_y = validate_pair("--display-origin", args.display_origin)
    x_offset += win_off_x
    y_offset += win_off_y

    profile = CoordProfile(
        x_scale=x_scale,
        y_scale=y_scale,
        x_offset=x_offset,
        y_offset=y_offset,
        display_origin_x=disp_x,
        display_origin_y=disp_y,
        name=args.name,
        created_at=utc_now(),
    )

    payload = asdict(profile)
    payload["display_origin"] = [profile.display_origin_x, profile.display_origin_y]
    payload["calibration_mode"] = mode

    output = args.output
    if args.dry_run:
        print(json.dumps(payload, ensure_ascii=False, indent=2))
        return

    output.parent.mkdir(parents=True, exist_ok=True)
    with output.open("w", encoding="utf-8") as f:
        json.dump(payload, f, ensure_ascii=False, indent=2)

    print(f"saved coord profile: {output}")
    print(describe_profile(profile))


def add_matcher_args(parser: argparse.ArgumentParser) -> None:
    parser.add_argument("--elements", type=Path, required=True, help="Path to elements JSON.")
    parser.add_argument(
        "--type",
        dest="types",
        action="append",
        default=[],
        help="Normalized type filter (repeatable or comma-separated).",
    )
    parser.add_argument(
        "--raw-type",
        dest="raw_types",
        action="append",
        default=[],
        help="Raw detector type filter (repeatable or comma-separated).",
    )
    parser.add_argument(
        "--text-contains",
        action="append",
        default=[],
        help="Case-insensitive text substring filter (AND semantics).",
    )
    parser.add_argument(
        "--text-regex",
        default=None,
        help="Case-insensitive regex against element text.",
    )
    parser.add_argument(
        "--id-regex",
        default=None,
        help="Case-insensitive regex against element id.",
    )
    parser.add_argument(
        "--clickable",
        choices=["any", "true", "false"],
        default="any",
        help="Filter by clickable flag.",
    )
    parser.add_argument(
        "--sort",
        choices=["yx", "id", "area"],
        default="yx",
        help="Sort order for matched results.",
    )


def add_click_runtime_args(
    parser: argparse.ArgumentParser, click_interval_option: str = "--interval"
) -> None:
    parser.add_argument("--duration", type=float, default=0.0, help="Mouse move duration.")
    parser.add_argument("--clicks", type=int, default=1, help="Number of clicks.")
    parser.add_argument(
        click_interval_option,
        dest="click_interval",
        type=float,
        default=0.0,
        help="Interval between clicks.",
    )
    parser.add_argument(
        "--button",
        choices=["left", "middle", "right"],
        default="left",
        help="Mouse button.",
    )
    parser.add_argument("--delay", type=float, default=0.2, help="Pre-action delay seconds.")
    parser.add_argument("--dry-run", action="store_true", help="Print action without execution.")
    parser.add_argument(
        "--disable-failsafe",
        action="store_true",
        help="Disable pyautogui failsafe.",
    )


def add_coord_profile_args(parser: argparse.ArgumentParser) -> None:
    parser.add_argument(
        "--coord-profile",
        type=Path,
        default=None,
        help="Path to coordinate profile JSON for parsed->screen transform.",
    )


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description=(
            "Operate UI from parsed element JSON. "
            "Headless-safe: list/find/wait/calibrate. "
            "GUI-required: click/click-xy/type/key/hotkey/screenshot/screen-info."
        )
    )
    sub = parser.add_subparsers(dest="command", required=True)

    list_p = sub.add_parser("list", help="List parsed elements.")
    list_p.add_argument("--elements", type=Path, required=True, help="Path to elements JSON.")
    list_p.add_argument("--limit", type=int, default=50, help="Number of lines to print.")
    list_p.set_defaults(func=cmd_list)

    find_p = sub.add_parser("find", help="Find elements by type/text/regex filters.")
    add_matcher_args(find_p)
    find_p.add_argument("--limit", type=int, default=20, help="Number of rows to print.")
    find_p.add_argument("--as-json", action="store_true", help="Print matches as JSON.")
    find_p.add_argument("--click", action="store_true", help="Click the selected match.")
    find_p.add_argument("--index", type=int, default=0, help="Match index used with --click.")
    find_p.add_argument("--offset-x", type=int, default=0, help="Parsed-space x offset.")
    find_p.add_argument("--offset-y", type=int, default=0, help="Parsed-space y offset.")
    add_coord_profile_args(find_p)
    add_click_runtime_args(find_p)
    find_p.set_defaults(func=cmd_find)

    wait_p = sub.add_parser("wait", help="Wait until element match appears or disappears.")
    add_matcher_args(wait_p)
    wait_p.add_argument(
        "--state",
        choices=["appear", "disappear"],
        default="appear",
        help="Success condition.",
    )
    wait_p.add_argument("--timeout", type=float, default=20.0, help="Timeout seconds.")
    wait_p.add_argument("--interval", type=float, default=1.0, help="Polling interval seconds.")
    wait_p.add_argument(
        "--refresh-cmd",
        default=None,
        help="Optional shell command run before each poll (e.g. re-parse screenshot).",
    )
    wait_p.add_argument(
        "--refresh-timeout",
        type=float,
        default=120.0,
        help="Timeout for refresh command.",
    )
    wait_p.add_argument(
        "--ignore-refresh-errors",
        action="store_true",
        help="Continue polling if refresh command exits non-zero.",
    )
    wait_p.add_argument(
        "--strict-read-errors",
        action="store_true",
        help="Fail immediately when elements JSON cannot be read during polling.",
    )
    wait_p.add_argument("--verbose", action="store_true", help="Print per-attempt status.")
    wait_p.add_argument("--click", action="store_true", help="Click matched element on success.")
    wait_p.add_argument("--index", type=int, default=0, help="Match index used with --click.")
    wait_p.add_argument("--offset-x", type=int, default=0, help="Parsed-space x offset.")
    wait_p.add_argument("--offset-y", type=int, default=0, help="Parsed-space y offset.")
    add_coord_profile_args(wait_p)
    add_click_runtime_args(wait_p, click_interval_option="--click-interval")
    wait_p.set_defaults(func=cmd_wait)

    click_p = sub.add_parser("click", help="Click by element id.")
    click_p.add_argument("--elements", type=Path, required=True, help="Path to elements JSON.")
    click_p.add_argument("--id", required=True, help="Element id, e.g. e_0007.")
    click_p.add_argument("--offset-x", type=int, default=0, help="Parsed-space x offset.")
    click_p.add_argument("--offset-y", type=int, default=0, help="Parsed-space y offset.")
    add_coord_profile_args(click_p)
    add_click_runtime_args(click_p)
    click_p.set_defaults(func=cmd_click)

    click_xy_p = sub.add_parser("click-xy", help="Click by coordinates.")
    click_xy_p.add_argument("--x", type=float, required=True)
    click_xy_p.add_argument("--y", type=float, required=True)
    click_xy_p.add_argument(
        "--coord-space",
        choices=["screen", "parsed"],
        default="screen",
        help="Coordinate space for x/y. parsed requires --coord-profile.",
    )
    add_coord_profile_args(click_xy_p)
    add_click_runtime_args(click_xy_p)
    click_xy_p.set_defaults(func=cmd_click_xy)

    type_p = sub.add_parser("type", help="Type text.")
    type_p.add_argument("--text", required=True, help="Text to type.")
    type_p.add_argument("--interval", type=float, default=0.02, help="Keystroke interval.")
    type_p.add_argument("--delay", type=float, default=0.2, help="Pre-action delay seconds.")
    type_p.add_argument("--dry-run", action="store_true", help="Print action without typing.")
    type_p.add_argument(
        "--disable-failsafe",
        action="store_true",
        help="Disable pyautogui failsafe.",
    )
    type_p.set_defaults(func=cmd_type)

    key_p = sub.add_parser("key", help="Press one key.")
    key_p.add_argument("--key", required=True, help="Key name, e.g. enter, tab, esc.")
    key_p.add_argument("--delay", type=float, default=0.2, help="Pre-action delay seconds.")
    key_p.add_argument("--dry-run", action="store_true", help="Print action without key press.")
    key_p.add_argument(
        "--disable-failsafe",
        action="store_true",
        help="Disable pyautogui failsafe.",
    )
    key_p.set_defaults(func=cmd_key)

    hotkey_p = sub.add_parser("hotkey", help="Press multi-key combo.")
    hotkey_p.add_argument(
        "--keys",
        nargs="+",
        required=True,
        help="Key list, e.g. --keys command c",
    )
    hotkey_p.add_argument("--delay", type=float, default=0.2, help="Pre-action delay seconds.")
    hotkey_p.add_argument("--dry-run", action="store_true", help="Print action without hotkey.")
    hotkey_p.add_argument(
        "--disable-failsafe",
        action="store_true",
        help="Disable pyautogui failsafe.",
    )
    hotkey_p.set_defaults(func=cmd_hotkey)

    shot_p = sub.add_parser("screenshot", help="Take screenshot.")
    shot_p.add_argument(
        "--output",
        default=None,
        help="Output image path. Default: user's temp directory.",
    )
    shot_p.add_argument(
        "--region",
        nargs=4,
        type=int,
        metavar=("X", "Y", "W", "H"),
        help="Optional region in pixels.",
    )
    shot_p.add_argument("--delay", type=float, default=0.2, help="Pre-action delay seconds.")
    shot_p.add_argument("--dry-run", action="store_true", help="Print action without screenshot.")
    shot_p.add_argument(
        "--disable-failsafe",
        action="store_true",
        help="Disable pyautogui failsafe.",
    )
    shot_p.set_defaults(func=cmd_screenshot)

    info_p = sub.add_parser("screen-info", help="Show primary screen and monitor info.")
    info_p.set_defaults(func=cmd_screen_info)

    cal_p = sub.add_parser("calibrate", help="Create coordinate transform profile.")
    cal_p.add_argument(
        "--output",
        type=Path,
        default=default_coord_profile_path(),
        help="Output profile path. Default: ./coord_profile.json",
    )
    cal_p.add_argument("--name", default="default", help="Profile name.")
    cal_p.add_argument(
        "--parsed-size",
        nargs=2,
        type=float,
        default=None,
        metavar=("W", "H"),
        help="Parsed screenshot size, e.g. 1515 2880.",
    )
    cal_p.add_argument(
        "--actual-size",
        nargs=2,
        type=float,
        default=None,
        metavar=("W", "H"),
        help="Actual screen/render size corresponding to parsed image.",
    )
    cal_p.add_argument(
        "--parsed-anchor",
        nargs=2,
        type=float,
        default=None,
        metavar=("X", "Y"),
        help="Parsed coordinate anchor point.",
    )
    cal_p.add_argument(
        "--actual-anchor",
        nargs=2,
        type=float,
        default=None,
        metavar=("X", "Y"),
        help="Actual screen coordinate for anchor point.",
    )
    cal_p.add_argument(
        "--parsed-anchor-2",
        nargs=2,
        type=float,
        default=None,
        metavar=("X", "Y"),
        help="Second parsed anchor for two-point calibration.",
    )
    cal_p.add_argument(
        "--actual-anchor-2",
        nargs=2,
        type=float,
        default=None,
        metavar=("X", "Y"),
        help="Second actual anchor for two-point calibration.",
    )
    cal_p.add_argument(
        "--window-offset",
        nargs=2,
        type=float,
        default=(0.0, 0.0),
        metavar=("DX", "DY"),
        help="Extra offset after scaling (window origin offset).",
    )
    cal_p.add_argument(
        "--display-origin",
        nargs=2,
        type=float,
        default=(0.0, 0.0),
        metavar=("X", "Y"),
        help="Display origin for multi-monitor coordinate systems.",
    )
    cal_p.add_argument("--dry-run", action="store_true", help="Print profile JSON only.")
    cal_p.set_defaults(func=cmd_calibrate)

    return parser


def main() -> None:
    parser = build_parser()
    args = parser.parse_args()
    args.func(args)


if __name__ == "__main__":
    main()


path: scripts/bootstrap_omniparser_env.sh
content: #!/usr/bin/env bash
set -euo pipefail

# Usage:
#   bootstrap_omniparser_env.sh [workdir] [venv_path] [omniparser_dir]
#
# Example:
#   skills/ui-element-ops/scripts/bootstrap_omniparser_env.sh "$PWD"

WORKDIR="${1:-$PWD}"
VENV_PATH="${2:-$WORKDIR/.venv}"
OMNIPARSER_DIR="${3:-/tmp/OmniParser}"

if ! command -v python3.12 >/dev/null 2>&1; then
  echo "python3.12 is required but not found." >&2
  exit 1
fi

mkdir -p "$WORKDIR"
python3.12 -m venv "$VENV_PATH"

"$VENV_PATH/bin/python" -m pip install --upgrade pip
"$VENV_PATH/bin/pip" install \
  pillow \
  requests \
  openai \
  numpy==1.26.4 \
  matplotlib \
  torch \
  torchvision \
  easyocr \
  supervision==0.18.0 \
  ultralytics==8.3.70 \
  "transformers==4.49.0" \
  accelerate \
  timm \
  einops==0.8.0 \
  opencv-python \
  pyautogui \
  screeninfo \
  huggingface_hub

if [ ! -d "$OMNIPARSER_DIR/.git" ]; then
  git clone --depth 1 https://github.com/microsoft/OmniParser "$OMNIPARSER_DIR"
fi

mkdir -p "$OMNIPARSER_DIR/weights" /tmp/hf /tmp/xdg

HF_HOME=/tmp/hf XDG_CACHE_HOME=/tmp/xdg \
  "$VENV_PATH/bin/hf" download microsoft/OmniParser-v2.0 \
  icon_detect/train_args.yaml \
  icon_detect/model.pt \
  icon_detect/model.yaml \
  icon_caption/config.json \
  icon_caption/generation_config.json \
  icon_caption/model.safetensors \
  --local-dir "$OMNIPARSER_DIR/weights"

if [ -d "$OMNIPARSER_DIR/weights/icon_caption" ] && [ ! -d "$OMNIPARSER_DIR/weights/icon_caption_florence" ]; then
  mv "$OMNIPARSER_DIR/weights/icon_caption" "$OMNIPARSER_DIR/weights/icon_caption_florence"
fi

echo "Environment ready:"
echo "  venv: $VENV_PATH"
echo "  OmniParser: $OMNIPARSER_DIR"
echo "  weights: $OMNIPARSER_DIR/weights"


path: scripts/parse_ui.py
content: #!/usr/bin/env python3
"""Parse UI screenshot into element types and coordinates using OmniParser.

Usage example:
  python3 parse_ui.py \
    --image ./screen.png \
    --output ./screen.elements.json \
    --overlay ./screen.overlay.png \
    --omniparser-dir /path/to/OmniParser
"""

from __future__ import annotations

import argparse
import base64
import json
import os
import sys
import types
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional


DEFAULT_TYPE_RULES: Dict[str, List[str]] = {
    "button": [
        "login",
        "log in",
        "sign in",
        "submit",
        "confirm",
        "ok",
        "next",
        "done",
        "save",
        "send",
        "cancel",
        "back",
        "continue",
        "buy",
        "pay",
        "install",
        "open",
        "allow",
    ],
    "input": [
        "search",
        "username",
        "password",
        "email",
        "phone",
        "type here",
        "enter",
        "placeholder",
    ],
    "checkbox": ["checkbox", "accept terms", "remember me", "agree"],
    "switch": ["toggle", "switch", "on", "off"],
    "tab": ["home", "feed", "profile", "settings", "notifications"],
    "menu": ["menu", "more", "hamburger", "options"],
    "link": ["http", ".com", ".cn", ".net", ".org", "www."],
    "list_item": ["item", "row", "option", "result"],
}


@dataclass
class ParserConfig:
    image: Path
    output: Path
    overlay: Optional[Path]
    omniparser_dir: Path
    som_model_path: str
    caption_model_name: str
    caption_model_path: str
    box_threshold: float
    iou_threshold: float
    use_paddleocr: bool
    ocr_text_threshold: float
    imgsz: Optional[int]
    type_rules_path: Optional[Path]


def parse_args() -> ParserConfig:
    parser = argparse.ArgumentParser(
        description="Parse UI screenshot into structured elements with OmniParser."
    )
    parser.add_argument("--image", required=True, type=Path, help="Input screenshot path.")
    parser.add_argument(
        "--output",
        type=Path,
        default=None,
        help="Output JSON path. Default: <image>.elements.json",
    )
    parser.add_argument(
        "--overlay",
        type=Path,
        default=None,
        help="Optional output path for OmniParser overlay image.",
    )
    parser.add_argument(
        "--omniparser-dir",
        type=Path,
        default=Path(os.environ.get("OMNIPARSER_DIR", "")),
        help="Local OmniParser repository directory.",
    )
    parser.add_argument(
        "--som-model-path",
        default="weights/icon_detect/model.pt",
        help="Path relative to --omniparser-dir if not absolute.",
    )
    parser.add_argument(
        "--caption-model-name",
        default="florence2",
        choices=["florence2", "blip2", "phi3_v"],
        help="Caption model name used by OmniParser.",
    )
    parser.add_argument(
        "--caption-model-path",
        default="weights/icon_caption_florence",
        help="Path relative to --omniparser-dir if not absolute.",
    )
    parser.add_argument(
        "--box-threshold",
        type=float,
        default=0.05,
        help="Detection confidence threshold.",
    )
    parser.add_argument(
        "--iou-threshold",
        type=float,
        default=0.7,
        help="IoU threshold for overlap filtering.",
    )
    parser.add_argument(
        "--use-paddleocr",
        action="store_true",
        help="Use PaddleOCR instead of EasyOCR.",
    )
    parser.add_argument(
        "--ocr-text-threshold",
        type=float,
        default=0.8,
        help="OCR text confidence threshold.",
    )
    parser.add_argument(
        "--imgsz",
        type=int,
        default=None,
        help="Optional icon detect image size.",
    )
    parser.add_argument(
        "--type-rules",
        type=Path,
        default=None,
        help="Optional JSON file to override type keyword rules.",
    )
    args = parser.parse_args()

    if not args.image.exists():
        raise FileNotFoundError(f"--image not found: {args.image}")
    if not args.omniparser_dir:
        raise ValueError("Provide --omniparser-dir or set OMNIPARSER_DIR.")
    if not args.omniparser_dir.exists():
        raise FileNotFoundError(f"--omniparser-dir not found: {args.omniparser_dir}")

    output = args.output
    if output is None:
        output = args.image.with_suffix("").with_suffix(".elements.json")

    return ParserConfig(
        image=args.image.resolve(),
        output=output.resolve(),
        overlay=args.overlay.resolve() if args.overlay else None,
        omniparser_dir=args.omniparser_dir.resolve(),
        som_model_path=args.som_model_path,
        caption_model_name=args.caption_model_name,
        caption_model_path=args.caption_model_path,
        box_threshold=args.box_threshold,
        iou_threshold=args.iou_threshold,
        use_paddleocr=args.use_paddleocr,
        ocr_text_threshold=args.ocr_text_threshold,
        imgsz=args.imgsz,
        type_rules_path=args.type_rules.resolve() if args.type_rules else None,
    )


def resolve_model_path(base_dir: Path, value: str) -> str:
    path = Path(value)
    if path.is_absolute():
        return str(path)
    return str((base_dir / value).resolve())


def ensure_optional_paddleocr_stub() -> None:
    """Provide a lightweight fallback when paddleocr is unavailable.

    OmniParser imports PaddleOCR at module import time even when `use_paddleocr=False`.
    This stub allows the import path to succeed without installing paddleocr.
    """
    try:
        __import__("paddleocr")
        return
    except Exception:
        pass

    stub = types.ModuleType("paddleocr")

    class PaddleOCR:  # type: ignore[no-redef]
        def __init__(self, *args: Any, **kwargs: Any) -> None:
            pass

        def ocr(self, *_args: Any, **_kwargs: Any) -> List[Any]:
            return [[]]

    stub.PaddleOCR = PaddleOCR  # type: ignore[attr-defined]
    sys.modules["paddleocr"] = stub


def load_type_rules(path: Optional[Path]) -> Dict[str, List[str]]:
    if path is None:
        return DEFAULT_TYPE_RULES
    with path.open("r", encoding="utf-8") as f:
        data = json.load(f)
    if not isinstance(data, dict):
        raise ValueError("--type-rules must be a JSON object.")
    rules: Dict[str, List[str]] = {}
    for key, value in data.items():
        if not isinstance(value, list) or not all(isinstance(x, str) for x in value):
            raise ValueError(f"Invalid type rule for '{key}', expected list[str].")
        rules[key] = value
    return rules


def contains_any(text: str, keywords: List[str]) -> bool:
    return any(k in text for k in keywords)


def normalize_type(
    raw_type: str,
    clickable: bool,
    content: Optional[str],
    bbox_norm: List[float],
    rules: Dict[str, List[str]],
) -> str:
    text = (content or "").strip().lower()
    x1, y1, x2, y2 = bbox_norm
    width = max(1e-6, x2 - x1)
    height = max(1e-6, y2 - y1)
    aspect_ratio = width / height

    if raw_type == "text":
        if text and contains_any(text, rules.get("link", [])):
            return "link"
        return "text"

    if not clickable:
        if aspect_ratio > 1.8 and text:
            return "text"
        return "image"

    if text:
        if contains_any(text, rules.get("input", [])) and aspect_ratio > 1.6:
            return "input"
        if contains_any(text, rules.get("checkbox", [])):
            return "checkbox"
        if contains_any(text, rules.get("switch", [])):
            return "switch"
        if contains_any(text, rules.get("tab", [])):
            return "tab"
        if contains_any(text, rules.get("menu", [])):
            return "menu"
        if contains_any(text, rules.get("list_item", [])):
            return "list_item"
        if contains_any(text, rules.get("button", [])):
            return "button"

    # Fallbacks for interactable regions from detector.
    if aspect_ratio > 2.2 and text:
        return "input"
    if text:
        return "button"
    return "icon"


def clamp_bbox_norm(bbox_norm: List[float]) -> List[float]:
    x1, y1, x2, y2 = bbox_norm
    x1 = min(max(x1, 0.0), 1.0)
    y1 = min(max(y1, 0.0), 1.0)
    x2 = min(max(x2, 0.0), 1.0)
    y2 = min(max(y2, 0.0), 1.0)
    if x2 < x1:
        x1, x2 = x2, x1
    if y2 < y1:
        y1, y2 = y2, y1
    return [x1, y1, x2, y2]


def norm_to_px(bbox_norm: List[float], width: int, height: int) -> List[int]:
    x1, y1, x2, y2 = bbox_norm
    return [
        max(0, min(width - 1, int(round(x1 * width)))),
        max(0, min(height - 1, int(round(y1 * height)))),
        max(0, min(width - 1, int(round(x2 * width)))),
        max(0, min(height - 1, int(round(y2 * height)))),
    ]


def parse_ui(config: ParserConfig) -> Dict[str, Any]:
    sys.path.insert(0, str(config.omniparser_dir))
    ensure_optional_paddleocr_stub()

    try:
        from PIL import Image
        from util.utils import (  # type: ignore[import-not-found]
            check_ocr_box,
            get_caption_model_processor,
            get_som_labeled_img,
            get_yolo_model,
        )
    except Exception as exc:
        raise RuntimeError(
            "Failed to import OmniParser modules. Make sure dependencies are installed "
            "inside your Python environment."
        ) from exc

    type_rules = load_type_rules(config.type_rules_path)

    image = Image.open(config.image).convert("RGB")
    width, height = image.size

    box_overlay_ratio = max(width, height) / 3200
    draw_bbox_config = {
        "text_scale": 0.8 * box_overlay_ratio,
        "text_thickness": max(int(2 * box_overlay_ratio), 1),
        "text_padding": max(int(3 * box_overlay_ratio), 1),
        "thickness": max(int(3 * box_overlay_ratio), 1),
    }

    som_model_path = resolve_model_path(config.omniparser_dir, config.som_model_path)
    caption_model_path = resolve_model_path(config.omniparser_dir, config.caption_model_path)

    som_model = get_yolo_model(model_path=som_model_path)
    caption_model_processor = get_caption_model_processor(
        model_name=config.caption_model_name,
        model_name_or_path=caption_model_path,
    )

    (ocr_text, ocr_bbox), _ = check_ocr_box(
        image,
        display_img=False,
        output_bb_format="xyxy",
        goal_filtering=None,
        easyocr_args={"paragraph": False, "text_threshold": config.ocr_text_threshold},
        use_paddleocr=config.use_paddleocr,
    )

    som_image_base64, _label_coordinates, parsed_content = get_som_labeled_img(
        image,
        model=som_model,
        BOX_TRESHOLD=config.box_threshold,
        output_coord_in_ratio=True,
        ocr_bbox=ocr_bbox,
        draw_bbox_config=draw_bbox_config,
        caption_model_processor=caption_model_processor,
        ocr_text=ocr_text,
        use_local_semantics=True,
        iou_threshold=config.iou_threshold,
        scale_img=False,
        imgsz=config.imgsz,
        batch_size=128,
    )

    elements: List[Dict[str, Any]] = []
    for idx, item in enumerate(parsed_content):
        raw_bbox = item.get("bbox")
        if raw_bbox is None or len(raw_bbox) != 4:
            continue

        bbox_norm = clamp_bbox_norm([float(v) for v in raw_bbox])
        bbox_px = norm_to_px(bbox_norm, width, height)

        raw_type = str(item.get("type", "unknown"))
        clickable = bool(item.get("interactivity", False))
        content = item.get("content")
        content_str = str(content) if content is not None else None

        norm_type = normalize_type(
            raw_type=raw_type,
            clickable=clickable,
            content=content_str,
            bbox_norm=bbox_norm,
            rules=type_rules,
        )

        cx = int(round((bbox_px[0] + bbox_px[2]) / 2))
        cy = int(round((bbox_px[1] + bbox_px[3]) / 2))

        elements.append(
            {
                "id": f"e_{idx:04d}",
                "type": norm_type,
                "raw_type": raw_type,
                "bbox_px": bbox_px,
                "bbox_norm": [round(v, 6) for v in bbox_norm],
                "center_px": [cx, cy],
                "text": content_str,
                "clickable": clickable,
                "confidence": None,
                "source": item.get("source", "omniparser"),
            }
        )

    result: Dict[str, Any] = {
        "schema_version": "1.0.0",
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "pipeline": {
            "name": "omniparser",
            "box_threshold": config.box_threshold,
            "iou_threshold": config.iou_threshold,
            "use_paddleocr": config.use_paddleocr,
            "ocr_text_threshold": config.ocr_text_threshold,
            "caption_model_name": config.caption_model_name,
            "som_model_path": som_model_path,
            "caption_model_path": caption_model_path,
        },
        "image": {
            "path": str(config.image),
            "width": width,
            "height": height,
        },
        "counts": {
            "total": len(elements),
            "clickable": sum(1 for e in elements if e["clickable"]),
            "by_type": _count_by_type(elements),
        },
        "elements": elements,
    }

    if config.overlay:
        config.overlay.parent.mkdir(parents=True, exist_ok=True)
        with config.overlay.open("wb") as f:
            f.write(base64.b64decode(som_image_base64))

    config.output.parent.mkdir(parents=True, exist_ok=True)
    with config.output.open("w", encoding="utf-8") as f:
        json.dump(result, f, ensure_ascii=False, indent=2)

    return result


def _count_by_type(elements: List[Dict[str, Any]]) -> Dict[str, int]:
    out: Dict[str, int] = {}
    for e in elements:
        t = str(e.get("type", "unknown"))
        out[t] = out.get(t, 0) + 1
    return out


def main() -> None:
    config = parse_args()
    result = parse_ui(config)
    print(
        f"Parsed {result['counts']['total']} elements "
        f"from {config.image} -> {config.output}"
    )
    if config.overlay:
        print(f"Overlay image saved to {config.overlay}")


if __name__ == "__main__":
    main()


path: scripts/capture_and_parse.sh
content: #!/usr/bin/env bash
set -euo pipefail

# Capture a screenshot and parse it with randomized file names.
#
# Usage:
#   capture_and_parse.sh [--dry-run] [output_dir] [prefix]
#   capture_and_parse.sh [--dry-run] /abs/path/to/screenshot.png
#
# Note:
#   If a screenshot path is provided and the file already exists, the script
#   skips capture and only runs parsing.
#
# Examples:
#   skills/ui-element-ops/scripts/capture_and_parse.sh
#   skills/ui-element-ops/scripts/capture_and_parse.sh /tmp ui
#   skills/ui-element-ops/scripts/capture_and_parse.sh /Users/me/Downloads/screen.png
#   skills/ui-element-ops/scripts/capture_and_parse.sh --dry-run /tmp ui
#
# Env (optional):
#   VENV_PATH: Python venv path (default: $PWD/.venv)
#   OMNIPARSER_DIR: OmniParser repo path (default: /tmp/OmniParser)
#   TYPE_RULES: type rules json path

DRY_RUN=0
if [ "${1:-}" = "--dry-run" ]; then
  DRY_RUN=1
  shift
fi

OUTPUT_DIR="/tmp"
PREFIX="ui"
EXPLICIT_IMAGE_PATH=""

if [ "${1:-}" != "" ]; then
  case "$1" in
    *.png|*.jpg|*.jpeg|*.webp)
      EXPLICIT_IMAGE_PATH="$1"
      shift
      ;;
    *)
      OUTPUT_DIR="$1"
      shift
      PREFIX="${1:-ui}"
      ;;
  esac
fi

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

if command -v uuidgen >/dev/null 2>&1; then
  RAND="$(uuidgen | tr '[:upper:]' '[:lower:]' | tr -d '-' | cut -c1-8)"
else
  RAND="$(LC_ALL=C tr -dc 'a-f0-9' </dev/urandom | head -c8)"
fi

if [ -n "$EXPLICIT_IMAGE_PATH" ]; then
  IMG="$EXPLICIT_IMAGE_PATH"
  BASE="${IMG%.*}"
  if [ -f "$IMG" ]; then
    CAPTURE_NEEDED=0
  else
    CAPTURE_NEEDED=1
  fi
else
  ID="$(date +%Y%m%d-%H%M%S)-${RAND}"
  BASE="${OUTPUT_DIR%/}/${PREFIX}-${ID}"
  IMG="${BASE}.png"
  CAPTURE_NEEDED=1
fi

JSON="${BASE}.elements.json"
OVERLAY="${BASE}.overlay.png"

mkdir -p "$(dirname "$IMG")"

if [ "$DRY_RUN" -eq 1 ]; then
  echo "[dry-run] screenshot: $IMG"
  echo "[dry-run] elements:   $JSON"
  echo "[dry-run] overlay:    $OVERLAY"
  exit 0
fi

if [ "$CAPTURE_NEEDED" -eq 1 ]; then
  python3 "$SCRIPT_DIR/operate_ui.py" screenshot --output "$IMG"
else
  echo "use existing screenshot: $IMG"
fi
"$SCRIPT_DIR/run_parse_ui.sh" "$IMG" "$JSON" "$OVERLAY"

echo "screenshot: $IMG"
echo "elements:   $JSON"
echo "overlay:    $OVERLAY"


path: agents/openai.yaml
content: interface:
  display_name: "UI Element Ops"
  short_description: "Parse, find, wait, click, and calibrate UI"
  default_prompt: "Use $ui-element-ops to parse UI screenshots, find/wait for elements, perform UI actions, and calibrate coordinates."


path: references/type_rules.example.json
content: {
  "button": [
    "login",
    "log in",
    "sign in",
    "submit",
    "confirm",
    "ok",
    "next",
    "done",
    "save",
    "send",
    "cancel",
    "back",
    "continue",
    "buy",
    "pay",
    "install",
    "open",
    "allow"
  ],
  "input": [
    "search",
    "username",
    "password",
    "email",
    "phone",
    "type here",
    "enter",
    "placeholder"
  ],
  "checkbox": [
    "checkbox",
    "accept terms",
    "remember me",
    "agree"
  ],
  "switch": [
    "toggle",
    "switch",
    "on",
    "off"
  ],
  "tab": [
    "home",
    "feed",
    "profile",
    "settings",
    "notifications"
  ],
  "menu": [
    "menu",
    "more",
    "hamburger",
    "options"
  ],
  "link": [
    "http",
    ".com",
    ".cn",
    ".net",
    ".org",
    "www."
  ],
  "list_item": [
    "item",
    "row",
    "option",
    "result"
  ]
}


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (HIGH)