Skill: appergb/desktop-agent-ops (version 1.0.3)

--- name: desktop-agent-ops description: Execute cross-platform desktop tasks through a packaged desktop automation skill that guides the main agent to observe the screen, focus apps and windows, call helper scripts for screenshots and input actions, verify each step, clean up task context, and only escalate to multi-agent collaboration when tasks become clearly multi-window or multi-app. Use when the user wants desktop GUI control, native app operation, window focus, screenshots, click and type flows, or cross-platform desktop workflows on macOS, Windows, or Linux. version: 1.0.2 metadata: openclaw: requires: bins: [python3] anyBins: [cliclick, xdotool] emoji: 🖥️ os: [macos, windows, linux] install: brew: [cliclick, tesseract] --- # Desktop Agent Ops Use this skill as a **main-agent operating manual** for desktop GUI tasks. --- ## MANDATORY: Auto-setup gate (FIRST ACTION, every time) ```bash python3 <SKILL_DIR>/scripts/first_run_setup.py --check ``` If `"ready": false`, run setup (installs EVERYTHING automatically): ```bash python3 <SKILL_DIR>/scripts/first_run_setup.py ``` **Auto-installs on first run:** 1. Platform detection (macOS / Windows / Linux) 2. `cliclick` + `tesseract` (macOS via brew; Linux guide printed) 3. OCR language packs auto-detected from system locale (中文→chi_sim, 日本語→jpn, etc.) 4. Python venv + pillow, pyautogui, pytesseract, opencv-python, numpy (via uv or pip) 5. OS permissions (Screen Recording, Accessibility, Automation) with auto-open System Settings 6. Smoke test (screenshot + mouse move verification) After setup, set `$PY` for ALL subsequent calls: ``` PY=<output.env.DESKTOP_AGENT_OPS_PYTHON> ``` **Do NOT proceed if setup is not ready.** --- ## Core Execution Loop Every desktop task follows this loop. No exceptions. ``` 1. auto-setup gate ← run once per session 2. init task context ← create isolated temp directory 3. FOCUS the target app ← bring app to front, confirm frontmost 4. GET window bounds ← know exact position and size 5. CAPTURE that window ← screenshot ONLY the target window 6. ANALYZE the capture ← read screenshot or run OCR 7. LOCATE target via OCR ← find text/button within window bounds 8. VERIFY before acting ← move cursor, screenshot with cursor, confirm 9. EXECUTE one action ← click, type, scroll, press key 10. CAPTURE again ← screenshot to see result 11. VERIFY the result ← did the UI change as expected? 12. → if more steps, go to 5 13. CLEANUP ← remove task temp directory ``` **Key principles:** - One action at a time. Never chain blind actions. - Always verify after each action. If verification fails, recapture and retry — do NOT guess. - Always work within a specific window. Never click based on full-screen assumptions. --- ## Window-Scoped Targeting (THE CORRECT WAY) **NEVER do OCR or clicking on a full-screen screenshot.** Always scope to the target app window. ### The 6-Step Pipeline ``` ┌─────────────────────────────────────────────────────────┐ │ Step 1: FOCUS the target app │ │ $PY desktop_ops.py focus-app --name "AppName" │ │ → brings app to front │ ├─────────────────────────────────────────────────────────┤ │ Step 2: GET window bounds │ │ $PY desktop_ops.py front-window-bounds --app "AppName"│ │ → {x, y, width, height} in logical coordinates │ ├─────────────────────────────────────────────────────────┤ │ Step 3: CAPTURE only that window │ │ $PY desktop_ops.py capture-region --x X --y Y │ │ --width W --height H --output /tmp/window.png │ ├─────────────────────────────────────────────────────────┤ │ Step 4: OCR within the window │ │ $PY ocr_text.py --app "AppName" --python $PY │ │ → abs_box coordinates are INSIDE the window │ ├─────────────────────────────────────────────────────────┤ │ Step 5: VERIFY before clicking │ │ $PY desktop_ops.py move --x TX --y TY │ │ $PY desktop_ops.py screenshot --with-cursor │ │ → confirm cursor is on the right element │ ├─────────────────────────────────────────────────────────┤ │ Step 6: CLICK only if verified │ │ $PY desktop_ops.py click --x TX --y TY │ │ $PY desktop_ops.py screenshot → verify result │ └─────────────────────────────────────────────────────────┘ ``` ### Shortcut (RECOMMENDED for most targeting): ```bash $PY scripts/target_resolver.py --app "AppName" --text "按钮文字" --python $PY ``` This single command: focuses app → gets bounds → OCR within window → returns `best_candidate` with `{x, y, within_window}`. ### Why window-scoped matters: | Approach | Risk | |----------|------| | ❌ Full-screen OCR | "搜索" in WeChat AND Chrome → clicks wrong app | | ✅ Window-scoped | "搜索" ONLY in WeChat window → correct click | --- ## Failure Recovery (CRITICAL) When something fails, follow these rules: ### OCR finds nothing 1. Re-focus the app: `focus-app --name "AppName"` 2. Re-get bounds: `front-window-bounds --app "AppName"` (window may have moved/resized) 3. Take a fresh screenshot and read it visually 4. Try a different region label (e.g. `content_area` instead of `bottom_input`) 5. Try lowering OCR confidence: `--min-conf 30` ### Click doesn't work 1. Screenshot with cursor to check cursor position 2. The window may have moved — re-get bounds 3. Try clicking a few pixels offset from the OCR center 4. Check if a dialog/popup is blocking the target ### App state changed (login screen, dialog, etc.) 1. ALWAYS re-get window bounds after any major UI change 2. ALWAYS re-run OCR after navigation or state change 3. Never reuse old coordinates — they may be stale ### General retry rule - Maximum 3 retries per action - Each retry must recapture fresh state - If 3 retries fail, report the failure with screenshots and stop --- ## Generalization: How to Apply This to ANY App The pipeline works for any desktop application. Here is how to reason about new apps: ### Step-by-step for ANY new app: 1. **Identify the app name** exactly as it appears in the system (e.g. "Google Chrome", "微信", "System Settings") 2. **Focus and get bounds** — this tells you the window's exact position 3. **Screenshot the window** — look at what's on screen 4. **Identify the target** — what text, button, or area do you need to interact with? 5. **Use OCR to find it** — `target_resolver.py --app "AppName" --text "target text"` 6. **Verify and click** ### Common patterns across apps: | Task | How to do it | |------|-------------| | Click a button | OCR find text → verify → click | | Type in a field | OCR find field label → click field → `type --text` | | Search for something | OCR find search box → click → type query → press return | | Scroll a list | Get window bounds → scroll at window center with `--x --y` | | Switch between apps | `focus-app --name "OtherApp"` → re-get bounds | | Handle a dialog | Screenshot → OCR for dialog buttons → click appropriate one | | Navigate menus | Click menu item → wait → screenshot → OCR new menu → click | | Select from dropdown | Click dropdown → wait → OCR options → click selection | | Read screen content | OCR the window → extract all text boxes | | Verify an action | Screenshot before and after → compare or OCR for expected text | ### App-specific adaptations: | App type | Special considerations | |----------|----------------------| | Chat apps (WeChat, Slack, etc.) | Verify conversation title before typing; use `insert-newline` for multi-line; verify send mechanism | | Browsers (Chrome, Safari, etc.) | Address bar at top; content area varies; may need to handle tabs | | System Settings | Deep navigation; panels change; re-get bounds after each navigation | | File managers (Finder, Explorer) | Sidebar + content area; double-click to open; path bar for navigation | | Editors (VS Code, TextEdit, etc.) | Tab bar + editor area; use hotkeys for save/undo; type in editor area | --- ## Text Input and Send Rules ### Typing text ```bash $PY scripts/desktop_ops.py type --text "your message" ``` - Uses **clipboard paste** as primary method on all platforms — reliable for all languages including CJK - macOS: `set the clipboard to` + `Cmd+V` (single osascript call) - Windows: PowerShell `Set-Clipboard` + `Ctrl+V` (falls back to `clip.exe`) - Linux: `xclip` + `Ctrl+V` - First click on the input field to focus it before typing ### Multi-line messages ```bash $PY scripts/desktop_ops.py type --text "first line" $PY scripts/desktop_ops.py insert-newline $PY scripts/desktop_ops.py type --text "second line" ``` - Use `insert-newline` for literal line breaks - Do NOT use `\n` in `type --text` — it may trigger send in some apps ### Sending a message 1. **Preferred**: Look for a visible send button (e.g., `发送`) via OCR, then click it 2. **Alternative**: Use `press --key return` ONLY when the app is verified to use Enter-to-send 3. **Never guess** which send method to use — verify first ### Backend priority (macOS) | Operation | Primary | Fallback | |-----------|---------|----------| | `type` | Clipboard paste | cliclick (ASCII only) | | `press` | AppleScript `key code` | cliclick `kp:` | | `hotkey` | cliclick `kd:/t:/ku:` | pyautogui | | `click` | cliclick | pyautogui | > **Important**: cliclick `kp:return` is NOT recognized by WeChat — always use AppleScript for key press. > **Important**: cliclick `t:` silently drops CJK characters — always use clipboard paste for text input. --- ## DPI / HiDPI / Retina (All Platforms) **Handled automatically.** No manual DPI work needed. | Platform | Common scales | Detection method | |----------|---------------|-----------------| | macOS Retina | 2.0x | screenshot pixels ÷ logical screen bounds | | Windows HiDPI | 1.25x, 1.5x, 2.0x | screenshot pixels ÷ pyautogui.size() | | Linux X11 | 1.0x, 1.5x, 2.0x | screenshot pixels ÷ pyautogui.size() | OCR output: `box` = logical (use for mouse), `pixel_box` = raw pixels, `dpi_scale` = factor. --- ## CLI Quick Reference (EXACT parameter names) **CRITICAL**: Use EXACTLY these names. Do NOT guess. ### desktop_ops.py ```bash $PY scripts/desktop_ops.py screenshot [--output PATH] [--x X --y Y --width W --height H] [--with-cursor] $PY scripts/desktop_ops.py capture-region --x X --y Y --width W --height H [--output PATH] [--with-cursor] $PY scripts/desktop_ops.py frontmost $PY scripts/desktop_ops.py list-apps $PY scripts/desktop_ops.py front-window-bounds [--app NAME] $PY scripts/desktop_ops.py focus-app --name "App Name" $PY scripts/desktop_ops.py move --x X --y Y [--duration SECONDS] $PY scripts/desktop_ops.py click [--x X --y Y] [--button left|right|middle] $PY scripts/desktop_ops.py double-click [--x X --y Y] [--button left|right|middle] $PY scripts/desktop_ops.py drag --x1 X1 --y1 Y1 --x2 X2 --y2 Y2 [--duration SEC] [--button left] $PY scripts/desktop_ops.py scroll --amount N [--x X --y Y] [--direction vertical|horizontal] $PY scripts/desktop_ops.py mouse-position $PY scripts/desktop_ops.py press --key KEY $PY scripts/desktop_ops.py type --text "text to type" $PY scripts/desktop_ops.py insert-newline [--count N] $PY scripts/desktop_ops.py hotkey --keys cmd c $PY scripts/desktop_ops.py screen-size $PY scripts/desktop_ops.py pixel-color --x X --y Y ``` ### ocr_text.py ```bash $PY scripts/ocr_text.py --app "AppName" --python $PY [--region-label LABEL] [--lang auto] $PY scripts/ocr_text.py --image /path/to/capture.png --python $PY [--lang auto] ``` ### target_resolver.py ```bash $PY scripts/target_resolver.py --app "AppName" --text "text" --python $PY $PY scripts/target_resolver.py --app "AppName" --template /path/icon.png --python $PY $PY scripts/target_resolver.py --app "AppName" --text "text" --region-label LABEL --python $PY ``` ### task_context.py / cleanup_task.py ```bash $PY scripts/task_context.py init --task-id "my-task" # aliases: create, --name $PY scripts/task_context.py show --task-id "my-task" $PY scripts/cleanup_task.py --task-id "my-task" ``` ### window_regions.py ```bash $PY scripts/window_regions.py --window-x X --window-y Y --window-width W --window-height H [--label LABEL] ``` Labels: `top_search`, `left_sidebar`, `left_sidebar_top`, `title_header`, `content_area`, `toolbar_row`, `bottom_input`, `primary_action` --- ## Workflow Examples ### Example 1: Click a button by text (any app) ``` 1. $PY first_run_setup.py --check → ready: true 2. $PY task_context.py init --task-id "click-button" 3. $PY desktop_ops.py focus-app --name "AppName" 4. $PY desktop_ops.py front-window-bounds --app "AppName" → {x, y, w, h} 5. $PY target_resolver.py --app "AppName" --text "OK" --python $PY → best_candidate: {x:450, y:520, within_window:true} 6. $PY desktop_ops.py move --x 450 --y 520 7. $PY desktop_ops.py screenshot --with-cursor → verify cursor on "OK" 8. $PY desktop_ops.py click --x 450 --y 520 9. $PY desktop_ops.py screenshot → verify result 10. $PY cleanup_task.py --task-id "click-button" ``` ### Example 2: Type and search ``` 1. $PY desktop_ops.py focus-app --name "Safari" 2. $PY target_resolver.py --app "Safari" --text "Search" --region-label top_search --python $PY → {x:300, y:80, within_window:true} 3. $PY desktop_ops.py click --x 300 --y 80 4. $PY desktop_ops.py type --text "hello world" 5. $PY desktop_ops.py press --key return 6. $PY desktop_ops.py screenshot → verify search results ``` ### Example 3: Send a chat message (WeChat, Slack, etc.) ``` 1. $PY desktop_ops.py focus-app --name "WeChat" 2. $PY desktop_ops.py front-window-bounds --app "WeChat" 3. # Navigate to the right conversation (OCR sidebar or search) 4. $PY target_resolver.py --app "WeChat" --text "ContactName" --region-label left_sidebar --python $PY 5. $PY desktop_ops.py click --x <found_x> --y <found_y> 6. # Verify conversation is open 7. $PY desktop_ops.py screenshot → confirm conversation title 8. # Click the input field 9. $PY target_resolver.py --app "WeChat" --text "" --region-label bottom_input --python $PY OR: click at the bottom center of the window 10. $PY desktop_ops.py type --text "Hello!" 11. # Send: prefer visible send button; if not available, use press --key return 12. $PY target_resolver.py --app "WeChat" --text "发送" --python $PY IF found: $PY desktop_ops.py click --x <x> --y <y> ELSE: $PY desktop_ops.py press --key return 13. $PY desktop_ops.py screenshot → verify message sent ``` ### Example 4: Scroll a list and find an item ``` 1. $PY desktop_ops.py focus-app --name "AppName" 2. $PY desktop_ops.py front-window-bounds --app "AppName" → {x:100, y:50, w:800, h:600} 3. # Scroll down in the window center $PY desktop_ops.py scroll --amount -5 --x 500 --y 350 4. $PY desktop_ops.py screenshot → check if target visible 5. $PY target_resolver.py --app "AppName" --text "target item" --python $PY 6. IF not found: scroll more and retry (max 5 scrolls) 7. IF found: click it ``` ### Example 5: Handle an unexpected dialog ``` 1. # During any operation, if the expected UI doesn't match: 2. $PY desktop_ops.py screenshot → examine what's on screen 3. # If a dialog is visible, OCR it: $PY ocr_text.py --app "AppName" --python $PY 4. # Find and click the appropriate button (OK, Cancel, Allow, etc.) $PY target_resolver.py --app "AppName" --text "OK" --python $PY 5. $PY desktop_ops.py click --x <x> --y <y> 6. # After dialog is dismissed, re-get window bounds and continue $PY desktop_ops.py front-window-bounds --app "AppName" ``` --- ## Reference Documents Load as needed: | Document | When to read | |----------|-------------| | `references/workflow.md` | Core 8-step closed loop | | `references/platform-macos.md` | macOS-specific tools and permissions | | `references/platform-windows.md` | Windows setup | | `references/platform-linux.md` | Linux X11/Wayland setup | | `references/operation-patterns.md` | Reusable task templates | | `references/validation-patterns.md` | Two-stage validation | | `references/precise-targeting.md` | 5-layer precision targeting | | `references/target-providers.md` | Provider ordering and fallback contract | | `references/coordinate-reconstruction.md` | Rebuild click coordinates from screenshot evidence | | `references/chat-app-macos.md` | Chat app workflow | | `references/app-wechat-desktop.md` | Cross-platform WeChat guidance | | `references/cleanup-rules.md` | Cleanup timing and scope | | `references/collaboration-rules.md` | When multi-agent collaboration is justified | | `references/example-cases.md` | Repeatable task examples | | `references/reproducible-setup.md` | Host bring-up checklist | ## Scope Use this skill for: chat apps, browsers, file managers, editors, office apps, system settings, any closed desktop software with no usable API. ## Hard Rules 1. **Always run auto-setup gate first** 2. **Always use EXACT parameter names from CLI reference — never guess** 3. **Always scope OCR to the target app window — NEVER full-screen OCR** 4. **Always: focus-app → front-window-bounds → OCR within window → verify → act** 5. **Always pass `--python $PY` to ocr_text.py and target_resolver.py** 6. **Always verify coordinates are within window bounds before clicking** 7. **Always re-get window bounds after any UI state change (login, dialog, navigation)** 8. **Use `insert-newline` for line breaks; never use `\n` in `type --text`** 9. **For send actions: prefer visible send button; use `press --key return` only when verified** 10. **One action at a time; verify after each** 11. **Maximum 3 retries per action; each retry must recapture fresh state** 12. **Cleanup is mandatory at task end** 13. **If verification fails, recapture and rebuild — do not retry blindly**

path: agents/openai.yaml
content: interface:
  display_name: "Desktop Agent Ops"
  short_description: "Cross-platform desktop GUI automation for AI agents"
  default_prompt: "Use $desktop-agent-ops to operate a desktop app safely with window-scoped OCR, verified targeting, and step-by-step desktop actions."

policy:
  allow_implicit_invocation: true


path: references/app-wechat-desktop.md
content: # WeChat Desktop

Use this reference when the task is specifically about WeChat Desktop on macOS or Windows.

## Core send decision rule

Treat sending as a decision ladder, not a hardcoded single action:

1. verify the correct conversation is open
2. verify the draft text is visible in the true composer
3. check whether a visible `发送` send button is available
4. if the send button is visible and verified, click the send button
5. if there is no verified send button, and the host is confirmed to be direct-Enter-to-send, run `desktop_ops.py press --key return`
6. if neither path is verified, do not send yet

This is the required WeChat workflow for this skill, and it should also be generalized to similar desktop chat apps.

## Message composition rule

- use `desktop_ops.py type --text` for ordinary text input
- use `desktop_ops.py insert-newline` for literal line breaks inside the draft
- do not use `type --text` with `\n` as a send surrogate
- keep literal line breaks and send actions as separate operations

## Platform guidance

### Windows WeChat

Windows WeChat commonly exposes a visible `发送` button.

- prefer the verified `发送` button as the primary send action
- only fall back to `press --key return` when no send button is available and Enter-to-send is already confirmed for the host

### macOS WeChat

macOS WeChat may rely more often on direct-Enter-to-send.

- if a verified `发送` button is available, it is still safe to prefer that explicit UI action
- if no verified send button is available, use `desktop_ops.py press --key return` only after the draft is visibly in the true composer and Enter-to-send is known for the host

## Recommended workflow

1. focus WeChat
2. verify the correct conversation is active
3. click the true composer region
4. type message text with `desktop_ops.py type --text`
5. if the message needs a real line break, use `desktop_ops.py insert-newline`
6. verify the full draft is visible in the composer
7. look for a verified visible `发送` button
8. if found, click it
9. otherwise, if Enter-to-send is verified for this host, run `desktop_ops.py press --key return`
10. capture again and verify the outgoing message bubble appears

## Important caution

Do not guess the send action.

- visible send button beats inferred key behavior
- verified Enter-to-send beats unverified assumptions
- if neither path is verified, stop and recapture before sending

## Generalization rule

Apply the same reasoning to similar chat apps:

- first prefer the app's explicit visible send affordance
- if that affordance is absent, use the verified send key path for that host and app
- if the message needs multiple lines, use `insert-newline` and reserve the final send action for the verified send step


path: references/app-wechat-macos.md
content: # WeChat on macOS

Use `app-wechat-desktop.md` as the primary WeChat reference.

This file remains only as a compatibility pointer for older reference lists.


path: references/app-wechat-windows.md
content: # WeChat on Windows

Use `app-wechat-desktop.md` as the primary WeChat reference.

This file remains only as a compatibility pointer for older reference lists.


path: references/chat-app-macos.md
content: # Chat App on macOS

Use this reference for desktop chat software on macOS, including but not limited to WeChat, QQ, Feishu desktop, Telegram Desktop, and Discord Desktop.

## Goal

Provide a reusable workflow for reading context, selecting the right conversation, composing text, sending, and verifying the result without relying on private APIs.

For chat apps, sending should follow a decision ladder instead of a guessed shortcut:

1. first check for the app's explicit visible send affordance
2. if that affordance is visible and verified, use it
3. if no explicit send affordance is available, use the verified send key path for that host and app
4. if neither send path is verified, do not send yet

## Standard chat workflow

1. focus target app
2. wait briefly if the window was occluded
3. obtain front-window bounds
4. derive semantic regions relative to the window:
   - top search region
   - left conversation list region
   - title/header region
   - message transcript region
   - bottom composer region
5. select target conversation
6. validate active conversation before typing
7. read visible context
8. draft a reply compatible with that context
9. click true composer region
10. verify typed text appears in the composer
11. choose the verified send action: explicit send button first, otherwise verified `desktop_ops.py press --key return`
12. wait briefly for UI commit
13. verify sent result

## Required validation before send

Before sending a message, confirm all of these:
- the app is the intended app
- **the left sidebar conversation row has been clicked** (not just the search box)
- the active conversation title matches the intended target
- the visible transcript context matches the intended conversation
- the input text is visible in the true text composer

If any one of these is not confirmed, do not send. If title cannot be confirmed (no OCR/title read), recapture sidebar + header and request user confirmation.

For the typical chat apps covered by this reference, prefer a verified visible send button when it exists. If no verified send button exists, use `desktop_ops.py press --key return` only when direct-Enter-to-send is verified for that host and app.

Do not try to send by appending a newline inside `type --text`. That path behaves like text insertion and may create a new line instead of sending.

When a message really needs a line break, use `desktop_ops.py insert-newline` so the app receives inserted text instead of a send key event.

## Region guidance

### Conversation list region
Use window-relative coordinates and click within the interior of the row.
Do not click borders or row separators.

### Title/header region
Use this region to confirm the active conversation after selecting a row.

### Composer region
Treat the toolbar/attachment row and the true text input area as separate targets.
Typing is valid only when the text is visible in the true text input area.

## Send verification

For chat apps, a send key event alone is not sufficient proof.
Always verify via a post-send capture.

For the apps covered here, do not hardcode a single send trigger. Check for a verified send button first, then fall back to a verified send key path.

Recommended timing:
- first verification capture after ~0.3-0.8 seconds
- second verification capture at about 1 second total if the first is ambiguous

## Public-safe examples

Safe neutral payloads:
- hello from desktop agent
- test message
- automation check

Do not hardcode private contacts, chat names, or sensitive messages into public skill references.


path: references/cleanup-rules.md
content: # Cleanup Rules

Cleanup is mandatory.

## Clean when

Run cleanup when a task is:

- completed
- failed and terminated
- cancelled
- clearly replaced by a new unrelated task

## Clean these items

At minimum clear:

- temporary screenshots
- region captures
- temporary OCR outputs if created later
- recent target / anchor state
- focus hints
- retry counters
- task-local metadata that should not leak into unrelated work

## Keep these items only if useful

You may keep lightweight audit artifacts only when they help future debugging, but do not let them behave as active task context.

Examples of safe retained items:

- a final result summary
- a final success screenshot if intentionally preserved
- a short log for debugging

## Context isolation rule

If a task is finished and a later request is unrelated, do not reuse:

- last screenshot
- last target name
- last focused app assumption
- last fallback path

Create a new task context instead.

## Suggested rule of thumb

If the previous task is done and enough time has passed that the new request is functionally unrelated, start fresh rather than trying to resume old context.

## Script usage

Use `scripts/cleanup_task.py` to perform actual cleanup and return a structured result.


path: references/collaboration-rules.md
content: # Collaboration Rules

Collaboration is optional and should be rare in the MVP.

## Default

The main agent should do the task itself.

## Escalate only when clearly helpful

Consider collaboration only when one or more of these are true:

- two or more apps need coordinated attention
- two or more windows need simultaneous tracking
- copy/compare/transfer work is easier split by role
- repeated single-agent attempts have already failed
- a subtask is well-bounded and can be delegated cleanly

## Control rule

Even in collaboration mode, keep the main agent as controller.

- the main agent defines the subtask
- the sub-agent returns observations or a bounded result
- the main agent decides the next step

## Avoid in MVP

Do not overbuild collaboration before the single-agent flow is stable.


path: references/coordinate-reconstruction.md
content: # Coordinate Reconstruction

Use this file when the agent must decide **where** to click from screenshot evidence instead of relying on fixed guesses.

## Goal

Reconstruct a reliable click target from:

- full-screen screenshots
- region screenshots
- screen size
- live mouse coordinates
- pixel samples at candidate points

## Principle

Do not trust a guessed absolute coordinate until you verify it.

Preferred loop:

1. get current screen size
2. capture full screen
3. identify a candidate UI region visually
4. narrow to a region capture if needed
5. estimate a target point inside the region
6. optionally sample the pixel color at that point
7. move to the target point
8. read back the real mouse position
9. click only if the pointer landed where expected
10. capture again and verify UI change

## Why this matters

Absolute coordinates alone are fragile because:

- the frontmost window can change
- window positions can move
- screen scaling can change perceived layout
- a control’s visible center may not be its true hit target

## Practical commands

Use helper commands like:

- `screen-size`
- `screenshot`
- `capture-region`
- `mouse-position`
- `pixel-color --x ... --y ...`
- `move --x ... --y ...`
- `click --x ... --y ...`

## Reconstruction strategy

### 1. Global locate

Use a full screenshot to find the rough area of the app and the target control.

### 2. Region refine

Crop a smaller region around the target and reason there instead of on the full screen.

### 3. Pick an anchor point

Choose a point such as:

- center of a button
- center of a row
- center of an icon
- slightly inward from a visible border

### 4. Validate with move

Move first, then read back `mouse-position`.
If the pointer did not land where expected, stop and reassess.

### 5. Validate with UI change

A successful click must be verified by a fresh screenshot, not assumed from the command return.

## Future upgrades

Later, add OCR, text anchors, or UI detection. For MVP, coordinate reconstruction should still work with screenshot + region crop + move/readback + pixel sampling.

When OCR is available, prefer text anchors for buttons, tabs, and titles:

- run OCR on a cropped region
- match text to a query
- derive the click point from the matched text box

This reduces reliance on layout percentages when labels are visible.


path: references/eval-scenarios.md
content: # Eval Scenarios

Use this file to keep the skill public, testable, and improvable.

## Rule

Only include public-safe scenarios here.

Do not include:
- private contacts
- real account data
- sensitive conversations
- political or otherwise sensitive payloads
- machine-specific secrets

## Why this matters

Competitor systems improve because they do not stop at raw actions.
They also define repeatable scenarios, known initial states, and pass/fail checks.

## Scenario design format

Each scenario should define:

1. name
2. platform
3. target app class
4. preconditions
5. initial-state instructions
6. allowed actions
7. success criteria
8. failure hints

## Recommended public starter scenarios for macOS

### Scenario A: Focus and inspect app
- focus a target app
- capture screenshot
- confirm frontmost app and front-window bounds

### Scenario B: Search then open result
- focus app
- target top search region
- click verified candidate point
- type known safe query
- verify results change
- open first intended result

### Scenario C: Type into bottom composer
- open a safe conversation or scratch chat
- derive bottom input region from front window
- test candidate points until cursor is confirmed in text field
- type public-safe text
- verify text appears in input box

### Scenario D: Send a safe test message
- precondition: a send path is verified for the app under test
- message text must be generic and non-sensitive
- verify text appears in input box first
- if a verified visible send control exists, use it
- otherwise, if direct-Enter-to-send is verified, run `desktop_ops.py press --key return`
- wait a short fraction of a second for UI commit
- verify message bubble appears in transcript
- if the first capture is inconclusive, wait until about 1 second total and capture again

If the scenario needs a multi-line draft before sending, insert the line break with `desktop_ops.py insert-newline` instead of using a send key.

## Learning scenes to embed

The skill should include learning scenes in increasing difficulty:

1. focus-only
2. bounds-only
3. region capture
4. candidate-point click
5. text entry
6. send action
7. recovery from popup/dialog
8. multi-step app workflow

## Public-safe macOS WeChat test

For a public skill, define the WeChat example generically:
- use a clearly safe contact in a test account
- use a neutral payload such as `hello from desktop agent`
- never hardcode private names or IDs into the skill itself

## Evaluation style borrowed from strong competitors

From benchmark-style systems, keep these ideas:
- reproducible initial state
- execution-based verification
- task-specific success checks
- explicit exclusions for tasks that require private accounts or unstable network state


path: references/operation-patterns.md
content: # Operation Patterns

Use these as repeatable task templates.

## Pattern: Open an app and start from known state

1. check frontmost app
2. if needed, focus target app
3. capture state
4. verify the expected app is visible
5. continue with the task

## Pattern: Find a person or object in an app

When the UI has a search box, prefer search over manual scrolling.

Before clicking a candidate row, keep the target app frontmost and use move -> mouse-position -> click -> recapture instead of blind clicking.

1. capture current state
2. if target is visibly present, click it
3. otherwise locate the app search area
4. click search field
5. type target text
6. capture again
7. verify results appeared
8. click the intended result
9. capture again to verify the correct view opened

## Pattern: Send a message

1. confirm the correct conversation is open
2. if the app window can move, prefer window-relative region targeting instead of full-screen absolute points
3. derive the bottom input region from the front window when possible
4. generate 3-5 candidate points inside the input region and avoid the attachment/tool row above the text box
5. click one candidate, then recapture to verify the real text cursor landed in the text field
6. type message text
7. if a real line break is needed, run `desktop_ops.py insert-newline`
8. capture and verify the text is visible in the actual input field
9. check whether a verified visible send control exists
10. if it exists, use that explicit send control
11. otherwise, if direct-Enter-to-send is verified for the host and app, run `desktop_ops.py press --key return`
12. capture again and verify the message appears sent

For WeChat and similar chat apps, the preferred order is explicit visible send control first, verified send key second.

If the intended payload needs multiple lines, use `desktop_ops.py insert-newline` for the line break and reserve the final send action for the verified send step.

For instant-messaging apps like WeChat, do not re-capture immediately after pressing Enter. The UI may need a short fraction of a second to commit the outgoing message bubble. Standard rule: wait briefly after send (for example ~0.3-0.8s), then capture; if still ambiguous, wait until about 1 second total and capture again before declaring failure.

## Pattern: File selection flow

1. ensure file picker is frontmost
2. use search if available
3. otherwise navigate carefully one step at a time
4. capture after every meaningful navigation change
5. confirm intended file is selected
6. click open/confirm
7. capture again to verify picker closed or target app updated

## Pattern: Region-first precise interaction

Use this when the target is small or the screen is visually dense.

1. capture full screen or target app state
2. narrow to a region capture around the area of interest
3. reason over the smaller region
4. execute one action
5. recapture the region or full state and verify

## Pattern: Recovery after a failed click

1. do not keep clicking blindly
2. capture current state again
3. verify frontmost app is still correct
4. refocus if needed
5. reduce scope to a region capture
6. retry one carefully chosen step
7. if still unstable, escalate or pause

## Pattern: Right-click context menu

1. right-click target: `click --x X --y Y --button right`
2. wait 0.3s for menu to render
3. capture and OCR the context menu
4. click the target option (left-click)
5. capture and verify action result
6. if menu disappeared without action, press `escape` and retry

Context menus use absolute screen coordinates and disappear on any stray click — do not click outside the menu accidentally.

## Pattern: Drag and drop

1. identify source position (OCR or known region)
2. identify destination position
3. use `drag --x1 SX --y1 SY --x2 DX --y2 DY --duration 0.5`
4. longer `--duration` (0.5–1.0s) for apps that need sustained hold
5. capture and verify element moved
6. if drag failed, try longer duration or click-hold first

## Pattern: Multi-field form fill

1. capture form area
2. for each field: OCR label → click input → verify cursor → type value → verify
3. use `tab` key between fields when layout is vertical and sequential
4. after all fields: locate submit button → click → verify result
5. if validation error appears, OCR the error message and correct the field

## Pattern: Dropdown selection

1. click dropdown control to open
2. wait 0.2s for options to render
3. capture and OCR option list
4. click target option
5. verify dropdown shows selected value
6. if option not visible, scroll within dropdown before clicking

## Pattern: Cross-app copy-paste

1. focus source app → select content → `hotkey --keys cmd c`
2. focus destination app → click target field → `hotkey --keys cmd v`
3. capture and verify pasted content
4. always re-focus and re-get bounds when switching apps

## Pattern: Tab switching (browser/editor)

1. use keyboard shortcuts: `hotkey --keys cmd t` (new tab), `hotkey --keys cmd w` (close tab)
2. switch tabs: `hotkey --keys cmd shift ]` (next) or `hotkey --keys cmd 1` (specific tab)
3. after switching, capture and verify tab title/content via OCR
4. tab operations do not change window bounds — no need to re-get bounds

## Pattern: System settings navigation

1. focus settings app
2. prefer search if available: click search → type setting name → select result
3. if no search: OCR sidebar → click category → verify panel changed
4. re-get window bounds after deep navigation changes
5. interact with control (toggle, slider, input) → verify state changed


path: references/platform-linux.md
content: # Linux Path

This file defines the intended Linux branch for the skill.

## First branch

Always detect whether Linux is:

- X11
- Wayland

Use `scripts/platform_probe.py` before assuming available tools.

## Preferred direction

For Linux helper logic, tool families include:

- X11: `xdotool`, `wmctrl`, screenshot tools
- Wayland: compositor-specific screenshot and input tools, with stricter limits

If X11 tools are missing, report which dependency is absent and suggest installation.

## MVP rule

Linux helpers are now **best-effort** via `xdotool`/`wmctrl` for frontmost/list/focus/bounds. If tools are missing (or under Wayland), commands return structured errors. Be explicit about dependency requirements when failures occur.

## Behavior rule

Keep the skill workflow identical where possible; only the helper implementation should branch.


path: references/platform-macos.md
content: # macOS Path

## Preferred tools

For macOS, prefer these primitives:

- screenshot: `/usr/sbin/screencapture`
- app activation: `osascript` (AppleScript)
- mouse: `cliclick` (preferred) → `pyautogui` (fallback)
- text input: clipboard paste via `set the clipboard to` + `Cmd+V` (handles all languages including CJK)
- key press: AppleScript `key code` (preferred, most reliable for apps like WeChat) → `cliclick kp:` (fallback)
- hotkey: `cliclick kd:/t:/ku:` (modifier keys + character) → `pyautogui.hotkey` (fallback)
- focus-app: fast path if already frontmost; Dock click to restore minimized windows; `activate` + `AXRaise`

Use them through `scripts/desktop_ops.py`, not ad hoc each time.

For local setup, prefer a dedicated virtual environment outside the packaged skill directory and install runtime deps there before testing action commands. When running diagnostics, pass that interpreter through an environment variable such as `DESKTOP_AGENT_OPS_PYTHON`.

## Required permissions

macOS desktop automation may require:

- Accessibility
- Screen Recording
- Automation permissions for app control

If capture or input fails unexpectedly, suspect permissions first.

## Minimum macOS command set for MVP

Implement and use these first:

- `screenshot`
- `capture-region`
- `frontmost`
- `list-apps`
- `focus-app`
- `move`
- `click`
- `double-click`
- `drag`
- `scroll`
- `press`
- `type`
- `hotkey`
- `mouse-position`

## Recommended action order on macOS

## Timing notes for interactive apps

On macOS, app activation, focus transfer, and message-send UI updates may lag slightly behind the input event.

When working with chat apps such as WeChat:
- after `focus-app`, prefer a short settle wait before capturing if the window was previously occluded
- `type --text` always uses clipboard paste (reliable for CJK); cliclick `t:` silently drops non-ASCII characters
- `press --key return` uses AppleScript `key code 36` (not cliclick) — cliclick's `kp:return` is not recognized by WeChat and some other apps
- for WeChat and similar instant-messaging apps, first check for a verified visible send control; if none exists, use `desktop_ops.py press --key return` only when direct-Enter-to-send is verified for that host
- when a multi-line message is needed, use `desktop_ops.py insert-newline` for the line break and keep `press --key return` for the actual send event
- after a send trigger, wait a short moment (0.3–0.5s) before verification capture
- if the first verification capture is ambiguous, recapture once more at about 1 second total elapsed before treating it as a failure
- `focus-app` skips Dock traversal if app is already frontmost (fast path ~0.3s vs full ~1s)


### Bring an app forward

1. check frontmost app
2. if wrong, run `focus-app`
3. capture again
4. verify app switched

### Work inside a target UI

1. capture state
2. if target is unclear, capture a smaller region
3. perform one action
4. capture again
5. verify before continuing

## Notes

- `focus-window` may be added later, but `focus-app` is enough for MVP
- prefer explicit re-capture after activation because animation and focus changes can lag
- when a text search UI exists, use it instead of manual scanning/scrolling
ts/doctor.py`


path: references/platform-windows.md
content: # Windows Path

This file defines the intended Windows branch for the skill.

## Preferred direction

For Windows, prefer a helper-script path that wraps:

- `pygetwindow` (Win32 window enumeration/activation, restore minimized windows)
- screenshot tooling such as Pillow/MSS
- pyautogui for mouse and keyboard input
- text input: clipboard paste via PowerShell `Set-Clipboard` + `Ctrl+V` (handles all languages including CJK; falls back to `clip.exe` if PowerShell unavailable)
- key press / hotkey: `pyautogui.press()` / `pyautogui.hotkey()`

## MVP rule

Windows helpers are now **best-effort** via `pygetwindow` (frontmost, list windows, focus, bounds). If `pygetwindow` or its dependencies are missing, commands will return structured errors. Be explicit about dependency requirements when failures occur.

## Expected Windows command surface

Aim to match the macOS helper surface:

- screenshot
- capture-region
- frontmost
- list-apps or windows
- focus-app
- click / double-click / drag / scroll
- type / press / hotkey

## Behavior rule

Keep the top-level workflow the same across platforms; only the helper implementation should differ.

## Windows WeChat note

For WeChat on Windows, first check for the visible `发送` button and prefer it as the send trigger.

- type message text with `desktop_ops.py type --text`
- if a literal line break is needed, use `desktop_ops.py insert-newline`
- resolve the visible `发送` button with `target_resolver.py --text "发送"` and click it when verified
- only fall back to `press --key return` if no verified send button exists and Enter-to-send is already confirmed for that host


path: references/target-providers.md
content: # Target Providers

This file defines a small, explicit contract for different targeting strategies.

## Goal

Allow the system to choose among multiple targeting providers and fall back safely.

## Provider order

Default order for high confidence:

1. accessibility provider
2. OCR or text anchor provider
3. template or image match provider
4. heuristic region provider

## Provider contract

Each provider should output:

- target type
- confidence score
- bounding box (absolute)
- one or more candidate click points
- validation hints

## Implemented scripts

- `scripts/target_resolver.py` orchestrates provider selection and returns a ranked candidate.
- `scripts/ocr_text.py` extracts text boxes and coordinates from screenshots.
- `scripts/template_match.py` matches a template image within a region.

## Notes

- Accessibility is currently a stub and should be implemented per platform.
- OCR and template matching are optional and depend on external packages.
- Heuristic regions remain as the last fallback only.


path: scripts/cleanup_task.py
content: #!/usr/bin/env python3
import argparse
import json
import shutil

from task_paths import resolve_task_dir


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--task-id', required=True)
    args = ap.parse_args()
    target = resolve_task_dir(args.task_id)
    existed = target.exists()
    if existed:
        shutil.rmtree(target, ignore_errors=True)
    print(json.dumps({'ok': True, 'task_id': args.task_id, 'removed': existed}, ensure_ascii=False))


if __name__ == '__main__':
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)