Skill: chillybean/pipx-desktop-agent (version 1.3.2)

--- name: Desktop Control description: Control mouse, keyboard, and screen for desktop automation tasks --- # Desktop Control Skill This skill provides comprehensive desktop automation capabilities through PyAutoGUI, allowing AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment. ## How to Use This Skill As an AI agent, you can invoke desktop automation commands using the `uvx desktop-agent` CLI. ### Command Structure All commands follow this pattern: ```bash uvx desktop-agent <category> <command> [arguments] [options] ``` **Categories:** - `mouse` - Mouse control - `keyboard` - Keyboard input - `screen` - Screenshots and screen analysis - `message` - User dialogs - `app` - Application control (open, focus, list windows) ## Available Commands ### 🖱️ Mouse Control (`mouse`) Control cursor movement and clicks. ```bash # Move cursor to coordinates uvx desktop-agent mouse move <x> <y> [--duration SECONDS] # Click at current position or specific coordinates uvx desktop-agent mouse click [x] [y] [--button left|right|middle] [--clicks N] # Specialized clicks uvx desktop-agent mouse double-click [x] [y] uvx desktop-agent mouse right-click [x] [y] uvx desktop-agent mouse middle-click [x] [y] # Drag to coordinates uvx desktop-agent mouse drag <x> <y> [--duration SECONDS] [--button BUTTON] # Scroll (positive=up, negative=down) uvx desktop-agent mouse scroll <clicks> [x] [y] # Get current mouse position uvx desktop-agent mouse position ``` **Examples:** ```bash # Move to center of 1920x1080 screen uvx desktop-agent mouse move 960 540 --duration 0.5 # Right-click at specific location uvx desktop-agent mouse right-click 500 300 # Scroll down 5 clicks uvx desktop-agent mouse scroll -5 ``` ### ⌨️ Keyboard Control (`keyboard`) Type text and execute keyboard shortcuts. ```bash # Type text uvx desktop-agent keyboard write "<text>" [--interval SECONDS] # Press keys uvx desktop-agent keyboard press <key> [--presses N] [--interval SECONDS] # Execute hotkey combination (comma-separated) uvx desktop-agent keyboard hotkey "<key1>,<key2>,..." # Hold/release keys uvx desktop-agent keyboard keydown <key> uvx desktop-agent keyboard keyup <key> ``` **Examples:** ```bash # Type text with natural delay uvx desktop-agent keyboard write "Hello World" --interval 0.05 # Copy selected text uvx desktop-agent keyboard hotkey "ctrl,c" # Open Task Manager uvx desktop-agent keyboard hotkey "ctrl,shift,esc" # Press Enter 3 times uvx desktop-agent keyboard press enter --presses 3 ``` **Common Key Names:** - Modifiers: `ctrl`, `shift`, `alt`, `win` - Special: `enter`, `tab`, `esc`, `space`, `backspace`, `delete` - Function: `f1` through `f12` - Arrows: `up`, `down`, `left`, `right` ### 🖼️ Screen & Screenshots (`screen`) Capture screenshots and analyze screen content. Supports targeting specific windows. ```bash # Take screenshot uvx desktop-agent screen screenshot <filename> [--region "x,y,width,height"] [--window <title>] [--active] # Locate image on screen or within window uvx desktop-agent screen locate <image_path> [--confidence 0.0-1.0] [--window <title>] [--active] uvx desktop-agent screen locate-center <image_path> [--confidence 0.0-1.0] [--window <title>] [--active] # Locate text using OCR within window uvx desktop-agent screen locate-text-coordinates <text> [--window <title>] [--active] uvx desktop-agent screen read-all-text [--window <title>] [--active] # Utility commands uvx desktop-agent screen pixel <x> <y> uvx desktop-agent screen size uvx desktop-agent screen on-screen <x> <y> ``` **Examples:** ```bash # Screenshot of active window uvx desktop-agent screen screenshot active.png --active # Screenshot of a specific application uvx desktop-agent screen screenshot chrome.png --window "Google Chrome" # Locate image within Notepad uvx desktop-agent screen locate-center button.png --window "Notepad" ``` ### 💬 Message Dialogs (`message`) Display user interaction dialogs. ```bash # Show alert uvx desktop-agent message alert "<text>" [--title TITLE] [--button BUTTON] # Show confirmation dialog uvx desktop-agent message confirm "<text>" [--title TITLE] [--buttons "OK,Cancel"] # Prompt for input uvx desktop-agent message prompt "<text>" [--title TITLE] [--default TEXT] # Password input uvx desktop-agent message password "<text>" [--title TITLE] [--mask CHAR] ``` **Examples:** ```bash # Simple alert uvx desktop-agent message alert "Task completed!" # Get user confirmation uvx desktop-agent message confirm "Continue with operation?" # Ask for user input uvx desktop-agent message prompt "Enter your name:" ``` ### 📱 Application Control (`app`) Control applications across Windows, macOS, and Linux. ```bash # Open an application by name uvx desktop-agent app open <name> [--arg ARGS...] # Focus on a window by title/name uvx desktop-agent app focus <name> # List all visible windows uvx desktop-agent app list ``` **Examples:** ```bash # Windows: Open Notepad uvx desktop-agent app open notepad # Windows: Open Chrome with a URL uvx desktop-agent app open "chrome" --arg "https://google.com" # macOS: Open Safari uvx desktop-agent app open "Safari" # Focus on a specific window uvx desktop-agent app focus "Untitled - Notepad" # List all open windows uvx desktop-agent app list ``` ## Common Automation Workflows ### Workflow 1: Open Application and Type ```bash # Open notepad directly (cross-platform) uvx desktop-agent app open notepad # Wait for app to open, then focus it uvx desktop-agent app focus notepad # Type some text uvx desktop-agent keyboard write "Hello from Desktop Skill!" ``` ### Workflow 2: Screenshot + Analysis ```bash # Get screen size first uvx desktop-agent screen size # Take full screenshot uvx desktop-agent screen screenshot current_screen.png # Check if specific UI element is visible uvx desktop-agent screen locate save_button.png ``` ### Workflow 3: Form Filling ```bash # Click first field uvx desktop-agent mouse click 300 200 # Fill field uvx desktop-agent keyboard write "John Doe" # Tab to next field uvx desktop-agent keyboard press tab # Fill second field uvx desktop-agent keyboard write "[REDACTED_SECRET]" # Submit form (Enter) uvx desktop-agent keyboard press enter ``` ### Workflow 4: Copy/Paste Operations ```bash # Select all text uvx desktop-agent keyboard hotkey "ctrl,a" # Copy uvx desktop-agent keyboard hotkey "ctrl,c" # Click destination uvx desktop-agent mouse click 500 600 # Paste uvx desktop-agent keyboard hotkey "ctrl,v" ``` ## Safety Considerations When using this skill, AI agents should: 1. **Verify coordinates**: Use `screen size` and `on-screen` before clicking 2. **Add delays**: Insert appropriate delays between commands for UI responsiveness 3. **Validate images**: Ensure image files exist before using `locate` commands 4. **Handle failures**: Commands may fail if windows change or elements move 5. **User safety**: Always confirm destructive actions with user via `message confirm` ## Troubleshooting ### PyAutoGUI Fail-Safe PyAutoGUI has a fail-safe: moving mouse to screen corner aborts operations. This is a safety feature. ### Image not found When using `screen locate`, ensure: - Image file exists and path is correct - Adjust `--confidence` (try 0.7-0.9) - Image matches exact screen appearance (resolution, colors) ## Getting Help ```bash # Show all available commands uvx desktop-agent --help # Show commands for specific category uvx desktop-agent mouse --help uvx desktop-agent keyboard --help uvx desktop-agent screen --help uvx desktop-agent message --help # Show help for specific command uvx desktop-agent mouse move --help ``` ## Integration Tips for AI Agents 1. **Always check screen size first** when working with absolute coordinates 2. **Use relative positioning** when possible (e.g., get current position, calculate offset) 3. **Combine commands** for complex workflows 4. **Validate before executing** (e.g., check if image exists on screen) 5. **Provide user feedback** using message dialogs for important operations 6. **Handle errors gracefully** - commands may fail if UI state changes ## Performance Notes - Mouse movements with `--duration` are animated and take time - Image location (`locate`) can be slow on large screens - use regions when possible - Keyboard commands are generally fast (< 100ms) - Screenshots depend on screen resolution and region size ## Output Format All commands output structured JSON by default, ideal for programmatic use by AI agents: ```bash uvx desktop-agent mouse position # Output: {"success": true, "command": "mouse.position", "timestamp": "2026-01-31T10:00:00Z", "duration_ms": 5, "data": {"position": {"x": 960, "y": 540}}} ``` ### Response Schema All JSON responses follow this schema: ```json { "success": true, "command": "category.command", "timestamp": "2026-01-31T10:00:00Z", "duration_ms": 150, "data": { ... }, "error": null } ``` ### Error Response Schema ```json { "success": false, "command": "category.command", "timestamp": "2026-01-31T10:00:00Z", "duration_ms": 50, "data": null, "error": { "code": "image_not_found", "message": "Image file 'button.png' not found", "details": {}, "recoverable": true } } ``` ### Error Codes | Code | Description | |------|-------------| | `success` | Command succeeded | | `invalid_argument` | Invalid command arguments | | `coordinates_out_of_bounds` | Coordinates outside screen | | `image_not_found` | Image file not found or not on screen | | `window_not_found` | Target window not found | | `ocr_failed` | OCR operation failed | | `application_not_found` | Application not found | | `permission_denied` | Permission denied | | `platform_not_supported` | Platform not supported | | `timeout` | Operation timed out | | `unknown_error` | Unknown error | **Mouse move:** ```bash uvx desktop-agent mouse move 960 540 ``` ```json {"success": true, "command": "mouse.move", "timestamp": "...", "duration_ms": 150, "data": {"x": 960, "y": 540, "duration": 0}, "error": null} ``` **Screen size:** ```bash uvx desktop-agent screen size ``` ```json {"success": true, "command": "screen.size", "timestamp": "...", "duration_ms": 5, "data": {"size": {"width": 1920, "height": 1080}}, "error": null} ``` **Locate image:** ```bash uvx desktop-agent screen locate button.png ``` ```json {"success": true, "command": "screen.locate", "timestamp": "...", "duration_ms": 250, "data": {"image_found": true, "bounding_box": {"left": 100, "top": 200, "width": 50, "height": 30, "center_x": 125, "center_y": 215}}, "error": null} ``` **List windows:** ```bash uvx desktop-agent app list ``` ```json {"success": true, "command": "app.list", "timestamp": "...", "duration_ms": 100, "data": {"windows": ["Untitled - Notepad", "Google Chrome", "Visual Studio Code"]}, "error": null} ``` **Error example:** ```bash uvx desktop-agent screen locate missing.png ``` ```json {"success": false, "command": "screen.locate", "timestamp": "...", "duration_ms": 50, "data": null, "error": {"code": "image_not_found", "message": "Image file 'missing.png' not found", "details": {}, "recoverable": true}} ``` ## Effective Usage Guide for AI Agents This section teaches AI agents how to use this skill effectively with optimal command sequences and best practices. ### 🎯 Core Strategy: Observe First, Then Act **Always** understand the current state before performing actions. This avoids clicking wrong coordinates or typing in the wrong window. **Recommended Initial Sequence:** ```bash # 1. Get screen dimensions to understand your workspace uvx desktop-agent screen size uvx desktop-agent app list uvx desktop-agent mouse position ``` ### 📋 Recommended Command Sequences by Task #### Open and Interact with Application ```bash # ✅ CORRECT: Open, wait, verify, then interact uvx desktop-agent app open notepad # Step 1: Open app uvx desktop-agent app list uvx desktop-agent app focus "Notepad" uvx desktop-agent keyboard write "Hello World" # Step 4: Now safe to type # ❌ WRONG: Type immediately without verification uvx desktop-agent app open notepad uvx desktop-agent keyboard write "Hello World" # May type in wrong window! ``` #### Find and Click UI Element (Image-Based) ```bash # ✅ CORRECT: Locate first, click if found uvx desktop-agent screen locate-center button.png --confidence 0.8 # Check if success=true and coordinates are valid uvx desktop-agent mouse click 125 215 # Use returned coordinates # ❌ WRONG: Click without verifying element exists uvx desktop-agent mouse click 125 215 # Might click wrong area! ``` #### Find and Click UI Element (Text-Based with OCR) ```bash # ✅ CORRECT: Read screen text, then locate specific text uvx desktop-agent screen read-all-text --active uvx desktop-agent screen locate-text-coordinates "Save" --active # Use returned coordinates to click # For window-specific OCR: uvx desktop-agent screen locate-text-coordinates "OK" --window "Dialog Title" ``` #### Fill a Form with Multiple Fields ```bash # ✅ CORRECT: Click each field explicitly before typing uvx desktop-agent mouse click 300 200 # Click first field uvx desktop-agent keyboard write "John Doe" uvx desktop-agent mouse click 300 250 # Click second field (more reliable) uvx desktop-agent keyboard write "[REDACTED_SECRET]" uvx desktop-agent mouse click 300 300 # Click third field uvx desktop-agent keyboard write "555-1234" # OR use Tab navigation (less reliable if field order changes) uvx desktop-agent mouse click 300 200 uvx desktop-agent keyboard write "John Doe" uvx desktop-agent keyboard press tab uvx desktop-agent keyboard write "[REDACTED_SECRET]" uvx desktop-agent keyboard press tab uvx desktop-agent keyboard write "555-1234" uvx desktop-agent keyboard press enter # Submit ``` #### Take Targeted Screenshots for Analysis ```bash # ✅ CORRECT: Screenshot specific windows for faster processing uvx desktop-agent app list --json # Find exact window title uvx desktop-agent screen screenshot app.png --window "Google Chrome" # For active window only uvx desktop-agent screen screenshot active.png --active # Full screen only when necessary (slower, larger file) uvx desktop-agent screen size uvx desktop-agent screen screenshot full.png ``` #### Safe Drag and Drop ```bash # ✅ CORRECT: Move to start, verify position, then drag uvx desktop-agent mouse move 100 200 # Move to source uvx desktop-agent mouse position # Verify position uvx desktop-agent mouse drag 500 400 --duration 0.5 # Drag to destination # For precision, use slower duration uvx desktop-agent mouse drag 500 400 --duration 1.0 ``` ### 🔄 Error Recovery Patterns #### When Window Not Found ```bash # Pattern: List windows, find closest match, retry uvx desktop-agent app focus "Chrome" # Fails with window_not_found uvx desktop-agent app list # See actual window titles # Output shows: "Google Chrome - My Page" uvx desktop-agent app focus "Google Chrome" # Use correct title ``` #### When Image Not Found ```bash # Pattern: Adjust confidence or take new screenshot uvx desktop-agent screen locate button.png --confidence 0.9 uvx desktop-agent screen locate button.png --confidence 0.7 # If still failing, capture current state for analysis uvx desktop-agent screen screenshot current.png --active ``` #### When Click Seems to Miss ```bash # Pattern: Verify coordinates are on screen uvx desktop-agent screen size # Get screen bounds uvx desktop-agent screen on-screen 1500 900 # Check if coords are valid uvx desktop-agent mouse move 1500 900 # Move first to visualize uvx desktop-agent mouse click # Then click at current position ``` ### ⚡ Performance Optimization #### Minimize Screenshots ```bash # ✅ GOOD: Screenshot only the region you need uvx desktop-agent screen screenshot button_area.png --region "100,200,200,100" # ✅ GOOD: Screenshot specific window instead of full screen uvx desktop-agent screen screenshot chrome.png --window "Google Chrome" # ❌ SLOW: Full screen capture when you only need a small area uvx desktop-agent screen screenshot full.png ``` #### Batch Keyboard Input ```bash # ✅ FASTER: Write entire text at once uvx desktop-agent keyboard write "This is a complete sentence with all the text." # ❌ SLOWER: Multiple write commands uvx desktop-agent keyboard write "This is " uvx desktop-agent keyboard write "a complete " uvx desktop-agent keyboard write "sentence." ``` #### Use Hotkeys Over Mouse When Possible ```bash # ✅ FASTER: Use keyboard shortcuts uvx desktop-agent keyboard hotkey "ctrl,s" # Save uvx desktop-agent keyboard hotkey "ctrl,a" # Select all uvx desktop-agent keyboard hotkey "ctrl,shift,s" # Save as # ❌ SLOWER: Navigate menu with mouse uvx desktop-agent mouse click 50 30 # Click File menu uvx desktop-agent mouse click 60 80 # Click Save option ``` ### 🛡️ Defensive Programming Patterns #### Always Verify Critical Actions ```bash # Before destructive action, confirm with user uvx desktop-agent message confirm "This will delete all files. Continue?" --title "Warning" # Check output: if "Cancel" was clicked, abort operation ``` #### Use JSON Mode for Reliable Parsing ```bash # ✅ RELIABLE: Parse structured JSON output uvx desktop-agent screen locate button.png # Parse: {"success": true, "data": {"center_x": 125, "center_y": 215}} # ❌ FRAGILE: Parse text output uvx desktop-agent screen locate button.png # Parse: "Found at: Box(left=100, top=200, width=50, height=30)" ``` #### Validate Before Multi-Step Operations ```bash # Multi-step file operation with validation uvx desktop-agent app list uvx desktop-agent screen locate-text-coordinates "File" --active uvx desktop-agent mouse click <returned_x> <returned_y> uvx desktop-agent screen locate-text-coordinates "Save As" --active uvx desktop-agent mouse click <returned_x> <returned_y> ``` ### 🎮 Platform-Specific Considerations #### Windows ```bash # Common Windows shortcuts uvx desktop-agent keyboard hotkey "win,d" # Show desktop uvx desktop-agent keyboard hotkey "win,e" # Open Explorer uvx desktop-agent keyboard hotkey "alt,tab" # Switch windows uvx desktop-agent keyboard hotkey "win,r" # Run dialog # Open apps by name uvx desktop-agent app open notepad uvx desktop-agent app open calc uvx desktop-agent app open mspaint ``` #### macOS ```bash # Common macOS shortcuts (use 'command' for Cmd key) uvx desktop-agent keyboard hotkey "command,space" # Spotlight uvx desktop-agent keyboard hotkey "command,tab" # App switcher uvx desktop-agent keyboard hotkey "command,q" # Quit app uvx desktop-agent keyboard hotkey "command,shift,3" # Screenshot # Open apps uvx desktop-agent app open "Safari" uvx desktop-agent app open "TextEdit" ``` #### Linux ```bash # Open apps (uses xdg-open or direct command) uvx desktop-agent app open firefox uvx desktop-agent app open gedit # Common shortcuts may vary by DE uvx desktop-agent keyboard hotkey "alt,f2" # Run dialog (many DEs) ``` ### 📊 Decision Tree: Choosing the Right Command ``` Want to interact with an app? ├── App not running → `app open <name>` ├── App running but not focused → `app focus <name>` └── Need to verify windows → `app list` Want to find a UI element? ├── Have reference image → `screen locate-center <image>` ├── Know the text label → `screen locate-text-coordinates "<text>"` └── Need to see all text → `screen read-all-text --active` Want to click something? ├── Know exact coordinates → `mouse click <x> <y>` ├── Need to find first → Use locate commands above, then click returned coords └── Not sure if on screen → `screen on-screen <x> <y>` first Want to type something? ├── Regular text → `keyboard write "<text>"` ├── Keyboard shortcut → `keyboard hotkey "<key1>,<key2>"` ├── Single key press → `keyboard press <key>` └── Multiple of same key → `keyboard press <key> --presses N` ``` ## Integration Tips for AI Agents 1. **Always check screen size first** when working with absolute coordinates 2. **Use relative positioning** when possible (e.g., get current position, calculate offset) 3. **Combine commands** for complex workflows 4. **Validate before executing** (e.g., check if image exists on screen) 5. **Provide user feedback** using message dialogs for important operations 6. **Handle errors gracefully** - commands may fail if UI state changes

path: AGENTS.md
content: <!-- OPENSPEC:START -->
# OpenSpec Instructions

These instructions are for AI assistants working in this project.

Always open `@/openspec/AGENTS.md` when the request:
- Mentions planning or proposals (words like proposal, spec, change, plan)
- Introduces new capabilities, breaking changes, architecture shifts, or big performance/security work
- Sounds ambiguous and you need the authoritative spec before coding

Use `@/openspec/AGENTS.md` to learn:
- How to create and apply change proposals
- Spec format and conventions
- Project structure and guidelines

Keep this managed block so 'openspec update' can refresh the instructions.

<!-- OPENSPEC:END -->

path: README.md
content: # Desktop Control Skill

🤖 **AI Agent Skill** for desktop automation using PyAutoGUI.

Control mouse, keyboard, and screen programmatically through a simple CLI interface.

## ⚡ Quick Install

### For AI Agents

Add this skill to your AI coding agent with a single command:

```bash
npx skills add patrickporto/desktop-agent
```

### For Users

Install the CLI with pipx (recommended):

```bash
pipx install desktop-agent
```

Or run without installing using uvx:

```bash
uvx desktop-agent --help
```

Or using pip:

```bash
pip install desktop-agent
```

---

## 🎯 For AI Agents

This project is packaged as an **AI Agent Skill**. To use it:

1. **Read the skill documentation**: [SKILL.md](SKILL.md)
2. **Install the CLI**: Use `pip install desktop-agent` or `pipx install desktop-agent`
3. **Invoke commands**: Use `desktop-agent <category> <command>`

**Quick Reference for Agents:**
- All commands documented in [SKILL.md](SKILL.md)
- Practical examples in [examples/automation_examples.md](examples/automation_examples.md)
- Help system: `desktop-agent --help`
- All commands return JSON output by default

---

## 📦 Installation

### Using pipx (Recommended)

```bash
pipx install desktop-agent
```

### Using pip

```bash
pip install desktop-agent
```

### Run without installing (using uvx)

```bash
uvx desktop-agent
```


## Usage

The CLI is organized into command categories:

### 🖱️ Mouse (`mouse`)

```bash
# Move mouse to coordinates
desktop-agent mouse move 100 200

# Move with duration (animation)
desktop-agent mouse move 100 200 --duration 1.0

# Click at current position
desktop-agent mouse click

# Click at specific coordinates
desktop-agent mouse click 500 500

# Right click
desktop-agent mouse right-click

# Double click
desktop-agent mouse double-click 300 400

# Drag to coordinates
desktop-agent mouse drag 200 300

# Scroll (positive = up, negative = down)
desktop-agent mouse scroll 5
desktop-agent mouse scroll -3

# Get current mouse position
desktop-agent mouse position
```

### ⌨️ Keyboard (`keyboard`)

```bash
# Write text
desktop-agent keyboard write "Hello World"

# Write with interval between keys
desktop-agent keyboard write "Slow typing" --interval 0.1

# Press a key
desktop-agent keyboard press enter

# Press multiple times
desktop-agent keyboard press a --presses 5

# Execute keyboard shortcut
desktop-agent keyboard hotkey "ctrl,c"
desktop-agent keyboard hotkey "ctrl,shift,esc"

# Hold/release key
desktop-agent keyboard keydown shift
desktop-agent keyboard keyup shift
```

### 🖼️ Screen (`screen`)

```bash
# Capture screenshot (full screen)
desktop-agent screen screenshot my_screen.png

# Take screenshot of active window
desktop-agent screen screenshot active_window.png --active

# Take screenshot of specific window
desktop-agent screen screenshot notepad.png --window "Notepad"

# Screenshot of specific region (x,y,width,height)
desktop-agent screen screenshot region.png --region "100,100,500,400"

# Locate image within active window
desktop-agent screen locate button.png --active

# Locate center of image on screen
desktop-agent screen locate-center button.png --confidence 0.8

# Find text coordinates within active window
desktop-agent screen locate-text-coordinates "OK" --active

# Find text in specific image
desktop-agent screen locate-text-coordinates "Confirm" --image screenshot.png

# Case-sensitive search
desktop-agent screen locate-text-coordinates "Login" --case-sensitive

# Read all text from screen
desktop-agent screen read-all-text

# Read text from image
desktop-agent screen read-all-text --image capture.png

# Specify languages for OCR (default: pt,en)
desktop-agent screen locate-text-coordinates "Button" --lang "en"
```

### 💬 Messages (`message`)

```bash
# Show alert
desktop-agent message alert "Hello!"

# Confirmation
desktop-agent message confirm "Are you sure?"

# Input prompt
desktop-agent message prompt "Enter your name:"

# Password
desktop-agent message password "Enter your password:"
```

### 📱 Applications (`app`)

```bash
# Open an application (cross-platform)
desktop-agent app open notepad
desktop-agent app open "Google Chrome"

# Open with arguments
desktop-agent app open chrome --arg "https://google.com"

# Focus on a window by title
desktop-agent app focus "Untitled - Notepad"

# List all visible windows
desktop-agent app list
```

## Automation Examples

### Open Notepad and write

```bash
desktop-agent app open notepad
desktop-agent app focus notepad
desktop-agent keyboard write "Hello from Desktop Skill!"
```

### Capture screenshot and analyze

```bash
desktop-agent screen screenshot full_screen.png
desktop-agent screen pixel 500 500
```

## Available Commands

Run `desktop-agent --help` to see all commands:

```bash
desktop-agent --help
desktop-agent mouse --help
desktop-agent keyboard --help
desktop-agent screen --help
desktop-agent message --help
```

## Project Structure

```
desktop-skill/
├── desktop_agent/       # Main package
│   ├── __init__.py
│   ├── commands/        # Command modules
│   │   ├── __init__.py
│   │   ├── mouse.py    # Mouse commands
│   │   ├── keyboard.py # Keyboard commands
│   │   ├── screen.py   # Screen/screenshot/OCR commands
│   │   └── message.py  # Message boxes
├── pyproject.toml      # Project configuration
└── README.md           # This documentation
```

## Technologies

- **PyAutoGUI**: GUI automation
- **EasyOCR**: Optical character recognition
- **Typer**: Modern CLI framework


path: commands/keyboard.py
content: """Keyboard control commands."""
import typer
import pyautogui

app = typer.Typer(help="Keyboard control commands")


@app.command()
def write(
    text: str = typer.Argument(..., help="Text to type"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between keystrokes"),
):
    """Type text with optional interval between keys."""
    pyautogui.write(text, interval=interval)
    typer.echo(f"Typed: {text}")


@app.command()
def press(
    keys: str = typer.Argument(..., help="Key(s) to press (comma-separated for sequence)"),
    presses: int = typer.Option(1, "--presses", "-p", help="Number of times to press"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between presses"),
):
    """Press one or more keys."""
    key_list = [k.strip() for k in keys.split(",")]
    for key in key_list:
        pyautogui.press(key, presses=presses, interval=interval)
    typer.echo(f"Pressed: {keys} ({presses}x)")


@app.command()
def hotkey(
    keys: str = typer.Argument(..., help="Keys for hotkey (comma-separated, e.g., 'ctrl,c')"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between key presses"),
):
    """Execute a hotkey combination."""
    key_list = [k.strip() for k in keys.split(",")]
    pyautogui.hotkey(*key_list, interval=interval)
    typer.echo(f"Executed hotkey: {' + '.join(key_list)}")


@app.command()
def keydown(
    key: str = typer.Argument(..., help="Key to hold down"),
):
    """Hold down a key."""
    pyautogui.keyDown(key)
    typer.echo(f"Key down: {key}")


@app.command()
def keyup(
    key: str = typer.Argument(..., help="Key to release"),
):
    """Release a held key."""
    pyautogui.keyUp(key)
    typer.echo(f"Key up: {key}")


path: commands/message.py
content: """Message box commands."""
import typer
import pyautogui

app = typer.Typer(help="Message box commands")


@app.command()
def alert(
    text: str = typer.Argument(..., help="Alert message"),
    title: str = typer.Option("Alert", "--title", "-t", help="Window title"),
    button: str = typer.Option("OK", "--button", "-b", help="Button text"),
):
    """Display an alert message box."""
    result = pyautogui.alert(text=text, title=title, button=button)
    typer.echo(f"Alert shown: {result}")


@app.command()
def confirm(
    text: str = typer.Argument(..., help="Confirmation message"),
    title: str = typer.Option("Confirm", "--title", "-t", help="Window title"),
    buttons: str = typer.Option("OK,Cancel", "--buttons", "-b", help="Button texts (comma-separated)"),
):
    """Display a confirmation dialog."""
    button_list = [b.strip() for b in buttons.split(",")]
    result = pyautogui.confirm(text=text, title=title, buttons=button_list)
    typer.echo(f"User selected: {result}")


@app.command()
def prompt(
    text: str = typer.Argument(..., help="Prompt message"),
    title: str = typer.Option("Input", "--title", "-t", help="Window title"),
    default: str = typer.Option("", "--default", "-d", help="Default value"),
):
    """Display a prompt dialog for text input."""
    result = pyautogui.prompt(text=text, title=title, default=default)
    if result is not None:
        typer.echo(f"User entered: {result}")
    else:
        typer.echo("User cancelled")


@app.command()
def password(
    text: str = typer.Argument(..., help="Password prompt message"),
    title: str = typer.Option("Password", "--title", "-t", help="Window title"),
    default: str = typer.Option("", "--default", "-d", help="Default value"),
    mask: str = typer.Option("*", "--mask", "-m", help="Mask character"),
):
    """Display a password input dialog."""
    result = pyautogui.password(text=text, title=title, default=default, mask=mask)
    if result is not None:
        typer.echo(f"Password entered (length: {len(result)})")
    else:
        typer.echo("User cancelled")


path: commands/mouse.py
content: """Mouse control commands."""
import typer
import pyautogui

app = typer.Typer(help="Mouse control commands")


@app.command()
def move(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
    duration: float = typer.Option(0.0, "--duration", "-d", help="Duration in seconds"),
):
    """Move mouse to specified coordinates."""
    pyautogui.moveTo(x, y, duration=duration)
    typer.echo(f"Mouse moved to ({x}, {y})")


@app.command()
def click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
    button: str = typer.Option("left", "--button", "-b", help="Mouse button: left, right, middle"),
    clicks: int = typer.Option(1, "--clicks", "-c", help="Number of clicks"),
):
    """Click at current position or specified coordinates."""
    if x is not None and y is not None:
        pyautogui.click(x, y, clicks=clicks, button=button)
        typer.echo(f"{button.capitalize()} clicked {clicks}x at ({x}, {y})")
    else:
        pyautogui.click(clicks=clicks, button=button)
        typer.echo(f"{button.capitalize()} clicked {clicks}x at current position")


@app.command()
def double_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Double click at current position or specified coordinates."""
    if x is not None and y is not None:
        pyautogui.doubleClick(x, y)
        typer.echo(f"Double clicked at ({x}, {y})")
    else:
        pyautogui.doubleClick()
        typer.echo("Double clicked at current position")


@app.command()
def right_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Right click at current position or specified coordinates."""
    if x is not None and y is not None:
        pyautogui.rightClick(x, y)
        typer.echo(f"Right clicked at ({x}, {y})")
    else:
        pyautogui.rightClick()
        typer.echo("Right clicked at current position")


@app.command()
def middle_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Middle click at current position or specified coordinates."""
    if x is not None and y is not None:
        pyautogui.middleClick(x, y)
        typer.echo(f"Middle clicked at ({x}, {y})")
    else:
        pyautogui.middleClick()
        typer.echo("Middle clicked at current position")


@app.command()
def drag(
    x: int = typer.Argument(..., help="Target X coordinate"),
    y: int = typer.Argument(..., help="Target Y coordinate"),
    duration: float = typer.Option(0.0, "--duration", "-d", help="Duration in seconds"),
    button: str = typer.Option("left", "--button", "-b", help="Mouse button: left, right, middle"),
):
    """Drag mouse to specified coordinates."""
    pyautogui.drag(x, y, duration=duration, button=button)
    typer.echo(f"Dragged to ({x}, {y}) with {button} button")


@app.command()
def scroll(
    clicks: int = typer.Argument(..., help="Number of scroll clicks (negative for down)"),
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Scroll at current position or specified coordinates."""
    if x is not None and y is not None:
        pyautogui.scroll(clicks, x, y)
        typer.echo(f"Scrolled {clicks} clicks at ({x}, {y})")
    else:
        pyautogui.scroll(clicks)
        typer.echo(f"Scrolled {clicks} clicks at current position")


@app.command()
def position():
    """Get current mouse position."""
    pos = pyautogui.position()
    typer.echo(f"Mouse position: ({pos.x}, {pos.y})")


path: commands/screen.py
content: """Screen and screenshot commands."""
import typer
import pyautogui
from pathlib import Path

app = typer.Typer(help="Screen and screenshot commands")


@app.command()
def screenshot(
    filename: str = typer.Argument("screenshot.png", help="Output filename"),
    region: str = typer.Option(None, "--region", "-r", help="Region as 'x,y,width,height'"),
):
    """Take a screenshot of the entire screen or a region."""
    if region:
        try:
            x, y, width, height = map(int, region.split(","))
            img = pyautogui.screenshot(region=(x, y, width, height))
            img.save(filename)
            typer.echo(f"Screenshot saved to {filename} (region: {region})")
        except ValueError:
            typer.echo("Error: Region must be in format 'x,y,width,height'", err=True)
            raise typer.Exit(1)
    else:
        img = pyautogui.screenshot()
        img.save(filename)
        typer.echo(f"Screenshot saved to {filename}")


@app.command()
def locate(
    image: str = typer.Argument(..., help="Path to image to locate"),
    confidence: float = typer.Option(0.9, "--confidence", "-c", help="Match confidence (0.0-1.0)"),
):
    """Locate an image on the screen."""
    try:
        location = pyautogui.locateOnScreen(image, confidence=confidence)
        if location:
            typer.echo(f"Found at: x={location.left}, y={location.top}, width={location.width}, height={location.height}")
        else:
            typer.echo("Image not found on screen")
    except pyautogui.ImageNotFoundException:
        typer.echo("Image not found on screen")
    except Exception as e:
        typer.echo(f"Error: {e}", err=True)
        raise typer.Exit(1)


@app.command()
def locate_center(
    image: str = typer.Argument(..., help="Path to image to locate"),
    confidence: float = typer.Option(0.9, "--confidence", "-c", help="Match confidence (0.0-1.0)"),
):
    """Get the center coordinates of an image on the screen."""
    try:
        location = pyautogui.locateCenterOnScreen(image, confidence=confidence)
        if location:
            typer.echo(f"Center at: ({location.x}, {location.y})")
        else:
            typer.echo("Image not found on screen")
    except pyautogui.ImageNotFoundException:
        typer.echo("Image not found on screen")
    except Exception as e:
        typer.echo(f"Error: {e}", err=True)
        raise typer.Exit(1)


@app.command()
def pixel(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
):
    """Get the RGB color of a pixel at specified coordinates."""
    color = pyautogui.pixel(x, y)
    typer.echo(f"Pixel at ({x}, {y}): RGB{color}")


@app.command()
def size():
    """Get the screen size."""
    screen_size = pyautogui.size()
    typer.echo(f"Screen size: {screen_size.width}x{screen_size.height}")


@app.command()
def on_screen(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
):
    """Check if coordinates are on the screen."""
    is_on_screen = pyautogui.onScreen(x, y)
    if is_on_screen:
        typer.echo(f"({x}, {y}) is on screen")
    else:
        typer.echo(f"({x}, {y}) is NOT on screen")


path: desktop_agent/__init__.py
content: """Desktop Agent - Main entry point."""
import typer
from desktop_agent.commands import mouse, keyboard, screen, app as app_cmd, message

app = typer.Typer(help="Desktop automation agent")

app.add_typer(mouse.app, name="mouse")
app.add_typer(keyboard.app, name="keyboard")
app.add_typer(screen.app, name="screen")
app.add_typer(app_cmd.app, name="app")
app.add_typer(message.app, name="message")

def app_cli():
    app()

if __name__ == "__main__":
    app()


path: desktop_agent/commands/app.py
content: """Application control commands - cross-platform app launching and focusing."""
import platform
import subprocess
import time
import typer
from typing import Optional
from desktop_agent.utils import CommandResponse, ErrorCode, DesktopAgentError

app = typer.Typer(help="Application control commands")


def _get_platform() -> str:
    """Get the current platform."""
    system = platform.system().lower()
    if system == "darwin":
        return "macos"
    elif system == "windows":
        return "windows"
    else:
        return "linux"


def _handle_command(command: str, func, *args, **kwargs):
    start = time.time()
    try:
        result = func(*args, **kwargs)
        duration_ms = int((time.time() - start) * 1000)
        response = CommandResponse.success_response(
            command=command,
            data=result,
            duration_ms=duration_ms,
        )
        response.print()
    except Exception as e:
        duration_ms = int((time.time() - start) * 1000)
        error = DesktopAgentError(
            code=ErrorCode.from_exception(e),
            message=str(e),
        )
        response = CommandResponse.error_response(
            command=command,
            code=error.code.to_string(),
            message=error.message,
            details=error.details,
            duration_ms=duration_ms,
         )
        response.print()
        raise sys.exit(error.exit_code())


@app.command()
def open(
    name: str = typer.Argument(..., help="Application name or path to open"),
    args: Optional[list[str]] = typer.Option(None, "--arg", "-a", help="Arguments to pass to the application"),
):
    """Open an application by name or path."""
    def execute():
        current_platform = _get_platform()
        args_list = args or []

        if current_platform == "windows":
            if args_list:
                subprocess.Popen(
                    f'start "" "{name}" {" ".join(args_list)}',
                    shell=True,
                )
            else:
                subprocess.Popen(f'start "" "{name}"', shell=True)

        elif current_platform == "macos":
            cmd = ["open", "-a", name]
            if args_list:
                cmd.extend(["--args"] + args_list)
            subprocess.Popen(cmd)

        else:
            cmd = [name] + args_list
            subprocess.Popen(
                cmd,
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
            )

        return {"application": name, "args": args_list}
    _handle_command("app.open", execute)


@app.command()
def focus(name: str = typer.Argument(..., help="Window title or application name to focus")):
    """Focus on a window by title or application name."""
    def execute():
        current_platform = _get_platform()

        if current_platform == "windows":
            import ctypes
            from ctypes import wintypes

            user32 = ctypes.windll.user32

            EnumWindowsProc = ctypes.WINFUNCTYPE(
                ctypes.c_bool,
                wintypes.HWND,
                wintypes.LPARAM
            )

            found_hwnd = None

            def enum_callback(hwnd, lparam):
                nonlocal found_hwnd
                if user32.IsWindowVisible(hwnd):
                    length = user32.GetWindowTextLengthW(hwnd)
                    if length > 0:
                        buffer = ctypes.create_unicode_buffer(length + 1)
                        user32.GetWindowTextW(hwnd, buffer, length + 1)
                        title = buffer.value
                        if name.lower() in title.lower():
                            found_hwnd = hwnd
                            return False
                return True

            user32.EnumWindows(EnumWindowsProc(enum_callback), 0)

            if found_hwnd:
                SW_RESTORE = 9
                user32.ShowWindow(found_hwnd, SW_RESTORE)
                user32.SetForegroundWindow(found_hwnd)
                return {"window_title": name, "focused": True}
            else:
                raise DesktopAgentError(
                    code=ErrorCode.WINDOW_NOT_FOUND,
                    message=f"Window '{name}' not found",
                )

        elif current_platform == "macos":
            script = f'''
            tell application "{name}"
                activate
            end tell
            '''
            result = subprocess.run(
                ["osascript", "-e", script],
                capture_output=True,
                text=True,
            )
            if result.returncode == 0:
                return {"application": name, "focused": True}
            else:
                raise DesktopAgentError(
                    code=ErrorCode.WINDOW_NOT_FOUND,
                    message=f"Could not focus '{name}'",
                    details={"stderr": result.stderr},
                )

        else:
            try:
                result = subprocess.run(
                    ["wmctrl", "-a", name],
                    capture_output=True,
                    text=True,
                )
                if result.returncode == 0:
                    return {"window_title": name, "focused": True}
                else:
                    raise FileNotFoundError("wmctrl failed")
            except FileNotFoundError:
                result = subprocess.run(
                    ["xdotool", "search", "--name", name, "windowactivate"],
                    capture_output=True,
                    text=True,
                )
                if result.returncode == 0:
                    return {"window_title": name, "focused": True}
                else:
                    raise DesktopAgentError(
                        code=ErrorCode.WINDOW_NOT_FOUND,
                        message=f"Could not focus '{name}'. Install wmctrl or xdotool.",
                    )
    _handle_command("app.focus", execute)


@app.command()
def list():
    """List all visible windows."""
    def execute():
        current_platform = _get_platform()
        windows = []

        if current_platform == "windows":
            import ctypes
            from ctypes import wintypes

            user32 = ctypes.windll.user32

            EnumWindowsProc = ctypes.WINFUNCTYPE(
                ctypes.c_bool,
                wintypes.HWND,
                wintypes.LPARAM
            )

            def enum_callback(hwnd, lparam):
                if user32.IsWindowVisible(hwnd):
                    length = user32.GetWindowTextLengthW(hwnd)
                    if length > 0:
                        buffer = ctypes.create_unicode_buffer(length + 1)
                        user32.GetWindowTextW(hwnd, buffer, length + 1)
                        title = buffer.value
                        if title.strip():
                            windows.append(title)
                return True

            user32.EnumWindows(EnumWindowsProc(enum_callback), 0)

        elif current_platform == "macos":
            script = '''
            tell application "System Events"
                set windowList to {}
                repeat with proc in (every process whose background only is false)
                    repeat with win in (every window of proc)
                        set end of windowList to (name of proc) & " - " & (name of win)
                    end repeat
                end repeat
                return windowList
            end tell
            '''
            result = subprocess.run(
                ["osascript", "-e", script],
                capture_output=True,
                text=True,
            )
            if result.returncode == 0 and result.stdout.strip():
                output = result.stdout.strip()
                if output:
                    windows = [w.strip() for w in output.split(",")]

        else:
            result = subprocess.run(
                ["wmctrl", "-l"],
                capture_output=True,
                text=True,
            )
            if result.returncode == 0:
                for line in result.stdout.strip().split("\n"):
                    if line:
                        parts = line.split(None, 3)
                        if len(parts) >= 4:
                            windows.append(parts[3])
            else:
                raise DesktopAgentError(
                    code=ErrorCode.PLATFORM_NOT_SUPPORTED,
                    message="wmctrl not found. Install it with: sudo apt install wmctrl",
                )

        return {"windows": windows}
    _handle_command("app.list", execute)


import sys


path: desktop_agent/commands/keyboard.py
content: """Keyboard control commands."""
import time
import typer
import pyautogui
from desktop_agent.utils import CommandResponse, ErrorCode, DesktopAgentError

app = typer.Typer(help="Keyboard control commands")


def _handle_command(command: str, func, *args, **kwargs):
    start = time.time()
    try:
        result = func(*args, **kwargs)
        duration_ms = int((time.time() - start) * 1000)
        response = CommandResponse.success_response(
            command=command,
            data=result,
            duration_ms=duration_ms,
        )
        response.print()
    except Exception as e:
        duration_ms = int((time.time() - start) * 1000)
        error = DesktopAgentError(
            code=ErrorCode.from_exception(e),
            message=str(e),
        )
        response = CommandResponse.error_response(
            command=command,
            code=error.code.to_string(),
            message=error.message,
            details=error.details,
            duration_ms=duration_ms,
        )
        response.print()
        raise sys.exit(error.exit_code())


@app.command()
def write(
    text: str = typer.Argument(..., help="Text to type"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between keystrokes"),
):
    """Type text with optional interval between keys."""
    def execute():
        pyautogui.write(text, interval=interval)
        return {"text": text, "interval": interval}
    _handle_command("keyboard.write", execute)


@app.command()
def press(
    keys: str = typer.Argument(..., help="Key(s) to press (comma-separated for sequence)"),
    presses: int = typer.Option(1, "--presses", "-p", help="Number of times to press"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between presses"),
):
    """Press one or more keys."""
    def execute():
        key_list = [k.strip() for k in keys.split(",")]
        for key in key_list:
            pyautogui.press(key, presses=presses, interval=interval)
        return {"keys": key_list, "presses": presses, "interval": interval}
    _handle_command("keyboard.press", execute)


@app.command()
def hotkey(
    keys: str = typer.Argument(..., help="Keys for hotkey (comma-separated, e.g., 'ctrl,c')"),
    interval: float = typer.Option(0.0, "--interval", "-i", help="Interval between key presses"),
):
    """Execute a hotkey combination."""
    def execute():
        key_list = [k.strip() for k in keys.split(",")]
        pyautogui.hotkey(*key_list, interval=interval)
        return {"keys": key_list, "interval": interval}
    _handle_command("keyboard.hotkey", execute)


@app.command()
def keydown(key: str = typer.Argument(..., help="Key to hold down")):
    """Hold down a key."""
    def execute():
        pyautogui.keyDown(key)
        return {"key": key}
    _handle_command("keyboard.keydown", execute)


@app.command()
def keyup(key: str = typer.Argument(..., help="Key to release")):
    """Release a held key."""
    def execute():
        pyautogui.keyUp(key)
        return {"key": key}
    _handle_command("keyboard.keyup", execute)


import sys


path: desktop_agent/commands/message.py
content: """Message box commands."""
import time
import typer
import pyautogui
from desktop_agent.utils import CommandResponse, ErrorCode, DesktopAgentError

app = typer.Typer(help="Message box commands")


def _handle_command(command: str, func, *args, **kwargs):
    start = time.time()
    try:
        result = func(*args, **kwargs)
        duration_ms = int((time.time() - start) * 1000)
        response = CommandResponse.success_response(
            command=command,
            data=result,
            duration_ms=duration_ms,
        )
        response.print()
    except Exception as e:
        duration_ms = int((time.time() - start) * 1000)
        error = DesktopAgentError(
            code=ErrorCode.from_exception(e),
            message=str(e),
        )
        response = CommandResponse.error_response(
            command=command,
            code=error.code.to_string(),
            message=error.message,
            details=error.details,
            duration_ms=duration_ms,
        )
        response.print()
        raise sys.exit(error.exit_code())


@app.command()
def alert(
    text: str = typer.Argument(..., help="Alert message"),
    title: str = typer.Option("Alert", "--title", "-t", help="Window title"),
    button: str = typer.Option("OK", "--button", "-b", help="Button text"),
):
    """Display an alert message box."""
    def execute():
        result = pyautogui.alert(text=text, title=title, button=button)
        return {"button_pressed": result}
    _handle_command("message.alert", execute)


@app.command()
def confirm(
    text: str = typer.Argument(..., help="Confirmation message"),
    title: str = typer.Option("Confirm", "--title", "-t", help="Window title"),
    buttons: str = typer.Option("OK,Cancel", "--buttons", "-b", help="Button texts (comma-separated)"),
):
    """Display a confirmation dialog."""
    def execute():
        button_list = [b.strip() for b in buttons.split(",")]
        result = pyautogui.confirm(text=text, title=title, buttons=button_list)
        return {"button_pressed": result}
    _handle_command("message.confirm", execute)


@app.command()
def prompt(
    text: str = typer.Argument(..., help="Prompt message"),
    title: str = typer.Option("Input", "--title", "-t", help="Window title"),
    default: str = typer.Option("", "--default", "-d", help="Default value"),
):
    """Display a prompt dialog for text input."""
    def execute():
        result = pyautogui.prompt(text=text, title=title, default=default)
        if result is not None:
            return {"user_input": result, "cancelled": False}
        else:
            return {"user_input": None, "cancelled": True}
    _handle_command("message.prompt", execute)


@app.command()
def password(
    text: str = typer.Argument(..., help="Password prompt message"),
    title: str = typer.Option("Password", "--title", "-t", help="Window title"),
    default: str = typer.Option("", "--default", "-d", help="Default value"),
    mask: str = typer.Option("*", "--mask", "-m", help="Mask character"),
):
    """Display a password input dialog."""
    def execute():
        result = pyautogui.password(text=text, title=title, default=default, mask=mask)
        if result is not None:
            return {"entered": True, "length": len(result)}
        else:
            return {"entered": False, "length": 0}
    _handle_command("message.password", execute)


import sys


path: desktop_agent/commands/mouse.py
content: """Mouse control commands."""
import time
import typer
import pyautogui
from desktop_agent.utils import CommandResponse, ErrorCode, DesktopAgentError

app = typer.Typer(help="Mouse control commands")


def _handle_command(command: str, func, *args, **kwargs):
    start = time.time()
    try:
        result = func(*args, **kwargs)
        duration_ms = int((time.time() - start) * 1000)
        response = CommandResponse.success_response(
            command=command,
            data=result,
            duration_ms=duration_ms,
        )
        response.print()
    except Exception as e:
        duration_ms = int((time.time() - start) * 1000)
        error = DesktopAgentError(
            code=ErrorCode.from_exception(e),
            message=str(e),
        )
        response = CommandResponse.error_response(
            command=command,
            code=error.code.to_string(),
            message=error.message,
            details=error.details,
            duration_ms=duration_ms,
        )
        response.print()
        raise sys.exit(error.exit_code())


@app.command()
def move(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
    duration: float = typer.Option(0.0, "--duration", "-d", help="Duration in seconds"),
):
    """Move mouse to specified coordinates."""
    def execute():
        pyautogui.moveTo(x, y, duration=duration)
        return {"x": x, "y": y, "duration": duration}
    _handle_command("mouse.move", execute)


@app.command()
def click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
    button: str = typer.Option("left", "--button", "-b", help="Mouse button: left, right, middle"),
    clicks: int = typer.Option(1, "--clicks", "-c", help="Number of clicks"),
):
    """Click at current position or specified coordinates."""
    def execute():
        if x is not None and y is not None:
            pyautogui.click(x, y, clicks=clicks, button=button)
            return {"position": {"x": x, "y": y}, "button": button, "clicks": clicks}
        else:
            pyautogui.click(clicks=clicks, button=button)
            return {"position": None, "button": button, "clicks": clicks}
    _handle_command("mouse.click", execute)


@app.command()
def double_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Double click at current position or specified coordinates."""
    def execute():
        if x is not None and y is not None:
            pyautogui.doubleClick(x, y)
            return {"position": {"x": x, "y": y}}
        else:
            pyautogui.doubleClick()
            return {"position": None}
    _handle_command("mouse.double_click", execute)


@app.command()
def right_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Right click at current position or specified coordinates."""
    def execute():
        if x is not None and y is not None:
            pyautogui.rightClick(x, y)
            return {"position": {"x": x, "y": y}}
        else:
            pyautogui.rightClick()
            return {"position": None}
    _handle_command("mouse.right_click", execute)


@app.command()
def middle_click(
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Middle click at current position or specified coordinates."""
    def execute():
        if x is not None and y is not None:
            pyautogui.middleClick(x, y)
            return {"position": {"x": x, "y": y}}
        else:
            pyautogui.middleClick()
            return {"position": None}
    _handle_command("mouse.middle_click", execute)


@app.command()
def drag(
    x: int = typer.Argument(..., help="Target X coordinate"),
    y: int = typer.Argument(..., help="Target Y coordinate"),
    duration: float = typer.Option(0.0, "--duration", "-d", help="Duration in seconds"),
    button: str = typer.Option("left", "--button", "-b", help="Mouse button: left, right, middle"),
):
    """Drag mouse to specified coordinates."""
    def execute():
        pyautogui.drag(x, y, duration=duration, button=button)
        return {"x": x, "y": y, "duration": duration, "button": button}
    _handle_command("mouse.drag", execute)


@app.command()
def scroll(
    clicks: int = typer.Argument(..., help="Number of scroll clicks (negative for down)"),
    x: int = typer.Argument(None, help="X coordinate (optional)"),
    y: int = typer.Argument(None, help="Y coordinate (optional)"),
):
    """Scroll at current position or specified coordinates."""
    def execute():
        if x is not None and y is not None:
            pyautogui.scroll(clicks, x, y)
            return {"clicks": clicks, "position": {"x": x, "y": y}}
        else:
            pyautogui.scroll(clicks)
            return {"clicks": clicks, "position": None}
    _handle_command("mouse.scroll", execute)


@app.command()
def position():
    """Get current mouse position."""
    def execute():
        pos = pyautogui.position()
        return {"position": {"x": pos.x, "y": pos.y}}
    _handle_command("mouse.position", execute)


import sys


path: desktop_agent/commands/screen.py
content: """Screen and screenshot commands."""
import time
import typer
import pyautogui
from pathlib import Path
from typing import Optional
import json
import pywinctl
from desktop_agent.utils import CommandResponse, ErrorCode, DesktopAgentError

app = typer.Typer(help="Screen and screenshot commands")


def _get_window_region(window_name: Optional[str] = None, active: bool = False) -> Optional[tuple[int, int, int, int]]:
    """Get the region of a specific or active window using PyWinCtl."""
    try:
        if active:
            window = pywinctl.getActiveWindow()
            if not window:
                return None
        elif window_name:
            windows = pywinctl.getWindowsWithTitle(window_name)
            if not windows:
                return None
            window = windows[0]
        else:
            return None

        return (int(window.left), int(window.top), int(window.width), int(window.height))
    except Exception:
        return None


def _handle_command(command: str, func, *args, **kwargs):
    start = time.time()
    try:
        result = func(*args, **kwargs)
        duration_ms = int((time.time() - start) * 1000)
        response = CommandResponse.success_response(
            command=command,
            data=result,
            duration_ms=duration_ms,
        )
        response.print()
    except Exception as e:
        duration_ms = int((time.time() - start) * 1000)
        error = DesktopAgentError(
            code=ErrorCode.from_exception(e),
            message=str(e),
        )
        response = CommandResponse.error_response(
            command=command,
            code=error.code.to_string(),
            message=error.message,
            details=error.details,
            duration_ms=duration_ms,
        )
        response.print()
        raise sys.exit(error.exit_code())


@app.command()
def screenshot(
    filename: str = typer.Argument("screenshot.png", help="Output filename"),
    region: str = typer.Option(None, "--region", "-r", help="Region as 'x,y,width,height'"),
    window: Optional[str] = typer.Option(None, "--window", "-w", help="Target window title"),
    active: bool = typer.Option(False, "--active", "-a", help="Target active window"),
):
    """Take a screenshot of the entire screen, a window, or a specific region."""
    def execute():
        target_region = None

        window_region = _get_window_region(window, active)
        if window_region:
            target_region = window_region

        if region:
            try:
                target_region = tuple(map(int, region.split(",")))
            except ValueError:
                raise DesktopAgentError(
                    code=ErrorCode.INVALID_ARGUMENT,
                    message="Region must be in format 'x,y,width,height'",
                )

        if target_region:
            img = pyautogui.screenshot(region=target_region)
            img.save(filename)
            return {
                "filename": filename,
                "region": {
                    "x": target_region[0],
                    "y": target_region[1],
                    "width": target_region[2],
                    "height": target_region[3],
                },
            }
        else:
            img = pyautogui.screenshot()
            img.save(filename)
            return {"filename": filename, "region": None}
    _handle_command("screen.screenshot", execute)


@app.command()
def locate(
    image: str = typer.Argument(..., help="Path to image to locate"),
    confidence: float = typer.Option(0.9, "--confidence", "-c", help="Match confidence (0.0-1.0)"),
    window: Optional[str] = typer.Option(None, "--window", "-w", help="Search within a specific window"),
    active: bool = typer.Option(False, "--active", "-a", help="Search within the active window"),
):
    """Locate an image on the screen or within a targeted window."""
    def execute():
        region = _get_window_region(window, active)

        if not Path(image).exists():
            raise DesktopAgentError(
                code=ErrorCode.IMAGE_NOT_FOUND,
                message=f"Image file '{image}' not found",
            )

        location = pyautogui.locateOnScreen(image, confidence=confidence, region=region)
        if location:
            return {
                "image_found": True,
                "bounding_box": {
                    "left": location.left,
                    "top": location.top,
                    "width": location.width,
                    "height": location.height,
                    "center_x": location.left + location.width // 2,
                    "center_y": location.top + location.height // 2,
                },
            }
        else:
            return {"image_found": False}
    _handle_command("screen.locate", execute)


@app.command()
def locate_center(
    image: str = typer.Argument(..., help="Path to image to locate"),
    confidence: float = typer.Option(0.9, "--confidence", "-c", help="Match confidence (0.0-1.0)"),
    window: Optional[str] = typer.Option(None, "--window", "-w", help="Search within a specific window"),
    active: bool = typer.Option(False, "--active", "-a", help="Search within the active window"),
):
    """Get the center coordinates of an image on the screen or within a window."""
    def execute():
        region = _get_window_region(window, active)

        if not Path(image).exists():
            raise DesktopAgentError(
                code=ErrorCode.IMAGE_NOT_FOUND,
                message=f"Image file '{image}' not found",
            )

        location = pyautogui.locateCenterOnScreen(image, confidence=confidence, region=region)
        if location:
            return {"position": {"x": location.x, "y": location.y}}
        else:
            return {"image_found": False}
    _handle_command("screen.locate_center", execute)


@app.command()
def pixel(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
):
    """Get the RGB color of a pixel at specified coordinates."""
    def execute():
        color = pyautogui.pixel(x, y)
        return {
            "pixel": {
                "r": color[0],
                "g": color[1],
                "b": color[2],
                "hex": f"#{color[0]:02x}{color[1]:02x}{color[2]:02x}",
            }
        }
    _handle_command("screen.pixel", execute)


@app.command()
def size():
    """Get the screen size."""
    def execute():
        screen_size = pyautogui.size()
        return {
            "size": {
                "width": screen_size.width,
                "height": screen_size.height,
            }
        }
    _handle_command("screen.size", execute)


@app.command()
def on_screen(
    x: int = typer.Argument(..., help="X coordinate"),
    y: int = typer.Argument(..., help="Y coordinate"),
):
    """Check if coordinates are on the screen."""
    def execute():
        is_on_screen = pyautogui.onScreen(x, y)
        return {"on_screen": is_on_screen}
    _handle_command("screen.on_screen", execute)


_reader = None
_reader_langs = None


def _get_system_language() -> str:
    import locale
    try:
        system_locale = locale.getdefaultlocale()[0]
        if system_locale:
            lang_code = system_locale.split('_')[0].lower()
            return lang_code
    except Exception:
        pass
    return 'en'


def _get_default_languages() -> list[str]:
    system_lang = _get_system_language()
    if system_lang == 'en':
        return ['en']
    return [system_lang, 'en']


def get_reader(lang: Optional[list[str]] = None):
    global _reader, _reader_langs
    langs = lang or _get_default_languages()
    if _reader is None or set(langs) != set(_reader_langs or []):
        import easyocr
        _reader = easyocr.Reader(langs)
        _reader_langs = langs
    return _reader


@app.command(name="locate-text-coordinates")
def locate_text_coordinates(
    search: str = typer.Argument(..., help="Text to search for (partial match)"),
    image: Optional[str] = typer.Option(None, "--image", "-i", help="Path to image (if not provided, takes screenshot)"),
    lang: Optional[str] = typer.Option(None, "--lang", "-l", help="Languages to use (comma-separated, default: system language + en)"),
    case_sensitive: bool = typer.Option(False, "--case-sensitive", "-c", help="Case sensitive search"),
    window: Optional[str] = typer.Option(None, "--window", "-w", help="Search within a specific window"),
    active: bool = typer.Option(False, "--active", "-a", help="Search within the active window"),
):
    """Locate text coordinates on screen, within a window, or in an image using OCR."""
    def execute():
        region = _get_window_region(window, active)

        if image:
            if not Path(image).exists():
                raise DesktopAgentError(
                    code=ErrorCode.IMAGE_NOT_FOUND,
                    message=f"Image file '{image}' not found",
                )
            image_path = image
        else:
            screenshot_path = "temp_screenshot.png"
            img = pyautogui.screenshot(region=region)
            img.save(screenshot_path)
            image_path = screenshot_path

        languages = lang.split(',') if lang else None
        reader = get_reader(languages)

        results = reader.readtext(image_path)

        search_text = search if case_sensitive else search.lower()
        matches = []

        for (bbox, text, confidence) in results:
            compare_text = text if case_sensitive else text.lower()
            if search_text in compare_text:
                top_left = bbox[0]
                bottom_right = bbox[2]

                x1, y1 = int(top_left[0]), int(top_left[1])
                x2, y2 = int(bottom_right[0]), int(bottom_right[1])
                width = x2 - x1
                height = y2 - y1
                center_x = int((x1 + x2) / 2)
                center_y = int((y1 + y2) / 2)

                match = {
                    "text": text,
                    "confidence": float(confidence),
                    "bounding_box": {
                        "x": x1,
                        "y": y1,
                        "width": width,
                        "height": height,
                        "center_x": center_x,
                        "center_y": center_y,
                    },
                }
                matches.append(match)

        if not image and Path("temp_screenshot.png").exists():
            Path("temp_screenshot.png").unlink()

        return {"matches": matches}
    _handle_command("screen.locate_text_coordinates", execute)


@app.command(name="read-all-text")
def read_all_text(
    image: Optional[str] = typer.Option(None, "--image", "-i", help="Path to image (if not provided, takes screenshot)"),
    lang: Optional[str] = typer.Option(None, "--lang", "-l", help="Languages to use (comma-separated, default: system language + en)"),
    window: Optional[str] = typer.Option(None, "--window", "-w", help="Read from a specific window"),
    active: bool = typer.Option(False, "--active", "-a", help="Read from the active window"),
):
    """Read all text from screen, a targeted window, or an image using OCR."""
    def execute():
        region = _get_window_region(window, active)

        if image:
            if not Path(image).exists():
                raise DesktopAgentError(
                    code=ErrorCode.IMAGE_NOT_FOUND,
                    message=f"Image file '{image}' not found",
                )
            image_path = image
        else:
            screenshot_path = "temp_screenshot.png"
            img = pyautogui.screenshot(region=region)
            img.save(screenshot_path)
            image_path = screenshot_path

        languages = lang.split(',') if lang else None
        reader = get_reader(languages)

        results = reader.readtext(image_path)

        all_text = []
        for (bbox, text, confidence) in results:
            top_left = bbox[0]
            bottom_right = bbox[2]

            x1, y1 = int(top_left[0]), int(top_left[1])
            x2, y2 = int(bottom_right[0]), int(bottom_right[1])
            width = x2 - x1
            height = y2 - y1
            center_x = int((x1 + x2) / 2)
            center_y = int((y1 + y2) / 2)

            item = {
                "text": text,
                "confidence": float(confidence),
                "bounding_box": {
                    "x": x1,
                    "y": y1,
                    "width": width,
                    "height": height,
                    "center_x": center_x,
                    "center_y": center_y,
                },
            }
            all_text.append(item)

        if not image and Path("temp_screenshot.png").exists():
            Path("temp_screenshot.png").unlink()

        return {"text_items": all_text}
    _handle_command("screen.read_all_text", execute)


import sys


path: desktop_agent/specs/v1/schema.json
content: {
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Desktop Agent Command Response",
  "description": "Schema for all desktop-agent command responses",
  "type": "object",
  "properties": {
    "success": {
      "type": "boolean",
      "description": "Whether the command executed successfully"
    },
    "command": {
      "type": "string",
      "description": "Full command path (e.g., 'mouse.move', 'screen.screenshot')"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp in UTC"
    },
    "duration_ms": {
      "type": "integer",
      "minimum": 0,
      "description": "Execution time in milliseconds"
    },
    "data": {
      "type": "object",
      "description": "Command-specific result data",
      "properties": {
        "x": { "type": "integer", "description": "X coordinate" },
        "y": { "type": "integer", "description": "Y coordinate" },
        "position": {
          "type": "object",
          "properties": {
            "x": { "type": "integer" },
            "y": { "type": "integer" }
          }
        },
        "size": {
          "type": "object",
          "properties": {
            "width": { "type": "integer" },
            "height": { "type": "integer" }
          }
        },
        "windows": {
          "type": "array",
          "items": { "type": "string" },
          "description": "List of window titles"
        },
        "text": { "type": "string", "description": "Extracted or input text" },
        "pixel": {
          "type": "object",
          "properties": {
            "r": { "type": "integer" },
            "g": { "type": "integer" },
            "b": { "type": "integer" },
            "hex": { "type": "string" }
          }
        },
        "message": { "type": "string", "description": "Status message" },
        "result": { "type": "string", "description": "Generic result" },
        "on_screen": { "type": "boolean", "description": "Whether coordinates are on screen" },
        "image_found": { "type": "boolean", "description": "Whether image was found" },
        "bounding_box": {
          "type": "object",
          "properties": {
            "left": { "type": "integer" },
            "top": { "type": "integer" },
            "width": { "type": "integer" },
            "height": { "type": "integer" },
            "center_x": { "type": "integer" },
            "center_y": { "type": "integer" }
          }
        }
      }
    },
    "error": {
      "type": "object",
      "description": "Error details if command failed",
      "properties": {
        "code": {
          "type": "string",
          "description": "Error code (e.g., 'invalid_argument', 'image_not_found')"
        },
        "message": {
          "type": "string",
          "description": "Human-readable error message"
        },
        "details": {
          "type": "object",
          "description": "Additional error context"
        },
        "recoverable": {
          "type": "boolean",
          "description": "Whether the error can be recovered from"
        }
      }
    }
  },
  "required": ["success", "command", "timestamp"]
}



path: desktop_agent/utils/__init__.py
content: from .errors import ErrorCode, DesktopAgentError
from .response import CommandResponse


path: desktop_agent/utils/errors.py
content: from enum import Enum
from typing import Optional, Any

class ErrorCode(Enum):
    SUCCESS = 0
    INVALID_ARGUMENT = 1
    COORDINATES_OUT_OF_BOUNDS = 2
    IMAGE_NOT_FOUND = 3
    WINDOW_NOT_FOUND = 4
    OCR_FAILED = 5
    APPLICATION_NOT_FOUND = 6
    PERMISSION_DENIED = 7
    PLATFORM_NOT_SUPPORTED = 8
    TIMEOUT = 9
    UNKNOWN_ERROR = 99

    def to_string(self) -> str:
        return self.name.lower()

    @classmethod
    def from_exception(cls, exc: Exception) -> "ErrorCode":
        exc_type = type(exc).__name__.lower()
        mapping = {
            "valueerror": cls.INVALID_ARGUMENT,
            "typeerror": cls.INVALID_ARGUMENT,
            "oserror": cls.PERMISSION_DENIED,
            "filenotfounderror": cls.IMAGE_NOT_FOUND,
            "timeouterror": cls.TIMEOUT,
        }
        return mapping.get(exc_type, cls.UNKNOWN_ERROR)

class DesktopAgentError(Exception):
    def __init__(
        self,
        code: ErrorCode,
        message: str,
        details: Optional[dict] = None,
        recoverable: bool = True,
    ):
        self.code = code
        self.message = message
        self.details = details or {}
        self.recoverable = recoverable
        super().__init__(f"[{code.to_string()}] {message}")

    def to_dict(self) -> dict[str, Any]:
        return {
            "code": self.code.to_string(),
            "message": self.message,
            "details": self.details,
            "recoverable": self.recoverable,
        }

    def exit_code(self) -> int:
        return self.code.value if self.code != ErrorCode.SUCCESS else 0


path: desktop_agent/utils/response.py
content: from datetime import datetime, timezone
from typing import Any, Optional
from pydantic import BaseModel, Field

class CommandResponse(BaseModel):
    success: bool = Field(..., description="Whether the command succeeded")
    command: str = Field(..., description="Full command path (e.g., 'mouse.move')")
    timestamp: str = Field(
        default_factory=lambda: datetime.now(timezone.utc).isoformat(),
        description="ISO 8601 timestamp in UTC"
    )
    duration_ms: Optional[int] = Field(None, description="Execution time in milliseconds")
    data: Optional[dict[str, Any]] = Field(None, description="Command-specific data")
    error: Optional[dict[str, Any]] = Field(None, description="Error details if failed")

    def to_json(self) -> str:
        return self.model_dump_json()

    def to_text(self) -> str:
        if self.success:
            if self.data:
                if "position" in self.data:
                    pos = self.data["position"]
                    return f"Position: ({pos.get('x', '?')}, {pos.get('y', '?')})"
                if "text" in self.data:
                    return self.data["text"]
                if "size" in self.data:
                    s = self.data["size"]
                    return f"Size: {s.get('width', '?')}x{s.get('height', '?')}"
                if "message" in self.data:
                    return self.data["message"]
                if "windows" in self.data:
                    count = len(self.data["windows"])
                    return f"Found {count} window(s)"
                if "result" in self.data:
                    return str(self.data["result"])
            return "OK"
        else:
            error_msg = self.error.get("message", "Unknown error") if self.error else "Unknown error"
            return f"Error: {error_msg}"

    def print(self) -> None:
        print(self.to_json())

    @classmethod
    def success_response(
        cls,
        command: str,
        data: Optional[dict[str, Any]] = None,
        duration_ms: Optional[int] = None,
    ) -> "CommandResponse":
        return cls(
            success=True,
            command=command,
            duration_ms=duration_ms,
            data=data,
            error=None,
        )

    @classmethod
    def error_response(
        cls,
        command: str,
        code: str,
        message: str,
        details: Optional[dict[str, Any]] = None,
        duration_ms: Optional[int] = None,
    ) -> "CommandResponse":
        return cls(
            success=False,
            command=command,
            duration_ms=duration_ms,
            data=None,
            error={
                "code": code,
                "message": message,
                "details": details,
            },
        )


path: examples/automation_examples.md
content: # Automation Examples

Collection of practical automation examples using the Desktop Control Skill.

## Example 1: Open and Configure Notepad

```bash
# Open Run dialog
python main.py keyboard hotkey "win,r"

# Small delay for dialog to appear (200ms)
# In actual automation, add sleep between steps

# Type notepad
python main.py keyboard write "notepad"

# Press Enter to open
python main.py keyboard press enter

# Wait for Notepad to open (500ms recommended)

# Type some text
python main.py keyboard write "Hello from Desktop Control Skill!"

# Select all
python main.py keyboard hotkey "ctrl,a"

# Change to uppercase (in many text editors)
python main.py keyboard hotkey "shift,f3"
```

## Example 2: Screenshot Workflow

```bash
# Get screen dimensions
python main.py screen size
# Output: Screen size: 1920x1080

# Take full screenshot
python main.py screen screenshot full_desktop.png

# Take screenshot of top-left quadrant
python main.py screen screenshot quadrant.png --region "0,0,960,540"

# Get pixel color at specific location
python main.py screen pixel 100 100
# Output: Pixel at (100, 100): RGB(45, 45, 48)
```

## Example 3: Form Filling Automation

Automate filling a form with tab navigation:

```bash
# Click first field (adjust coordinates for your form)
python main.py mouse click 300 200

# Fill first name
python main.py keyboard write "John"

# Tab to next field
python main.py keyboard press tab

# Fill last name
python main.py keyboard write "Doe"

# Tab to email field
python main.py keyboard press tab

# Fill email
python main.py keyboard write "[REDACTED_SECRET]"

# Tab to next field
python main.py keyboard press tab

# Fill phone
python main.py keyboard write "555-1234"

# Submit form (or click submit button)
python main.py keyboard press enter
```

## Example 4: Image-Based Button Clicking

Find and click a button using image recognition:

```bash
# First, save a screenshot of the button you want to click
# Name it 'button.png' and place it in the project directory

# Locate the button on screen
python main.py screen locate-center button.png --confidence 0.9
# Output: Center at: (450, 320)

# If found, click at those coordinates
python main.py mouse click 450 320
```

## Example 5: Copy File Path from Explorer

```bash
# Assuming file is already selected in Windows Explorer

# Copy path: Alt+D (focus address bar) then Ctrl+C
python main.py keyboard hotkey "alt,d"

# Small delay

# Copy address
python main.py keyboard hotkey "ctrl,c"

# Path is now in clipboard
```

## Example 6: Multiple Window Management

```bash
# Show all windows (Windows + Tab)
python main.py keyboard hotkey "win,tab"

# Navigate with arrow keys
python main.py keyboard press right
python main.py keyboard press right

# Select window
python main.py keyboard press enter

# Or use Alt+Tab for quick switching
python main.py keyboard hotkey "alt,tab"
```

## Example 7: Text Manipulation

```bash
# Select current line (Home, Shift+End)
python main.py keyboard press home
python main.py keyboard hotkey "shift,end"

# Copy it
python main.py keyboard hotkey "ctrl,c"

# Move to end of document
python main.py keyboard hotkey "ctrl,end"

# Paste
python main.py keyboard hotkey "ctrl,v"
```

## Example 8: Screen Region Analysis

```bash
# Capture specific region for analysis
python main.py screen screenshot taskbar.png --region "0,1040,1920,40"

# Get colors at multiple points
python main.py screen pixel 10 10
python main.py screen pixel 100 100
python main.py screen pixel 500 500

# Verify a coordinate is on screen before clicking
python main.py screen on-screen 2000 2000
# Output: (2000, 2000) is NOT on screen
```

## Example 9: User Interaction

```bash
# Ask for confirmation before proceeding
python main.py message confirm "Do you want to proceed with the operation?"
# User clicks OK or Cancel

# Get user input
python main.py message prompt "Enter the filename:" --default "document.txt"
# Returns user input

# Show completion message
python main.py message alert "Operation completed successfully!"
```

## Example 10: Drawing/Painting Automation

```bash
# Open Paint
python main.py keyboard hotkey "win,r"
python main.py keyboard write "mspaint"
python main.py keyboard press enter

# Wait for Paint to open

# Select pencil tool (keyboard shortcut)
python main.py keyboard press p

# Draw a square by dragging
python main.py mouse move 200 200
python main.py mouse drag 400 200 --duration 0.5
python main.py mouse drag 400 400 --duration 0.5
python main.py mouse drag 200 400 --duration 0.5
python main.py mouse drag 200 200 --duration 0.5
```

## Tips for Reliable Automation

1. **Add Delays**: Always add small delays between commands when automating UI
   - After opening applications: 500-1000ms
   - After clicking: 100-200ms
   - After typing: 50-100ms per character or use `--interval`

2. **Verify Before Acting**: 
   - Use `screen size` to calculate safe coordinates
   - Use `screen on-screen` to validate coordinates
   - Use `screen locate` to find UI elements

3. **Error Handling**:
   - Save screenshots before/after operations for debugging
   - Use `message confirm` for critical operations
   - Validate paths and files exist before using them

4. **Resolution Independence**:
   - Get screen size first
   - Calculate relative positions (e.g., center = width/2, height/2)
   - Use image recognition instead of fixed coordinates when possible

5. **Keyboard Shortcuts**:
   - Prefer keyboard shortcuts over mouse clicks when possible
   - More reliable and faster
   - Less dependent on screen resolution


path: main.py
content: """Desktop Agent - Backwards compatibility wrapper.

For production use: uvx desktop-agent <command>
For development: python main.py <command> or python -m desktop_agent <command>
"""
from desktop_agent import app

if __name__ == "__main__":
    app()



path: openspec/AGENTS.md
content: # OpenSpec Instructions

Instructions for AI coding assistants using OpenSpec for spec-driven development.

## TL;DR Quick Checklist

- Search existing work: `openspec spec list --long`, `openspec list` (use `rg` only for full-text search)
- Decide scope: new capability vs modify existing capability
- Pick a unique `change-id`: kebab-case, verb-led (`add-`, `update-`, `remove-`, `refactor-`)
- Scaffold: `proposal.md`, `tasks.md`, `design.md` (only if needed), and delta specs per affected capability
- Write deltas: use `## ADDED|MODIFIED|REMOVED|RENAMED Requirements`; include at least one `#### Scenario:` per requirement
- Validate: `openspec validate [change-id] --strict` and fix issues
- Request approval: Do not start implementation until proposal is approved

## Three-Stage Workflow

### Stage 1: Creating Changes
Create proposal when you need to:
- Add features or functionality
- Make breaking changes (API, schema)
- Change architecture or patterns  
- Optimize performance (changes behavior)
- Update security patterns

Triggers (examples):
- "Help me create a change proposal"
- "Help me plan a change"
- "Help me create a proposal"
- "I want to create a spec proposal"
- "I want to create a spec"

Loose matching guidance:
- Contains one of: `proposal`, `change`, `spec`
- With one of: `create`, `plan`, `make`, `start`, `help`

Skip proposal for:
- Bug fixes (restore intended behavior)
- Typos, formatting, comments
- Dependency updates (non-breaking)
- Configuration changes
- Tests for existing behavior

**Workflow**
1. Review `openspec/project.md`, `openspec list`, and `openspec list --specs` to understand current context.
2. Choose a unique verb-led `change-id` and scaffold `proposal.md`, `tasks.md`, optional `design.md`, and spec deltas under `openspec/changes/<id>/`.
3. Draft spec deltas using `## ADDED|MODIFIED|REMOVED Requirements` with at least one `#### Scenario:` per requirement.
4. Run `openspec validate <id> --strict` and resolve any issues before sharing the proposal.

### Stage 2: Implementing Changes
Track these steps as TODOs and complete them one by one.
1. **Read proposal.md** - Understand what's being built
2. **Read design.md** (if exists) - Review technical decisions
3. **Read tasks.md** - Get implementation checklist
4. **Implement tasks sequentially** - Complete in order
5. **Confirm completion** - Ensure every item in `tasks.md` is finished before updating statuses
6. **Update checklist** - After all work is done, set every task to `- [x]` so the list reflects reality
7. **Approval gate** - Do not start implementation until the proposal is reviewed and approved

### Stage 3: Archiving Changes
After deployment, create separate PR to:
- Move `changes/[name]/` → `changes/archive/YYYY-MM-DD-[name]/`
- Update `specs/` if capabilities changed
- Use `openspec archive <change-id> --skip-specs --yes` for tooling-only changes (always pass the change ID explicitly)
- Run `openspec validate --strict` to confirm the archived change passes checks

## Before Any Task

**Context Checklist:**
- [ ] Read relevant specs in `specs/[capability]/spec.md`
- [ ] Check pending changes in `changes/` for conflicts
- [ ] Read `openspec/project.md` for conventions
- [ ] Run `openspec list` to see active changes
- [ ] Run `openspec list --specs` to see existing capabilities

**Before Creating Specs:**
- Always check if capability already exists
- Prefer modifying existing specs over creating duplicates
- Use `openspec show [spec]` to review current state
- If request is ambiguous, ask 1–2 clarifying questions before scaffolding

### Search Guidance
- Enumerate specs: `openspec spec list --long` (or `--json` for scripts)
- Enumerate changes: `openspec list` (or `openspec change list --json` - deprecated but available)
- Show details:
  - Spec: `openspec show <spec-id> --type spec` (use `--json` for filters)
  - Change: `openspec show <change-id> --json --deltas-only`
- Full-text search (use ripgrep): `rg -n "Requirement:|Scenario:" openspec/specs`

## Quick Start

### CLI Commands

```bash
# Essential commands
openspec list                  # List active changes
openspec list --specs          # List specifications
openspec show [item]           # Display change or spec
openspec validate [item]       # Validate changes or specs
openspec archive <change-id> [--yes|-y]   # Archive after deployment (add --yes for non-interactive runs)

# Project management
openspec init [path]           # Initialize OpenSpec
openspec update [path]         # Update instruction files

# Interactive mode
openspec show                  # Prompts for selection
openspec validate              # Bulk validation mode

# Debugging
openspec show [change] --json --deltas-only
openspec validate [change] --strict
```

### Command Flags

- `--json` - Machine-readable output
- `--type change|spec` - Disambiguate items
- `--strict` - Comprehensive validation
- `--no-interactive` - Disable prompts
- `--skip-specs` - Archive without spec updates
- `--yes`/`-y` - Skip confirmation prompts (non-interactive archive)

## Directory Structure

```
openspec/
├── project.md              # Project conventions
├── specs/                  # Current truth - what IS built
│   └── [capability]/       # Single focused capability
│       ├── spec.md         # Requirements and scenarios
│       └── design.md       # Technical patterns
├── changes/                # Proposals - what SHOULD change
│   ├── [change-name]/
│   │   ├── proposal.md     # Why, what, impact
│   │   ├── tasks.md        # Implementation checklist
│   │   ├── design.md       # Technical decisions (optional; see criteria)
│   │   └── specs/          # Delta changes
│   │       └── [capability]/
│   │           └── spec.md # ADDED/MODIFIED/REMOVED
│   └── archive/            # Completed changes
```

## Creating Change Proposals

### Decision Tree

```
New request?
├─ Bug fix restoring spec behavior? → Fix directly
├─ Typo/format/comment? → Fix directly  
├─ New feature/capability? → Create proposal
├─ Breaking change? → Create proposal
├─ Architecture change? → Create proposal
└─ Unclear? → Create proposal (safer)
```

### Proposal Structure

1. **Create directory:** `changes/[change-id]/` (kebab-case, verb-led, unique)

2. **Write proposal.md:**
```markdown
# Change: [Brief description of change]

## Why
[1-2 sentences on problem/opportunity]

## What Changes
- [Bullet list of changes]
- [Mark breaking changes with **BREAKING**]

## Impact
- Affected specs: [list capabilities]
- Affected code: [key files/systems]
```

3. **Create spec deltas:** `specs/[capability]/spec.md`
```markdown
## ADDED Requirements
### Requirement: New Feature
The system SHALL provide...

#### Scenario: Success case
- **WHEN** user performs action
- **THEN** expected result

## MODIFIED Requirements
### Requirement: Existing Feature
[Complete modified requirement]

## REMOVED Requirements
### Requirement: Old Feature
**Reason**: [Why removing]
**Migration**: [How to handle]
```
If multiple capabilities are affected, create multiple delta files under `changes/[change-id]/specs/<capability>/spec.md`—one per capability.

4. **Create tasks.md:**
```markdown
## 1. Implementation
- [ ] 1.1 Create database schema
- [ ] 1.2 Implement API endpoint
- [ ] 1.3 Add frontend component
- [ ] 1.4 Write tests
```

5. **Create design.md when needed:**
Create `design.md` if any of the following apply; otherwise omit it:
- Cross-cutting change (multiple services/modules) or a new architectural pattern
- New external dependency or significant data model changes
- Security, performance, or migration complexity
- Ambiguity that benefits from technical decisions before coding

Minimal `design.md` skeleton:
```markdown
## Context
[Background, constraints, stakeholders]

## Goals / Non-Goals
- Goals: [...]
- Non-Goals: [...]

## Decisions
- Decision: [What and why]
- Alternatives considered: [Options + rationale]

## Risks / Trade-offs
- [Risk] → Mitigation

## Migration Plan
[Steps, rollback]

## Open Questions
- [...]
```

## Spec File Format

### Critical: Scenario Formatting

**CORRECT** (use #### headers):
```markdown
#### Scenario: User login success
- **WHEN** valid credentials provided
- **THEN** return JWT token
```

**WRONG** (don't use bullets or bold):
```markdown
- **Scenario: User login**  ❌
**Scenario**: User login     ❌
### Scenario: User login      ❌
```

Every requirement MUST have at least one scenario.

### Requirement Wording
- Use SHALL/MUST for normative requirements (avoid should/may unless intentionally non-normative)

### Delta Operations

- `## ADDED Requirements` - New capabilities
- `## MODIFIED Requirements` - Changed behavior
- `## REMOVED Requirements` - Deprecated features
- `## RENAMED Requirements` - Name changes

Headers matched with `trim(header)` - whitespace ignored.

#### When to use ADDED vs MODIFIED
- ADDED: Introduces a new capability or sub-capability that can stand alone as a requirement. Prefer ADDED when the change is orthogonal (e.g., adding "Slash Command Configuration") rather than altering the semantics of an existing requirement.
- MODIFIED: Changes the behavior, scope, or acceptance criteria of an existing requirement. Always paste the full, updated requirement content (header + all scenarios). The archiver will replace the entire requirement with what you provide here; partial deltas will drop previous details.
- RENAMED: Use when only the name changes. If you also change behavior, use RENAMED (name) plus MODIFIED (content) referencing the new name.

Common pitfall: Using MODIFIED to add a new concern without including the previous text. This causes loss of detail at archive time. If you aren’t explicitly changing the existing requirement, add a new requirement under ADDED instead.

Authoring a MODIFIED requirement correctly:
1) Locate the existing requirement in `openspec/specs/<capability>/spec.md`.
2) Copy the entire requirement block (from `### Requirement: ...` through its scenarios).
3) Paste it under `## MODIFIED Requirements` and edit to reflect the new behavior.
4) Ensure the header text matches exactly (whitespace-insensitive) and keep at least one `#### Scenario:`.

Example for RENAMED:
```markdown
## RENAMED Requirements
- FROM: `### Requirement: Login`
- TO: `### Requirement: User Authentication`
```

## Troubleshooting

### Common Errors

**"Change must have at least one delta"**
- Check `changes/[name]/specs/` exists with .md files
- Verify files have operation prefixes (## ADDED Requirements)

**"Requirement must have at least one scenario"**
- Check scenarios use `#### Scenario:` format (4 hashtags)
- Don't use bullet points or bold for scenario headers

**Silent scenario parsing failures**
- Exact format required: `#### Scenario: Name`
- Debug with: `openspec show [change] --json --deltas-only`

### Validation Tips

```bash
# Always use strict mode for comprehensive checks
openspec validate [change] --strict

# Debug delta parsing
openspec show [change] --json | jq '.deltas'

# Check specific requirement
openspec show [spec] --json -r 1
```

## Happy Path Script

```bash
# 1) Explore current state
openspec spec list --long
openspec list
# Optional full-text search:
# rg -n "Requirement:|Scenario:" openspec/specs
# rg -n "^#|Requirement:" openspec/changes

# 2) Choose change id and scaffold
CHANGE=add-two-factor-auth
mkdir -p openspec/changes/$CHANGE/{specs/auth}
printf "## Why\n...\n\n## What Changes\n- ...\n\n## Impact\n- ...\n" > openspec/changes/$CHANGE/proposal.md
printf "## 1. Implementation\n- [ ] 1.1 ...\n" > openspec/changes/$CHANGE/tasks.md

# 3) Add deltas (example)
cat > openspec/changes/$CHANGE/specs/auth/spec.md << 'EOF'
## ADDED Requirements
### Requirement: Two-Factor Authentication
Users MUST provide a second factor during login.

#### Scenario: OTP required
- **WHEN** valid credentials are provided
- **THEN** an OTP challenge is required
EOF

# 4) Validate
openspec validate $CHANGE --strict
```

## Multi-Capability Example

```
openspec/changes/add-2fa-notify/
├── proposal.md
├── tasks.md
└── specs/
    ├── auth/
    │   └── spec.md   # ADDED: Two-Factor Authentication
    └── notifications/
        └── spec.md   # ADDED: OTP email notification
```

auth/spec.md
```markdown
## ADDED Requirements
### Requirement: Two-Factor Authentication
...
```

notifications/spec.md
```markdown
## ADDED Requirements
### Requirement: OTP Email Notification
...
```

## Best Practices

### Simplicity First
- Default to <100 lines of new code
- Single-file implementations until proven insufficient
- Avoid frameworks without clear justification
- Choose boring, proven patterns

### Complexity Triggers
Only add complexity with:
- Performance data showing current solution too slow
- Concrete scale requirements (>1000 users, >100MB data)
- Multiple proven use cases requiring abstraction

### Clear References
- Use `file.ts:42` format for code locations
- Reference specs as `specs/auth/spec.md`
- Link related changes and PRs

### Capability Naming
- Use verb-noun: `user-auth`, `payment-capture`
- Single purpose per capability
- 10-minute understandability rule
- Split if description needs "AND"

### Change ID Naming
- Use kebab-case, short and descriptive: `add-two-factor-auth`
- Prefer verb-led prefixes: `add-`, `update-`, `remove-`, `refactor-`
- Ensure uniqueness; if taken, append `-2`, `-3`, etc.

## Tool Selection Guide

| Task | Tool | Why |
|------|------|-----|
| Find files by pattern | Glob | Fast pattern matching |
| Search code content | Grep | Optimized regex search |
| Read specific files | Read | Direct file access |
| Explore unknown scope | Task | Multi-step investigation |

## Error Recovery

### Change Conflicts
1. Run `openspec list` to see active changes
2. Check for overlapping specs
3. Coordinate with change owners
4. Consider combining proposals

### Validation Failures
1. Run with `--strict` flag
2. Check JSON output for details
3. Verify spec file format
4. Ensure scenarios properly formatted

### Missing Context
1. Read project.md first
2. Check related specs
3. Review recent archives
4. Ask for clarification

## Quick Reference

### Stage Indicators
- `changes/` - Proposed, not yet built
- `specs/` - Built and deployed
- `archive/` - Completed changes

### File Purposes
- `proposal.md` - Why and what
- `tasks.md` - Implementation steps
- `design.md` - Technical decisions
- `spec.md` - Requirements and behavior

### CLI Essentials
```bash
openspec list              # What's in progress?
openspec show [item]       # View details
openspec validate --strict # Is it correct?
openspec archive <change-id> [--yes|-y]  # Mark complete (add --yes for automation)
```

Remember: Specs are truth. Changes are proposals. Keep them in sync.


path: openspec/changes/rename-to-desktop-agent/design.md
content: # Design: Desktop-Agent Package Structure and UVX Distribution

## Overview
This change transforms the project from a local script-based tool to a properly packaged Python application that can be executed via `uvx` without installation.

## Architecture Decisions

### 1. Package Structure

**Current:**
```
desktop-skill/
├── main.py
├── commands/
│   ├── mouse.py
│   ├── keyboard.py
│   └── ...
└── pyproject.toml
```

**Proposed:**
```
desktop-agent/
├── desktop_agent/          # New package directory
│   ├── __init__.py        # Exports CLI app
│   ├── __main__.py        # Entry point for python -m
│   └── commands/          # Moved inside package
│       ├── __init__.py
│       ├── mouse.py
│       └── ...
├── pyproject.toml         # Updated config
└── scripts/
    └── install.py
```

**Rationale:**
- Standard Python package structure enables proper distribution
- `__main__.py` allows `python -m desktop_agent` execution
- Package name uses underscores (Python convention), CLI uses hyphens (CLI convention)

### 2. Entry Points Configuration

```toml
[project.scripts]
desktop-agent = "desktop_agent:app"
```

**How it works:**
- `uvx desktop-agent` → uvx downloads/installs package → runs `desktop_agent:app`
- `desktop-agent` → runs installed CLI (if installed locally)
- `python -m desktop_agent` → runs package as module

### 3. UVX Compatibility

**Requirements:**
- Valid `pyproject.toml` with build system
- Proper entry points defined
- Package name matches PyPI-compatible naming (even if not published yet)

**Benefits:**
- Zero-install execution: `uvx desktop-agent <command>` works immediately
- Version pinning: `uvx desktop-agent@0.1.0 <command>`
- Isolated environments: uvx creates temporary venv per execution
- Dependency management: uvx handles all dependencies automatically

### 4. Import Path Migration

**Before:**
```python
from commands import mouse, keyboard
```

**After:**
```python
from desktop_agent.commands import mouse, keyboard
```

**Strategy:**
- Update all imports systematically
- Ensure relative imports work within package
- Test each module independently

### 5. Backwards Compatibility

**Development:**
- Keep supporting `python main.py` during development
- `main.py` can be a thin wrapper that imports from package

**Production:**
- Users should migrate to `uvx desktop-agent` or `desktop-agent`
- Add deprecation notice in README for old invocation method

## Implementation Strategy

### Phase 1: Structure (Non-breaking)
1. Create `desktop_agent/` directory
2. Move code without changing imports yet
3. Setup can coexist with current structure

### Phase 2: Entry Points
1. Update `pyproject.toml`
2. Configure entry points
3. Test local installation

### Phase 3: Migration
1. Update all imports
2. Remove old structure
3. Update documentation

### Phase 4: Verification
1. Test uvx execution
2. Verify all commands work
3. Test edge cases

## Technical Considerations

### Python Package Naming
- **Package name**: `desktop_agent` (underscore)
- **Distribution name**: `desktop-agent` (hyphen)
- **CLI command**: `desktop-agent` (hyphen)

This follows PEP 8 and PyPA conventions.

### UVX Execution Model
```bash
uvx desktop-agent mouse move 100 200
```

UVX will:
1. Create isolated environment
2. Install desktop-agent and dependencies
3. Execute: `desktop_agent:app` with args `['mouse', 'move', '100', '200']`
4. Clean up environment (optional caching)

### Module Entry Point (`__main__.py`)
```python
from desktop_agent import app

if __name__ == "__main__":
    app()
```

Enables: `python -m desktop_agent <commands>`

## Testing Plan

### Manual Tests
```bash
# Test uvx execution (no install)
uvx --from . desktop-agent --help
uvx --from . desktop-agent mouse position

# Test local installation
uv pip install -e .
desktop-agent --help

# Test as module
python -m desktop_agent --help
```

### Verification Points
- [ ] All command categories accessible
- [ ] Help system works
- [ ] Arguments parse correctly
- [ ] Typer sub-apps load properly
- [ ] PyAutoGUI commands execute

## Risk Mitigation

**Risk**: Breaking imports during migration
- **Mitigation**: Phase-based approach, one module at a time

**Risk**: UVX not finding package
- **Mitigation**: Test with `uvx --from .` first (local testing)

**Risk**: Users confused by new invocation
- **Mitigation**: Clear migration guide, keep examples updated

## Future Enhancements (Out of Scope)
- Publish to PyPI for `uvx desktop-agent` without `--from`
- Add shell completions for better UX
- Multi-platform testing (macOS, Linux)


path: openspec/changes/rename-to-desktop-agent/proposal.md
content: # Proposal: Rename to Desktop-Agent and Configure UVX Distribution

## Summary
Rename the project from `desktop-skill` to `desktop-agent` and configure it for easy distribution via `uvx desktop-agent`, enabling one-command installation and execution.

## Motivation
- **Clearer naming**: "desktop-agent" better reflects that this is an AI agent skill for desktop control
- **Easy distribution**: Using `uvx` allows users and AI agents to run the tool without manual installation
- **Better discoverability**: Standard Python package naming improves searchability
- **Simplified usage**: `uvx desktop-agent <command>` is more intuitive than `python main.py <command>`

## Proposed Changes

### 1. Package Renaming
- Rename package from `desktop-skill` to `desktop-agent`
- Update all references in documentation, code, and configuration files
- Ensure backwards compatibility notes are added to README

### 2. UVX Distribution Configuration
- Configure `pyproject.toml` to support uvx execution
- Set up proper entry points for CLI execution
- Ensure the package can be installed from:
  - Local directory: `uv pip install .`
  - UVX direct execution: `uvx desktop-agent <command>`
  - Standard pip: `pip install desktop-agent` (future PyPI publication)

### 3. CLI Invocation Updates
- Primary invocation: `uvx desktop-agent <category> <command>`
- Alternative (local install): `desktop-agent <category> <command>`
- Development mode: `python -m desktop_agent <category> <command>`
- Backwards compat (dev only): `python main.py <category> <command>`

### 4. Documentation Updates
- Update SKILL.md with new invocation patterns
- Update README.md with uvx installation instructions
- Update examples to use `uvx desktop-agent` or `desktop-agent`
- Add migration guide for existing users (if any)

## Benefits
1. **One-command execution**: `uvx desktop-agent mouse move 100 200` works instantly
2. **No installation required**: uvx handles dependencies automatically
3. **Version control**: uvx can run specific versions: `uvx desktop-agent@0.2.0`
4. **Better for AI agents**: Clearer command structure and automatic dependency resolution

## Trade-offs
- Requires renaming files and updating all documentation
- Users with current installation will need to migrate
- Slightly longer command vs `python main.py` (but more explicit)

## Success Criteria
- [ ] Project renamed to `desktop-agent`
- [ ] `uvx desktop-agent --help` works without prior installation
- [ ] `uvx desktop-agent mouse position` executes successfully
- [ ] All documentation reflects new naming
- [ ] Installation script updated for new structure

## Out of Scope
- Publishing to PyPI (can be done later)
- Changing command structure or API
- Adding new features


path: openspec/changes/rename-to-desktop-agent/specs/cli-naming/spec.md
content: # CLI Naming Specification

## MODIFIED Requirements

### Requirement: CLI Command Naming
The CLI MUST be invoked using `desktop-agent` instead of `desktop-skill`.

#### Scenario: Primary Invocation
**Given** uvx or local installation  
**When** user runs `desktop-agent --help`  
**Then** help text should display  
**And** command should be recognized

#### Scenario: Category Commands
**Given** the CLI is available  
**When** user runs `desktop-agent mouse position`  
**Then** mouse position should be returned  
**And** same structure as before but with new name

### Requirement: Documentation Consistency  
All documentation MUST reference the new CLI name `desktop-agent`.

#### Scenario: SKILL.md References
**Given** an AI agent reading SKILL.md  
**When** looking for command examples  
**Then** all examples should use `desktop-agent`  
**And** no references to `desktop-skill` should exist

#### Scenario: README Instructions
**Given** a new user reading README  
**When** following installation instructions  
**Then** all commands should use `desktop-agent`  
**And** uvx invocation should be primary method

## REMOVED Requirements

### Requirement: Python Script Invocation (Deprecated)
The pattern `python main.py <command>` is DEPRECATED for production use.

#### Scenario: Development Only
**Given** development environment  
**When** using `python main.py`  
**Then** it may still work (backwards compat)  
**But** should not be documented as primary method


path: openspec/changes/rename-to-desktop-agent/specs/package-distribution/spec.md
content: # Package Distribution Specification

## ADDED Requirements

### Requirement: UVX-Compatible Package Structure
The package MUST be structured to support execution via `uvx` without prior installation.

#### Scenario: Zero-Install Execution
**Given** a user without the package installed  
**When** they run `uvx desktop-agent --help`  
**Then** the CLI help text should display  
**And** no manual installation steps should be required

#### Scenario: Command Execution via UVX
**Given** uvx is installed  
**When** user runs `uvx desktop-agent mouse position`  
**Then** the mouse position should be returned  
**And** all dependencies should be automatically resolved

### Requirement: Proper Python Package Structure
The package MUST follow standard Python package conventions for distribution.

#### Scenario: Package Import
**Given** the package is installed  
**When** a Python script imports `desktop_agent`  
**Then** the CLI app should be accessible  
**And** all command modules should be importable

#### Scenario: Module Execution
**Given** the package is installed  
**When** user runs `python -m desktop_agent --help`  
**Then** the CLI help should display  
**And** it should behave identically to `desktop-agent --help`

### Requirement: Entry Point Configuration
The package MUST define correct entry points for CLI execution.

#### Scenario: CLI Entry Point
**Given** the package is installed via pip/uv  
**When** user runs `desktop-agent <command>`  
**Then** the command should execute  
**And** the entry point should resolve to `desktop_agent:app`

## ADDED Requirements

### Requirement: Package Naming Convention
Package naming MUST follow Python and PyPA conventions.

#### Scenario: Module Name
**Given** the Python package  
**When** imported in code  
**Then** it should use `desktop_agent` (underscore)

#### Scenario: Distribution Name  
**Given** the package configuration  
**When** referenced in `pyproject.toml`  
**Then** it should use `desktop-agent` (hyphen)

#### Scenario: CLI Command Name
**Given** the installed CLI  
**When** invoked from terminal  
**Then** it should use `desktop-agent` (hyphen)

### Requirement: Dependency Management
All dependencies MUST be properly declared and automatically resolved.

#### Scenario: Automatic Dependency Installation
**Given** uvx is used to run the package  
**When** dependencies are not present  
**Then** uvx should automatically install them  
**And** the user should not see dependency errors

#### Scenario: Version Compatibility
**Given** the package requirements  
**When** checking Python version  
**Then** it should require Python 3.12+  
**And** should fail gracefully with clear message on older versions


path: openspec/changes/rename-to-desktop-agent/tasks.md
content: # Tasks: Rename to Desktop-Agent

## Phase 1: Package Structure
- [ ] Rename package directory from root to `desktop_agent` (Python package naming)
- [ ] Update `pyproject.toml` with new package name and entry points
- [ ] Configure `pyproject.toml` for uvx compatibility
- [ ] Create `desktop_agent/__init__.py` with CLI app
- [ ] Move `main.py` logic to `desktop_agent/__main__.py`

## Phase 2: Module Updates
- [ ] Update imports in all command modules (`commands/` → `desktop_agent/commands/`)
- [ ] Ensure all internal imports use new package name
- [ ] Update any hardcoded references to "desktop-skill"

## Phase 3: Documentation
- [ ] Update SKILL.md with new command invocations (`uvx desktop-agent`)
- [ ] Update README.md with uvx installation instructions
- [ ] Update examples/automation_examples.md with new command format
- [ ] Add migration notes for existing users

## Phase 4: Scripts
- [ ] Update `scripts/install.py` to reference new package name
- [ ] Add uvx-based installation verification
- [ ] Update quick start examples in install script

## Phase 5: Verification
- [ ] Test `uvx desktop-agent --help` (external invocation)
- [ ] Test `uvx desktop-agent mouse position`
- [ ] Test `uvx desktop-agent screen size`
- [ ] Test all command categories work via uvx
- [ ] Verify local development mode still works
- [ ] Run full command suite regression test

## Phase 6: Final Touches
- [ ] Update .gitignore if needed for new structure
- [ ] Verify README quickstart works end-to-end
- [ ] Archive old walkthrough and create new one


path: openspec/project.md
content: # Project Context

## Purpose
Desktop Control Skill is an AI agent skill that provides desktop automation capabilities through PyAutoGUI. It allows AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment programmatically.

## Tech Stack
- Python 3.12+
- PyAutoGUI (desktop automation library)
- Typer (CLI framework)
- UV (Python package installer and manager)

## Project Conventions

### Code Style
- Use descriptive function and variable names
- Include docstrings for all public functions
- Follow PEP 8 naming conventions
- Type hints where applicable

### Architecture Patterns
- Modular command organization: commands organized by category (mouse, keyboard, screen, message)
- CLI structure: Main entry point delegates to sub-applications
- Each command module is a Typer app registered with the main app

### Testing Strategy
- Manual verification of commands
- Integration testing through actual CLI invocation
- Validation of help text and command discovery

### Git Workflow
- Standard Git workflow with feature branches
- Clear commit messages describing changes

## Domain Context
- **Desktop Automation**: Commands interact with OS-level GUI elements
- **AI Agent Integration**: Designed to be easily discoverable and usable by AI agents
- **Cross-platform**: Primary target is Windows, but PyAutoGUI supports macOS and Linux

## Important Constraints
- Requires Python 3.12+
- PyAutoGUI has a fail-safe feature (moving mouse to screen corner aborts)
- Commands may fail if UI state changes between invocation and execution
- Screenshot and image location features depend on screen resolution

## External Dependencies
- PyAutoGUI: Core automation library
- Typer: CLI framework
- UV: Package management and distribution


path: package.json
content: name: @chillybean/pipx-desktop-agent
version: 1.3.2
description: Fixed Desktop Agent skill that installs correctly with pipx (includes missing __init__.py)
main: main.py

path: pyproject.toml
content: [project]
name = "desktop-agent"
version = "1.3.2"
description = "AI agent skill for desktop automation using PyAutoGUI"
readme = "README.md"
requires-python = ">=3.12"
authors = [
    { name = "Patrick Porto" }
]
keywords = ["desktop", "automation", "pyautogui", "ai", "agent", "skill"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Programming Language :: Python :: 3.12",
]
dependencies = [
    "easyocr>=1.7.2",
    "opencv-python>=4.13.0.90",
    "pillow>=12.1.0",
    "pyautogui>=0.9.54",
    "pywinctl>=0.4.1",
    "typer>=0.21.1",
    "pydantic>=2.0.0",
]

[project.scripts]
desktop-agent = "desktop_agent:app_cli"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["desktop_agent"]


path: scripts/install.py
content: """Installation script for desktop-skill.

This script helps AI agents and users install the desktop-skill easily.
"""
import subprocess
import sys
from pathlib import Path


def check_python_version():
    """Check if Python version is 3.12+."""
    version = sys.version_info
    if version.major < 3 or (version.major == 3 and version.minor < 12):
        print(f"❌ Python 3.12+ required. Current version: {version.major}.{version.minor}")
        print("   Please upgrade your Python installation.")
        return False
    print(f"✅ Python {version.major}.{version.minor}.{version.micro}")
    return True


def check_uv():
    """Check if uv is installed."""
    try:
        result = subprocess.run(
            ["uv", "--version"],
            capture_output=True,
            text=True,
            check=False,
        )
        if result.returncode == 0:
            print(f"✅ uv is installed: {result.stdout.strip()}")
            return True
        else:
            print("❌ uv not found")
            return False
    except FileNotFoundError:
        print("❌ uv not found")
        return False


def install_uv():
    """Provide instructions to install uv."""
    print("\n📦 To install uv, run:")
    print("   Windows (PowerShell): irm https://astral.sh/uv/install.ps1 | iex")
    print("   macOS/Linux: curl -LsSf https://astral.sh/uv/install.sh | sh")
    return False


def install_dependencies():
    """Install project dependencies using uv."""
    print("\n📦 Installing dependencies...")
    try:
        result = subprocess.run(
            ["uv", "sync"],
            check=True,
            capture_output=True,
            text=True,
        )
        print("✅ Dependencies installed successfully")
        return True
    except subprocess.CalledProcessError as e:
        print(f"❌ Failed to install dependencies: {e.stderr}")
        return False


def verify_installation():
    """Verify the installation by running help command."""
    print("\n🔍 Verifying installation...")
    try:
        result = subprocess.run(
            ["python", "main.py", "--help"],
            capture_output=True,
            text=True,
            check=True,
        )
        print("✅ Installation verified successfully")
        print("\n" + "="*50)
        print("Desktop Control Skill is ready to use!")
        print("="*50)
        print("\nQuick start:")
        print("  python main.py --help              # Show all commands")
        print("  python main.py mouse position      # Get mouse position")
        print("  python main.py screen size         # Get screen size")
        print("\nFor full documentation, see SKILL.md")
        return True
    except subprocess.CalledProcessError as e:
        print(f"❌ Verification failed: {e.stderr}")
        return False


def main():
    """Main installation process."""
    print("="*50)
    print("Desktop Control Skill - Installation")
    print("="*50)
    print()
    
    # Check Python version
    if not check_python_version():
        sys.exit(1)
    
    # Check for uv
    if not check_uv():
        if not install_uv():
            sys.exit(1)
    
    # Install dependencies
    if not install_dependencies():
        sys.exit(1)
    
    # Verify installation
    if not verify_installation():
        sys.exit(1)
    
    print("\n✨ Installation complete!")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis:  | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)