Skill: lurui808/web-desktop-automation (version 1.0.0)

--- name: web-desktop-automation description: Use when the user wants browser automation, web scraping, form filling, clicking, or desktop GUI automation, including mixed workflows that move between web pages and local applications. --- # Web + Desktop Automation Use this skill when a task may involve: - Opening or controlling websites - Reading or extracting page content - Filling forms, clicking buttons, logging in - Downloading or uploading files - Controlling desktop apps with mouse/keyboard - Combining browser steps with local app steps ## Core rule Prefer the simplest reliable path: 1. If the task can be done in the browser, use browser automation. 2. If the task needs local apps or OS-level interaction, use desktop automation. 3. If both are needed, split the job into clear phases and verify after each phase. ## Execution strategy ### 1) Classify the task Decide which of these applies: - Browser only - Desktop only - Mixed browser + desktop ### 2) Browser automation Use browser automation for: - Navigation - Search - Page reading - Form filling - Clicking controls - File upload/download - Logged-in web workflows Prefer stable selectors and explicit waits. Avoid brittle coordinate-based clicking when browser selectors exist. ### 3) Desktop automation Use desktop automation for: - Native apps - Window switching - Copy/paste between apps - File manager operations - UI flows outside the browser Prefer application/window-aware methods when available. Use image-based or coordinate-based control only when necessary. ### 4) Mixed workflows Break the task into phases: - Browser phase - Desktop phase - Browser phase again if needed After each phase, verify the result before continuing. ## Recovery rules If a step fails: 1. Re-check the current UI state 2. Re-locate the target element or window 3. Try a more stable selector or a different interaction method 4. If the task risks loss of data or irreversible action, stop and ask the user ## Best practices - Prefer deterministic steps over guessing - Avoid rapid blind retries - Capture key state when tasks are long or fragile - Keep flows small and modular - Use scripts for repeated actions - Use `scripts/browser_runner.py` for Playwright browser automation templates - Use `scripts/desktop_runner.py` for PyAutoGUI desktop automation templates - Use `scripts/mixed_orchestrator.py` for browser + desktop handoffs - Put browser-specific patterns in `references/browser-workflows.md` - Put desktop-specific patterns in `references/desktop-workflows.md` - Put mixed-flow orchestration examples in `references/mixed-flows.md` - Put dependency and installation notes in `references/dependencies.md` - Put a realistic browser-download → desktop-edit → browser-upload flow in `references/mixed-example.md` - See `requirements.txt` for a minimal install set - Put dependency and installation notes in `references/dependencies.md` - Put a realistic browser-download → desktop-edit → browser-upload flow in `references/mixed-example.md` - Put dependency and installation notes in `references/dependencies.md`

path: requirements.txt
content: playwright>=1.50.0
pyautogui>=0.9.54
opencv-python>=4.10.0.84
pillow>=10.4.0


path: scripts/browser_runner.py
content: """Playwright browser automation template.

Usage:
    python scripts/browser_runner.py https://example.com

Install:
    pip install playwright
    playwright install
"""

from __future__ import annotations

import argparse
from dataclasses import dataclass
from typing import Optional

try:
    from playwright.sync_api import Browser, Page, sync_playwright
except ImportError:  # pragma: no cover - graceful degradation when dependency is missing
    Browser = None  # type: ignore[assignment]
    Page = None  # type: ignore[assignment]
    sync_playwright = None  # type: ignore[assignment]


@dataclass
class BrowserConfig:
    headless: bool = False
    slow_mo_ms: int = 0
    timeout_ms: int = 30000


class BrowserUnavailableError(RuntimeError):
    pass


class BrowserRunner:
    def __init__(self, config: Optional[BrowserConfig] = None) -> None:
        self.config = config or BrowserConfig()
        self._pw = None
        self._browser: Optional[Browser] = None
        self._page: Optional[Page] = None

    @staticmethod
    def available() -> bool:
        return sync_playwright is not None

    def _ensure_available(self) -> None:
        if not self.available():
            raise BrowserUnavailableError(
                "Playwright is not installed. Run: pip install playwright && playwright install"
            )

    def start(self) -> Page:
        self._ensure_available()
        self._pw = sync_playwright().start()
        self._browser = self._pw.chromium.launch(
            headless=self.config.headless,
            slow_mo=self.config.slow_mo_ms,
        )
        context = self._browser.new_context()
        self._page = context.new_page()
        self._page.set_default_timeout(self.config.timeout_ms)
        return self._page

    @property
    def page(self) -> Page:
        if not self._page:
            raise RuntimeError("BrowserRunner not started")
        return self._page

    def goto(self, url: str) -> None:
        self.page.goto(url, wait_until="domcontentloaded")

    def search_text(self, text: str) -> bool:
        locator = self.page.get_by_text(text)
        return locator.count() > 0

    def click_text(self, text: str) -> None:
        self.page.get_by_text(text).first.click()

    def fill(self, selector: str, value: str) -> None:
        self.page.locator(selector).fill(value)

    def html(self) -> str:
        return self.page.content()

    def screenshot(self, path: str) -> None:
        self.page.screenshot(path=path, full_page=True)

    def close(self) -> None:
        if self._browser:
            self._browser.close()
        if self._pw:
            self._pw.stop()


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("url", help="Target URL")
    parser.add_argument("--headless", action="store_true")
    args = parser.parse_args()

    if not BrowserRunner.available():
        raise SystemExit(
            "Playwright is not installed. Run: pip install playwright && playwright install"
        )

    runner = BrowserRunner(BrowserConfig(headless=args.headless))
    try:
        page = runner.start()
        runner.goto(args.url)
        print(page.title())
        print(runner.html()[:1000])
    finally:
        runner.close()


if __name__ == "__main__":
    main()


path: scripts/desktop_runner.py
content: """PyAutoGUI desktop automation template.

Usage:
    python scripts/desktop_runner.py

Install:
    pip install pyautogui
"""

from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Optional, Tuple

import pyautogui


@dataclass
class DesktopConfig:
    pause_seconds: float = 0.2
    fail_safe: bool = True


class DesktopRunner:
    def __init__(self, config: Optional[DesktopConfig] = None) -> None:
        self.config = config or DesktopConfig()
        pyautogui.PAUSE = self.config.pause_seconds
        pyautogui.FAILSAFE = self.config.fail_safe

    def move(self, x: int, y: int, duration: float = 0.2) -> None:
        pyautogui.moveTo(x, y, duration=duration)

    def click(self, x: int, y: int, button: str = "left") -> None:
        pyautogui.click(x, y, button=button)

    def double_click(self, x: int, y: int) -> None:
        pyautogui.doubleClick(x, y)

    def write(self, text: str, interval: float = 0.02) -> None:
        pyautogui.write(text, interval=interval)

    def hotkey(self, *keys: str) -> None:
        pyautogui.hotkey(*keys)

    def press(self, key: str) -> None:
        pyautogui.press(key)

    def screenshot(self, path: str) -> None:
        pyautogui.screenshot(path)

    def locate(self, image_path: str, confidence: float = 0.9) -> Optional[Tuple[int, int, int, int]]:
        return pyautogui.locateOnScreen(image_path, confidence=confidence)

    def click_image(self, image_path: str, confidence: float = 0.9) -> bool:
        box = self.locate(image_path, confidence=confidence)
        if not box:
            return False
        center = pyautogui.center(box)
        pyautogui.click(center.x, center.y)
        return True

    def sleep(self, seconds: float) -> None:
        time.sleep(seconds)


if __name__ == "__main__":
    runner = DesktopRunner()
    runner.sleep(1)
    print(pyautogui.position())


path: scripts/mixed_orchestrator.py
content: """Mixed-flow orchestrator template.

This file shows how to chain browser and desktop phases.
"""

from __future__ import annotations

from dataclasses import dataclass

from browser_runner import BrowserConfig, BrowserRunner
from desktop_runner import DesktopRunner


@dataclass
class MixedFlowResult:
    downloaded_path: str | None = None
    notes: str = ""


def run_mixed_flow(url: str) -> MixedFlowResult:
    browser = BrowserRunner(BrowserConfig(headless=False))
    desktop = DesktopRunner()
    result = MixedFlowResult()

    try:
        if browser.available():
            page = browser.start()
            browser.goto(url)

            # Browser phase
            title = page.title()
            result.notes += f"Opened: {title}\n"
        else:
            result.notes += "Browser phase skipped: Playwright unavailable\n"

        # Desktop phase placeholder
        # Example: switch to a native app, paste content, or process downloaded file
        # desktop.hotkey("alt", "tab")
        # desktop.write("hello")

        return result
    finally:
        if browser.available():
            browser.close()


if __name__ == "__main__":
    print(run_mixed_flow("https://example.com"))


path: references/browser-workflows.md
content: # Browser Workflows

Use browser automation for:
- Search
- Web scraping
- Login flows
- Form submission
- File upload/download
- Reading page content

## Preferred approach
- Use selectors over screen coordinates
- Wait for page state instead of sleeping blindly
- Verify visible text after major actions
- Reuse authenticated sessions when allowed

## Common failure modes
- Modal dialogs blocking clicks
- Dynamic content not yet loaded
- Anti-bot or login challenges
- Stale selectors after page redesign


path: references/dependencies.md
content: # Dependencies and Installation

This skill uses Python-based automation tools.

## Required

### Browser automation
- Python 3.10+
- `playwright`
- Browser binaries installed via `playwright install`

Install:
```bash
pip install playwright
playwright install
```

### Desktop automation
- Python 3.10+
- `pyautogui`

Install:
```bash
pip install pyautogui
```

## Optional but recommended

### Image-based targeting
- `opencv-python`

Why:
- Enables confidence-based image matching in PyAutoGUI

Install:
```bash
pip install opencv-python
```

### Screenshot / image handling
- `pillow`

Install:
```bash
pip install pillow
```

## Suggested combined setup

```bash
pip install playwright pyautogui opencv-python pillow
playwright install
```

## Windows notes

- Run scripts from the same Python environment that has the packages installed
- If image matching is used, install `opencv-python` or confidence-based matching may fail
- Keep display scaling stable when using coordinate-based automation

## Quick verification

Browser:
```bash
python -c "from playwright.sync_api import sync_playwright; print('playwright ok')"
```

Desktop:
```bash
python -c "import pyautogui; print('pyautogui ok')"
```


path: references/desktop-workflows.md
content: # Desktop Workflows

Use desktop automation for:
- Native apps
- File manager
- Cross-window copy/paste
- Keyboard shortcuts
- GUI-only workflows

## Preferred approach
- Focus the correct window first
- Prefer app-specific shortcuts
- Verify the active window before typing
- Minimize reliance on screen coordinates

## Common failure modes
- Window focus lost
- Resolution changes
- App moved between monitors
- UI theme or scaling changes


path: references/mixed-example.md
content: # Real-World Mixed Example

## Scenario
Download a CSV report from a website, open it in a local spreadsheet app, make a small edit, save it, then upload the result back to the website.

## Why this example matters
This is the most common mixed workflow pattern:
- Browser prepares or retrieves the file
- Desktop app edits or transforms it
- Browser submits the result

## Phase 1: Browser download
1. Open the target website in Playwright.
2. Log in if needed.
3. Navigate to the report page.
4. Click the export/download button.
5. Wait until the file appears in the download folder.
6. Record the exact file path.

## Phase 2: Desktop processing
1. Open the downloaded file with the local spreadsheet app.
2. Make the required edit.
3. Save as a new file to avoid losing the original.
4. Verify the new file exists.

## Phase 3: Browser upload
1. Return to the browser.
2. Open the upload page.
3. Upload the edited file.
4. Submit.
5. Confirm success text or page state.

## Recommended safeguards
- Keep the original file unchanged.
- Use a new output filename.
- Verify each phase before continuing.
- If the browser phase fails, do not edit the file yet.
- If the desktop save fails, do not upload partial data.

## Template flow

```text
1. browser.goto(report_page)
2. browser.click(download)
3. wait for file
4. desktop.open(file)
5. desktop.edit_and_save(new_file)
6. browser.goto(upload_page)
7. browser.upload(new_file)
8. browser.submit()
9. verify success
```


path: references/mixed-flows.md
content: # Mixed Flows

Typical sequence:
1. Browser automation to fetch or prepare data
2. Desktop automation to use local software
3. Browser automation to upload or submit results

## Rules
- Split the flow into phases
- Confirm each phase before moving on
- Save intermediate artifacts when useful
- Use clear handoff data between browser and desktop steps


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)