Skill: jordaneparis/desktop-automation-100per100-local (version 2.0.1)

# Desktop Automation Skill v2.0 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![OpenClaw](https://img.shields.io/badge/OpenClaw-Skill-blue)](https://github.com/openclaw/openclaw) **Complete desktop automation for Windows/macOS/Linux. Zero-error edition.** --- ## ⚠️ **Privacy & Security** **CRITICAL**: This skill captures ALL keyboard and mouse events. - **NEVER record while entering passwords, credit cards, or secrets** - Recorded macros are stored as JSON in `recorded_macro/` directory - Always use **dry_run=true** to test before actual execution - Store macros in secure locations only - Enable safe mode by default (it is) --- ## 🎯 **What It Does** Automate desktop interactions without APIs: - ✅ Click, type, drag, scroll - ✅ Capture screenshots - ✅ Recognize images (OpenCV template matching) - ✅ Extract text (Tesseract OCR) - ✅ Record and replay macros - ✅ Find windows by title - ✅ Clipboard operations - ✅ Safe mode with dry_run for testing --- ## 🔐 **Safety Features (Built-In)** ### 1. **Safe Mode** (Default: ON) Blocks dangerous actions when enabled: - `type`, `press_key`, `click`, `drag` are monitored - Parameters are scanned for dangerous patterns: `rm `, `del `, `C:\Windows\`, `/etc/`, `sudo`, etc. - Blocked actions are logged ### 2. **Dry-Run Mode** All actions support `dry_run=true`: - Action is logged but NOT executed - Use for testing before running real automation ### 3. **Audit Logging** Every action logged to `~/.openclaw/skills/desktop-automation-logs/automation_YYYY-MM-DD.log` ### 4. **Thread Safety** All modules use locks to prevent race conditions. --- ## 📦 **Installation** ### 1. Extract Files Place `desktop-automation-ultra-local/` in: - Windows: `C:\Users\<User>\.openclaw\workspace\skills\` - Linux/macOS: `~/.openclaw/workspace/skills/` ### 2. Install Dependencies ```bash pip install -r requirements.txt ``` ### 3. Optional: Tesseract for OCR For `find_text_on_screen` functionality: - **Windows**: Download installer from https://github.com/UB-Mannheim/tesseract/wiki - **Linux**: `sudo apt install tesseract-ocr` - **macOS**: `brew install tesseract` ### 4. Restart OpenClaw ```bash openclaw gateway restart ``` --- ## 🚀 **Quick Start** ### Basic Click ```yaml action: click params: x: 100 y: 100 dry_run: true # Test first! ``` ### Type Text ```yaml action: type params: text: "Hello World" interval: 0.05 # Delay between keys dry_run: false ``` ### Find Image ```yaml action: find_image params: template_path: "templates/button.png" confidence: 0.95 ``` ### Extract Text (OCR) ```yaml action: read_text_ocr params: lang: "fra" # French ``` --- ## 📖 **Core Actions** ### **Mouse & Keyboard** | Action | Parameters | Returns | |--------|------------|---------| | `click` | `x`, `y`, `button="left"`, `dry_run` | `{status, x, y}` | | `type` | `text`, `interval=0.05`, `dry_run` | `{status, text}` | | `press_key` | `key`, `dry_run` | `{status, key}` | | `move_mouse` | `x`, `y`, `duration=0.5`, `dry_run` | `{status, x, y}` | | `scroll` | `amount=5`, `dry_run` | `{status, amount}` | | `drag` | `start_x`, `start_y`, `end_x`, `end_y`, `duration=0.5`, `dry_run` | `{status}` | | `copy_to_clipboard` | `text`, `dry_run` | `{status}` | | `paste_from_clipboard` | `dry_run` | `{status, length}` | ### **Screenshots & Windows** | Action | Parameters | Returns | |--------|------------|---------| | `screenshot` | `path="~/Desktop/screenshot.png"`, `dry_run` | `{status, path}` | | `get_active_window` | `dry_run` | `{status, title, x, y, width, height}` | | `list_windows` | `dry_run` | `{status, windows[], count}` | | `activate_window` | `title_substring`, `dry_run` | `{status, title}` | ### **Image Recognition** (requires OpenCV) | Action | Parameters | Returns | |--------|------------|---------| | `find_image` | `template_path`, `confidence=0.9`, `dry_run` | `{status, x, y, confidence}` | | `find_image_multiscale` | `template_path`, `confidence`, `scale_factors`, `dry_run` | `{status, x, y, confidence, scale}` | | `wait_for_image` | `template_path`, `timeout=30.0`, `interval=0.5`, `confidence=0.9`, `dry_run` | `{status, x, y, confidence}` | ### **OCR / Text Recognition** (requires Tesseract) | Action | Parameters | Returns | |--------|------------|---------| | `find_text_on_screen` | `text`, `lang="fra"`, `dry_run` | `{status, locations[], count}` | | `find_all_text_on_screen` | `text`, `lang="fra"`, `dry_run` | `{status, data[], count}` | | `read_text_ocr` | `lang="fra"`, `dry_run` | `{status, text, length}` | | `read_text_region` | `x`, `y`, `width`, `height`, `lang="fra"`, `dry_run` | `{status, text, length}` | | `extract_screen_data` | `region={}`, `output_format="json"`, `lang="fra"`, `dry_run` | `{status, data[], count}` | ### **Macros** | Action | Parameters | Returns | |--------|------------|---------| | `play_macro` | `macro_path`, `speed=1.0`, `dry_run` | `{status, executed, total, errors[]}` | | `stop_macro` | — | `{status}` | | `play_macro_with_subroutines` | `macro_path`, `speed=1.0`, `sub_macros_dir`, `dry_run` | `{status, executed, total, errors[]}` | ### **Safety Management** | Action | Parameters | Returns | |--------|------------|---------| | `set_safe_mode` | `enabled=true` | `{status, safe_mode}` | | `get_safety_status` | — | `{status, safe_mode_enabled, dangerous_patterns, dangerous_actions[]}` | --- ## 📝 **Macro Format** Recorded macros are JSON with this structure: ```json { "events": [ { "action": "click", "params": {"x": 100, "y": 50}, "wait": 500 }, { "action": "type", "params": {"text": "Hello"}, "wait": 200 }, { "action": "press_key", "params": {"key": "return"}, "wait": 100 } ] } ``` - `action` — action name - `params` — action parameters - `wait` — milliseconds to wait before next action --- ## 🔧 **Advanced: Mouse Move Debouncing** To avoid recording hundreds of `move_mouse` events during a smooth drag, the recorder uses **debouncing**: - When you move the mouse, events are **suppressed** during movement - After you **stop moving** for `N` seconds (default: 1 sec), the **final position** is recorded - This reduces macro size dramatically while preserving intended end positions - Configurable via GUI: set debounce time (0.1–10 seconds) **Example:** - Fast horizontal line → 1 `move_mouse` event (end coordinates) - Slow, stop-and-go → multiple `move_mouse` events (one per "stop") --- ## 🧪 **Testing** Run the unit test suite: ```bash python scripts/test_automation.py ``` Output: ``` test_dry_run_click ... ok test_get_active_window ... ok test_safe_mode_blocks_dangerous ... ok ... Ran 13 tests OK ``` --- ## 📊 **Logging** All actions logged to: `~/.openclaw/skills/desktop-automation-logs/automation_YYYY-MM-DD.log` Example: ``` [2026-03-15 10:23:45] [INFO] ActionManager: ActionManager initialized with safe_mode=True [2026-03-15 10:23:46] [INFO] ActionManager: Clicked at (100, 50) with left button [2026-03-15 10:23:47] [INFO] ActionManager: Typed: Hello World ``` --- ## ⚙️ **Configuration** ### Environment Variables ```bash # Override log directory export AUTOMATION_LOG_DIR=~/my_logs # Disable safe mode globally (NOT recommended) export AUTOMATION_SAFE_MODE=false ``` --- ## 🐛 **Troubleshooting** ### "pyautogui failsafe triggered" Move mouse to corner of screen to stop. ### OCR returns empty text - Ensure Tesseract is installed correctly - Check image quality (high contrast helps) - Try `read_text_ocr` instead of `find_text_on_screen` ### Image recognition not finding template - Ensure template image exists and is correct format (PNG, JPG) - Try lower confidence threshold (e.g., 0.85 instead of 0.95) - Use `find_image_multiscale` to detect at different scales ### Actions blocked by safe mode This is intentional. To run dangerous actions: ```yaml action: set_safe_mode params: enabled: false ``` Then execute your action. Re-enable safe mode immediately after: ```yaml action: set_safe_mode params: enabled: true ``` --- ## 📄 **License** MIT License. See LICENSE file. --- ## 📚 **Files Structure** ``` desktop-automation-ultra-local/ ├── SKILL.md (This file) ├── requirements.txt (Python dependencies) ├── lib/ │ ├── actions.py (Core click/type/drag actions) │ ├── image_recognition.py (OpenCV template matching) │ ├── ocr_engine.py (Tesseract OCR) │ ├── macro_player.py (Record/playback macros) │ ├── safety_manager.py (Safe mode, blocking) │ └── utils.py (Logging, helpers) ├── scripts/ │ └── test_automation.py (Unit tests) └── recorded_macro/ (Output: saved macros) ``` --- ## ✅ **Validation Checklist** - [x] All modules have proper error handling - [x] Thread safety implemented (locks) - [x] Safe mode enabled by default - [x] Dry-run mode on all actions - [x] Comprehensive logging - [x] Unit tests (13 tests) - [x] UTF-8 encoding for all text - [x] No hardcoded paths (uses expanduser) - [x] Graceful fallbacks for missing dependencies - [x] Documentation complete **Status: PRODUCTION READY** ✅ --- *Last updated: 2026-03-15* *Version: 2.0.0*


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)