Skill: ravenquasar/vision-helper (version 1.0.0)

--- name: vision-helper description: Analyze images using local or cloud vision models via Ollama. Use when you need to identify screenshots, analyze UI elements, read image content, or perform OCR. Triggers on: "analyze image", "screenshot", "what's in this image", "OCR", "识图", "看图", "截图识别". --- # 📸 Vision Helper — Image Analysis Analyze images using vision models via Ollama, with extended timeout support for cloud-based models. ## Why Not Use the Built-in `image` Tool? The built-in `image` tool has limited timeout settings that cause failures with cloud vision models (which often need 40–120 seconds). This skill calls the Ollama API directly with a 180-second timeout, supporting both local and cloud models reliably. It also bypasses the built-in tool's file path restrictions, allowing analysis of images from any readable directory. ## Usage ### Basic ```bash # Analyze an image (default: English description) python3 <skill-dir>/scripts/analyze_image.py <image_path> # With a custom prompt python3 <skill-dir>/scripts/analyze_image.py <image_path> "Is this a chess game? Describe the board state" # With a specific model python3 <skill-dir>/scripts/analyze_image.py <image_path> "Describe content" kimi-k2.5:cloud ``` > `<skill-dir>` resolves to your OpenClaw skill installation directory, typically `~/.openclaw/workspace/skills/vision-helper/`. ### In Conversation When you need to analyze an image, use the `exec` tool: ``` exec: python3 <skill-dir>/scripts/analyze_image.py /path/to/image.png "What do you see?" ``` **Important:** Set exec timeout to 120–180 seconds, as cloud vision models are slow. ### Screenshot + Analysis Workflow #### Option A: Browser screenshot → analyze ``` 1. browser(action="screenshot") → get screenshot path (MEDIA: xxx) 2. exec("<skill-dir>/scripts/analyze_image.py <screenshot_path> 'Describe this UI'") 3. Act on the analysis result ``` #### Option B: Desktop screenshot → analyze **macOS:** ``` 1. exec("screencapture -x /tmp/screen.png") 2. exec("<skill-dir>/scripts/analyze_image.py /tmp/screen.png 'Describe the desktop'") ``` **Linux:** ``` 1. exec("gnome-screenshot -f /tmp/screen.png") — or — exec("import /tmp/screen.png") # ImageMagick — or — exec("scrot /tmp/screen.png") 2. exec("<skill-dir>/scripts/analyze_image.py /tmp/screen.png 'Describe the desktop'") ``` #### Option C: Game/App UI → analyze → act ``` 1. Screenshot the current screen 2. Use vision-helper to identify UI elements, buttons, text 3. Execute clicks/input based on the analysis ``` ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `VISION_MODEL` | `gemma4:31b` | Default vision model | | `VISION_TIMEOUT` | `180` | Request timeout in seconds | | `OLLAMA_API_URL` | `http://localhost:11434/api/chat` | Ollama API endpoint | ## Supported Models | Model | Vision | Speed | Recommendation | |-------|--------|-------|----------------| | `gemma4:31b` | ✅ | Local, fast | ⭐ **Primary** (privacy, no API needed) | | `kimi-k2.6:cloud` | ✅ | 40–120s | 🔬 Advanced (high quality, cloud) | | `kimi-k2.5:cloud` | ✅ | 40–90s | Alternative cloud option | | `qwen3.5:cloud` | ✅ | 30–60s | Fast cloud recognition | | `qwen3.5:397b-cloud` | ✅ | 40–90s | High quality cloud | | `gemma4:31b` | ✅ | Local, fast | Privacy-first (runs offline) | Note: Cloud models require the model to be available in your Ollama instance. Use `VISION_MODEL` env var to switch. ## FAQ ### Q: Can I use the built-in `image` tool instead? A: It works for local models but will time out on cloud vision models. Always prefer this skill's script for reliable results. ### Q: What image formats are supported? A: PNG, JPG, JPEG, GIF, WebP, BMP, TIFF, SVG. Maximum file size: 20 MB. ### Q: Where should I save screenshots? A: Any readable directory works — `/tmp/`, your workspace, etc. This script has no path restrictions. ### Q: How do I use a Chinese prompt? A: Pass it as the second argument: `python3 <skill-dir>/scripts/analyze_image.py /tmp/img.png "请描述这张图片的内容"` ## Automation Ideas - **Game automation**: Screenshot → analyze game state → decide next action - **Browser verification**: Screenshot → verify page loaded correctly - **Desktop monitoring**: Periodic screenshots → detect changes - **UI testing**: Screenshot → verify rendered output - **OCR**: Extract text content from images

path: clawhub.json
content: {
  "name": "vision-helper",
  "displayName": "Vision Helper — AI Image Analysis",
  "version": "1.0.0",
  "description": "Analyze images using local or cloud vision models via Ollama. Bypasses built-in image tool timeout limits with extended 180s timeout, file validation, and cross-platform screenshot support.",
  "author": "Frieren",
  "license": "MIT",
  "pricing": {
    "model": "free"
  },
  "tags": ["vision", "image", "ocr", "screenshot", "ollama", "gemma", "kimi"],
  "minOpenClawVersion": "1.2.0"
}

path: scripts/analyze_image.py
content: #!/usr/bin/env python3
"""
Image analysis helper for OpenClaw.
Uses vision models (via Ollama) for image understanding, with extended timeout
to support cloud-based models.

Usage:
    python3 analyze_image.py <image_path> [prompt] [model]

Examples:
    python3 analyze_image.py /tmp/screenshot.png
    python3 analyze_image.py /tmp/screen.png "Is this a chess game?"
    python3 analyze_image.py /tmp/test.png "Describe this" kimi-k2.5:cloud

Output: Plain text description from the vision model.
"""

import sys
import json
import base64
import urllib.request
import urllib.error
import os

# Default Ollama API endpoint (override with OLLAMA_API_URL env var)
OLLAMA_API = os.environ.get("OLLAMA_API_URL", "http://localhost:11434/api/chat")

# Default model — local first for privacy (override with VISION_MODEL env var)
DEFAULT_MODEL = os.environ.get("VISION_MODEL", "gemma4:31b")
DEFAULT_PROMPT = "Describe this image in detail, including all visible objects, text, UI elements, and colors."
DEFAULT_TIMEOUT = int(os.environ.get("VISION_TIMEOUT", "180"))  # 3 minutes, override with env var

# File validation constants
MAX_FILE_SIZE = 20 * 1024 * 1024  # 20 MB
ALLOWED_EXTS = {'.png', '.jpg', '.jpeg', '.gif', '.webp', '.bmp', '.tiff', '.svg'}


def validate_image_path(image_path: str) -> str:
    """Validate and resolve image path. Returns resolved path or raises ValueError."""
    # Resolve ~ and normalize
    resolved = os.path.abspath(os.path.expanduser(image_path))

    # Check for path traversal
    if '..' in image_path:
        raise ValueError(f"Path traversal not allowed: {image_path}")

    # Check file extension
    _, ext = os.path.splitext(resolved)
    if ext.lower() not in ALLOWED_EXTS:
        raise ValueError(
            f"Unsupported image format: {ext}. "
            f"Allowed: {', '.join(sorted(ALLOWED_EXTS))}"
        )

    # Check file exists
    if not os.path.exists(resolved):
        raise ValueError(f"Image file not found: {image_path}")

    # Check file size
    file_size = os.path.getsize(resolved)
    if file_size > MAX_FILE_SIZE:
        raise ValueError(
            f"Image too large: {file_size / 1024 / 1024:.1f} MB "
            f"(max {MAX_FILE_SIZE / 1024 / 1024:.0f} MB)"
        )

    if file_size == 0:
        raise ValueError("Image file is empty")

    return resolved


def analyze_image(image_path: str, prompt: str = DEFAULT_PROMPT, model: str = DEFAULT_MODEL, timeout: int = DEFAULT_TIMEOUT) -> str:
    try:
        resolved_path = validate_image_path(image_path)
    except ValueError as e:
        return f"Error: {e}"

    # Read and encode image
    try:
        with open(resolved_path, "rb") as f:
            img_data = f.read()
    except OSError as e:
        return f"Error: Failed to read image file"

    b64 = base64.b64encode(img_data).decode()

    # Build Ollama chat request with image
    payload = {
        "model": model,
        "messages": [
            {
                "role": "user",
                "content": prompt,
                "images": [b64]
            }
        ],
        "stream": False
    }

    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        OLLAMA_API,
        data=data,
        headers={"Content-Type": "application/json"}
    )

    try:
        resp = urllib.request.urlopen(req, timeout=timeout)
        result = json.loads(resp.read().decode())
        content = result.get("message", {}).get("content", "")
        if not content:
            eval_count = result.get("eval_count", 0)
            if eval_count > 0:
                return "(Model returned empty content but ran successfully)"
            return "Error: No content returned from model"
        return content
    except urllib.error.HTTPError as e:
        return "HTTP request failed"
    except urllib.error.URLError:
        return "Connection failed: unable to reach the vision model API"
    except Exception:
        return "Analysis failed: unexpected error"


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python3 analyze_image.py <image_path> [prompt] [model]")
        print("  image_path  Path to image file (supports: png, jpg, jpeg, gif, webp, bmp, tiff, svg)")
        print("  prompt      Optional description prompt (default: general description in English)")
        print("  model       Optional Ollama model name (default: gemma4:31b, env: VISION_MODEL)")
        sys.exit(1)

    img_path = sys.argv[1]
    user_prompt = sys.argv[2] if len(sys.argv) > 2 else DEFAULT_PROMPT
    user_model = sys.argv[3] if len(sys.argv) > 3 else DEFAULT_MODEL

    result = analyze_image(img_path, user_prompt, user_model)
    print(result)

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)