Skill: bencoremans/cliproxy-media (version 1.0.3)

--- name: cliproxy-media description: Analyze images (jpg, png, gif, webp) and PDFs via CLIProxyAPI — a Claude Max proxy that routes requests through your subscription at zero extra cost. Use this skill whenever you need to analyze, describe, or extract information from an image or photo ("analyze image", "describe photo", "what is in this picture"), read or summarize a PDF document ("read PDF", "summary of this document"), or process any media file via a CLIProxy-compatible endpoint ("process media via proxy", "cliproxy vision", "cliproxy media"). NEVER use the built-in `image` or `pdf` tools when using CLIProxyAPI — they fall back to direct Anthropic API which requires separate credits. Use this skill instead for all vision and document analysis tasks. --- # cliproxy-media **Source:** https://github.com/bencoremans/site/tree/main/skills/cliproxy-media Analyze images and PDFs via CLIProxyAPI (Claude Max subscription, zero extra cost). ## Setup Set the endpoint to your CLIProxy instance: ```bash export CLIPROXY_URL=http://your-host:8317/v1/messages ``` For Docker setups, replace `your-host` with your container hostname (e.g. `cliproxyapi`, `localhost`, or the container IP). ## Quick start ```bash # Analyze an image python3 skills/cliproxy-media/scripts/analyze.py /path/to/image.jpg "What is in this image?" # Read a PDF python3 skills/cliproxy-media/scripts/analyze.py /path/to/document.pdf "Give a summary" # Compare multiple images python3 skills/cliproxy-media/scripts/analyze.py img1.jpg img2.jpg "Compare these images" # With streaming (output appears immediately) python3 skills/cliproxy-media/scripts/analyze.py --stream image.jpg "Describe in detail" # With system prompt python3 skills/cliproxy-media/scripts/analyze.py --system "You are a medical expert" scan.jpg "What do you see?" # With higher token limit python3 skills/cliproxy-media/scripts/analyze.py --max-tokens 4096 document.pdf "Extensive analysis" ``` ## What works ✅ / What doesn't ❌ ### ✅ Supported file types | Type | Format | Note | |------|--------|------| | Image | `.jpg` / `.jpeg` | Requires valid JPEG data | | Image | `.png` | Fully supported | | Image | `.gif` | Fully supported | | Image | `.webp` | Fully supported | | Document | `.pdf` | Base64-encoded, via `document` content type | | Image via URL | `http://` / `https://` | Direct URL reference, no download needed | **Multiple files at once:** Provide multiple paths before the question. Max ~100 per request (Anthropic limit). ### ❌ Not supported - **Office files** (`.docx`, `.xlsx`, `.pptx`) — Workaround: convert to PDF - **Audio** (`.mp3`, `.wav`, `.ogg`) — Use Whisper for transcription - **Video** (`.mp4`, `.mov`, `.avi`) — Not supported by the model - **Other document types** (`.txt`, `.html`, `.md` as document) — Send text directly as a string ## ⚠️ System prompt warning CLIProxyAPI accepts **only** the array notation for system prompts. The string notation is **silently ignored** — the model does not see it, but you also won't get an error message! ```python # ❌ DOES NOT WORK — ignored without error message payload["system"] = "You are an expert." # ✅ WORKS — always use array notation payload["system"] = [{"type": "text", "text": "You are an expert."}] ``` The `--system` argument in `analyze.py` automatically uses the correct array notation. ## Configuration (env vars) | Variable | Default | Description | |----------|---------|-------------| | `CLIPROXY_URL` | `http://localhost:8317/v1/messages` | Full endpoint URL | | `CLIPROXY_MODEL` | `claude-sonnet-4-6` | Model to use | Example: ```bash export CLIPROXY_URL=http://localhost:8317/v1/messages export CLIPROXY_MODEL=claude-opus-4-6 python3 skills/cliproxy-media/scripts/analyze.py image.jpg "question" ``` ## Additional options ``` --stream Streaming output via SSE (output appears immediately) --system TEXT System prompt (automatically sent as array) --max-tokens N Maximum output tokens (default: 1024) --model MODEL Model override (overrides CLIPROXY_MODEL) --url URL Endpoint override (overrides CLIPROXY_URL) ``` ## Compatibility This script works with any API that supports the Anthropic Messages format: | Provider | Compatible | Note | |----------|-----------|------| | **CLIProxyAPI** | ✅ Yes | Primarily tested, system prompt array required | | **OpenRouter** | ✅ Yes | Use Bearer token instead of `x-api-key: dummy` | | **LiteLLM** | ✅ Yes | As proxy for Anthropic format | | **Anthropic direct** | ✅ Yes | Use `ANTHROPIC_API_KEY` as x-api-key | **Note for non-CLIProxy endpoints:** Some proxies do accept string notation for system prompts. Always use array notation for maximum compatibility. ## Known limitations of CLIProxyAPI - `temperature` and `top_p` may **not** be used at the same time (HTTP 400) - PDF as document with URL source does not work (`Unable to download the file`) - Only `claude-sonnet-4-6` and `claude-opus-4-6` available (haiku is deprecated) - `inference_geo` is always `not_available` in the response ## Direct Python API If you want to call the script from your own Python code: ```python import subprocess, json result = subprocess.run( ["python3", "skills/cliproxy-media/scripts/analyze.py", "image.jpg", "Describe this"], capture_output=True, text=True ) print(result.stdout) ``` Or use the built-in exec tool: ``` exec: python3 skills/cliproxy-media/scripts/analyze.py /path/to/image.jpg "question" ```

path: scripts/analyze.py
content: #!/usr/bin/env python3
"""
analyze.py — Universal media analysis script for CLIProxyAPI

Usage:
  python3 analyze.py image.jpg "question"
  python3 analyze.py doc.pdf "summary"
  python3 analyze.py img1.jpg img2.jpg "compare"
  python3 analyze.py --stream image.jpg "describe"
  python3 analyze.py --system "you are an expert" image.jpg "question"
  python3 analyze.py --max-tokens 4096 doc.pdf "analysis"
  python3 analyze.py --model claude-opus-4-6 image.jpg "question"

Supported file types:
  Images:    .jpg/.jpeg, .png, .gif, .webp
  Documents: .pdf
  URLs:      http:// and https:// (referenced directly as image URL)

Configuration via env vars:
  CLIPROXY_URL    Endpoint URL (default: http://localhost:8317/v1/messages)
  CLIPROXY_MODEL  Model (default: claude-sonnet-4-6)

⚠️  System prompts are always sent as an array (CLIProxy requirement).
    String notation is silently ignored by the proxy.
"""

import argparse
import base64
import json
import mimetypes
import os
import sys
import urllib.error
import urllib.request
from pathlib import Path

# Standaard configuratie
DEFAULT_ENDPOINT = "http://localhost:8317/v1/messages"
DEFAULT_MODEL = "claude-sonnet-4-6"
DEFAULT_MAX_TOKENS = 1024

# Media type mapping op extensie
IMAGE_TYPES = {
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".png": "image/png",
    ".gif": "image/gif",
    ".webp": "image/webp",
}


def detect_media_type(path: str) -> str:
    """Bepaal media type op basis van bestandsextensie."""
    ext = Path(path).suffix.lower()
    if ext in IMAGE_TYPES:
        return IMAGE_TYPES[ext]
    if ext == ".pdf":
        return "application/pdf"
    # Fallback: probeer via mimetypes module
    mime, _ = mimetypes.guess_type(path)
    if mime:
        return mime
    raise ValueError(f"Onbekende bestandsextensie: {ext} — ondersteund: jpg/jpeg/png/gif/webp/pdf")


def load_file_as_base64(path: str) -> str:
    """Laad een bestand en codeer als base64 string."""
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


def build_content_block(file_path: str) -> dict:
    """
    Maak een content block voor een bestand.
    Ondersteunt: afbeeldingen (base64), PDFs (base64 document), URLs (image url).
    """
    # URL: directe referentie (alleen voor afbeeldingen — document URL werkt niet)
    if file_path.startswith("http://") or file_path.startswith("https://"):
        return {
            "type": "image",
            "source": {
                "type": "url",
                "url": file_path,
            },
        }

    # Lokaal bestand
    if not os.path.exists(file_path):
        raise FileNotFoundError(f"Bestand niet gevonden: {file_path}")

    media_type = detect_media_type(file_path)
    data = load_file_as_base64(file_path)

    if media_type == "application/pdf":
        # PDFs als document type (enige geldige document media type)
        return {
            "type": "document",
            "source": {
                "type": "base64",
                "media_type": "application/pdf",
                "data": data,
            },
        }
    elif media_type.startswith("image/"):
        # Afbeeldingen als image type
        return {
            "type": "image",
            "source": {
                "type": "base64",
                "media_type": media_type,
                "data": data,
            },
        }
    else:
        raise ValueError(
            f"Niet ondersteund media type: {media_type}\n"
            f"Ondersteund: afbeeldingen (jpg/png/gif/webp) en PDFs.\n"
            f"Workarounds:\n"
            f"  - Office bestanden: converteer naar PDF\n"
            f"  - Audio: gebruik Whisper voor transcriptie\n"
            f"  - Tekst: kopieer de inhoud direct in je vraag"
        )


def stream_response(resp) -> str:
    """Verwerk SSE streaming response, print naar stdout, retourneer volledige tekst."""
    full_text = ""
    for line in resp:
        line = line.decode("utf-8").rstrip()
        if not line.startswith("data: "):
            continue
        data_str = line[6:]
        if data_str == "[DONE]":
            break
        try:
            event = json.loads(data_str)
        except json.JSONDecodeError:
            continue

        event_type = event.get("type", "")

        if event_type == "content_block_delta":
            delta = event.get("delta", {})
            if delta.get("type") == "text_delta":
                chunk = delta.get("text", "")
                print(chunk, end="", flush=True)
                full_text += chunk
        elif event_type == "message_stop":
            break

    print()  # Newline na streaming
    return full_text


def analyze(
    files: list,
    question: str,
    system: str = None,
    max_tokens: int = DEFAULT_MAX_TOKENS,
    model: str = None,
    endpoint: str = None,
    stream: bool = False,
) -> str:
    """
    Analyseer een of meerdere bestanden met een vraag.

    Args:
        files:      Lijst van bestandspaden of URLs
        question:   De vraag of instructie
        system:     Optionele system prompt (wordt als array verstuurd — CLIProxy vereiste)
        max_tokens: Maximum output tokens
        model:      Model override (default: CLIPROXY_MODEL env var of claude-sonnet-4-6)
        endpoint:   Endpoint URL override (default: CLIPROXY_URL env var)
        stream:     Gebruik SSE streaming output

    Returns:
        De gegenereerde tekst als string
    """
    # Configuratie
    actual_endpoint = endpoint or os.environ.get("CLIPROXY_URL", DEFAULT_ENDPOINT)
    actual_model = model or os.environ.get("CLIPROXY_MODEL", DEFAULT_MODEL)

    # Bouw content blocks
    content = []
    for f in files:
        content.append(build_content_block(f))
    content.append({"type": "text", "text": question})

    # Bouw request payload
    payload = {
        "model": actual_model,
        "max_tokens": max_tokens,
        "messages": [{"role": "user", "content": content}],
    }

    # System prompt — ALTIJD array-notatie (string wordt silently genegeerd door CLIProxy)
    if system:
        payload["system"] = [{"type": "text", "text": system}]

    # Streaming flag
    if stream:
        payload["stream"] = True

    # HTTP request
    req = urllib.request.Request(
        actual_endpoint,
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "x-api-key": "dummy",  # CLIProxy gebruikt eigen auth
        },
    )

    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            if stream:
                return stream_response(resp)
            else:
                response_data = json.loads(resp.read())
                return response_data["content"][0]["text"]

    except urllib.error.HTTPError as e:
        error_body = e.read().decode("utf-8")
        try:
            error_json = json.loads(error_body)
            error_msg = error_json.get("error", {}).get("message", error_body)
        except json.JSONDecodeError:
            error_msg = error_body
        raise RuntimeError(f"HTTP {e.code}: {error_msg}") from e

    except urllib.error.URLError as e:
        raise RuntimeError(
            f"Kan endpoint niet bereiken: {actual_endpoint}\n"
            f"Fout: {e.reason}\n"
            f"Controleer of CLIProxyAPI draait en CLIPROXY_URL correct is."
        ) from e


def main():
    parser = argparse.ArgumentParser(
        description="Analyseer afbeeldingen en PDFs via CLIProxyAPI",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Voorbeelden:
  python3 analyze.py image.jpg "Wat staat er op deze foto?"
  python3 analyze.py document.pdf "Geef een samenvatting"
  python3 analyze.py img1.jpg img2.jpg "Vergelijk deze afbeeldingen"
  python3 analyze.py --stream image.jpg "Beschrijf in detail"
  python3 analyze.py --system "Je bent een medisch expert" scan.jpg "Wat zie je?"
  python3 analyze.py --max-tokens 4096 rapport.pdf "Uitgebreide analyse"
  python3 analyze.py https://example.com/photo.jpg "Beschrijf"

Env vars:
  CLIPROXY_URL    Endpoint URL (default: http://localhost:8317/v1/messages)
  CLIPROXY_MODEL  Model (default: claude-sonnet-4-6)
        """,
    )

    parser.add_argument(
        "--stream",
        action="store_true",
        help="Gebruik SSE streaming (output verschijnt direct)",
    )
    parser.add_argument(
        "--system",
        type=str,
        default=None,
        help="System prompt (wordt als array verstuurd)",
    )
    parser.add_argument(
        "--max-tokens",
        type=int,
        default=DEFAULT_MAX_TOKENS,
        help=f"Maximum output tokens (default: {DEFAULT_MAX_TOKENS})",
    )
    parser.add_argument(
        "--model",
        type=str,
        default=None,
        help="Model override (default: CLIPROXY_MODEL of claude-sonnet-4-6)",
    )
    parser.add_argument(
        "--url",
        type=str,
        default=None,
        dest="endpoint",
        help="Endpoint URL override (default: CLIPROXY_URL)",
    )
    parser.add_argument(
        "args",
        nargs="+",
        help="[bestand1.jpg] [bestand2.pdf] ... [vraag]",
    )

    parsed = parser.parse_args()

    # Splits args: alles behalve het laatste is een bestand/URL, laatste is de vraag
    if len(parsed.args) < 2:
        parser.error(
            "Geef minimaal één bestand en een vraag op.\n"
            "Gebruik: analyze.py [--opties] bestand.jpg 'vraag'"
        )

    files = parsed.args[:-1]
    question = parsed.args[-1]

    try:
        result = analyze(
            files=files,
            question=question,
            system=parsed.system,
            max_tokens=parsed.max_tokens,
            model=parsed.model,
            endpoint=parsed.endpoint,
            stream=parsed.stream,
        )
        if not parsed.stream:
            print(result)
    except (RuntimeError, FileNotFoundError, ValueError) as e:
        print(f"Fout: {e}", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)