Skill: jaredforreal/glmv-grounding (version 1.0.5)

--- name: glmv-grounding description: > A skill that uses GLM-V native grounding capabilities for coordinate conversion, bounding-box visualization, and more. GLM-V native grounding can locate any target specified by the prompt in an image and output relative coordinates normalized to 0-1000 based on image size. Coordinate formats include 2D bounding box (default), 2D points, and 3D bounding box. GLM-V also supports spatiotemporal localization and tracking of multiple prompt-specified targets in videos, outputting 2D bounding boxes per second. metadata: openclaw: requires: env: - ZHIPU_API_KEY - GLM_GROUNDING_TIMEOUT primaryEnv: ZHIPU_API_KEY emoji: "🖼️" source: https://github.com/zai-org/GLM-V/tree/main/skills/glmv-grounding homepage: https://github.com/zai-org/GLM-V/tree/main/skills/glmv-grounding --- # GLMV-Grounding Skill Extract and visualize grounding results produced by GLM-V. Depending on the user prompt, grounding coordinates in model outputs may appear in different forms, including 2D bounding boxes, Objects Detection JSON, 2D points, 3D bounding boxes, and target-tracking JSON. **Note**: GLM-V outputs coordinates where **x and y are relative coordinates normalized from pixel coordinates x_pixel and y_pixel using image width W and height H (range 0-1000), i.e., x=round(x_pixel/W*1000), y=round(y_pixel/H*1000)**. The origin of the pixel coordinate system is the top-left corner. **Note**: If the prompt does not explicitly specify a grounding format (for example, "find the location of xxx" or "draw a box around xxx"), treat the request as 2D bounding boxes by default. ## When to use - **Use GLM-V to ground targets in images**: obtain grounding results in an image for any prompt-described target, with output formats such as 2D bounding box (default), 2D points, and 3D bounding box. - **Use GLM-V to track targets in videos**: obtain tracking results in a video for any prompt-described target, with output format like {"0": [{"label": ..., "bbox_2d": ...}, ...], ...}. - **Use utility functions for extraction, conversion, and visualization**: extract coordinates, points, and JSON from natural text; normalize and de-normalize coordinates; visualize boxes, points, 3D boxes, and video tracking results. ## Setup your API Key Configure ZHIPU_API_KEY to call the GLM-V API. 1. Get your API key: https://www.bigmodel.cn/usercenter/proj-mgmt/apikeys 2. Configure it with: ``` python scripts/config_setup.py setup --api-key YOUR_KEY ``` ## Security & Transparency - **Primary API key env**: `ZHIPU_API_KEY` (required). - **Timeout env**: `GLM_GROUNDING_TIMEOUT` (optional, seconds, default `60`). - **API endpoint**: fixed to official Zhipu Chat Completions endpoint in CLI implementation. - **No dynamic key name switching**: the skill expects `ZHIPU_API_KEY` consistently. - **URL/local file handling**: the skill can read local files or fetch user-provided URLs for processing/visualization; URL inputs are restricted to public http/https targets (localhost/private network targets are rejected). ## Runtime Dependencies Install dependencies before use: ```bash pip install -r scripts/requirements.txt ``` Main packages used by this skill: - `requests` - `Pillow` - `opencv-python` - `numpy` - `matplotlib` - `decord` System dependency for video visualization: - `ffmpeg` ## General workflow ``` Input (image or video + Prompt) | ▼ Run glm_grounding_cli.py to get grounding results (natural language) | ▼ Return results (grounding results, visualized image or video) ``` ## How to Use ### Run glm_grounding_cli.py to get grounding results - Ground any target in an image ``` python scripts/glm_grounding_cli.py --image-url "URL provided by user" --prompt "description of target for grounding" ``` - Track any target in a video ``` python scripts/glm_grounding_cli.py --video-url /path/to/image.jpg --prompt "description of target for tracking" --visualize --visualization-dir "./vis" ``` ### Reply with grounding results After receiving a grounding prompt from the user, your direct reply should be natural language that includes grounding coordinates. Coordinates $x$ and $y$ are relative values in [0, 1000], computed as: $$ x = round(x_{pixel} / W * 1000) \\ y = round(y_{pixel}/H*1000) $$ where $x_{pixel}, y_{pixel}$ are pixel coordinates with origin (0, 0) at the top-left corner of the image, and W/H are the image width/height. **Unless otherwise specified, grounding results should use the following Python data formats**: - _2D bounding boxes_: `[[x1, y1, x2, y2], ...]`, extracted grounding result is a list of boxes, each box has 4 coordinate values - _2D points_: `[[x, y], ...]`, extracted grounding result is a list of points, each point has 2 coordinate values - _2D polygon_: `[[x1, y1], [x2, y2], ...]`, extracted grounding result is a polygon coordinate list, each vertex has 2 coordinate values - _3D bounding boxes_: `[{"bbox_3d":[x_center, y_center, z_center, x_size, y_size, z_size, roll, pitch, yaw],"label":"category"}, ...]`, extracted grounding result is a JSON list where each object contains a category label and one 3D box with 8 coordinate values - _Objects Detection JSON_: `[{'label': 'category', 'bbox_2d': [x1, y1, x2, y2]}, ...]`, extracted grounding result is a JSON list where each object contains a category label and one box - _Video Objects Tracking JSON_: `{0: [{'label': 'car-1', 'bbox_2d': [1,2,3,4]}, {'label': 'car-2', 'bbox_2d': [2,3,4,5]}], 1: [{'label': 'car-2', 'bbox_2d': [4,5,6,7]}, {'label': 'person-1', 'bbox_2d': [10,20,30,40]}]}`, extracted grounding result is a JSON object whose keys are video frame indices and values are lists of JSON objects, each containing a category label and one 2D box ## Python example ```shell # 1. User grounding request and your reply image=https://example.com/image.jpg prompt="Please box all people wearing Santa hats in the image and tell me their coordinates. Use red boxes, line thickness 3, and label format 'SantaHat-i'." # 2. Get grounding results python scripts/glm_grounding_cli.py --image-url $image --prompt $prompt --visualize --visualization-dir "./vis" # { # "ok": True, # "grounding_result": [[100, 200, 300, 400], [500, 600, 700, 800]], # "visualizations_result": ( # {"visualized_image": "./vis/image_vis.jpg"} # ), # "raw_result": "1. Person 1: box [100, 200, 300, 400]\n2. Person 2: box [500, 600, 700, 800]. The box format is [x1, y1, x2, y2], where (x1, y1) is the top-left corner and (x2, y2) is the bottom-right corner.", # "error": None, # "source": source, # } ``` ## Utility function quick reference | Function | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `parse_coordinates_from_response(response_str, coords_type='bbox', init_context_window=2000, max_context_window=-1)` | Parse and extract all coordinate results from model responses (supports 2D bbox, point, polygon) | | `parse_3d_boxes_from_response(response_str, max_context_window=-1)` | Parse and extract all 3D boxes and labels from model responses (strict and loose matching) | | `parse_detection_from_response(response_str, max_context_window=-1)` | Parse and extract all 2D detection results from model responses (Objects Detection JSON format) | | `parse_mot_from_response(response_str, max_context_window=-1)` | Parse and extract all video object tracking results from model responses (Video Objects Tracking JSON format) | | `visualize_boxes(img_path=None, img_bytes=None, boxes=[], labels=None, renormalize=False, save_path=None, return_b64=False, save_optimized=True, **kwargs)` | Draw 2D boxes on images with labels, custom colors, and line thickness | | `visualize_points(img_path=None, img_bytes=None, points=[], labels=None, renormalize=False, diameters=None, save_path=None, return_b64=False, save_optimized=True, distinct_colors=False, colors=None)` | Draw points on images with labels, custom size, and colors | | `visualize_3d_boxes_glmv_simple(image_path, cam_params, bbox_3d_list, image_bytes=None, coord_format='xyzwhlpyr', save_path=None, save_optimized=False, return_b64=False, **kwargs)` | Draw projected 3D boxes on images using camera intrinsics (supports rotation and multiple coordinate formats) | | `visualize_mot(video_path=None, video_bytes=None, mot_js=None, renormalize=False, save_path=None, return_b64=False, distinct_colors=True, **kwargs)` | Draw Video Objects Tracking boxes on each video frame with labels | ## Common errors - **Coordinate values exceed 1000**: if extracted coordinate values are greater than 1000, the model may have produced unnormalized coordinates due to prompt effects. Extract the target phrase from the user request (for example, "people wearing Santa hats"), then query the model again and explicitly require output coordinates to be relative values normalized to 0-1000 based on image size (for example, "Please box all people wearing Santa hats in the image and tell me their coordinates. Ensure the output coordinates are relative values normalized to 0-1000 based on image size.").

path: scripts/glm_grounding_cli.py
content: #!/usr/bin/env python3
"""
GLM-Grounding CLI

Command-line interface for GLM-V vision model.

Usage:
    python scripts/glm_grounding_cli.py --file-url "URL" --prompt "description of target for grounding"
    python scripts/glm_grounding_cli.py --file-url "URL" --prompt "description of target for grounding" --visualize --visualization-dir "./vis"
"""

import argparse
import base64
import io
import json
import logging
import mimetypes
import os
import socket
import sys
import ipaddress
from pathlib import Path
from typing import Any
from urllib.parse import urlparse

from utils_boxes import (
    parse_coordinates_from_response,
    visualize_boxes,
)
from utils_detection import parse_detection_from_response
from utils_video import parse_mot_from_response, visualize_mot

try:
    import requests
except ImportError:
    print(
        "Error: Missing dependency 'requests'. Install with: pip install -r scripts/requirements.txt",
        file=sys.stderr,
    )
    sys.exit(2)

# Fix Windows console encoding
if sys.platform == "win32":
    sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
    sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")

logger = logging.getLogger(__name__)

# =============================================================================
# Constants
# =============================================================================

DEFAULT_TIMEOUT = 60  # seconds
DEFAULT_MODEL = "glm-5v-turbo"
API_GUIDE_URL = "https://open.bigmodel.cn/usercenter/apikeys"
DEFAULT_API_URL = "https://open.bigmodel.cn/api/paas/v4/chat/completions"

# =============================================================================
# Environment
# =============================================================================

_env_loaded = False


def _load_env():
    """Load .env file if available."""
    global _env_loaded
    if _env_loaded:
        return

    env_file = Path(__file__).parent.parent / ".env"
    if env_file.exists():
        try:
            with open(env_file, "r", encoding="utf-8") as f:
                for line in f:
                    line = line.strip()
                    if not line or line.startswith("#") or "=" not in line:
                        continue
                    key, value = line.split("=", 1)
                    key = key.strip()
                    value = value.strip()
                    if key and key not in os.environ:
                        os.environ[key] = value
        except OSError as e:
            logger.debug(f"Failed to load .env: {e}")

    _env_loaded = True


def _get_env(key: str, *fallback_keys: str) -> str:
    """Get environment variable with fallback keys."""
    _load_env()
    value = os.getenv(key, "").strip()
    if value:
        return value
    for fallback in fallback_keys:
        value = os.getenv(fallback, "").strip()
        if value:
            logger.debug(f"Using fallback env var: {fallback}")
            return value
    return ""


def get_config() -> tuple[str, str]:
    """
    Get API URL and key from environment.

    Returns:
        tuple of (api_url, api_key)

    Raises:
        ValueError: If not configured
    """
    [REDACTED_SECRET]"ZHIPU_API_KEY")

    if not api_key:
        raise ValueError(
            f"ZHIPU_API_KEY not configured. Get your API key at: {API_GUIDE_URL}"
        )

    # Security: use fixed official endpoint to avoid key exfiltration via custom URL.
    return DEFAULT_API_URL, api_key


# =============================================================================
# File Utilities
# =============================================================================


def _is_url(path: str) -> bool:
    """Check if the given path is a URL."""
    try:
        result = urlparse(path)
        return all([result.scheme, result.netloc]) and result.scheme in (
            "http",
            "https",
        )
    except Exception:
        return False


def _is_public_url(url: str) -> tuple[bool, str]:
    """Validate URL is http/https and not localhost/private network target."""
    try:
        parsed = urlparse(url)
        if parsed.scheme not in ("http", "https") or not parsed.netloc:
            return False, "Only http/https URLs are supported"

        host = parsed.hostname
        if not host:
            return False, "Invalid URL host"

        host_l = host.lower()
        if host_l in {"localhost", "127.0.0.1", "::1"}:
            return False, "Localhost URLs are not allowed"

        # Resolve and block private/link-local/loopback/reserved addresses.
        infos = socket.getaddrinfo(host, None)
        for info in infos:
            ip_str = info[4][0]
            ip = ipaddress.ip_address(ip_str)
            if (
                ip.is_private
                or ip.is_loopback
                or ip.is_link_local
                or ip.is_multicast
                or ip.is_reserved
                or ip.is_unspecified
            ):
                return False, f"URL resolves to non-public IP: {ip}"
        return True, ""
    except Exception as e:
        return False, f"URL validation failed: {e}"


def _encode_file(file_path: str) -> tuple[str, str]:
    """
    Encode a local file to base64 data URI.

    Returns:
        tuple of (data_uri, error_message)
        On success, error_message is empty.
    """
    path = Path(file_path)
    if not path.exists():
        return "", f"File not found: {file_path}"
    if not path.is_file():
        return "", f"Not a regular file: {file_path}"

    # Guess MIME type, fallback to application/octet-stream
    mime, _ = mimetypes.guess_type(str(path))
    if not mime:
        mime = "application/octet-stream"

    try:
        data = path.read_bytes()
        b64 = base64.b64encode(data).decode("ascii")
        return f"data:{mime};base64,{b64}", ""
    except Exception as e:
        return "", f"Failed to read file: {e}"


def load_media_bytes(source: str):
    """Get bytes for a media URL or local file path."""
    if _is_url(source):
        ok, reason = _is_public_url(source)
        if not ok:
            return None, f"Rejected URL for security reasons: {reason}"
        try:
            resp = requests.get(source, timeout=10)
            resp.raise_for_status()
            return resp.content, ""
        except Exception as e:
            return None, f"Failed to download image: {e}"
    else:
        try:
            with open(source, "rb") as f:
                return f.read(), ""
        except OSError as e:
            return None, f"Failed to read file: {e}"


# =============================================================================
# API Request
# =============================================================================


def _make_api_request(api_url: str, api_key: str, payload: dict) -> dict:
    """
    Make GLM-Grounding API request.

    Args:
        api_url: API endpoint URL
        api_key: API key
        payload: Request payload

    Returns:
        API response dict

    Raises:
        RuntimeError: On API errors
    """
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}

    timeout = float(os.getenv("GLM_GROUNDING_TIMEOUT", str(DEFAULT_TIMEOUT)))

    try:
        resp = requests.post(api_url, json=payload, headers=headers, timeout=timeout)
    except requests.Timeout:
        raise RuntimeError(f"API request timed out after {timeout}s")
    except requests.RequestException as e:
        raise RuntimeError(f"API request failed: {e}")

    # Handle HTTP errors
    if resp.status_code != 200:
        error_detail = ""
        try:
            error_body = resp.json()
            if isinstance(error_body, dict):
                error_detail = str(
                    error_body.get("error", {}).get("message", "")
                ).strip()
        except Exception:
            pass

        if not error_detail:
            error_detail = (resp.text[:200] or "No response body").strip()

        if resp.status_code == 401 or resp.status_code == 403:
            raise RuntimeError(
                f"Authentication failed ({resp.status_code}): {error_detail}"
            )
        elif resp.status_code == 429:
            raise RuntimeError(f"API rate limit exceeded (429): {error_detail}")
        elif resp.status_code >= 500:
            raise RuntimeError(
                f"API service error ({resp.status_code}): {error_detail}"
            )
        else:
            raise RuntimeError(f"API error ({resp.status_code}): {error_detail}")

    # Parse response
    try:
        result = resp.json()
    except Exception:
        raise RuntimeError(f"Invalid JSON response: {resp.text[:200]}")

    # Check API-level error
    if "error" in result:
        raise RuntimeError(f"API error: {result['error']}")

    return result


# =============================================================================
# Main API
# =============================================================================


def get_grounding_results(
    image: str | None,
    prompt: str,
    model: str = DEFAULT_MODEL,
    video: str | None = None,
    visualize: bool = False,
    visualization_dir: str | None = None,
    **options,
) -> dict[str, Any]:
    """
    Get visual grounding results by prompting GLM-V with an image/video and target description.

    Args:
        image: a URL or local file path to an image.
        prompt: Target description for visual grounding.
        model: Model to use (glm-4.6v, etc.)
        video: A URL or local file path to a video file.
        visualize: Whether to visualize the results.
        visualization_dir: Directory to save visualizations.
        **options: Additional API options:
            - enable_thinking: True - Whether to enable GLM-V's thinking mode. Default True.

    Returns:
        {
            "ok": True,
            "grounding_result": "Generated grounded results in coordinates or JSON format.",
            "visualizations_result": {"original_image_url": "path/to/visualization.jpg", ...} if visualize else None,
            "raw_result": { raw API result },
            "error": None,
            "source": "original source",
        }
        or on error:
        {
            "ok": False,
            "grounding_result": "",
            "visualizations_result": None,
            "raw_result": None,
            "error": {"code": "...", "message": "..."},
            "source": "original source",
        }
    """
    # Get config
    try:
        api_url, [REDACTED_SECRET])
    except ValueError as e:
        return _error("CONFIG_ERROR", str(e), image or video or "")

    if not prompt or not prompt.strip():
        return _error("INPUT_ERROR", "Prompt cannot be empty", image or video or "")

    if bool(image) == bool(video):
        return _error(
            "INPUT_ERROR", "Provide exactly one of image or video", image or video or ""
        )

    source = image or video or ""

    content = [{"type": "text", "text": prompt}]

    mm_content = []
    if image:
        if not _is_url(image):
            encoded_image, encode_err = _encode_file(image)
            if encode_err:
                return _error("INPUT_ERROR", encode_err, source)
            image_for_api = encoded_image
        else:
            image_for_api = image
        mm_content.append({"type": "image_url", "image_url": {"url": image_for_api}})
    elif video is not None:  # video
        if not _is_url(video):
            encoded_video, encode_err = _encode_file(video)
            if encode_err:
                return _error("INPUT_ERROR", encode_err, source)
            video_for_api = encoded_video
        else:
            video_for_api = video
        mm_content.append({"type": "video_url", "video_url": {"url": video_for_api}})

    content = [*mm_content, *content]
    messages = [{"role": "user", "content": content}]

    # Build request payload
    payload = {
        "model": model,
        "messages": messages,
    }

    if options.get("enable_thinking", False):
        payload["thinking"] = {"type": "disabled"}

    # Call API
    try:
        raw_result = _make_api_request(api_url, api_key, payload)
    except RuntimeError as e:
        return _error("API_ERROR", str(e), source)

    # Parsers in utils expect text; normalize response to string once.
    raw_text = json.dumps(raw_result, ensure_ascii=False)

    # Parse grounding result
    if video:
        grounding_result = parse_mot_from_response(raw_text)
        boxes = []
        labels = []
    elif "bbox_2d" in raw_text or "bbox_3d" in raw_text:
        grounding_result = parse_detection_from_response(raw_text)
        boxes = [item["bbox_2d"] for item in grounding_result]
        labels = [item.get("label", "") for item in grounding_result]
    else:
        grounding_result = parse_coordinates_from_response(raw_text)
        boxes = grounding_result
        labels = [str(_) for _ in grounding_result]

    # Visualzie
    if visualize:
        visualization_dir = visualization_dir or "./visualization"
        os.makedirs(visualization_dir, exist_ok=True)
        media_bytes, media_err = load_media_bytes(source)
        if media_err:
            return _error("VISUALIZE_ERROR", media_err, source)

        if image:
            visualized_img_path = os.path.join(
                visualization_dir, f"{Path(source).stem}_grounding_visualized.jpg"
            )
            visualize_boxes(
                img_bytes=media_bytes,
                boxes=boxes,
                labels=labels,
                renormalize=True,
                save_path=visualized_img_path,
            )
        else:
            visualized_video_path = os.path.join(
                visualization_dir, f"{Path(source).stem}_grounding_visualized.mp4"
            )
            visualize_mot(
                video_bytes=media_bytes,
                mot_js=grounding_result,
                save_path=visualized_video_path,
            )

    return {
        "ok": True,
        "grounding": grounding_result,
        "visualizations_result": (
            {"visualized_image": visualized_img_path}
            if visualize and image
            else (
                {"visualized_video": visualized_video_path}
                if visualize and video
                else None
            )
        ),
        "raw_result": raw_result,
        "error": None,
        "source": source,
    }


def _error(code: str, message: str, source: str = "") -> dict:
    """Create error response."""
    return {
        "ok": False,
        "text": "",
        "layout_details": None,
        "result": None,
        "error": {"code": code, "message": message},
        "source": source,
        "source_type": "url" if _is_url(source) else "file",
    }


# =============================================================================
# CLI Interface
# =============================================================================


def main():
    parser = argparse.ArgumentParser(
        description="GLM-Grounding - Grounding anything in images/videos with GLM-V",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  # Run for remote URL
  python scripts/glm_grounding_cli.py --image-url "https://example.com/image.jpg" --prompt "description of target for grounding"


  # Run for local video with visualization
  python scripts/glm_grounding_cli.py --video-url "https://example.com/video.mp4" --prompt "description of target for trackinging" --visualize --visualization-dir "./vis"

Configuration:
  Run: python scripts/config_setup.py setup
    Or set in .env: ZHIPU_API_KEY
        """,
    )

    parser.add_argument("--image-url", help="URL or local path to image/PDF")
    parser.add_argument("--prompt", "-p", required=True, help="Prompt for grounding")
    parser.add_argument("--video-url", help="URL or local path to video")
    parser.add_argument(
        "--visualize", action="store_true", help="Visualize grounding results"
    )
    parser.add_argument("--visualization-dir", help="Directory to save visualizations")

    args = parser.parse_args()

    if bool(args.image_url) == bool(args.video_url):
        parser.error("Provide exactly one of --image-url or --video-url")

    result = get_grounding_results(
        image=args.image_url,
        prompt=args.prompt,
        video=args.video_url,
        visualize=args.visualize,
        visualization_dir=args.visualization_dir,
    )

    print(result)
    # Exit code based on result
    sys.exit(0 if result["ok"] else 1)


if __name__ == "__main__":
    main()


path: scripts/utils_3d.py
content: """
      v4_____________________v5
      /|                    /|
     / |                   / |
    /  |                  /  |
   /___|_________________/   |
v0|    |                 |v1 |
  |    |       y   z     |   |
  |    |       ↑ ↗       |   |
  |    |       · → x	   |   |
  |    |_________________|___|
  |   / v7               |   /v6
  |  /                   |  /
  | /                    | /
  |/_____________________|/
  v3                     v2
"""

import math, io, base64, json, ast, re
import numpy as np
import cv2
from PIL import Image
import matplotlib.pyplot as plt


def _to_number(x):
    if isinstance(x, (int, float)):
        return x
    if isinstance(x, str):
        x = x.strip()
        if not x:
            return None
        try:
            return float(x) if ("." in x or "e" in x.lower()) else int(x)
        except Exception:
            return None
    return None


def visualize_3d_boxes_glmv_simple(
    image_path,
    cam_params,
    bbox_3d_list,
    image_bytes=None,
    coord_format="xyzwhlpyr",
    save_path=None,
    save_optimized=False,
    return_b64=False,
    **kwargs,
):
    """Draw multiple 3D bounding boxes on the same image and return matplotlib figure.
    Args:
      format: 'xyzwhlpyr' or 'xyzXYZpyr', the former means (x,y,z,z_size,y_size,x_size,pitch,yaw,roll), and the later means (x,y,z,x_size,y_size,z_size,pitch,yaw,roll).
    """
    colors = kwargs.get("colors", ["red"] * len(bbox_3d_list))
    colors = [
        tuple(int(c * 255) for c in plt.matplotlib.colors.to_rgb(_)) for _ in colors
    ]
    thickness = kwargs.get("thickness", [2] * len(bbox_3d_list))

    def rotate_xyz(_point, _pitch, _yaw, _roll):
        # 1. rotate around x
        x0, y0, z0 = _point
        x1 = x0
        y1 = y0 * math.cos(_pitch) - z0 * math.sin(_pitch)
        z1 = y0 * math.sin(_pitch) + z0 * math.cos(_pitch)

        # 2. rotate around y
        x2 = x1 * math.cos(_yaw) + z1 * math.sin(_yaw)
        y2 = y1
        z2 = -x1 * math.sin(_yaw) + z1 * math.cos(_yaw)

        # 2. rotate around z
        x3 = x2 * math.cos(_roll) - y2 * math.sin(_roll)
        y3 = x2 * math.sin(_roll) + y2 * math.cos(_roll)
        z3 = z2
        return [x3, y3, z3]

    def convert_3dbbox(point, cam_params):
        """Convert 3D bounding box to 2D image coordinates"""
        x, y, z, x_size, y_size, z_size, pitch, yaw, roll = point
        hx, hy, hz = x_size / 2, y_size / 2, z_size / 2
        local_corners = [
            [hx, hy, hz],
            [hx, hy, -hz],
            [hx, -hy, hz],
            [hx, -hy, -hz],
            [-hx, hy, hz],
            [-hx, hy, -hz],
            [-hx, -hy, hz],
            [-hx, -hy, -hz],
        ]
        img_corners = []
        qwen_verts = []
        for corner in local_corners:
            # rotated = rotate_xyz(corner, np.deg2rad(pitch), np.deg2rad(yaw), np.deg2rad(roll))
            rotated = rotate_xyz(
                corner, pitch, yaw, roll
            )  # Qiji: different from Qwen3-vl who normalize radians by pi
            X, Y, Z = rotated[0] + x, rotated[1] + y, rotated[2] + z
            qwen_verts.append([X, Y, Z])
            if Z > 0:
                x_2d = cam_params["fx"] * (X / Z) + cam_params["cx"]
                y_2d = cam_params["fy"] * (Y / Z) + cam_params["cy"]
                img_corners.append([x_2d, y_2d])
            else:
                print(f"Z: {Z}, corner: {corner}")
        return img_corners

    # Read image
    if image_path:
        img = cv2.imread(image_path)
    else:
        image_np = np.frombuffer(image_bytes, dtype=np.uint8)
        img = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # Convert to RGB
    # annotated_image = cv2.imread(image_path)
    annotated_image = img
    if annotated_image is None:
        print(f"Error reading image: {image_path}")
        return None

    edges = [
        [0, 1],
        [2, 3],
        [4, 5],
        [6, 7],
        [0, 2],
        [1, 3],
        [4, 6],
        [5, 7],
        [0, 4],
        [1, 5],
        [2, 6],
        [3, 7],
    ]

    # Draw 3D box for each bbox
    for i, bbox_data in enumerate(bbox_3d_list):
        # Extract bbox_3d from the dictionary
        if isinstance(bbox_data, dict) and "bbox_3d" in bbox_data:
            bbox_3d = bbox_data["bbox_3d"]
        else:
            bbox_3d = bbox_data

        # Convert angles multiplied by 180 to degrees
        bbox_3d = list(bbox_3d)  # Convert to list for modification
        # bbox_3d[-3:] = [_x * 180 for _x in bbox_3d[-3:]]  # Qiji: different from Qwen3-vl who normalize radians by pi

        if coord_format == "xyzwhlpyr":
            bbox_3d = [
                *bbox_3d[:3],
                *bbox_3d[3:6][::-1],
                *bbox_3d[6:],
            ]  # convert to 'xyzwhlpyr'
        elif coord_format == "xyzXYZpyr":
            pass

        bbox_2d = convert_3dbbox(bbox_3d, cam_params)
        # print(f"bbox_2d: {bbox_2d}")

        if len(bbox_2d) >= 8:
            # Generate random color for each box
            # box_color = [random.randint(0, 255) for _ in range(3)]
            box_color = colors[i] if colors else [255, 0, 0]  # red
            for start, end in edges:
                try:
                    pt1 = tuple([int(_pt) for _pt in bbox_2d[start]])
                    pt2 = tuple([int(_pt) for _pt in bbox_2d[end]])
                    cv2.line(
                        annotated_image,
                        pt1,
                        pt2,
                        box_color,
                        thickness[i] if thickness else 2,
                    )
                except:
                    continue
            if "label" in bbox_data:
                cv2.putText(
                    annotated_image,
                    bbox_data["label"],
                    [round(_) for _ in bbox_2d[6]],
                    cv2.FONT_HERSHEY_SIMPLEX,
                    1.0,
                    box_color,
                    1,
                )
    if save_path:
        if save_optimized:
            Image.fromarray(annotated_image).save(
                save_path, format="JPEG", quality=85, optimize=True
            )
        else:
            Image.fromarray(annotated_image).save(save_path)

    # Convert BGR to RGB for matplotlib
    # annotated_image_rgb = cv2.cvtColor(annotated_image, cv2.COLOR_BGR2RGB)

    # Create matplotlib figure
    # fig, ax = plt.subplots(1, 1, figsize=(12, 8))
    # ax.imshow(annotated_image_rgb)
    # ax.axis('off')
    if return_b64:
        pil_img = Image.fromarray(annotated_image)
        buffered = io.BytesIO()
        pil_img.save(buffered, format="JPEG")
        annotated_image = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return annotated_image


def parse_3d_boxes_from_response(response_str, max_context_window=-1):
    """从模型回复文本中解析出3D bounding boxes: `[{\"bbox_3d\":[x_center, y_center, z_center, x_size, y_size, z_size, roll, pitch, yaw],\"label\":\"category\"}, ...]`，结果是一个由JSON对象组成的list，每个JSON对象包含一个类别标签和一个由8个坐标数值组成的3D坐标框。基本逻辑如下：
    - 解析出两个结果，一个是直接找到左、右括号字符，然后直接严格解析找到的字符串，作为strict_match结果；
    - 另一个是按括号匹配算法从后向前收集的结果，作为loose_match结果，匹配算法逻辑如下：
      + 从最后一个字符开始向前做栈式括号匹配算法，找到完整的[{"bbox_3d", [...]}, {}, ...]字符串。即从后向前查找，每逢右括号就入栈，每逢左括号就出栈，当匹配完整的JSON时（即{"bbox_3d": [...]})记录一个3D坐标框的字符串，直到找到最后一个完整的JSON字符串为止。这个过程可以确保即使坐标被截断了（如被分成两行输出），也能正确地找到完整的坐标字符串。当栈空或达到`max_context_window`时停止。

    最终，返回strict_match和loose_match。
    """

    def _normalize_items(obj):
        """将解析结果统一成 [{'bbox_3d': [...], 'label': ...}, ...]。"""
        if isinstance(obj, dict):
            obj = [obj]
        if not isinstance(obj, list):
            return []

        out = []
        for item in obj:
            if not isinstance(item, dict):
                continue
            if "bbox_3d" not in item:
                continue
            bbox = item.get("bbox_3d")
            if not isinstance(bbox, (list, tuple)):
                continue

            nums = []
            ok = True
            for v in bbox:
                n = _to_number(v)
                if n is None:
                    ok = False
                    break
                nums.append(n)
            if not ok or len(nums) < 8:  # 兼容8/9维
                continue

            rec = {"bbox_3d": nums}
            if "label" in item and item["label"] is not None:
                rec["label"] = str(item["label"])
            out.append(rec)
        return out

    def _parse_text_block(text):
        """严格 JSON + 宽松 literal_eval。"""
        if not isinstance(text, str):
            return []
        s = text.strip()
        if not s:
            return []

        for loader in (json.loads, ast.literal_eval):
            try:
                obj = loader(s)
                norm = _normalize_items(obj)
                if norm:
                    return norm
            except Exception:
                pass
        return []

    # 防御式处理输入
    if not isinstance(response_str, str) or not response_str.strip():
        return [], []

    text = response_str
    strict_match = []
    loose_match = []

    # ---------- strict_match ----------
    # 1) 先按正则匹配形如 [ {...}, {...}, ... ] 的候选片段
    #    这里要求对象中出现 bbox_3d，并优先选择可解析且数量最多的结果
    pattern = re.compile(
        r'\[\s*\{[\s\S]*?"bbox_3d"\s*:\s*\[[^\]]*\][\s\S]*?\}'
        r'(?:\s*,\s*\{[\s\S]*?"bbox_3d"\s*:\s*\[[^\]]*\][\s\S]*?\})*\s*\]',
        re.DOTALL,
    )

    best = []
    for m in pattern.finditer(text):
        seg = m.group(0)
        parsed = _parse_text_block(seg)
        if parsed and len(parsed) > len(best):
            best = parsed
    strict_match = best

    # ---------- loose_match ----------
    # 从后向前，栈式匹配，优先寻找“最右侧完整数组”
    n = len(text)
    window = (
        n
        if (max_context_window is None or max_context_window < 0)
        else min(n, int(max_context_window))
    )
    start_limit = max(0, n - window)

    stack = []  # [(char, idx)]
    pairs = {"[": "]", "{": "}"}
    started = False

    for i in range(n - 1, start_limit - 1, -1):
        ch = text[i]

        if ch in "]}":
            stack.append((ch, i))
            started = True
            continue

        if ch in "[{":
            if not stack:
                continue
            top_ch, top_idx = stack[-1]
            if pairs[ch] != top_ch:
                # 括号类型不一致，丢弃当前top继续
                stack.pop()
                continue

            stack.pop()  # 成功匹配 ch ... top_idx

            # 优先尝试数组片段
            if ch == "[":
                seg = text[i : top_idx + 1]
                if "bbox_3d" in seg:
                    parsed = _parse_text_block(seg)
                    if parsed:
                        loose_match = parsed
                        break

            # 如果已经进入匹配过程且栈清空，说明一个完整后缀段结束，可停止
            if started and not stack:
                break

    # 若数组未找到，退化为“从后向前抽取单个对象”
    if not loose_match:
        stack = []
        objs = []
        for i in range(n - 1, start_limit - 1, -1):
            ch = text[i]
            if ch == "}":
                stack.append((ch, i))
            elif ch == "{":
                if not stack:
                    continue
                top_ch, top_idx = stack[-1]
                if top_ch != "}":
                    stack.pop()
                    continue
                stack.pop()
                seg = text[i : top_idx + 1]
                if "bbox_3d" in seg:
                    one = _parse_text_block(seg)
                    if one:
                        # _parse_text_block(dict) 会返回长度1列表
                        objs.extend(one)
                if not stack and objs:
                    break
        if objs:
            loose_match = list(reversed(objs))

    return strict_match, loose_match


if __name__ == "__main__":

    # 测试parse_3d_boxes_from_response
    test_response_correct = """Here is the detected 3D bounding box:
[{"bbox_3d": [1.5, 2.0, 3.0, 1.0, 2.0, 3.0, 10, 20, 30], "label": "car"}]
Note: the coordinates might be split across lines, but the function should still be able to parse them correctly. For example:
[{"bbox_3d": [1.5, 2.0, 3.0,
1.0, 2.0, 3.0, 10, 20, 30], "label": "car"}]
And there might be some noise in the text as well."""
    test_response_incomplete = """Here is the detected 3D bounding box:
[{"bbox_3d": [1.5, 2.0, 3.0, 1.0, 2.0, 3.0, 10, 20, 30]}]
Note: the coordinates might be split across lines, but the function should still be able to parse them correctly. For example:
[{"bbox_3d": [1.5, 2.0, 3.0,
1.0, 2.0, 3.0], "label": "car"}]
And there might be some noise in the text as well."""

    strict, loose = parse_3d_boxes_from_response(test_response_incomplete)
    print("Strict match:", strict)
    print("Loose match:", loose)


path: scripts/utils_detection.py
content: import json
import re
from ast import literal_eval


def _to_number(v):
    """将数值/字符串安全转为 float。"""
    if isinstance(v, (int, float)):
        return float(v)
    if isinstance(v, str):
        s = v.strip()
        if not s:
            return None
        try:
            return float(s)
        except Exception:
            return None
    return None


def parse_detection_js(txt, match_all_keys=False):
    """Parse object detection results outputted by GLM-V, refer to https://zhipu-ai.feishu.cn/wiki/PxWtwH1rziRh1mkdcsvcgghinyg.
    The valid format should include keys of `label`, `bbox_2d` with values of label-name and detection-bbox (e.g., {'label':'person-1', 'bbox_2d': [1,2,3,4]}).
    """
    # 1. define kv match patterns & kv valid map
    kv_pattern = {
        "label": (
            r"[\'\"]label[\'\"]\s*:\s*[\"\']",
            r"(.*?)",
            r"[\"\'][,\s]+",
        ),  # (pre, mid, post)
        "bbox_2d": (
            r"[\'\"]bbox_2d[\'\"]\s*:\s*",
            r"(\[\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*\])",
            ".*",
        ),
        "bbox_3d": (
            r"[\'\"]bbox_3d[\'\"]\s*:\s*",
            r"(\[\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*\])",
            ".*",
        ),
    }
    kv_to_valid = {
        "label": lambda x: str(x) if x else "",
        "bbox_2d": lambda x: (
            [int(_) for _ in literal_eval(str(x))]
            if re.search(kv_pattern["bbox_2d"][1], str(x))
            else []
        ),
        "bbox_3d": lambda x: (
            [float(_) for _ in literal_eval(str(x))]
            if re.search(kv_pattern["bbox_3d"][1], str(x))
            else []
        ),
    }

    valid_js, match_js = {}, {}
    # 2. parse valid json
    try:
        _js = literal_eval(txt)
        if isinstance(_js, dict):
            for k, v in _js.items():
                valid_js[k] = kv_to_valid.get(k, lambda x: x)(v)
    except:
        pass
    # 3. fallback to match
    try:
        for m in re.finditer(
            r"[\"\']*(\w+)[\"\']*\s*:\s*[\"\']*([^\{\}\[\],\"\']+|[\{\[][^{}]*?[\}\]])[\"\']*",
            txt,
        ):  # match "key": value or 'key': value, where value can be a primitive or a list/dict, and strip quotes from string values
            k, v = m.groups()
            match_js[k] = kv_to_valid.get(k, lambda x: x)(v.strip())
        # for k, ptrs in kv_pattern.items():
        #     found = re.search(''.join(ptrs), txt)
        #     if found:
        #         match_js[k] =  kv_to_valid.get(k, lambda x:x)(found.group(1))
    except:
        pass
    # 3. check completeness for valid_js
    if match_all_keys:
        if not all([_ in valid_js for _ in kv_pattern.keys()]):
            valid_js = {}
    return valid_js, match_js


def parse_detection(txt, return_valid_only=False):
    """Extract a list of multiple detection results, where each det result refers to a JSON with 'label' and 'bbox_2d' keys."""
    valid_result, match_result = [], []
    if isinstance(txt, str):
        txt = txt.strip()
    # 1. parse valid detection strictly follow the format
    try:
        dets = literal_eval(txt)  # assume to be a list of dicts
        for d in dets:
            if isinstance(d, dict):
                d = json.dumps(d)
            valid_js, match_js = parse_detection_js(str(d))
            if valid_js:
                valid_result.append(valid_js)
    except:
        pass
    # 2. fallback to match
    try:
        dets = re.findall(r"\{[^{}]*\}", txt)
        for d in dets:
            valid_js, match_js = parse_detection_js(str(d))
            if valid_js:
                match_result.append(match_js)
    except:
        pass

    if return_valid_only:
        return valid_result
    return valid_result, match_result


def parse_detection_from_response(
    response_str, max_context_window=-1, required_keys=["bbox_2d"]
):
    """从模型回复文本中解析出Objects Detection JSON: `[{'label': 'category', 'bbox_2d': [x1, y1, x2, y2]}, ...]`，抽取出的grounding结果是一个JSON列表，每个JSON对象包含一个类别标签和一个4值坐标框。基本逻辑如下：
    - 解析出两个结果，一个是直接找到左、右括号字符，然后直接严格解析找到的字符串，作为strict_match结果；
    - 另一个是按括号匹配算法从后向前收集的结果，作为loose_match结果，匹配算法逻辑如下：
      + 从最后一个字符开始向前做栈式括号匹配算法，找到完整的[{"bbox_2d", [...]}, {}, ...]字符串。即从后向前查找，每逢右括号就入栈，每逢左括号就出栈，当匹配完整的JSON时（即{"bbox_2d": [...]})记录一个2D坐标框的字符串，直到找到最后一个完整的JSON字符串为止。这个过程可以确保即使坐标被截断了（如被分成两行输出），也能正确地找到完整的坐标字符串。当栈空或达到`max_context_window`时停止。

    最终，返回strict_match和loose_match。
    """

    def _iter_dets_text_candidates(s):
        """Linear-time extraction of complete [...] candidates containing bbox_2d."""
        stack = []
        for idx, ch in enumerate(s):
            if ch == "[":
                stack.append(idx)
            elif ch == "]" and stack:
                start = stack.pop()
                seg = s[start : idx + 1]
                if all(k in seg for k in required_keys) and "{" in seg:
                    yield seg

    def _is_valid_det(det):
        if not isinstance(det, dict):
            return False
        for k in required_keys:
            if k not in det:
                return False
            v = det.get(k)
            if v is None:
                return False
            if isinstance(v, (str, list, tuple, dict, set)) and len(v) == 0:
                return False
        return True

    # 防御式处理输入
    if not isinstance(response_str, str) or not response_str.strip():
        return [], []

    text = response_str
    if not all(k in text for k in required_keys):
        return [], []
    strict_match = []
    loose_match = []

    # ---------- strict_match ----------
    best = []
    for seg in _iter_dets_text_candidates(text):
        parsed = [
            d for d in parse_detection(seg, return_valid_only=True) if _is_valid_det(d)
        ]
        if parsed and len(parsed) > len(best):
            best = parsed
    strict_match = best

    # ---------- loose_match ----------
    # 从后向前，优先匹配“最右侧完整数组”，找不到再降级到对象级恢复。
    n = len(text)
    window = (
        n
        if (max_context_window is None or max_context_window < 0)
        else min(n, int(max_context_window))
    )
    start_limit = max(0, n - window)

    # Step 1: rightmost complete array
    sq_stack = []
    rightmost_array = []
    for i in range(n - 1, start_limit - 1, -1):
        ch = text[i]
        if ch == "]":
            sq_stack.append(i)
        elif ch == "[":
            if not sq_stack:
                continue
            j = sq_stack.pop()
            seg = text[i : j + 1]
            if not (all(k in seg for k in required_keys) and "{" in seg):
                continue
            parsed = [
                d
                for d in parse_detection(seg, return_valid_only=True)
                if _is_valid_det(d)
            ]
            if parsed:
                rightmost_array = parsed
                break

    if rightmost_array:
        loose_match = rightmost_array
    else:
        # Step 2: fallback object-level recovery (useful for truncated tail)
        obj_stack = []
        objs = []
        for i in range(n - 1, start_limit - 1, -1):
            ch = text[i]
            if ch == "}":
                obj_stack.append(i)
            elif ch == "{":
                if not obj_stack:
                    continue
                j = obj_stack.pop()
                seg = text[i : j + 1]
                if not (all(k in seg for k in required_keys) and "{" in seg):
                    continue
                valid_js, _ = parse_detection_js(seg)
                if _is_valid_det(valid_js):
                    objs.append(valid_js)
        if objs:
            loose_match = list(reversed(objs))

    # return strict_match, loose_match
    return loose_match


if __name__ == "__main__":

    # 测试parse_detection_from_response
    test_response_correct = """Here is the detected 2D bounding box:
[{"bbox_2d": [1.5, 2.0, 3.0, 1.0], "label": "car"}]
Note: the coordinates might be split across lines, but the function should still be able to parse them correctly. For example:
[{"bbox_2d": [1.5, 2.0, 3.0,
1.0, 2.0, 3.0], "label": "car"}]
And there might be some noise in the text as well."""
    test_response_incomplete = """Here is the detected 2D bounding box:
[{"bbox_2d": [1.5, 2.0, 3.0, 6.0], "label": "car"}]
Note: the coordinates might be split across lines, but the function should still be able to parse them correctly. For example:
[{"bbox_2d": [1.5, 2.0, 3.0, 1.0, 2.0, 3.0], "label": "car"}]
And there might be some noise in the text as well."""

    loose = parse_detection_from_response(
        test_response_incomplete, max_context_window=400
    )
    print("Loose match:", loose)


path: scripts/requirements.txt
content: requests>=2.31.0
Pillow>=10.0.0
opencv-python>=4.8.0
numpy>=1.24.0
matplotlib>=3.7.0
decord>=0.6.0


path: scripts/utils_video.py
content: import base64, cv2, io, os, math, tempfile, re, json, ast, shutil, subprocess, glob
import numpy as np
from pathlib import Path
from decord import VideoReader, cpu
import matplotlib.pyplot as plt
import traceback
import colorsys
from PIL import Image
from ast import literal_eval


def _to_number(x):
    if isinstance(x, (int, float)):
        return x
    if isinstance(x, str):
        x = x.strip()
        if not x:
            return None
        try:
            return float(x) if ("." in x or "e" in x.lower()) else int(x)
        except Exception:
            return None
    return None


def reverse_normalize_box(
    box, img_width=None, img_height=None, img_path=None, round_fn=lambda x: round(x, 2)
):
    if img_path is not None:
        with Image.open(img_path) as img:
            img_width, img_height = img.size
    box = [max(0, min(x, 999)) for x in box]
    box = [x / 1000.0 for x in box]
    # box = [box[0] * im_width, box[1] * img_height, box[2] * im_width, box[3] * img_height]
    box = [
        box[_] * img_width if _ % 2 == 0 else box[_] * img_height
        for _ in range(len(box))
    ]
    box = [round_fn(x) for x in box]  # round to 2 decimal places
    return box


def get_distinct_colors_hsv(n, bright=True, normalize=True):
    """
    Generate random colors.
    To get visually distinct colors, generate them in HSV space then
    convert to RGB.
    """
    brightness = 1.0 if bright else 0.7
    hsv = [(i / n, 1, brightness) for i in range(n)]
    colors = list(map(lambda c: colorsys.hsv_to_rgb(*c), hsv))
    if not normalize:
        colors = [(int(r * 255), int(g * 255), int(b * 255)) for r, g, b in colors]
    # random.shuffle(colors)
    return colors


def video_to_base64(video_path):
    with open(video_path, "rb") as video_file:
        # 读取视频文件并转换为 Base64 编码的字节数据
        encoded_string = base64.b64encode(video_file.read())
        return encoded_string.decode("utf-8")


def build_video_from_frames(
    frames: np.ndarray = None,
    frames_paths: str = None,
    output_path=None,
    fps=1,
    return_bytes=False,
):
    """Build video from frames.
    @frames: List of frames as numpy arrays.
    @frames_paths: List of frame file paths.
    """
    assert frames or frames_paths, "Either frames or frames_paths must be provided."
    if frames is None:
        frames = []
        for frame_path in frames_paths:
            frame = cv2.imread(frame_path)
            frames.append(frame)
    height, width, _ = frames[0].shape

    if output_path is None:
        output_temp = tempfile.NamedTemporaryFile(suffix=".mp4")
        output_path = output_temp.name

    # fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    fourcc = cv2.VideoWriter_fourcc(*"H264")
    video_writer = cv2.VideoWriter(output_path, fourcc, fps, (width, height))

    for idx, frame in enumerate(frames):
        if frame is None:
            print(f"Frame at index {idx} is None, skipping.")
            continue
        if len(frame.shape) == 2:  # grayscale to BGR
            frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2BGR)
        elif frame.shape[2] == 4:  # RGBA to BGR
            frame = cv2.cvtColor(frame, cv2.COLOR_RGBA2BGR)
        video_writer.write(frame)
    video_writer.release()
    print(f"✅ Video saved to {output_path}")

    # delete temp file if used
    if "output_temp" in locals():
        output_temp.close()

    if return_bytes:
        with open(output_path, "rb") as f:
            video_bytes = f.read()
        return video_bytes
    else:
        return output_path


def visualize_mot(
    video_path: str = None,
    video_bytes=None,
    mot_js=None,
    renormalize=True,
    save_path: str = None,
    return_b64=False,
    distinct_colors=True,
    **kwargs,
):
    """Visualize the mot results for a given video, where the format of mot_js should be {0: [{"label": xx, "bbox_2d": [x1,y1,x2,y2]}, ..], 1: ..} where the keys refer to the timestamp in seconds and values are detection results."""
    assert (
        video_path is not None or video_bytes is not None
    ), "Either video_path or video_bytes must be provided."
    assert mot_js is not None, "mot_js must be provided."

    label2colors = kwargs.get("label2colors", {})
    label2colors = {
        _k: tuple(int(c * 255) for c in plt.matplotlib.colors.to_rgb(_v))
        for _k, _v in label2colors.items()
    }
    label2thickness = kwargs.get("label2thickness", {})

    # --- Read original video
    if video_path is not None:
        vr = VideoReader(video_path, ctx=cpu(0))
    else:
        # video_bytes 情况：保存到临时文件再读取
        vr = VideoReader(io.BytesIO(video_bytes), ctx=cpu(0))
    fps, frame_count = vr.get_avg_fps(), len(vr)
    height, width = vr[0].shape[:2]

    output_path = save_path if save_path is not None else tempfile.mktemp(suffix=".mp4")

    # 定义 VideoWriter
    # writer = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))
    writer = cv2.VideoWriter(
        output_path, cv2.VideoWriter_fourcc(*"H264"), fps, (width, height)
    )
    if not writer.isOpened():
        print(f"Failed to open video writer for output_path: {output_path}")
        output_path = str(Path(output_path).with_suffix(".avi"))
        writer = cv2.VideoWriter(
            output_path, cv2.VideoWriter_fourcc(*"XVID"), fps, (width, height)
        )

    all_labels = set([_d["label"] for _f in mot_js for _d in mot_js[_f]])
    if distinct_colors:
        all_colors = get_distinct_colors_hsv(len(all_labels), normalize=False)
        label2colors = {str(_lbl): all_colors[_i] for _i, _lbl in enumerate(all_labels)}
    for frame_id in range(frame_count):
        frame = vr[frame_id].asnumpy()
        # 转成 RGB 方便绘制
        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        annotated_frame = frame_rgb.copy()
        # dets = mot_js.get(frame_id, []) or mot_js.get(str(frame_id), [])  # 当前秒的所有检测结果
        if frame_id % fps == 0:
            dets = mot_js.get(str(math.floor(frame_id / fps)), []) or mot_js.get(
                str(math.floor(frame_id // fps)), []
            )  # 当前秒的所有检测结果
        else:
            dets = []
        labels = []
        boxes = []
        for det in dets:
            lbl = str(det.get("label", ""))
            box = det.get("bbox_2d", [])
            try:
                if renormalize:
                    box = reverse_normalize_box(box, width, height, round_fn=int)
                else:
                    box = [int(_) for _ in box]
            except:
                import traceback

                print(
                    f"Error in processing bbox for frame_id: {frame_id}, label: {lbl}"
                )
                traceback.print_exc()
            labels.append(lbl)
            boxes.append(box)
        # 绘制每个框和标签
        for box, lbl in zip(boxes, labels):
            cv2.rectangle(
                annotated_frame,
                box[:2],
                box[2:],
                color=label2colors.get(str(lbl), (0, 255, 0)),
                thickness=label2thickness.get(str(lbl), 2),
            )
            cv2.putText(
                annotated_frame,
                lbl,
                (box[0], box[1]),
                cv2.FONT_HERSHEY_SIMPLEX,
                0.5,
                label2colors.get(str(lbl), (0, 255, 0)),
                thickness=label2thickness.get(str(lbl), 2),
            )
        # 再转成 BGR 给cv2写视频
        # annotated_frame = cv2.cvtColor(annotated_frame, cv2.COLOR_RGB2BGR)
        writer.write(annotated_frame)
    writer.release()

    # return
    if return_b64:
        with open(output_path, "rb") as f:
            b64_video = base64.b64encode(f.read()).decode("utf-8")
        if save_path is None:
            os.remove(output_path)
        return b64_video
    else:
        return output_path


def visualize_mot_per_frame(
    video_path: str = None,
    video_bytes=None,
    mot_js=None,
    renormalize=True,
    save_path: str = None,
    distinct_colors=True,
    thickness=3,
    **kwargs,
):
    """Visualize MOT results on EVERY frame (not just 1 per second).

    Uses visualize_boxes to draw boxes on each extracted frame, then ffmpeg
    to reassemble at the original framerate. Much faster than visualize_mot
    which writes frames one-by-one via OpenCV.

    Args:
        video_path: Path to input video.
        video_bytes: Bytes of input video (alternative to video_path).
        mot_js: MOT results dict with STRING keys: {"0": [{"label": "cat-1", "bbox_2d": [x1,y1,x2,y2]}], "1": ...}
        renormalize: If True, convert 0~1000 normalized coords to pixel coords.
        save_path: Output video path.
        distinct_colors: If True, auto-generate distinct colors per label.
        thickness: Box line thickness.
        **kwargs: Passed to visualize_boxes (colors, labels, etc.)

    Returns:
        Output video path.
    """
    from utils_boxes import visualize_boxes

    assert (
        video_path is not None or video_bytes is not None
    ), "Either video_path or video_bytes must be provided."
    assert mot_js is not None, "mot_js must be provided."

    # --- Read video info ---
    if video_path is not None:
        vr = VideoReader(video_path, ctx=cpu(0))
    else:
        vr = VideoReader(io.BytesIO(video_bytes), ctx=cpu(0))
    fps = vr.get_avg_fps()
    frame_count = len(vr)
    height, width = vr[0].shape[:2]

    # --- Ensure string keys in mot_js ---
    mot_str = {}
    for k, v in mot_js.items():
        mot_str[str(k)] = v

    # --- Setup colors ---
    label2colors = kwargs.get("label2colors", {})
    if not label2colors and distinct_colors:
        all_labels = sorted(set(d["label"] for dets in mot_str.values() for d in dets))
        # Use matplotlib named colors for visualize_boxes compatibility
        named_colors = [
            "red",
            "blue",
            "green",
            "orange",
            "purple",
            "cyan",
            "magenta",
            "yellow",
            "lime",
            "pink",
        ]
        label2colors = {
            lbl: named_colors[i % len(named_colors)] for i, lbl in enumerate(all_labels)
        }

    # --- Extract all frames ---
    tmp_dir = tempfile.mkdtemp(prefix="mot_viz_")
    raw_dir = os.path.join(tmp_dir, "raw")
    out_dir = os.path.join(tmp_dir, "out")
    os.makedirs(raw_dir)
    os.makedirs(out_dir)

    if video_path is not None:
        # Extract all frames
        raw_pattern = os.path.join(raw_dir, "frame_%05d.jpg")
        subprocess.run(
            ["ffmpeg", "-y", "-i", video_path, "-q:v", "2", raw_pattern],
            capture_output=True,
        )
    else:
        # Save bytes to temp file then extract
        tmp_video = os.path.join(tmp_dir, "input.mov")
        with open(tmp_video, "wb") as f:
            f.write(video_bytes)
        raw_pattern = os.path.join(raw_dir, "frame_%05d.jpg")
        subprocess.run(
            ["ffmpeg", "-y", "-i", tmp_video, "-q:v", "2", raw_pattern],
            capture_output=True,
        )

    # --- Draw boxes on every frame ---
    frame_files = sorted(glob.glob(os.path.join(raw_dir, "frame_*.jpg")))
    total_frames = len(frame_files)

    for i, f in enumerate(frame_files):
        sec_idx = str(
            min(int(i / fps), max(int(k) for k in mot_str.keys()) if mot_str else 0)
        )
        dets = mot_str.get(sec_idx, [])
        if not dets:
            # Try neighboring seconds
            for offset in [-1, 1]:
                alt = str(int(sec_idx) + offset)
                if alt in mot_str:
                    dets = mot_str[alt]
                    break

        if dets:
            boxes = [d["bbox_2d"] for d in dets]
            labels = [d["label"] for d in dets]
            colors = [label2colors.get(lbl, "red") for lbl in labels]
            thicknesses = [thickness] * len(boxes)
            save_p = os.path.join(out_dir, f"frame_{i:05d}.jpg")
            visualize_boxes(
                img_path=f,
                boxes=boxes,
                labels=labels,
                renormalize=renormalize,
                save_path=save_p,
                colors=colors,
                thickness=thicknesses,
            )
        else:
            # No detections, copy original frame
            shutil.copy(f, os.path.join(out_dir, f"frame_{i:05d}.jpg"))

        if (i + 1) % 100 == 0:
            print(f"  visualize_mot_per_frame: {i+1}/{total_frames} frames done")

    # --- Reassemble with ffmpeg ---
    output_path = save_path if save_path else os.path.join(tmp_dir, "output.mp4")
    out_pattern = os.path.join(out_dir, "frame_%05d.jpg")
    subprocess.run(
        [
            "ffmpeg",
            "-y",
            "-framerate",
            str(fps),
            "-i",
            out_pattern,
            "-c:v",
            "libx264",
            "-pix_fmt",
            "yuv420p",
            output_path,
        ],
        capture_output=True,
    )

    # Cleanup temp dirs
    shutil.rmtree(tmp_dir, ignore_errors=True)

    print(
        f"✅ visualize_mot_per_frame: video saved to {output_path} ({total_frames} frames at {fps:.1f}fps)"
    )
    return output_path


def parse_detection_js(txt, match_all_keys=False):
    """Parse object detection results outputted by GLM-V, refer to https://zhipu-ai.feishu.cn/wiki/PxWtwH1rziRh1mkdcsvcgghinyg.
    The valid format should include keys of `label`, `bbox_2d` with values of label-name and detection-bbox (e.g., {'label':'person-1', 'bbox_2d': [1,2,3,4]}).
    """
    # 1. define kv match patterns & kv valid map
    kv_pattern = {
        "label": (
            r"[\'\"]label[\'\"]\s*:\s*[\"\']",
            r"(.*?)",
            r"[\"\'][,\s]+",
        ),  # (pre, mid, post)
        "bbox_2d": (
            r"[\'\"]bbox_2d[\'\"]\s*:\s*",
            r"(\[\s*\d+\s*,\s*\d+\s*,\s*\d+\s*,\s*\d+\s*\])",
            ".*",
        ),
        "bbox_3d": (
            r"[\'\"]bbox_3d[\'\"]\s*:\s*",
            r"(\[\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*,\s*-*\d+\.*\d+\s*\])",
            ".*",
        ),
    }
    kv_to_valid = {
        "label": lambda x: str(x) if x else "",
        "bbox_2d": lambda x: (
            [int(_) for _ in literal_eval(str(x))]
            if re.search(kv_pattern["bbox_2d"][1], str(x))
            else []
        ),
        "bbox_3d": lambda x: (
            [float(_) for _ in literal_eval(str(x))]
            if re.search(kv_pattern["bbox_3d"][1], str(x))
            else []
        ),
    }

    valid_js, match_js = {}, {}
    # 2. parse valid json
    try:
        _js = literal_eval(txt)
        if isinstance(_js, dict):
            for k, v in _js.items():
                valid_js[k] = kv_to_valid.get(k, lambda x: x)(v)
    except:
        pass
    # 3. fallback to match
    try:
        for m in re.finditer(
            r"[\"\']*(\w+)[\"\']*\s*:\s*[\"\']*([^\{\}\[\],\"\']+|[\{\[][^{}]*?[\}\]])[\"\']*",
            txt,
        ):  # match "key": value or 'key': value, where value can be a primitive or a list/dict, and strip quotes from string values
            k, v = m.groups()
            match_js[k] = kv_to_valid.get(k, lambda x: x)(v.strip())
        # for k, ptrs in kv_pattern.items():
        #     found = re.search(''.join(ptrs), txt)
        #     if found:
        #         match_js[k] =  kv_to_valid.get(k, lambda x:x)(found.group(1))
    except:
        pass
    # 3. check completeness for valid_js
    if match_all_keys:
        if not all([_ in valid_js for _ in kv_pattern.keys()]):
            valid_js = {}
    return valid_js, match_js


def parse_detection(txt, return_valid_only=False):
    """Extract a list of multiple detection results, where each det result refers to a JSON with 'label' and 'bbox_2d' keys."""
    valid_result, match_result = [], []
    if isinstance(txt, str):
        txt = txt.strip()
    # 1. parse valid detection strictly follow the format
    try:
        dets = literal_eval(txt)  # assume to be a list of dicts
        for d in dets:
            if isinstance(d, dict):
                d = json.dumps(d)
            valid_js, match_js = parse_detection_js(str(d))
            if valid_js:
                valid_result.append(valid_js)
    except:
        pass
    # 2. fallback to match
    try:
        dets = re.findall(r"\{[^{}]*\}", txt)
        for d in dets:
            valid_js, match_js = parse_detection_js(str(d))
            if valid_js:
                match_result.append(match_js)
    except:
        pass

    if return_valid_only:
        return valid_result
    return valid_result, match_result


def parse_detection_from_response(
    response_str, max_context_window=-1, required_keys=["bbox_2d"]
):
    """从模型回复文本中解析出Objects Detection JSON: `[{'label': 'category', 'bbox_2d': [x1, y1, x2, y2]}, ...]`，抽取出的grounding结果是一个JSON列表，每个JSON对象包含一个类别标签和一个4值坐标框。基本逻辑如下：
    - 解析出两个结果，一个是直接找到左、右括号字符，然后直接严格解析找到的字符串，作为strict_match结果；
    - 另一个是按括号匹配算法从后向前收集的结果，作为loose_match结果，匹配算法逻辑如下：
      + 从最后一个字符开始向前做栈式括号匹配算法，找到完整的[{"bbox_2d", [...]}, {}, ...]字符串。即从后向前查找，每逢右括号就入栈，每逢左括号就出栈，当匹配完整的JSON时（即{"bbox_2d": [...]})记录一个2D坐标框的字符串，直到找到最后一个完整的JSON字符串为止。这个过程可以确保即使坐标被截断了（如被分成两行输出），也能正确地找到完整的坐标字符串。当栈空或达到`max_context_window`时停止。

    最终，返回strict_match和loose_match。
    """

    def _iter_dets_text_candidates(s):
        """Linear-time extraction of complete [...] candidates containing bbox_2d."""
        stack = []
        for idx, ch in enumerate(s):
            if ch == "[":
                stack.append(idx)
            elif ch == "]" and stack:
                start = stack.pop()
                seg = s[start : idx + 1]
                if all(k in seg for k in required_keys) and "{" in seg:
                    yield seg

    def _is_valid_det(det):
        if not isinstance(det, dict):
            return False
        for k in required_keys:
            if k not in det:
                return False
            v = det.get(k)
            if v is None:
                return False
            if isinstance(v, (str, list, tuple, dict, set)) and len(v) == 0:
                return False
        return True

    # 防御式处理输入
    if not isinstance(response_str, str) or not response_str.strip():
        return [], []

    text = response_str
    if not all(k in text for k in required_keys):
        return [], []
    strict_match = []
    loose_match = []

    # ---------- strict_match ----------
    best = []
    for seg in _iter_dets_text_candidates(text):
        parsed = [
            d for d in parse_detection(seg, return_valid_only=True) if _is_valid_det(d)
        ]
        if parsed and len(parsed) > len(best):
            best = parsed
    strict_match = best

    # ---------- loose_match ----------
    # 从后向前，优先匹配“最右侧完整数组”，找不到再降级到对象级恢复。
    n = len(text)
    window = (
        n
        if (max_context_window is None or max_context_window < 0)
        else min(n, int(max_context_window))
    )
    start_limit = max(0, n - window)

    # Step 1: rightmost complete array
    sq_stack = []
    rightmost_array = []
    for i in range(n - 1, start_limit - 1, -1):
        ch = text[i]
        if ch == "]":
            sq_stack.append(i)
        elif ch == "[":
            if not sq_stack:
                continue
            j = sq_stack.pop()
            seg = text[i : j + 1]
            if not (all(k in seg for k in required_keys) and "{" in seg):
                continue
            parsed = [
                d
                for d in parse_detection(seg, return_valid_only=True)
                if _is_valid_det(d)
            ]
            if parsed:
                rightmost_array = parsed
                break

    if rightmost_array:
        loose_match = rightmost_array
    else:
        # Step 2: fallback object-level recovery (useful for truncated tail)
        obj_stack = []
        objs = []
        for i in range(n - 1, start_limit - 1, -1):
            ch = text[i]
            if ch == "}":
                obj_stack.append(i)
            elif ch == "{":
                if not obj_stack:
                    continue
                j = obj_stack.pop()
                seg = text[i : j + 1]
                if not (all(k in seg for k in required_keys) and "{" in seg):
                    continue
                valid_js, _ = parse_detection_js(seg)
                if _is_valid_det(valid_js):
                    objs.append(valid_js)
        if objs:
            loose_match = list(reversed(objs))

    return strict_match, loose_match


def parse_mot_from_response(
    response_str, max_context_window=256000, required_keys=["bbox_2d"]
):
    """从模型回复文本中解析出Video Objects Tracking JSON: `{"0": [{'label': 'car-1', 'bbox_2d': [1,2,3,4]}, {'label': 'car-2', 'bbox_2d': [2,3,4,5]}], '1': [{'label': 'car-2', 'bbox_2d': [4,5,6,7]}, {'label': 'person-1', 'bbox_2d': [10,20,30,40]}]}`, 抽取出的grounding结果是一个JSON对象，键为视频帧索引，值为一个列表，每个列表元素是一个JSON对象，包含一个类别标签和一个2D坐标框。基本逻辑：
    - 解析出两个结果，一个是直接找到左、右括号字符，然后直接严格解析找到的字符串，作为strict_match结果；s
    - 另一个从后向前找到每一个完整的以序号为key以object detectionJSON对象（即{"label": ..., "bbox_2d": [...]})为value的字符串，然后调用parse_detection_from_response的解析函数解析出detection结果，最终合成一个mot结果，作为loose_match结果。这个过程可以确保即使坐标被截断了（如被分成两行输出），也能正确地找到完整的坐标字符串。当栈空或达到`max_context_window`时停止。

    最终，返回strict_match和loose_match。
    """
    mot_pattern = r"[\"\']*(\d+)[\"\']*\s*\:\s*(\[\s*\{.*?\}\s*\])"

    def _normalize_frame_key(k):
        if isinstance(k, int):
            return str(k)
        if isinstance(k, str):
            kk = k.strip().strip('"').strip("'")
            if re.fullmatch(r"-?\d+", kk):
                return str(int(kk))
        return None

    def _sort_mot_keys(mot):
        return {k: mot[k] for k in sorted(mot, key=lambda x: int(x))}

    def _parse_det_value(v):
        if isinstance(v, str):
            s = v
        elif isinstance(v, (list, dict)):
            s = repr(v)
        else:
            return []
        strict_dets, loose_dets = parse_detection_from_response(
            s,
            max_context_window=-1,
            required_keys=required_keys,
        )
        return strict_dets or loose_dets

    def _normalize_mot_obj(obj):
        if not isinstance(obj, dict):
            return {}
        out = {}
        for k, v in obj.items():
            key = _normalize_frame_key(k)
            if key is None:
                continue
            dets = _parse_det_value(v)
            if dets:
                out[key] = dets
        return _sort_mot_keys(out)

    def _parse_mot_js(txt, strict_only=False):
        # 1) 先走结构化解析（更准确）
        for loader in (json.loads, ast.literal_eval):
            try:
                norm = _normalize_mot_obj(loader(txt.strip()))
                if norm:
                    return norm
            except:
                pass

        if strict_only:
            return {}

        # 2) 兜底正则（兼容半结构化文本）
        out = {}
        for m in re.finditer(mot_pattern, txt):
            res_tid, dets = m.groups()
            key = _normalize_frame_key(res_tid)
            if key is None:
                continue
            parsed = _parse_det_value(dets)
            if parsed:
                out[key] = parsed
        return _sort_mot_keys(out)

    def _iter_dets_text_candidates(s):
        """从后向前，仅在栈清空时产出顶层闭合块候选，避免内层{}反复正则。"""
        stack = []
        for idx in range(len(s) - 1, -1, -1):
            ch = s[idx]
            if ch == "}":
                stack.append(idx)
            elif ch == "{":
                if not stack:
                    continue
                end_idx = stack.pop()
                if stack:
                    continue
                seg = s[idx : end_idx + 1]
                if all(k in seg for k in required_keys) and re.search(mot_pattern, seg):
                    yield seg

    if not isinstance(response_str, str) or not response_str.strip():
        return {}, {}

    n = len(response_str)
    window = (
        n
        if (max_context_window is None or max_context_window < 0)
        else min(n, int(max_context_window))
    )
    text = response_str[-window:] if window > 0 else response_str

    strict_match, loose_match = {}, {}

    # === strict match ===
    try:
        for seg in _iter_dets_text_candidates(text):
            strict_match = _parse_mot_js(seg, strict_only=True)
            # 只找一个完整的MOT对象，找到后就
            break
    except:
        pass

    # === loose match ===
    for seg in _iter_dets_text_candidates(text):
        parsed_mot = _parse_mot_js(seg, strict_only=False)
        if parsed_mot and len(parsed_mot) > len(loose_match):
            loose_match = parsed_mot

    # return strict_match, loose_match
    return loose_match


if __name__ == "__main__":
    # 测试 parse_detection_from_response
    test_str = """
    Here are the detections: [{"label": "car", "bbox_2d": [100, 150, 200, 250]}, {"label": "person", "bbox_2d": [300, 350, 400, 450]}]
    And some more text.
    """
    loose = parse_detection_from_response(test_str)
    print("Loose:", loose)

    # 测试 parse_mot_from_response
    test_mot_str = """
    Tracking results: {0: [{"label": "car-1", "bbox_2d": [1,2,3,4]}, {"label": "car-2", "bbox_2d": [2,3,4,5]}], 1: [{"label": "car-2", "bbox_2d": [4,5,6,7]}, {"label": "person-1", "bbox_2d": xx}]}
    """
    loose_mot = parse_mot_from_response(test_mot_str)
    print("Loose MOT:", loose_mot)


path: scripts/config_setup.py
content: #!/usr/bin/env python3
"""
Environment Variable Configuration Setup
Helps users set up their .env file for GLM-Grounding skill
"""

import sys
from pathlib import Path
from typing import Optional


def _get_skill_root() -> Path:
    """Get the skill root directory"""
    # Get the directory where this script is located
    script_dir = Path(__file__).resolve().parent
    return script_dir.parent


def _get_env_path(skill_root: Optional[Path] = None) -> Path:
    """Get the path to the .env file"""
    if skill_root is None:
        skill_root = _get_skill_root()
    return skill_root / ".env"


def _env_exists(skill_root: Optional[Path] = None) -> bool:
    """Check if .env file exists"""
    return _get_env_path(skill_root).exists()


def _read_env_file(skill_root: Optional[Path] = None) -> dict:
    """
    Read existing .env file and return as dictionary

    Returns:
        Dictionary of environment variables
    """
    env_path = _get_env_path(skill_root)
    env_vars = {}

    if not env_path.exists():
        return env_vars

    with open(env_path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            # Skip comments and empty lines
            if not line or line.startswith("#"):
                continue
            # Parse KEY=VALUE
            if "=" in line:
                key, value = line.split("=", 1)
                env_vars[key.strip()] = value.strip()

    return env_vars


def _write_env_file(env_vars: dict, skill_root: Optional[Path] = None) -> None:
    """
    Write environment variables to .env file

    Args:
        env_vars: Dictionary of environment variables to write
        skill_root: Skill root directory path
    """
    env_path = _get_env_path(skill_root)

    with open(env_path, "w", encoding="utf-8") as f:
        # Write header
        f.write("# GLM-V API Configuration\n")
        f.write("# Auto-generated by config_setup.py\n")
        f.write("# DO NOT commit this file to version control\n\n")

        # Write environment variables
        for key, value in env_vars.items():
            f.write(f"{key}={value}\n")


def _setup_interactive() -> dict:
    """
    Interactive setup for environment variables

    Returns:
        Dictionary of configured environment variables
    """
    env_vars = {}
    existing = _read_env_file()

    print("=" * 60)
    print("GLM-Grounding Skill - Environment Configuration")
    print("=" * 60)
    print()

    # API Key
    current_key = existing.get("ZHIPU_API_KEY", "")
    if current_key:
        print(f"Current API Key: {current_key[:8]}...{current_key[-4:]}")

    [REDACTED_SECRET]
        f"Enter GLM-V API Key{' (press Enter to keep current)' if current_key else ''}: "
    ).strip()

    if not api_key and current_key:
        [REDACTED_SECRET]
    elif not api_key and not current_key:
        print("Error: API Key is required!")
        sys.exit(1)

    env_vars["ZHIPU_API_KEY"] = api_key

    print()
    print("=" * 60)
    print("Configuration Summary:")
    print("=" * 60)
    print(
        f"API Key: {env_vars['ZHIPU_API_KEY'][:8]}...{env_vars['ZHIPU_API_KEY'][-4:]}"
    )
    print()

    return env_vars


def _setup_from_args(api_key: str) -> dict:
    """
    Setup environment variables from command line arguments

    Args:
        api_key: GLM-V API key
        api_url: Optional API URL

    Returns:
        Dictionary of environment variables
    """
    env_vars = {"ZHIPU_API_KEY": api_key}
    return env_vars


def _validate_config(env_vars: dict) -> bool:
    """
    Validate the configuration

    Args:
        env_vars: Dictionary of environment variables

    Returns:
        True if valid, False otherwise
    """
    if "ZHIPU_API_KEY" not in env_vars or not env_vars["ZHIPU_API_KEY"]:
        print("Error: ZHIPU_API_KEY is required")
        return False

    [REDACTED_SECRET]"ZHIPU_API_KEY"]
    if len(api_key) < 10:
        print("Warning: API key seems too short")

    return True


def _show_config() -> None:
    """Display current configuration"""
    env_path = _get_env_path()

    if not env_path.exists():
        print("No .env file found. Run 'setup' to create one.")
        return

    env_vars = _read_env_file()

    if not env_vars:
        print(".env file exists but is empty.")
        return

    print("Current Configuration:")
    print("=" * 60)

    for key, value in env_vars.items():
        if "KEY" in key or "SECRET" in key or "TOKEN" in key:
            # Show only partial value for security
            if len(value) > 12:
                display = f"{value[:8]}...{value[-4:]}"
            else:
                display = "***"
        else:
            display = value
        print(f"{key} = {display}")

    print("=" * 60)


def main():
    """Main CLI interface"""
    import argparse

    parser = argparse.ArgumentParser(
        description="Configure environment variables for GLM-Grounding skill",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  # Interactive setup
  python config_setup.py setup

  # Setup with arguments
  python config_setup.py setup --api-key YOUR_KEY

  # Show current configuration
  python config_setup.py show
        """,
    )

    subparsers = parser.add_subparsers(dest="command", help="Available commands")

    # Setup command
    setup_parser = subparsers.add_parser(
        "setup", help="Set up environment configuration"
    )
    setup_parser.add_argument("--api-key", "-k", help="GLM-V API key")
    setup_parser.add_argument(
        "--non-interactive",
        "-n",
        action="store_true",
        help="Non-interactive mode (requires --api-key)",
    )

    # Show command
    subparsers.add_parser("show", help="Show current configuration")

    # Validate command
    subparsers.add_parser("validate", help="Validate current configuration")

    args = parser.parse_args()

    if not args.command:
        parser.print_help()
        sys.exit(1)

    skill_root = _get_skill_root()

    if args.command == "setup":
        # Get configuration
        if args.[REDACTED_SECRET] = _setup_from_args(args.api_key)
        elif args.non_interactive:
            print("Error: --api-key is required in non-interactive mode")
            sys.exit(1)
        else:
            env_vars = _setup_interactive()

        # Validate
        if not _validate_config(env_vars):
            sys.exit(1)

        # Write to .env
        _write_env_file(env_vars, skill_root)

        print()
        print(f"Configuration saved to: {_get_env_path(skill_root)}")
        print()

        # Remind about .gitignore
        gitignore_path = skill_root / ".gitignore"
        if gitignore_path.exists():
            with open(gitignore_path, "r") as f:
                gitignore_content = f.read()
            if ".env" not in gitignore_content:
                print("Tip: Add '.env' to your .gitignore to keep API keys secure")
        else:
            print("Tip: Create a .gitignore file with '.env' to keep API keys secure")

    elif args.command == "show":
        _show_config()

    elif args.command == "validate":
        env_vars = _read_env_file(skill_root)

        if not env_vars:
            print("No configuration found. Run 'setup' first.")
            sys.exit(1)

        print("Validating configuration...")
        if _validate_config(env_vars):
            print("Configuration is valid!")
        else:
            print("Configuration is invalid!")
            sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/utils_boxes.py
content: #!/usr/bin/env python3
"""
本工具脚本包含：从模型的回复文本中解析坐标框，对坐标框做反向归一化，在图片上可视化画框等。
"""

import math, os, io, base64, json, ast, copy, numbers, bisect, re
from PIL import Image, ImageDraw, ImageFont
import cv2
from enum import Enum, auto
import numpy as np
import matplotlib.pyplot as plt
import colorsys


class BracketsStyle(Enum):
    """Enum for different styles of brackets."""

    SQUARE = {"left": "[", "right": "]"}  # []
    CURLY = {"left": "{", "right": "}"}  # {}
    ANGLE = {"left": "<", "right": ">"}  # <>
    PARENTHESES = {"left": "(", "right": ")"}  # ()
    BBOX = {"left": "<bbox>", "right": "</bbox>"}  # <bbox> </bbox>


class NestedStyle(Enum):
    """Enum for different styles of nested brackets."""

    DOUBLE = auto()  # [[]] or {{}}
    SINGLE = auto()  # [] or {}


def _safe_number(v):
    """Safely coerce scalar to number without eval."""
    if isinstance(v, numbers.Number):
        return v
    if isinstance(v, str):
        s = v.strip()
        if not s:
            return None
        # Support simple fractional form like "1/2"
        if re.fullmatch(r"[+-]?\d+\s*/\s*[+-]?\d+", s):
            try:
                num, den = s.split("/", 1)
                den_v = float(den)
                if den_v == 0:
                    return None
                return float(num) / den_v
            except Exception:
                return None
        # Standard int/float parsing
        try:
            if re.fullmatch(r"[+-]?\d+", s):
                return int(s)
            return float(s)
        except Exception:
            return None
    return None


def _coerce_coord_list(lst):
    """Convert a list of coordinates to numeric values safely."""
    out = []
    for item in lst:
        n = _safe_number(item)
        if n is None:
            return None
        out.append(n)
    return out


def is_single_list(lst):
    if isinstance(lst, list) and (len(lst) == 0 or not isinstance(lst[0], list)):
        return True
    return False


def is_nested_list(lst):
    if isinstance(lst, list) and len(lst) > 0 and isinstance(lst[0], list):
        return True
    return False


def flat_boxes(boxes):
    res_boxes = []

    def _flat(lst):
        if not isinstance(lst, list) or len(lst) == 0:
            return
        elif not isinstance(lst[0], list):
            res_boxes.append(lst)
        else:
            for l in lst:
                _flat(l)

    _flat(boxes)
    return res_boxes


def get_distinct_colors_hsv(n, bright=True, normalize=True):
    """
    Generate random colors.
    To get visually distinct colors, generate them in HSV space then
    convert to RGB.
    """
    brightness = 1.0 if bright else 0.7
    hsv = [(i / n, 1, brightness) for i in range(n)]
    colors = list(map(lambda c: colorsys.hsv_to_rgb(*c), hsv))
    if not normalize:
        colors = [(int(r * 255), int(g * 255), int(b * 255)) for r, g, b in colors]
    # random.shuffle(colors)
    return colors


def reverse_normalize_box(
    box, img_width=None, img_height=None, img_path=None, round_fn=lambda x: round(x, 2)
):
    if img_path is not None:
        with Image.open(img_path) as img:
            img_width, img_height = img.size
    box = [max(0, min(x, 999)) for x in box]
    box = [x / 1000.0 for x in box]
    # box = [box[0] * im_width, box[1] * img_height, box[2] * im_width, box[3] * img_height]
    box = [
        box[_] * img_width if _ % 2 == 0 else box[_] * img_height
        for _ in range(len(box))
    ]
    box = [round_fn(x) for x in box]  # round to 2 decimal places
    return box


def find_coordinates(
    text,
    coords_type="bbox",
    brackets_style=BracketsStyle.SQUARE,
    nested_style=NestedStyle.SINGLE,
    return_matches=False,
):
    """Extracts coordinates from a sentence containing bracketed representations.

    Args:
        text (str): The input sentence containing coordinates in bracketed format.
        coords_type (str, optional): Type of coordinates to extract: 'bbox' (4 values), 'point' (2 values),
            or 'polygon' (3+ values). Defaults to 'bbox'.
        brackets_style (BracketsStyle, optional): The style of brackets used to enclose coordinates.
            Defaults to BracketsStyle.SQUARE.
        nested_style (NestedStyle, optional): The nesting style (single or double brackets).
            Defaults to NestedStyle.SINGLE.
        return_matches (bool, optional): If True, also returns the matched strings and their spans.
            Defaults to False.

    Returns:
        list: If return_matches is False, returns a list of extracted coordinates.
        tuple: If return_matches is True, returns a tuple (coords, matched_strings, spans).

    Example:
        >>> find_coordinates("Two apples {{1, 2, 3, 4}, {5,6,7,8}}", coords_type='bbox')
        [[[1, 2, 3, 4], [5, 6, 7, 8]]]
    """
    # Determine expected coord length based on type
    coord_lengths = {"bbox": 4, "point": 2, "polygon": lambda x: x >= 3}
    expected_len = coord_lengths.get(coords_type, 4)

    LB, RB = brackets_style.value["left"], brackets_style.value["right"]
    LB_p, RB_p = (
        (f"\{LB}", f"\{RB}")
        if brackets_style
        in [BracketsStyle.SQUARE, BracketsStyle.CURLY, BracketsStyle.PARENTHESES]
        else (LB, RB)
    )
    if nested_style == NestedStyle.DOUBLE:
        pattern = r"(%s\s*%s.*?%s\s*%s)" % (LB_p, LB_p, RB_p, RB_p)
    else:
        pattern = r"(%s\s*[\d\.,\s]+\s*%s)" % (LB_p, RB_p)

    res_coords, res_strs, res_spans = [], [], []
    for item in re.finditer(pattern, text):
        try:
            coords_lst = item.group(1).replace(LB, "[").replace(RB, "]")
            coords = ast.literal_eval(coords_lst)

            # Check based on coords_type
            if is_nested_list(coords):
                if callable(expected_len):
                    is_valid = all([expected_len(len(c)) for c in coords])
                else:
                    is_valid = all([len(c) == expected_len for c in coords])
                if is_valid:
                    parsed = [_coerce_coord_list(c) for c in coords]
                    if any(p is None for p in parsed):
                        continue
                    coords = parsed
                else:
                    continue
            elif is_single_list(coords):
                if callable(expected_len):
                    is_valid = expected_len(len(coords))
                else:
                    is_valid = len(coords) == expected_len
                if is_valid:
                    parsed = _coerce_coord_list(coords)
                    if parsed is None:
                        continue
                    coords = parsed
                else:
                    continue
            else:
                continue

            res_coords.append(coords)
            res_strs.append(item.group(1))
            res_spans.append(item.span())
        except:
            pass

    if return_matches:
        return res_coords, res_strs, res_spans
    else:
        return res_coords


def find_coordinates_all(text, coords_type="bbox", flat=True):
    found, ps_l, ps_r = [], [], []
    for nested_style in NestedStyle:  # Double first
        for brackets_style in BracketsStyle:
            for bbx, bbx_str, bbx_span in zip(
                *find_coordinates(
                    text,
                    coords_type,
                    brackets_style,
                    nested_style=nested_style,
                    return_matches=True,
                )
            ):
                ls = bisect.bisect_right(ps_l, bbx_span[0])
                if ls > 0 and bbx_span[1] <= ps_r[ls - 1]:
                    continue  # fall within existing span
                found.append((bbx, bbx_str, bbx_span))
                ps_l.insert(ls, bbx_span[0])
                ps_r.insert(ls, bbx_span[1])
    # ordering
    found_sort = []
    for bbx_str_span in sorted(found, key=lambda x: x[2][0]):
        found_sort.append(bbx_str_span[0])
    if len(found_sort) == 0:
        found_sort = [[]]  # default behavior
    if flat:
        found_sort = flat_boxes(found_sort)
    return found_sort


def draw_texts_on_frame(
    annotated_frame, points, labels, colors, renormalize=False, imgsize_wh=None
):
    # Draw labels with PIL (supports Chinese/UTF-8)
    pil_img = Image.fromarray(annotated_frame)
    draw = ImageDraw.Draw(pil_img)
    font_size = 18
    font = None
    for font_path in [
        "/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc",
        "/usr/share/fonts/opentype/unifont/unifont.otf",
        "/usr/share/fonts/truetype/droid/DroidSansFallbackFull.ttf",
    ]:
        if os.path.exists(font_path):
            font = ImageFont.truetype(font_path, font_size)
            break
    if font is None:
        font = ImageFont.load_default()
    # Re-apply renormalization for label positions
    for i, (point, lbl) in enumerate(zip(points, labels)):
        if renormalize:
            point = reverse_normalize_box(
                point, imgsize_wh[0], imgsize_wh[1], round_fn=lambda x: int(x)
            )
        text_pos = (point[0], max(point[1] - font_size - 4, 0))
        draw.text(text_pos, lbl, font=font, fill=colors[i])
    annotated_frame = np.array(pil_img)
    return annotated_frame


def visualize_boxes(
    img_path: str = None,
    img_bytes=None,
    boxes: list = [],
    labels: list = None,
    renormalize=False,
    save_path: str = None,
    return_b64=False,
    save_optimized=True,
    **kwargs,
):
    """
    Visualizes bounding boxes and labels on an image.

    Args:
        img_path (str, optional): Path to the input image file. Either `img_path` or `img_bytes` must be provided.
        img_bytes (bytes, optional): Image data in bytes. Used if `img_path` is not provided.
        boxes (list): List of bounding boxes, each box as [x1, y1, x2, y2].
        labels (list, optional): List of labels corresponding to each box. If None, empty labels are used.
        renormalize (bool, optional): If True, renormalizes box coordinates based on image size.
        save_path (str, optional): Path to save the annotated image. If None, image is not saved.
        return_b64 (bool, optional): If True, returns the annotated image as a base64-encoded string.
        save_optimized (bool, optional): If True, saves the image with JPEG optimization.
        **kwargs:
            colors (list, optional): List of colors for each box. Default is 'red' for all boxes.
            thickness (list, optional): List of thickness values for each box. Default is 2 for all boxes.

    Returns:
        np.ndarray or str: Annotated image as a numpy array (RGB) or base64 string if `return_b64` is True.

    Raises:
        AssertionError: If neither `img_path` nor `img_bytes` is provided.

    Example:
        >>> visualize_boxes(img_path="image.jpg", boxes=[[10, 20, 100, 200]], labels=["cat"], save_path="out.jpg")
    """
    assert (
        img_path is not None or img_bytes is not None
    ), "Either img_path or img_bytes must be provided."
    colors = kwargs.get("colors", ["red"] * len(boxes))
    thickness = kwargs.get("thickness", [2] * len(boxes))
    if img_path:
        img = cv2.imread(img_path)
    else:
        image_np = np.frombuffer(img_bytes, dtype=np.uint8)
        img = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # Convert to RGB
    image_h, image_w, _ = img.shape

    labels = [""] * len(boxes) if labels is None else labels
    colors = [
        tuple(int(c * 255) for c in plt.matplotlib.colors.to_rgb(_)) for _ in colors
    ]

    annotated_frame = img.copy()
    for i, (box, lbl) in enumerate(zip(boxes, labels)):
        box = [int(_) for _ in box]
        box = (
            reverse_normalize_box(box, image_w, image_h, round_fn=lambda x: int(x))
            if renormalize
            else box
        )
        cv2.rectangle(
            annotated_frame, box[:2], box[2:], color=colors[i], thickness=thickness[i]
        )
        # cv2.putText does not support Chinese; use PIL instead
        # cv2.putText(annotated_frame, lbl, (box[0], box[1]-8), cv2.FONT_HERSHEY_SIMPLEX, 0.5, colors[i],  1)

    # Draw labels with PIL (supports Chinese/UTF-8)
    annotated_frame = draw_texts_on_frame(
        annotated_frame,
        boxes,
        labels,
        colors,
        renormalize=renormalize,
        imgsize_wh=(image_w, image_h),
    )

    if save_path:
        pil_img = Image.fromarray(annotated_frame)
        if save_optimized:
            pil_img.save(save_path, format="JPEG", quality=85, optimize=True)
        else:
            pil_img.save(save_path)
    if return_b64:
        pil_img = Image.fromarray(annotated_frame)
        buffered = io.BytesIO()
        pil_img.save(buffered, format="JPEG")
        annotated_frame = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return annotated_frame


def visualize_points(
    img_path: str = None,
    img_bytes=None,
    points: list = [],
    labels: list = None,
    renormalize=False,
    diameters=None,
    save_path: str = None,
    return_b64=False,
    save_optimized=True,
    distinct_colors=False,
    colors=None,
):
    assert (
        img_path is not None or img_bytes is not None
    ), "Either img_path or img_bytes must be provided."
    if img_path:
        img = cv2.imread(img_path)
    else:
        image_np = np.frombuffer(img_bytes, dtype=np.uint8)
        img = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # Convert to RGB
    image_h, image_w, _ = img.shape
    diameters = diameters or [max(3, min(img.shape) / 40)] * len(points)

    labels = [""] * len(points) if labels is None else labels
    colors = (
        [(255, 0, 0)] * len(points)
        if colors is None
        else [
            tuple(int(c * 255) for c in plt.matplotlib.colors.to_rgb(_)) for _ in colors
        ]
    )
    if distinct_colors:
        colors = get_distinct_colors_hsv(len(points), normalize=False)

    annotated_frame = img.copy()
    for i, (point, lbl) in enumerate(zip(points, labels)):
        point = [int(_) for _ in point]
        point = (
            reverse_normalize_box(point, image_w, image_h, round_fn=lambda x: int(x))
            if renormalize
            else point
        )
        # Draw points
        cv2.circle(
            annotated_frame,
            (point[0], point[1]),
            radius=diameters[i],
            color=colors[i],
            thickness=-1,
        )  # thickness=-1表示填充
    # Draw labels
    annotated_frame = draw_texts_on_frame(
        annotated_frame,
        points,
        labels,
        colors,
        renormalize=renormalize,
        imgsize_wh=(image_w, image_h),
    )

    if save_path:
        if save_optimized:
            Image.fromarray(annotated_frame).save(
                save_path, format="JPEG", quality=85, optimize=True
            )
        else:
            Image.fromarray(annotated_frame).save(save_path)
    if return_b64:
        pil_img = Image.fromarray(annotated_frame)
        buffered = io.BytesIO()
        pil_img.save(buffered, format="JPEG")
        annotated_frame = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return annotated_frame


def visualize_polys(
    img_path: str = None,
    img_bytes=None,
    polys: list = [],
    labels: list = None,
    is_closed=True,
    renormalize=False,
    save_path: str = None,
    return_b64=False,
    save_optimized=True,
):
    """polys: [[x1,y1], [x2,y2], ...], where the [x1,y2] is the top-left corner."""
    assert (
        img_path is not None or img_bytes is not None
    ), "Either img_path or img_bytes must be provided."
    if img_path:
        img = cv2.imread(img_path)
    else:
        image_np = np.frombuffer(img_bytes, dtype=np.uint8)
        img = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # Convert to RGB
    image_h, image_w, _ = img.shape

    labels = [""] * len(polys) if labels is None else labels

    annotated_frame = img.copy()
    text_poss = []
    for poly, lbl in zip(polys, labels):
        poly = [[int(_) for _ in _p] for _p in poly]
        poly = (
            [
                reverse_normalize_box(_p, image_w, image_h, round_fn=lambda x: int(x))
                for _p in poly
            ]
            if renormalize
            else poly
        )
        # box = [int(box[0]*image_w), int(box[1]*image_h), int(box[2]*image_w), int(box[3]*image_h)]
        # Draw boxes
        cv2.polylines(
            annotated_frame,
            np.array([poly], dtype=np.int32),
            isClosed=is_closed,
            color=(144, 238, 144),
            thickness=2,
        )
        # Draw labels
        # cv2.putText(annotated_frame, label, coord, font, scale, (0, 0, 0),  1)
        # cv2.putText(annotated_frame, lbl, poly[0], cv2.FONT_HERSHEY_SIMPLEX, 0.5, (144,238,144),  1)
        text_poss.append(poly[0])
    # Draw label texts
    annotated_frame = draw_texts_on_frame(
        annotated_frame,
        text_poss,
        labels,
        [(144, 238, 144)] * len(polys),
        renormalize=renormalize,
        imgsize_wh=(image_w, image_h),
    )

    if save_path:
        if save_optimized:
            Image.fromarray(annotated_frame).save(
                save_path, format="JPEG", quality=85, optimize=True
            )
        else:
            Image.fromarray(annotated_frame).save(save_path)
    if return_b64:
        pil_img = Image.fromarray(annotated_frame)
        buffered = io.BytesIO()
        pil_img.save(buffered, format="JPEG")
        annotated_frame = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return annotated_frame


def parse_coordinates_from_response(
    response_str, coords_type="bbox", init_context_window=2000, max_context_window=-1
):
    """从模型回复文本中解析出有坐标值数组（coordinates组成的list，坐标值可能是bbox、point或polygon）。基本逻辑如下：
    初始，检查从后向前数第`init_context_window`个字符是否是bbox的一部分（即被截断），如果是，则向前拓展`init_context_window`，直至前一个字符不是bbox的一部分，或达到了`max_context_window`。
    然后，从选取的context_window中调用find_coordinates_all方法，找到所有坐标，flat之后返回list。
    """
    if not isinstance(response_str, str):
        return []
    context_window = init_context_window
    max_context_window = (
        len(response_str) if max_context_window < 0 else max_context_window
    )
    start_idx = max(0, len(response_str) - context_window)
    # 向前拓展context_window，直到前一个字符不是bbox的一部分
    bbox_chars = set("0123456789.,[]{}()<>' ")
    # 从枚举类BracketsStyle中获取所有括号字符
    for style in BracketsStyle:
        bbox_chars.update(style.value["left"])
        bbox_chars.update(style.value["right"])
    for style in BracketsStyle:
        bbox_chars.update(style.value["left"])
        bbox_chars.update(style.value["right"])
    while (
        start_idx > 0
        and response_str[start_idx - 1] in bbox_chars
        and context_window < max_context_window
    ):
        context_window += 1
        start_idx = max(0, len(response_str) - context_window)
    context_str = response_str[start_idx:]

    # 按不同的坐标类型解析
    res_coords = find_coordinates_all(context_str, coords_type=coords_type, flat=True)
    return res_coords


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)