Skill: highnoonoffice/agent-tollbooth (version 2.2.0)

--- name: agent-tollbooth version: 2.0.0 description: "Web access privileges for your agent. So your agent stops hitting walls." homepage: https://github.com/highnoonoffice/agent-tollbooth source: https://github.com/highnoonoffice/agent-tollbooth license: MIT metadata: ~ config: - $OPENCLAW_WORKSPACE/data/agent-tollbooth/web-access-log.json — event log written by scripts/web-log.py - $OPENCLAW_WORKSPACE/data/agent-tollbooth/cache/ — price and response cache written by fetch-crypto.py and fetch-prices.py --- # Agent Tollbooth You're mid-task. Your agent fires a Yahoo Finance request. Gets a 429. Stops. You don't know if it's rate limits, a bad endpoint, a missing header, or just bad luck. You try again. Same thing. You start debugging blind. Tollbooth is the field notes that stop this from happening twice. Observed operating profiles for 16 external services — safe endpoints, sleep intervals, caching patterns, auth requirements — built from real API friction. Your agent checks the profile before calling, follows the safe pattern, and logs what happens. Next time it already knows. Every external service has a threshold. This skill provides the map so agents learn them once and stop hitting them twice. ## Core Pattern Before calling any external service: 1. Check `references/profiles.md` for an existing profile 2. Follow its safe pattern — endpoint, sleep, cache, auth 3. If no profile exists, observe behavior and add an entry after ## Caching Always cache prices and API responses locally when TTL allows. - Default TTL: 300s (5 minutes) for prices - Cache file: `$OPENCLAW_WORKSPACE/data/agent-tollbooth/cache/` - Serve from cache first — only hit the API when stale or forced **Script:** `scripts/fetch-prices.py` implements cache + sequential Yahoo Finance fetching. Use it instead of raw requests. CoinGecko and other services are covered by profiles in `references/profiles.md`. ## How It Learns Tollbooth grows with your usage. Three scripts form the learning loop: **Before any external call:** ```bash python3 scripts/check-profile.py coingecko.com ``` Returns the safe pattern if a profile exists. If not, logs the miss and returns exit code 1 — your agent can continue, but observation has started. **During any call — log what happens:** ```python from scripts.web_log import log_event log_event("my-api.com", "429", "hit rate limit at 10 req/min", worked=None) log_event("my-api.com", "success", "sequential 500ms sleep worked", worked="sequential + 500ms sleep") ``` **After enough observations — promote to a profile:** ```bash python3 scripts/promote-profile.py # dry run, see what's ready python3 scripts/promote-profile.py --write # append drafts to profiles.md ``` Default threshold: 5 events. Auto-drafted profiles include all observed friction and working patterns. Review before trusting — they're drafts, not finished entries. Events are written to `$OPENCLAW_WORKSPACE/data/agent-tollbooth/web-access-log.json` — outside the skill bundle, never modifying packaged files. Cache files go to `$OPENCLAW_WORKSPACE/data/agent-tollbooth/cache/`. Set `OPENCLAW_WORKSPACE` before running (standard on any OpenClaw install). ## Workspace Access This skill writes event logs and cache files to `$OPENCLAW_WORKSPACE/data/agent-tollbooth/`. No credentials are accessed. No sensitive data is written. No files outside this directory are touched. ## Service Profiles See `references/profiles.md` for all current profiles: - Yahoo Finance - CoinGecko - Ghost Admin API - ClawHub API - Telegram Bot API - Replicate - OpenAI API - Anthropic API - GitHub API - Brave Search API - Serper (Google Search) - Notion API - Airtable API - Stripe API - HuggingFace Inference API - Firecrawl

path: references/profiles.md
content: # Service Profiles — Observed Operating Behavior

Each entry: safe pattern, known failure modes, first observed date.

---

## Yahoo Finance

- **Endpoint:** `query2.finance.yahoo.com` — preferred. `query1` rate-limits faster.
- **Auth:** None — User-Agent header required (`Mozilla/5.0 ...`)
- **Safe pattern:** Sequential requests, 0.6s sleep between tickers
- **Never:** Parallel requests — triggers 429 immediately
- **Cache:** 5 minutes minimum — serve from cache whenever possible
- **Crypto:** Use CoinGecko instead — single batch call, more reliable
- **Script:** `scripts/fetch-prices.py` wraps all of this automatically
- **First observed:** 2026-04-17

---

## CoinGecko

- **Endpoint:** `api.coingecko.com/api/v3/simple/price`
- **Auth:** None on free tier
- **Safe pattern:** Batch all coins in one call — `ids=bitcoin,ethereum,solana&vs_currencies=usd&include_24hr_change=true`
- **Free tier:** Generous — rarely rate-limits on normal usage
- **Cache:** 5 minutes via `fetch-prices.py`
- **First observed:** 2026-04-17

---

## Ghost Admin API

- **Endpoint:** `https://<site>.ghost.io/ghost/api/admin/`
- **Auth:** JWT token generated fresh each call from Admin API key
- **Key location:** your credentials file → field `key` (format: `{"key": "<id>:<secret>"}`)
- **Permission wall:** Integration tokens cannot write to `/settings/` — owner-only. Browser automation required for code injection and theme uploads.
- **Image uploads:** Python `requests` only — curl multipart breaks on macOS zsh
- **First observed:** 2026-03-17

---

## ClawHub API

- **Endpoint:** `https://clawhub.ai/api/v1/skills`
- **Auth:** Bearer token from your credentials file
- **Safe pattern:** Python `requests` with multipart — `data={"payload": json.dumps(payload)}` + `files=[...]`
- **Never:** curl — JSON quoting breaks silently on zsh, returns `Invalid JSON payload` every time
- **Never:** Browser importer — intermittent server errors, unreliable
- **First observed:** 2026-03-18

---

## Telegram Bot API

- **Pattern:** Use OpenClaw `message` tool — never raw curl
- **File delivery:** `sendDocument` endpoint for PDFs and media
- **Rate limits:** 30 messages/second global, 1 message/second per chat
- **First observed:** 2026-03-14

---

## Replicate (FLUX image gen)

- **Auth:** API token via OpenClaw config
- **Error quirk:** "Less than $5.0 in credit" error message is unreliable — check dashboard for real balance
- **Cost:** FLUX 1.1 Pro ~$0.02–0.04/image — $15 balance = 350–700 images. Never needs a top-up for normal usage.
- **First observed:** 2026-03-19

---

## OpenAI API

- **Endpoint:** `api.openai.com/v1/`
- **Auth:** Bearer token — `Authorization: Bearer sk-...`
- **Rate limits:** Tier-dependent. Free tier: 3 RPM / 200 RPD. Tier 1+: 500–3,500 RPM depending on model. Check `x-ratelimit-remaining-requests` response header.
- **Token limits separate from request limits:** A single request can burn your TPM ceiling without hitting RPM. Watch both.
- **Safe pattern:** Exponential backoff on 429. Header `retry-after` tells you exactly how long to wait — use it.
- **Streaming vs. non-streaming:** Both count equally against rate limits. Streaming does NOT get preferential treatment.
- **Model-specific ceilings:** GPT-4o has lower RPM than GPT-4o-mini on the same tier. Always check the model's specific limit, not the account's general limit.
- **Never:** Parallel requests on free/tier-1 — you'll 429 immediately and waste retries
- **Cache:** Cache completions aggressively. Same prompt = same response at temperature 0. Save tokens and money.
- **First observed:** 2026-04-18

---

## Anthropic API

- **Endpoint:** `api.anthropic.com/v1/messages`
- **Auth:** `x-api-key` header — not Bearer format
- **Rate limits:** Two separate ceilings: RPM (requests per minute) AND TPM (tokens per minute). Hitting either triggers 429.
- **TPM is usually the binding constraint** — a single large context window can eat your TPM budget before RPM becomes relevant
- **Model-specific limits:** Haiku has higher RPM/TPM ceilings than Sonnet, which has higher than Opus. Route cheap tasks to Haiku.
- **Prompt caching:** Prefix caching cuts token cost 90% on cached portions. Cache prefix must be identical to the character — a single token difference breaks the cache.
- **Safe pattern:** Check `x-ratelimit-remaining-tokens` and `x-ratelimit-remaining-requests` headers on every response
- **Streaming:** Use streaming for long responses — avoids gateway timeouts on responses > 30s
- **Never:** Assume Haiku limits apply to Sonnet or vice versa — they're tracked independently
- **First observed:** 2026-04-18

---

## GitHub API

- **Endpoint:** `api.github.com`
- **Auth:** `Authorization: Bearer <PAT>` — classic or fine-grained PAT
- **Primary rate limit:** 5,000 req/hr authenticated. 60 req/hr unauthenticated. Check `x-ratelimit-remaining` header.
- **Secondary rate limits (the real gotcha):** Separate from the primary limit. Triggered by: too many write operations in a short window, too many concurrent requests, too many requests to a single endpoint. Returns 403, not 429.
- **Safe pattern:** Sequential writes with 1s sleep between. Never fire parallel POST/PATCH/DELETE calls.
- **Search API:** Separate limit — 30 req/min authenticated, 10 unauthenticated. Completely independent of the main 5,000/hr.
- **GraphQL:** Single endpoint `api.github.com/graphql` — counts as one request regardless of query complexity, but has its own point budget
- **Pagination:** Always paginate — default page size is 30, max 100. Never assume you got all results on page 1.
- **Never:** Use unauthenticated requests for anything beyond occasional public repo reads
- **First observed:** 2026-04-18

---

## Brave Search API

- **Endpoint:** `api.search.brave.com/res/v1/web/search`
- **Auth:** `X-Subscription-Token` header
- **Model:** Credit-based, NOT rate-based. Each query costs credits regardless of response size.
- **Free tier:** 2,000 queries/month. Does not reset on 429 — it hard-stops when credits are exhausted.
- **Failure mode:** Returns 429 when monthly credits are gone. This is a budget wall, not a speed wall — waiting does not help. Credits reset on billing date.
- **Safe pattern:** Cache results aggressively. Same query within a session = serve from cache, don't re-query.
- **No retry value on 429:** Unlike most APIs, retrying a Brave 429 immediately is useless — you're out of credits, not over a speed limit.
- **Results freshness:** `freshness` param accepts `day`, `week`, `month`, `year` — use it to filter stale results on time-sensitive queries
- **First observed:** 2026-04-18

---

## Serper (Google Search API)

- **Endpoint:** `google.serper.dev/search`
- **Auth:** `X-API-KEY` header
- **Model:** Credit-based. Free tier: 2,500 credits. Each search = 1 credit.
- **Failure mode:** Returns 403 when credits exhausted. Credits don't reset — must purchase more.
- **Safe pattern:** Batch intent before querying. One well-crafted query beats three exploratory ones.
- **Result types:** `/search` (web), `/images`, `/news`, `/places`, `/scholar` — each costs 1 credit regardless of type
- **Never:** Fire exploratory parallel queries — each one costs a credit whether useful or not
- **vs. Brave:** Serper returns Google results (higher coverage), Brave returns Brave index (more privacy-friendly, often enough for research). Use Brave first, Serper when Google coverage matters.
- **First observed:** 2026-04-18

---

## Notion API

- **Endpoint:** `api.notion.com/v1/`
- **Auth:** Bearer integration token + `Notion-Version: 2022-06-28` header (required)
- **Rate limit:** 3 requests/second per integration. Hard wall — 429 with no retry-after header.
- **Safe pattern:** 350ms sleep between requests. Never parallel.
- **Pagination:** All list endpoints are paginated — `has_more` + `next_cursor` pattern. Max page size: 100. Always loop until `has_more: false`.
- **Block vs. page:** Fetching a page gives you metadata only, not content. Content lives in blocks — separate `GET /blocks/{id}/children` call required.
- **Write quirk:** Appending blocks is additive — there's no replace operation. To update content you must delete existing blocks first, then append.
- **Never:** Assume a page fetch includes body content — it never does
- **First observed:** 2026-04-18

---

## Airtable API

- **Endpoint:** `api.airtable.com/v0/<base_id>/<table_name>`
- **Auth:** `Authorization: Bearer <personal-access-token>`
- **Rate limit:** 5 requests/second per base. Across all tables in that base combined.
- **Safe pattern:** 200ms sleep between requests. Batch reads using `filterByFormula` instead of fetching all and filtering client-side.
- **Pagination:** `offset` token in response — keep fetching until no offset returned. Default page size: 100 records.
- **Bulk writes:** No true bulk insert. Create up to 10 records per request via array in `records` field. 10 at a time, sleep between batches.
- **Formula quirk:** `filterByFormula` uses Airtable formula syntax, not SQL. Strings need curly braces: `{Field Name} = 'value'`
- **Never:** Fetch all records then filter in code — use `filterByFormula` to push filtering server-side
- **First observed:** 2026-04-18

---

## Stripe API

- **Endpoint:** `api.stripe.com/v1/`
- **Auth:** Bearer token (secret key) — separate test and live keys, never mix environments
- **Rate limit:** 100 read requests/second, 100 write requests/second. Rarely hit in normal agent workflows.
- **Idempotency keys (critical):** Always pass `Idempotency-Key` header on POST requests. Same key = same result, safe to retry. Without it, a network timeout can trigger duplicate operations.
- **Test vs. live:** Test keys hit a completely separate environment. Test mode objects don't exist in live mode. Never assume an ID from test works in live.
- **Pagination:** Cursor-based — `starting_after` and `ending_before` params. Default 10 objects, max 100.
- **Webhooks:** Verify `Stripe-Signature` header on every incoming webhook — do not trust unverified events.
- **Safe pattern:** Always use idempotency keys on writes. Log the key with the request so you can retry safely.
- **Never:** Mix test and live keys, or skip idempotency keys on sensitive operations
- **First observed:** 2026-04-18

---

## HuggingFace Inference API

- **Endpoint:** `api-inference.huggingface.co/models/<model-id>`
- **Auth:** `Authorization: Bearer hf_...`
- **Free tier:** Rate-limited, no hard quota published — in practice ~10-30 req/min before throttling
- **Cold start (the real issue):** Models on free tier spin down when idle. First request after idle returns 503 with `{"error": "Model ... is currently loading"}` and an `estimated_time` field in seconds. This is NOT a failure — wait and retry.
- **Safe pattern:** On 503 with `estimated_time`, sleep that many seconds + 5s buffer, then retry once. If still 503, wait another 30s.
- **Never:** Treat a loading 503 as a hard failure — it's a cold start, not an error
- **Dedicated endpoints:** If you need reliable latency, use Inference Endpoints (paid) — no cold starts, dedicated hardware
- **Model-specific behavior:** Some models require specific input formats. Always check the model card before querying a new model.
- **First observed:** 2026-04-18

---

## Firecrawl

- **Endpoint:** `api.firecrawl.dev/v1/scrape` (single URL) or `/v1/crawl` (multi-page)
- **Auth:** `Authorization: Bearer fc-...`
- **Model:** Credit-based. Free tier: 500 credits/month. Single scrape = 1 credit. Crawl = 1 credit per page crawled.
- **JS rendering:** Enabled by default — adds 3-8s latency per page vs. static fetch. Use `formats: ["markdown"]` to get clean LLM-ready output.
- **vs. web_fetch:** Firecrawl handles JS-rendered pages, paywalls, and complex layouts. Use web_fetch for simple static pages — faster and free. Escalate to Firecrawl when web_fetch returns garbled or incomplete content.
- **Crawl vs. scrape:** `/scrape` is one URL, instant response. `/crawl` is async — returns a job ID, poll `/v1/crawl/<id>` until `status: completed`.
- **Safe pattern:** Scrape first. Only crawl if you need multiple pages from the same domain.
- **Never:** Use `/crawl` when `/scrape` is sufficient — crawl burns credits per page
- **First observed:** 2026-04-18


path: scripts/web-log.py
content: #!/usr/bin/env python3
"""Shared event logger for web access/fetch scripts."""

from __future__ import annotations

import argparse
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional

ALLOWED_EVENT_TYPES = {
    "429",
    "timeout",
    "auth_failure",
    "cache_hit",
    "success",
    "workaround",
}


SCRIPT_DIR = Path(__file__).resolve().parent
MODULE_DIR = SCRIPT_DIR.parent

# Write operational data outside the skill bundle.
# Resolves to $OPENCLAW_WORKSPACE/data/agent-tollbooth/ on any standard OpenClaw install.
# Falls back to a local data/ dir if the workspace is not found.
import os
_WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", Path.home() / ".openclaw" / "workspace"))
DATA_DIR = _WORKSPACE / "data" / "agent-tollbooth"
LOG_PATH = DATA_DIR / "web-access-log.json"
PROFILES_PATH = DATA_DIR / "profiles-annotations.md"


def _ensure_log_file() -> None:
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    if not LOG_PATH.exists():
        LOG_PATH.write_text("[]\n", encoding="utf-8")


def _format_timestamp_for_profile(timestamp: str) -> str:
    try:
        parsed = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
    except ValueError:
        return timestamp
    return parsed.astimezone(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")


def annotate_profile(service: str, event_type: str, detail: str, timestamp: str) -> None:
    if not PROFILES_PATH.exists():
        return

    lines = PROFILES_PATH.read_text(encoding="utf-8").splitlines()
    if not lines:
        return

    target_heading = f"## {service}".strip().lower()
    section_start = None
    for idx, line in enumerate(lines):
        if line.strip().lower() == target_heading:
            section_start = idx
            break
    if section_start is None:
        return

    section_end = len(lines)
    for idx in range(section_start + 1, len(lines)):
        if lines[idx].startswith("## "):
            section_end = idx
            break

    formatted_ts = _format_timestamp_for_profile(timestamp)
    line_prefix = f"- [{formatted_ts}] {event_type}:"
    for existing in lines[section_start:section_end]:
        if existing.startswith(line_prefix):
            return

    annotation = f"- [{formatted_ts}] {event_type}: {detail}"
    lines.insert(section_end, annotation)
    PROFILES_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")


def log_event(
    service: str,
    event_type: str,
    detail: str,
    worked: Optional[str] = None,
) -> None:
    """Append a web access event to data/web-access-log.json."""
    if event_type not in ALLOWED_EVENT_TYPES:
        raise ValueError(f"Unsupported event_type: {event_type}")

    _ensure_log_file()
    try:
        events = json.loads(LOG_PATH.read_text(encoding="utf-8") or "[]")
    except json.JSONDecodeError:
        events = []

    entry = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "service": str(service),
        "event_type": event_type,
        "detail": str(detail),
        "worked": None if worked is None else str(worked),
    }
    events.append(entry)
    LOG_PATH.write_text(json.dumps(events, indent=2) + "\n", encoding="utf-8")
    annotate_profile(
        service=entry["service"],
        event_type=entry["event_type"],
        detail=entry["detail"],
        timestamp=entry["timestamp"],
    )

def _cli() -> int:
    parser = argparse.ArgumentParser(description="Append an entry to web-access-log.json")
    parser.add_argument("service")
    parser.add_argument("event_type", choices=sorted(ALLOWED_EVENT_TYPES))
    parser.add_argument("detail")
    parser.add_argument("--worked", default=None)
    parser.add_argument("--verbose", action="store_true")
    args = parser.parse_args()

    log_event(service=args.service, event_type=args.event_type, detail=args.detail, worked=args.worked)
    if args.verbose:
        _ensure_log_file()
        events = json.loads(LOG_PATH.read_text(encoding="utf-8") or "[]")
        if events:
            print(json.dumps(events[-1], indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(_cli())


path: scripts/fetch-prices.py
content: #!/usr/bin/env python3
"""Fetch stock prices with lightweight disk caching and structured web-access logging."""

from __future__ import annotations

import argparse
import hashlib
import importlib.util
import json
import os
import socket
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List
from urllib.error import HTTPError, URLError
from urllib.request import urlopen

SCRIPT_DIR = Path(__file__).resolve().parent
MODULE_DIR = SCRIPT_DIR.parent

# Write cache outside the skill bundle.
# Resolves to $OPENCLAW_WORKSPACE/data/agent-tollbooth/ on any standard OpenClaw install.
# Falls back to a local data/ dir if the workspace is not found.
_WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", Path.home() / ".openclaw" / "workspace"))
OCLAW_DATA_DIR = _WORKSPACE / "data" / "agent-tollbooth"
DATA_DIR = OCLAW_DATA_DIR if _WORKSPACE.exists() else MODULE_DIR / "data"
CACHE_DIR = DATA_DIR / "cache"


def _load_logger():
    logger_path = SCRIPT_DIR / "web-log.py"
    spec = importlib.util.spec_from_file_location("web_log", logger_path)
    if spec is None or spec.loader is None:
        raise RuntimeError(f"Unable to load logger at {logger_path}")
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.log_event


log_event = _load_logger()


def _cache_path(symbol: str) -> Path:
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
    digest = hashlib.sha256(symbol.encode("utf-8")).hexdigest()[:16]
    return CACHE_DIR / f"{symbol.upper()}-{digest}.json"


def _load_cached(symbol: str, ttl_seconds: int) -> Dict[str, Any] | None:
    path = _cache_path(symbol)
    if not path.exists():
        return None

    try:
        payload = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError:
        return None

    ts = payload.get("cached_at")
    if not ts:
        return None

    try:
        cached_at = datetime.fromisoformat(ts)
    except ValueError:
        return None

    age = (datetime.now(timezone.utc) - cached_at).total_seconds()
    if age <= ttl_seconds:
        log_event(
            service="Yahoo Finance",
            event_type="cache_hit",
            detail=f"Used cached quote for {symbol}",
            worked="true",
        )
        return payload.get("quote")

    return None


def _save_cache(symbol: str, quote: Dict[str, Any]) -> None:
    payload = {
        "cached_at": datetime.now(timezone.utc).isoformat(),
        "quote": quote,
    }
    _cache_path(symbol).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")


def _fetch_quote(symbol: str, timeout_seconds: float) -> Dict[str, Any]:
    url = f"https://query2.finance.yahoo.com/v8/finance/chart/{symbol}?interval=1d&range=5d"

    try:
        with urlopen(url, timeout=timeout_seconds) as response:
            body = response.read().decode("utf-8")
    except HTTPError as exc:
        if exc.code == 429:
            log_event(
                service="Yahoo Finance",
                event_type="429",
                detail=f"HTTP 429 while fetching {symbol}",
                worked="false",
            )
        elif exc.code in {401, 403}:
            log_event(
                service="Yahoo Finance",
                event_type="auth_failure",
                detail=f"HTTP {exc.code} while fetching {symbol}",
                worked="false",
            )
        raise
    except (TimeoutError, socket.timeout) as exc:
        log_event(
            service="Yahoo Finance",
            event_type="timeout",
            detail=f"Timeout while fetching {symbol}: {exc}",
            worked="false",
        )
        raise
    except URLError as exc:
        if isinstance(exc.reason, socket.timeout):
            log_event(
                service="Yahoo Finance",
                event_type="timeout",
                detail=f"URLError timeout while fetching {symbol}: {exc.reason}",
                worked="false",
            )
        raise

    payload = json.loads(body)
    results = payload.get("chart", {}).get("result", [])
    if not results:
        raise RuntimeError(f"No quote data returned for {symbol}")

    meta = results[0].get("meta", {})
    quote = {
        "symbol": meta.get("symbol", symbol.upper()),
        "regularMarketPrice": meta.get("regularMarketPrice"),
        "currency": meta.get("currency"),
        "regularMarketTime": meta.get("regularMarketTime"),
    }
    log_event(
        service="Yahoo Finance",
        event_type="success",
        detail=f"Fetched live quote for {symbol}",
        worked="true",
    )
    return quote


def fetch_price(symbol: str, timeout_seconds: float, cache_ttl_seconds: int) -> Dict[str, Any]:
    symbol = symbol.upper().strip()

    cached = _load_cached(symbol, cache_ttl_seconds)
    if cached is not None:
        return cached

    quote = _fetch_quote(symbol, timeout_seconds)
    _save_cache(symbol, quote)
    return quote


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Fetch quotes from Yahoo Finance")
    parser.add_argument("symbols", nargs="+", help="Ticker symbols (e.g., AAPL MSFT TSLA)")
    parser.add_argument("--timeout", type=float, default=8.0, help="HTTP timeout in seconds")
    parser.add_argument("--cache-ttl", type=int, default=300, help="Cache TTL in seconds")
    parser.add_argument("--sleep", type=float, default=0.0, help="Sleep between requests")
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    output: List[Dict[str, Any]] = []

    for idx, symbol in enumerate(args.symbols):
        quote = fetch_price(symbol, timeout_seconds=args.timeout, cache_ttl_seconds=args.cache_ttl)
        output.append(
            {
                "symbol": quote.get("symbol", symbol.upper()),
                "price": quote.get("regularMarketPrice"),
                "currency": quote.get("currency"),
                "asOf": quote.get("regularMarketTime"),
            }
        )

        if args.sleep and idx < len(args.symbols) - 1:
            time.sleep(args.sleep)

    print(json.dumps(output, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/fetch-crypto.py
content: #!/usr/bin/env python3
"""Fetch CoinGecko crypto prices with disk caching and structured web-access logging."""

from __future__ import annotations

import argparse
import hashlib
import importlib.util
import json
import os
import socket
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import urlopen

SCRIPT_DIR = Path(__file__).resolve().parent
OPENCLAW_WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", Path.home() / ".openclaw" / "workspace")).expanduser()
DATA_DIR = OPENCLAW_WORKSPACE / "data" / "agent-tollbooth"
CACHE_DIR = DATA_DIR / "cache"


def _load_logger():
    logger_path = SCRIPT_DIR / "web-log.py"
    spec = importlib.util.spec_from_file_location("web_log", logger_path)
    if spec is None or spec.loader is None:
        raise RuntimeError(f"Unable to load logger at {logger_path}")
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.log_event


log_event = _load_logger()


def _cache_path(coins: List[str]) -> Path:
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
    normalized = ",".join(sorted({coin.strip().lower() for coin in coins if coin.strip()}))
    digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest()[:16]
    return CACHE_DIR / f"coingecko-{digest}.json"


def _load_cached(coins: List[str], ttl_seconds: int) -> Dict[str, Any] | None:
    path = _cache_path(coins)
    if not path.exists():
        return None

    try:
        payload = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError:
        return None

    ts = payload.get("cached_at")
    if not ts:
        return None

    try:
        cached_at = datetime.fromisoformat(ts)
    except ValueError:
        return None

    age = (datetime.now(timezone.utc) - cached_at).total_seconds()
    if age <= ttl_seconds:
        requested = ",".join(sorted({coin.strip().lower() for coin in coins if coin.strip()}))
        log_event(
            service="CoinGecko",
            event_type="cache_hit",
            detail=f"Used cached batch quote for {requested}",
            worked="true",
        )
        return payload.get("prices")

    return None


def _save_cache(coins: List[str], prices: Dict[str, Any]) -> None:
    payload = {
        "cached_at": datetime.now(timezone.utc).isoformat(),
        "prices": prices,
    }
    _cache_path(coins).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")


def _fetch_prices(coins: List[str], timeout_seconds: float) -> Dict[str, Any]:
    requested = [coin.strip().lower() for coin in coins if coin.strip()]
    joined = ",".join(requested)
    encoded_ids = quote(joined, safe=",")
    url = (
        "https://api.coingecko.com/api/v3/simple/price"
        f"?ids={encoded_ids}&vs_currencies=usd&include_24hr_change=true"
    )

    try:
        with urlopen(url, timeout=timeout_seconds) as response:
            body = response.read().decode("utf-8")
    except HTTPError as exc:
        if exc.code == 429:
            log_event(
                service="CoinGecko",
                event_type="429",
                detail=f"HTTP 429 while fetching ids={joined}",
                worked="false",
            )
        elif exc.code in {401, 403}:
            log_event(
                service="CoinGecko",
                event_type="auth_failure",
                detail=f"HTTP {exc.code} while fetching ids={joined}",
                worked="false",
            )
        raise
    except (TimeoutError, socket.timeout) as exc:
        log_event(
            service="CoinGecko",
            event_type="timeout",
            detail=f"Timeout while fetching ids={joined}: {exc}",
            worked="false",
        )
        raise
    except URLError as exc:
        if isinstance(exc.reason, socket.timeout):
            log_event(
                service="CoinGecko",
                event_type="timeout",
                detail=f"URLError timeout while fetching ids={joined}: {exc.reason}",
                worked="false",
            )
        raise

    payload = json.loads(body)
    log_event(
        service="CoinGecko",
        event_type="success",
        detail=f"Fetched live batch quote for ids={joined}",
        worked="true",
    )
    return payload


def fetch_crypto(coins: List[str], timeout_seconds: float, cache_ttl_seconds: int) -> Dict[str, Any]:
    cached = _load_cached(coins, cache_ttl_seconds)
    if cached is not None:
        return cached

    prices = _fetch_prices(coins, timeout_seconds)
    _save_cache(coins, prices)
    return prices


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Fetch CoinGecko prices in a single batch")
    parser.add_argument("coins", nargs="+", help="CoinGecko IDs (e.g., bitcoin ethereum solana)")
    parser.add_argument("--timeout", type=float, default=10.0, help="HTTP timeout in seconds")
    parser.add_argument("--cache-ttl", type=int, default=300, help="Cache TTL in seconds")
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    requested = [coin.strip().lower() for coin in args.coins if coin.strip()]
    prices = fetch_crypto(requested, timeout_seconds=args.timeout, cache_ttl_seconds=args.cache_ttl)
    as_of = datetime.now(timezone.utc).isoformat()

    output: List[Dict[str, Any]] = []
    for coin in requested:
        quote = prices.get(coin, {})
        output.append(
            {
                "symbol": coin,
                "price": quote.get("usd"),
                "change24h": quote.get("usd_24h_change"),
                "asOf": as_of,
            }
        )

    print(json.dumps(output, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/check-profile.py
content: #!/usr/bin/env python3
"""
check-profile.py — Pre-flight lookup for Agent Tollbooth profiles.

Usage:
    python3 scripts/check-profile.py <service-name-or-domain>

Returns the safe pattern for a known service, or a "no profile yet" notice
that seeds the log so promote-profile.py can eventually build one.

Exit codes:
    0 — profile found and printed
    1 — no profile found (observation logged, safe to continue)
"""

from __future__ import annotations

import os
import sys
import json
import re
from pathlib import Path
from datetime import datetime, timezone

SCRIPT_DIR = Path(__file__).resolve().parent
MODULE_DIR = SCRIPT_DIR.parent
PROFILES_PATH = MODULE_DIR / "references" / "profiles.md"

_WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", Path.home() / ".openclaw" / "workspace"))
DATA_DIR = _WORKSPACE / "data" / "agent-tollbooth"
LOG_PATH = DATA_DIR / "web-access-log.json"

# Domain → canonical service name aliases
ALIASES: dict[str, str] = {
    "openai.com": "OpenAI API",
    "api.openai.com": "OpenAI API",
    "anthropic.com": "Anthropic API",
    "api.anthropic.com": "Anthropic API",
    "github.com": "GitHub API",
    "api.github.com": "GitHub API",
    "search.brave.com": "Brave Search API",
    "api.search.brave.com": "Brave Search API",
    "google.serper.dev": "Serper (Google Search API)",
    "serper.dev": "Serper (Google Search API)",
    "notion.com": "Notion API",
    "api.notion.com": "Notion API",
    "airtable.com": "Airtable API",
    "api.airtable.com": "Airtable API",
    "stripe.com": "Stripe API",
    "api.stripe.com": "Stripe API",
    "huggingface.co": "HuggingFace Inference API",
    "api-inference.huggingface.co": "HuggingFace Inference API",
    "firecrawl.dev": "Firecrawl",
    "api.firecrawl.dev": "Firecrawl",
    "finance.yahoo.com": "Yahoo Finance",
    "query2.finance.yahoo.com": "Yahoo Finance",
    "coingecko.com": "CoinGecko",
    "api.coingecko.com": "CoinGecko",
    "ghost.io": "Ghost Admin API",
    "clawhub.ai": "ClawHub API",
    "api.telegram.org": "Telegram Bot API",
    "replicate.com": "Replicate (FLUX image gen)",
    "api.replicate.com": "Replicate (FLUX image gen)",
}


def _parse_profiles(path: Path) -> dict[str, list[str]]:
    """Parse profiles.md into {service_name: [bullet lines]}."""
    if not path.exists():
        return {}

    profiles: dict[str, list[str]] = {}
    current: str | None = None

    for line in path.read_text(encoding="utf-8").splitlines():
        if line.startswith("## "):
            current = line[3:].strip()
            profiles[current] = []
        elif current is not None and line.startswith("- "):
            profiles[current].append(line)

    return profiles


def _resolve_service(query: str, profiles: dict[str, list[str]]) -> str | None:
    """
    Match query against known profile names and aliases.
    Returns the canonical profile name or None.
    """
    q = query.strip().lower()

    # Exact alias match (domain or shorthand)
    for alias, canonical in ALIASES.items():
        if q == alias.lower():
            return canonical

    # Exact profile name match
    for name in profiles:
        if q == name.lower():
            return name

    # Fuzzy: query is a substring of a profile name or vice versa
    for name in profiles:
        if q in name.lower() or name.lower() in q:
            return name

    # Fuzzy: alias value contains query
    for alias, canonical in ALIASES.items():
        if q in alias.lower():
            return canonical

    return None


def _log_miss(service_raw: str) -> None:
    """Log a cache miss so promote-profile.py can track unknown services."""
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    if not LOG_PATH.exists():
        LOG_PATH.write_text("[]", encoding="utf-8")

    try:
        events = json.loads(LOG_PATH.read_text(encoding="utf-8") or "[]")
    except json.JSONDecodeError:
        events = []

    events.append({
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "service": service_raw,
        "event_type": "profile_miss",
        "detail": f"No profile found for '{service_raw}' — observation started",
        "worked": None,
    })
    LOG_PATH.write_text(json.dumps(events, indent=2) + "\n", encoding="utf-8")


def main() -> int:
    if len(sys.argv) < 2:
        print("Usage: python3 check-profile.py <service-name-or-domain>", file=sys.stderr)
        return 2

    query = " ".join(sys.argv[1:])
    profiles = _parse_profiles(PROFILES_PATH)
    match = _resolve_service(query, profiles)

    if match and profiles.get(match):
        print(f"✓ Profile found: {match}")
        print()
        for line in profiles[match]:
            print(line)
        return 0
    else:
        print(f"✗ No profile for '{query}'")
        print("  Logging observation. Use promote-profile.py after a few interactions to build one.")
        _log_miss(query)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/promote-profile.py
content: #!/usr/bin/env python3
"""
promote-profile.py — Mine the event log and draft new profile entries.

Reads web-access-log.json, finds services with enough observations to
characterize, and prints a draft profile in profiles.md format.

Usage:
    python3 scripts/promote-profile.py
    python3 scripts/promote-profile.py --min-events 3
    python3 scripts/promote-profile.py --service "my-api.com"
    python3 scripts/promote-profile.py --write   # append drafts to profiles.md

The agent or user reviews the output and decides whether to keep it.
Nothing is written to profiles.md without --write flag.
"""

from __future__ import annotations

import os
import sys
import json
import argparse
from pathlib import Path
from datetime import datetime, timezone
from collections import defaultdict

SCRIPT_DIR = Path(__file__).resolve().parent
MODULE_DIR = SCRIPT_DIR.parent

_WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", Path.home() / ".openclaw" / "workspace"))
DATA_DIR = _WORKSPACE / "data" / "agent-tollbooth"

# Read-only reference bundled with the skill — never written to
BUNDLE_PROFILES_PATH = MODULE_DIR / "references" / "profiles.md"
# Write target — lives outside the skill bundle in workspace data dir
PROFILES_PATH = DATA_DIR / "profiles.md"
LOG_PATH = DATA_DIR / "web-access-log.json"

# Minimum events before a service is considered promotable
DEFAULT_MIN_EVENTS = 5

# Event types that signal friction (worth documenting)
FRICTION_EVENTS = {"429", "timeout", "auth_failure"}
# Event types that signal working patterns
SUCCESS_EVENTS = {"success", "cache_hit", "workaround"}


def _load_log() -> list[dict]:
    if not LOG_PATH.exists():
        return []
    try:
        return json.loads(LOG_PATH.read_text(encoding="utf-8") or "[]")
    except json.JSONDecodeError:
        return []


def _load_known_profiles() -> set[str]:
    """Return lowercased set of service names already in either profiles file."""
    known = set()
    for path in (BUNDLE_PROFILES_PATH, PROFILES_PATH):
        if not path.exists():
            continue
        for line in path.read_text(encoding="utf-8").splitlines():
            if line.startswith("## "):
                known.add(line[3:].strip().lower())
    return known


def _analyze_service(service: str, events: list[dict]) -> dict:
    """Summarize observations for a single service."""
    friction: list[dict] = []
    successes: list[dict] = []
    misses: list[dict] = []

    for e in events:
        et = e.get("event_type", "")
        if et in FRICTION_EVENTS:
            friction.append(e)
        elif et in SUCCESS_EVENTS:
            successes.append(e)
        elif et == "profile_miss":
            misses.append(e)

    # Derive safe pattern hints from success events
    worked_hints = [e["worked"] for e in successes if e.get("worked")]

    # Derive failure hints from friction events
    failure_hints = [e["detail"] for e in friction]

    # First and last seen
    all_ts = [e["timestamp"] for e in events if e.get("timestamp")]
    first_seen = min(all_ts)[:10] if all_ts else "unknown"

    return {
        "service": service,
        "total_events": len(events),
        "friction_count": len(friction),
        "success_count": len(successes),
        "miss_count": len(misses),
        "worked_hints": worked_hints,
        "failure_hints": failure_hints,
        "first_seen": first_seen,
    }


def _draft_profile(analysis: dict) -> str:
    """Generate a profile.md-format draft from analysis data."""
    s = analysis["service"]
    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")

    lines = [
        f"## {s}",
        "",
        f"- **Endpoint:** `{s}` (observed — verify exact endpoint)",
        f"- **Auth:** Unknown — check service docs",
    ]

    if analysis["worked_hints"]:
        unique_hints = list(dict.fromkeys(analysis["worked_hints"]))  # dedup, preserve order
        for hint in unique_hints[:3]:
            lines.append(f"- **Observed working:** {hint}")

    if analysis["failure_hints"]:
        unique_failures = list(dict.fromkeys(analysis["failure_hints"]))
        for hint in unique_failures[:3]:
            lines.append(f"- **Observed failure:** {hint}")

    if analysis["friction_count"] > 0:
        lines.append(f"- **Safe pattern:** Sequential requests recommended — {analysis['friction_count']} friction events logged")
    else:
        lines.append(f"- **Safe pattern:** No friction logged — {analysis['success_count']} successful calls observed")

    lines.append(f"- **Cache:** Recommended — add TTL appropriate to data freshness needs")
    lines.append(f"- **First observed:** {analysis['first_seen']}")
    lines.append(f"- **Auto-drafted:** {today} by promote-profile.py from {analysis['total_events']} events — review and refine before trusting")
    lines.append("")

    return "\n".join(lines)


def _append_to_profiles(draft: str) -> None:
    """Append a draft profile to the workspace profiles file (never the bundle)."""
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    content = PROFILES_PATH.read_text(encoding="utf-8") if PROFILES_PATH.exists() else ""
    if not content.endswith("\n"):
        content += "\n"
    content += "\n---\n\n" + draft
    PROFILES_PATH.write_text(content, encoding="utf-8")


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Promote frequently-observed services to profile entries"
    )
    parser.add_argument(
        "--min-events",
        type=int,
        default=DEFAULT_MIN_EVENTS,
        help=f"Minimum events before promoting (default: {DEFAULT_MIN_EVENTS})",
    )
    parser.add_argument(
        "--service",
        default=None,
        help="Analyze a specific service name instead of scanning all",
    )
    parser.add_argument(
        "--write",
        action="store_true",
        help="Append draft profiles to references/profiles.md (default: dry-run, print only)",
    )
    parser.add_argument(
        "--all",
        action="store_true",
        help="Include services already in profiles.md (re-draft existing profiles)",
    )
    args = parser.parse_args()

    events = _load_log()
    if not events:
        print("No events in log yet. Hit some walls first.")
        return 0

    known_profiles = _load_known_profiles()

    # Group events by service
    by_service: dict[str, list[dict]] = defaultdict(list)
    for e in events:
        svc = e.get("service", "unknown")
        by_service[svc].append(e)

    # Filter to target service if specified
    if args.service:
        svc_lower = args.service.lower()
        by_service = {
            k: v for k, v in by_service.items()
            if svc_lower in k.lower()
        }
        if not by_service:
            print(f"No events found for service matching '{args.service}'")
            return 1

    candidates = []
    for service, svc_events in by_service.items():
        if len(svc_events) < args.min_events:
            continue
        if not args.all and service.lower() in known_profiles:
            continue  # Already profiled
        analysis = _analyze_service(service, svc_events)
        candidates.append(analysis)

    if not candidates:
        print(f"No unprofiled services with {args.min_events}+ events found.")
        print(f"Total services in log: {len(by_service)}")
        for svc, evts in sorted(by_service.items(), key=lambda x: -len(x[1])):
            status = "✓ profiled" if svc.lower() in known_profiles else f"{len(evts)} events (need {args.min_events})"
            print(f"  {svc}: {status}")
        return 0

    # Sort by total events descending (most-observed first)
    candidates.sort(key=lambda x: -x["total_events"])

    drafted = []
    for analysis in candidates:
        draft = _draft_profile(analysis)
        drafted.append((analysis["service"], draft))

    if args.write:
        for service, draft in drafted:
            _append_to_profiles(draft)
            print(f"✓ Drafted and appended profile for: {service}")
        print(f"\nWrote {len(drafted)} profile(s) to {PROFILES_PATH} (workspace — bundle is untouched)")
        print("Review them — auto-drafted profiles need human verification before trusting.")
    else:
        print(f"Found {len(drafted)} promotable service(s). Dry run — use --write to append.\n")
        print("=" * 60)
        for service, draft in drafted:
            print(draft)
            print("-" * 60)

    return 0


if __name__ == "__main__":
    raise SystemExit(main())


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)