Skill: ademczuk/skillmetricscraper (version 1.3.0)

--- name: skills-weekly description: "OpenClaw Skills Weekly — tracks trending ClawHub skills, generates GitHubAwesome-style YouTube video scripts with two-track ranking (Movers + Rockets)." emoji: "\U0001F4CA" user-invocable: true metadata: openclaw: version: "1.2.0" requires: bins: ["python3"] env: ["ANTHROPIC_API_KEY"] triggers: - "skills weekly" - "trending skills" - "clawhub trending" - "skill metrics" - "weekly report" - "skill snapshot" - "openclaw skills" - "generate report" - "video script" --- # OpenClaw Skills Weekly Automated pipeline for tracking trending ClawHub skills and generating YouTube-ready video scripts in the GitHubAwesome format. ## What This Skill Does 1. **ClawHub API Discovery** — Fetches all ~13K skills from `GET https://clawhub.ai/api/v1/skills` with cursor pagination. No auth required. 2. **SQLite Time-Series Snapshots** — Records daily metrics (installs, downloads, stars) to build 7-day velocity history. 3. **Two-Track Ranking** — MOVERS (established skills, 30+ days, ranked by install velocity) and ROCKETS (new skills <30 days, with recency bonus). Author diversity cap prevents one author from dominating. 4. **Content Harvesting** — Fetches documentation and author info from ClawHub detail API for top-ranked skills. 5. **YouTube Script Generation** — Generates GitHubAwesome-style video segments via Claude Haiku: hook-first, technical specs, no popularity metrics, dry newscast tone. 6. **Dual Output** — Markdown report (`.md`) + voice-ready video script (`.txt`). ## Commands Parse the user's request and route to the correct mode: | User says | Mode | What happens | |---|---|---| | `weekly report` or `full report` or `generate report` | Full Pipeline | Discovery → snapshot → rank → harvest → scripts → output | | `snapshot` or `daily snapshot` | Snapshot Only | Record ClawHub metrics to DB (no scripts) | | `trending` or `what's trending` | Quick Trending | Show top 10 from existing DB data | | `status` or `db status` | Status | Show DB health and snapshot history | | `video script` or `generate script` | Script Only | Re-generate scripts from last snapshot (no re-fetch) | ## Full Pipeline (Weekly Report) First, install dependencies if not already present: ```bash cd "${SKILL_ROOT}" && pip install -r requirements.txt --quiet 2>/dev/null || pip3 install -r requirements.txt --quiet ``` Then run the full pipeline: ```bash cd "${SKILL_ROOT}" && python3 run_weekly.py --top 10 --episode ${EPISODE_NUM:-1} ``` Replace `${EPISODE_NUM}` with the episode number the user specifies, or default to 1. If the user says `--skip-x` or doesn't want X/Twitter capture, add `--skip-x`: ```bash cd "${SKILL_ROOT}" && python3 run_weekly.py --top 10 --skip-x --episode ${EPISODE_NUM:-1} ``` **What this produces:** - `openclaw_weekly_YYYYMMDD.md` — Data-rich markdown report with metrics, rankings, and scripts - `openclaw_weekly_YYYYMMDD_script.txt` — Voice-ready video script in GitHubAwesome format Present both file paths to the user when done. **Expected output:** ``` ============================================================ OpenClaw Skills Weekly — Full Pipeline (v4) Week of Mar 01, 2026 ============================================================ PHASE 1: Multi-Source Signal Capture (X + Reddit + HN) PHASE 2: ClawHub Data Pipeline [1/5] Discovering ClawHub skills... [2/5] Saving snapshot... [3/5] Ranking by 7-day velocity... [4/5] Harvesting content... [5/5] Generating YouTube scripts... DONE: Report: openclaw_weekly_20260301.md Script: openclaw_weekly_20260301_script.txt ``` ## Snapshot Only (Daily Cron) For daily snapshot accumulation without script generation: ```bash cd "${SKILL_ROOT}" && python3 run_weekly.py --snapshot-only --skip-x ``` Tell the user how many skills were captured and how many snapshot dates exist in the DB. ## Quick Trending Show what's trending from existing DB data without re-fetching: ```bash cd "${SKILL_ROOT}" && python3 main.py --list-db ``` ## Status ```bash cd "${SKILL_ROOT}" && python3 main.py --list-db ``` Shows: DB path, total snapshot rows, distinct dates, top skills by current installs. ## CLI Options Reference | Flag | Default | Description | |---|---|---| | `--top N` | 10 | Number of top movers to include | | `--days N` | 7 | Trailing days for velocity calculation | | `--episode N` | 1 | Episode number for video script cold open | | `--skip-x` | false | Skip X/Twitter signal capture | | `--snapshot-only` | false | Just record snapshot, no scripts | | `--max-pages N` | 0 (all) | Limit API pages (for testing) | | `--model MODEL` | claude-haiku-4-5-20251001 | Anthropic model for script gen | | `--output FILE` | auto-dated | Custom output file path | | `--mock` | false | Use synthetic data (offline dev) | ## Environment Variables | Variable | Required | Description | |---|---|---| | `ANTHROPIC_API_KEY` | Yes | For YouTube script generation via Claude | | `GITHUB_TOKEN` | No | For fetching source READMEs from GitHub | | `XAI_API_KEY` | No | For X/Twitter signal capture via xAI | | `CLAWHUB_BASE_URL` | No | Override ClawHub URL (default: https://clawhub.ai) | ## Video Script Format Scripts follow the GitHubAwesome "GitHub Trending Weekly" format: - **Cold open**: "It is time for OpenClaw Skills Weekly, episode number N..." - **Per-skill segments** (~20 sec each): Hook first → technical specs → sharp closer - **No popularity metrics** in narration (no download/install/star counts) - **Dry, confident newscast tone** — no hype, no superlatives - **No outro** — last item ends the episode ## Architecture ``` run_weekly.py # Full pipeline orchestrator main.py # Alternative CLI with --list-db discovery.py # ClawHub API cursor pagination (~13K skills) storage.py # SQLite time-series (slug-scoped dedup, CTE velocity) ranker.py # Two-track: Movers + Rockets, author diversity cap harvester.py # ClawHub detail API content + author extraction script_generator.py # LLM script gen + markdown + video script rendering community_signals.py # Multi-source signal loading and rendering x_capture.py # xAI x_search API integration reddit_capture.py # Reddit + Hacker News signal capture hourly_heartbeat.py # Top-500 hourly snapshot + project metadata project_tracker.py # OpenClaw GitHub repo metadata tracking data/metrics.db # SQLite database (auto-created) ```

path: community_signals.py
content: """
community_signals.py — Weekly X/Twitter community intel for the report

Loads curated signal data from a JSON file (data/weekly_signals.json).
Each week, you drop in the week's X threads, blog posts, and notable
discussions before running the pipeline. The report renders them as a
"Community Buzz" section.

File format (data/weekly_signals.json):
[
  {
    "date": "2026-02-28",
    "category": "tutorial",
    "title": "How to create a simple custom OpenClaw Skill",
    "url": "https://x.com/i/status/...",
    "summary": "Video walkthrough of building a custom skill from scratch.",
    "tags": ["custom-skills", "beginner"],
    "source": "x"
  },
  ...
]

Categories: tutorial, security, ecosystem, showcase, discussion, market
Sources: x, blog, docs, youtube, reddit, hackernews
"""

import json
from datetime import date, timedelta
from pathlib import Path

SIGNALS_PATH = Path(__file__).parent / "data" / "weekly_signals.json"

CATEGORY_EMOJI = {
    "tutorial": "📚",
    "security": "🔒",
    "ecosystem": "🌐",
    "showcase": "🎨",
    "discussion": "💬",
    "market": "📈",
}

SOURCE_ICON = {
    "x": "𝕏",
    "reddit": "🔴",
    "hackernews": "🟠",
    "blog": "📝",
    "docs": "📄",
    "youtube": "▶️",
}

CATEGORY_LABEL = {
    "tutorial": "Tutorial",
    "security": "Security",
    "ecosystem": "Ecosystem",
    "showcase": "Showcase",
    "discussion": "Discussion",
    "market": "Market Signal",
}


def load_signals() -> list[dict]:
    """Load weekly community signals from JSON file."""
    if not SIGNALS_PATH.exists():
        return []
    try:
        data = json.loads(SIGNALS_PATH.read_text(encoding="utf-8"))
        return data if isinstance(data, list) else []
    except (json.JSONDecodeError, OSError):
        return []


def render_community_section(signals: list[dict], ttl_days: int = 7) -> str:
    """Render the Community Buzz markdown section. Filters to last ttl_days only."""
    if not signals:
        return ""

    # Filter by TTL — only show signals from the last N days
    cutoff = (date.today() - timedelta(days=ttl_days)).isoformat()
    signals = [s for s in signals if s.get("date", "9999") >= cutoff]

    if not signals:
        return ""

    # Count signals by source for subtitle
    sources = {s.get("source", "x") for s in signals}
    source_labels = []
    if "x" in sources:
        source_labels.append("X")
    if "reddit" in sources:
        source_labels.append("Reddit")
    if "hackernews" in sources:
        source_labels.append("Hacker News")
    if "blog" in sources:
        source_labels.append("blogs")
    subtitle = ", ".join(source_labels) or "community discussions"

    lines = [
        "## Community Buzz This Week",
        "",
        f"*Curated from {subtitle}.*",
        "",
    ]

    # Group by category
    by_cat: dict[str, list[dict]] = {}
    for s in signals:
        cat = s.get("category", "discussion")
        by_cat.setdefault(cat, []).append(s)

    for cat in ["showcase", "tutorial", "security", "ecosystem", "discussion", "market"]:
        items = by_cat.get(cat, [])
        if not items:
            continue

        emoji = CATEGORY_EMOJI.get(cat, "")
        label = CATEGORY_LABEL.get(cat, cat.title())
        lines.append(f"### {emoji} {label}")
        lines.append("")

        for item in items:
            title = item.get("title", "Untitled")
            url = item.get("url", "")
            summary = item.get("summary", "")
            item_date = item.get("date", "")
            tags = item.get("tags", [])
            source = item.get("source", "")

            link = f"[{title}]({url})" if url else title
            tag_str = " ".join(f"`#{t}`" for t in tags) if tags else ""
            date_str = f" ({item_date})" if item_date else ""
            source_icon = SOURCE_ICON.get(source, "")
            source_prefix = f"{source_icon} " if source_icon else ""

            lines.append(f"- {source_prefix}**{link}**{date_str}")
            if summary:
                lines.append(f"  {summary}")
            if tag_str:
                lines.append(f"  {tag_str}")
            lines.append("")

    lines.append("---")
    lines.append("")

    return "\n".join(lines)


path: discovery.py
content: """
discovery.py — ClawHub Ingestion Engine (v3 — Real API)

Uses the confirmed ClawHub REST API:
    GET https://clawhub.ai/api/v1/skills?sort=...&limit=100&cursor=...

No authentication required for reads. Cursor-based pagination.
Rate limit: undocumented — we use 0.75s delay between pages.

Configure via .env:
    CLAWHUB_BASE_URL=https://clawhub.ai   (override for testing)
"""

import os
import time
import httpx
from datetime import datetime, timezone


DEFAULT_BASE_URL = "https://clawhub.ai"
API_PATH = "/api/v1/skills"
REQUEST_TIMEOUT = 30.0
PAGE_DELAY = 0.75  # seconds between pages
PAGE_LIMIT = 100
MAX_RETRIES = 3
RETRY_BACKOFF = 2.0  # seconds, doubles on each retry

HEADERS = {
    "User-Agent": "OpenclawSkillsWeekly/1.0",
    "Accept": "application/json",
}


def _normalise(item: dict) -> dict:
    """Flatten a ClawHub API skill item into our storage-compatible format."""
    stats = item.get("stats") or {}
    lv = item.get("latestVersion") or {}
    meta = item.get("metadata") or {}

    slug = item.get("slug", "")
    # Author is the first part of slug (e.g. "username/skill-name")
    author = slug.split("/")[0] if "/" in slug else ""

    return {
        "slug": slug,
        "display_name": item.get("displayName") or slug,
        "summary": item.get("summary") or "",
        "author": author,
        "downloads": stats.get("downloads", 0) or 0,
        "stars": stats.get("stars", 0) or 0,
        "installs_current": stats.get("installsCurrent", 0) or 0,
        "installs_all_time": stats.get("installsAllTime", 0) or 0,
        "versions": stats.get("versions", 0) or 0,
        "comments": stats.get("comments", 0) or 0,
        "created_at": item.get("createdAt", 0) or 0,
        "updated_at": item.get("updatedAt", 0) or 0,
        "latest_version": lv.get("version", ""),
        "os_support": ",".join(meta.get("os") or []),
        "systems": ",".join(meta.get("systems") or []),
        "clawhub_url": f"{DEFAULT_BASE_URL}/skills/{slug}" if slug else "",
    }


def _fetch_page(client: httpx.Client, url: str, params: dict) -> tuple[list[dict], str | None]:
    """Fetch one page with retry logic. Returns (items, nextCursor)."""
    for attempt in range(MAX_RETRIES):
        try:
            resp = client.get(url, params=params)

            if resp.status_code == 429:
                wait = RETRY_BACKOFF * (2 ** attempt)
                print(f"  [RATE LIMIT] 429 — backing off {wait:.0f}s (attempt {attempt + 1}/{MAX_RETRIES})")
                time.sleep(wait)
                continue

            resp.raise_for_status()
            data = resp.json()
            items = data.get("items", [])
            cursor = data.get("nextCursor")
            return items, cursor

        except httpx.TimeoutException:
            wait = RETRY_BACKOFF * (2 ** attempt)
            print(f"  [TIMEOUT] Retrying in {wait:.0f}s (attempt {attempt + 1}/{MAX_RETRIES})")
            time.sleep(wait)
        except httpx.HTTPStatusError as e:
            print(f"  [HTTP ERROR] {e.response.status_code}: {e}")
            return [], None
        except Exception as e:
            print(f"  [ERROR] {e}")
            return [], None

    print(f"  [FAILED] Max retries exceeded")
    return [], None


def discover(
    clawhub_url: str = "",
    sort: str = "downloads",
    mock: bool = False,
    max_pages: int = 0,
) -> list[dict]:
    """
    Fetch all skills from ClawHub via the public REST API.

    Args:
        clawhub_url: Base URL override (default: https://clawhub.ai)
        sort: Sort order — 'downloads', 'trending', 'installsCurrent', 'updated', 'stars'
        mock: If True, return synthetic data for offline dev
        max_pages: Limit pages fetched (0 = unlimited). Useful for testing.

    Returns:
        List of normalised skill dicts ready for storage.upsert_skills()
    """
    if mock:
        return _mock_skills()

    base = clawhub_url or os.getenv("CLAWHUB_BASE_URL", DEFAULT_BASE_URL)
    url = f"{base.rstrip('/')}{API_PATH}"

    print(f"[DISCOVERY] ClawHub API: {url}")
    print(f"[DISCOVERY] Sort: {sort}, page limit: {max_pages or 'unlimited'}")

    all_skills = []
    cursor = None
    page = 0

    with httpx.Client(headers=HEADERS, timeout=REQUEST_TIMEOUT, follow_redirects=True) as client:
        while True:
            params = {"sort": sort, "limit": PAGE_LIMIT}
            if cursor:
                params["cursor"] = cursor

            items, next_cursor = _fetch_page(client, url, params)

            if not items and page == 0:
                print("[DISCOVERY] No items returned on first page — API may be down")
                break

            for item in items:
                all_skills.append(_normalise(item))

            page += 1
            print(f"  Page {page}: {len(items)} skills (total: {len(all_skills)})")

            cursor = next_cursor
            if not cursor or not items:
                break
            if max_pages and page >= max_pages:
                print(f"  [DISCOVERY] Reached page limit ({max_pages})")
                break

            time.sleep(PAGE_DELAY)

    print(f"[DISCOVERY] Done: {len(all_skills)} skills fetched in {page} pages")
    return all_skills


def discover_incremental(
    clawhub_url: str = "",
    since_ts_ms: int = 0,
) -> list[dict]:
    """
    Fetch only skills updated since a given timestamp (milliseconds).
    Sorts by 'updated' and stops when we hit skills older than cutoff.
    """
    base = clawhub_url or os.getenv("CLAWHUB_BASE_URL", DEFAULT_BASE_URL)
    url = f"{base.rstrip('/')}{API_PATH}"

    print(f"[DISCOVERY] Incremental fetch since {since_ts_ms}")

    all_skills = []
    cursor = None
    page = 0

    with httpx.Client(headers=HEADERS, timeout=REQUEST_TIMEOUT, follow_redirects=True) as client:
        while True:
            params = {"sort": "updated", "limit": PAGE_LIMIT}
            if cursor:
                params["cursor"] = cursor

            items, next_cursor = _fetch_page(client, url, params)
            if not items:
                break

            hit_cutoff = False
            for item in items:
                if (item.get("updatedAt", 0) or 0) < since_ts_ms:
                    hit_cutoff = True
                    break
                all_skills.append(_normalise(item))

            page += 1
            cursor = next_cursor

            if hit_cutoff or not cursor:
                break

            time.sleep(PAGE_DELAY)

    print(f"[DISCOVERY] Incremental: {len(all_skills)} updated skills in {page} pages")
    return all_skills


# ---------------------------------------------------------------------------
# Mock mode (for offline development only)
# ---------------------------------------------------------------------------

def _mock_skills() -> list[dict]:
    """Synthetic stub data for development when ClawHub is unreachable."""
    print("  [MOCK] Using synthetic stub data for offline development")
    now_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
    stubs = [
        ("openclaw-team/browser-control", "Browser Control Skill", "Automate any browser interaction via natural language", 8420, 142, 210, 8420, 4.8),
        ("alexsjones/code-reviewer", "Code Reviewer", "AI-powered code review with inline suggestions", 6130, 98, 370, 6130, 4.6),
        ("openclaw-team/slack-notifier", "Slack Notifier", "Push structured notifications to Slack channels", 5890, 76, 180, 5890, 4.5),
        ("aidanpark/data-pipeline", "Data Pipeline Builder", "Construct ETL pipelines via agent conversation", 4200, 55, 120, 4200, 4.3),
        ("openpawz/web-scraper", "Web Scraper", "Extract structured data from any public webpage", 3980, 110, 290, 3980, 4.7),
        ("openclaw-team/image-analyser", "Image Analyser", "Vision-powered image description and tagging", 3100, 45, 95, 3100, 4.4),
        ("alexsjones/github-pr-agent", "GitHub PR Agent", "Automate PR reviews, labels, and merges", 2750, 88, 150, 2750, 4.2),
        ("contrib-user1/pdf-extractor", "PDF Extractor", "Parse and structure PDF documents into JSON", 2200, 32, 60, 2200, 4.1),
        ("contrib-user2/email-drafter", "Email Drafter", "Draft context-aware professional emails", 1800, 28, 40, 1800, 4.0),
        ("aidanpark/sql-query-agent", "SQL Query Agent", "Natural language to SQL with safety guardrails", 950, 65, 85, 950, 4.6),
    ]
    return [
        {
            "slug": slug,
            "display_name": name,
            "summary": desc,
            "author": slug.split("/")[0],
            "downloads": downloads,
            "stars": stars,
            "installs_current": inst_cur,
            "installs_all_time": inst_all,
            "versions": 3,
            "comments": 5,
            "created_at": now_ms - 86400000 * 30,
            "updated_at": now_ms,
            "latest_version": "1.0.0",
            "os_support": "darwin,linux,win32",
            "systems": "claude",
            "clawhub_url": f"https://clawhub.ai/skills/{slug}",
        }
        for slug, name, desc, downloads, stars, inst_cur, inst_all, _ in stubs
    ]


path: harvester.py
content: """
harvester.py — Content Harvester (v3)

Fetches skill documentation from ClawHub detail page or linked source repo.
Priority order:
  1. ClawHub skill detail page (GET /api/v1/skills/{slug})
  2. Linked GitHub/source repo README
  3. Falls back to summary-only if neither is available
"""

import time
import httpx

CLAWHUB_BASE = "https://clawhub.ai"
GITHUB_RAW = "https://raw.githubusercontent.com"
README_CANDIDATES = ["README.md", "readme.md", "SKILL.md", "README.rst", "README"]
MAX_CONTENT_CHARS = 6000
REQUEST_TIMEOUT = 15


def _fetch_clawhub_detail(slug: str) -> dict:
    """Fetch skill detail via ClawHub API. Returns {content, author} dict."""
    if not slug:
        return {"content": "", "author": ""}
    url = f"{CLAWHUB_BASE}/api/v1/skills/{slug}"
    try:
        resp = httpx.get(url, timeout=REQUEST_TIMEOUT, follow_redirects=True,
                         headers={"Accept": "application/json"})
        if resp.status_code != 200:
            return {"content": "", "author": ""}
        data = resp.json()

        # Extract author from owner field (only available in detail endpoint)
        owner = data.get("owner") or {}
        author = owner.get("handle", "") or owner.get("displayName", "")

        # Build content from skill summary + changelog
        parts = []
        skill_data = data.get("skill") or {}
        summary = skill_data.get("summary") or data.get("summary") or ""
        if summary:
            parts.append(summary)
        if data.get("description"):
            parts.append(data["description"])
        lv = data.get("latestVersion") or {}
        if lv.get("changelog"):
            parts.append(f"Latest changelog: {lv['changelog']}")
        content = "\n\n".join(parts)[:MAX_CONTENT_CHARS]

        return {"content": content, "author": author}
    except Exception:
        return {"content": "", "author": ""}


def _fetch_github_readme(source_url: str, github_token: str) -> str:
    """Fetch README from a GitHub source repo URL."""
    if not source_url or "github.com" not in source_url:
        return ""

    parts = source_url.rstrip("/").replace("https://github.com/", "").split("/")
    if len(parts) < 2:
        return ""
    owner, repo = parts[0], parts[1]

    headers = {"User-Agent": "OpenclawSkillsWeekly/1.0"}
    if github_token:
        headers["Authorization"] = f"Bearer {github_token}"

    # Get default branch
    try:
        api_resp = httpx.get(
            f"https://api.github.com/repos/{owner}/{repo}",
            headers={**headers, "Accept": "application/vnd.github+json"},
            timeout=REQUEST_TIMEOUT,
        )
        branch = api_resp.json().get("default_branch", "main") if api_resp.status_code == 200 else "main"
    except Exception:
        branch = "main"

    for filename in README_CANDIDATES:
        url = f"{GITHUB_RAW}/{owner}/{repo}/{branch}/{filename}"
        try:
            resp = httpx.get(url, headers=headers, timeout=REQUEST_TIMEOUT)
            if resp.status_code == 200:
                return resp.text[:MAX_CONTENT_CHARS]
        except Exception:
            continue
    return ""


def harvest(skills: list[dict], github_token: str = "") -> list[dict]:
    """
    Fetch documentation for each skill.
    Returns list of enriched skill dicts with added 'content' field.
    """
    results = []

    for i, skill in enumerate(skills, 1):
        name = skill.get("display_name", skill.get("slug", "?"))
        slug = skill.get("slug", "")
        print(f"[HARVEST] {i}/{len(skills)} {name}")

        # 1. Try ClawHub API detail (also extracts author)
        detail = _fetch_clawhub_detail(slug)
        content = detail["content"]
        author = detail["author"]

        if content:
            print(f"  Source: ClawHub API ({len(content)} chars)")
        else:
            # 2. Try GitHub source repo
            content = _fetch_github_readme(skill.get("source_url", ""), github_token)
            if content:
                print(f"  Source: GitHub README ({len(content)} chars)")
            else:
                # 3. Fall back to summary
                content = skill.get("summary", "")
                print(f"  Source: summary only ({len(content)} chars)")

        enriched = {**skill, "content": content}
        # Merge author from detail endpoint if discovery didn't provide one
        if author and not skill.get("author"):
            enriched["author"] = author
            print(f"  Author: {author}")

        results.append(enriched)

        if i < len(skills):
            time.sleep(0.3)

    return results


path: hourly_heartbeat.py
content: """
hourly_heartbeat.py — Top-500 Skill Snapshot + Project Metadata

Runs every hour via Windows Task Scheduler.
- Snapshots the top 500 ClawHub skills (5 pages, ~4 seconds)
- Tracks OpenClaw ecosystem GitHub repos (stars, PRs, releases)
- Append-only inserts — no dedup destruction of earlier hourly rows

Usage:
    python hourly_heartbeat.py              # Normal run
    python hourly_heartbeat.py --pages 10   # More skills
    python hourly_heartbeat.py --skip-meta  # Skip GitHub metadata
"""

import argparse
import os
import sys
from datetime import datetime, timezone
from pathlib import Path

from dotenv import load_dotenv

load_dotenv()

# Ensure project root is in path
sys.path.insert(0, str(Path(__file__).parent))

import discovery
import storage
import project_tracker


def main():
    parser = argparse.ArgumentParser(description="Hourly heartbeat snapshot")
    parser.add_argument("--pages", type=int, default=5, help="API pages to fetch (100 skills/page)")
    parser.add_argument("--skip-meta", action="store_true", help="Skip GitHub project metadata")
    args = parser.parse_args()

    now = datetime.now(timezone.utc)
    print(f"[HEARTBEAT] {now.strftime('%Y-%m-%d %H:%M')} UTC — top-{args.pages * 100} snapshot")

    # Phase 1: ClawHub skill snapshot
    storage.init_db()

    skills = discovery.discover(
        sort="downloads",
        max_pages=args.pages,
    )

    if not skills:
        print("[HEARTBEAT] No skills returned — aborting")
        return

    storage.upsert_skills(skills)
    storage.append_snapshot(skills)

    # Phase 2: OpenClaw ecosystem metadata
    if not args.skip_meta:
        print("[HEARTBEAT] Tracking OpenClaw ecosystem repos...")
        try:
            project_tracker.capture()
        except Exception as e:
            print(f"[HEARTBEAT] Project tracker failed (non-fatal): {e}")

    print(f"[HEARTBEAT] Complete — {len(skills)} skills, {now.strftime('%H:%M')} UTC")


if __name__ == "__main__":
    main()


path: main.py
content: """
main.py — SkillMetricScraper v3 CLI (ClawHub Real API Edition)

Usage:
    python main.py [--top N] [--days N] [--output FILE] [--week LABEL]
    python main.py --snapshot-only       # Record snapshot, skip script generation
    python main.py --list-db             # Show current DB state
    python main.py --mock                # Use synthetic data (offline dev)
    python main.py --max-pages 5         # Limit API pages (testing)
    python main.py --sort trending       # Sort: downloads, trending, installsCurrent, updated, stars

Environment (.env):
    CLAWHUB_BASE_URL    ClawHub registry URL (default: https://clawhub.ai)
    GITHUB_TOKEN        For fetching source READMEs from GitHub
    ANTHROPIC_API_KEY   For script generation
"""

import argparse
import os
import sys
from datetime import datetime
from pathlib import Path

from dotenv import load_dotenv

import discovery
import storage
import ranker
import harvester
import script_generator


def parse_args():
    parser = argparse.ArgumentParser(
        description="Discover, rank, and generate YouTube scripts for trending ClawHub skills."
    )
    parser.add_argument("--top", type=int, default=10, help="Top N skills to script (default: 10)")
    parser.add_argument("--days", type=int, default=7, help="Trailing days for velocity (default: 7)")
    parser.add_argument("--output", type=str, default=None, help="Output markdown file path")
    parser.add_argument("--week", type=str, default=None, help="Week label for report header")
    parser.add_argument("--model", type=str, default="claude-haiku-4-5-20251001", help="Anthropic model")
    parser.add_argument("--sort", type=str, default="downloads", help="ClawHub sort: downloads, trending, installsCurrent, updated, stars")
    parser.add_argument("--max-pages", type=int, default=0, help="Limit API pages (0 = unlimited)")
    parser.add_argument("--snapshot-only", action="store_true", help="Record snapshot only, skip LLM")
    parser.add_argument("--list-db", action="store_true", help="Print DB snapshot count and exit")
    parser.add_argument("--mock", action="store_true", help="Use synthetic data (offline dev)")
    parser.add_argument("--episode", type=int, default=1, help="Episode number for video script")
    return parser.parse_args()


def load_env() -> dict:
    load_dotenv()
    return {
        "github_token": os.getenv("GITHUB_TOKEN", ""),
        "anthropic_key": os.getenv("ANTHROPIC_API_KEY", ""),
        "clawhub_url": os.getenv("CLAWHUB_BASE_URL", ""),
    }


def main():
    args = parse_args()
    env = load_env()

    # --list-db
    if args.list_db:
        storage.init_db()
        count = storage.snapshot_count()
        latest = storage.get_latest_snapshot()
        dates = storage.distinct_snapshot_dates()
        print(f"DB: {storage.DB_PATH}")
        print(f"Total snapshot rows: {count}")
        print(f"Distinct snapshot dates: {len(dates)}")
        for d in dates[-5:]:
            print(f"  {d}")
        print(f"Skills with data: {len(latest)}")
        for row in latest[:15]:
            print(
                f"  {row['display_name']:35s} "
                f"installs={row['installs_current']:>6,}  "
                f"downloads={row['downloads']:>8,}  "
                f"stars={row['stars']:>4}"
            )
        return

    week_label = args.week or f"Week of {datetime.now().strftime('%b %d, %Y')}"
    output_path = args.output or f"openclaw_weekly_{datetime.now().strftime('%Y%m%d')}.md"

    print("=" * 60)
    print("  SkillMetricScraper v3 — ClawHub Real API Edition")
    print(f"  {week_label}")
    print("=" * 60)

    # Step 1: Initialise DB
    storage.init_db()

    # Step 2: Discover
    print(f"\n[1/5] Discovering ClawHub skills (sort={args.sort})...")
    skills = discovery.discover(
        clawhub_url=env["clawhub_url"],
        sort=args.sort,
        mock=args.mock,
        max_pages=args.max_pages,
    )
    if not skills:
        print("[ERROR] No skills discovered. Check network or use --mock for offline dev.")
        sys.exit(1)

    # Step 3: Upsert skills + record snapshot
    print(f"\n[2/5] Saving snapshot of {len(skills)} skills to DB...")
    storage.upsert_skills(skills)
    storage.record_snapshot(skills)

    if args.snapshot_only:
        dates = storage.distinct_snapshot_dates()
        print(f"\n[DONE] Snapshot recorded ({len(skills)} skills).")
        print(f"       Total snapshots in DB: {storage.snapshot_count()} rows across {len(dates)} date(s).")
        print(f"       Run without --snapshot-only to generate scripts.")
        print(f"       Tip: run daily with --snapshot-only to build velocity history.")
        return

    # Step 4: Rank by velocity (two-track)
    print(f"\n[3/5] Ranking by {args.days}-day velocity (movers + rockets)...")
    top_movers, top_rockets = ranker.rank(skills, db_days=args.days, top_n=args.top)

    all_top = top_movers + top_rockets

    # Step 5: Harvest content
    print(f"\n[4/5] Harvesting content for {len(all_top)} skills...")
    harvested = harvester.harvest(all_top, github_token=env["github_token"])

    movers_h = harvested[:len(top_movers)]
    rockets_h = harvested[len(top_movers):]

    # Step 6: Generate scripts
    if not env["anthropic_key"]:
        print("[WARN] No ANTHROPIC_API_KEY — skipping script generation")
        movers_s = [{**s, "script": "[No API key]"} for s in movers_h]
        rockets_s = [{**s, "script": "[No API key]"} for s in rockets_h]
    else:
        print(f"\n[5/5] Generating YouTube scripts via {args.model}...")
        all_scripted = script_generator.generate_scripts(harvested, env["anthropic_key"], model=args.model)
        movers_s = all_scripted[:len(top_movers)]
        rockets_s = all_scripted[len(top_movers):]

    # Render output: markdown report
    markdown = script_generator.render_markdown(movers_s, rockets_s, week_label=week_label)
    Path(output_path).write_text(markdown, encoding="utf-8")

    # Render output: voice-ready video script
    script_path = output_path.replace(".md", "_script.txt")
    video_script = script_generator.render_video_script(
        movers_s, rockets_s,
        episode_num=args.episode,
        week_label=week_label,
    )
    Path(script_path).write_text(video_script, encoding="utf-8")

    snap_count = storage.snapshot_count()
    dates = storage.distinct_snapshot_dates()
    total = len(movers_s) + len(rockets_s)
    print(f"\n[DONE] Report: {output_path}")
    print(f"       Script: {script_path}")
    print(f"       {len(movers_s)} movers + {len(rockets_s)} rockets | DB: {snap_count} rows across {len(dates)} date(s)")


if __name__ == "__main__":
    main()


path: openclaw-skill/last30days-weekly/scripts/clawhub_snapshot.py
content: #!/usr/bin/env python3
"""
clawhub_snapshot.py — ClawHub Skills Tracker (stdlib-only, runs inside OpenClaw container)

Fetches all skills from the ClawHub REST API, stores daily snapshots in SQLite,
calculates 7-day install velocity for trending analysis.

Usage:
    python3 clawhub_snapshot.py snapshot                # Daily: record snapshot
    python3 clawhub_snapshot.py report [--top N]        # Weekly: snapshot + rank + markdown
    python3 clawhub_snapshot.py trending [--top N]      # Quick: show top N from DB
    python3 clawhub_snapshot.py status                  # Show DB health
    python3 clawhub_snapshot.py community [--save]      # Placeholder for community signals

No pip dependencies. Uses urllib, json, sqlite3 (all Python stdlib).
"""

import argparse
import json
import os
import sqlite3
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path

# --- Config ---
CLAWHUB_API = "https://clawhub.ai/api/v1/skills"
PAGE_LIMIT = 100
PAGE_DELAY = 0.75
MAX_RETRIES = 3
RETRY_BACKOFF = 2.0

DATA_DIR = Path(os.environ.get(
    "SKILLS_WEEKLY_DATA",
    # Use workspace volume (writable) inside Docker, fallback for host
    "/home/node/.openclaw/workspace/data/skills-weekly"
    if Path("/home/node/.openclaw/workspace").exists()
    else os.path.expanduser("~/.local/share/skills-weekly")
))
DB_PATH = DATA_DIR / "metrics.db"
SIGNALS_PATH = DATA_DIR / "weekly_signals.json"

# Also write reports to host mount if available (set HOST_OUTPUT_DIR env var)
HOST_OUTPUT = Path(os.environ.get("HOST_OUTPUT_DIR", "/mnt/host/skills-weekly"))


# --- Database ---

def init_db():
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    conn = sqlite3.connect(str(DB_PATH))
    conn.row_factory = sqlite3.Row
    conn.execute("PRAGMA journal_mode=WAL")
    conn.executescript("""
        CREATE TABLE IF NOT EXISTS skills (
            slug TEXT PRIMARY KEY,
            display_name TEXT NOT NULL,
            summary TEXT DEFAULT '',
            author TEXT DEFAULT '',
            clawhub_url TEXT DEFAULT '',
            created_at INTEGER DEFAULT 0,
            updated_at INTEGER DEFAULT 0,
            latest_version TEXT DEFAULT '',
            os_support TEXT DEFAULT '',
            systems TEXT DEFAULT ''
        );
        CREATE TABLE IF NOT EXISTS metrics_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            slug TEXT NOT NULL REFERENCES skills(slug),
            timestamp TEXT NOT NULL,
            downloads INTEGER NOT NULL DEFAULT 0,
            stars INTEGER NOT NULL DEFAULT 0,
            installs_current INTEGER NOT NULL DEFAULT 0,
            installs_all_time INTEGER NOT NULL DEFAULT 0,
            versions INTEGER NOT NULL DEFAULT 0,
            comments INTEGER NOT NULL DEFAULT 0
        );
        CREATE INDEX IF NOT EXISTS idx_mh_slug_ts
            ON metrics_history (slug, timestamp);
    """)
    conn.close()
    return DB_PATH


def get_conn():
    conn = sqlite3.connect(str(DB_PATH))
    conn.row_factory = sqlite3.Row
    return conn


# --- ClawHub API ---

def fetch_page(url, params):
    """Fetch one page from ClawHub API with retry logic."""
    qs = "&".join(f"{k}={v}" for k, v in params.items() if v)
    full_url = f"{url}?{qs}" if qs else url

    for attempt in range(MAX_RETRIES):
        try:
            req = urllib.request.Request(full_url, headers={
                "User-Agent": "OpenclawSkillsWeekly/1.0",
                "Accept": "application/json",
            })
            with urllib.request.urlopen(req, timeout=30) as resp:
                data = json.loads(resp.read().decode())
                return data.get("items", []), data.get("nextCursor")
        except urllib.error.HTTPError as e:
            if e.code == 429:
                wait = RETRY_BACKOFF * (2 ** attempt)
                print(f"  [429] Rate limited, backing off {wait:.0f}s")
                time.sleep(wait)
                continue
            print(f"  [HTTP {e.code}] {e.reason}")
            return [], None
        except Exception as e:
            wait = RETRY_BACKOFF * (2 ** attempt)
            print(f"  [ERROR] {e}, retrying in {wait:.0f}s")
            time.sleep(wait)
    return [], None


def normalise(item):
    """Flatten ClawHub API item into DB-compatible dict."""
    stats = item.get("stats") or {}
    lv = item.get("latestVersion") or {}
    meta = item.get("metadata") or {}
    slug = item.get("slug", "")
    author = slug.split("/")[0] if "/" in slug else ""
    return {
        "slug": slug,
        "display_name": item.get("displayName") or slug,
        "summary": item.get("summary") or "",
        "author": author,
        "downloads": stats.get("downloads", 0) or 0,
        "stars": stats.get("stars", 0) or 0,
        "installs_current": stats.get("installsCurrent", 0) or 0,
        "installs_all_time": stats.get("installsAllTime", 0) or 0,
        "versions": stats.get("versions", 0) or 0,
        "comments": stats.get("comments", 0) or 0,
        "created_at": item.get("createdAt", 0) or 0,
        "updated_at": item.get("updatedAt", 0) or 0,
        "latest_version": lv.get("version", ""),
        "os_support": ",".join(meta.get("os") or []),
        "systems": ",".join(meta.get("systems") or []),
        "clawhub_url": f"https://clawhub.ai/skills/{slug}" if slug else "",
    }


def discover_all(sort="downloads", max_pages=0):
    """Paginate through all ClawHub skills."""
    print(f"[SNAPSHOT] Fetching from {CLAWHUB_API} (sort={sort})...")
    all_skills = []
    cursor = None
    page = 0

    while True:
        params = {"sort": sort, "limit": str(PAGE_LIMIT)}
        if cursor:
            params["cursor"] = cursor

        items, next_cursor = fetch_page(CLAWHUB_API, params)
        if not items and page == 0:
            print("[SNAPSHOT] No items on first page — API may be down")
            break

        for item in items:
            all_skills.append(normalise(item))

        page += 1
        print(f"  Page {page}: {len(items)} skills (total: {len(all_skills)})")

        cursor = next_cursor
        if not cursor or not items:
            break
        if max_pages and page >= max_pages:
            break
        time.sleep(PAGE_DELAY)

    print(f"[SNAPSHOT] Done: {len(all_skills)} skills fetched in {page} pages")
    return all_skills


def record_snapshot(skills):
    """Upsert skills and record metrics snapshot."""
    now = datetime.now(timezone.utc).isoformat()
    conn = get_conn()
    try:
        for s in skills:
            conn.execute("""
                INSERT INTO skills (slug, display_name, summary, author, clawhub_url,
                                    created_at, updated_at, latest_version, os_support, systems)
                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ON CONFLICT(slug) DO UPDATE SET
                    display_name=excluded.display_name, summary=excluded.summary,
                    author=excluded.author, clawhub_url=excluded.clawhub_url,
                    updated_at=excluded.updated_at, latest_version=excluded.latest_version,
                    os_support=excluded.os_support, systems=excluded.systems
            """, (s["slug"], s["display_name"], s["summary"], s["author"],
                  s["clawhub_url"], s["created_at"], s["updated_at"],
                  s["latest_version"], s["os_support"], s["systems"]))

        conn.executemany("""
            INSERT INTO metrics_history (slug, timestamp, downloads, stars,
                                         installs_current, installs_all_time, versions, comments)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
        """, [(s["slug"], now, s["downloads"], s["stars"], s["installs_current"],
               s["installs_all_time"], s["versions"], s["comments"]) for s in skills])
        conn.commit()
    finally:
        conn.close()

    print(f"[SNAPSHOT] Recorded {len(skills)} skills at {now[:19]}")


# --- Velocity / Trending ---

def get_velocity(slug, days=7):
    """Calculate install velocity for one skill."""
    cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
    conn = get_conn()
    rows = conn.execute("""
        SELECT installs_current, downloads, stars, timestamp
        FROM metrics_history WHERE slug = ? AND timestamp >= ?
        ORDER BY timestamp ASC
    """, (slug, cutoff)).fetchall()
    conn.close()

    if len(rows) < 2:
        r = rows[-1] if rows else None
        return {
            "slug": slug,
            "installs_delta": 0,
            "downloads_delta": 0,
            "stars_delta": 0,
            "pct_increase": 0.0,
            "snapshots": len(rows),
            "latest_installs": r["installs_current"] if r else 0,
            "latest_downloads": r["downloads"] if r else 0,
            "latest_stars": r["stars"] if r else 0,
        }

    e, l = rows[0], rows[-1]
    id_ = l["installs_current"] - e["installs_current"]
    dd = l["downloads"] - e["downloads"]
    sd = l["stars"] - e["stars"]
    base = e["installs_current"]
    pct = (id_ / base * 100) if base > 0 else (100.0 if id_ > 0 else 0.0)

    return {
        "slug": slug,
        "installs_delta": id_, "downloads_delta": dd, "stars_delta": sd,
        "pct_increase": round(pct, 2), "snapshots": len(rows),
        "latest_installs": l["installs_current"],
        "latest_downloads": l["downloads"],
        "latest_stars": l["stars"],
    }


def rank_trending(top_n=10, days=7):
    """Rank all skills by trending velocity."""
    conn = get_conn()
    slugs = [r["slug"] for r in conn.execute("SELECT slug FROM skills").fetchall()]
    conn.close()

    scored = []
    for slug in slugs:
        v = get_velocity(slug, days)
        # On first run, use raw installs as proxy
        if v["snapshots"] <= 1:
            v["installs_delta"] = v["latest_installs"]
            v["downloads_delta"] = v["latest_downloads"]

        # Composite: 35% installs delta + 15% downloads delta + 30% pct + 20% stars
        id_n = min(max(v["installs_delta"], 0), 2000) / 2000
        dd_n = min(max(v["downloads_delta"], 0), 10000) / 10000
        pn = min(max(v["pct_increase"], 0), 500) / 500
        sn = min(v["latest_stars"], 200) / 200
        score = 0.35 * id_n + 0.15 * dd_n + 0.30 * pn + 0.20 * sn

        v["score"] = round(score, 4)

        # Get metadata
        conn2 = get_conn()
        row = conn2.execute("SELECT * FROM skills WHERE slug = ?", (slug,)).fetchone()
        conn2.close()
        if row:
            v["display_name"] = row["display_name"]
            v["summary"] = row["summary"]
            v["author"] = row["author"]
            v["clawhub_url"] = row["clawhub_url"]
        scored.append(v)

    scored.sort(key=lambda x: x["score"], reverse=True)
    return scored[:top_n]


# --- Output ---

def render_report(ranked, week_label=""):
    """Generate markdown report."""
    if not week_label:
        week_label = f"Week of {datetime.now().strftime('%b %d, %Y')}"

    lines = [
        f"# OpenClaw Skills Weekly — {week_label}",
        "",
        f"*Top {len(ranked)} trending ClawHub skills ranked by 7-day install velocity.*",
        "",
        "| # | Skill | Installs | 7d Delta | DL Delta | Stars | Score |",
        "|---|-------|----------|----------|----------|-------|-------|",
    ]
    for i, r in enumerate(ranked, 1):
        name = r.get("display_name", r["slug"])
        snap = f"({r['snapshots']}d)" if r["snapshots"] > 1 else "(new)"
        lines.append(
            f"| {i} | [{name}]({r.get('clawhub_url', '')}) "
            f"| {r['latest_installs']:,} "
            f"| +{r['installs_delta']:,} {snap} "
            f"| +{r['downloads_delta']:,} "
            f"| {r['latest_stars']} "
            f"| {r['score']:.4f} |"
        )

    lines += ["", "---", ""]

    # Append community signals if available
    if SIGNALS_PATH.exists():
        try:
            signals = json.loads(SIGNALS_PATH.read_text())
            if signals:
                lines.append("## Community Buzz This Week")
                lines.append("")
                for s in signals:
                    title = s.get("title", "")
                    url = s.get("url", "")
                    summary = s.get("summary", "")
                    cat = s.get("category", "discussion")
                    link = f"[{title}]({url})" if url else title
                    lines.append(f"- **[{cat}]** {link}")
                    if summary:
                        lines.append(f"  {summary}")
                    lines.append("")
                lines.append("---")
        except (json.JSONDecodeError, OSError):
            pass

    return "\n".join(lines)


def db_status():
    """Print DB health info."""
    conn = get_conn()
    total = conn.execute("SELECT COUNT(*) FROM metrics_history").fetchone()[0]
    dates = conn.execute(
        "SELECT DISTINCT DATE(timestamp) as d FROM metrics_history ORDER BY d"
    ).fetchall()
    skill_count = conn.execute("SELECT COUNT(*) FROM skills").fetchone()[0]

    top = conn.execute("""
        SELECT m.slug, m.installs_current, m.downloads, m.stars, s.display_name
        FROM metrics_history m JOIN skills s ON s.slug = m.slug
        WHERE m.timestamp = (SELECT MAX(timestamp) FROM metrics_history WHERE slug = m.slug)
        ORDER BY m.installs_current DESC LIMIT 15
    """).fetchall()
    conn.close()

    print(f"DB: {DB_PATH}")
    print(f"Skills: {skill_count}")
    print(f"Snapshot rows: {total}")
    print(f"Snapshot dates: {len(dates)}")
    for d in dates[-7:]:
        print(f"  {d['d']}")
    print(f"\nTop 15 by current installs:")
    for r in top:
        print(f"  {r['display_name']:35s} installs={r['installs_current']:>6,}  dl={r['downloads']:>8,}  stars={r['stars']:>4}")


# --- Main ---

def main():
    parser = argparse.ArgumentParser(description="ClawHub Skills Tracker")
    parser.add_argument("mode", choices=["snapshot", "report", "trending", "status", "community"],
                        help="Operation mode")
    parser.add_argument("--top", type=int, default=10)
    parser.add_argument("--days", type=int, default=7)
    parser.add_argument("--sort", default="downloads")
    parser.add_argument("--max-pages", type=int, default=0)
    parser.add_argument("--save", action="store_true")
    args = parser.parse_args()

    init_db()

    if args.mode == "status":
        db_status()
        return

    if args.mode == "snapshot":
        skills = discover_all(sort=args.sort, max_pages=args.max_pages)
        if skills:
            record_snapshot(skills)
            # Print summary
            conn = get_conn()
            dates = conn.execute(
                "SELECT DISTINCT DATE(timestamp) as d FROM metrics_history"
            ).fetchall()
            total = conn.execute("SELECT COUNT(*) FROM metrics_history").fetchone()[0]
            conn.close()
            print(f"[SNAPSHOT] DB: {len(dates)} snapshot dates, {total} total rows")
        return

    if args.mode == "trending":
        ranked = rank_trending(top_n=args.top, days=args.days)
        print(f"\nTop {len(ranked)} trending skills ({args.days}-day velocity):\n")
        for i, r in enumerate(ranked, 1):
            snap = f"({r['snapshots']} snapshots)" if r["snapshots"] > 1 else "(first run)"
            print(f"  #{i:2d} {r.get('display_name', r['slug']):35s} "
                  f"score={r['score']:.4f} "
                  f"installs_delta=+{r['installs_delta']:,} "
                  f"pct=+{r['pct_increase']:.1f}% "
                  f"stars={r['latest_stars']} {snap}")
        return

    if args.mode == "report":
        # Snapshot first
        skills = discover_all(sort=args.sort, max_pages=args.max_pages)
        if skills:
            record_snapshot(skills)

        # Rank
        ranked = rank_trending(top_n=args.top, days=args.days)

        # Render
        md = render_report(ranked)
        report_name = f"openclaw_weekly_{datetime.now().strftime('%Y%m%d')}.md"

        # Save locally
        local_path = DATA_DIR / report_name
        local_path.write_text(md)
        print(f"[REPORT] Saved: {local_path}")

        # Also save to host mount if available
        if HOST_OUTPUT.exists():
            host_path = HOST_OUTPUT / report_name
            host_path.write_text(md)
            print(f"[REPORT] Also saved to host: {host_path}")

        # Print to stdout for Claude to see
        print("\n" + md)
        return

    if args.mode == "community":
        print("[COMMUNITY] Community signal capture")
        print("  This mode is invoked by the OpenClaw gateway's WebSearch tool.")
        print("  The SKILL.md instructs Claude to search and save signals.")
        print(f"  Signals file: {SIGNALS_PATH}")
        if SIGNALS_PATH.exists():
            signals = json.loads(SIGNALS_PATH.read_text())
            print(f"  Current signals: {len(signals)}")
            for s in signals:
                print(f"    [{s.get('category', '?')}] {s.get('title', '?')}")
        else:
            print("  No signals captured yet.")
        return


if __name__ == "__main__":
    main()


path: project_tracker.py
content: """
project_tracker.py — OpenClaw Ecosystem GitHub Metadata Tracker

Tracks stars, forks, PRs, releases, and commit activity for core OpenClaw repos.
Uses GitHub API via GITHUB_TOKEN from .env.
Called from hourly_heartbeat.py.
"""

import json
import os
import subprocess
from datetime import datetime, timezone

import storage

# Core OpenClaw ecosystem repos to track
REPOS = [
    "openclaw/openclaw",    # Main framework (245K+ stars)
    "openclaw/clawhub",     # Skill registry CLI (3.5K stars)
    "openclaw/skills",      # Archived skill versions
]


def _gh_api(endpoint: str) -> dict | list | None:
    """Call GitHub API via gh CLI. Returns parsed JSON or None on error."""
    try:
        result = subprocess.run(
            ["gh", "api", endpoint],
            capture_output=True, text=True, timeout=15, encoding="utf-8",
        )
        if result.returncode != 0:
            print(f"  [WARN] gh api {endpoint}: {(result.stderr or '').strip()[:100]}")
            return None
        if not result.stdout:
            return None
        return json.loads(result.stdout)
    except (subprocess.TimeoutExpired, FileNotFoundError, json.JSONDecodeError) as e:
        print(f"  [WARN] gh api {endpoint}: {e}")
        return None


def snapshot_repos() -> list[dict]:
    """Fetch metadata for all tracked repos. Returns list of dicts for storage."""
    results = []

    for repo in REPOS:
        print(f"  [PROJECT] {repo}...")
        data = _gh_api(f"repos/{repo}")
        if not data:
            continue

        entry = {
            "repo": repo,
            "stars": data.get("stargazers_count", 0),
            "forks": data.get("forks_count", 0),
            "open_issues": data.get("open_issues_count", 0),
            "watchers": data.get("subscribers_count", 0),
        }

        # Open PRs count
        prs = _gh_api(f"search/issues?q=repo:{repo}+type:pr+state:open")
        entry["open_prs"] = prs.get("total_count", 0) if prs else 0

        # Latest release
        releases = _gh_api(f"repos/{repo}/releases?per_page=1")
        if releases and len(releases) > 0:
            entry["latest_release"] = releases[0].get("tag_name", "")
            entry["latest_release_date"] = releases[0].get("published_at", "")[:10]
        else:
            entry["latest_release"] = ""
            entry["latest_release_date"] = ""

        # Weekly commit count (from commit_activity stats)
        stats = _gh_api(f"repos/{repo}/stats/commit_activity")
        if stats and len(stats) > 0:
            entry["weekly_commits"] = stats[-1].get("total", 0)
        else:
            entry["weekly_commits"] = 0

        results.append(entry)

    return results


def capture() -> int:
    """Full capture: snapshot repos and store to DB."""
    storage.init_project_metadata()
    entries = snapshot_repos()
    if not entries:
        print("[PROJECT] No repo data captured")
        return 0
    return storage.record_project_metadata(entries)


if __name__ == "__main__":
    capture()


path: ranker.py
content: """
ranker.py — Trending Velocity Calculator (v5)

Two-track ranking system:
  MOVERS   — established skills ranked by installs_all_time delta (cumulative growth)
  ROCKETS  — new skills (<30 days) ranked by growth + velocity with recency bonus

Uses installs_all_time (monotonically increasing counter) instead of
installs_current (gauge that oscillates near zero for established skills).

Cold-start handling: when only 1 snapshot exists, uses age-normalized proxy
(average daily rate * 7) instead of raw cumulative totals. This prevents
the rankings from being a popularity contest and reduces day-2 cliff.
"""

import storage
from datetime import datetime, timezone, timedelta


# --- MOVER scoring weights ---
W_IAT_DELTA = 0.35       # installs_all_time delta (primary trending signal)
W_DOWNLOADS_DELTA = 0.15  # Download growth (secondary)
W_PCT_INCREASE = 0.25     # Relative velocity (breakout detection)
W_STARS = 0.15            # Stars as quality signal
W_STARS_DELTA = 0.10      # Star growth (community momentum)

# Normalisation caps (calibrated for weekly deltas, not cumulative)
MAX_IAT_DELTA = 500       # installs_all_time weekly growth
MAX_DOWNLOADS_DELTA = 5000
MAX_PCT = 200.0
MAX_STARS = 1000
MAX_STARS_DELTA = 50

# Author diversity: max entries per author in top N
MAX_PER_AUTHOR = 3


def _skill_age_days(skill: dict) -> int:
    """Return skill age in days from created_at, or 365 if unknown."""
    created_ms = skill.get("created_at", 0)
    if not created_ms:
        return 365
    created = datetime.fromtimestamp(created_ms / 1000, tz=timezone.utc)
    return max((datetime.now(timezone.utc) - created).days, 1)


def _mover_score(vel: dict, stars: int) -> float:
    """Composite score for established skills (30+ days old)."""
    iat_delta = vel.get("installs_all_time_delta", vel.get("installs_delta", 0))
    iat_norm = min(max(iat_delta, 0), MAX_IAT_DELTA) / MAX_IAT_DELTA
    dl_norm = min(max(vel["downloads_delta"], 0), MAX_DOWNLOADS_DELTA) / MAX_DOWNLOADS_DELTA
    pct = vel.get("pct_increase", 0.0)
    pct_norm = min(max(pct, 0), MAX_PCT) / MAX_PCT if pct >= 0 else 0.0
    stars_norm = min(stars, MAX_STARS) / MAX_STARS
    stars_d_norm = min(max(vel.get("stars_delta", 0), 0), MAX_STARS_DELTA) / MAX_STARS_DELTA

    return (
        W_IAT_DELTA * iat_norm
        + W_DOWNLOADS_DELTA * dl_norm
        + W_PCT_INCREASE * pct_norm
        + W_STARS * stars_norm
        + W_STARS_DELTA * stars_d_norm
    )


def _rocket_score(skill: dict, vel: dict) -> float:
    """Score for new skills (<30 days). Uses velocity when available, recency bonus applied."""
    age_days = _skill_age_days(skill)
    recency_bonus = 1 + (1 / (1 + age_days))  # day 0 = 2x, day 7 = 1.125x

    snaps = vel.get("snapshots_used", 0)

    if snaps >= 2:
        # Real velocity data available — use it
        iat_delta = vel.get("installs_all_time_delta", 0)
        dl_delta = vel.get("downloads_delta", 0)
        stars = vel.get("latest_stars", skill.get("stars", 0))
        iat_norm = min(max(iat_delta, 0), 100) / 100
        dl_norm = min(max(dl_delta, 0), 1000) / 1000
        stars_norm = min(stars, 50) / 50
    else:
        # Cold start: use age-normalized average weekly rate
        iat = skill.get("installs_all_time", skill.get("installs_current", 0))
        dl = skill.get("downloads", 0)
        stars = skill.get("stars", 0)
        weekly_rate_iat = iat / age_days * 7
        weekly_rate_dl = dl / age_days * 7
        iat_norm = min(weekly_rate_iat, 100) / 100
        dl_norm = min(weekly_rate_dl, 1000) / 1000
        stars_norm = min(stars, 50) / 50

    raw = 0.40 * iat_norm + 0.30 * dl_norm + 0.30 * stars_norm
    return raw * recency_bonus


def _is_new(skill: dict, days: int = 30) -> bool:
    """Check if skill was created within the last N days."""
    created_ms = skill.get("created_at", 0)
    if not created_ms:
        return False
    created = datetime.fromtimestamp(created_ms / 1000, tz=timezone.utc)
    return (datetime.now(timezone.utc) - created).days < days


def _apply_author_diversity(ranked: list[dict], max_per: int, total: int) -> list[dict]:
    """Cap entries per author in the top list to ensure diversity."""
    result = []
    author_count: dict[str, int] = {}
    for skill in ranked:
        if len(result) >= total:
            break
        author = (skill.get("author") or "").lower()
        if not author:
            result.append(skill)
            continue
        count = author_count.get(author, 0)
        if count < max_per:
            result.append(skill)
            author_count[author] = count + 1
    return result


def rank(skills: list[dict], db_days: int = 7, top_n: int = 10) -> list[dict]:
    """
    Rank skills by trending velocity using two-track system.

    Returns top_n MOVERS (author-diversified) plus up to 5 ROCKETS (new skills <30 days).
    """
    velocity_map = {v["slug"]: v for v in storage.get_all_velocities(days=db_days)}

    movers = []
    rockets = []

    for skill in skills:
        slug = skill["slug"]
        vel = velocity_map.get(slug, {
            "slug": slug,
            "installs_delta": 0,
            "installs_all_time_delta": 0,
            "downloads_delta": 0,
            "pct_increase": 0.0,
            "latest_installs": skill.get("installs_current", 0),
            "latest_installs_all_time": skill.get("installs_all_time", 0),
            "latest_downloads": skill.get("downloads", 0),
            "latest_stars": skill.get("stars", 0),
            "stars_delta": 0,
            "snapshots_used": 0,
        })

        snaps = vel.get("snapshots_used", 0)

        # Cold-start proxy: age-normalized weekly estimate instead of raw cumulative
        if snaps <= 1:
            age_days = _skill_age_days(skill)
            iat_raw = skill.get("installs_all_time", skill.get("installs_current", 0))
            dl_raw = skill.get("downloads", 0)
            vel["installs_all_time_delta"] = round(iat_raw / age_days * 7)
            vel["downloads_delta"] = round(dl_raw / age_days * 7)
            vel["pct_increase"] = -1.0  # sentinel: "no data"

        stars = vel.get("latest_stars", skill.get("stars", 0))

        if _is_new(skill, days=30):
            score = _rocket_score(skill, vel)
            rockets.append({
                **skill,
                "_score": round(score, 4),
                "_track": "rocket",
                "_installs_delta": vel.get("installs_all_time_delta", 0),
                "_downloads_delta": vel["downloads_delta"],
                "_stars_delta": vel.get("stars_delta", 0),
                "_pct_increase": vel["pct_increase"],
                "_snapshots_used": snaps,
                "_cold_start": snaps <= 1,
            })
        else:
            score = _mover_score(vel, stars)
            movers.append({
                **skill,
                "_score": round(score, 4),
                "_track": "mover",
                "_installs_delta": vel.get("installs_all_time_delta", 0),
                "_downloads_delta": vel["downloads_delta"],
                "_stars_delta": vel.get("stars_delta", 0),
                "_pct_increase": vel["pct_increase"],
                "_snapshots_used": snaps,
                "_cold_start": snaps <= 1,
            })

    movers.sort(key=lambda s: s["_score"], reverse=True)
    rockets.sort(key=lambda s: s["_score"], reverse=True)

    # Author diversity: prevent one author from dominating the list
    top_movers = _apply_author_diversity(movers, MAX_PER_AUTHOR, top_n)
    top_rockets = rockets[:5]

    def _safe(text: str) -> str:
        return text.encode("ascii", errors="replace").decode("ascii")

    print(f"[RANKER] {len(movers)} movers, {len(rockets)} rockets")
    print(f"[RANKER] Top {len(top_movers)} movers:")
    for i, s in enumerate(top_movers, 1):
        hist = f"({s['_snapshots_used']}d)" if s["_snapshots_used"] > 1 else "(est)"
        print(
            f"  #{i} {_safe(s['display_name'])} score={s['_score']} "
            f"iat_delta=+{s['_installs_delta']} dl_delta=+{s['_downloads_delta']} "
            f"stars={s.get('stars', 0)} {hist}"
        )

    if top_rockets:
        print(f"[RANKER] Top {len(top_rockets)} rockets (new <30d):")
        for i, s in enumerate(top_rockets, 1):
            age = "new"
            if s.get("created_at"):
                age_d = (datetime.now(timezone.utc) - datetime.fromtimestamp(s["created_at"] / 1000, tz=timezone.utc)).days
                age = f"{age_d}d old"
            print(f"  #{i} {_safe(s['display_name'])} score={s['_score']} {age}")

    return top_movers, top_rockets


path: reddit_capture.py
content: """
reddit_capture.py — Reddit + Hacker News Signal Capture

Fetches OpenClaw/ClawHub discussions from Reddit and Hacker News.
Structures them via Claude Haiku into the standard signal schema.
Appends to data/weekly_signals.json (same file as x_capture.py).

All APIs are free and require no authentication.
"""

import json
import os
import time
from datetime import datetime, timezone, timedelta
from pathlib import Path
from urllib.request import Request, urlopen
from urllib.error import URLError

import anthropic

SIGNALS_PATH = Path(__file__).parent / "data" / "weekly_signals.json"

# Reddit search queries (no auth needed)
REDDIT_QUERIES = [
    "openclaw OR clawhub",
    "openclaw skills",
    "clawhub skill marketplace",
]

REDDIT_SUBREDDITS = [
    "ClaudeAI",
    "LocalLLaMA",
    "artificial",
]

# Hacker News queries (Algolia API, free)
HN_QUERIES = ["openclaw", "clawhub"]

HAIKU_MODEL = "claude-haiku-4-5-20251001"

STRUCTURE_PROMPT = """You are a signal classifier for an AI agent skills newsletter.

Given these community posts about OpenClaw/ClawHub, extract structured signals.

For EACH relevant post, output a JSON object with:
- "date": ISO date string (YYYY-MM-DD)
- "category": one of "tutorial", "security", "ecosystem", "showcase", "discussion", "market"
- "title": concise descriptive title (not the raw post title)
- "url": the post URL
- "summary": 1-2 sentence summary of what's discussed
- "tags": list of 2-4 lowercase tags
- "source": "{source}"

Skip posts that are:
- Not about OpenClaw, ClawHub, or AI agent skills
- Spam, low-effort, or duplicates
- Job postings or self-promotion without substance

Output a JSON array of signal objects. If no posts are relevant, output [].
"""


def _fetch_json(url: str) -> dict | list | None:
    """Fetch JSON from a URL with User-Agent header."""
    try:
        req = Request(url, headers={
            "User-Agent": "OpenClawSkillsWeekly/1.0 (signal capture bot)",
        })
        with urlopen(req, timeout=15) as resp:
            return json.loads(resp.read().decode("utf-8"))
    except (URLError, json.JSONDecodeError, TimeoutError) as e:
        print(f"  [WARN] Fetch failed {url[:80]}: {e}")
        return None


def _search_reddit(days: int = 7) -> list[dict]:
    """Search Reddit for OpenClaw/ClawHub posts. No auth required."""
    posts = []
    seen_ids = set()

    for query in REDDIT_QUERIES:
        url = (
            f"https://www.reddit.com/search.json"
            f"?q={query.replace(' ', '+')}&sort=new&limit=50&t=week"
        )
        data = _fetch_json(url)
        if not data or "data" not in data:
            continue

        for child in data["data"].get("children", []):
            post = child.get("data", {})
            post_id = post.get("id", "")
            if post_id in seen_ids:
                continue
            seen_ids.add(post_id)

            created = datetime.fromtimestamp(post.get("created_utc", 0), tz=timezone.utc)
            posts.append({
                "title": post.get("title", ""),
                "url": f"https://reddit.com{post.get('permalink', '')}",
                "subreddit": post.get("subreddit", ""),
                "score": post.get("score", 0),
                "num_comments": post.get("num_comments", 0),
                "date": created.strftime("%Y-%m-%d"),
                "selftext": (post.get("selftext", "") or "")[:500],
            })
        time.sleep(1)  # Reddit rate limit politeness

    # Also check specific subreddits
    for sub in REDDIT_SUBREDDITS:
        url = (
            f"https://www.reddit.com/r/{sub}/search.json"
            f"?q=openclaw+OR+clawhub&sort=new&limit=25&t=week&restrict_sr=on"
        )
        data = _fetch_json(url)
        if not data or "data" not in data:
            continue

        for child in data["data"].get("children", []):
            post = child.get("data", {})
            post_id = post.get("id", "")
            if post_id in seen_ids:
                continue
            seen_ids.add(post_id)

            created = datetime.fromtimestamp(post.get("created_utc", 0), tz=timezone.utc)
            posts.append({
                "title": post.get("title", ""),
                "url": f"https://reddit.com{post.get('permalink', '')}",
                "subreddit": post.get("subreddit", ""),
                "score": post.get("score", 0),
                "num_comments": post.get("num_comments", 0),
                "date": created.strftime("%Y-%m-%d"),
                "selftext": (post.get("selftext", "") or "")[:500],
            })
        time.sleep(1)

    print(f"  [REDDIT] Found {len(posts)} posts")
    return posts


def _search_hackernews(days: int = 7) -> list[dict]:
    """Search Hacker News via Algolia API. Free, no auth."""
    cutoff = int((datetime.now(timezone.utc) - timedelta(days=days)).timestamp())
    posts = []
    seen_ids = set()

    for query in HN_QUERIES:
        url = (
            f"https://hn.algolia.com/api/v1/search_by_date"
            f"?query={query}&tags=story"
            f"&numericFilters=created_at_i>{cutoff}"
        )
        data = _fetch_json(url)
        if not data:
            continue

        for hit in data.get("hits", []):
            hn_id = hit.get("objectID", "")
            if hn_id in seen_ids:
                continue
            seen_ids.add(hn_id)

            posts.append({
                "title": hit.get("title", ""),
                "url": hit.get("url") or f"https://news.ycombinator.com/item?id={hn_id}",
                "hn_url": f"https://news.ycombinator.com/item?id={hn_id}",
                "points": hit.get("points", 0),
                "num_comments": hit.get("num_comments", 0),
                "date": datetime.fromtimestamp(
                    hit.get("created_at_i", 0), tz=timezone.utc
                ).strftime("%Y-%m-%d"),
            })

    print(f"  [HN] Found {len(posts)} stories")
    return posts


def _fix_json(text: str) -> str:
    """Try to fix common LLM JSON issues (trailing commas, unescaped quotes)."""
    import re
    # Remove trailing commas before ] or }
    text = re.sub(r',\s*([}\]])', r'\1', text)
    # Fix unescaped newlines inside strings
    text = text.replace('\n', '\\n')
    # Restore structural newlines (between array items and object braces)
    text = re.sub(r'\\n(\s*[{\[\]},])', lambda m: '\n' + m.group(1), text)
    return text


def _structure_batch(posts: list[dict], source: str, client) -> list[dict]:
    """Structure a single batch of posts via Haiku."""
    post_text = "\n\n".join(
        f"Title: {p['title']}\nURL: {p['url']}\nDate: {p['date']}\n"
        f"Score: {p.get('score', p.get('points', 0))}, "
        f"Comments: {p.get('num_comments', 0)}\n"
        f"{'Subreddit: ' + p['subreddit'] if 'subreddit' in p else ''}\n"
        f"{p.get('selftext', '')[:200]}"
        for p in posts
    )

    prompt = STRUCTURE_PROMPT.replace("{source}", source) + f"\n\nPosts:\n{post_text}"

    try:
        msg = client.messages.create(
            model=HAIKU_MODEL,
            max_tokens=4000,
            messages=[{"role": "user", "content": prompt}],
        )
        text = msg.content[0].text.strip()
        start = text.find("[")
        end = text.rfind("]") + 1
        if start >= 0 and end > start:
            raw = text[start:end]
            try:
                return json.loads(raw)
            except json.JSONDecodeError:
                return json.loads(_fix_json(raw))
    except (anthropic.APIError, json.JSONDecodeError) as e:
        print(f"  [WARN] Batch structuring failed for {source}: {e}")

    return []


def _structure_signals(posts: list[dict], source: str, api_key: str) -> list[dict]:
    """Use Haiku to classify and structure raw posts into signal schema.
    Batches posts in chunks of 30 to avoid LLM output truncation."""
    if not posts:
        return []

    client = anthropic.Anthropic([REDACTED_SECRET])
    batch_size = 30
    all_signals = []

    for i in range(0, len(posts), batch_size):
        batch = posts[i:i + batch_size]
        signals = _structure_batch(batch, source, client)
        all_signals.extend(signals)
        if i + batch_size < len(posts):
            time.sleep(0.5)  # Rate limit politeness

    print(f"  [STRUCTURE] {len(all_signals)} signals from {source} ({len(posts)} posts in {(len(posts) - 1) // batch_size + 1} batches)")
    return all_signals


def _dedup_and_append(new_signals: list[dict]) -> int:
    """Deduplicate by URL and append to weekly_signals.json."""
    SIGNALS_PATH.parent.mkdir(parents=True, exist_ok=True)

    existing = []
    if SIGNALS_PATH.exists():
        try:
            existing = json.loads(SIGNALS_PATH.read_text(encoding="utf-8"))
        except json.JSONDecodeError:
            existing = []

    existing_urls = {s.get("url", "") for s in existing}
    added = 0

    for sig in new_signals:
        url = sig.get("url", "")
        if url and url not in existing_urls:
            existing.append(sig)
            existing_urls.add(url)
            added += 1

    SIGNALS_PATH.write_text(json.dumps(existing, indent=2), encoding="utf-8")
    print(f"  [DEDUP] {added} new signals added ({len(existing)} total)")
    return added


def capture(days: int = 7, skip_haiku: bool = False) -> int:
    """
    Full capture: Reddit + HN search → structure → dedup → append.
    Returns number of new signals added.
    """
    [REDACTED_SECRET]"ANTHROPIC_API_KEY", "")

    print("[REDDIT+HN] Searching for OpenClaw/ClawHub signals...")

    # Fetch from both sources
    reddit_posts = _search_reddit(days=days)
    hn_posts = _search_hackernews(days=days)

    all_signals = []

    if reddit_posts and api_key and not skip_haiku:
        all_signals.extend(_structure_signals(reddit_posts, "reddit", api_key))
    elif reddit_posts:
        # Fallback: create basic signals without Haiku structuring
        for p in reddit_posts:
            all_signals.append({
                "date": p["date"],
                "category": "discussion",
                "title": p["title"][:100],
                "url": p["url"],
                "summary": p.get("selftext", "")[:200] or p["title"],
                "tags": ["reddit", p.get("subreddit", "").lower()],
                "source": "reddit",
            })

    if hn_posts and api_key and not skip_haiku:
        all_signals.extend(_structure_signals(hn_posts, "hackernews", api_key))
    elif hn_posts:
        for p in hn_posts:
            all_signals.append({
                "date": p["date"],
                "category": "discussion",
                "title": p["title"][:100],
                "url": p.get("hn_url", p["url"]),
                "summary": p["title"],
                "tags": ["hackernews"],
                "source": "hackernews",
            })

    if not all_signals:
        print("[REDDIT+HN] No signals found")
        return 0

    return _dedup_and_append(all_signals)


if __name__ == "__main__":
    from dotenv import load_dotenv
    load_dotenv()
    capture()


path: requirements.txt
content: httpx>=0.27.0
python-dotenv>=1.0.0
anthropic>=0.49.0
beautifulsoup4>=4.12.0


path: run_weekly.py
content: """
run_weekly.py — One-command weekly report generation (v5)

Runs the full pipeline in order:
  0. bridge: Copy container DB (has daily snapshot history) to host
  1. x_capture: Scan X/Twitter for OpenClaw signals (append to existing)
  2. discovery: Fetch all skills from ClawHub API
  3. storage: Record snapshot to SQLite time-series DB
  4. ranker: Two-track ranking (Movers + Rockets)
  5. harvester: Fetch docs for top skills
  6. script_generator: Generate YouTube pitches via LLM

Usage:
    python run_weekly.py                     # Full run, top 10, all pages
    python run_weekly.py --top 5             # Top 5 only
    python run_weekly.py --skip-x            # Skip X/Twitter capture
    python run_weekly.py --snapshot-only     # Just snapshot, no scripts
    python run_weekly.py --max-pages 3       # Limit API pages (testing)
    python run_weekly.py --no-bridge         # Skip container DB copy

For daily cron (snapshot accumulation only):
    python run_weekly.py --snapshot-only --skip-x
"""

import argparse
import json
import os
import shutil
import subprocess
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path

from dotenv import load_dotenv

load_dotenv()

CONTAINER_NAME = "openclaw-gateway-secure"
CONTAINER_DB = "/home/node/.openclaw/workspace/data/skills-weekly/metrics.db"


def _bridge_container_db():
    """Copy container's metrics.db (with daily snapshot history) to host."""
    host_db = Path(__file__).parent / "data" / "metrics.db"
    host_db.parent.mkdir(parents=True, exist_ok=True)

    # Check container health
    try:
        result = subprocess.run(
            ["docker", "ps", "--filter", f"name={CONTAINER_NAME}",
             "--format", "{{.Status}}"],
            capture_output=True, text=True, timeout=10,
        )
        if "(healthy)" not in result.stdout:
            print(f"  [WARN] Container {CONTAINER_NAME} not healthy, skipping DB bridge")
            return False
    except (subprocess.TimeoutExpired, FileNotFoundError):
        print("  [WARN] Docker not available, skipping DB bridge")
        return False

    # Backup existing host DB
    if host_db.exists():
        backup = host_db.with_suffix(".db.bak")
        shutil.copy2(host_db, backup)
        print(f"  Host DB backed up to {backup.name}")

    # Copy container DB to host
    try:
        result = subprocess.run(
            ["docker", "cp", f"{CONTAINER_NAME}:{CONTAINER_DB}", str(host_db)],
            capture_output=True, text=True, timeout=30,
        )
        if result.returncode != 0:
            print(f"  [WARN] docker cp failed: {result.stderr.strip()}")
            # Restore backup
            backup = host_db.with_suffix(".db.bak")
            if backup.exists():
                shutil.copy2(backup, host_db)
                print("  Restored backup")
            return False
        print(f"  Container DB copied to host ({host_db.stat().st_size // 1024}KB)")
        return True
    except subprocess.TimeoutExpired:
        print("  [WARN] docker cp timed out")
        return False


def main():
    parser = argparse.ArgumentParser(description="Full weekly pipeline runner")
    parser.add_argument("--top", type=int, default=10)
    parser.add_argument("--days", type=int, default=7)
    parser.add_argument("--output", type=str, default=None)
    parser.add_argument("--week", type=str, default=None)
    parser.add_argument("--model", type=str, default="claude-haiku-4-5-20251001")
    parser.add_argument("--sort", type=str, default="downloads")
    parser.add_argument("--max-pages", type=int, default=0)
    parser.add_argument("--snapshot-only", action="store_true")
    parser.add_argument("--skip-x", action="store_true", help="Skip X/Twitter capture")
    parser.add_argument("--mock", action="store_true")
    parser.add_argument("--no-bridge", action="store_true", help="Skip container DB copy")
    parser.add_argument("--episode", type=int, default=1, help="Episode number for video script")
    args = parser.parse_args()

    now = datetime.now(timezone.utc)
    week_label = args.week or f"Week of {now.strftime('%b %d, %Y')}"

    print("=" * 60)
    print("  OpenClaw Skills Weekly — Full Pipeline (v4)")
    print(f"  {week_label}")
    print("=" * 60)

    # Step 1: Multi-source signal capture (X + Reddit + HN)
    if not args.skip_x and not args.snapshot_only:
        print("\n" + "=" * 40)
        print("  PHASE 1: Multi-Source Signal Capture")
        print("=" * 40)

        # X/Twitter
        try:
            import x_capture
            date_from = (now - timedelta(days=args.days)).strftime("%Y-%m-%d")
            date_to = now.strftime("%Y-%m-%d")
            x_capture.capture(date_from=date_from, date_to=date_to, append=True)
        except Exception as e:
            print(f"[WARN] X capture failed (non-fatal): {e}")

        # Reddit + Hacker News
        try:
            import reddit_capture
            reddit_capture.capture(days=args.days)
        except Exception as e:
            print(f"[WARN] Reddit/HN capture failed (non-fatal): {e}")

    # Step 0: Bridge container DB (get real daily snapshot history)
    if not args.no_bridge:
        print("\n" + "=" * 40)
        print("  PHASE 0: Container DB Bridge")
        print("=" * 40)
        _bridge_container_db()

    # Rollup old hourly data (keep last 30 days granular, compact the rest)
    try:
        import storage as _storage_rollup
        _storage_rollup.init_db()
        rolled = _storage_rollup.rollup_hourly()
        if rolled > 0:
            print(f"  [ROLLUP] Compacted {rolled} old hourly rows")
    except Exception as e:
        print(f"  [WARN] Rollup failed (non-fatal): {e}")

    # OpenClaw project metadata snapshot (GitHub repo tracking)
    if not args.snapshot_only:
        try:
            import project_tracker
            print("\n  [PROJECT] Capturing OpenClaw ecosystem metadata...")
            project_tracker.capture()
        except Exception as e:
            print(f"  [WARN] Project tracker failed (non-fatal): {e}")

    # Step 2-6: ClawHub pipeline
    print("\n" + "=" * 40)
    print("  PHASE 2: ClawHub Data Pipeline")
    print("=" * 40)

    import discovery
    import storage
    import ranker
    import harvester
    import script_generator

    env = {
        "github_token": os.getenv("GITHUB_TOKEN", ""),
        "anthropic_key": os.getenv("ANTHROPIC_API_KEY", ""),
        "clawhub_url": os.getenv("CLAWHUB_BASE_URL", ""),
    }

    storage.init_db()

    print(f"\n[1/5] Discovering ClawHub skills (sort={args.sort})...")
    skills = discovery.discover(
        clawhub_url=env["clawhub_url"],
        sort=args.sort,
        mock=args.mock,
        max_pages=args.max_pages,
    )
    if not skills:
        print("[ERROR] No skills discovered.")
        sys.exit(1)

    print(f"\n[2/5] Saving snapshot of {len(skills)} skills to DB...")
    storage.upsert_skills(skills)
    storage.record_snapshot(skills)

    if args.snapshot_only:
        dates = storage.distinct_snapshot_dates()
        print(f"\n[DONE] Snapshot recorded ({len(skills)} skills).")
        print(f"       DB: {storage.snapshot_count()} rows across {len(dates)} date(s).")
        return

    print(f"\n[3/5] Ranking by {args.days}-day velocity (two-track: movers + rockets)...")
    top_movers, top_rockets = ranker.rank(skills, db_days=args.days, top_n=args.top)

    # Combine for harvesting and script gen
    all_top = top_movers + top_rockets

    print(f"\n[4/5] Harvesting content for {len(all_top)} skills...")
    harvested = harvester.harvest(all_top, github_token=env["github_token"])

    # Split back into movers and rockets after harvest
    movers_h = harvested[:len(top_movers)]
    rockets_h = harvested[len(top_movers):]

    if not env["anthropic_key"]:
        print("[WARN] No ANTHROPIC_API_KEY — skipping script generation")
        movers_s = [{**s, "script": "[No API key]"} for s in movers_h]
        rockets_s = [{**s, "script": "[No API key]"} for s in rockets_h]
    else:
        print(f"\n[5/5] Generating YouTube scripts via {args.model}...")
        all_scripted = script_generator.generate_scripts(harvested, env["anthropic_key"], model=args.model)
        movers_s = all_scripted[:len(top_movers)]
        rockets_s = all_scripted[len(top_movers):]

    # Time-series enrichment for report + JSON
    print("\n[6/6] Building time-series data...")

    # Per-skill history (sparkline data for each top skill)
    for skill in movers_s + rockets_s:
        slug = skill.get("slug", "")
        if slug:
            skill["history"] = storage.get_skill_history(slug, days=args.days)

    # Catalog-level aggregates (total skills, downloads, installs over time)
    catalog_history = storage.get_catalog_history(days=args.days)

    # OpenClaw project metadata (stars, PRs, releases)
    project_history = storage.get_project_history(days=args.days)

    # Output 1: Markdown report (data-rich, for reference)
    output_path = args.output or f"openclaw_weekly_{now.strftime('%Y%m%d')}.md"
    markdown = script_generator.render_markdown(
        movers_s, rockets_s, week_label=week_label,
        catalog_history=catalog_history, project_history=project_history,
    )
    Path(output_path).write_text(markdown, encoding="utf-8")

    # Output 2: Voice-ready video script (GitHubAwesome style)
    script_path = output_path.replace(".md", "_script.txt")
    video_script = script_generator.render_video_script(
        movers_s, rockets_s,
        episode_num=args.episode,
        week_label=week_label,
    )
    Path(script_path).write_text(video_script, encoding="utf-8")

    # Output 3: Structured JSON for Remotion video pipeline
    json_path = output_path.replace(".md", ".json")
    json_data = {
        "episode": args.episode,
        "week_label": week_label,
        "generated_at": now.isoformat(),
        "movers": movers_s,
        "rockets": rockets_s,
        "catalog": catalog_history,
        "openclaw_project": project_history,
    }
    Path(json_path).write_text(json.dumps(json_data, indent=2, default=str), encoding="utf-8")

    dates = storage.distinct_snapshot_dates()
    total = len(movers_s) + len(rockets_s)
    print(f"\n{'=' * 60}")
    print(f"  DONE:")
    print(f"    Report: {output_path}")
    print(f"    Script: {script_path}")
    print(f"    JSON:   {json_path}")
    print(f"  {len(movers_s)} movers + {len(rockets_s)} rockets | DB: {storage.snapshot_count()} rows across {len(dates)} date(s)")
    print(f"{'=' * 60}")


if __name__ == "__main__":
    main()


path: script_generator.py
content: """
script_generator.py — LLM Script Generator (v7 — GitHubAwesome Format)

Generates YouTube segments for top ClawHub skills.
Format cloned from GitHubAwesome's GitHub Trending Weekly (episodes 24-25):
  - ONE hook sentence (problem-first / absurdist / provocateur / punchy)
  - ONE-TWO sentences of what it does with exact technical details
  - 30-50 words per segment, no transitions, no filler
  - Dry, slightly sardonic — a knowledgeable friend reporting back

Supports two tracks: MOVERS (established) and ROCKETS (new <30 days).
Uses Anthropic SDK (claude-haiku-4-5 by default).
"""

import re

import anthropic
import community_signals
from datetime import datetime, timezone


# --- GitHubAwesome-style system prompt ---
# Calibrated from verbatim analysis of GitHubAwesome episodes 24 and 25
SYSTEM_PROMPT = (
    "You write YouTube video segments about AI agent skills. "
    "Your tone is dry, precise, and slightly sardonic — a knowledgeable friend "
    "who has already seen everything, reporting back with deadpan efficiency. "
    "The wit is in the framing, not in adjectives."
    "\n\n"
    "FORMAT — exactly this structure, nothing else:\n"
    "1. ONE hook sentence. Use one of these patterns:\n"
    "   - Problem-first: a specific scenario the listener recognises\n"
    "   - Absurdist-specific: a weird real detail that grabs attention\n"
    "   - Provocateur: a statement that makes the listener lean in\n"
    "   - Punchy tagline: a short declarative that frames the whole project\n"
    "2. ONE to TWO sentences of what it does. Name exact technologies, "
    "CLI commands, file formats, languages, or integrations. "
    "Describe active behaviour, not capability.\n"
    "3. STOP. No closer, no call to action, no summary sentence.\n"
    "\n"
    "BANNED:\n"
    "- Markdown, headers, bold, bullets, formatting of any kind\n"
    "- Download counts, install counts, star counts, popularity metrics\n"
    "- 'allows you to', 'enables you to', 'lets you' — use active voice instead\n"
    "- 'check it out', 'game changer', 'Welcome back', 'worth noting'\n"
    "- Superlatives: 'incredible', 'amazing', 'revolutionary', 'powerful', 'robust'\n"
    "- Generic problem framing like 'managing X is hard' — be specific\n"
    "- Transitional phrases, enthusiasm injection, filler of any kind\n"
    "\n"
    "HARD LIMIT: 30-50 words. Count them. Single paragraph. No line breaks.\n"
    "\n"
    "EXAMPLES of the exact style (from GitHub Trending Weekly):\n"
    "\n"
    "\"One thousand job applications in two days. Fully autonomous. "
    "JobBot fills forms, uploads resumes, and tracks responses across "
    "LinkedIn, Indeed, and Glassdoor via headless Chromium.\"\n"
    "\n"
    "\"A dad got tired of walking past his kid's tablet hearing gaming YouTubers. "
    "BrainRotGuard is a self-hosted YouTube approval system — sends Telegram "
    "notifications with Approve and Deny buttons. No YouTube account needed.\"\n"
    "\n"
    "\"Surge is a download manager built in Go for the terminal. "
    "Simultaneous connections, intelligent worker optimisation, three modes: "
    "interactive TUI, headless server, and CLI client.\""
)

USER_TEMPLATE_VELOCITY = """Skill: {display_name}
Author: {author}
Track: {track}

Summary: {summary}

Documentation:
{content}

Write one segment in the exact style shown. Hook sentence, then what it does. 30-50 words. Stop after that."""

USER_TEMPLATE_COLD = """Skill: {display_name} (brand new)
Author: {author}
Track: {track}

Summary: {summary}

Documentation:
{content}

Write one segment in the exact style shown. Hook sentence, then what it does — emphasise what's novel. 30-50 words. Stop after that."""


def _strip_markdown(text: str) -> str:
    """Remove any markdown formatting from LLM output for voice-ready text."""
    text = re.sub(r"^#+\s+.*\n?", "", text, flags=re.MULTILINE)  # headers
    text = re.sub(r"\*\*\[?[A-Z ]+\]?\*\*\n?", "", text)  # **[HOOK]** style labels
    text = re.sub(r"\*\*(.+?)\*\*", r"\1", text)  # **bold**
    text = re.sub(r"\*(.+?)\*", r"\1", text)  # *italic*
    text = re.sub(r"^[-*]\s+", "", text, flags=re.MULTILINE)  # bullet points
    text = re.sub(r"\n{2,}", " ", text)  # collapse multiple newlines
    return text.strip()


def _build_prompt(skill: dict) -> str:
    track = skill.get("_track", "mover")
    track_label = "NEW THIS WEEK" if track == "rocket" else "TOP MOVER"
    cold = skill.get("_cold_start", False)
    template = USER_TEMPLATE_COLD if cold else USER_TEMPLATE_VELOCITY

    return template.format(
        display_name=skill.get("display_name", "Unknown"),
        author=skill.get("author", "Unknown"),
        summary=skill.get("summary") or "No description.",
        content=(skill.get("content") or "No documentation available.")[:3000],
        track=track_label,
    )


def generate_scripts(
    harvested: list[dict],
    api_key: str,
    model: str = "claude-haiku-4-5-20251001",
) -> list[dict]:
    """
    Generate YouTube segment for each harvested skill.
    Returns list with 'script' field added.
    """
    client = anthropic.Anthropic([REDACTED_SECRET])
    results = []

    for i, skill in enumerate(harvested, 1):
        name = skill.get("display_name", skill.get("slug", "?"))
        print(f"[SCRIPT] {i}/{len(harvested)} Generating segment for {name}...")

        try:
            message = client.messages.create(
                model=model,
                max_tokens=150,
                system=SYSTEM_PROMPT,
                messages=[{"role": "user", "content": _build_prompt(skill)}],
            )
            script = _strip_markdown(message.content[0].text)
        except anthropic.APIError as e:
            print(f"  [WARN] API error for {name}: {e}")
            script = f"[Script generation failed: {e}]"

        skill["script"] = script
        results.append(skill)
        print(f"  Script: {script[:80]}...")

    return results


def render_markdown(
    movers: list[dict],
    rockets: list[dict] | None = None,
    week_label: str = "",
    catalog_history: list[dict] | None = None,
    project_history: list[dict] | None = None,
) -> str:
    """Render the final report with MOVERS and ROCKETS sections."""
    rockets = rockets or []

    # Determine if any movers have real velocity data
    has_velocity = any(r.get("_snapshots_used", 0) > 1 for r in movers)

    lines = [
        f"# OpenClaw Skills Weekly{' — ' + week_label if week_label else ''}",
        "",
    ]

    # --- Ecosystem Overview ---
    if catalog_history or project_history:
        lines += ["## Ecosystem Pulse", ""]

        if catalog_history and len(catalog_history) >= 1:
            latest = catalog_history[-1]
            lines.append(
                f"**{latest.get('total_skills', 0):,}** skills on ClawHub | "
                f"**{latest.get('total_downloads', 0):,}** total downloads | "
                f"**{latest.get('total_installs', 0):,}** total installs | "
                f"**{latest.get('total_stars', 0):,}** total stars"
            )
            if len(catalog_history) >= 2:
                prev = catalog_history[0]
                skill_delta = latest.get("total_skills", 0) - prev.get("total_skills", 0)
                dl_delta = latest.get("total_downloads", 0) - prev.get("total_downloads", 0)
                install_delta = latest.get("total_installs", 0) - prev.get("total_installs", 0)
                lines.append("")
                lines.append(
                    f"*{len(catalog_history)}-day trend:* "
                    f"+{skill_delta:,} skills | "
                    f"+{dl_delta:,} downloads | "
                    f"+{install_delta:,} installs"
                )
            lines.append("")

        if project_history:
            # Group by repo, show latest snapshot
            repos: dict[str, dict] = {}
            for row in project_history:
                repos[row["repo"]] = row  # last row wins (sorted ASC)
            for repo, data in repos.items():
                lines.append(
                    f"- **{repo}**: {data.get('stars', 0):,} stars, "
                    f"{data.get('forks', 0):,} forks, "
                    f"{data.get('open_prs', 0):,} open PRs, "
                    f"{data.get('weekly_commits', 0)} weekly commits"
                    f"{', release ' + data['latest_release'] if data.get('latest_release') else ''}"
                )
            lines.append("")

        lines += ["---", ""]

    # --- MOVERS section ---
    if has_velocity:
        ranking_desc = "*Ranked by 7-day install velocity (installs_all_time delta).*"
    else:
        ranking_desc = "*Ranked by estimated weekly velocity (age-normalized from lifetime data — building daily history for true 7-day deltas).*"

    lines += [
        f"## Top {len(movers)} Trending Skills",
        "",
        ranking_desc,
        "",
    ]

    for i, r in enumerate(movers, 1):
        snaps = r.get("_snapshots_used", 0)
        cold = r.get("_cold_start", snaps <= 1)
        author = r.get("author", "")
        author_str = f"**Author:** {author} | " if author else ""

        if cold:
            hist = "estimated"
            pct_str = "est."
            growth_label = "Est. weekly"
        else:
            hist = f"{snaps}d data"
            pct_str = f"{r.get('_pct_increase', 0.0):.1f}%"
            growth_label = "7d Growth"

        lines += [
            f"### #{i} [{r.get('display_name', r.get('slug', '?'))}]({r.get('clawhub_url', '')})",
            "",
            (
                f"{author_str}"
                f"**Downloads:** {r.get('downloads', 0):,} | "
                f"**Installs:** {r.get('installs_current', 0):,} (all-time: {r.get('installs_all_time', 0):,}) | "
                f"**Stars:** {r.get('stars', 0)} | "
                f"**{growth_label}:** +{r.get('_installs_delta', 0):,} ({pct_str}) | "
                f"**Score:** {r.get('_score', 0.0):.4f} | "
                f"*{hist}*"
            ),
            "",
            f"> {r.get('summary', '')}" if r.get("summary") else "",
            "",
            "**YouTube Script**",
            "",
            f"_{r.get('script', '')}_",
            "",
            "---",
            "",
        ]

    # --- ROCKETS section (new skills) ---
    if rockets:
        lines += [
            "## New This Week",
            "",
            "*Brand new skills (<30 days old) showing early traction.*",
            "",
        ]
        for i, r in enumerate(rockets, 1):
            age = ""
            if r.get("created_at"):
                age_d = (datetime.now(timezone.utc) - datetime.fromtimestamp(r["created_at"] / 1000, tz=timezone.utc)).days
                age = f" | *{age_d} days old*"
            author = r.get("author", "")
            author_str = f"**Author:** {author} | " if author else ""
            lines += [
                f"### [{r.get('display_name', r.get('slug', '?'))}]({r.get('clawhub_url', '')})",
                "",
                (
                    f"{author_str}"
                    f"**Downloads:** {r.get('downloads', 0):,} | "
                    f"**Installs:** {r.get('installs_current', 0):,} | "
                    f"**Stars:** {r.get('stars', 0)} | "
                    f"**Score:** {r.get('_score', 0.0):.4f}"
                    f"{age}"
                ),
                "",
                f"> {r.get('summary', '')}" if r.get("summary") else "",
                "",
                "**YouTube Script**",
                "",
                f"_{r.get('script', '')}_",
                "",
                "---",
                "",
            ]

    # --- Community signals section ---
    signals = community_signals.load_signals()
    if signals:
        lines.append(community_signals.render_community_section(signals))

    return "\n".join(lines)


def render_video_script(
    movers: list[dict],
    rockets: list[dict] | None = None,
    episode_num: int = 1,
    week_label: str = "",
) -> str:
    """
    Render a voice-ready video script in GitHubAwesome style.

    Plain text, no markdown. Ready to read aloud or feed to TTS.
    Structure: cold open → movers → rockets → hard stop (no outro).
    """
    rockets = rockets or []
    total_skills = len(movers) + len(rockets)

    lines = []

    # --- Cold open (GitHubAwesome style) ---
    lines += [
        f"[COLD OPEN]",
        "",
        f"It is time for OpenClaw Skills Weekly, episode number {episode_num}, "
        f"featuring {total_skills} of the AI agent skills trending on ClawHub right now.",
        "",
        "",
    ]

    # --- Movers section ---
    if movers:
        lines += [
            "[TOP MOVERS]",
            "",
        ]
        for i, r in enumerate(movers, 1):
            name = r.get("display_name", r.get("slug", "?"))
            script = r.get("script", "")
            lines += [
                f"[{i}] {name}",
                "",
                script,
                "",
                "",
            ]

    # --- Rockets section ---
    if rockets:
        lines += [
            "[NEW THIS WEEK]",
            "",
        ]
        for i, r in enumerate(rockets, 1):
            name = r.get("display_name", r.get("slug", "?"))
            script = r.get("script", "")
            lines += [
                f"[NEW {i}] {name}",
                "",
                script,
                "",
                "",
            ]

    # --- No outro — last item ends the episode (GitHubAwesome style) ---
    lines += [
        "[END]",
    ]

    return "\n".join(lines)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)