Skill: bibaofeng/last30days-aisa (version 1.0.4)

--- name: last30days version: "1.0.4" description: "Research the last 30 days across Reddit, X/Twitter, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and web search. Use when: you need recent social research, company updates, person profiles, competitor comparisons, launch reactions, or trend scans. Supports AISA-powered planning, clustering, reranking, and JSON output." argument-hint: "last30days OpenAI Agents SDK, last30days Peter Steinberger, last30days OpenClaw" allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch homepage: https://github.com/AIsa-team/agent-skills repository: https://github.com/AIsa-team/agent-skills author: mvanhorn license: MIT user-invocable: true metadata: openclaw: emoji: "📰" requires: env: - AISA_API_KEY bins: - python3 - bash primaryEnv: AISA_API_KEY files: - "scripts/*" --- # last30days Research recent evidence across social platforms, community forums, prediction markets, GitHub, and grounded web results, then merge everything into one brief. ## When to use - Use when you need a last-30-days research brief on a person, company, product, market, tool, or trend. - Use when you want a recent competitor comparison, launch reaction summary, creator/community sentiment scan, or shipping update. - Use when you want structured JSON with `query_plan`, `ranked_candidates`, `clusters`, and `items_by_source`. ## When NOT to use - Do not use for timeless encyclopedia questions with no recent evidence requirement. - Do not use when you need only one official source and do not want social/community signals. ## Capabilities - AISA-hosted planning, reranking, synthesis, grounded web search, X/Twitter search, YouTube search, and Polymarket search. - Public Reddit and Hacker News retrieval with fail-soft behavior. - Official GitHub API search when `GH_TOKEN` or `GITHUB_TOKEN` is available. - Hosted discovery for TikTok, Instagram, Threads, and Pinterest when enabled in runtime config. ## Setup - `AISA_API_KEY` is the main hosted credential. - `GH_TOKEN` or `GITHUB_TOKEN` is optional for GitHub search only. - Python `3.12+` is required. ```bash for py in /usr/local/python3.12/bin/python3.12 python3.14 python3.13 python3.12 python3; do command -v "$py" >/dev/null 2>&1 || continue "$py" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)' || continue LAST30DAYS_PYTHON="$py" break done ``` ## Quick Reference ```bash bash "${SKILL_ROOT}/scripts/run-last30days.sh" "$ARGUMENTS" --emit=compact "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=json "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --quick "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --deep "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --search=reddit,x,grounding "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" --diagnose ``` ## Inputs And Outputs - Input: a topic or comparison query such as `OpenAI Agents SDK`, `OpenClaw vs Codex`, or `Peter Steinberger`. - Output: synthesized research plus `provider_runtime`, `query_plan`, `ranked_candidates`, `clusters`, and `items_by_source`. ## Example Queries - `last30days OpenAI Agents SDK` - `last30days Peter Steinberger` - `last30days OpenClaw vs Codex` - `last30days Kanye West --quick`

path: pyproject.toml
content: [project]
name = "last30days-skill"
version = "1.0.2"
description = "AISA-only multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
  "requests>=2.32,<3",
]

[dependency-groups]
dev = [
  "pytest>=9,<10",
  "pytest-cov>=7,<8",
]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = [
  "-q",
  "--tb=short",
]

[tool.coverage.run]
branch = true
source = ["scripts", "tests"]
omit = [
  "scripts/lib/vendor/*",
  "dist/*",
]

[tool.coverage.report]
skip_empty = true
show_missing = true
omit = [
  "scripts/lib/vendor/*",
  "dist/*",
]



path: README.md
content: # last30days

English ClawHub publish bundle for `last30days`.

Included:

- runtime scripts
- `SKILL.md`
- `.codex-plugin/plugin.json`
- license and package metadata

Excluded on purpose:

- tests
- historical docs
- hooks
- fixtures
- Gemini-specific extension metadata

Runtime summary:

- `AISA_API_KEY` powers hosted planning, reranking, synthesis, X/Twitter, YouTube, Polymarket, and grounded web search.
- Reddit and Hacker News use public paths.
- GitHub stays on the official GitHub API path and may need `GH_TOKEN` or `GITHUB_TOKEN`.



path: scripts/briefing.py
content: #!/usr/bin/env python3
"""Morning briefing generator for last30days.

Synthesizes accumulated findings into formatted briefings.
The Python script collects the data; the agent (via SKILL.md) does the
beautiful synthesis. This script provides the structured data.

Usage:
    bash scripts/run-briefing.sh generate              # Daily briefing data
    bash scripts/run-briefing.sh generate --weekly     # Weekly digest data
    bash scripts/run-briefing.sh show [--date DATE]    # Show saved briefing
"""

import argparse
import json
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

import store

BRIEFS_DIR = Path.home() / ".local" / "share" / "last30days" / "briefs"


def _parse_sqlite_utc_timestamp(value: str) -> datetime:
    return datetime.strptime(value, "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)


def generate_daily(since: str = None) -> dict:
    """Generate daily briefing data.

    Returns structured data for the agent to synthesize into a beautiful briefing.
    """
    store.init_db()
    topics = store.list_topics()

    if not topics:
        return {
            "status": "no_topics",
            "message": "No watchlist topics yet. Add one with: last30days watch add \"your topic\"",
        }

    enabled = [t for t in topics if t["enabled"]]
    if not enabled:
        return {
            "status": "no_enabled",
            "message": "All topics are paused. Enable a topic to generate briefings.",
        }

    # Default: findings since yesterday
    if not since:
        since = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")

    briefing_topics = []
    total_new = 0

    for topic in enabled:
        findings = store.get_new_findings(topic["id"], since)
        last_run = topic.get("last_run")
        last_status = topic.get("last_status", "unknown")

        # Calculate staleness
        stale = False
        hours_ago = None
        if last_run:
            try:
                run_dt = _parse_sqlite_utc_timestamp(last_run)
                hours_ago = (datetime.now(timezone.utc) - run_dt).total_seconds() / 3600
                stale = hours_ago > 36  # Stale if > 36 hours
            except (ValueError, TypeError):
                stale = True

        topic_data = {
            "name": topic["name"],
            "findings": findings,
            "new_count": len(findings),
            "last_run": last_run,
            "last_status": last_status,
            "stale": stale,
            "hours_ago": round(hours_ago, 1) if hours_ago else None,
        }

        # Extract top finding by engagement
        if findings:
            top = max(findings, key=lambda f: f.get("engagement_score", 0))
            topic_data["top_finding"] = {
                "title": top.get("source_title", ""),
                "source": top.get("source", ""),
                "author": top.get("author", ""),
                "engagement": top.get("engagement_score", 0),
                "content": top.get("content", "")[:300],
            }

        briefing_topics.append(topic_data)
        total_new += len(findings)

    # Cost info
    daily_cost = store.get_daily_cost()
    budget = float(store.get_setting("daily_budget", "5.00"))

    # Find the single top finding across all topics (for TL;DR)
    all_findings = []
    for t in briefing_topics:
        for f in t["findings"]:
            f["_topic"] = t["name"]
            all_findings.append(f)

    top_overall = None
    if all_findings:
        top_overall = max(all_findings, key=lambda f: f.get("engagement_score", 0))

    result = {
        "status": "ok",
        "date": datetime.now().strftime("%Y-%m-%d"),
        "since": since,
        "topics": briefing_topics,
        "total_new": total_new,
        "total_topics": len(briefing_topics),
        "top_finding": {
            "title": top_overall.get("source_title", ""),
            "topic": top_overall.get("_topic", ""),
            "engagement": top_overall.get("engagement_score", 0),
        } if top_overall else None,
        "cost": {
            "daily": daily_cost,
            "budget": budget,
        },
        "failed_topics": [
            t["name"] for t in briefing_topics if t["last_status"] == "failed"
        ],
    }

    # Save briefing data
    _save_briefing(result)

    return result


def generate_weekly() -> dict:
    """Generate weekly digest data with trend analysis."""
    store.init_db()

    week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
    two_weeks_ago = (datetime.now() - timedelta(days=14)).strftime("%Y-%m-%d")

    topics = store.list_topics()
    if not topics:
        return {"status": "no_topics", "message": "No watchlist topics."}

    weekly_topics = []

    for topic in topics:
        if not topic["enabled"]:
            continue

        # This week's findings
        this_week = store.get_new_findings(topic["id"], week_ago)

        # Last week's findings (for comparison)
        conn = store._connect()
        try:
            last_week_rows = conn.execute(
                """SELECT * FROM findings
                   WHERE topic_id = ? AND first_seen >= ? AND first_seen < ? AND dismissed = 0
                   ORDER BY engagement_score DESC""",
                (topic["id"], two_weeks_ago, week_ago),
            ).fetchall()
            last_week = [dict(r) for r in last_week_rows]
        finally:
            conn.close()

        this_engagement = sum(f.get("engagement_score", 0) for f in this_week)
        last_engagement = sum(f.get("engagement_score", 0) for f in last_week)

        # Trend calculation
        if last_engagement > 0:
            engagement_change = ((this_engagement - last_engagement) / last_engagement) * 100
        else:
            engagement_change = 100 if this_engagement > 0 else 0

        weekly_topics.append({
            "name": topic["name"],
            "this_week_count": len(this_week),
            "last_week_count": len(last_week),
            "this_week_engagement": this_engagement,
            "last_week_engagement": last_engagement,
            "engagement_change_pct": round(engagement_change, 1),
            "top_findings": this_week[:5],  # Top 5 by engagement (already sorted)
        })

    result = {
        "status": "ok",
        "type": "weekly",
        "week_of": week_ago,
        "topics": weekly_topics,
    }

    _save_briefing(result, suffix="-weekly")

    return result


def show_briefing(date: str = None) -> dict:
    """Load a saved briefing by date."""
    if not date:
        date = datetime.now().strftime("%Y-%m-%d")

    path = BRIEFS_DIR / f"{date}.json"
    if not path.exists():
        # Try weekly
        path = BRIEFS_DIR / f"{date}-weekly.json"

    if not path.exists():
        return {"status": "not_found", "message": f"No briefing found for {date}."}

    with open(path, encoding="utf-8") as f:
        return json.load(f)


def _save_briefing(data: dict, suffix: str = ""):
    """Save briefing data to local archive."""
    BRIEFS_DIR.mkdir(parents=True, exist_ok=True)
    date = datetime.now().strftime("%Y-%m-%d")
    path = BRIEFS_DIR / f"{date}{suffix}.json"
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, default=str)


def main():
    parser = argparse.ArgumentParser(description="Generate last30days briefings")
    sub = parser.add_subparsers(dest="command")

    # generate
    g = sub.add_parser("generate", help="Generate a briefing")
    g.add_argument("--weekly", action="store_true", help="Weekly digest")
    g.add_argument("--since", help="Findings since date (YYYY-MM-DD)")

    # show
    s = sub.add_parser("show", help="Show a saved briefing")
    s.add_argument("--date", help="Date (YYYY-MM-DD, default: today)")

    args = parser.parse_args()

    if args.command == "generate":
        if args.weekly:
            result = generate_weekly()
        else:
            result = generate_daily(since=args.since)
        print(json.dumps(result, indent=2, default=str))

    elif args.command == "show":
        result = show_briefing(date=args.date)
        print(json.dumps(result, indent=2, default=str))

    else:
        parser.print_help()
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/compare.sh
content: #!/bin/bash
# Safe local comparison helper for last30days.
# Usage: bash scripts/compare.sh "Kanye West"

set -euo pipefail

if [ $# -eq 0 ]; then
  echo "Usage: bash scripts/compare.sh <topic>"
  echo "  Example: bash scripts/compare.sh Kevin Rose"
  exit 1
fi

TOPIC="$*"
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
OUT_DIR="${LAST30DAYS_COMPARE_DIR:-$ROOT/.last30days-compare}"

mkdir -p "$OUT_DIR"

echo "=============================================="
echo " Local compare: $TOPIC"
echo " Output dir: $OUT_DIR"
echo "=============================================="
echo ""

echo "[1/2] Running quick profile..."
bash "$ROOT/scripts/run-last30days.sh" "$TOPIC" --emit=json --quick > "$OUT_DIR/quick.json"
echo "  ✓ Wrote $OUT_DIR/quick.json"
echo ""

echo "[2/2] Running deep profile..."
bash "$ROOT/scripts/run-last30days.sh" "$TOPIC" --emit=json --deep > "$OUT_DIR/deep.json"
echo "  ✓ Wrote $OUT_DIR/deep.json"
echo ""

echo "Compare these files:"
echo "  $OUT_DIR/quick.json"
echo "  $OUT_DIR/deep.json"


path: scripts/dev-python.sh
content: #!/usr/bin/env bash
set -euo pipefail

for py in /usr/local/python3.12/bin/python3.12 python3.14 python3.13 python3.12 python3; do
  command -v "$py" >/dev/null 2>&1 || continue
  "$py" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)' >/dev/null 2>&1 || continue
  printf '%s\n' "$py"
  exit 0
done

echo "ERROR: last30days requires Python 3.12+." >&2
exit 1


path: scripts/evaluate_search_quality.py
content: #!/usr/bin/env python3
"""Compare two last30days revisions on the v3 ranked candidate output.

Preferred entrypoint:
    bash scripts/run-evaluate.sh ...
"""

from __future__ import annotations

import argparse
import json
import math
import os
import subprocess
import sys
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

sys.path.insert(0, str(Path(__file__).parent))

from lib import aisa
from lib import env as envlib
from lib import providers
from lib import schema


REPO_ROOT = Path(__file__).resolve().parent.parent
EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json"


def _load_default_topics() -> list[tuple[str, str]]:
    if EVAL_TOPICS_FILE.exists():
        rows = json.loads(EVAL_TOPICS_FILE.read_text())
        return [(row["topic"], row["query_type"]) for row in rows]
    return [
        ("nano banana pro prompting", "product"),
        ("codex vs claude code", "comparison"),
        ("openclaw vs nanoclaw vs ironclaw", "comparison"),
        ("anthropic odds", "prediction"),
        ("kanye west", "breaking_news"),
        ("remotion animations for Claude Code", "how_to"),
    ]


DEFAULT_TOPICS = _load_default_topics()
DEFAULT_SEARCH = ""
DEFAULT_JUDGE_MODEL = providers.AISA_DEFAULT


def stable_item_key(item: dict[str, Any]) -> str:
    return str(item.get("candidate_id") or item.get("url") or item.get("title") or "")


def row_sources(row: dict[str, Any]) -> list[str]:
    candidate = schema.candidate_from_dict(row)
    return schema.candidate_sources(candidate)


def row_best_date(row: dict[str, Any]) -> str | None:
    candidate = schema.candidate_from_dict(row)
    return schema.candidate_best_published_at(candidate)


V2_SOURCE_KEYS = [
    ("reddit", "title"),
    ("x", "text"),
    ("youtube", "title"),
    ("tiktok", "text"),
    ("instagram", "text"),
    ("hackernews", "title"),
    ("polymarket", "question"),
    ("web", "title"),
]


def build_ranked_items(report: dict[str, Any], limit: int) -> list[dict[str, Any]]:
    # v3 format: ranked_candidates list
    if report.get("ranked_candidates"):
        ranked = []
        for row in report["ranked_candidates"][:limit]:
            candidate_sources = row_sources(row)
            ranked.append({
                "key": stable_item_key(row),
                "source": ", ".join(candidate_sources),
                "sources": candidate_sources,
                "url": str(row.get("url") or ""),
                "text": str(row.get("title") or ""),
                "date": row_best_date(row),
                "score": float(row.get("final_score") or 0.0),
            })
        return ranked

    # v2 format: per-source lists (reddit, x, youtube, etc.)
    all_items = []
    for source_key, text_field in V2_SOURCE_KEYS:
        for item in report.get(source_key) or []:
            if not isinstance(item, dict):
                continue
            all_items.append({
                "key": str(item.get("url") or item.get("id") or item.get(text_field) or ""),
                "source": source_key,
                "sources": [source_key],
                "url": str(item.get("url") or ""),
                "text": str(item.get(text_field) or item.get("title") or ""),
                "date": item.get("date"),
                "score": float(item.get("score") or 0.0),
            })
    all_items.sort(key=lambda x: x["score"], reverse=True)
    return all_items[:limit]


def source_sets(report: dict[str, Any], limit: int) -> dict[str, set[str]]:
    grouped: dict[str, set[str]] = {}
    for item in build_ranked_items(report, limit):
        for source in item["sources"]:
            grouped.setdefault(source, set()).add(item["key"])
    return grouped


def jaccard(left: set[str], right: set[str]) -> float:
    if not left and not right:
        return 1.0
    union = left | right
    if not union:
        return 1.0
    return len(left & right) / len(union)


def retention(left: set[str], right: set[str]) -> float:
    if not left:
        return 1.0
    return len(left & right) / len(left)


def precision_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int) -> float:
    top = ranking[:k]
    if not top:
        return 0.0
    return sum(1 for item in top if judgments.get(item["key"], 0) >= 2) / len(top)


def ndcg_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int, judged_pool: list[dict[str, Any]]) -> float:
    top = ranking[:k]
    if not top:
        return 0.0

    def dcg(grades: list[int]) -> float:
        total = 0.0
        for index, grade in enumerate(grades, start=1):
            total += (2**grade - 1) / math.log2(index + 1)
        return total

    actual = [judgments.get(item["key"], 0) for item in top]
    ideal = sorted((judgments.get(item["key"], 0) for item in judged_pool), reverse=True)[: len(top)]
    ideal_score = dcg(ideal)
    if ideal_score == 0:
        return 0.0
    return dcg(actual) / ideal_score


def source_coverage_recall(ranking: list[dict[str, Any]], judged_pool: list[dict[str, Any]], judgments: dict[str, int]) -> float:
    good_sources = {
        source
        for item in judged_pool
        if judgments.get(item["key"], 0) >= 2
        for source in item["sources"]
    }
    if not good_sources:
        return 1.0
    hit_sources = {
        source
        for item in ranking
        if judgments.get(item["key"], 0) >= 2
        for source in item["sources"]
    }
    return len(hit_sources & good_sources) / len(good_sources)


def resolve_judge_api_key(config: dict[str, Any]) -> str | None:
    return os.environ.get("AISA_API_KEY") or config.get("AISA_API_KEY")


def extract_judge_text(payload: dict[str, Any]) -> str:
    text = providers.extract_openai_text(payload)
    if text:
        return text
    raise ValueError("Judge response did not contain text.")


def call_aisa_judge(api_key: str, model: str, prompt: str) -> dict[str, Any]:
    try:
        payload = aisa.chat_completion(
            api_key,
            model,
            prompt,
            response_mime_type="application/json",
        )
    except Exception as exc:
        raise RuntimeError(f"AISA judge request failed: {exc}") from exc
    return json.loads(extract_judge_text(payload))


def build_judge_prompt(topic: str, query_type: str, items: list[dict[str, Any]]) -> str:
    item_lines = []
    for item in items:
        item_lines.append(
            "\n".join([
                f"- id: {item['key']}",
                f"  source: {item['source']}",
                f"  title: {item['text'][:220]}",
                f"  url: {item['url']}",
                f"  date: {item.get('date') or 'unknown'}",
            ])
        )
    return f"""
Judge search-result relevance for a last-30-days research tool.

Topic: {topic}
Query type: {query_type}

Score each item on this 0-3 scale:
- 0 = off-topic or clearly bad
- 1 = weak or tangential
- 2 = relevant and useful
- 3 = highly relevant, one of the best results

Return JSON only:
{{
  "judgments": [
    {{"id": "ITEM_ID", "grade": 0}}
  ]
}}

Items:
{chr(10).join(item_lines)}
""".strip()


def get_judgments(
    *,
    output_dir: Path,
    slug: str,
    topic: str,
    query_type: str,
    items: list[dict[str, Any]],
    judge_model: str,
    judge_api_key: str | None,
) -> dict[str, int]:
    cache_file = output_dir / "judgments" / f"{slug}.json"
    cache_file.parent.mkdir(parents=True, exist_ok=True)
    if cache_file.exists():
        payload = json.loads(cache_file.read_text())
        return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}
    if not judge_api_key or not items:
        return {}
    payload = call_aisa_judge(judge_api_key, judge_model, build_judge_prompt(topic, query_type, items))
    cache_file.write_text(json.dumps(payload, indent=2))
    return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}


def create_eval_env() -> dict[str, str]:
    config = envlib.get_config()
    passthrough = {
        "PATH": os.environ.get("PATH", ""),
        "LANG": os.environ.get("LANG", "en_US.UTF-8"),
        "LC_ALL": os.environ.get("LC_ALL", ""),
        "TMPDIR": os.environ.get("TMPDIR", ""),
        "PYTHONUTF8": "1",
        "LAST30DAYS_CONFIG_DIR": "",
    }
    for key in ("AISA_API_KEY",):
        value = os.environ.get(key) or config.get(key)
        if value:
            passthrough[key] = value
    return passthrough


def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]:
    cmd = [sys.executable, "scripts/last30days.py", topic, "--emit=json"]
    if search:
        cmd.extend(["--search", search])
    if quick:
        cmd.append("--quick")
    if mock:
        cmd.append("--mock")
    result = subprocess.run(
        cmd,
        cwd=repo_dir,
        env=env,
        capture_output=True,
        text=True,
        timeout=timeout_seconds,
        check=False,
    )
    if result.returncode != 0:
        raise RuntimeError(f"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\n{result.stderr.strip()}")
    return json.loads(result.stdout)


def create_worktree(rev: str) -> Path:
    worktree_dir = Path(tempfile.mkdtemp(prefix="last30days-eval-"))
    subprocess.run(
        ["git", "worktree", "add", "--detach", str(worktree_dir), rev],
        cwd=REPO_ROOT,
        check=True,
        capture_output=True,
        text=True,
    )
    return worktree_dir


def resolve_repo_dir(label: str) -> tuple[Path, bool]:
    """Resolve a benchmark label into a repo directory and whether it is temporary."""
    if label == "WORKTREE":
        return REPO_ROOT, False
    return create_worktree(label), True


def remove_worktree(path: Path) -> None:
    subprocess.run(
        ["git", "worktree", "remove", "--force", str(path)],
        cwd=REPO_ROOT,
        check=False,
        capture_output=True,
        text=True,
    )
    try:
        os.rmdir(path)
    except OSError:
        pass


def summarize_topic(topic: str, query_type: str, baseline_report: dict[str, Any], candidate_report: dict[str, Any], judgments: dict[str, int], judged_pool: list[dict[str, Any]], limit: int) -> dict[str, Any]:
    baseline_ranked = build_ranked_items(baseline_report, limit)
    candidate_ranked = build_ranked_items(candidate_report, limit)
    baseline_sets = source_sets(baseline_report, limit)
    candidate_sets = source_sets(candidate_report, limit)
    overall_left = set().union(*baseline_sets.values()) if baseline_sets else set()
    overall_right = set().union(*candidate_sets.values()) if candidate_sets else set()
    sources = sorted(set(baseline_sets) | set(candidate_sets))
    return {
        "topic": topic,
        "query_type": query_type,
        "baseline": {
            "precision_at_5": precision_at_k(baseline_ranked, judgments, 5),
            "ndcg_at_5": ndcg_at_k(baseline_ranked, judgments, 5, judged_pool),
            "source_coverage_recall": source_coverage_recall(baseline_ranked, judged_pool, judgments),
        },
        "candidate": {
            "precision_at_5": precision_at_k(candidate_ranked, judgments, 5),
            "ndcg_at_5": ndcg_at_k(candidate_ranked, judgments, 5, judged_pool),
            "source_coverage_recall": source_coverage_recall(candidate_ranked, judged_pool, judgments),
        },
        "stability": {
            "overall_jaccard": jaccard(overall_left, overall_right),
            "overall_retention_vs_baseline": retention(overall_left, overall_right),
            "per_source": {
                source: {
                    "baseline_count": len(baseline_sets.get(source, set())),
                    "candidate_count": len(candidate_sets.get(source, set())),
                    "jaccard": jaccard(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
                    "retention_vs_baseline": retention(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
                }
                for source in sources
            },
        },
    }


def write_summary(output_dir: Path, baseline_label: str, candidate_label: str, summaries: list[dict[str, Any]]) -> None:
    output_dir.mkdir(parents=True, exist_ok=True)
    payload = {
        "generated_at": datetime.now().isoformat(timespec="seconds"),
        "baseline": baseline_label,
        "candidate": candidate_label,
        "topics": summaries,
    }
    (output_dir / "metrics.json").write_text(json.dumps(payload, indent=2))

    lines = [
        "# Search Quality Evaluation",
        "",
        f"- Baseline: `{baseline_label}`",
        f"- Candidate: `{candidate_label}`",
        f"- Generated: {payload['generated_at']}",
        "",
        "| Topic | Base P@5 | Cand P@5 | Base nDCG@5 | Cand nDCG@5 | Jaccard | Retention |",
        "|---|---:|---:|---:|---:|---:|---:|",
    ]
    for row in summaries:
        lines.append(
            "| {topic} | {bp:.2f} | {cp:.2f} | {bn:.2f} | {cn:.2f} | {jac:.2f} | {ret:.2f} |".format(
                topic=row["topic"],
                bp=row["baseline"]["precision_at_5"],
                cp=row["candidate"]["precision_at_5"],
                bn=row["baseline"]["ndcg_at_5"],
                cn=row["candidate"]["ndcg_at_5"],
                jac=row["stability"]["overall_jaccard"],
                ret=row["stability"]["overall_retention_vs_baseline"],
            )
        )
    (output_dir / "summary.md").write_text("\n".join(lines) + "\n")


def write_failure_summary(
    output_dir: Path,
    baseline_label: str,
    candidate_label: str,
    summaries: list[dict[str, Any]],
    failures: list[dict[str, Any]],
) -> None:
    write_summary(output_dir, baseline_label, candidate_label, summaries)
    metrics_path = output_dir / "metrics.json"
    payload = json.loads(metrics_path.read_text()) if metrics_path.exists() else {
        "generated_at": datetime.now().isoformat(timespec="seconds"),
        "baseline": baseline_label,
        "candidate": candidate_label,
        "topics": [],
    }
    payload["failures"] = failures
    metrics_path.write_text(json.dumps(payload, indent=2))

    summary_path = output_dir / "summary.md"
    lines = summary_path.read_text().splitlines() if summary_path.exists() else ["# Search Quality Evaluation", ""]
    if failures:
        lines.extend([
            "",
            "## Failures",
            "",
        ])
        for failure in failures:
            lines.append(f"- `{failure['topic']}`: {failure['error']}")
    summary_path.write_text("\n".join(lines).rstrip() + "\n")


def parse_topics_file(path: Path) -> list[tuple[str, str]]:
    rows = json.loads(path.read_text())
    return [(str(row["topic"]), str(row.get("query_type") or "general")) for row in rows]


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Compare two last30days revisions on ranked candidate quality")
    parser.add_argument("--baseline", default="HEAD~1")
    parser.add_argument("--candidate", default="WORKTREE")
    parser.add_argument("--search", default=DEFAULT_SEARCH)
    parser.add_argument("--output-dir", default="tmp/search-quality")
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--timeout", type=int, default=240)
    parser.add_argument("--limit", type=int, default=20)
    parser.add_argument("--mock", action="store_true")
    parser.add_argument("--quick", action="store_true")
    parser.add_argument("--topics-file")
    return parser


def main() -> int:
    args = build_parser().parse_args()
    topics = parse_topics_file(Path(args.topics_file)) if args.topics_file else DEFAULT_TOPICS
    output_dir = Path(args.output_dir).resolve()
    config = envlib.get_config()
    judge_api_key = resolve_judge_api_key(config)
    run_env = create_eval_env()

    baseline_dir, baseline_temp = resolve_repo_dir(args.baseline)
    candidate_dir, candidate_temp = resolve_repo_dir(args.candidate)
    try:
        summaries = []
        failures = []
        for topic, query_type in topics:
            try:
                baseline_report = run_last30days(
                    baseline_dir,
                    topic,
                    search=args.search,
                    timeout_seconds=args.timeout,
                    quick=args.quick,
                    mock=args.mock,
                    env=run_env,
                )
                candidate_report = run_last30days(
                    candidate_dir,
                    topic,
                    search=args.search,
                    timeout_seconds=args.timeout,
                    quick=args.quick,
                    mock=args.mock,
                    env=run_env,
                )
                judged_pool_map = {
                    item["key"]: item
                    for item in build_ranked_items(baseline_report, args.limit) + build_ranked_items(candidate_report, args.limit)
                }
                judged_pool = list(judged_pool_map.values())
                judgments = get_judgments(
                    output_dir=output_dir,
                    slug="".join(char.lower() if char.isalnum() else "-" for char in topic).strip("-"),
                    topic=topic,
                    query_type=query_type,
                    items=judged_pool,
                    judge_model=args.judge_model,
                    judge_api_key=judge_api_key,
                )
                summaries.append(summarize_topic(topic, query_type, baseline_report, candidate_report, judgments, judged_pool, args.limit))
            except Exception as exc:
                failures.append({"topic": topic, "query_type": query_type, "error": str(exc)})
        write_failure_summary(output_dir, args.baseline, args.candidate, summaries, failures)
    finally:
        if baseline_temp:
            remove_worktree(baseline_dir)
        if candidate_temp:
            remove_worktree(candidate_dir)
    result = {"output_dir": str(output_dir), "topics": len(topics), "failures": len(failures)}
    print(json.dumps(result, indent=2))
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/generate-synthesis-inputs.py
content: #!/usr/bin/env python3
"""Legacy compatibility wrapper that renders compact markdown from saved reports."""

from __future__ import annotations

import json
import sys
from pathlib import Path

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import render, schema

JSON_DIR = SCRIPT_DIR.parent / "outputs" / "json"
COMPACT_DIR = SCRIPT_DIR.parent / "outputs" / "compact"


def main() -> int:
    COMPACT_DIR.mkdir(parents=True, exist_ok=True)
    for json_path in sorted(JSON_DIR.glob("*.json")):
        report = schema.report_from_dict(json.loads(json_path.read_text()))
        output = f"# last30days v3.0.0: {report.topic}\n\n" + render.render_compact(report)
        out_path = COMPACT_DIR / f"{json_path.stem}.md"
        out_path.write_text(output)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/last30days.py
content: #!/usr/bin/env python3
# ruff: noqa: E402
"""last30days v1.0.2 CLI."""

from __future__ import annotations

import argparse
import atexit
import json
import os
import re
import signal
import sys
import threading
import time
from pathlib import Path

MIN_PYTHON = (3, 12)


def ensure_supported_python(version_info: tuple[int, int, int] | object | None = None) -> None:
    if version_info is None:
        version_info = sys.version_info
    major, minor, micro = tuple(version_info[:3])
    if (major, minor) >= MIN_PYTHON:
        return
    sys.stderr.write(
        "last30days v1 requires Python 3.12+.\n"
        f"Detected Python {major}.{minor}.{micro}.\n"
        "Use /usr/local/python3.12/bin/python3.12 or another Python 3.12+ interpreter, then rerun this command.\n"
    )
    raise SystemExit(1)


ensure_supported_python()

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import env, pipeline, render, schema, ui

_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()


def register_child_pid(pid: int) -> None:
    with _child_pids_lock:
        _child_pids.add(pid)


def unregister_child_pid(pid: int) -> None:
    with _child_pids_lock:
        _child_pids.discard(pid)


def _cleanup_children() -> None:
    with _child_pids_lock:
        pids = list(_child_pids)
    for pid in pids:
        try:
            os.killpg(os.getpgid(pid), signal.SIGTERM)
        except (ProcessLookupError, PermissionError, OSError):
            continue


atexit.register(_cleanup_children)


def parse_search_flag(raw: str) -> list[str]:
    sources = []
    for source in raw.split(","):
        source = source.strip().lower()
        if not source:
            continue
        normalized = pipeline.SEARCH_ALIAS.get(source, source)
        if normalized not in pipeline.MOCK_AVAILABLE_SOURCES:
            raise SystemExit(f"Unknown search source: {source}")
        if normalized not in sources:
            sources.append(normalized)
    if not sources:
        raise SystemExit("--search requires at least one source.")
    return sources


def slugify(value: str) -> str:
    slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
    return slug or "last30days"


def save_output(report: schema.Report, emit: str, save_dir: str, suffix: str = "") -> Path:
    from datetime import datetime
    path = Path(save_dir).expanduser().resolve()
    path.mkdir(parents=True, exist_ok=True)
    slug = slugify(report.topic)
    extension = "json" if emit == "json" else "md"
    suffix_part = f"-{suffix}" if suffix else ""
    out_path = path / f"{slug}-raw{suffix_part}.{extension}"
    if out_path.exists():
        out_path = path / f"{slug}-raw{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
    # Save a user-requested debug artifact only when --save-dir is provided.
    if emit == "json":
        content = emit_output(report, emit)
    else:
        content = render.render_full(report)
    out_path.write_text(content)
    return out_path


def emit_output(report: schema.Report, emit: str, fun_level: str = "medium") -> str:
    if emit == "json":
        return json.dumps(schema.to_dict(report), indent=2, sort_keys=True)
    if emit in {"compact", "md"}:
        return render.render_compact(report, fun_level=fun_level)
    if emit == "context":
        return render.render_context(report)
    raise SystemExit(f"Unsupported emit mode: {emit}")


def persist_report(report: schema.Report) -> dict[str, int]:
    import store

    store.init_db()
    topic_row = store.add_topic(report.topic)
    topic_id = topic_row["id"]
    source_mode = ",".join(sorted(report.items_by_source)) or "v3"
    run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
    try:
        findings = store.findings_from_report(report)
        counts = store.store_findings(run_id, topic_id, findings)
        store.update_run(
            run_id,
            status="completed",
            findings_new=counts["new"],
            findings_updated=counts["updated"],
        )
        return counts
    except Exception as exc:
        store.update_run(run_id, status="failed", error_message=str(exc)[:500])
        raise


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Research a topic across live social, market, and grounded web sources.")
    parser.add_argument("topic", nargs="*", help="Research topic")
    parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md"])
    parser.add_argument("--search", help="Comma-separated source list")
    parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
    parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
    parser.add_argument("--debug", action="store_true", help="Enable HTTP debug logging")
    parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
    parser.add_argument("--diagnose", action="store_true", help="Print provider and source availability")
    parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
    parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
    parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
    parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
    parser.add_argument("--web-backend", default="auto",
                        choices=["auto", "aisa", "none"],
                        help="Web search backend (default: auto, prefers AISA)")
    parser.add_argument("--deep-research", action="store_true",
                        help="Use the AISA-hosted deep web research path for in-depth analysis.")
    parser.add_argument("--plan", help="JSON query plan (skips internal LLM planner). Can be a JSON string or a file path.")
    parser.add_argument("--save-suffix", help="Suffix for saved output filename (e.g., 'aisa' → kanye-west-raw-aisa.md)")
    parser.add_argument("--subreddits", help="Comma-separated subreddit names to search (e.g., SaaS,Entrepreneur)")
    parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
    parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
    parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
    parser.add_argument("--lookback-days", type=int, default=30, help="Number of days to look back for research (default: 30, watchlist uses 90)")
    parser.add_argument("--auto-resolve", action="store_true",
                        help="Use web search to discover subreddits/handles before planning (for platforms without WebSearch)")
    parser.add_argument("--github-user", help="GitHub username for person-mode search (e.g., steipete)")
    parser.add_argument("--github-repo", help="Comma-separated owner/repo for project-mode search (e.g., openclaw/openclaw,paperclipai/paperclip)")
    return parser


def _missing_sources_for_promo(diag: dict[str, object]) -> str | None:
    available = set(diag.get("available_sources") or [])
    missing = []
    if "reddit" not in available:
        missing.append("reddit")
    if "x" not in available:
        missing.append("x")
    if "grounding" not in available:
        missing.append("web")
    if not missing:
        return None
    if "reddit" in missing and "x" in missing:
        return "both"
    return missing[0]


def _show_runtime_ui(report: schema.Report, progress: ui.ProgressDisplay, diag: dict[str, object]) -> None:
    counts = {source: len(items) for source, items in report.items_by_source.items()}
    display_sources = list(
        dict.fromkeys(
            [
                *report.query_plan.source_weights.keys(),
                *report.items_by_source.keys(),
                *report.errors_by_source.keys(),
            ]
        )
    )
    progress.end_processing()
    progress.show_complete(
        source_counts=counts,
        display_sources=display_sources,
    )
    promo = _missing_sources_for_promo(diag)
    if promo:
        progress.show_promo(promo, diag=diag)


def main() -> int:
    parser = build_parser()
    # Use parse_known_args so setup sub-flags (--device-auth, --github,
    # --openclaw) pass through without argparse hard-exiting.
    args, extra_argv = parser.parse_known_args()
    if args.debug:
        os.environ["LAST30DAYS_DEBUG"] = "1"

    config = env.get_config()

    # Handle setup subcommand
    topic = " ".join(args.topic).strip()
    if topic.lower() == "setup":
        from lib import setup_wizard
        if "--openclaw" in extra_argv:
            results = setup_wizard.run_openclaw_setup(config)
            print(json.dumps(results))
            return 0
        if "--github" in extra_argv:
            results = setup_wizard.run_github_auth()
            print(json.dumps(results))
            return 0
        if "--device-auth" in extra_argv:
            results = setup_wizard.run_full_device_auth()
            print(json.dumps(results))
            return 0
        sys.stderr.write("Running auto-setup...\n")
        results = setup_wizard.run_auto_setup(config)
        setup_wizard.write_setup_config(env.CONFIG_FILE)
        results["env_written"] = True
        sys.stderr.write(setup_wizard.get_setup_status_text(results) + "\n")
        return 0

    requested_sources = parse_search_flag(args.search) if args.search else None
    diag = pipeline.diagnose(config, requested_sources)

    if args.diagnose:
        print(json.dumps(diag, indent=2, sort_keys=True))
        return 0

    if not topic:
        parser.print_usage(sys.stderr)
        return 2

    progress = ui.ProgressDisplay(topic, show_banner=True)
    progress.start_processing()

    depth = "deep" if args.deep else "quick" if args.quick else "default"
    try:
        x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
        subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
        tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
        tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
        ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
        # Parse external plan if provided via --plan flag
        external_plan = None
        if args.plan:
            import json as _json
            plan_str = args.plan
            if os.path.isfile(plan_str):
                plan_str = open(plan_str).read()
            try:
                external_plan = _json.loads(plan_str)
            except _json.JSONDecodeError as exc:
                sys.stderr.write(f"[Planner] Invalid --plan JSON: {exc}\n")

        # Auto-resolve: use web search to discover subreddits/handles before planning.
        # This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
        # without WebSearch (OpenClaw, Codex, raw CLI).
        if args.auto_resolve and not external_plan:
            from lib import resolve
            resolution = resolve.auto_resolve(topic, config)
            if resolution.get("subreddits") and not subreddits:
                subreddits = resolution["subreddits"]
                sys.stderr.write(f"[AutoResolve] Subreddits: {', '.join(subreddits)}\n")
            if resolution.get("x_handle") and not args.x_handle:
                args.x_handle = resolution["x_handle"]
                sys.stderr.write(f"[AutoResolve] X handle: @{args.x_handle}\n")
            if resolution.get("github_user") and not args.github_user:
                args.github_user = resolution["github_user"]
                sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
            if resolution.get("github_repos") and not args.github_repo:
                args.github_repo = ",".join(resolution["github_repos"])
                sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
            if resolution.get("context"):
                # Inject context into external_plan metadata for the planner to use
                if not external_plan:
                    external_plan = None  # planner will use its own, but with context
                # Store context for the planner prompt injection
                config["_auto_resolve_context"] = resolution["context"]
                sys.stderr.write(f"[AutoResolve] Context: {resolution['context'][:80]}...\n")

        github_user = args.github_user.lstrip("@").lower() if args.github_user else None
        github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None

        # --deep-research: keep grounded research on the hosted AISA path
        if args.deep_research:
            if not config.get("AISA_API_KEY"):
                print("Error: --deep-research requires AISA_API_KEY", file=sys.stderr)
                sys.exit(1)
            config["_deep_research"] = True
            include = config.get("INCLUDE_SOURCES") or ""
            if "grounding" not in include.lower():
                config["INCLUDE_SOURCES"] = f"{include},grounding" if include else "grounding"

        stage_start = time.time()
        sys.stderr.write("[last30days] pipeline start\n")
        sys.stderr.flush()
        report = pipeline.run(
            topic=topic,
            config=config,
            depth=depth,
            requested_sources=requested_sources,
            mock=args.mock,
            x_handle=args.x_handle,
            x_related=x_related,
            web_backend=args.web_backend,
            external_plan=external_plan,
            subreddits=subreddits,
            tiktok_hashtags=tiktok_hashtags,
            tiktok_creators=tiktok_creators,
            ig_creators=ig_creators,
            lookback_days=args.lookback_days,
            github_user=github_user,
            github_repos=github_repos,
        )
        sys.stderr.write(
            f"[last30days] pipeline done in {time.time() - stage_start:.2f}s "
            f"clusters={len(report.clusters)} candidates={len(report.ranked_candidates)}\n"
        )
        sys.stderr.flush()
    except Exception as exc:
        progress.end_processing()
        progress.show_error(str(exc))
        raise
    _show_runtime_ui(report, progress, diag)
    if args.store:
        counts = persist_report(report)
        sys.stderr.write(
            f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
        )
        sys.stderr.flush()

    # Show quality nudge if applicable
    try:
        from lib import quality_nudge
        quality = quality_nudge.compute_quality_score(config, {})
        if quality.get("nudge_text"):
            sys.stderr.write(f"\n{quality['nudge_text']}\n")
            sys.stderr.flush()
    except Exception:
        pass

    fun_level = config.get("FUN_LEVEL", "medium").lower()
    rendered = emit_output(report, args.emit, fun_level=fun_level)
    if args.save_dir:
        save_path = save_output(report, args.emit, args.save_dir, suffix=args.save_suffix or "")
        sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
        sys.stderr.flush()
    print(rendered)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/run-briefing.sh
content: #!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PYTHON="$("$ROOT/scripts/dev-python.sh")"

cd "$ROOT"
exec "$PYTHON" "$ROOT/scripts/briefing.py" "$@"


path: scripts/run-evaluate.sh
content: #!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PYTHON="$("$ROOT/scripts/dev-python.sh")"

cd "$ROOT"
exec "$PYTHON" "$ROOT/scripts/evaluate_search_quality.py" "$@"


path: scripts/run-last30days.sh
content: #!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PYTHON="$("$ROOT/scripts/dev-python.sh")"

cd "$ROOT"
exec "$PYTHON" "$ROOT/scripts/last30days.py" "$@"


path: scripts/run-tests.sh
content: #!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PYTHON="$("$ROOT/scripts/dev-python.sh")"

cd "$ROOT"
exec "$PYTHON" -m pytest --capture=no "$@"


path: scripts/run-watchlist.sh
content: #!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PYTHON="$("$ROOT/scripts/dev-python.sh")"

cd "$ROOT"
exec "$PYTHON" "$ROOT/scripts/watchlist.py" "$@"


path: scripts/store.py
content: #!/usr/bin/env python3
"""SQLite research accumulator for last30days.

Stores topics, research runs, and findings with:
- WAL mode for safe concurrent access (cron + user)
- FTS5 full-text search with porter+unicode61 tokenizer
- URL-based dedup with engagement metric updates on re-sighting
- Lightweight schema migrations without external dependencies

Database location: ./.last30days-data/research.db by default,
or LAST30DAYS_DATA_DIR/research.db when explicitly configured.

Preferred wrappers:
- bash scripts/run-last30days.sh
- bash scripts/run-watchlist.sh
- bash scripts/run-briefing.sh
"""

import argparse
import json
import os
import sqlite3
import sys
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Dict, List, Optional

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import schema

DB_DIR = Path(os.environ.get("LAST30DAYS_DATA_DIR") or (Path.cwd() / ".last30days-data"))
DB_PATH = DB_DIR / "research.db"

# Allow override for testing
_db_override = None


def _get_db_path() -> Path:
    return _db_override or DB_PATH


SCHEMA_V1 = """
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA cache_size=-64000;

CREATE TABLE IF NOT EXISTS schema_version (
    version INTEGER PRIMARY KEY,
    applied_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE IF NOT EXISTS topics (
    id INTEGER PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    search_queries TEXT,
    schedule TEXT,
    enabled INTEGER DEFAULT 1,
    created_at TEXT DEFAULT (datetime('now')),
    updated_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE IF NOT EXISTS research_runs (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER REFERENCES topics(id),
    run_date TEXT NOT NULL,
    source_mode TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    token_cost REAL,
    duration_seconds REAL,
    status TEXT DEFAULT 'completed',
    error_message TEXT,
    findings_new INTEGER DEFAULT 0,
    findings_updated INTEGER DEFAULT 0,
    created_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE IF NOT EXISTS findings (
    id INTEGER PRIMARY KEY,
    run_id INTEGER REFERENCES research_runs(id),
    topic_id INTEGER REFERENCES topics(id),
    source TEXT NOT NULL,
    source_url TEXT UNIQUE,
    source_title TEXT,
    author TEXT,
    content TEXT,
    summary TEXT,
    engagement_score REAL,
    relevance_score REAL,
    first_seen TEXT DEFAULT (datetime('now')),
    last_seen TEXT DEFAULT (datetime('now')),
    sighting_count INTEGER DEFAULT 1,
    dismissed INTEGER DEFAULT 0
);

CREATE INDEX IF NOT EXISTS idx_findings_topic ON findings(topic_id, first_seen);
CREATE INDEX IF NOT EXISTS idx_findings_source ON findings(source, topic_id);
CREATE INDEX IF NOT EXISTS idx_findings_url ON findings(source_url);

CREATE VIRTUAL TABLE IF NOT EXISTS findings_fts USING fts5(
    content, summary, source_title, author,
    tokenize='porter unicode61',
    content='findings',
    content_rowid='id'
);

CREATE TRIGGER IF NOT EXISTS findings_ai AFTER INSERT ON findings BEGIN
    INSERT INTO findings_fts(rowid, content, summary, source_title, author)
    VALUES (new.id, new.content, new.summary, new.source_title, new.author);
END;

CREATE TRIGGER IF NOT EXISTS findings_ad AFTER DELETE ON findings BEGIN
    INSERT INTO findings_fts(findings_fts, rowid, content, summary, source_title, author)
    VALUES ('delete', old.id, old.content, old.summary, old.source_title, old.author);
END;

CREATE TRIGGER IF NOT EXISTS findings_au AFTER UPDATE ON findings BEGIN
    INSERT INTO findings_fts(findings_fts, rowid, content, summary, source_title, author)
    VALUES ('delete', old.id, old.content, old.summary, old.source_title, old.author);
    INSERT INTO findings_fts(rowid, content, summary, source_title, author)
    VALUES (new.id, new.content, new.summary, new.source_title, new.author);
END;

CREATE TABLE IF NOT EXISTS settings (
    key TEXT PRIMARY KEY,
    value TEXT,
    updated_at TEXT DEFAULT (datetime('now'))
);
"""

SCHEMA_V1_DEFAULTS = """
INSERT OR IGNORE INTO schema_version (version) VALUES (1);
INSERT OR IGNORE INTO settings (key, value) VALUES ('daily_budget', '5.00');
INSERT OR IGNORE INTO settings (key, value) VALUES ('delivery_channel', '');
INSERT OR IGNORE INTO settings (key, value) VALUES ('delivery_mode', 'announce');
INSERT OR IGNORE INTO settings (key, value) VALUES ('briefing_format', 'concise');
INSERT OR IGNORE INTO settings (key, value) VALUES ('default_schedule', '0 8 * * *');
"""

_UPDATABLE_RUN_COLUMNS = frozenset({
    "source_mode",
    "prompt_tokens",
    "completion_tokens",
    "token_cost",
    "duration_seconds",
    "status",
    "error_message",
    "findings_new",
    "findings_updated",
})

_UPDATABLE_FINDING_COLUMNS = frozenset({
    "source",
    "source_url",
    "source_title",
    "author",
    "content",
    "summary",
    "engagement_score",
    "relevance_score",
    "last_seen",
    "sighting_count",
    "dismissed",
})

# Future migrations keyed by version number
MIGRATIONS: Dict[int, str] = {}


def _connect(db_path: Optional[Path] = None) -> sqlite3.Connection:
    """Open a connection with WAL mode and row factory."""
    path = db_path or _get_db_path()
    conn = sqlite3.connect(str(path))
    conn.row_factory = sqlite3.Row
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("PRAGMA synchronous=NORMAL")
    conn.execute("PRAGMA foreign_keys=ON")
    return conn


def init_db(db_path: Optional[Path] = None) -> Path:
    """Create database and tables if they don't exist. Returns the DB path."""
    path = db_path or _get_db_path()
    path.parent.mkdir(parents=True, exist_ok=True)

    conn = _connect(path)
    try:
        conn.executescript(SCHEMA_V1)
        conn.executescript(SCHEMA_V1_DEFAULTS)
        _run_migrations(conn)
        conn.commit()
    finally:
        conn.close()

    return path


def _run_migrations(conn: sqlite3.Connection):
    """Apply pending schema migrations."""
    current = conn.execute(
        "SELECT MAX(version) FROM schema_version"
    ).fetchone()[0] or 0

    for version in sorted(MIGRATIONS.keys()):
        if version > current:
            conn.executescript(MIGRATIONS[version])
            conn.execute(
                "INSERT INTO schema_version (version) VALUES (?)", (version,)
            )


# --- Topics ---


def add_topic(
    name: str,
    search_queries: Optional[List[str]] = None,
    schedule: str = "0 8 * * *",
) -> Dict[str, Any]:
    """Add a topic to the watchlist. Returns the topic dict."""
    init_db()
    conn = _connect()
    try:
        queries_json = json.dumps(search_queries) if search_queries else None
        conn.execute(
            """INSERT INTO topics (name, search_queries, schedule)
               VALUES (?, ?, ?)
               ON CONFLICT(name) DO UPDATE SET
                   search_queries = excluded.search_queries,
                   schedule = excluded.schedule,
                   updated_at = datetime('now')""",
            (name, queries_json, schedule),
        )
        conn.commit()
        row = conn.execute(
            "SELECT * FROM topics WHERE name = ?", (name,)
        ).fetchone()
        return dict(row)
    finally:
        conn.close()


def remove_topic(name: str) -> bool:
    """Remove a topic from the watchlist. Returns True if found."""
    init_db()
    conn = _connect()
    try:
        row = conn.execute(
            "SELECT id FROM topics WHERE name = ?", (name,)
        ).fetchone()
        if not row:
            return False
        topic_id = row["id"]
        # Delete findings and runs for this topic
        conn.execute("DELETE FROM findings WHERE topic_id = ?", (topic_id,))
        conn.execute("DELETE FROM research_runs WHERE topic_id = ?", (topic_id,))
        conn.execute("DELETE FROM topics WHERE id = ?", (topic_id,))
        conn.commit()
        return True
    finally:
        conn.close()


def list_topics() -> List[Dict[str, Any]]:
    """List all topics with stats."""
    init_db()
    conn = _connect()
    try:
        rows = conn.execute(
            """SELECT t.*,
                      (SELECT COUNT(*) FROM findings WHERE topic_id = t.id) as finding_count,
                      (SELECT MAX(run_date) FROM research_runs WHERE topic_id = t.id) as last_run,
                      (SELECT status FROM research_runs WHERE topic_id = t.id
                       ORDER BY created_at DESC LIMIT 1) as last_status
               FROM topics t
               ORDER BY t.name"""
        ).fetchall()
        return [dict(r) for r in rows]
    finally:
        conn.close()


def get_topic(name: str) -> Optional[Dict[str, Any]]:
    """Get a topic by name."""
    init_db()
    conn = _connect()
    try:
        row = conn.execute(
            "SELECT * FROM topics WHERE name = ?", (name,)
        ).fetchone()
        return dict(row) if row else None
    finally:
        conn.close()


# --- Research Runs ---


def record_run(
    topic_id: int,
    source_mode: str = "both",
    status: str = "completed",
    error_message: Optional[str] = None,
    duration_seconds: float = 0,
    prompt_tokens: int = 0,
    completion_tokens: int = 0,
    token_cost: float = 0,
) -> int:
    """Record a research run. Returns the run ID."""
    conn = _connect()
    try:
        cursor = conn.execute(
            """INSERT INTO research_runs
               (topic_id, run_date, source_mode, status, error_message,
                duration_seconds, prompt_tokens, completion_tokens, token_cost)
               VALUES (?, datetime('now'), ?, ?, ?, ?, ?, ?, ?)""",
            (
                topic_id, source_mode, status, error_message,
                duration_seconds, prompt_tokens, completion_tokens, token_cost,
            ),
        )
        conn.commit()
        return cursor.lastrowid
    finally:
        conn.close()


def update_run(run_id: int, **kwargs):
    """Update a research run's fields."""
    conn = _connect()
    try:
        invalid_columns = sorted(set(kwargs) - _UPDATABLE_RUN_COLUMNS)
        if invalid_columns:
            raise ValueError(
                f"Invalid run update fields: {', '.join(invalid_columns)}"
            )
        sets = ", ".join(f"{k} = ?" for k in kwargs)
        values = list(kwargs.values()) + [run_id]
        conn.execute(f"UPDATE research_runs SET {sets} WHERE id = ?", values)
        conn.commit()
    finally:
        conn.close()


# --- Findings ---


def store_findings(
    run_id: int,
    topic_id: int,
    findings: List[Dict[str, Any]],
) -> Dict[str, int]:
    """Store findings with URL-based dedup. Returns counts of new/updated."""
    conn = _connect()
    new_count = 0
    updated_count = 0

    try:
        for f in findings:
            url = f.get("source_url") or f.get("url")
            if not url:
                continue

            existing = conn.execute(
                "SELECT id, engagement_score, sighting_count FROM findings WHERE source_url = ?",
                (url,),
            ).fetchone()

            if existing:
                # Update engagement and re-sighting info
                new_engagement = f.get("engagement_score", 0)
                conn.execute(
                    """UPDATE findings SET
                           last_seen = datetime('now'),
                           sighting_count = sighting_count + 1,
                           engagement_score = ?,
                           run_id = ?
                       WHERE id = ?""",
                    (
                        max(new_engagement, existing["engagement_score"] or 0),
                        run_id,
                        existing["id"],
                    ),
                )
                updated_count += 1
            else:
                # New finding
                conn.execute(
                    """INSERT INTO findings
                       (run_id, topic_id, source, source_url, source_title,
                        author, content, summary, engagement_score, relevance_score)
                       VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
                    (
                        run_id,
                        topic_id,
                        f.get("source", "unknown"),
                        url,
                        f.get("source_title") or f.get("title", ""),
                        f.get("author", ""),
                        f.get("content") or f.get("text", ""),
                        f.get("summary", ""),
                        f.get("engagement_score", 0),
                        f.get("relevance_score", 0),
                    ),
                )
                new_count += 1

        # Update run stats
        conn.execute(
            "UPDATE research_runs SET findings_new = ?, findings_updated = ? WHERE id = ?",
            (new_count, updated_count, run_id),
        )
        conn.commit()
    finally:
        conn.close()

    return {"new": new_count, "updated": updated_count}


def get_new_findings(
    topic_id: int,
    since: Optional[str] = None,
) -> List[Dict[str, Any]]:
    """Get findings for a topic, optionally since a date."""
    conn = _connect()
    try:
        if since:
            rows = conn.execute(
                """SELECT * FROM findings
                   WHERE topic_id = ? AND first_seen >= ? AND dismissed = 0
                   ORDER BY first_seen DESC""",
                (topic_id, since),
            ).fetchall()
        else:
            rows = conn.execute(
                """SELECT * FROM findings
                   WHERE topic_id = ? AND dismissed = 0
                   ORDER BY first_seen DESC""",
                (topic_id,),
            ).fetchall()
        return [dict(r) for r in rows]
    finally:
        conn.close()


def search_findings(query: str, limit: int = 20) -> List[Dict[str, Any]]:
    """FTS5 search across all findings with BM25 ranking."""
    conn = _connect()
    try:
        rows = conn.execute(
            """SELECT f.*, bm25(findings_fts) as rank, t.name as topic_name
               FROM findings_fts
               JOIN findings f ON f.id = findings_fts.rowid
               LEFT JOIN topics t ON t.id = f.topic_id
               WHERE findings_fts MATCH ?
               ORDER BY rank
               LIMIT ?""",
            (query, limit),
        ).fetchall()
        return [dict(r) for r in rows]
    finally:
        conn.close()


def update_finding(finding_id: int, **kwargs):
    """Update a finding's fields."""
    conn = _connect()
    try:
        invalid_columns = sorted(set(kwargs) - _UPDATABLE_FINDING_COLUMNS)
        if invalid_columns:
            raise ValueError(
                f"Invalid finding update fields: {', '.join(invalid_columns)}"
            )
        sets = ", ".join(f"{k} = ?" for k in kwargs)
        values = list(kwargs.values()) + [finding_id]
        conn.execute(f"UPDATE findings SET {sets} WHERE id = ?", values)
        conn.commit()
    finally:
        conn.close()


def delete_finding(finding_id: int):
    """Delete a finding."""
    conn = _connect()
    try:
        conn.execute("DELETE FROM findings WHERE id = ?", (finding_id,))
        conn.commit()
    finally:
        conn.close()


def dismiss_finding(finding_id: int):
    """Mark a finding as dismissed."""
    update_finding(finding_id, dismissed=1)


# --- Cost Tracking ---


def get_daily_cost(date: Optional[str] = None) -> float:
    """Get total token cost for a given day (default: today)."""
    conn = _connect()
    try:
        if not date:
            date = datetime.now().strftime("%Y-%m-%d")
        row = conn.execute(
            """SELECT COALESCE(SUM(token_cost), 0) as total
               FROM research_runs
               WHERE date(run_date) = date(?)""",
            (date,),
        ).fetchone()
        return row["total"]
    finally:
        conn.close()


# --- Settings ---


def get_setting(key: str, default: Optional[str] = None) -> Optional[str]:
    """Get a setting value."""
    init_db()
    conn = _connect()
    try:
        row = conn.execute(
            "SELECT value FROM settings WHERE key = ?", (key,)
        ).fetchone()
        return row["value"] if row else default
    finally:
        conn.close()


def set_setting(key: str, value: str):
    """Set a setting value."""
    init_db()
    conn = _connect()
    try:
        conn.execute(
            """INSERT INTO settings (key, value, updated_at)
               VALUES (?, ?, datetime('now'))
               ON CONFLICT(key) DO UPDATE SET
                   value = excluded.value,
                   updated_at = datetime('now')""",
            (key, value),
        )
        conn.commit()
    finally:
        conn.close()


# --- Stats ---


def get_stats() -> Dict[str, Any]:
    """Get overall database stats."""
    conn = _connect()
    try:
        topic_count = conn.execute("SELECT COUNT(*) FROM topics WHERE enabled = 1").fetchone()[0]
        finding_count = conn.execute("SELECT COUNT(*) FROM findings").fetchone()[0]

        week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
        runs_7d = conn.execute(
            "SELECT COUNT(*) FROM research_runs WHERE run_date >= ?", (week_ago,)
        ).fetchone()[0]
        successful_7d = conn.execute(
            "SELECT COUNT(*) FROM research_runs WHERE run_date >= ? AND status = 'completed'",
            (week_ago,),
        ).fetchone()[0]
        failed_7d = conn.execute(
            "SELECT COUNT(*) FROM research_runs WHERE run_date >= ? AND status = 'failed'",
            (week_ago,),
        ).fetchone()[0]
        cost_7d = conn.execute(
            "SELECT COALESCE(SUM(token_cost), 0) FROM research_runs WHERE run_date >= ?",
            (week_ago,),
        ).fetchone()[0]

        # Source breakdown
        sources = {}
        for row in conn.execute(
            "SELECT source, COUNT(*) as cnt FROM findings GROUP BY source"
        ).fetchall():
            sources[row["source"]] = row["cnt"]

        db_path = _get_db_path()
        db_size = db_path.stat().st_size if db_path.exists() else 0

        return {
            "topics_active": topic_count,
            "total_findings": finding_count,
            "db_size_bytes": db_size,
            "runs_7d": runs_7d,
            "successful_7d": successful_7d,
            "failed_7d": failed_7d,
            "cost_7d": cost_7d,
            "sources": sources,
            "daily_budget": get_setting("daily_budget", "5.00"),
        }
    finally:
        conn.close()


def get_trending(days: int = 7) -> List[Dict[str, Any]]:
    """Get topics ranked by recent finding activity."""
    conn = _connect()
    try:
        since = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
        rows = conn.execute(
            """SELECT t.name, t.id,
                      COUNT(f.id) as new_findings,
                      COALESCE(SUM(f.engagement_score), 0) as total_engagement
               FROM topics t
               LEFT JOIN findings f ON f.topic_id = t.id AND f.first_seen >= ?
               WHERE t.enabled = 1
               GROUP BY t.id
               ORDER BY new_findings DESC""",
            (since,),
        ).fetchall()
        return [dict(r) for r in rows]
    finally:
        conn.close()


def finding_from_candidate(candidate: schema.Candidate) -> Dict[str, Any]:
    """Convert a ranked candidate into a persisted finding."""
    primary_item = schema.candidate_primary_item(candidate)
    corroborating_sources = [
        source for source in schema.candidate_sources(candidate)
        if source and source != candidate.source
    ]
    summary = candidate.explanation or candidate.snippet or ""
    if corroborating_sources:
        prefix = f"Also seen in: {', '.join(corroborating_sources)}."
        summary = f"{prefix} {summary}".strip()
    body = (
        primary_item.body
        if primary_item and primary_item.body
        else candidate.snippet or candidate.title
    )
    author = primary_item.author if primary_item and primary_item.author else ""
    return {
        "source": candidate.source or "unknown",
        "source_url": candidate.url,
        "source_title": candidate.title,
        "author": author,
        "content": body,
        "summary": summary,
        "engagement_score": candidate.engagement or 0,
        "relevance_score": candidate.final_score or candidate.rerank_score or candidate.local_relevance,
    }


def findings_from_report(
    report: schema.Report,
    *,
    limit: Optional[int] = None,
) -> List[Dict[str, Any]]:
    """Convert report into persisted findings.
    
    Uses ranked candidates (post-rerank) when available for quality scores and explanations.
    Supplements with raw items from items_by_source for HN/PM that didn't rank highly
    but are valuable for watchlist persistence.
    """
    findings = []
    seen_urls = set()
    
    # Phase 1: Process ranked candidates (high-quality data with explanations and corroboration)
    for candidate in report.ranked_candidates:
        finding = finding_from_candidate(candidate)
        findings.append(finding)
        seen_urls.add(candidate.url)
    
    # Phase 2: Add source items not already captured in ranked candidates.
    # This keeps watchlist persistence useful even when ranked_candidates is
    # empty or thin, which is common for smaller or fail-fast runs.
    for source_name, items in (report.items_by_source or {}).items():
        for item in items:
            if item.url in seen_urls:
                continue  # Already captured with rich data
            findings.append({
                "source": source_name,
                "source_url": item.url,
                "source_title": item.title,
                "author": item.author or "",
                "content": item.body or "",
                "summary": item.snippet or (item.body[:500] if item.body else ""),
                "engagement_score": item.engagement_score or 0.0,
                "relevance_score": item.local_relevance or 0.5,
            })
            seen_urls.add(item.url)
    
    # Apply global limit after collecting all findings (fix: was per-source, now global)
    return findings[:limit] if limit is not None else findings


# --- CLI interface ---


def _cli_query(args):
    """Handle CLI query command."""
    topic = get_topic(args.topic)
    if not topic:
        print(json.dumps({"error": f"Topic not found: {args.topic}"}))
        return

    since = None
    if args.since:
        # Parse duration like "7d", "30d"
        days = int(args.since.rstrip("d"))
        since = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")

    findings = get_new_findings(topic["id"], since)
    print(json.dumps({"topic": topic["name"], "findings": findings, "count": len(findings)}, default=str))


def _cli_search(args):
    """Handle CLI search command."""
    results = search_findings(args.query, limit=args.limit)
    print(json.dumps({"query": args.query, "results": results, "count": len(results)}, default=str))


def _cli_trending(args):
    """Handle CLI trending command."""
    results = get_trending(args.days)
    print(json.dumps({"trending": results}, default=str))


def _cli_stats(args):
    """Handle CLI stats command."""
    stats = get_stats()
    print(json.dumps(stats, default=str))


def main():
    parser = argparse.ArgumentParser(description="Query the last30days research database")
    sub = parser.add_subparsers(dest="command")

    # query
    q = sub.add_parser("query", help="Query findings for a topic")
    q.add_argument("topic", help="Topic name")
    q.add_argument("--since", help="Duration like '7d' or '30d'")
    q.set_defaults(func=_cli_query)

    # search
    s = sub.add_parser("search", help="Full-text search across findings")
    s.add_argument("query", help="Search query")
    s.add_argument("--limit", type=int, default=20, help="Max results")
    s.set_defaults(func=_cli_search)

    # trending
    t = sub.add_parser("trending", help="Show trending topics")
    t.add_argument("--days", type=int, default=7, help="Look back N days")
    t.set_defaults(func=_cli_trending)

    # stats
    st = sub.add_parser("stats", help="Show database stats")
    st.set_defaults(func=_cli_stats)

    args = parser.parse_args()
    if not args.command:
        parser.print_help()
        sys.exit(1)

    # Ensure DB exists
    init_db()
    args.func(args)


if __name__ == "__main__":
    main()


path: scripts/sync.sh
content: #!/usr/bin/env bash
# sync.sh - Safe local file sync helper
# Usage: bash scripts/sync.sh /absolute/or/relative/target-dir
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "Usage: bash scripts/sync.sh <target-dir>"
  exit 1
fi

SRC="$(cd "$(dirname "$0")/.." && pwd)"
TARGET="$1"
PYTHON="$("$SRC/scripts/dev-python.sh")"

mkdir -p "$TARGET/scripts/lib"
cp "$SRC/SKILL.md" "$TARGET/SKILL.md"
cp "$SRC/scripts/last30days.py" "$TARGET/scripts/"
cp "$SRC/scripts/watchlist.py" "$TARGET/scripts/"
cp "$SRC/scripts/briefing.py" "$TARGET/scripts/"
cp "$SRC/scripts/store.py" "$TARGET/scripts/"
cp "$SRC/scripts/lib/"*.py "$TARGET/scripts/lib/"

echo "Synced skill files to $TARGET"
(
  cd "$TARGET/scripts" &&
  "$PYTHON" -c "import briefing, store, watchlist; from lib import youtube_yt, render, ui; print('Import check: OK')"
)


path: scripts/test-aisa-planner.py
content: #!/usr/bin/env python3
"""End-to-end AISA planner request probe for last30days v1.0.2.

This script mirrors the current planner -> provider -> AISA Chat path:
pipeline -> planner.plan_query() -> providers.AIsaClient -> aisa.chat_completion()

Usage:
    AISA_API_KEY=... /usr/local/python3.12/bin/python3.12 scripts/test-aisa-planner.py \
      --topic "OpenAI Agents SDK" \
      --depth quick \
      --available x,youtube,grounding,reddit,hackernews,polymarket \
      --requested x,youtube,grounding
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from pathlib import Path


SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import aisa, http, planner, providers  # noqa: E402


def _csv_list(raw: str | None) -> list[str]:
    if not raw:
        return []
    return [item.strip() for item in raw.split(",") if item.strip()]


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Probe the live AISA planner request used by last30days.")
    parser.add_argument("--topic", required=True, help="Research topic passed to the planner.")
    parser.add_argument("--depth", default="quick", choices=["quick", "default", "deep"])
    parser.add_argument(
        "--available",
        default="reddit,x,youtube,hackernews,polymarket,grounding",
        help="Comma-separated available sources as seen by planner.",
    )
    parser.add_argument(
        "--requested",
        default="",
        help="Comma-separated requested sources. Leave empty for auto.",
    )
    parser.add_argument(
        "--model",
        default=os.environ.get("LAST30DAYS_PLANNER_MODEL") or providers.AISA_DEFAULT,
        help="AISA chat model to test.",
    )
    parser.add_argument(
        "--context",
        default="",
        help="Optional web-resolved context appended to the planner prompt.",
    )
    parser.add_argument(
        "--print-prompt",
        action="store_true",
        help="Print the full planner prompt before sending.",
    )
    return parser


def main() -> int:
    args = build_parser().parse_args()
    [REDACTED_SECRET]"AISA_API_KEY", "").strip()
    if not [REDACTED_SECRET]"AISA_API_KEY is required.\n")
        return 2

    available_sources = _csv_list(args.available)
    requested_sources = _csv_list(args.requested) or None

    prompt = planner._build_prompt(
        args.topic,
        available_sources,
        requested_sources,
        args.depth,
    )
    if args.context:
        prompt += f"\n\nCurrent context (from web search): {args.context}"

    payload: dict[str, object] = {
        "model": args.model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "response_format": {"type": "json_object"},
    }

    print("[Planner Probe] endpoint=", aisa.AISA_CHAT_COMPLETIONS_URL, sep="")
    print("[Planner Probe] timeout=", f"{aisa.AISA_CHAT_TIMEOUT}s", sep="")
    print("[Planner Probe] retries=", aisa.AISA_CHAT_RETRIES, sep="")
    print("[Planner Probe] model=", args.model, sep="")
    print("[Planner Probe] topic=", args.topic, sep="")
    print("[Planner Probe] depth=", args.depth, sep="")
    print("[Planner Probe] available_sources=", json.dumps(available_sources, ensure_ascii=False), sep="")
    print(
        "[Planner Probe] requested_sources=",
        json.dumps(requested_sources, ensure_ascii=False),
        sep="",
    )
    print("[Planner Probe] prompt_chars=", len(prompt), sep="")
    print("[Planner Probe] request_payload=")
    print(json.dumps(payload, indent=2, ensure_ascii=False))

    if args.print_prompt:
        print("[Planner Probe] full_prompt=")
        print(prompt)

    start = time.time()
    try:
        response = aisa.chat_completion(
            api_key,
            args.model,
            prompt,
            response_mime_type="application/json",
        )
        elapsed = time.time() - start
        text = providers.extract_openai_text(response)
        parsed = providers.extract_json(text) if text else {}
        print(f"[Planner Probe] elapsed={elapsed:.2f}s")
        print("[Planner Probe] response_text=")
        print(text)
        print("[Planner Probe] parsed_json=")
        print(json.dumps(parsed, indent=2, ensure_ascii=False))
        return 0
    except Exception as exc:  # pragma: no cover - runtime probe
        elapsed = time.time() - start
        error_class = http.classify_error(exc)
        print(f"[Planner Probe] elapsed={elapsed:.2f}s", file=sys.stderr)
        print(
            f"[Planner Probe] failed class={error_class} type={type(exc).__name__}: {exc}",
            file=sys.stderr,
        )
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/test-aisa-provider.py
content: #!/usr/bin/env python3
"""Minimal AISA provider connectivity check for last30days v1.0.2."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path


SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import aisa, http, providers  # noqa: E402


def main() -> int:
    [REDACTED_SECRET]"AISA_API_KEY", "").strip()
    if not [REDACTED_SECRET]"AISA_API_KEY is required.\n")
        return 2

    model = os.environ.get("LAST30DAYS_PLANNER_MODEL") or providers.AISA_DEFAULT
    prompt = os.environ.get(
        "AISA_TEST_PROMPT",
        "Reply with JSON only: {\"ok\": true, \"provider\": \"aisa\"}",
    )

    print(f"[AISA Test] model={model}")
    print(f"[AISA Test] url={aisa.AISA_CHAT_COMPLETIONS_URL}")
    print(
        "[AISA Test] timeout="
        f"{aisa.AISA_CHAT_TIMEOUT}s retries={aisa.AISA_CHAT_RETRIES} "
        f"prompt_chars={len(prompt)}"
    )
    try:
        payload = aisa.chat_completion(
            api_key,
            model,
            prompt,
            response_mime_type="application/json",
        )
        text = providers.extract_openai_text(payload)
        parsed = providers.extract_json(text) if text else {}
        print("[AISA Test] request=ok")
        print(json.dumps({"text": text, "parsed": parsed}, indent=2, ensure_ascii=False))
        return 0
    except Exception as exc:  # pragma: no cover - runtime utility
        error_class = http.classify_error(exc)
        print(
            f"[AISA Test] request=failed class={error_class} "
            f"type={type(exc).__name__}: {exc}",
            file=sys.stderr,
        )
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/test-v1-vs-v2.sh
content: #!/bin/bash
set -euo pipefail

# Safe v1/v2 local comparison harness.
# Runs the same queries against two explicit scripts and stores JSON outputs locally.

if [ $# -lt 2 ]; then
  echo "Usage: bash scripts/test-v1-vs-v2.sh <baseline-script> <candidate-script>"
  exit 1
fi

BASELINE_SCRIPT="$1"
CANDIDATE_SCRIPT="$2"
REPO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
OUT_DIR="$REPO_DIR/.last30days-compare/v1-vs-v2-${TIMESTAMP}"
V1_DIR="$OUT_DIR/v1"
V2_DIR="$OUT_DIR/v2"

mkdir -p "$V1_DIR" "$V2_DIR"

echo "📁 Output directory: $OUT_DIR"
echo ""

QUERIES=(
  "prompting techniques for chatgpt for legal questions"
  "best clawdbot use cases"
  "how to best setup clawdbot"
  "prompting tips for nano banana pro for ios designs"
  "top claude code skills"
  "using ChatGPT to make images of dogs"
  "research best practices for beautiful remotion animation videos in claude code"
  "photorealistic people in nano banana pro"
  "What are the best rap songs lately"
  "what are people saying about DeepSeek R1"
  "best practices for cursor rules files for Cursor"
  "prompt advice for using suno to make killer songs in simple mode"
  "how do I use Codex with Claude Code on same app to make it better"
  "kanye west"
  "howie.ai"
  "open claw"
  "nano banana pro prompting"
)

TYPES=(
  "PROMPTING+TOOL"
  "RECOMMENDATIONS"
  "HOW-TO"
  "PROMPTING+TOOL"
  "RECOMMENDATIONS"
  "GENERAL"
  "PROMPTING"
  "PROMPTING"
  "RECOMMENDATIONS"
  "NEWS"
  "PROMPTING"
  "PROMPTING"
  "HOW-TO"
  "NEWS"
  "GENERAL"
  "GENERAL"
  "PROMPTING"
)

slugify() {
  echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | cut -c1-50
}

run_version() {
  local version="$1"
  local outdir="$2"
  local script_path="$3"
  local total=${#QUERIES[@]}

  echo ""
  echo "=========================================="
  echo "  Running $version — $total queries"
  echo "=========================================="
  echo ""

  for i in "${!QUERIES[@]}"; do
    local query="${QUERIES[$i]}"
    local type="${TYPES[$i]}"
    local slug
    slug=$(slugify "$query")
    local num=$((i + 1))
    local outfile="$outdir/${num}-${slug}.txt"
    local errfile="$outdir/${num}-${slug}.stderr.txt"

    echo "[$version] ($num/$total) $query [$type]"

    local start_time
    start_time=$(date +%s)

    if /usr/local/python3.12/bin/python3.12 "$script_path" \
      "$query" --emit=json --quick \
      > "$outfile" 2>"$errfile"; then
      local end_time
      end_time=$(date +%s)
      local duration=$((end_time - start_time))
      local lines
      lines=$(wc -l < "$outfile")
      echo "  ✅ Done — ${lines} lines, ${duration}s"
    else
      local exit_code=$?
      echo "  ❌ Failed (exit $exit_code)" | tee -a "$outfile"
    fi

    # Brief pause between queries to avoid rate limits
    sleep 3
  done
}

run_version "V1" "$V1_DIR" "$BASELINE_SCRIPT"
run_version "V2" "$V2_DIR" "$CANDIDATE_SCRIPT"

# === Phase 3: Generate summary ===
echo ""
echo "=========================================="
echo "  Generating comparison summary"
echo "=========================================="

SUMMARY="$OUT_DIR/comparison-summary.md"

cat > "$SUMMARY" << EOF
# V1 vs V2 Comparison Results

Generated: $(date)
Output directory: $OUT_DIR

## Output Files

| # | Query | Type | V1 Lines | V2 Lines | V1 Time | V2 Time |
|---|-------|------|----------|----------|---------|---------|
EOF

for i in "${!QUERIES[@]}"; do
  query="${QUERIES[$i]}"
  type="${TYPES[$i]}"
  slug=$(slugify "$query")
  num=$((i + 1))

  v1file="$V1_DIR/${num}-${slug}.txt"
  v2file="$V2_DIR/${num}-${slug}.txt"

  v1lines=$(wc -l < "$v1file" 2>/dev/null || echo "ERR")
  v2lines=$(wc -l < "$v2file" 2>/dev/null || echo "ERR")

  echo "| $num | \`$query\` | $type | $v1lines | $v2lines | — | — |" >> "$SUMMARY"
done

cat >> "$SUMMARY" << 'EOF'

## Quick Check: Key Features

For each query, check these v2 improvements:

- [ ] Query parsing display (`🔍 **{TOPIC}** · {QUERY_TYPE}`)
- [ ] Sparse citations (not every sentence)
- [ ] Bold topic headers in summary
- [ ] Emoji stats tree (`├─ 🟠 Reddit:`)
- [ ] Quality checklist applied to prompts
- [ ] Self-check (research grounding, not generic)

## Scoring Guide

Use the full scoring rubric from:
`docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md`

## Next Step

Have Claude read all 34 output files and generate scored comparison:
```
Read all files in docs/test-results/v1-vs-v2-*/v1/ and v2/
Score each on the 7 dimensions from the test plan
Write the final analysis to docs/test-results/v1-vs-v2-*/analysis.md
```
EOF

echo ""
echo "✅ All done!"
echo ""
echo "📁 Results:  $OUT_DIR"
echo "📊 Summary:  $SUMMARY"
echo "📄 V1 files: $V1_DIR/"
echo "📄 V2 files: $V2_DIR/"
echo ""
echo "To review:"
echo "  open $OUT_DIR"


path: scripts/verify_v3.py
content: #!/usr/bin/env python3
"""Run the v3 verification bundle for last30days."""

from __future__ import annotations

import argparse
import json
import os
import statistics
import subprocess
import sys
import time
from pathlib import Path


REPO_ROOT = Path(__file__).resolve().parent.parent


def resolve_dev_python() -> str:
    for candidate in (
        "/usr/local/python3.12/bin/python3.12",
        "python3.14",
        "python3.13",
        "python3.12",
        sys.executable,
        "python3",
    ):
        try:
            probe = subprocess.run(
                [candidate, "-c", "import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)"],
                cwd=REPO_ROOT,
                text=True,
                capture_output=True,
                check=False,
            )
        except FileNotFoundError:
            continue
        if probe.returncode == 0:
            return candidate
    raise RuntimeError("last30days verification requires Python 3.12+")


PYTHON = resolve_dev_python()
LIVE_SMOKE_TIMEOUT = int(os.environ.get("LAST30DAYS_VERIFY_SMOKE_TIMEOUT", "180"))
MOCK_SMOKE_TIMEOUT = int(os.environ.get("LAST30DAYS_VERIFY_MOCK_TIMEOUT", "120"))

SMOKE_TOPIC = "openclaw skills"
SMOKE_CASES = [
    ("aisa", ["--quick", "--search=grounding,hackernews"]),
    ("aisa", ["--quick", "--search=reddit,hackernews"]),
    ("aisa", ["--quick", "--search=reddit,grounding,hackernews"]),
    ("auto", ["--quick", "--search=reddit,grounding,hackernews"]),
]

LATENCY_TOPICS = [
    "openclaw skills",
    "codex vs claude code",
    "anthropic odds",
]
LATENCY_PROFILES = [
    ("quick", ["--quick", "--search=grounding,hackernews"]),
    ("default", ["--search=grounding,hackernews"]),
    ("deep", ["--deep", "--search=grounding,hackernews"]),
]

CURATED_TESTS = [
    "tests/test_aisa.py",
    "tests/test_env_v3.py",
    "tests/test_providers_v3.py",
    "tests/test_grounding_v3.py",
    "tests/test_setup_openclaw.py",
    "tests/test_setup_wizard.py",
    "tests/test_ui_v3.py",
    "tests/test_quality_nudge.py",
    "tests/test_tiktok.py",
    "tests/test_instagram.py",
    "tests/test_social_sources_v3.py",
    "tests/test_xquik.py",
    "tests/test_pipeline_v3.py",
]


def run_command(cmd: list[str], *, env: dict[str, str] | None = None, timeout: int = 600) -> subprocess.CompletedProcess[str]:
    return subprocess.run(
        cmd,
        cwd=REPO_ROOT,
        env=env,
        text=True,
        capture_output=True,
        timeout=timeout,
        check=True,
    )


def verify_unit() -> dict[str, str]:
    existing_tests = [path for path in CURATED_TESTS if (REPO_ROOT / path).exists()]
    if existing_tests:
        run_command([PYTHON, "-m", "pytest", "--capture=no", *existing_tests], timeout=600)
    compile_roots = ["scripts"]
    if (REPO_ROOT / "tests").exists():
        compile_roots.append("tests")
    run_command(
        [
            PYTHON,
            "-m",
            "py_compile",
            *subprocess.run(
                ["rg", "--files", *compile_roots, "-g", "*.py", "-g", "!scripts/lib/vendor/**"],
                cwd=REPO_ROOT,
                text=True,
                capture_output=True,
                check=True,
            ).stdout.split(),
        ],
        timeout=600,
    )
    return {"status": "ok" if existing_tests else "skipped-no-tests"}


def verify_diagnose() -> dict[str, object]:
    result = run_command([PYTHON, "scripts/last30days.py", "--diagnose"], timeout=120)
    return json.loads(result.stdout)


def verify_smoke() -> list[dict[str, object]]:
    rows: list[dict[str, object]] = []
    has_live_hosted = bool(os.environ.get("AISA_API_KEY"))
    smoke_timeout = LIVE_SMOKE_TIMEOUT if has_live_hosted else MOCK_SMOKE_TIMEOUT
    for provider, extra in SMOKE_CASES:
        env = os.environ.copy()
        env["LAST30DAYS_REASONING_PROVIDER"] = provider
        smoke_args = [PYTHON, "scripts/last30days.py", SMOKE_TOPIC, "--emit=json", *extra]
        if not has_live_hosted:
            smoke_args.append("--mock")
        start = time.time()
        print(
            f"[verify_v3] smoke start provider={provider} mode={'live' if has_live_hosted else 'mock'} timeout={smoke_timeout}s args={' '.join(extra)}",
            file=sys.stderr,
            flush=True,
        )
        try:
            result = run_command(smoke_args, env=env, timeout=smoke_timeout)
        except subprocess.TimeoutExpired:
            duration = round(time.time() - start, 2)
            print(
                f"[verify_v3] smoke timeout provider={provider} after {duration}s",
                file=sys.stderr,
                flush=True,
            )
            rows.append(
                {
                    "provider": provider,
                    "mode": "live" if has_live_hosted else "mock",
                    "duration_seconds": duration,
                    "reasoning_provider": "aisa" if has_live_hosted else "aisa",
                    "cluster_count": 0,
                    "candidate_count": 0,
                    "error_sources": ["timeout"],
                }
            )
            continue
        duration = round(time.time() - start, 2)
        report = json.loads(result.stdout)
        row = {
            "provider": provider,
            "mode": "live" if has_live_hosted else "mock",
            "duration_seconds": duration,
            "reasoning_provider": (report.get("provider_runtime") or {}).get("reasoning_provider"),
            "cluster_count": len(report.get("clusters") or []),
            "candidate_count": len(report.get("ranked_candidates") or []),
            "error_sources": sorted((report.get("errors_by_source") or {}).keys()),
        }
        print(
            f"[verify_v3] smoke done provider={provider} duration={duration}s candidates={row['candidate_count']} clusters={row['cluster_count']} errors={','.join(row['error_sources']) or 'none'}",
            file=sys.stderr,
            flush=True,
        )
        rows.append(row)
    return rows


def verify_latency() -> dict[str, dict[str, object]]:
    results: dict[str, dict[str, object]] = {}
    for profile, extra in LATENCY_PROFILES:
        timings = []
        for topic in LATENCY_TOPICS:
            start = time.time()
            run_command(
                [PYTHON, "scripts/last30days.py", topic, "--emit=json", *extra],
                timeout=300,
            )
            timings.append(time.time() - start)
        results[profile] = {
            "times": [round(value, 2) for value in timings],
            "median_seconds": round(statistics.median(timings), 2),
            "max_seconds": round(max(timings), 2),
        }
    return results


def verify_eval(
    *,
    baseline: str,
    candidate: str,
    output_dir: str,
    quick: bool,
    limit: int,
    timeout: int,
) -> dict[str, object]:
    cmd = [
        PYTHON,
        "scripts/evaluate_search_quality.py",
        f"--baseline={baseline}",
        f"--candidate={candidate}",
        f"--output-dir={output_dir}",
        f"--limit={limit}",
        f"--timeout={timeout}",
    ]
    if quick:
        cmd.append("--quick")
    run_command(cmd, timeout=max(timeout * 8, 600))
    output = Path(output_dir)
    metrics = json.loads((output / "metrics.json").read_text())
    summary = (output / "summary.md").read_text()
    return {"metrics": metrics, "summary": summary}


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Run the v3 verification bundle")
    parser.add_argument("--skip-eval", action="store_true", help="Skip the judged evaluator")
    parser.add_argument("--skip-latency", action="store_true", help="Skip live latency sampling")
    parser.add_argument("--baseline", default="HEAD~1")
    parser.add_argument("--candidate", default="WORKTREE")
    parser.add_argument("--output-dir", default="/tmp/last30days-v3-verify")
    parser.add_argument("--quick-eval", action="store_true", help="Use evaluator quick mode")
    parser.add_argument("--eval-limit", type=int, default=20)
    parser.add_argument("--eval-timeout", type=int, default=240)
    return parser


def main() -> int:
    args = build_parser().parse_args()
    summary: dict[str, object] = {}
    summary["unit"] = verify_unit()
    summary["diagnose"] = verify_diagnose()
    summary["smoke"] = verify_smoke()
    if not args.skip_latency:
        summary["latency"] = verify_latency()
    if not args.skip_eval:
        summary["eval"] = verify_eval(
            baseline=args.baseline,
            candidate=args.candidate,
            output_dir=args.output_dir,
            quick=args.quick_eval,
            limit=args.eval_limit,
            timeout=args.eval_timeout,
        )
    print(json.dumps(summary, indent=2, sort_keys=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/watchlist.py
content: #!/usr/bin/env python3
"""Topic watchlist management for last30days.

Preferred entrypoint:
    bash scripts/run-watchlist.sh ...
"""

from __future__ import annotations

import argparse
import json
import subprocess
import sys
import time
from pathlib import Path

try:
    import requests
except ImportError:
    requests = None

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

import store
from lib import schema


# --- Webhook Delivery Functions ---

def _deliver_findings(topic_name: str, counts: dict) -> None:
    """Send webhook notification if delivery is configured and there are new findings."""
    channel = store.get_setting("delivery_channel", "")
    if not channel or counts.get("new", 0) == 0:
        return
    
    mode = store.get_setting("delivery_mode", "announce")
    message = _format_delivery_message(topic_name, counts, mode)
    
    try:
        if "hooks.slack.com" in channel:
            _send_slack_webhook(channel, message)
        elif channel.startswith("https://"):
            _send_generic_webhook(channel, message)
    except Exception as e:
        # Don't fail the research run if delivery fails
        print(f"Delivery failed: {e}", file=sys.stderr)


def _format_delivery_message(topic: str, counts: dict, mode: str) -> str:
    """Format notification message based on delivery mode."""
    new = counts.get("new", 0)
    updated = counts.get("updated", 0)
    
    if mode == "announce":
        return f"📰 *last30days update: {topic}*\n{new} new, {updated} updated"
    elif mode == "silent":
        return f"last30days: {new} new findings for '{topic}'"
    else:
        return f"last30days: Research complete for '{topic}'"


def _send_slack_webhook(url: str, text: str) -> None:
    """POST to Slack incoming webhook."""
    if not requests:
        raise RuntimeError("requests library not available for webhook delivery")
    
    response = requests.post(
        url,
        json={"text": text},
        headers={"Content-Type": "application/json"},
        timeout=10,
    )
    response.raise_for_status()


def _send_generic_webhook(url: str, text: str) -> None:
    """POST JSON payload to generic webhook."""
    if not requests:
        raise RuntimeError("requests library not available for webhook delivery")
    
    response = requests.post(
        url,
        json={
            "message": text,
            "source": "last30days",
            "timestamp": time.time(),
        },
        headers={"Content-Type": "application/json"},
        timeout=10,
    )
    response.raise_for_status()


# --- Command Handlers ---

def cmd_add(args):
    schedule = "0 8 * * 1" if args.weekly else (args.schedule or "0 8 * * *")
    queries = [query.strip() for query in (args.queries or "").split(",") if query.strip()] or None
    topic = store.add_topic(args.topic, search_queries=queries, schedule=schedule)
    sched_desc = "weekly (Mondays 8am)" if args.weekly else f"daily ({schedule})"
    print(json.dumps({
        "action": "added",
        "topic": topic["name"],
        "schedule": sched_desc,
        "message": f'Added "{topic["name"]}" to watchlist. Schedule: {sched_desc}.',
    }, default=str))


def cmd_remove(args):
    removed = store.remove_topic(args.topic)
    if not removed:
        print(json.dumps({"action": "not_found", "topic": args.topic, "message": f'Topic not found: "{args.topic}"'}))
        return
    remaining = store.list_topics()
    print(json.dumps({
        "action": "removed",
        "topic": args.topic,
        "message": f'Removed "{args.topic}" from watchlist.',
        "remaining": len(remaining),
    }))


def cmd_list(args):
    del args
    topics = store.list_topics()
    budget_used = store.get_daily_cost()
    budget_limit = float(store.get_setting("daily_budget", "5.00"))
    print(json.dumps({
        "topics": topics,
        "budget_used": budget_used,
        "budget_limit": budget_limit,
    }, default=str))


def cmd_run_one(args):
    topic = store.get_topic(args.topic)
    if not topic:
        print(json.dumps({"error": f'Topic not found: "{args.topic}"'}))
        sys.exit(1)
    print(json.dumps(_run_topic(topic), default=str))


def cmd_run_all(args):
    del args
    topics = [topic for topic in store.list_topics() if topic["enabled"]]
    if not topics:
        print(json.dumps({"message": "No enabled topics to research."}))
        return

    budget_limit = float(store.get_setting("daily_budget", "5.00"))
    results = []
    for topic in topics:
        if store.get_daily_cost() >= budget_limit:
            results.append({
                "topic": topic["name"],
                "status": "skipped",
                "reason": f"Budget exceeded: ${store.get_daily_cost():.2f}/${budget_limit:.2f}",
            })
            continue
        results.append(_run_topic(topic))

    print(json.dumps({
        "action": "run_all",
        "results": results,
        "budget_used": store.get_daily_cost(),
        "budget_limit": budget_limit,
    }, default=str))


def _run_topic(topic: dict) -> dict:
    start_time = time.time()
    topic_id = topic["id"]
    run_id = store.record_run(topic_id, source_mode="v3", status="running")

    try:
        search_queries = json.loads(topic["search_queries"]) if topic.get("search_queries") else None
        search_term = search_queries[0] if search_queries else topic["name"]
        result = subprocess.run(
            [
                sys.executable,
                str(SCRIPT_DIR / "last30days.py"),
                search_term,
                "--emit=json",
                "--quick",
                "--lookback-days",
                "90",
            ],
            capture_output=True,
            text=True,
            timeout=300,
        )
        duration = time.time() - start_time
        if result.returncode != 0:
            store.update_run(
                run_id,
                status="failed",
                error_message=result.stderr[:500],
                duration_seconds=duration,
            )
            return {
                "topic": topic["name"],
                "status": "failed",
                "error": result.stderr[:200],
                "duration": duration,
            }

        report = schema.report_from_dict(json.loads(result.stdout))
        findings = store.findings_from_report(report, limit=25)
        counts = store.store_findings(run_id, topic_id, findings)
        store.update_run(
            run_id,
            status="completed",
            duration_seconds=duration,
            findings_new=counts["new"],
            findings_updated=counts["updated"],
        )
        
        # Deliver webhook notification if configured
        _deliver_findings(topic["name"], counts)
        
        return {
            "topic": topic["name"],
            "status": "completed",
            "new": counts["new"],
            "updated": counts["updated"],
            "duration": duration,
        }
    except subprocess.TimeoutExpired:
        duration = time.time() - start_time
        store.update_run(
            run_id,
            status="failed",
            error_message="Research timed out after 300s",
            duration_seconds=duration,
        )
        return {"topic": topic["name"], "status": "failed", "error": "timeout"}
    except json.JSONDecodeError as exc:
        duration = time.time() - start_time
        store.update_run(
            run_id,
            status="failed",
            error_message=f"Invalid JSON output: {exc}",
            duration_seconds=duration,
        )
        return {"topic": topic["name"], "status": "failed", "error": f"parse error: {exc}"}
def cmd_config(args):
    if args.key == "budget":
        store.set_setting("daily_budget", str(args.value))
        print(json.dumps({"action": "config", "key": "daily_budget", "value": str(args.value)}))
        return
    if args.key == "delivery":
        store.set_setting("delivery_channel", str(args.value))
        print(json.dumps({"action": "config", "key": "delivery_channel", "value": str(args.value)}))
        return
    raise SystemExit(f"Unknown config key: {args.key}")


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Manage the last30days watchlist")
    sub = parser.add_subparsers(dest="command")

    add = sub.add_parser("add")
    add.add_argument("topic")
    add.add_argument("--schedule")
    add.add_argument("--weekly", action="store_true")
    add.add_argument("--queries")
    add.set_defaults(func=cmd_add)

    remove = sub.add_parser("remove")
    remove.add_argument("topic")
    remove.set_defaults(func=cmd_remove)

    list_parser = sub.add_parser("list")
    list_parser.set_defaults(func=cmd_list)

    run_one = sub.add_parser("run-one")
    run_one.add_argument("topic")
    run_one.set_defaults(func=cmd_run_one)

    run_all = sub.add_parser("run-all")
    run_all.set_defaults(func=cmd_run_all)

    config = sub.add_parser("config")
    config.add_argument("key", choices=["delivery", "budget"])
    config.add_argument("value")
    config.set_defaults(func=cmd_config)

    return parser


def main() -> int:
    parser = build_parser()
    args = parser.parse_args()
    if not getattr(args, "command", None):
        parser.print_help()
        return 1
    args.func(args)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/lib/aisa.py
content: """AIsa API helpers for chat, search, and social retrieval."""

from __future__ import annotations

import math
import urllib.parse
from typing import Any

from . import dates, http

AISA_BASE_URL = "https://api.aisa.one"
AISA_CHAT_COMPLETIONS_URL = f"{AISA_BASE_URL}/v1/chat/completions"
AISA_TWITTER_SEARCH_URL = f"{AISA_BASE_URL}/apis/v1/twitter/tweet/advanced_search"
AISA_YOUTUBE_SEARCH_URL = f"{AISA_BASE_URL}/apis/v1/youtube/search"
AISA_TAVILY_SEARCH_URL = f"{AISA_BASE_URL}/apis/v1/tavily/search"
AISA_POLYMARKET_MARKETS_URL = f"{AISA_BASE_URL}/apis/v1/polymarket/markets"
AISA_CHAT_TIMEOUT = 20
AISA_CHAT_RETRIES = 1

DEPTH_LIMITS = {
    "quick": 6,
    "default": 12,
    "deep": 24,
}


def _headers(api_key: str) -> dict[str, str]:
    return {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }


def chat_completion(
    api_key: str,
    model: str,
    prompt: str,
    *,
    response_mime_type: str | None = None,
) -> dict[str, Any]:
    payload: dict[str, Any] = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }
    if response_mime_type == "application/json":
        payload["response_format"] = {"type": "json_object"}
    return http.post(
        AISA_CHAT_COMPLETIONS_URL,
        payload,
        headers=_headers(api_key),
        timeout=AISA_CHAT_TIMEOUT,
        retries=AISA_CHAT_RETRIES,
        max_429_retries=1,
    )


def search_twitter(
    api_key: str,
    query: str,
    from_date: str,
    depth: str = "default",
) -> dict[str, Any]:
    params = urllib.parse.urlencode(
        {
            "query": f"{query} since:{from_date}",
            "queryType": "Top",
            "count": DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"]),
        }
    )
    return http.get(
        f"{AISA_TWITTER_SEARCH_URL}?{params}",
        headers=_headers(api_key),
        timeout=45,
    )


def search_youtube(
    api_key: str,
    query: str,
    depth: str = "default",
) -> dict[str, Any]:
    params = urllib.parse.urlencode(
        {
            "engine": "youtube",
            "q": query,
            "num": DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"]),
        }
    )
    return http.get(
        f"{AISA_YOUTUBE_SEARCH_URL}?{params}",
        headers=_headers(api_key),
        timeout=45,
    )


def search_tavily(
    api_key: str,
    query: str,
    date_range: tuple[str, str] | None = None,
    *,
    limit: int | None = None,
    count: int = 5,
) -> dict[str, Any]:
    max_results = limit if limit is not None else count
    payload: dict[str, Any] = {
        "query": query,
        "topic": "news",
        "max_results": max_results,
        "include_raw_content": False,
    }
    if date_range:
        payload["start_date"] = date_range[0]
        payload["end_date"] = date_range[1]
    return http.post(
        AISA_TAVILY_SEARCH_URL,
        payload,
        headers=_headers(api_key),
        timeout=45,
    )


def search_polymarket(
    api_key: str,
    query: str,
) -> dict[str, Any]:
    params = urllib.parse.urlencode({"search": query, "status": "open"})
    return http.get(
        f"{AISA_POLYMARKET_MARKETS_URL}?{params}",
        headers=_headers(api_key),
        timeout=30,
    )


def parse_twitter_response(response: dict[str, Any], *, query: str = "") -> list[dict[str, Any]]:
    raw_items = (
        response.get("tweets")
        or response.get("results")
        or response.get("data")
        or response.get("items")
        or []
    )
    items: list[dict[str, Any]] = []
    for index, item in enumerate(raw_items):
        if not isinstance(item, dict):
            continue
        text = _first(item, "full_text", "text", "content")
        url = _first(item, "url", "tweet_url")
        if not url:
            tweet_id = _first(item, "tweet_id", "id", "rest_id")
            handle = _handle_from_item(item)
            if tweet_id and handle:
                url = f"https://x.com/{handle}/status/{tweet_id}"
        if not url:
            continue
        items.append(
            {
                "id": f"AX{index + 1}",
                "text": str(text or "").strip()[:500],
                "url": url,
                "author_handle": _handle_from_item(item),
                "date": _normalize_date(_first(item, "created_at", "date", "published_at")),
                "engagement": {
                    "likes": _to_int(_first(item, "favorite_count", "likes", "like_count")),
                    "reposts": _to_int(_first(item, "retweet_count", "reposts")),
                    "replies": _to_int(_first(item, "reply_count", "replies")),
                    "quotes": _to_int(_first(item, "quote_count", "quotes")),
                },
                "why_relevant": "AIsa Twitter search",
                "relevance": 0.8 if not query else _query_relevance(query, str(text or "")),
            }
        )
    return items


def parse_youtube_response(response: dict[str, Any], *, topic: str, from_date: str) -> list[dict[str, Any]]:
    raw_items = (
        response.get("video_results")
        or response.get("videos")
        or response.get("items")
        or response.get("results")
        or []
    )
    items: list[dict[str, Any]] = []
    for index, item in enumerate(raw_items):
        if not isinstance(item, dict):
            continue
        video_id = _first(item, "videoId", "id")
        url = _first(item, "link", "url")
        if not url and video_id:
            url = f"https://www.youtube.com/watch?v={video_id}"
        if not url:
            continue
        title = str(_first(item, "title", "name") or "")
        snippet = str(_first(item, "snippet", "description") or "")[:500]
        items.append(
            {
                "video_id": video_id or f"aisa-{index + 1}",
                "title": title,
                "url": url,
                "channel_name": str(_first(item, "channel", "channel_name", "author") or ""),
                "date": _normalize_date(_first(item, "publishedDate", "published_at", "date")),
                "engagement": {
                    "views": _to_int(_first(item, "views", "view_count")) or 0,
                    "likes": _to_int(_first(item, "likes", "like_count")),
                    "comments": None,
                },
                "duration": _first(item, "duration", "length"),
                "relevance": _youtube_relevance(topic, title, snippet),
                "why_relevant": "AIsa YouTube search",
                "description": snippet,
            }
        )
    recent = [item for item in items if item.get("date") and item["date"] >= from_date]
    return recent if len(recent) >= 3 else items


def parse_tavily_response(response: dict[str, Any], *, date_range: tuple[str, str]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
    raw_items = response.get("results") or response.get("data") or []
    items: list[dict[str, Any]] = []
    for index, item in enumerate(raw_items):
        if not isinstance(item, dict):
            continue
        url = _first(item, "url", "link")
        if not url:
            continue
        pub_date = _normalize_date(_first(item, "published_date", "date"))
        if pub_date and not (date_range[0] <= pub_date <= date_range[1]):
            continue
        items.append(
            {
                "id": f"AT{index + 1}",
                "title": str(_first(item, "title") or ""),
                "url": url,
                "source_domain": urllib.parse.urlparse(url).netloc.strip().lower(),
                "snippet": str(_first(item, "content", "snippet", "raw_content") or "")[:500],
                "date": pub_date,
                "relevance": 0.8,
                "why_relevant": "AIsa Tavily search",
            }
        )
    artifact = {"label": "aisa-tavily", "webSearchQueries": [response.get("query", "")], "resultCount": len(items)}
    return items, artifact


def parse_polymarket_response(response: dict[str, Any], *, topic: str = "") -> list[dict[str, Any]]:
    raw_items = response.get("markets") or response.get("data") or response.get("results") or []
    items: list[dict[str, Any]] = []
    for index, item in enumerate(raw_items):
        if not isinstance(item, dict):
            continue
        question = str(_first(item, "question", "title", "name") or "").strip()
        if not question:
            continue
        url = _first(item, "url")
        slug = _first(item, "slug")
        market_id = _first(item, "id", "market_id")
        if not url:
            if slug:
                url = f"https://polymarket.com/event/{slug}"
            elif market_id:
                url = f"https://polymarket.com/event/{market_id}"
        if not url:
            continue
        items.append(
            {
                "id": f"AP{index + 1}",
                "title": question,
                "url": url,
                "date": _normalize_date(_first(item, "updatedAt", "end_date", "date")),
                "probability": _normalize_probability(_first(item, "probability", "yes_price", "price")),
                "volume": _to_float(_first(item, "volume")),
                "container": "Polymarket",
                "why_relevant": "AIsa Polymarket search",
                "relevance": _query_relevance(topic, question) if topic else 0.8,
            }
        )
    return items


def _handle_from_item(item: dict[str, Any]) -> str:
    return str(
        _first(item, "screen_name", "author_handle", "username", "user_name", "handle") or ""
    ).lstrip("@")


def _first(item: dict[str, Any], *keys: str) -> Any:
    for key in keys:
        value = item.get(key)
        if value not in (None, ""):
            return value
    return None


def _normalize_date(value: object) -> str | None:
    if value is None:
        return None
    parsed = dates.parse_date(str(value).strip())
    if not parsed:
        return None
    return parsed.date().isoformat()


def _to_int(value: Any) -> int | None:
    if value in (None, ""):
        return None
    try:
        return int(value)
    except (TypeError, ValueError):
        return None


def _to_float(value: Any) -> float | None:
    if value in (None, ""):
        return None
    try:
        return float(value)
    except (TypeError, ValueError):
        return None


def _normalize_probability(value: Any) -> float | None:
    prob = _to_float(value)
    if prob is None:
        return None
    if prob > 1:
        return prob / 100.0
    return prob


def _query_relevance(topic: str, text: str) -> float:
    topic_tokens = {token for token in topic.lower().split() if len(token) > 2}
    if not topic_tokens:
        return 0.8
    text_lower = text.lower()
    overlap = sum(1 for token in topic_tokens if token in text_lower)
    return min(1.0, max(0.4, overlap / max(1, len(topic_tokens))))


def _youtube_relevance(topic: str, title: str, snippet: str) -> float:
    text = f"{title} {snippet}".strip()
    base = _query_relevance(topic, text)
    if re_match := any(marker in text.lower() for marker in ("review", "breakdown", "reaction", "explained")):
        return min(1.0, base + 0.1)
    return base


path: scripts/lib/cluster.py
content: """Candidate clustering and representative selection."""

from __future__ import annotations

import re

from . import dedupe, schema

CLUSTERABLE_INTENTS = {"breaking_news", "opinion", "comparison", "prediction"}

# Words too common to signal shared topic between clusters.
_ENTITY_STOPWORDS = frozenset({
    "the", "a", "an", "to", "for", "how", "is", "in", "of", "on", "and",
    "with", "from", "by", "at", "this", "that", "it", "what", "are", "do",
    "can", "his", "her", "he", "she", "its", "was", "has", "new", "just",
    "says", "said", "will", "about", "after", "now", "all", "been", "here",
    "not", "out", "up", "more", "also", "but", "who", "year", "first",
    "make", "being", "making", "over", "into", "than", "they", "their",
    "would", "could", "get", "got", "some", "like", "back", "going",
    "breaking", "https", "http", "www", "com",
})


def _candidate_text(candidate: schema.Candidate) -> str:
    return " ".join(part for part in [candidate.title, candidate.snippet] if part).strip()


def _extract_entities(text: str) -> set[str]:
    """Extract significant words (proper nouns, numbers, capitalized words) from text.

    Used for cross-source cluster merging where phrasing differs but entities overlap.
    """
    # Normalize but preserve word boundaries
    words = re.sub(r"[^\w\s]", " ", text).split()
    entities = set()
    for word in words:
        lower = word.lower()
        if lower in _ENTITY_STOPWORDS or len(word) <= 2:
            continue
        # Keep words that are: capitalized, ALL CAPS, contain digits, or 4+ chars
        if word[0].isupper() or word.isupper() or any(c.isdigit() for c in word) or len(word) >= 4:
            entities.add(lower)
    return entities


def _entity_overlap(entities_a: set[str], entities_b: set[str]) -> float:
    """Jaccard-style overlap on extracted entities."""
    if not entities_a or not entities_b:
        return 0.0
    intersection = entities_a & entities_b
    smaller = min(len(entities_a), len(entities_b))
    # Use overlap coefficient (intersection / min) instead of Jaccard,
    # because a short tweet about the same event as a long Reddit post
    # will have fewer total entities but high overlap with the larger set.
    return len(intersection) / smaller if smaller > 0 else 0.0


def _mmr_representatives(
    candidates: list[schema.Candidate],
    text_cache: dict[str, dedupe._PreparedText],
    limit: int = 3,
    diversity_lambda: float = 0.75,
) -> list[str]:
    selected: list[schema.Candidate] = []
    remaining_set = {c.candidate_id for c in candidates}
    remaining = list(candidates)
    while remaining and len(selected) < limit:
        if not selected:
            best = max(remaining, key=lambda candidate: candidate.final_score)
            selected.append(best)
            remaining_set.discard(best.candidate_id)
            remaining = [c for c in remaining if c.candidate_id in remaining_set]
            continue

        selected_preps = [text_cache[c.candidate_id] for c in selected]

        def score(candidate: schema.Candidate) -> float:
            prep = text_cache[candidate.candidate_id]
            diversity_penalty = max(
                dedupe.prepared_similarity(prep, sp) for sp in selected_preps
            )
            return (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)

        best = max(remaining, key=score)
        selected.append(best)
        remaining_set.discard(best.candidate_id)
        remaining = [c for c in remaining if c.candidate_id in remaining_set]
    return [candidate.candidate_id for candidate in selected]


def cluster_candidates(
    candidates: list[schema.Candidate],
    plan: schema.QueryPlan,
) -> list[schema.Cluster]:
    """Greedy clustering around high-ranked leaders."""
    if plan.intent not in CLUSTERABLE_INTENTS or plan.cluster_mode == "none":
        clusters = []
        for index, candidate in enumerate(candidates, start=1):
            cluster_id = f"cluster-{index}"
            candidate.cluster_id = cluster_id
            clusters.append(
                schema.Cluster(
                    cluster_id=cluster_id,
                    title=candidate.title,
                    candidate_ids=[candidate.candidate_id],
                    representative_ids=[candidate.candidate_id],
                    sources=sorted(schema.candidate_sources(candidate)),
                    score=candidate.final_score,
                    uncertainty=None,
                )
            )
        return clusters

    text_cache: dict[str, dedupe._PreparedText] = {
        c.candidate_id: dedupe._PreparedText(_candidate_text(c))
        for c in candidates
    }

    groups: list[list[schema.Candidate]] = []
    # Lower threshold for breaking_news: related articles share fewer exact
    # words but cover the same event.
    threshold = 0.42 if plan.intent == "breaking_news" else 0.48
    for candidate in candidates:
        assigned = False
        cand_prep = text_cache[candidate.candidate_id]
        for group in groups:
            leader = group[0]
            similarity = dedupe.prepared_similarity(cand_prep, text_cache[leader.candidate_id])
            if similarity >= threshold:
                group.append(candidate)
                assigned = True
                break
        if not assigned:
            groups.append([candidate])

    clusters: list[schema.Cluster] = []
    for index, group in enumerate(groups, start=1):
        group.sort(key=lambda candidate: candidate.final_score, reverse=True)
        cluster_id = f"cluster-{index}"
        representatives = _mmr_representatives(group, text_cache)
        for candidate in group:
            candidate.cluster_id = cluster_id
        clusters.append(
            schema.Cluster(
                cluster_id=cluster_id,
                title=group[0].title,
                candidate_ids=[candidate.candidate_id for candidate in group],
                representative_ids=representatives,
                sources=sorted({source for candidate in group for source in schema.candidate_sources(candidate)}),
                score=max(candidate.final_score for candidate in group),
                uncertainty=_cluster_uncertainty(group),
            )
        )

    # Second pass: merge small clusters that share entities across sources.
    clusters = _merge_entity_clusters(clusters, candidates)

    return sorted(clusters, key=lambda cluster: cluster.score, reverse=True)


def _merge_entity_clusters(
    clusters: list[schema.Cluster],
    all_candidates: list[schema.Candidate],
) -> list[schema.Cluster]:
    """Merge small clusters that cover the same story across different sources.

    The initial greedy pass uses text similarity which misses cross-source
    matches where phrasing differs. This second pass looks at entity overlap
    (proper nouns, names, numbers) to catch cases like:
      - Reddit: "Kanye West to headline all three nights of Wireless Festival 2026"
      - X: "BREAKING: Kanye West (Ye) is making his massive UK comeback!"
    """
    if len(clusters) < 2:
        return clusters

    candidate_map = {c.candidate_id: c for c in all_candidates}

    # Build entity sets per cluster
    cluster_entities: list[set[str]] = []
    for cl in clusters:
        entities: set[str] = set()
        for cid in cl.candidate_ids:
            cand = candidate_map.get(cid)
            if cand:
                entities |= _extract_entities(_candidate_text(cand))
        cluster_entities.append(entities)

    # Only merge clusters with <= 3 items (don't merge already-large clusters)
    merged_into: dict[int, int] = {}  # index -> merge target index
    for i in range(len(clusters)):
        if i in merged_into or len(clusters[i].candidate_ids) > 3:
            continue
        for j in range(i + 1, len(clusters)):
            if j in merged_into or len(clusters[j].candidate_ids) > 3:
                continue
            # Require different sources to merge (same-source should already be grouped)
            sources_i = set(clusters[i].sources)
            sources_j = set(clusters[j].sources)
            if sources_i == sources_j and len(sources_i) == 1:
                continue
            # Prevent Polymarket clusters from merging with non-Polymarket
            # clusters. Prediction markets about "Sam Altman equity" should not
            # merge into a news cluster about "Sam Altman rivalry" just because
            # both mention the same entity.
            poly_i = "polymarket" in sources_i
            poly_j = "polymarket" in sources_j
            if poly_i != poly_j:
                continue

            overlap = _entity_overlap(cluster_entities[i], cluster_entities[j])
            if overlap >= 0.45:
                merged_into[j] = i

    if not merged_into:
        return clusters

    # Build merged cluster list
    result: list[schema.Cluster] = []
    for i, cl in enumerate(clusters):
        if i in merged_into:
            continue
        # Collect all clusters merged into this one
        merge_sources = [i] + [j for j, target in merged_into.items() if target == i]
        if len(merge_sources) == 1:
            result.append(cl)
            continue

        # Combine candidates from all merged clusters
        combined_cids: list[str] = []
        combined_sources: set[str] = set()
        best_score = 0.0
        for idx in merge_sources:
            combined_cids.extend(clusters[idx].candidate_ids)
            combined_sources.update(clusters[idx].sources)
            best_score = max(best_score, clusters[idx].score)

        # Pick representatives from combined pool
        combined_candidates = [candidate_map[cid] for cid in combined_cids if cid in candidate_map]
        combined_candidates.sort(key=lambda c: c.final_score, reverse=True)
        merge_text_cache = {
            c.candidate_id: dedupe._PreparedText(_candidate_text(c))
            for c in combined_candidates
        }
        reps = _mmr_representatives(combined_candidates, merge_text_cache)

        cluster_id = cl.cluster_id
        for cid in combined_cids:
            cand = candidate_map.get(cid)
            if cand:
                cand.cluster_id = cluster_id

        result.append(schema.Cluster(
            cluster_id=cluster_id,
            title=combined_candidates[0].title if combined_candidates else cl.title,
            candidate_ids=combined_cids,
            representative_ids=reps,
            sources=sorted(combined_sources),
            score=best_score,
            uncertainty=_cluster_uncertainty(combined_candidates),
        ))

    return result


def _cluster_uncertainty(group: list[schema.Candidate]) -> str | None:
    sources = {source for candidate in group for source in schema.candidate_sources(candidate)}
    if len(sources) == 1:
        return "single-source"
    if max(candidate.final_score for candidate in group) < 55:
        return "thin-evidence"
    return None


path: scripts/lib/dates.py
content: """Date utilities for last30days skill."""

from datetime import datetime, timedelta, timezone
from typing import Optional, Tuple


def get_date_range(days: int = 30) -> Tuple[str, str]:
    """Get the date range for the last N days.

    Returns:
        Tuple of (from_date, to_date) as YYYY-MM-DD strings
    """
    today = datetime.now(timezone.utc).date()
    from_date = today - timedelta(days=days)
    return from_date.isoformat(), today.isoformat()


def parse_date(date_str: Optional[str]) -> Optional[datetime]:
    """Parse a date string in various formats.

    Supports: YYYY-MM-DD, ISO 8601, Unix timestamp
    """
    if not date_str:
        return None

    # Try Unix timestamp (from Reddit)
    try:
        ts = float(date_str)
        return datetime.fromtimestamp(ts, tz=timezone.utc)
    except (ValueError, TypeError):
        pass

    # Try ISO formats
    formats = [
        "%Y-%m-%d",
        "%Y-%m-%dT%H:%M:%S",
        "%Y-%m-%dT%H:%M:%SZ",
        "%Y-%m-%dT%H:%M:%S%z",
        "%Y-%m-%dT%H:%M:%S.%f%z",
    ]

    for fmt in formats:
        try:
            dt = datetime.strptime(date_str, fmt)
            if dt.tzinfo is not None:
                return dt.astimezone(timezone.utc)
            return dt.replace(tzinfo=timezone.utc)
        except ValueError:
            continue

    return None


def timestamp_to_date(ts: Optional[float]) -> Optional[str]:
    """Convert Unix timestamp to YYYY-MM-DD string."""
    if ts is None:
        return None
    try:
        dt = datetime.fromtimestamp(ts, tz=timezone.utc)
        return dt.date().isoformat()
    except (ValueError, TypeError, OSError):
        return None


def get_date_confidence(date_str: Optional[str], from_date: str, to_date: str) -> str:
    """Determine confidence level for a date.

    Args:
        date_str: The date to check (YYYY-MM-DD or None)
        from_date: Start of valid range (YYYY-MM-DD)
        to_date: End of valid range (YYYY-MM-DD)

    Returns:
        'high', 'med', or 'low'
    """
    if not date_str:
        return 'low'

    try:
        dt = datetime.strptime(date_str, "%Y-%m-%d").date()
        start = datetime.strptime(from_date, "%Y-%m-%d").date()
        end = datetime.strptime(to_date, "%Y-%m-%d").date()

        return 'high' if start <= dt <= end else 'low'
    except ValueError:
        return 'low'


def days_ago(date_str: Optional[str]) -> Optional[int]:
    """Calculate how many days ago a date is.

    Returns None if date is invalid or missing.
    """
    if not date_str:
        return None

    try:
        dt = datetime.strptime(date_str, "%Y-%m-%d").date()
        today = datetime.now(timezone.utc).date()
        delta = today - dt
        return delta.days
    except ValueError:
        return None


def recency_score(date_str: Optional[str], max_days: int = 30) -> int:
    """Calculate recency score (0-100).

    0 days ago = 100, max_days ago = 0, clamped.
    """
    age = days_ago(date_str)
    if age is None:
        return 0  # Unknown date gets worst score

    if age < 0:
        return 100  # Future date (treat as today)
    if age >= max_days:
        return 0

    return int(100 * (1 - age / max_days))


path: scripts/lib/dedupe.py
content: """Within-source near-duplicate detection."""

from __future__ import annotations

import re

from . import schema

STOPWORDS = frozenset(
    {
        "the",
        "a",
        "an",
        "to",
        "for",
        "how",
        "is",
        "in",
        "of",
        "on",
        "and",
        "with",
        "from",
        "by",
        "at",
        "this",
        "that",
        "it",
        "what",
        "are",
        "do",
        "can",
    }
)


def normalize_text(text: str) -> str:
    text = re.sub(r"[^\w\s]", " ", text.lower())
    return re.sub(r"\s+", " ", text).strip()


def get_ngrams(text: str, n: int = 3) -> set[str]:
    text = normalize_text(text)
    if len(text) < n:
        return {text} if text else set()
    return {text[index:index + n] for index in range(len(text) - n + 1)}


def jaccard_similarity(left: set[str], right: set[str]) -> float:
    if not left or not right:
        return 0.0
    union = left | right
    if not union:
        return 0.0
    return len(left & right) / len(union)


def token_jaccard(text_a: str, text_b: str) -> float:
    tokens_a = {
        token
        for token in normalize_text(text_a).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    tokens_b = {
        token
        for token in normalize_text(text_b).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    return jaccard_similarity(tokens_a, tokens_b)


def hybrid_similarity(text_a: str, text_b: str) -> float:
    return max(
        jaccard_similarity(get_ngrams(text_a), get_ngrams(text_b)),
        token_jaccard(text_a, text_b),
    )


def _tokenize(normalized: str) -> frozenset[str]:
    return frozenset(
        tok for tok in normalized.split()
        if len(tok) > 1 and tok not in STOPWORDS
    )


class _PreparedText:
    """Pre-computed text representations for fast repeated similarity checks."""

    __slots__ = ("ngrams", "tokens")

    def __init__(self, raw: str) -> None:
        norm = normalize_text(raw)
        self.ngrams = get_ngrams(norm) if norm else set()
        self.tokens = _tokenize(norm)


def prepared_similarity(a: _PreparedText, b: _PreparedText) -> float:
    return max(
        jaccard_similarity(a.ngrams, b.ngrams),
        jaccard_similarity(a.tokens, b.tokens),
    )


def item_text(item: schema.SourceItem) -> str:
    parts = [item.title, item.body, item.author or "", item.container or ""]
    return " ".join(part for part in parts if part).strip()


def dedupe_items(items: list[schema.SourceItem], threshold: float = 0.7) -> list[schema.SourceItem]:
    """Remove near-duplicates while keeping earlier, better-scored items."""
    kept: list[schema.SourceItem] = []
    kept_prepared: list[_PreparedText] = []
    for item in items:
        text = item_text(item)
        if not text:
            kept.append(item)
            continue
        prep = _PreparedText(text)
        is_duplicate = False
        for existing_prep in kept_prepared:
            if prepared_similarity(prep, existing_prep) >= threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            kept.append(item)
            kept_prepared.append(prep)
    return kept


path: scripts/lib/entity_extract.py
content: """Entity extraction from initial search results for supplemental searches."""

import re
from collections import Counter
from typing import Any, Dict, List

# Handles that appear too frequently to be useful for targeted search.
# These are generic/platform accounts, not topic-specific voices.
GENERIC_HANDLES = {
    "elonmusk", "openai", "google", "microsoft", "apple", "meta",
    "github", "youtube", "x", "twitter", "reddit", "wikipedia",
    "nytimes", "washingtonpost", "cnn", "bbc", "reuters",
    "verified", "jack", "sundarpichai",
}


def extract_entities(
    reddit_items: List[Dict[str, Any]],
    x_items: List[Dict[str, Any]],
    max_handles: int = 5,
    max_hashtags: int = 3,
    max_subreddits: int = 5,
) -> Dict[str, List[str]]:
    """Extract key entities from Phase 1 results for supplemental searches.

    Parses X results for @handles and #hashtags, Reddit results for subreddit
    names and cross-referenced communities.

    Args:
        reddit_items: Raw Reddit item dicts from Phase 1
        x_items: Raw X item dicts from Phase 1
        max_handles: Maximum handles to return
        max_hashtags: Maximum hashtags to return
        max_subreddits: Maximum subreddits to return

    Returns:
        Dict with keys: x_handles, x_hashtags, reddit_subreddits
    """
    handles = _extract_x_handles(x_items)
    hashtags = _extract_x_hashtags(x_items)
    subreddits = _extract_subreddits(reddit_items)

    return {
        "x_handles": handles[:max_handles],
        "x_hashtags": hashtags[:max_hashtags],
        "reddit_subreddits": subreddits[:max_subreddits],
    }


def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank @handles from X results.

    Sources handles from:
    1. author_handle field (who posted)
    2. @mentions in post text (who they're talking about/to)

    Returns handles ranked by frequency, filtered for generic accounts.
    """
    handle_counts = Counter()

    for item in x_items:
        # Author handle
        author = item.get("author_handle", "").strip().lstrip("@").lower()
        if author and author not in GENERIC_HANDLES:
            handle_counts[author] += 1

        # @mentions in text
        text = item.get("text", "")
        mentions = re.findall(r'@(\w{1,15})', text)
        for mention in mentions:
            mention_lower = mention.lower()
            if mention_lower not in GENERIC_HANDLES:
                handle_counts[mention_lower] += 1

    # Return all handles ranked by frequency
    return [h for h, _ in handle_counts.most_common()]


def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank #hashtags from X results.

    Returns hashtags ranked by frequency.
    """
    hashtag_counts = Counter()

    for item in x_items:
        text = item.get("text", "")
        tags = re.findall(r'#(\w{2,30})', text)
        for tag in tags:
            hashtag_counts[tag.lower()] += 1

    # Return all hashtags ranked by frequency
    return [f"#{t}" for t, _ in hashtag_counts.most_common()]


def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank subreddits from Reddit results.

    Sources from:
    1. subreddit field on each result
    2. Cross-references in comment text (e.g., "check out r/localLLaMA")

    Returns subreddits ranked by frequency.
    """
    sub_counts = Counter()

    for item in reddit_items:
        # Primary subreddit
        sub = item.get("subreddit", "").strip().lstrip("r/")
        if sub:
            sub_counts[sub] += 1

        # Cross-references in comment insights
        for insight in item.get("comment_insights", []):
            cross_refs = re.findall(r'r/(\w{2,30})', insight)
            for ref in cross_refs:
                sub_counts[ref] += 1

        # Cross-references in top comments
        for comment in item.get("top_comments", []):
            excerpt = comment.get("excerpt", "")
            cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
            for ref in cross_refs:
                sub_counts[ref] += 1

    # Return subreddits ranked by frequency
    return [sub for sub, _ in sub_counts.most_common()]


path: scripts/lib/env.py
content: """Environment and runtime configuration for the AISA-only last30days skill."""

from __future__ import annotations

import os
import sys
from pathlib import Path
from typing import Any

# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
# Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
_config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
if _config_override == "":
    # Empty string = no config file (clean mode)
    CONFIG_DIR = None
    CONFIG_FILE = None
elif _config_override:
    CONFIG_DIR = Path(_config_override)
    CONFIG_FILE = CONFIG_DIR / ".env"
else:
    CONFIG_DIR = Path.home() / ".config" / "last30days"
    CONFIG_FILE = CONFIG_DIR / ".env"

def _check_file_permissions(path: Path) -> None:
    """Warn to stderr if a secrets file has overly permissive permissions."""
    try:
        mode = path.stat().st_mode
        # Check if group or other can read (bits 0o044)
        if mode & 0o044:
            sys.stderr.write(
                f"[last30days] WARNING: {path} is readable by other users. "
                f"Run: chmod 600 {path}\n"
            )
            sys.stderr.flush()
    except OSError as exc:
        sys.stderr.write(f"[last30days] WARNING: could not stat {path}: {exc}\n")
        sys.stderr.flush()


def load_env_file(path: Path) -> dict[str, str]:
    """Load environment variables from a file."""
    env = {}
    if not path or not path.exists():
        return env
    _check_file_permissions(path)

    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            if '=' in line:
                key, _, value = line.partition('=')
                key = key.strip()
                value = value.strip()
                # Remove quotes if present
                if value and value[0] in ('"', "'") and value[-1] == value[0]:
                    value = value[1:-1]
                if key and value:
                    env[key] = value
    return env

def _find_project_env() -> Path | None:
    """Find per-project .env by walking up from cwd.

    Searches for .claude/last30days.env in each parent directory,
    stopping at the user's home directory or filesystem root.
    """
    cwd = Path.cwd()
    for parent in [cwd, *cwd.parents]:
        candidate = parent / '.claude' / 'last30days.env'
        if candidate.exists():
            return candidate
        # Stop at filesystem root or home
        if parent == Path.home() or parent == parent.parent:
            break
    return None


def get_config() -> dict[str, Any]:
    """Load configuration from multiple sources.

    Priority (highest wins):
      1. Environment variables (os.environ)
      2. .claude/last30days.env (per-project config)
      3. ~/.config/last30days/.env (global config)
    """
    # Load from global config file
    file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}

    # Load from per-project config (overrides global)
    project_env_path = _find_project_env()
    project_env = load_env_file(project_env_path) if project_env_path else {}

    # Merge: project overrides global
    merged_env = {**file_env, **project_env}

    # Build config: process.env > project .env > global .env
    config = {
        'AISA_API_KEY': os.environ.get('AISA_API_KEY') or merged_env.get('AISA_API_KEY'),
        'AISA_BASE_URL': os.environ.get('AISA_BASE_URL') or merged_env.get('AISA_BASE_URL', 'https://api.aisa.one'),
        'GITHUB_TOKEN': (
            os.environ.get('GITHUB_TOKEN')
            or os.environ.get('GH_TOKEN')
            or merged_env.get('GITHUB_TOKEN')
            or merged_env.get('GH_TOKEN')
        ),
    }

    keys = [
        ('XIAOHONGSHU_API_BASE', None),
        ('LAST30DAYS_REASONING_PROVIDER', 'auto'),
        ('LAST30DAYS_PLANNER_MODEL', None),
        ('LAST30DAYS_RERANK_MODEL', None),
        ('LAST30DAYS_X_BACKEND', None),
        ('SETUP_COMPLETE', None),
        ('INCLUDE_SOURCES', None),
        ('LAST30DAYS_YOUTUBE_TRANSCRIPTS', None),
        ('LAST30DAYS_REDDIT_COMMENTS', None),
    ]

    for key, default in keys:
        config[key] = os.environ.get(key) or merged_env.get(key, default)

    # Track which config source was used
    if project_env_path:
        config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
    elif CONFIG_FILE and CONFIG_FILE.exists():
        config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
    else:
        config['_CONFIG_SOURCE'] = 'env_only'

    return config


def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
    """Return (source, method) for X search in the AISA-only runtime."""
    if config.get("AISA_API_KEY"):
        return "aisa", "aisa"
    return None, "none"


def config_exists() -> bool:
    """Check if any configuration source exists."""
    if _find_project_env():
        return True
    if CONFIG_FILE:
        return CONFIG_FILE.exists()
    return False


def is_reddit_available(config: dict[str, Any]) -> bool:
    """Check if Reddit search is available.

    Public Reddit is always available.
    """
    del config
    return True


def get_reddit_source(config: dict[str, Any]) -> str | None:
    """Determine which Reddit backend to use."""
    del config
    return 'public'


def get_x_source(config: dict[str, Any]) -> str | None:
    """Determine the active X backend for the AISA-only runtime."""
    preferred = (config.get('LAST30DAYS_X_BACKEND') or '').lower()
    if preferred == 'aisa':
        return 'aisa' if config.get('AISA_API_KEY') else None
    if config.get('AISA_API_KEY'):
        return 'aisa'
    return None


def is_ytdlp_available() -> bool:
    """Legacy compatibility probe for older transcript helpers."""
    from . import youtube_yt
    return youtube_yt.is_ytdlp_installed()


def is_youtube_comments_available(config: dict[str, Any]) -> bool:
    """YouTube comment enrichment is not exposed in the AISA-only runtime."""
    del config
    return False


def is_youtube_sc_available(config: dict[str, Any]) -> bool:
    """Check if AISA YouTube search is available."""
    return bool(config.get('AISA_API_KEY'))


def is_hackernews_available() -> bool:
    """Check if Hacker News source is available.

    Always returns True - HN uses free Algolia API, no key needed.
    """
    return True


def is_polymarket_available() -> bool:
    """Check if Polymarket source is available.

    AISA is required for the hosted Polymarket integration.
    """
    return bool(os.environ.get("AISA_API_KEY"))


def is_tiktok_available(config: dict[str, Any]) -> bool:
    """Check if TikTok source is available."""
    return bool(config.get('AISA_API_KEY'))


def get_tiktok_token(config: dict[str, Any]) -> str:
    """Get the AISA token for TikTok discovery."""
    return config.get('AISA_API_KEY') or ''


def _parse_include_sources(config: dict[str, Any]) -> set[str]:
    """Parse INCLUDE_SOURCES config value into a set of lowercase source names."""
    raw = config.get('INCLUDE_SOURCES') or ''
    return {s.strip().lower() for s in raw.split(',') if s.strip()}


def is_threads_available(config: dict[str, Any]) -> bool:
    """Check if Threads source is available."""
    if not config.get('AISA_API_KEY'):
        return False
    return 'threads' in _parse_include_sources(config)


def is_instagram_available(config: dict[str, Any]) -> bool:
    """Check if Instagram source is available."""
    return bool(config.get('AISA_API_KEY'))


def get_instagram_token(config: dict[str, Any]) -> str:
    """Get the AISA token for Instagram discovery."""
    return config.get('AISA_API_KEY') or ''


def get_xiaohongshu_api_base(config: dict[str, Any]) -> str:
    """Get Xiaohongshu HTTP API base URL.

    Defaults to host.docker.internal so OpenClaw Docker can reach host service.
    """
    return (config.get('XIAOHONGSHU_API_BASE') or "http://host.docker.internal:18060").rstrip("/")


def is_xiaohongshu_available(config: dict[str, Any]) -> bool:
    """Check whether Xiaohongshu HTTP API is reachable and logged in."""
    # Import here to avoid heavy imports at module load.
    from . import http

    base = get_xiaohongshu_api_base(config)
    try:
        # Keep health probe snappy, but allow one retry for transient hiccups.
        health = http.get(f"{base}/health", timeout=3, retries=2)
        if not isinstance(health, dict):
            return False
        if not health.get("success"):
            return False

        # Login probe can be slower on some deployments (browser/session checks),
        # so use a slightly longer timeout to avoid false negatives.
        login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=2)
        is_logged_in = (
            login.get("data", {}).get("is_logged_in")
            if isinstance(login, dict) else False
        )
        return bool(is_logged_in)
    except (OSError, http.HTTPError):
        return False
    except Exception as exc:
        sys.stderr.write(
            f"[last30days] WARNING: unexpected error checking Xiaohongshu: "
            f"{type(exc).__name__}: {exc}\n"
        )
        sys.stderr.flush()
        return False


# Backward compat alias
is_apify_available = is_tiktok_available


def get_x_source_status(config: dict[str, Any]) -> dict[str, Any]:
    """Get detailed X source status for UI decisions."""
    if config.get('AISA_API_KEY'):
        source = 'aisa'
    else:
        source = None

    return {
        "source": source,
        "bird_installed": False,
        "bird_authenticated": False,
        "bird_username": "",
        "aisa_available": bool(config.get('AISA_API_KEY')),
        "xai_available": False,
        "can_install_bird": False,
    }


# Pinterest
def is_pinterest_available(config: dict[str, Any]) -> bool:
    """Check if Pinterest source is available."""
    if not config.get('AISA_API_KEY'):
        return False
    return 'pinterest' in _parse_include_sources(config)


def get_pinterest_token(config: dict[str, Any]) -> str:
    """Get the AISA token for Pinterest discovery."""
    return config.get('AISA_API_KEY') or ''


# Xquik
def is_xquik_available(config: dict[str, Any]) -> bool:
    """Xquik is no longer exposed in the AISA-only runtime."""
    del config
    return False


def get_xquik_token(config: dict[str, Any]) -> str:
    """Xquik is retired from the default runtime surface."""
    del config
    return ''


path: scripts/lib/fusion.py
content: """Weighted reciprocal rank fusion for per-(subquery, source) streams."""

from __future__ import annotations

from urllib.parse import parse_qs, urlencode, urlparse, urlunparse

from . import schema

# Standard RRF smoothing constant (Cormack et al. 2009)
RRF_K = 60


def _candidate_sort_key(c: schema.Candidate) -> tuple:
    return (-c.rrf_score, -c.local_relevance, -c.freshness, schema.candidate_source_label(c), c.title)


def _normalize_url(url: str) -> str:
    """Normalize URL for dedup: lowercase, strip www/old/m prefixes, remove tracking params."""
    parsed = urlparse(url.strip().lower())
    netloc = parsed.netloc
    for prefix in ("www.", "old.", "m."):
        if netloc.startswith(prefix):
            netloc = netloc[len(prefix):]
    # Strip tracking params
    params = parse_qs(parsed.query)
    clean_params = {k: v for k, v in params.items() if not k.startswith("utm_")}
    query = urlencode(clean_params, doseq=True)
    return urlunparse((parsed.scheme, netloc, parsed.path.rstrip("/"), "", query, ""))


def candidate_key(item: schema.SourceItem) -> str:
    if item.url:
        return _normalize_url(item.url)
    return f"{item.source}:{item.item_id}"


_DIVERSITY_RELEVANCE_THRESHOLD = 0.25

# Per-author cap: no single author/handle should dominate the pool.
_MAX_ITEMS_PER_AUTHOR = 3


def _extract_author(candidate: schema.Candidate) -> str | None:
    """Return a normalized author key from a candidate's source items."""
    for item in candidate.source_items:
        if item.author:
            return item.author.strip().lower()
    return None


def _apply_per_author_cap(
    candidates: list[schema.Candidate],
    max_per_author: int = _MAX_ITEMS_PER_AUTHOR,
) -> list[schema.Candidate]:
    """Keep at most *max_per_author* items from any single author.

    Candidates are assumed to already be sorted by quality (rrf_score etc.),
    so the first N encountered per author are the best ones.
    """
    author_counts: dict[str, int] = {}
    result: list[schema.Candidate] = []
    for c in candidates:
        author = _extract_author(c)
        if author is None:
            result.append(c)
            continue
        count = author_counts.get(author, 0)
        if count < max_per_author:
            result.append(c)
            author_counts[author] = count + 1
    return result


def _diversify_pool(
    fused: list[schema.Candidate],
    pool_limit: int,
    min_per_source: int = 2,
) -> list[schema.Candidate]:
    """Ensure at least *min_per_source* items per qualifying source survive truncation.

    Sources only qualify for reserved slots if their best item exceeds
    the relevance threshold. Low-relevance sources compete on merit only.
    """
    max_relevance: dict[str, float] = {}
    for c in fused:
        current = max_relevance.get(c.source, 0.0)
        if c.local_relevance > current:
            max_relevance[c.source] = c.local_relevance

    reserved: dict[str, list[schema.Candidate]] = {}
    remainder: list[schema.Candidate] = []
    for c in fused:
        qualifies = max_relevance.get(c.source, 0.0) >= _DIVERSITY_RELEVANCE_THRESHOLD
        bucket = reserved.setdefault(c.source, [])
        if qualifies and len(bucket) < min_per_source:
            bucket.append(c)
        else:
            remainder.append(c)
    pool = [c for per_source in reserved.values() for c in per_source]
    seen = {c.candidate_id for c in pool}
    for c in remainder:
        if len(pool) >= pool_limit:
            break
        if c.candidate_id not in seen:
            pool.append(c)
    pool.sort(key=_candidate_sort_key)
    return pool[:pool_limit]


def weighted_rrf(
    streams: dict[tuple[str, str], list[schema.SourceItem]],
    plan: schema.QueryPlan,
    *,
    pool_limit: int,
) -> list[schema.Candidate]:
    """Fuse ranked lists into a single candidate pool."""
    subqueries = {subquery.label: subquery for subquery in plan.subqueries}
    candidates: dict[str, schema.Candidate] = {}

    for (label, source), items in streams.items():
        subquery = subqueries[label]
        weight = subquery.weight * plan.source_weights.get(source, 1.0)
        for rank, item in enumerate(items, start=1):
            key = candidate_key(item)
            score = weight / (RRF_K + rank)
            item_local_relevance = item.local_relevance if item.local_relevance is not None else float(item.metadata.get("local_relevance", item.relevance_hint))
            item_freshness = item.freshness if item.freshness is not None else int(item.metadata.get("freshness", 0))
            item_source_quality = item.source_quality if item.source_quality is not None else float(item.metadata.get("source_quality", 0.6))
            if key not in candidates:
                candidates[key] = schema.Candidate(
                    candidate_id=key,
                    item_id=item.item_id,
                    source=item.source,
                    title=item.title,
                    url=item.url,
                    snippet=item.snippet,
                    subquery_labels=[label],
                    native_ranks={f"{label}:{source}": rank},
                    local_relevance=item_local_relevance,
                    freshness=item_freshness,
                    engagement=item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score"),
                    source_quality=item_source_quality,
                    rrf_score=score,
                    sources=[item.source],
                    source_items=[item],
                    metadata={
                        "provenance": [
                            {
                                "source": source,
                                "subquery_label": label,
                                "native_rank": rank,
                                "item_id": item.item_id,
                            }
                        ]
                    },
                )
                continue

            candidate = candidates[key]
            candidate.rrf_score += score
            previous_primary_score = (candidate.local_relevance * 100.0) + candidate.freshness + (candidate.source_quality * 10.0)
            incoming_primary_score = (item_local_relevance * 100.0) + item_freshness + (item_source_quality * 10.0)
            candidate.local_relevance = max(
                candidate.local_relevance,
                item_local_relevance,
            )
            candidate.freshness = max(candidate.freshness, item_freshness)
            item_eng = item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score")
            if candidate.engagement is None:
                candidate.engagement = item_eng
            elif item_eng is not None:
                candidate.engagement = max(candidate.engagement, item_eng)
            candidate.source_quality = max(
                candidate.source_quality,
                item_source_quality,
            )
            candidate.native_ranks[f"{label}:{source}"] = rank
            if label not in candidate.subquery_labels:
                candidate.subquery_labels.append(label)
            if item.source not in candidate.sources:
                candidate.sources.append(item.source)
            if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items):
                candidate.source_items.append(item)
            candidate.metadata.setdefault("provenance", []).append(
                {
                    "source": source,
                    "subquery_label": label,
                    "native_rank": rank,
                    "item_id": item.item_id,
                }
            )
            if incoming_primary_score > previous_primary_score:
                candidate.item_id = item.item_id
                candidate.source = item.source
                candidate.title = item.title
                candidate.snippet = item.snippet
            if len(candidate.snippet.split()) < len(item.snippet.split()):
                candidate.snippet = item.snippet

    fused = sorted(candidates.values(), key=_candidate_sort_key)
    fused = _apply_per_author_cap(fused)
    return _diversify_pool(fused, pool_limit)


path: scripts/lib/grounding.py
content: """Web search retrieval via the AIsa Tavily proxy."""

from __future__ import annotations

from urllib.parse import urlparse

from . import aisa, dates


def web_search(
    query: str,
    date_range: tuple[str, str],
    config: dict,
    backend: str = "auto",
) -> tuple[list[dict], dict]:
    """Run grounded web search through AIsa only."""
    if backend == "none":
        return [], {}
    if backend not in {"auto", "aisa"}:
        raise ValueError(f"Unsupported web backend: {backend!r}")
    key = config.get("AISA_API_KEY")
    if not key:
        if backend == "aisa":
            raise RuntimeError("AISA_API_KEY is required when web_backend='aisa'")
        return [], {}
    response = aisa.search_tavily(key, query, date_range)
    items, artifact = aisa.parse_tavily_response(response, date_range=date_range)
    if items:
        return items, artifact
    artifact = dict(artifact or {})
    artifact.setdefault("label", "aisa-tavily")
    artifact["warning"] = (
        "AISA Tavily search returned 0 items. Check whether this AISA/Tavily key has web-search access "
        "or whether the query/date window is too strict."
    )
    return items, artifact


def _normalize_date(value: object) -> str | None:
    if value is None:
        return None
    parsed = dates.parse_date(str(value).strip())
    if not parsed:
        return None
    return parsed.date().isoformat()


def _in_date_range(pub_date: str | None, date_range: tuple[str, str]) -> bool:
    if not pub_date:
        return False
    return date_range[0] <= pub_date <= date_range[1]


def _domain(url: str) -> str:
    return urlparse(url).netloc.strip().lower()


path: scripts/lib/hackernews.py
content: """Hacker News search via Algolia API (free, no auth required).

Uses hn.algolia.com/api/v1 for story discovery and comment enrichment.
No API key needed - just HTTP calls via stdlib urllib.
"""

import datetime
import html
import math
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional

import re

from . import http, log
from .query import extract_core_subject
from .relevance import token_overlap_relevance

# Common HN prefixes that can cause false-positive keyword matches
_HN_PREFIXES = re.compile(r"^(Tell HN|Show HN|Ask HN|Launch HN)\s*:\s*", re.IGNORECASE)

ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"

DEPTH_CONFIG = {
    "quick": 15,
    "default": 30,
    "deep": 60,
}

ENRICH_LIMITS = {
    "quick": 3,
    "default": 5,
    "deep": 10,
}


def _log(msg: str):
    log.source_log("HN", msg)


def _date_to_unix(date_str: str) -> int:
    """Convert YYYY-MM-DD to Unix timestamp (start of day UTC)."""
    parts = date_str.split("-")
    year, month, day = int(parts[0]), int(parts[1]), int(parts[2])
    dt = datetime.datetime(year, month, day, tzinfo=datetime.timezone.utc)
    return int(dt.timestamp())


def _unix_to_date(ts: int) -> str:
    """Convert Unix timestamp to YYYY-MM-DD."""
    dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
    return dt.strftime("%Y-%m-%d")


def _strip_html(text: str) -> str:
    """Strip HTML tags and decode entities from HN comment text."""
    import re
    text = html.unescape(text)
    text = re.sub(r'<p>', '\n', text)
    text = re.sub(r'<[^>]+>', '', text)
    return text.strip()


def search_hackernews(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
) -> Dict[str, Any]:
    """Search Hacker News via Algolia API.

    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'

    Returns:
        Dict with Algolia response (contains 'hits' list).
    """
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    from_ts = _date_to_unix(from_date)
    to_ts = _date_to_unix(to_date) + 86400  # Include the end date

    # Use extracted core subject instead of raw topic for cleaner Algolia matching
    core = extract_core_subject(topic)
    _log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")

    # Use relevance-sorted search with minimum engagement filter.
    # NOTE: restrictSearchableAttributes=title omitted intentionally — it would
    # miss Ask HN/Show HN threads where the topic appears in the body.
    params = {
        "query": core,
        "tags": "story",
        "numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
        "hitsPerPage": str(count),
    }

    from urllib.parse import urlencode
    url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"

    try:
        response = http.request("GET", url, timeout=30)
    except http.HTTPError as e:
        _log(f"Search failed: {e}")
        return {"hits": [], "error": str(e)}
    except Exception as e:
        _log(f"Search failed: {e}")
        return {"hits": [], "error": str(e)}

    hits = response.get("hits", [])
    _log(f"Found {len(hits)} stories")
    return response


def _title_matches_query(title: str, query: str, author: str = "") -> bool:
    """Check if the query term appears in the title content, not just an HN prefix or author.

    Returns True if the query (or any multi-word token) appears in the title
    after stripping "Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes
    and ignoring the author name.  Returns True when query is empty (no filter).
    """
    if not query:
        return True
    stripped = _HN_PREFIXES.sub("", title).strip()
    # Also check that the match isn't solely in the author's username
    check_text = stripped.lower()
    query_lower = query.lower()
    # Check each word of the query independently; all must appear somewhere
    # in the stripped title (not just the prefix).
    query_words = query_lower.split()
    for word in query_words:
        if word in check_text:
            continue
        # Word not found in stripped title — reject
        return False
    return True


def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
    """Parse Algolia response into normalized item dicts.

    Args:
        response: Algolia search response
        query: Original search query for token-overlap relevance scoring

    Returns:
        List of item dicts ready for normalization.
    """
    hits = response.get("hits", [])
    # Post-filter: remove items where query only matched an HN prefix like "Tell HN:"
    if query:
        before = len(hits)
        hits = [
            h for h in hits
            if _title_matches_query(h.get("title", ""), query, h.get("author", ""))
        ]
        dropped = before - len(hits)
        if dropped:
            _log(f"Prefix filter removed {dropped}/{before} false-positive hits for '{query}'")
    items = []

    for i, hit in enumerate(hits):
        object_id = hit.get("objectID", "")
        points = hit.get("points") or 0
        num_comments = hit.get("num_comments") or 0
        created_at_i = hit.get("created_at_i")

        date_str = None
        if created_at_i:
            date_str = _unix_to_date(created_at_i)

        # Article URL vs HN discussion URL
        article_url = hit.get("url") or ""
        hn_url = f"https://news.ycombinator.com/item?id={object_id}"

        # Relevance: blend Algolia rank with token-overlap content matching
        rank_score = max(0.3, 1.0 - (i * 0.02))  # 1.0 -> 0.3 over 35 items
        engagement_boost = min(0.2, math.log1p(points) / 40)
        if query:
            content_score = token_overlap_relevance(query, hit.get("title", ""))
            relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
        else:
            relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)

        items.append({
            "id": object_id,
            "title": hit.get("title", ""),
            "url": article_url,
            "hn_url": hn_url,
            "author": hit.get("author", ""),
            "date": date_str,
            "engagement": {
                "points": points,
                "comments": num_comments,
            },
            "relevance": round(relevance, 2),
            "why_relevant": f"HN story about {hit.get('title', 'topic')[:60]}",
        })

    return items


def _fetch_item_comments(object_id: str, max_comments: int = 5) -> Dict[str, Any]:
    """Fetch top-level comments for a story from Algolia items endpoint.

    Args:
        object_id: HN story ID
        max_comments: Max comments to return

    Returns:
        Dict with 'comments' list and 'comment_insights' list.
    """
    url = f"{ALGOLIA_ITEM_URL}/{object_id}"

    try:
        data = http.request("GET", url, timeout=15)
    except Exception as e:
        _log(f"Failed to fetch comments for {object_id}: {e}")
        return {"comments": [], "comment_insights": []}

    children = data.get("children", [])

    # Sort by points (highest first), filter to actual comments
    real_comments = [
        c for c in children
        if c.get("text") and c.get("author")
    ]
    real_comments.sort(key=lambda c: c.get("points") or 0, reverse=True)

    comments = []
    insights = []
    for c in real_comments[:max_comments]:
        text = _strip_html(c.get("text", ""))
        excerpt = text[:300] + "..." if len(text) > 300 else text
        comments.append({
            "author": c.get("author", ""),
            "text": excerpt,
            "points": c.get("points") or 0,
        })
        # First sentence as insight
        first_sentence = text.split(". ")[0].split("\n")[0][:200]
        if first_sentence:
            insights.append(first_sentence)

    return {"comments": comments, "comment_insights": insights}


def enrich_top_stories(
    items: List[Dict[str, Any]],
    depth: str = "default",
) -> List[Dict[str, Any]]:
    """Fetch comments for top N stories by points.

    Args:
        items: Parsed HN items
        depth: Research depth (controls how many to enrich)

    Returns:
        Items with top_comments and comment_insights added.
    """
    if not items:
        return items

    limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])

    # Sort by points to enrich the most popular stories
    by_points = sorted(
        range(len(items)),
        key=lambda i: items[i].get("engagement", {}).get("points", 0),
        reverse=True,
    )
    to_enrich = by_points[:limit]

    _log(f"Enriching top {len(to_enrich)} stories with comments")

    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = {
            executor.submit(
                _fetch_item_comments,
                items[idx]["id"],
            ): idx
            for idx in to_enrich
        }

        for future in as_completed(futures):
            idx = futures[future]
            try:
                result = future.result(timeout=15)
                items[idx]["top_comments"] = result["comments"]
                items[idx]["comment_insights"] = result["comment_insights"]
            except (KeyError, TypeError, OSError) as exc:
                _log(f"Comment enrichment failed for story {items[idx].get('id', '?')}: {type(exc).__name__}: {exc}")
                items[idx]["top_comments"] = []
                items[idx]["comment_insights"] = []

    return items


path: scripts/lib/log.py
content: """Shared logging utilities for last30days skill."""

import os
import sys

DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")


def debug(msg: str) -> None:
    """Log debug message to stderr (only when LAST30DAYS_DEBUG is set)."""
    if DEBUG:
        sys.stderr.write(f"[DEBUG] {msg}\n")
        sys.stderr.flush()


def source_log(prefix: str, msg: str, *, tty_only: bool = True) -> None:
    """Log a source module message to stderr.

    Args:
        prefix: Source label (e.g. "Reddit", "Bird").
        msg: Message text.
        tty_only: If True, only log when stderr is a TTY (avoids cluttering
                  non-interactive output like Claude Code).
    """
    if tty_only and not sys.stderr.isatty():
        return
    sys.stderr.write(f"[{prefix}] {msg}\n")
    sys.stderr.flush()


path: scripts/lib/__init__.py
content: # last30days library modules


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)