Skill: cnm-byd/unbrowse-openclaw (version 1.0.0)

--- name: unbrowse description: Analyze any website's network traffic and turn it into reusable API skills backed by a shared marketplace. Skills discovered by any agent are published, scored, and reusable by all agents. Capture network traffic, discover API endpoints, learn patterns, execute learned skills, and manage auth for gated sites. Use when someone wants to extract structured data from a website, discover API endpoints, automate web interactions, or work without official API documentation. user-invocable: true metadata: {"openclaw": {"requires": {"bins": ["bun"]}, "emoji": "🔍", "homepage": "https://github.com/unbrowse-ai/unbrowse"}} --- # Unbrowse — Drop-in Browser Replacement for Agents Browse once, cache the APIs, reuse them instantly. First call discovers and learns the site's APIs (~20-80s). Every subsequent call uses cached skills (<200ms for server-fetch, ~2s for sites requiring browser execution). **IMPORTANT: Always use the CLI (`bun src/cli.ts`). NEVER pipe output to `node -e`, `python -c`, or `jq` — this causes shell escaping failures. Use `--path`, `--extract`, and `--limit` flags instead.** ## Server Startup ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts health ``` If not running, the CLI auto-starts the server. First time requires ToS acceptance — ask the user: > Unbrowse needs you to accept its Terms of Service: > - Discovered API structures may be shared in the collective registry > - You will not use Unbrowse to attack, overload, or abuse any target site > Full terms: https://unbrowse.ai/terms After consent, the CLI handles startup automatically. First run also needs the browser engine: ```bash cd ~/.agents/skills/unbrowse && npx agent-browser install ``` ## Core Workflow ### Step 1: Resolve an intent ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts resolve \ --intent "get feed posts" \ --url "https://www.linkedin.com/feed/" \ --pretty ``` This returns `available_endpoints` — a ranked list of discovered API endpoints. Pick the right one by URL pattern (e.g., `MainFeed` for feed, `HomeTimeline` for tweets). ### Step 2: Execute with extraction ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts execute \ --skill {skill_id} \ --endpoint {endpoint_id} \ --path "data.included[]" \ --extract "author:actor.name.text,text:commentary.text.text,posted:actor.subDescription.text" \ --limit 20 \ --pretty ``` **This is the key pattern — `--path` + `--extract` + `--limit` replace ALL piping to jq/node/python.** ### Step 3: Submit feedback (MANDATORY) ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts feedback \ --skill {skill_id} \ --endpoint {endpoint_id} \ --rating 5 \ --outcome success ``` **Rating:** 5=right+fast, 4=right+slow(>5s), 3=incomplete, 2=wrong endpoint, 1=useless. ## Data Extraction Flags These flags eliminate the need to pipe output to any external parser: | Flag | Example | What it does | |------|---------|--------------| | `--path` | `"data.home.timeline.instructions[].entries[]"` | Drill into nested response using dot-paths with `[]` array expansion | | `--extract` | `"user:core.user.name,text:legacy.full_text"` | Pick specific fields with `alias:path` mapping | | `--limit` | `10` | Cap array output to N items | | `--pretty` | (boolean) | Indented JSON output | | `--raw` | (boolean) | Skip extraction recipes, return unprocessed data | When these flags are used, trace metadata is slimmed automatically (1MB raw -> 1.5KB output typical). ### Examples ```bash # X timeline — extract tweets with user, text, likes bun src/cli.ts execute --skill {id} --endpoint {id} \ --path "data.home.home_timeline_urt.instructions[].entries[].content.itemContent.tweet_results.result" \ --extract "user:core.user_results.result.legacy.screen_name,text:legacy.full_text,likes:legacy.favorite_count" \ --limit 20 --pretty # LinkedIn feed — extract posts from included[] bun src/cli.ts execute --skill {id} --endpoint {id} \ --path "data.included[]" \ --extract "author:actor.name.text,text:commentary.text.text,likes:socialDetail.totalSocialActivityCounts.numLikes" \ --limit 20 --pretty # Simple case — just limit results bun src/cli.ts execute --skill {id} --endpoint {id} --limit 10 --pretty ``` ## Extraction Recipes For responses you parse repeatedly, submit a recipe so future calls return clean data automatically (for ALL agents): ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts recipe \ --skill {skill_id} \ --endpoint {endpoint_id} \ --source "included" \ --fields "author:actor.name.text,text:commentary.text.text,posted:actor.subDescription.text" \ --require "commentary" \ --compact \ --description "Extract posts from LinkedIn feed" ``` When a recipe exists, future executions auto-return clean data. Use `--raw` to bypass recipes. | Recipe flag | Description | |-------------|-------------| | `--source "path"` | Dot-path to the source array | | `--fields "alias:path,..."` | Field mappings | | `--filter '{"field":"type","equals":"post"}'` | Filter array items (JSON) | | `--require "field1,field2"` | Required non-null fields | | `--compact` | Strip nulls and empty values | ## Authentication **Automatic.** Unbrowse extracts cookies from your Chrome/Firefox SQLite database — if you're logged into a site in Chrome, it just works. If `auth_required` is returned: ```bash cd ~/.agents/skills/unbrowse && bun src/cli.ts login --url "https://example.com/login" ``` User completes login in the browser window. Cookies are stored and reused automatically. ## Other Commands ```bash bun src/cli.ts skills # List all skills bun src/cli.ts skill {id} # Get skill details bun src/cli.ts search --intent "..." --domain "..." # Search marketplace bun src/cli.ts sessions --domain "linkedin.com" # Debug session logs bun src/cli.ts health # Server health check ``` ## Mutations Always `--dry-run` first, ask user before `--confirm-unsafe`: ```bash bun src/cli.ts execute --skill {id} --endpoint {id} --dry-run bun src/cli.ts execute --skill {id} --endpoint {id} --confirm-unsafe ``` ## REST API Reference For cases where the CLI doesn't cover your needs, the raw REST API is at `http://localhost:6969`: | Method | Endpoint | Description | |--------|----------|-------------| | POST | `/v1/intent/resolve` | Resolve intent -> search/capture/execute | | POST | `/v1/skills/:id/execute` | Execute a specific skill | | POST | `/v1/auth/login` | Interactive browser login | | POST | `/v1/feedback` | Submit feedback with diagnostics | | POST | `/v1/search` | Search marketplace globally | | POST | `/v1/search/domain` | Search marketplace by domain | | POST | `/v1/skills/:id/endpoints/:eid/recipe` | Submit extraction recipe | | GET | `/v1/skills/:id` | Get skill details | | GET | `/v1/sessions/:domain` | Debug session logs | | GET | `/health` | Health check | ## Rules 1. **Always use the CLI** — never pipe to `node -e`, `python -c`, or `jq`. Use `--path`/`--extract`/`--limit` instead. 2. Always try `resolve` first — it handles the full marketplace search -> capture pipeline 3. Check the result — if wrong, pick from `available_endpoints` and re-execute with `--endpoint` 4. If `auth_required`, use `login` then retry 5. Always `--dry-run` before mutations 6. **Always submit feedback after every resolve call — this is mandatory** 7. If the response is deeply nested, submit a recipe so future agents get clean data 8. **Report bugs and issues on GitHub** — when something breaks, is slow, or behaves unexpectedly, file an issue: ```bash gh issue create --repo unbrowse-ai/unbrowse \ --title "bug: {short description}" \ --body "## What happened\n{description}\n\n## Expected\n{what should have happened}\n\n## Context\n- Skill: {skill_id}\n- Endpoint: {endpoint_id}\n- Domain: {domain}\n- Error: {error message or status code}" ``` Categories: `bug:` (broken/wrong data), `perf:` (slow), `auth:` (login/cookie issues), `feat:` (missing capability)

path: package.json
content: {
  "name": "unbrowse-skill",
  "version": "1.0.0",
  "description": "Reverse-engineer any website into reusable API skills. Claude Code skill + local engine.",
  "type": "module",
  "main": "src/index.ts",
  "bin": {
    "unbrowse": "src/cli.ts"
  },
  "scripts": {
    "start": "bun src/index.ts",
    "dev": "bun --watch src/index.ts"
  },
  "dependencies": {
    "fastify": "^5.7.4",
    "@fastify/cors": "^11.2.0",
    "@fastify/rate-limit": "^10.3.0",
    "nanoid": "^5.1.6",
    "agent-browser": "^0.12.0"
  }
}


path: README.md
content: # unbrowse-skill

Analyze any website's network traffic and turn it into reusable API skills, backed by a shared marketplace. Skills discovered by any agent are available to all.

## Install

Clone and run the setup script — it installs deps, auto-registers as an agent, accepts ToS, and starts the server. No manual API key configuration needed:

```bash
git clone <repo-url> ~/.agents/skills/unbrowse
bash ~/.agents/skills/unbrowse/scripts/setup.sh
```

Credentials are auto-generated and cached in `~/.unbrowse/config.json`.

## Run

If the server isn't already running:

```bash
bash ~/.agents/skills/unbrowse/scripts/setup.sh
```

Or manually:

```bash
cd ~/.agents/skills/unbrowse
UNBROWSE_ACCEPT_TOS=1 bun src/index.ts
```

The server starts on `http://localhost:6969`.

## Usage with Claude Code

Install the `SKILL.md` as a Claude Code skill. It tells Claude how to call the local server's API to capture sites, discover endpoints, and execute learned skills. The skill expects the engine at `~/.agents/skills/unbrowse`.

## How it works

1. You provide a URL and intent (e.g. "get trending searches on Google")
2. The marketplace is searched for an existing skill matching your intent
3. If found, the skill executes immediately (50-200ms)
4. If not found, a headless browser navigates to the URL and records all network traffic
5. API endpoints are extracted, scored, and filtered from the traffic
6. A reusable "skill" is published to the shared marketplace with endpoint schemas
7. The skill is executed and results are returned
8. Future calls -- from any agent -- reuse the learned skill instantly

## Marketplace

Skills are stored in a shared marketplace at `beta-api.unbrowse.ai`. On first startup the server auto-registers as an agent and caches credentials in `~/.unbrowse/config.json`. Skills published by any agent are discoverable via semantic search by all agents.

See [SKILL.md](./SKILL.md) for the full API reference including search, feedback, and issue reporting.


path: src/api/routes.ts
content: import type { FastifyInstance } from "fastify";
import { TRACE_VERSION, CODE_HASH, GIT_SHA } from "../version.js";
import { resolveAndExecute } from "../orchestrator/index.js";
import { getSkill } from "../marketplace/index.js";
import { executeSkill } from "../execution/index.js";
import { storeCredential } from "../vault/index.js";
import { interactiveLogin, extractBrowserAuth } from "../auth/index.js";
import { publishSkill } from "../marketplace/index.js";
import { recordFeedback, recordDiagnostics, getApiKey } from "../client/index.js";
import { ROUTE_LIMITS } from "../ratelimit/index.js";
import type { ProjectionOptions, ExtractionRecipe } from "../types/index.js";
import { writeFileSync, existsSync, mkdirSync } from "fs";
import { join } from "path";

const BETA_API_URL = "https://beta-api.unbrowse.ai";

const TRACES_DIR = process.env.TRACES_DIR ?? join(process.cwd(), "traces");

export async function registerRoutes(app: FastifyInstance) {
  // Auth gate: block all routes except /health when no API key is configured
  app.addHook("onRequest", async (req, reply) => {
    if (req.url === "/health") return;

    const key = getApiKey();
    if (!key) {
      return reply.code(401).send({
        error: "api_key_required",
        message: "No API key configured. Restart the server to auto-register, or run: bash scripts/setup.sh",
        docs_url: "https://unbrowse.ai",
      });
    }
  });

  // POST /v1/intent/resolve
  app.post("/v1/intent/resolve", { config: { rateLimit: ROUTE_LIMITS["/v1/intent/resolve"] } }, async (req, reply) => {
    const { intent, params, context, projection, confirm_unsafe, dry_run, force_capture } = req.body as {
      intent: string;
      params?: Record<string, unknown>;
      context?: { url?: string; domain?: string };
      projection?: ProjectionOptions;
      confirm_unsafe?: boolean;
      dry_run?: boolean;
      force_capture?: boolean;
    };
    if (!intent) return reply.code(400).send({ error: "intent required" });
    try {
      const result = await resolveAndExecute(intent, params ?? {}, context, projection, { confirm_unsafe, dry_run, force_capture });

      // Surface timing breakdown
      const res = result as unknown as Record<string, unknown>;
      if (result.timing) {
        res.timing = result.timing;
      }

      // If the orchestrator already included available_endpoints in result (deferral),
      // also append them at the top level for backward compatibility.
      const innerResult = result.result as Record<string, unknown> | null;
      if (innerResult?.available_endpoints && !res.available_endpoints) {
        res.available_endpoints = innerResult.available_endpoints;
      }

      return reply.send(result);
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // GET /v1/skills/:skill_id — local route so skill lookups hit disk cache before proxying to backend
  app.get("/v1/skills/:skill_id", async (req, reply) => {
    const { skill_id } = req.params as { skill_id: string };
    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });
    return reply.send(skill);
  });

  // POST /v1/skills/:skill_id/execute
  app.post("/v1/skills/:skill_id/execute", { config: { rateLimit: ROUTE_LIMITS["/v1/skills/:skill_id/execute"] } }, async (req, reply) => {
    const { skill_id } = req.params as { skill_id: string };
    const { params, projection, confirm_unsafe, dry_run, intent } = req.body as {
      params?: Record<string, unknown>;
      projection?: ProjectionOptions;
      confirm_unsafe?: boolean;
      dry_run?: boolean;
      intent?: string;
    };
    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });
    try {
      const execResult = await executeSkill(skill, params ?? {}, projection, { confirm_unsafe, dry_run, intent });
      saveTrace(execResult.trace);

      // Auto-recovery: if endpoint returned 404 (stale), re-capture via orchestrator
      if (
        execResult.trace.status_code === 404 &&
        skill.domain &&
        skill.intent_signature &&
        skill.execution_type !== "browser-capture"
      ) {
        try {
          const freshResult = await resolveAndExecute(
            intent || skill.intent_signature,
            { ...(params ?? {}), url: `https://${skill.domain}` },
            { url: `https://${skill.domain}` },
            projection,
            { confirm_unsafe, dry_run, intent: intent || skill.intent_signature }
          );
          saveTrace(freshResult.trace);
          return reply.send({
            ...freshResult,
            _recovery: {
              reason: "stale_endpoint_404",
              original_skill_id: skill_id,
              message: "Original endpoint returned 404. Auto-recovered with fresh capture.",
            },
          });
        } catch {
          // Recovery failed — return original 404 with guidance
        }
      }

      return reply.send(execResult);
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // POST /v1/skills/:skill_id/auth -- store credentials (cookies/headers) for a skill
  app.post("/v1/skills/:skill_id/auth", async (req, reply) => {
    const { skill_id } = req.params as { skill_id: string };
    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });

    const body = req.body as {
      cookies?: Array<{ name: string; value: string; domain: string; path?: string }>;
      headers?: Record<string, string>;
    };
    if (!body.cookies && !body.headers) {
      return reply.code(400).send({ error: "Provide cookies or headers" });
    }

    const ref = `${skill.domain}-session`;
    await storeCredential(ref, JSON.stringify({ cookies: body.cookies ?? [], headers: body.headers ?? {} }));

    // Patch the skill manifest to reference the stored credentials
    if (!skill.auth_profile_ref) {
      await publishSkill({ ...skill, auth_profile_ref: ref });
    }

    return reply.send({ ok: true, auth_profile_ref: ref });
  });

  // POST /v1/auth/login — interactive OAuth flow or direct browser cookie extraction
  app.post("/v1/auth/login", { config: { rateLimit: ROUTE_LIMITS["/v1/auth/login"] } }, async (req, reply) => {
    const { url } = req.body as { url: string };
    if (!url) return reply.code(400).send({ error: "url required" });
    try {
      const result = await interactiveLogin(url);
      return reply.send(result);
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // POST /v1/auth/steal — extract cookies from Chrome/Firefox SQLite DBs.
  // No browser launch, Chrome can stay open. Higher rate limit since it's instant.
  app.post("/v1/auth/steal", { config: { rateLimit: { max: 30, timeWindow: "1 minute" } } }, async (req, reply) => {
    const { url, chrome_profile, firefox_profile } = req.body as {
      url: string;
      chrome_profile?: string;
      firefox_profile?: string;
    };
    if (!url) return reply.code(400).send({ error: "url required" });
    try {
      const domain = new URL(url).hostname;
      const result = await extractBrowserAuth(domain, {
        chromeProfile: chrome_profile,
        firefoxProfile: firefox_profile,
      });
      return reply.send(result);
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // POST /v1/skills/:skill_id/verify — trigger verification
  app.post("/v1/skills/:skill_id/verify", async (req, reply) => {
    const { skill_id } = req.params as { skill_id: string };
    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });
    try {
      const { verifySkill } = await import("../verification/index.js");
      const results = await verifySkill(skill);
      return reply.send({ skill_id, verification: results });
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // POST /v1/feedback — submit execution feedback with optional diagnostics
  app.post("/v1/feedback", async (req, reply) => {
    const { skill_id, target_id, endpoint_id, rating, outcome, diagnostics } = req.body as {
      skill_id?: string;
      target_id?: string;
      endpoint_id?: string;
      rating?: number;
      outcome?: string;
      diagnostics?: {
        total_ms?: number;
        bottleneck?: string;
        wrong_endpoint?: boolean;
        expected_data?: string;
        got_data?: string;
        trace_version?: string;
      };
    };
    const resolvedSkillId = skill_id ?? target_id;
    if (!resolvedSkillId || !endpoint_id || rating == null) {
      return reply.code(400).send({ error: "skill_id, endpoint_id, and rating required" });
    }
    try {
      const avg_rating = await recordFeedback(resolvedSkillId, endpoint_id, rating);
      // Forward diagnostics to backend for version-grouped analysis
      if (diagnostics) {
        recordDiagnostics(resolvedSkillId, endpoint_id, diagnostics).catch(() => {});
      }
      return reply.send({ ok: true, avg_rating });
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // POST /v1/skills/:skill_id/endpoints/:endpoint_id/recipe — submit extraction recipe
  app.post("/v1/skills/:skill_id/endpoints/:endpoint_id/recipe", async (req, reply) => {
    const { skill_id, endpoint_id } = req.params as { skill_id: string; endpoint_id: string };
    const { recipe } = req.body as { recipe: ExtractionRecipe };
    if (!recipe) return reply.code(400).send({ error: "recipe object required in body" });

    const { validateRecipe } = await import("../transform/recipe.js");
    const errors = validateRecipe(recipe);
    if (errors.length > 0) return reply.code(422).send({ error: "Invalid recipe", details: errors });

    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });

    const endpoint = skill.endpoints.find(e => e.endpoint_id === endpoint_id);
    if (!endpoint) return reply.code(404).send({ error: "Endpoint not found" });

    endpoint.extraction_recipe = { ...recipe, updated_at: new Date().toISOString() };
    skill.updated_at = new Date().toISOString();

    try {
      await publishSkill(skill);
      return reply.send({ ok: true, skill_id, endpoint_id, recipe: endpoint.extraction_recipe });
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // DELETE /v1/skills/:skill_id/endpoints/:endpoint_id/recipe — remove extraction recipe
  app.delete("/v1/skills/:skill_id/endpoints/:endpoint_id/recipe", async (req, reply) => {
    const { skill_id, endpoint_id } = req.params as { skill_id: string; endpoint_id: string };

    const skill = await getSkill(skill_id);
    if (!skill) return reply.code(404).send({ error: "Skill not found" });

    const endpoint = skill.endpoints.find(e => e.endpoint_id === endpoint_id);
    if (!endpoint) return reply.code(404).send({ error: "Endpoint not found" });

    delete endpoint.extraction_recipe;
    skill.updated_at = new Date().toISOString();

    try {
      await publishSkill(skill);
      return reply.send({ ok: true, message: "Recipe removed" });
    } catch (err) {
      return reply.code(500).send({ error: (err as Error).message });
    }
  });

  // GET /health
  app.get("/health", async (_req, reply) => reply.send({ status: "ok", trace_version: TRACE_VERSION, code_hash: CODE_HASH, git_sha: GIT_SHA }));

  // Catch-all proxy: forward unmatched /v1/* routes to beta-api.unbrowse.ai
  app.all("/v1/*", async (req, reply) => {
    const key = getApiKey();
    const upstream = `${BETA_API_URL}${req.url}`;
    const headers: Record<string, string> = { "Content-Type": "application/json" };
    if (key) headers["Authorization"] = `Bearer ${key}`;

    try {
      const res = await fetch(upstream, {
        method: req.method,
        headers,
        body: req.method !== "GET" && req.method !== "HEAD" ? JSON.stringify(req.body) : undefined,
      });
      const text = await res.text();
      try {
        return reply.code(res.status).send(JSON.parse(text));
      } catch {
        return reply.code(res.status).send({ error: text || `Upstream returned ${res.status}` });
      }
    } catch (err) {
      return reply.code(502).send({ error: `Proxy to beta-api failed: ${(err as Error).message}` });
    }
  });
}

function saveTrace(trace: unknown) {
  if (!existsSync(TRACES_DIR)) mkdirSync(TRACES_DIR, { recursive: true });
  const t = trace as { trace_id: string };
  writeFileSync(join(TRACES_DIR, `${t.trace_id}.json`), JSON.stringify(trace, null, 2));
}


path: src/auth/browser-cookies.ts
content: /**
 * Extract cookies directly from Chrome/Firefox SQLite databases.
 * Adapted from github.com/jawond/bird — generalized for any domain.
 *
 * Chrome cookies are AES-128-CBC encrypted with a key from the macOS keychain.
 * Firefox cookies are stored unencrypted.
 *
 * This avoids needing to launch a browser or close Chrome (reads a copy of the DB).
 */

import { execSync, execFileSync } from "node:child_process";
import { createDecipheriv, pbkdf2Sync } from "node:crypto";
import { copyFileSync, existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs";
import { tmpdir, homedir, platform } from "node:os";
import { join } from "node:path";
import { log } from "../logger.js";
import { getRegistrableDomain, isDomainMatch } from "../domain.js";

export interface BrowserCookie {
  name: string;
  value: string;
  domain: string;
  path: string;
  secure: boolean;
  httpOnly: boolean;
  sameSite: string;
  expires: number;
}

export interface ExtractionResult {
  cookies: BrowserCookie[];
  source: string | null;
  warnings: string[];
}

// ---------------------------------------------------------------------------
// Path helpers
// ---------------------------------------------------------------------------

function getChromeCookiesPath(profile?: string): string {
  const home = homedir();
  const profileDir = profile || "Default";
  if (platform() === "darwin") {
    return join(home, "Library", "Application Support", "Google", "Chrome", profileDir, "Cookies");
  }
  if (platform() === "win32") {
    const appData = process.env.LOCALAPPDATA ?? join(home, "AppData", "Local");
    return join(appData, "Google", "Chrome", "User Data", profileDir, "Cookies");
  }
  return join(home, ".config", "google-chrome", profileDir, "Cookies");
}

function getFirefoxProfilesRoot(): string | null {
  const home = homedir();
  if (platform() === "darwin") {
    return join(home, "Library", "Application Support", "Firefox", "Profiles");
  }
  if (platform() === "linux") {
    return join(home, ".mozilla", "firefox");
  }
  if (platform() === "win32") {
    const appData = process.env.APPDATA;
    if (!appData) return null;
    return join(appData, "Mozilla", "Firefox", "Profiles");
  }
  return null;
}

function pickFirefoxProfile(profilesRoot: string, profile?: string): string | null {
  if (profile) {
    const candidate = join(profilesRoot, profile, "cookies.sqlite");
    return existsSync(candidate) ? candidate : null;
  }
  const entries = readdirSync(profilesRoot, { withFileTypes: true });
  const defaultRelease = entries.find((e) => e.isDirectory() && e.name.includes("default-release"));
  const targetDir = defaultRelease?.name ?? entries.find((e) => e.isDirectory())?.name;
  if (!targetDir) return null;
  const candidate = join(profilesRoot, targetDir, "cookies.sqlite");
  return existsSync(candidate) ? candidate : null;
}

function getFirefoxCookiesPath(profile?: string): string | null {
  const profilesRoot = getFirefoxProfilesRoot();
  if (!profilesRoot || !existsSync(profilesRoot)) return null;
  return pickFirefoxProfile(profilesRoot, profile);
}

// ---------------------------------------------------------------------------
// Chrome decryption (macOS — uses keychain + PBKDF2 + AES-128-CBC)
// ---------------------------------------------------------------------------

let _chromeKeyCache: Buffer | null = null;

function getChromeDecryptionKey(): Buffer | null {
  if (_chromeKeyCache) return _chromeKeyCache;
  if (platform() !== "darwin") return null; // TODO: Linux/Windows support

  try {
    const keyOutput = execSync(
      'security find-generic-password -s "Chrome Safe Storage" -w 2>/dev/null || echo ""',
      { encoding: "utf8" },
    ).trim();
    if (!keyOutput) return null;

    _chromeKeyCache = pbkdf2Sync(keyOutput, "saltysalt", 1003, 16, "sha1");
    return _chromeKeyCache;
  } catch {
    return null;
  }
}

function decryptChromeValue(encryptedHex: string): string | null {
  try {
    const buf = Buffer.from(encryptedHex, "hex");
    if (buf.length < 4) return null;

    const version = buf.subarray(0, 3).toString("utf8");
    if (version !== "v10" && version !== "v11") {
      // Not encrypted
      return buf.toString("utf8");
    }

    const key = getChromeDecryptionKey();
    if (!key) return null;

    const payload = buf.subarray(3);

    // Modern Chrome (v131+) prepends a 32-byte header (key derivation nonce)
    // before the actual AES-128-CBC ciphertext.  The second 16-byte block of
    // the raw payload acts as the CBC IV for the remaining ciphertext.
    // Fallback: legacy format has no header (IV = 16 × 0x20 space bytes).
    if (payload.length >= 48) {
      try {
        const iv = payload.subarray(16, 32);
        const encrypted = payload.subarray(32);
        const decipher = createDecipheriv("aes-128-cbc", key, iv);
        decipher.setAutoPadding(true);
        const decrypted = Buffer.concat([decipher.update(encrypted), decipher.final()]);
        const val = decrypted.toString("utf8").replace(/[^\x20-\x7E]/g, "");
        if (val.length > 0) return val;
      } catch { /* fall through to legacy */ }
    }

    // Legacy format: IV = 16 bytes of space, ciphertext starts right after version
    const iv = Buffer.alloc(16, 0x20);
    const decipher = createDecipheriv("aes-128-cbc", key, iv);
    decipher.setAutoPadding(true);
    const decrypted = Buffer.concat([decipher.update(payload), decipher.final()]);
    return decrypted.toString("utf8").replace(/[^\x20-\x7E]/g, "");
  } catch {
    return null;
  }
}

// ---------------------------------------------------------------------------
// SQLite helpers — copy DB to temp dir, query, cleanup
// ---------------------------------------------------------------------------

function withTempCopy<T>(dbPath: string, fn: (tempPath: string) => T): T {
  const tempDir = mkdtempSync(join(tmpdir(), "unbrowse-cookies-"));
  const tempDb = join(tempDir, "cookies.db");
  try {
    copyFileSync(dbPath, tempDb);
    // Copy WAL/SHM so we get the latest committed state even while Chrome is open
    for (const ext of ["-wal", "-shm"]) {
      const src = dbPath + ext;
      if (existsSync(src)) copyFileSync(src, tempDb + ext);
    }
    return fn(tempDb);
  } finally {
    try { rmSync(tempDir, { recursive: true, force: true }); } catch { /* ignore */ }
  }
}

function sqliteQuery(dbPath: string, sql: string): string {
  return execFileSync("sqlite3", ["-separator", "|", dbPath, sql], {
    encoding: "utf8",
    maxBuffer: 4 * 1024 * 1024,
  }).trim();
}

// ---------------------------------------------------------------------------
// Domain matching helpers for SQL WHERE clauses
// ---------------------------------------------------------------------------

function buildDomainWhereClause(domain: string, column: string): string {
  const reg = getRegistrableDomain(domain);
  // Match exact domains: .example.com, example.com, plus common subdomains
  const variants = new Set([
    reg,
    `.${reg}`,
    domain,
    `.${domain}`,
    `www.${reg}`,
    `.www.${reg}`,
  ]);
  const escaped = [...variants].map((d) => `'${d.replace(/'/g, "''")}'`);
  // Also match any subdomain via LIKE (e.g. .api.example.com, .sg.example.com)
  const likePattern = `'%.${reg.replace(/'/g, "''")}'`;
  return `(${column} IN (${escaped.join(", ")}) OR ${column} LIKE ${likePattern})`;
}

// ---------------------------------------------------------------------------
// Chrome extraction
// ---------------------------------------------------------------------------

export function extractFromChrome(
  domain: string,
  opts?: { profile?: string },
): ExtractionResult {
  const warnings: string[] = [];
  const dbPath = getChromeCookiesPath(opts?.profile);

  if (!existsSync(dbPath)) {
    warnings.push(`Chrome cookies DB not found at ${dbPath}`);
    return { cookies: [], source: null, warnings };
  }

  try {
    const cookies = withTempCopy(dbPath, (tempDb) => {
      const where = buildDomainWhereClause(domain, "host_key");
      const sql = `SELECT name, hex(encrypted_value) as ev, host_key, path, is_secure, is_httponly, samesite, expires_utc FROM cookies WHERE ${where};`;
      const rows = sqliteQuery(tempDb, sql);
      if (!rows) return [];

      const results: BrowserCookie[] = [];
      for (const line of rows.split("\n")) {
        const parts = line.split("|");
        if (parts.length < 8) continue;
        const [name, encHex, host, cookiePath, secure, httpOnly, sameSite, expiresUtc] = parts;
        const value = decryptChromeValue(encHex);
        if (!value) continue;

        results.push({
          name,
          value,
          domain: host,
          path: cookiePath || "/",
          secure: secure === "1",
          httpOnly: httpOnly === "1",
          sameSite: sameSite === "0" ? "None" : sameSite === "1" ? "Lax" : "Strict",
          // Chrome stores expiry as microseconds since 1601-01-01
          expires: expiresUtc === "0" ? -1 : Math.floor(
            (Number(expiresUtc) - 11644473600000000) / 1000000
          ),
        });
      }
      return results;
    });

    const source = opts?.profile ? `Chrome profile "${opts.profile}"` : "Chrome default profile";
    if (cookies.length === 0) {
      warnings.push(`No cookies for ${domain} found in ${source}`);
    }
    log("auth", `extracted ${cookies.length} cookies for ${domain} from ${source}`);
    return { cookies, source: cookies.length > 0 ? source : null, warnings };
  } catch (err) {
    warnings.push(`Chrome extraction failed: ${err instanceof Error ? err.message : err}`);
    return { cookies: [], source: null, warnings };
  }
}

// ---------------------------------------------------------------------------
// Firefox extraction
// ---------------------------------------------------------------------------

export function extractFromFirefox(
  domain: string,
  opts?: { profile?: string },
): ExtractionResult {
  const warnings: string[] = [];
  const dbPath = getFirefoxCookiesPath(opts?.profile);

  if (!dbPath) {
    warnings.push("Firefox cookies DB not found");
    return { cookies: [], source: null, warnings };
  }

  try {
    const cookies = withTempCopy(dbPath, (tempDb) => {
      const where = buildDomainWhereClause(domain, "host");
      const sql = `SELECT name, value, host, path, isSecure, isHttpOnly, sameSite, expiry FROM moz_cookies WHERE ${where};`;
      const rows = sqliteQuery(tempDb, sql);
      if (!rows) return [];

      const results: BrowserCookie[] = [];
      for (const line of rows.split("\n")) {
        const parts = line.split("|");
        if (parts.length < 8) continue;
        const [name, value, host, cookiePath, secure, httpOnly, sameSite, expiry] = parts;
        if (!name || !value) continue;

        results.push({
          name,
          value,
          domain: host,
          path: cookiePath || "/",
          secure: secure === "1",
          httpOnly: httpOnly === "1",
          sameSite: sameSite === "0" ? "None" : sameSite === "1" ? "Lax" : "Strict",
          expires: Number(expiry) || -1,
        });
      }
      return results;
    });

    const source = opts?.profile ? `Firefox profile "${opts.profile}"` : "Firefox default profile";
    if (cookies.length === 0) {
      warnings.push(`No cookies for ${domain} found in ${source}`);
    }
    log("auth", `extracted ${cookies.length} cookies for ${domain} from ${source}`);
    return { cookies, source: cookies.length > 0 ? source : null, warnings };
  } catch (err) {
    warnings.push(`Firefox extraction failed: ${err instanceof Error ? err.message : err}`);
    return { cookies: [], source: null, warnings };
  }
}

// ---------------------------------------------------------------------------
// Unified extraction — tries Firefox first, then Chrome (bird's priority)
// ---------------------------------------------------------------------------

export function extractBrowserCookies(
  domain: string,
  opts?: { chromeProfile?: string; firefoxProfile?: string },
): ExtractionResult {
  // Try Firefox first (no decryption needed, more reliable)
  const ff = extractFromFirefox(domain, { profile: opts?.firefoxProfile });
  if (ff.cookies.length > 0) return ff;

  // Fall back to Chrome
  const chrome = extractFromChrome(domain, { profile: opts?.chromeProfile });
  chrome.warnings.push(...ff.warnings);
  return chrome;
}


path: src/auth/index.ts
content: import { BrowserManager } from "agent-browser/dist/browser.js";
import { executeCommand } from "agent-browser/dist/actions.js";
import { storeCredential, getCredential, deleteCredential } from "../vault/index.js";
import { nanoid } from "nanoid";
import { isDomainMatch, getRegistrableDomain } from "../domain.js";
import { log } from "../logger.js";
import path from "node:path";
import os from "node:os";
import fs from "node:fs";

const LOGIN_TIMEOUT_MS = 120_000;
const POLL_INTERVAL_MS = 1_000;

/**
 * Returns the persistent profile directory for a given domain.
 * Stored under ~/.unbrowse/profiles/<registrableDomain>.
 * Exporting so capture/execute can also launch with the profile if needed.
 */
export function getProfilePath(domain: string): string {
  return path.join(os.homedir(), ".unbrowse", "profiles", getRegistrableDomain(domain));
}
export interface LoginResult {
  success: boolean;
  domain: string;
  cookies_stored: number;
  error?: string;
}

/**
 * Open a visible browser for the user to complete login.
 * Waits up to 120s for navigation back to the target domain, then captures cookies.
 * Uses an isolated persistent profile per domain.
 */
export async function interactiveLogin(
  url: string,
  domain?: string,
): Promise<LoginResult> {
  const targetDomain = domain ?? new URL(url).hostname;
  const profileDir = getProfilePath(targetDomain);

  const browser = new BrowserManager();
  log("auth", `interactiveLogin — url: ${url}, domain: ${targetDomain}`);

  try {
    fs.mkdirSync(profileDir, { recursive: true });
    await browser.launch({
      action: "launch",
      id: nanoid(),
      headless: false,
      profile: profileDir,
      userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    });
    await executeCommand({ action: "navigate", id: nanoid(), url }, browser);

    const page = browser.getPage();
    const startTime = Date.now();

    // Wait for user to complete login — detect navigation back to target domain
    let loggedIn = false;
    let lastLoggedUrl = "";
    while (Date.now() - startTime < LOGIN_TIMEOUT_MS) {
      await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
      try {
        const currentUrl = page.url();
        const currentDomain = new URL(currentUrl).hostname.toLowerCase();
        const targetNorm = targetDomain.toLowerCase();

        if (currentUrl !== lastLoggedUrl) {
          log("auth", `navigated to: ${currentUrl}`);
          lastLoggedUrl = currentUrl;
        }

        const isOnTarget = currentDomain === targetNorm || currentDomain.endsWith("." + targetNorm);
        if (isOnTarget) {
          const isStillLogin = /\/(login|signin|sign-in|sso|auth|oauth|uas\/login|checkpoint)/.test(new URL(currentUrl).pathname);
          if (!isStillLogin) {
            loggedIn = true;
            log("auth", `login complete — ${currentUrl}`);
            break;
          }
        }
      } catch { /* page navigating */ }
    }

    if (!loggedIn) {
      return { success: false, domain: targetDomain, cookies_stored: 0, error: "Login timeout (120s)" };
    }

    // Extract and store cookies
    const context = browser.getContext();
    const cookies = context ? await context.cookies() : [];
    const domainCookies = cookies.filter((c) => isDomainMatch(c.domain, targetDomain));

    if (domainCookies.length === 0) {
      return { success: false, domain: targetDomain, cookies_stored: 0, error: "No cookies captured for domain" };
    }

    const storableCookies = domainCookies.map((c) => ({
      name: c.name, value: c.value, domain: c.domain, path: c.path,
      secure: c.secure, httpOnly: c.httpOnly, sameSite: c.sameSite, expires: c.expires,
    }));

    const vaultKey = `auth:${getRegistrableDomain(targetDomain)}`;
    await storeCredential(vaultKey, JSON.stringify({ cookies: storableCookies }));
    log("auth", `stored ${storableCookies.length} cookies under ${vaultKey}`);

    return { success: true, domain: targetDomain, cookies_stored: storableCookies.length };
  } finally {
    try {
      const context = browser.getContext();
      if (context) await Promise.race([context.close(), new Promise<void>((r) => setTimeout(r, 4000))]);
    } catch { /* ignore */ }
  }
}

/**
 * Extract cookies directly from Chrome/Firefox SQLite databases.
 * No browser launch needed, Chrome can stay open.
 * Stores extracted cookies in the vault for subsequent use.
 * Always stores under the registrable domain key for consistency.
 */
export async function extractBrowserAuth(
  domain: string,
  opts?: { chromeProfile?: string; firefoxProfile?: string }
): Promise<LoginResult> {
  const { extractBrowserCookies } = await import("./browser-cookies.js");

  const result = extractBrowserCookies(domain, opts);

  if (result.cookies.length === 0) {
    return {
      success: false,
      domain,
      cookies_stored: 0,
      error: result.warnings.join("; ") || "No cookies found in any browser",
    };
  }

  // Store in vault under same format as interactiveLogin
  const storableCookies = result.cookies.map((c) => ({
    name: c.name,
    value: c.value,
    domain: c.domain,
    path: c.path,
    secure: c.secure,
    httpOnly: c.httpOnly,
    sameSite: c.sameSite,
    expires: c.expires,
  }));

  // Normalize: always store under registrable domain for consistent lookups
  const vaultKey = `auth:${getRegistrableDomain(domain)}`;
  await storeCredential(
    vaultKey,
    JSON.stringify({ cookies: storableCookies })
  );

  log("auth", `stored ${storableCookies.length} cookies for ${domain} (key: ${vaultKey}) from ${result.source}`);
  return { success: true, domain, cookies_stored: storableCookies.length };
}

type AuthCookie = {
  name: string;
  value: string;
  domain: string;
  path?: string;
  secure?: boolean;
  httpOnly?: boolean;
  sameSite?: string;
  expires?: number;
};

/** Filter out expired cookies. Session cookies (expires <= 0) are kept. */
function filterExpired(cookies: AuthCookie[]): AuthCookie[] {
  const now = Math.floor(Date.now() / 1000);
  return cookies.filter((c) => {
    if (c.expires == null || c.expires <= 0) return true; // session cookie
    return c.expires > now;
  });
}

/**
 * Retrieve stored auth cookies for a domain from the vault.
 * Filters out expired cookies automatically.
 * Checks both registrable domain key and exact domain key for backward compat.
 */
export async function getStoredAuth(
  domain: string
): Promise<AuthCookie[] | null> {
  // Try registrable domain key first (new normalized format), then exact domain
  const regDomain = getRegistrableDomain(domain);
  const keysToTry = [`auth:${regDomain}`];
  if (domain !== regDomain) keysToTry.push(`auth:${domain}`);

  for (const key of keysToTry) {
    const stored = await getCredential(key);
    if (!stored) continue;
    try {
      const parsed = JSON.parse(stored) as { cookies?: AuthCookie[] };
      const cookies = parsed.cookies;
      if (!cookies || cookies.length === 0) continue;

      const valid = filterExpired(cookies);
      if (valid.length === 0) {
        log("auth", `all ${cookies.length} cookies for ${domain} (key: ${key}) are expired — deleting`);
        await deleteCredential(key);
        continue;
      }
      if (valid.length < cookies.length) {
        log("auth", `filtered ${cookies.length - valid.length} expired cookies for ${domain}`);
      }
      return valid;
    } catch {
      continue;
    }
  }
  return null;
}

/**
 * Bird-style unified cookie resolution with auto-extract fallback.
 *
 * Fallback chain:
 *   1. Vault cookies (fast path)
 *   2. Auto-extract from Chrome/Firefox SQLite (bird pattern — always fresh)
 *
 * This ensures cookies are available without requiring the user to manually
 * call /v1/auth/steal first.
 */
export async function getAuthCookies(
  domain: string
): Promise<AuthCookie[] | null> {
  // 1. Try vault (fast)
  const vaultCookies = await getStoredAuth(domain);
  if (vaultCookies && vaultCookies.length > 0) return vaultCookies;

  // 2. Auto-extract from browser (bird pattern)
  log("auth", `no vault cookies for ${domain} — auto-extracting from browser`);
  try {
    const result = await extractBrowserAuth(domain);
    if (result.success && result.cookies_stored > 0) {
      return getStoredAuth(domain);
    }
  } catch (err) {
    log("auth", `browser auto-extract failed for ${domain}: ${err instanceof Error ? err.message : err}`);
  }

  return null;
}

/**
 * Refresh credentials from browser after a 401/403.
 * Returns true if fresh cookies were stored.
 */
export async function refreshAuthFromBrowser(domain: string): Promise<boolean> {
  log("auth", `401/403 received — attempting to refresh auth for ${domain} from browser`);
  try {
    const result = await extractBrowserAuth(domain);
    if (result.success && result.cookies_stored > 0) {
      log("auth", `refreshed ${result.cookies_stored} cookies for ${domain} from browser`);
      return true;
    }
  } catch (err) {
    log("auth", `browser refresh failed for ${domain}: ${err instanceof Error ? err.message : err}`);
  }
  return false;
}




path: src/capture/index.ts
content: import { BrowserManager } from "agent-browser/dist/browser.js";
import { executeCommand } from "agent-browser/dist/actions.js";
import { nanoid } from "nanoid";
import { getRegistrableDomain } from "../domain.js";
import { log } from "../logger.js";

// BUG-GC-012: Use a real Chrome UA — HeadlessChrome is actively blocked by Google and others.
const CHROME_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";

// Browser launch semaphore: max 3 concurrent browsers
const MAX_CONCURRENT_BROWSERS = 3;
let activeBrowsers = 0;
const waitQueue: Array<() => void> = [];

// Active browser registry — tracked for graceful shutdown
const activeBrowserRegistry = new Set<InstanceType<typeof BrowserManager>>();

// Hard timeout per capture: 90s prevents stuck browsers from holding slots forever
const CAPTURE_TIMEOUT_MS = 90_000;

async function acquireBrowserSlot(): Promise<void> {
  if (activeBrowsers < MAX_CONCURRENT_BROWSERS) {
    activeBrowsers++;
    return;
  }
  return new Promise<void>((resolve) => {
    waitQueue.push(() => { activeBrowsers++; resolve(); });
  });
}

function releaseBrowserSlot(browser?: InstanceType<typeof BrowserManager>): void {
  if (browser) activeBrowserRegistry.delete(browser);
  activeBrowsers--;
  const next = waitQueue.shift();
  if (next) next();
}

/** Close all active browser instances — called on server shutdown. */
export async function shutdownAllBrowsers(): Promise<void> {
  await Promise.allSettled([...activeBrowserRegistry].map((b) => b.close()));
  activeBrowserRegistry.clear();
}

export interface CapturedWsMessage {
  url: string;
  direction: "sent" | "received";
  data: string;
  timestamp: string;
}

export interface CaptureResult {
  requests: RawRequest[];
  har_lineage_id: string;
  domain: string;
  cookies?: Array<{ name: string; value: string; domain: string; path?: string; httpOnly?: boolean; secure?: boolean }>;
  final_url: string;
  ws_messages?: CapturedWsMessage[];
  html?: string;
}

export interface RawRequest {
  url: string;
  method: string;
  request_headers: Record<string, string>;
  request_body?: string;
  response_status: number;
  response_headers: Record<string, string>;
  response_body?: string;
  timestamp: string;
}

/**
 * Extract a route hint keyword from a URL path for intent-aware API waiting.
 * e.g., "/i/bookmarks" → "bookmark", "/dashboard/analytics" → "analytic"
 * Returns null if the path is too generic (root, single char, common SPA prefixes).
 */
function extractRouteHint(url: string): string | null {
  try {
    const pathname = new URL(url).pathname;
    // Walk path segments from the end — the last meaningful segment is most specific
    const segments = pathname.split("/").filter(Boolean);
    // Skip generic SPA prefixes
    const GENERIC = /^(i|app|dashboard|page|view|en|es|fr|de|#|_|index|home|main)$/i;
    for (let i = segments.length - 1; i >= 0; i--) {
      const seg = segments[i];
      if (seg.length <= 2 || GENERIC.test(seg)) continue;
      // Return lowercased stem (strip trailing 's' for simple plural handling)
      return seg.toLowerCase().replace(/s$/, "");
    }
  } catch { /* bad URL */ }
  return null;
}

/**
 * Adaptive content-ready wait. Replaces flat 5s timeout.
 * Phase 1: 2s initial settle
 * Phase 2: If Cloudflare challenge detected, poll up to 15s for clearance
 * Phase 3: Wait for network idle (SPA API calls to complete)
 * Phase 4: Intent-aware API wait — if a route hint exists, wait for a matching API response
 * Worst case: 2 + 15 + 8 + 5 = 30s, well within CAPTURE_TIMEOUT_MS (90s).
 */
async function waitForContentReady(
  browser: InstanceType<typeof BrowserManager>,
  captureUrl?: string,
  capturedUrls?: Set<string>,
): Promise<void> {
  // Phase 1: Initial settle — let the page start rendering
  await new Promise((r) => setTimeout(r, 2000));

  // Phase 2: Cloudflare challenge detection and wait
  try {
    const page = browser.getPage();
    const hasCfChallenge = await page.evaluate(() => {
      const html = document.documentElement.innerHTML;
      return (
        html.includes("challenge-platform") ||
        html.includes("cf_chl_opt") ||
        document.title === "Just a moment..." ||
        !!document.querySelector("#challenge-running, #challenge-form, .cf-browser-verification")
      );
    });

    if (hasCfChallenge) {
      log("capture", "Cloudflare challenge detected, waiting for clearance...");
      const CF_POLL_INTERVAL = 1500;
      const CF_MAX_WAIT = 15000;
      const cfStart = Date.now();

      while (Date.now() - cfStart < CF_MAX_WAIT) {
        await new Promise((r) => setTimeout(r, CF_POLL_INTERVAL));
        const stillBlocked = await page.evaluate(() => {
          return (
            document.title === "Just a moment..." ||
            !!document.querySelector("#challenge-running, #challenge-form, .cf-browser-verification")
          );
        }).catch(() => false);

        if (!stillBlocked) {
          log("capture", `Cloudflare cleared after ${Date.now() - cfStart}ms`);
          break;
        }
      }
    }
  } catch {
    // Page not available — skip CF detection
  }

  // Phase 3: Wait for network idle (SPA API calls to settle)
  try {
    const page = browser.getPage();
    await page.waitForLoadState("networkidle", { timeout: 8000 });
  } catch {
    // networkidle timeout is non-fatal — some sites never fully idle
  }

  // Phase 4: Intent-aware API wait — catch SPA lazy-loaded route-specific APIs
  // SPAs like x.com fire route-specific GraphQL queries (e.g., Bookmarks) after
  // the initial homepage APIs. If we have a route hint, wait for a matching response.
  if (captureUrl && capturedUrls) {
    const hint = extractRouteHint(captureUrl);
    if (hint) {
      // Check if any already-captured URL contains the hint
      const alreadyHave = [...capturedUrls].some((u) => u.toLowerCase().includes(hint));
      if (!alreadyHave) {
        log("capture", `intent-aware wait: looking for API matching "${hint}" (from ${captureUrl})`);
        try {
          const page = browser.getPage();
          await new Promise<void>((resolve) => {
            const timeout = setTimeout(resolve, 15000);
            const handler = (response: { url(): string }) => {
              if (response.url().toLowerCase().includes(hint)) {
                log("capture", `intent-aware wait: matched ${response.url().substring(0, 120)}`);
                clearTimeout(timeout);
                page.off("response", handler);
                // Give the response body time to arrive
                setTimeout(resolve, 500);
              }
            };
            page.on("response", handler);
          });
        } catch {
          // Non-fatal — proceed with what we have
        }
      } else {
        log("capture", `intent-aware wait: already captured API matching "${hint}", skipping`);
      }
    }
  }
}

export async function captureSession(
  url: string,
  authHeaders?: Record<string, string>,
  cookies?: Array<{ name: string; value: string; domain: string; path?: string; secure?: boolean; httpOnly?: boolean; sameSite?: string; expires?: number }>
): Promise<CaptureResult> {
  await acquireBrowserSlot();
  const browser = new BrowserManager();
  activeBrowserRegistry.add(browser);
  let captureTimedOut = false;
  const timeoutHandle = setTimeout(async () => {
    captureTimedOut = true;
    await browser.close().catch(() => {});
  }, CAPTURE_TIMEOUT_MS);
  try {
  const domain = new URL(url).hostname;

  // Always ephemeral headless — cookies are injected by the caller (bird pattern).
  // Persistent profiles are only for interactiveLogin where the user is present.
  await browser.launch({ action: "launch", id: nanoid(), headless: true, userAgent: CHROME_UA });

  // Override Chromium's auto-set Client Hints headers — without this, Chromium 145+
  // sends sec-ch-ua: "HeadlessChrome" even when user-agent is spoofed to Chrome 131,
  // which is how LinkedIn/Google/Cloudflare detect headless browsers.
  const CLIENT_HINT_HEADERS: Record<string, string> = {
    "sec-ch-ua": '"Not_A Brand";v="8", "Chromium";v="131", "Google Chrome";v="131"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"macOS"',
  };
  await browser.setExtraHeaders({ ...CLIENT_HINT_HEADERS, ...(authHeaders ?? {}) });
  if (cookies && cookies.length > 0) {
    await injectCookies(browser, cookies);
  }

  await browser.startHarRecording();
  browser.startRequestTracking();

  // Hook page.on('response') BEFORE navigation to capture all response bodies
  // including XHR/fetch calls made during initial page load
  const responseBodies = new Map<string, string>();
  const MAX_BODY_SIZE = 512 * 1024; // 512KB
  try {
    const page = browser.getPage();
    page.on("response", async (response) => {
      try {
        const ct = response.headers()["content-type"] ?? "";
        const respUrl = response.url();
        // Capture JSON, protobuf, and batch RPC responses (Google batchexecute etc.)
        const isDataResponse =
          ct.includes("application/json") ||
          ct.includes("+json") ||
          ct.includes("application/x-protobuf") ||
          ct.includes("text/plain") ||
          respUrl.includes("batchexecute") ||
          respUrl.includes("/api/");
        if (!isDataResponse) return;
        // Skip static assets
        if (/\.(js|css|woff2?|png|jpg|svg|ico)(\?|$)/.test(respUrl)) return;
        const body = await response.body();
        if (body.length > MAX_BODY_SIZE) return;
        responseBodies.set(respUrl, body.toString("utf8"));
      } catch {
        // Response body may be unavailable for redirects/aborted
      }
    });
  } catch {
    // page not available — skip body capture
  }

  // CDP-based WebSocket capture
  const wsMessages: CapturedWsMessage[] = [];
  const wsUrlMap = new Map<string, string>(); // requestId -> url
  try {
    const page = browser.getPage();
    const cdp = await page.context().newCDPSession(page);
    await cdp.send("Network.enable");

    cdp.on("Network.webSocketCreated", (params: { requestId: string; url: string }) => {
      wsUrlMap.set(params.requestId, params.url);
    });

    cdp.on("Network.webSocketFrameReceived", (params: { requestId: string; timestamp: number; response: { payloadData: string } }) => {
      wsMessages.push({
        url: wsUrlMap.get(params.requestId) ?? params.requestId,
        direction: "received",
        data: params.response.payloadData,
        timestamp: new Date(params.timestamp * 1000).toISOString(),
      });
    });

    cdp.on("Network.webSocketFrameSent", (params: { requestId: string; timestamp: number; response: { payloadData: string } }) => {
      wsMessages.push({
        url: wsUrlMap.get(params.requestId) ?? params.requestId,
        direction: "sent",
        data: params.response.payloadData,
        timestamp: new Date(params.timestamp * 1000).toISOString(),
      });
    });
  } catch {
    // CDP session unavailable — skip WS capture
  }

  await executeCommand({ action: "navigate", id: nanoid(), url }, browser);

  // Adaptive wait: handle Cloudflare challenges + SPA content loading + intent-aware API wait
  const capturedUrlKeys = new Set(responseBodies.keys());
  await waitForContentReady(browser, url, capturedUrlKeys);

  // BUG-008: Share Cloudflare clearance cookies across subdomains
  try {
    const context = browser.getContext();
    if (context) {
      const allCookies = await context.cookies();
      const cfCookies = allCookies.filter(
        (c) => c.name === "__cf_bm" || c.name === "cf_clearance" || c.name.startsWith("__cf")
      );
      if (cfCookies.length > 0) {
        const baseDomain = getRegistrableDomain(new URL(url).hostname);
        const subdomainCookies = cfCookies.map((c) => ({
          ...c,
          domain: `.${baseDomain}`,
        }));
        try {
          await context.addCookies(subdomainCookies);
        } catch { /* CF cookie sharing is best-effort */ }
      }
    }
  } catch { /* context unavailable */ }

  const trackedRequests = browser.getRequests();
  const har_lineage_id = nanoid();

  // Debug: log all captured request URLs and response body map keys
  log("capture", `tracked ${trackedRequests.length} requests, ${responseBodies.size} response bodies`);
  for (const [bodyUrl] of responseBodies) {
    log("capture", `response body captured: ${bodyUrl.substring(0, 150)}`);
  }

  let final_url = url;
  let html: string | undefined;
  try {
    const page = browser.getPage();
    final_url = page.url();
    html = await page.content();
  } catch {}

  const requests: RawRequest[] = trackedRequests.map((r) => ({
    url: r.url,
    method: r.method,
    request_headers: r.headers,
    response_status: 0,
    response_headers: {},
    response_body: responseBodies.get(r.url),
    timestamp: new Date(r.timestamp).toISOString(),
  }));

  // Synthesize RawRequests for response bodies captured by the response listener
  // but missed by request tracking (e.g., SPA API calls during intent-aware wait).
  const trackedUrls = new Set(trackedRequests.map((r) => r.url));
  for (const [bodyUrl, body] of responseBodies) {
    if (!trackedUrls.has(bodyUrl)) {
      requests.push({
        url: bodyUrl,
        method: "GET",
        request_headers: {},
        response_status: 200,
        response_headers: {},
        response_body: body,
        timestamp: new Date().toISOString(),
      });
    }
  }

  // Extract session cookies so callers can persist auth for future executions
  const ctx = browser.getContext();
  const rawCookies = ctx ? await ctx.cookies().catch(() => []) : [];
  const sessionCookies = rawCookies.map((c) => ({
    name: c.name,
    value: c.value,
    domain: c.domain,
    path: c.path,
    httpOnly: c.httpOnly,
    secure: c.secure,
  }));

  if (captureTimedOut) throw new Error(`captureSession timed out after ${CAPTURE_TIMEOUT_MS}ms for ${url}`);
  return { requests, har_lineage_id, domain, cookies: sessionCookies.length > 0 ? sessionCookies : undefined, final_url, ws_messages: wsMessages.length > 0 ? wsMessages : undefined, html };
  } finally {
    clearTimeout(timeoutHandle);
    releaseBrowserSlot(browser);
  }
}

export async function executeInBrowser(
  url: string,
  method: string,
  requestHeaders: Record<string, string>,
  body?: unknown,
  authHeaders?: Record<string, string>,
  cookies?: Array<{ name: string; value: string; domain: string; path?: string; secure?: boolean; httpOnly?: boolean; sameSite?: string; expires?: number }>
): Promise<{ status: number; data: unknown; trace_id: string }> {
  await acquireBrowserSlot();
  const browser = new BrowserManager();
  activeBrowserRegistry.add(browser);
  try {
  await browser.launch({ action: "launch", id: nanoid(), headless: true, userAgent: CHROME_UA });

  const allHeaders = { ...authHeaders, ...requestHeaders };
  if (Object.keys(allHeaders).length > 0) {
    await browser.setExtraHeaders(allHeaders);
  }
  if (cookies && cookies.length > 0) {
    await injectCookies(browser, cookies);
  }

  // No auth: use in-page fetch (original path)
  browser.startRequestTracking();
  const origin = new URL(url).origin;
  await executeCommand({ action: "navigate", id: nanoid(), url: origin }, browser);

  const page = browser.getPage();
  const result = await page.evaluate(
    async ({ url, method, headers, body }: { url: string; method: string; headers: Record<string, string>; body: unknown }) => {
      const res = await fetch(url, {
        method,
        headers,
        ...(body ? { body: JSON.stringify(body) } : {}),
      });
      const text = await res.text();
      let data: unknown;
      try { data = JSON.parse(text); } catch { data = text; }
      return { status: res.status, data };
    },
    { url, method, headers: requestHeaders, body }
  );

  return { ...result, trace_id: nanoid() };
  } finally {
    releaseBrowserSlot(browser);
  }
}

/**
 * Trigger-and-intercept execution: navigate to the page that originally
 * triggered an API call, let the site's own JS make the request (passing
 * CSRF checks, TLS fingerprinting, etc.), and intercept the response.
 *
 * This is the generalized version of bird's approach:
 * - Bird works because Twitter doesn't do TLS fingerprinting
 * - Sites like LinkedIn do, so we let their JS make the call
 * - The site's own code handles CSRF tokens, session state, etc.
 */
export async function triggerAndIntercept(
  triggerUrl: string,
  targetUrlPattern: string,
  cookies: Array<{ name: string; value: string; domain: string; path?: string; secure?: boolean; httpOnly?: boolean; sameSite?: string; expires?: number }>,
  authHeaders?: Record<string, string>,
): Promise<{ status: number; data: unknown; trace_id: string }> {
  await acquireBrowserSlot();
  const browser = new BrowserManager();
  activeBrowserRegistry.add(browser);
  try {
    await browser.launch({ action: "launch", id: nanoid(), headless: true, userAgent: CHROME_UA });

    // Set extra headers (client hints to avoid headless detection)
    const headers: Record<string, string> = {
      "sec-ch-ua": '"Not_A Brand";v="8", "Chromium";v="131", "Google Chrome";v="131"',
      "sec-ch-ua-mobile": "?0",
      "sec-ch-ua-platform": '"macOS"',
      ...authHeaders,
    };
    await browser.setExtraHeaders(headers);
    await injectCookies(browser, cookies);

    const page = browser.getPage();

    // Build a URL matcher: strip template vars and match on the base path.
    // For graphql endpoints, preserve the queryId param so we match the specific query,
    // not just any /graphql call (e.g. voyagerFeedDashMainFeed vs voyagerMessagingDash...).
    const targetBase = targetUrlPattern.replace(/\{[^}]+\}/g, "").split("?")[0];
    // For graphql endpoints, extract the queryId name (before the hash) for matching.
    // e.g. "voyagerFeedDashMainFeed.923020905727c01..." → "voyagerFeedDashMainFeed"
    let targetQueryId: string | null = null;
    try {
      const tu = new URL(targetUrlPattern.replace(/\{[^}]+\}/g, "x"));
      const rawQueryId = tu.searchParams.get("queryId");
      // Strip hash suffix — keep only the semantic name for matching
      targetQueryId = rawQueryId ? rawQueryId.split(".")[0] : null;
    } catch { /* skip */ }

    // Set up response interception BEFORE navigation
    let captured: { status: number; data: unknown } | null = null;
    const interceptPromise = new Promise<{ status: number; data: unknown }>((resolve) => {
      const timeout = setTimeout(() => {
        resolve({ status: 0, data: { error: "trigger_timeout", message: "Target API call not intercepted within 15s" } });
      }, 15000);

      page.on("response", async (response) => {
        const respUrl = response.url();
        // Match if the response URL contains the target base path (and queryId if graphql)
        const baseMatch = respUrl.includes(targetBase);
        const queryIdMatch = !targetQueryId || respUrl.includes(targetQueryId);
        if (baseMatch && queryIdMatch && !captured) {
          captured = { status: response.status(), data: null };
          try {
            const body = await response.text();
            try { captured.data = JSON.parse(body); } catch { captured.data = body; }
          } catch {
            captured.data = null;
          }
          clearTimeout(timeout);
          resolve(captured);
        }
      });
    });

    // Navigate to the trigger page — the site's JS will make the API call.
    // Use domcontentloaded (not networkidle) — we only need the specific API call to fire,
    // not all network activity to settle. The interceptPromise resolves as soon as it's captured.
    log("capture", `trigger-and-intercept: navigating to ${triggerUrl}, waiting for ${targetBase}${targetQueryId ? `?queryId=${targetQueryId.slice(0, 40)}` : ""}`);
    page.goto(triggerUrl, { waitUntil: "domcontentloaded", timeout: 20000 }).catch(() => null);

    const result = await interceptPromise;
    log("capture", `trigger-and-intercept: got status ${result.status} for ${targetBase}`);
    return { ...result, trace_id: nanoid() };
  } finally {
    releaseBrowserSlot(browser);
  }
}

/**
 * Sanitize and inject cookies into browser context.
 * Strips all fields except { name, value, domain, path } to avoid
 * Playwright CDP protocol errors from unexpected cookie properties.
 * Falls back to per-cookie injection if batch fails.
 */
async function injectCookies(
  browser: InstanceType<typeof BrowserManager>,
  cookies: Array<{
    name: string;
    value: string;
    domain: string;
    path?: string;
    secure?: boolean;
    httpOnly?: boolean;
    sameSite?: string;
    expires?: number;
  }>
): Promise<void> {
  const context = browser.getContext();
  if (!context) return;

  const sanitized = cookies.map((c) => ({
    name: c.name,
    value: c.value,
    domain: c.domain.startsWith(".") ? c.domain : `.${c.domain}`,
    path: c.path ?? "/",
    ...(c.secure != null ? { secure: c.secure } : {}),
    ...(c.httpOnly != null ? { httpOnly: c.httpOnly } : {}),
    ...(c.sameSite != null ? { sameSite: c.sameSite as "Strict" | "Lax" | "None" } : {}),
    ...(c.expires != null && c.expires > 0 ? { expires: c.expires } : {}),
  }));

  log("capture", `injecting ${sanitized.length} cookies for domains: ${[...new Set(sanitized.map((c) => c.domain))].join(", ")}`);
  try {
    await context.addCookies(sanitized);
  } catch (batchErr) {
    log("capture", `batch cookie injection failed: ${batchErr instanceof Error ? batchErr.message : batchErr} — falling back to per-cookie`);
    let injected = 0;
    for (const cookie of sanitized) {
      try {
        await context.addCookies([cookie]);
        injected++;
      } catch (err) {
        log("capture", `failed to inject cookie "${cookie.name}" for ${cookie.domain}: ${err instanceof Error ? err.message : err}`);
      }
    }
    log("capture", `per-cookie fallback: ${injected}/${sanitized.length} injected`);
  }
}


path: src/cli.ts
content: #!/usr/bin/env bun
/**
 * Unbrowse CLI — shell-safe wrapper for the local API.
 * Eliminates curl + jq escaping issues. All JSON is constructed
 * and parsed in TypeScript, never through shell interpolation.
 *
 * Usage: unbrowse <command> [flags]
 */

const BASE_URL = process.env.UNBROWSE_URL || "http://localhost:6969";

// ---------------------------------------------------------------------------
// Arg parser
// ---------------------------------------------------------------------------

export function parseArgs(argv: string[]): { command: string; args: string[]; flags: Record<string, string | boolean> } {
  const raw = argv.slice(2); // skip runtime + script
  const command = raw[0] && !raw[0].startsWith("--") ? raw[0] : "help";
  const rest = command === "help" ? raw : raw.slice(1);
  const positional: string[] = [];
  const flags: Record<string, string | boolean> = {};
  for (let i = 0; i < rest.length; i++) {
    const a = rest[i];
    if (a.startsWith("--")) {
      const key = a.slice(2);
      const next = rest[i + 1];
      if (!next || next.startsWith("--")) {
        flags[key] = true;
      } else {
        flags[key] = next;
        i++;
      }
    } else {
      positional.push(a);
    }
  }
  return { command, args: positional, flags };
}

// ---------------------------------------------------------------------------
// HTTP helpers
// ---------------------------------------------------------------------------

async function api(method: string, path: string, body?: unknown): Promise<unknown> {
  const res = await fetch(`${BASE_URL}${path}`, {
    method,
    headers: body ? { "Content-Type": "application/json" } : undefined,
    body: body ? JSON.stringify(body) : undefined,
  });
  if (!res.ok && res.headers.get("content-type")?.includes("json")) {
    return res.json();
  }
  if (!res.ok) {
    return { error: `HTTP ${res.status}: ${await res.text()}` };
  }
  return res.json();
}

function output(data: unknown, pretty = false): void {
  process.stdout.write((pretty ? JSON.stringify(data, null, 2) : JSON.stringify(data)) + "\n");
}

function die(msg: string): never {
  output({ error: msg });
  process.exit(1);
}

function info(msg: string): void {
  process.stderr.write(`[unbrowse] ${msg}\n`);
}

// ---------------------------------------------------------------------------
// Path resolution — drill into nested structures with [] array expansion
// ---------------------------------------------------------------------------

/** Resolve a dot-path like "data.items[].name" against an object. */
function resolvePath(obj: unknown, path: string): unknown {
  if (!path || obj == null) return obj;
  const segments = path.split(".");
  let cur: unknown = obj;
  for (let i = 0; i < segments.length; i++) {
    if (cur == null) return undefined;
    const seg = segments[i];
    if (seg.endsWith("[]")) {
      const key = seg.slice(0, -2);
      const arr = key ? (cur as Record<string, unknown>)[key] : cur;
      if (!Array.isArray(arr)) return undefined;
      const remaining = segments.slice(i + 1).join(".");
      if (!remaining) return arr;
      return arr.flatMap((item) => {
        const v = resolvePath(item, remaining);
        return v === undefined ? [] : Array.isArray(v) ? v : [v];
      });
    }
    cur = (cur as Record<string, unknown>)[seg];
  }
  return cur;
}

/** Apply --extract fields to data. Each field is "alias:deep.path" or just "field".
 *  When processing arrays, rows where ALL extracted fields are null/undefined are dropped.
 *  This handles decorator-pattern APIs (e.g. LinkedIn included[]) where heterogeneous
 *  item types coexist and only some items match the requested fields. */
function extractFields(data: unknown, fields: string[]): unknown {
  if (data == null) return data;

  function mapItem(item: unknown): Record<string, unknown> {
    const out: Record<string, unknown> = {};
    for (const f of fields) {
      const colonIdx = f.indexOf(":");
      const alias = colonIdx >= 0 ? f.slice(0, colonIdx) : f.split(".").pop()!;
      const path = colonIdx >= 0 ? f.slice(colonIdx + 1) : f;
      out[alias] = resolvePath(item, path);
    }
    return out;
  }

  if (Array.isArray(data)) {
    return data.map(mapItem).filter((row) => Object.values(row).some((v) => v != null));
  }
  return mapItem(data);
}

/** Apply --path, --extract, --limit to a result object. */
function applyTransforms(result: unknown, flags: Record<string, string | boolean>): unknown {
  let data = result;

  // --path: drill into nested structure
  const pathFlag = flags.path as string | undefined;
  if (pathFlag) {
    data = resolvePath(data, pathFlag);
  }

  // --extract: pick specific fields
  const extractFlag = flags.extract as string | undefined;
  if (extractFlag) {
    const fields = extractFlag.split(",").map((f) => f.trim());
    data = extractFields(data, fields);
  }

  // --limit: cap array output
  const limitFlag = flags.limit as string | undefined;
  if (limitFlag && Array.isArray(data)) {
    data = data.slice(0, Number(limitFlag));
  }

  return data;
}

/** Slim down trace when transforms are applied — keep only essential metadata. */
function slimTrace(obj: Record<string, unknown>): Record<string, unknown> {
  const trace = obj.trace as Record<string, unknown> | undefined;
  if (!trace) return obj;
  return {
    ...obj,
    trace: {
      trace_id: trace.trace_id,
      skill_id: trace.skill_id,
      endpoint_id: trace.endpoint_id,
      success: trace.success,
      status_code: trace.status_code,
      trace_version: trace.trace_version,
    },
  };
}

// ---------------------------------------------------------------------------
// Server lifecycle
// ---------------------------------------------------------------------------

async function ensureServer(noAutoStart: boolean): Promise<void> {
  try {
    const res = await fetch(`${BASE_URL}/health`, { signal: AbortSignal.timeout(2000) });
    if (res.ok) return;
  } catch { /* not running */ }

  if (noAutoStart) die("Server not running. Start with: cd ~/.agents/skills/unbrowse && bun src/index.ts");

  info("Server not running. Starting...");
  const skillDir = process.env.SKILL_DIR ?? `${process.env.HOME}/.agents/skills/unbrowse`;
  const { spawn } = await import("child_process");
  spawn("bun", ["src/index.ts"], {
    cwd: skillDir,
    detached: true,
    stdio: ["ignore", "ignore", "ignore"],
    env: { ...process.env, UNBROWSE_NON_INTERACTIVE: "1", UNBROWSE_TOS_ACCEPTED: "1" },
  }).unref();

  for (let i = 0; i < 15; i++) {
    await new Promise((r) => setTimeout(r, 1000));
    try {
      const res = await fetch(`${BASE_URL}/health`, { signal: AbortSignal.timeout(2000) });
      if (res.ok) { info("Server ready."); return; }
    } catch { /* keep polling */ }
  }
  die("Server failed to start. Check /tmp/unbrowse.log");
}

// ---------------------------------------------------------------------------
// Commands
// ---------------------------------------------------------------------------

async function cmdHealth(flags: Record<string, string | boolean>): Promise<void> {
  output(await api("GET", "/health"), !!flags.pretty);
}

async function cmdResolve(flags: Record<string, string | boolean>): Promise<void> {
  const intent = flags.intent as string;
  if (!intent) die("--intent is required");

  const body: Record<string, unknown> = { intent };
  const url = flags.url as string | undefined;
  const domain = flags.domain as string | undefined;

  if (url) {
    body.params = { url };
    body.context = { url };
  }
  if (domain) {
    body.context = { ...(body.context as Record<string, unknown> ?? {}), domain };
  }
  if (flags["endpoint-id"]) {
    body.params = { ...(body.params as Record<string, unknown> ?? {}), endpoint_id: flags["endpoint-id"] };
  }
  if (flags.params) {
    body.params = { ...(body.params as Record<string, unknown> ?? {}), ...JSON.parse(flags.params as string) };
  }
  if (flags["dry-run"]) body.dry_run = true;
  if (flags["force-capture"]) body.force_capture = true;
  if (flags.raw) body.projection = { raw: true };

  let result = await api("POST", "/v1/intent/resolve", body) as Record<string, unknown>;

  // --path / --extract / --limit: transform .result in-place
  const hasTransforms = !!(flags.path || flags.extract || flags.limit);
  if (hasTransforms && result.result != null) {
    result = slimTrace({ ...result, result: applyTransforms(result.result, flags) });
  }

  // Append CLI hint for feedback
  const skill = result.skill as Record<string, unknown> | undefined;
  const trace = result.trace as Record<string, unknown> | undefined;
  if (skill?.skill_id && trace) {
    (result as Record<string, unknown>)._feedback = `unbrowse feedback --skill ${skill.skill_id} --endpoint ${trace.endpoint_id || "?"} --rating <1-5>`;
  }

  output(result, !!flags.pretty);
}

async function cmdExecute(flags: Record<string, string | boolean>): Promise<void> {
  const skillId = flags.skill as string;
  if (!skillId) die("--skill is required");

  const body: Record<string, unknown> = { params: {} };
  if (flags.endpoint) {
    (body.params as Record<string, unknown>).endpoint_id = flags.endpoint;
  }
  if (flags.params) {
    body.params = { ...(body.params as Record<string, unknown>), ...JSON.parse(flags.params as string) };
  }
  if (flags["dry-run"]) body.dry_run = true;
  if (flags["confirm-unsafe"]) body.confirm_unsafe = true;
  if (flags.raw) body.projection = { raw: true };

  let result = await api("POST", `/v1/skills/${skillId}/execute`, body) as Record<string, unknown>;

  // --path / --extract / --limit: transform .result in-place
  const hasTransforms = !!(flags.path || flags.extract || flags.limit);
  if (hasTransforms && result.result != null) {
    result = slimTrace({ ...result, result: applyTransforms(result.result, flags) });
  }

  output(result, !!flags.pretty);
}

async function cmdFeedback(flags: Record<string, string | boolean>): Promise<void> {
  const skillId = flags.skill as string;
  const endpointId = flags.endpoint as string;
  const rating = Number(flags.rating);
  if (!skillId || !endpointId || !rating) die("--skill, --endpoint, and --rating are required");

  const body: Record<string, unknown> = {
    skill_id: skillId,
    endpoint_id: endpointId,
    rating,
  };
  if (flags.outcome) body.outcome = flags.outcome;
  if (flags.diagnostics) body.diagnostics = JSON.parse(flags.diagnostics as string);

  output(await api("POST", "/v1/feedback", body), !!flags.pretty);
}

async function cmdLogin(flags: Record<string, string | boolean>): Promise<void> {
  const url = flags.url as string;
  if (!url) die("--url is required");
  output(await api("POST", "/v1/auth/login", { url }), !!flags.pretty);
}

async function cmdSkills(flags: Record<string, string | boolean>): Promise<void> {
  output(await api("GET", "/v1/skills"), !!flags.pretty);
}

async function cmdSkill(args: string[], flags: Record<string, string | boolean>): Promise<void> {
  const id = args[0] ?? flags.id as string;
  if (!id) die("skill <id> or --id required");
  output(await api("GET", `/v1/skills/${id}`), !!flags.pretty);
}

async function cmdSearch(flags: Record<string, string | boolean>): Promise<void> {
  const intent = flags.intent as string;
  if (!intent) die("--intent is required");
  const domain = flags.domain as string | undefined;
  const path = domain ? "/v1/search/domain" : "/v1/search";
  const body: Record<string, unknown> = { intent, k: Number(flags.k) || 5 };
  if (domain) body.domain = domain;
  output(await api("POST", path, body), !!flags.pretty);
}

async function cmdRecipe(flags: Record<string, string | boolean>): Promise<void> {
  const skillId = flags.skill as string;
  const endpointId = flags.endpoint as string;
  if (!skillId || !endpointId) die("--skill and --endpoint are required");

  const recipe: Record<string, unknown> = {};
  if (flags.source) recipe.source = flags.source;
  if (flags.fields) {
    // Parse "alias:path,alias:path" into { alias: "path" }
    const fields: Record<string, string> = {};
    for (const pair of (flags.fields as string).split(",")) {
      const [key, ...rest] = pair.trim().split(":");
      fields[key] = rest.join(":") || key;
    }
    recipe.fields = fields;
  }
  if (flags.filter) recipe.filter = JSON.parse(flags.filter as string);
  if (flags.require) recipe.require = (flags.require as string).split(",");
  if (flags.compact) recipe.compact = true;
  if (flags.description) recipe.description = flags.description;

  output(await api("POST", `/v1/skills/${skillId}/endpoints/${endpointId}/recipe`, { recipe }), !!flags.pretty);
}

async function cmdSessions(flags: Record<string, string | boolean>): Promise<void> {
  const domain = flags.domain as string;
  if (!domain) die("--domain is required");
  const limit = flags.limit ?? "10";
  output(await api("GET", `/v1/sessions/${domain}?limit=${limit}`), !!flags.pretty);
}

// ---------------------------------------------------------------------------
// Help
// ---------------------------------------------------------------------------

function printHelp(): void {
  const help = `unbrowse — shell-safe CLI for the local API

Commands:
  health                                       Server health check
  resolve  --intent "..." --url "..." [opts]   Resolve intent → search/capture/execute
  execute  --skill ID --endpoint ID [opts]     Execute a specific endpoint
  feedback --skill ID --endpoint ID --rating N Submit feedback (mandatory after resolve)
  login    --url "..."                         Interactive browser login
  skills                                       List all skills
  skill    <id>                                Get skill details
  search   --intent "..." [--domain "..."]     Search marketplace
  recipe   --skill ID --endpoint ID [opts]     Submit extraction recipe
  sessions --domain "..." [--limit N]          Debug session logs

Global flags:
  --pretty          Indented JSON output
  --no-auto-start   Don't auto-start server
  --raw             Skip extraction recipes, return raw data

resolve/execute flags:
  --path "data.items[]"                       Drill into result before extract/output
  --extract "field1,alias:deep.path.to.val"   Pick specific fields (no piping needed)
  --limit N                                   Cap array output to N items
  --endpoint-id ID                            Pick a specific endpoint
  --dry-run                                   Preview mutations
  --force-capture                             Bypass caches, re-capture
  --params '{...}'                            Extra params as JSON

recipe flags:
  --source "data.items"                       Dot-path to source array
  --fields "name:user.name,text:body.text"    Field mappings (alias:path)
  --filter '{"field":"type","equals":"post"}' Filter criteria as JSON
  --require "id,text"                         Required non-null fields
  --compact                                   Strip nulls/empties
  --description "..."                         Human description

Examples:
  unbrowse resolve --intent "get timeline" --url "https://x.com"
  unbrowse execute --skill abc --endpoint def --pretty
  unbrowse execute --skill abc --endpoint def --extract "user,text,likes" --limit 10
  unbrowse execute --skill abc --endpoint def --path "data.included[]" --extract "name:actor.name,text:commentary.text" --limit 20
  unbrowse feedback --skill abc --endpoint def --rating 5
  unbrowse recipe --skill abc --endpoint def --source "included" --fields "author:actor.name,text:commentary.text" --compact
`;
  process.stderr.write(help);
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------

async function main(): Promise<void> {
  const { command, args, flags } = parseArgs(process.argv);
  const pretty = !!flags.pretty;
  const noAutoStart = !!flags["no-auto-start"];

  if (command === "help" || flags.help) {
    printHelp();
    process.exit(0);
  }

  // Health check doesn't need auto-start
  if (command !== "health") {
    await ensureServer(noAutoStart);
  }

  switch (command) {
    case "health": return cmdHealth(flags);
    case "resolve": return cmdResolve(flags);
    case "execute": case "exec": return cmdExecute(flags);
    case "feedback": case "fb": return cmdFeedback(flags);
    case "login": return cmdLogin(flags);
    case "skills": return cmdSkills(flags);
    case "skill": return cmdSkill(args, flags);
    case "search": return cmdSearch(flags);
    case "recipe": return cmdRecipe(flags);
    case "sessions": return cmdSessions(flags);
    default: info(`Unknown command: ${command}`); printHelp(); process.exit(1);
  }
}

main().catch((err) => {
  die((err as Error).message);
});


path: src/client/index.ts
content: import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync } from "fs";
import { join } from "path";
import { homedir, hostname } from "os";
import { randomBytes } from "crypto";
import { createInterface } from "readline";
import type { EndpointStats, ExecutionTrace, OrchestrationTiming, SkillManifest, ValidationResult } from "../types/index.js";

const API_URL = "https://beta-api.unbrowse.ai";
const CONFIG_DIR = join(homedir(), ".unbrowse");
const CONFIG_PATH = join(CONFIG_DIR, "config.json");
const SKILL_CACHE_DIR = join(CONFIG_DIR, "skill-cache");

interface UnbrowseConfig {
  [REDACTED_SECRET];
  agent_id: string;
  agent_name: string;
  registered_at: string;
  tos_accepted_version: string | null;
  tos_accepted_at: string | null;
}

function loadConfig(): UnbrowseConfig | null {
  try {
    if (existsSync(CONFIG_PATH)) {
      return JSON.parse(readFileSync(CONFIG_PATH, "utf-8"));
    }
  } catch { /* corrupt file, re-register */ }
  return null;
}

function saveConfig(config: UnbrowseConfig): void {
  if (!existsSync(CONFIG_DIR)) mkdirSync(CONFIG_DIR, { recursive: true });
  writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2), { mode: 0o600 });
}

export function getApiKey(): string {
  // Env var takes priority, then cached config
  if (process.env.UNBROWSE_API_KEY) return process.env.UNBROWSE_API_KEY;
  const config = loadConfig();
  if (config?.api_key) {
    process.env.UNBROWSE_API_KEY = config.api_key;
    return config.api_key;
  }
  return "";
}

async function api<T = unknown>(method: string, path: string, body?: unknown): Promise<T> {
  const key = getApiKey();
  const res = await fetch(`${API_URL}${path}`, {
    method,
    headers: {
      "Content-Type": "application/json",
      ...(key ? { Authorization: `Bearer ${key}` } : {}),
    },
    body: body ? JSON.stringify(body) : undefined,
  });
  let data: T & { error?: string };
  try {
    data = await res.json() as T & { error?: string };
  } catch {
    // Backend returned a non-JSON response (e.g. CF Worker error page)
    throw new Error(`API error ${res.status} from ${path}`);
  }

  // Handle ToS update required — tell user to restart
  if (res.status === 403 && (data as Record<string, unknown>).error === "tos_update_required") {
    console.warn("\n[unbrowse] The Terms of Service have been updated.");
    console.warn("[unbrowse] Please restart the unbrowse service to accept the new terms.");
    throw new Error("ToS update required. Restart unbrowse to accept new terms.");
  }

  if (!res.ok) throw new Error((data as { error?: string }).error ?? `API HTTP ${res.status}`);
  return data;
}

// --- ToS acceptance ---

async function promptTosAcceptance(summary: string, tosUrl: string): Promise<boolean> {
  // Non-interactive mode: skip the readline prompt, return false.
  // The calling agent is expected to show the ToS to the user and ask for consent,
  // then re-run with UNBROWSE_TOS_ACCEPTED=1 after the user agrees.
  if (process.env.UNBROWSE_NON_INTERACTIVE === "1") {
    if (process.env.UNBROWSE_TOS_ACCEPTED === "1") {
      console.log("[unbrowse] ToS accepted by user via agent.");
      return true;
    }
    console.log("[unbrowse] ToS acceptance required. Set UNBROWSE_TOS_ACCEPTED=1 after user consents.");
    console.log(`[unbrowse] ToS summary:\n${summary}`);
    console.log(`[unbrowse] Full terms: ${tosUrl}`);
    return false;
  }

  const rl = createInterface({ input: process.stdin, output: process.stdout });

  console.log("\n" + "=".repeat(60));
  console.log("UNBROWSE TERMS OF SERVICE");
  console.log("=".repeat(60));
  console.log(summary);
  console.log("=".repeat(60));

  return new Promise<boolean>((resolve) => {
    rl.question("\nDo you accept the Terms of Service? (y/n): ", (answer) => {
      rl.close();
      resolve(answer.trim().toLowerCase() === "y" || answer.trim().toLowerCase() === "yes");
    });
  });
}

async function checkTosStatus(): Promise<void> {
  const config = loadConfig();

  let tosInfo: { version: string; summary: string; url: string };
  try {
    tosInfo = await api<{ version: string; summary: string; url: string }>("GET", "/v1/tos/current");
  } catch {
    // Offline — allow usage with whatever ToS was previously accepted.
    // Backend will enforce on next actual API call anyway.
    return;
  }

  if (config?.tos_accepted_version === tosInfo.version) {
    return; // Already accepted current version
  }

  // Need re-acceptance
  console.log("\nThe Unbrowse Terms of Service have been updated.");
  const accepted = await promptTosAcceptance(tosInfo.summary, tosInfo.url);
  if (!accepted) {
    console.log("You must accept the updated Terms of Service to continue using Unbrowse.");
    process.exit(1);
  }

  // Call accept-tos endpoint
  try {
    await api("POST", "/v1/agents/accept-tos", { tos_version: tosInfo.version });

    // Update local config
    if (config) {
      config.tos_accepted_version = tosInfo.version;
      config.tos_accepted_at = new Date().toISOString();
      saveConfig(config);
    }
    console.log("Terms of Service accepted.");
  } catch (err) {
    console.warn(`Failed to record ToS acceptance: ${(err as Error).message}`);
    // Don't block — backend will enforce on next call
  }
}

/** Auto-register with the backend if no API key is configured. Persists to ~/.unbrowse/config.json. */
export async function ensureRegistered(): Promise<void> {
  if (getApiKey()) {
    // Already have a key — check if ToS re-acceptance is needed
    await checkTosStatus();
    return;
  }

  // Step 1: Fetch current ToS version from backend
  let tosInfo: { version: string; summary: string; url: string };
  try {
    tosInfo = await api<{ version: string; summary: string; url: string }>("GET", "/v1/tos/current");
  } catch {
    console.warn("[unbrowse] Cannot reach unbrowse API. Registration requires internet access.");
    console.warn("[unbrowse] Set UNBROWSE_API_KEY manually or try again when online.");
    return;
  }

  // Step 2: Prompt for ToS acceptance
  const accepted = await promptTosAcceptance(tosInfo.summary, tosInfo.url);
  if (!accepted) {
    console.log("You must accept the Terms of Service to use Unbrowse.");
    process.exit(1);
  }

  // Step 3: Register with ToS version
  const name = `${hostname()}-${randomBytes(3).toString("hex")}`;
  console.log(`Registering as "${name}"...`);

  try {
    const { agent_id, api_key } = await api<{ agent_id: string; [REDACTED_SECRET] }>(
      "POST", "/v1/agents/register", { name, tos_version: tosInfo.version }
    );

    process.env.UNBROWSE_API_KEY = api_key;
    saveConfig({
      api_key,
      agent_id,
      agent_name: name,
      registered_at: new Date().toISOString(),
      tos_accepted_version: tosInfo.version,
      tos_accepted_at: new Date().toISOString(),
    });

    console.log(`Registered as agent ${agent_id}. API key saved to ~/.unbrowse/config.json`);
  } catch (err) {
    console.warn(`Registration failed: ${(err as Error).message}`);
    console.warn("Set UNBROWSE_API_KEY manually or try again.");
    process.exit(1);
  }
}

// --- Skill CRUD ---

// Disk-backed skill cache — survives server restarts.
// Backend has eventual consistency so GET /v1/skills/:id can 404 for
// recently published skills. This cache is the safety net.

function skillCachePath(skillId: string): string {
  return join(SKILL_CACHE_DIR, `${skillId}.json`);
}

function readSkillCache(skillId: string): SkillManifest | null {
  try {
    const raw = readFileSync(skillCachePath(skillId), "utf-8");
    return JSON.parse(raw) as SkillManifest;
  } catch { return null; }
}

function writeSkillCache(skill: SkillManifest): void {
  try {
    if (!existsSync(SKILL_CACHE_DIR)) mkdirSync(SKILL_CACHE_DIR, { recursive: true });
    // Preserve local-only fields (exec_strategy) that the backend doesn't know about
    const existing = readSkillCache(skill.skill_id);
    if (existing) {
      for (const ep of skill.endpoints) {
        const cached = existing.endpoints.find(e => e.endpoint_id === ep.endpoint_id);
        if (!ep.exec_strategy && cached?.exec_strategy) {
          ep.exec_strategy = cached.exec_strategy;
          console.log(`[cache] preserved exec_strategy=${cached.exec_strategy} for ${ep.endpoint_id}`);
        }
        if (!ep.extraction_recipe && cached?.extraction_recipe) {
          ep.extraction_recipe = cached.extraction_recipe;
        }
      }
    }
    const hasStrategy = skill.endpoints.some(e => e.exec_strategy);
    if (hasStrategy) console.log(`[cache] writing skill ${skill.skill_id} with exec_strategy`);
    writeFileSync(skillCachePath(skill.skill_id), JSON.stringify(skill), "utf-8");
  } catch { /* non-critical — best effort */ }
}

export function cachePublishedSkill(skill: SkillManifest): void {
  writeSkillCache(skill);
}

/**
 * Find an existing cached skill for the same domain, so re-captures update
 * the existing skill instead of creating duplicates. Preserves skill_id and
 * exec_strategy across re-captures and server restarts.
 */
export function findExistingSkillForDomain(domain: string): SkillManifest | null {
  try {
    if (!existsSync(SKILL_CACHE_DIR)) return null;
    const files = readdirSync(SKILL_CACHE_DIR);
    for (const f of files) {
      if (!f.endsWith(".json") || f === "browser-capture.json") continue;
      try {
        const raw = readFileSync(join(SKILL_CACHE_DIR, f), "utf-8");
        const skill = JSON.parse(raw) as SkillManifest;
        if (skill.domain === domain && skill.execution_type === "http") {
          return skill;
        }
      } catch { /* skip corrupt files */ }
    }
  } catch { /* cache dir doesn't exist */ }
  return null;
}

export async function getSkill(skillId: string): Promise<SkillManifest | null> {
  // Cache-first: return disk cache immediately, async-refresh from backend
  const cached = readSkillCache(skillId);
  if (cached) {
    api<SkillManifest>("GET", `/v1/skills/${skillId}`)
      .then(skill => {
        // Preserve locally-learned exec_strategy — backend doesn't store these
        if (cached.endpoints) {
          for (const ep of skill.endpoints) {
            const local = cached.endpoints.find(e => e.endpoint_id === ep.endpoint_id);
            if (local?.exec_strategy && !ep.exec_strategy) {
              ep.exec_strategy = local.exec_strategy;
            }
            if (local?.extraction_recipe && !ep.extraction_recipe) {
              ep.extraction_recipe = local.extraction_recipe;
            }
          }
        }
        writeSkillCache(skill);
      })
      .catch(() => {});
    return cached;
  }
  // No cache — must fetch from backend
  try {
    const skill = await api<SkillManifest>("GET", `/v1/skills/${skillId}`);
    writeSkillCache(skill);
    return skill;
  } catch {
    return null;
  }
}

export async function listSkills(): Promise<SkillManifest[]> {
  const data = await api<{ skills: SkillManifest[] }>("GET", "/v1/skills");
  return data.skills;
}

export async function publishSkill(
  draft: Omit<SkillManifest, "skill_id" | "created_at" | "updated_at" | "version"> & {
    skill_id?: string;
    version?: string;
  }
): Promise<SkillManifest & { warnings: string[] }> {
  return api("POST", "/v1/skills", draft);
}

export async function deprecateSkill(skillId: string): Promise<void> {
  await api("DELETE", `/v1/skills/${skillId}`, undefined);
}

export async function updateEndpointScore(
  skillId: string,
  endpointId: string,
  score: number,
  status?: string
): Promise<void> {
  await api("PATCH", `/v1/skills/${skillId}/endpoints/${endpointId}`, { score, status });
}

export async function getEndpointSchema(
  skillId: string,
  endpointId: string
): Promise<unknown | null> {
  try {
    return await api("GET", `/v1/skills/${skillId}/endpoints/${endpointId}/schema`);
  } catch {
    return null;
  }
}

// --- Search ---

export async function searchIntent(
  intent: string,
  k = 5
): Promise<Array<{ id: number; score: number; metadata: Record<string, unknown> }>> {
  const data = await api<{ results: Array<{ id: number; score: number; metadata: Record<string, unknown> }> }>(
    "POST", "/v1/search", { intent, k }
  );
  return data.results;
}

export async function searchIntentInDomain(
  intent: string,
  domain: string,
  k = 5
): Promise<Array<{ id: number; score: number; metadata: Record<string, unknown> }>> {
  const data = await api<{ results: Array<{ id: number; score: number; metadata: Record<string, unknown> }> }>(
    "POST", "/v1/search/domain", { intent, domain, k }
  );
  return data.results;
}

// --- Stats ---

export async function recordExecution(
  skillId: string,
  endpointId: string,
  trace: ExecutionTrace
): Promise<void> {
  // Strip actual API response data — only send metadata for scoring
  const { result: _result, ...metadata } = trace;
  await api("POST", "/v1/stats/execution", {
    skill_id: skillId,
    endpoint_id: endpointId,
    trace: metadata,
  });
}

export async function recordFeedback(
  skillId: string,
  endpointId: string,
  rating: number
): Promise<number> {
  const data = await api<{ avg_rating: number }>("POST", "/v1/stats/feedback", {
    skill_id: skillId,
    endpoint_id: endpointId,
    rating,
  });
  return data.avg_rating;
}

// --- Diagnostics ---

export async function recordDiagnostics(
  skillId: string,
  endpointId: string,
  diagnostics: Record<string, unknown>
): Promise<void> {
  await api("POST", "/v1/stats/diagnostics", {
    skill_id: skillId,
    endpoint_id: endpointId,
    ...diagnostics,
  });
}

// --- Orchestration Perf ---

export async function recordOrchestrationPerf(timing: OrchestrationTiming): Promise<void> {
  await api("POST", "/v1/stats/perf", timing);
}

// --- Validation ---

export async function validateManifest(manifest: unknown): Promise<ValidationResult> {
  return api<ValidationResult>("POST", "/v1/validate", manifest);
}

// --- Agent Registration ---

export async function registerAgent(name: string): Promise<{ agent_id: string; [REDACTED_SECRET] }> {
  return api<{ agent_id: string; [REDACTED_SECRET] }>("POST", "/v1/agents/register", { name });
}

export async function getAgent(agentId: string): Promise<{ agent_id: string; name: string; created_at: string; skills_discovered: string[]; total_executions: number; total_feedback_given: number } | null> {
  try {
    return await api("GET", `/v1/agents/${agentId}`);
  } catch {
    return null;
  }
}

export async function getMyProfile(): Promise<{ agent_id: string; name: string; created_at: string; skills_discovered: string[]; total_executions: number; total_feedback_given: number }> {
  return api("GET", "/v1/agents/me", undefined);
}


path: src/domain.ts
content: /**
 * Shared domain utilities for ccTLD-aware domain parsing and suffix matching.
 */

/** Common ccTLD patterns (country-code second-level domains). */
const CC_TLDS = new Set([
  "co.uk", "co.nz", "co.jp", "co.kr", "co.in",
  "com.br", "com.au", "com.cn", "com.mx", "com.ar", "com.tw",
  "org.uk", "gov.uk", "ac.uk", "net.au", "org.au",
]);

/**
 * Get the registrable domain, handling ccTLDs like .co.uk, .com.br, etc.
 * e.g., "api.example.co.uk" → "example.co.uk"
 *        "sub.example.com"   → "example.com"
 */
export function getRegistrableDomain(hostname: string): string {
  const parts = hostname.split(".");
  if (parts.length >= 3) {
    const lastTwo = parts.slice(-2).join(".");
    if (CC_TLDS.has(lastTwo)) {
      return parts.slice(-3).join(".");
    }
  }
  return parts.slice(-2).join(".");
}

/**
 * Proper domain suffix matching.
 * Returns true if cookieDomain matches targetDomain or is a parent of it.
 * e.g., isDomainMatch(".google.com", "mail.google.com") → true
 *        isDomainMatch("notgoogle.com", "google.com") → false
 */
export function isDomainMatch(cookieDomain: string, targetDomain: string): boolean {
  const c = cookieDomain.replace(/^\./, "").toLowerCase();
  const t = targetDomain.replace(/^\./, "").toLowerCase();
  return t === c || t.endsWith("." + c);
}

path: src/execution/retry.ts
content: const RETRYABLE_STATUSES = new Set([500, 502, 503, 504, 429]);
const MAX_RETRIES = 2;
const BASE_DELAY_MS = 1000;
const MAX_DELAY_MS = 10000;

export interface RetryOptions {
  maxRetries?: number;
  baseDelay?: number;
  maxDelay?: number;
}

/**
 * Wrap an async function with exponential backoff retry.
 * Only retries if isRetryable returns true.
 */
export async function withRetry<T>(
  fn: () => Promise<T>,
  isRetryable: (result: T) => boolean,
  opts?: RetryOptions
): Promise<T> {
  const maxRetries = opts?.maxRetries ?? MAX_RETRIES;
  const baseDelay = opts?.baseDelay ?? BASE_DELAY_MS;
  const maxDelay = opts?.maxDelay ?? MAX_DELAY_MS;

  let lastResult: T | undefined;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const result = await fn();
    lastResult = result;

    if (!isRetryable(result) || attempt === maxRetries) {
      return result;
    }

    // Exponential backoff with jitter
    const delay = Math.min(baseDelay * Math.pow(2, attempt), maxDelay);
    const jitter = delay * 0.5 * Math.random();
    await new Promise((r) => setTimeout(r, delay + jitter));
  }

  return lastResult!;
}

export function isRetryableStatus(status: number): boolean {
  return RETRYABLE_STATUSES.has(status);
}

path: src/index.ts
content: import "dotenv/config";
import { execSync } from "node:child_process";
import { existsSync } from "node:fs";
import Fastify from "fastify";
import cors from "@fastify/cors";
import { registerRoutes } from "./api/routes.js";
import { registerRateLimiter } from "./ratelimit/index.js";
import { schedulePeriodicVerification } from "./verification/index.js";
import { ensureRegistered } from "./client/index.js";
import { shutdownAllBrowsers } from "./capture/index.js";

// Kill any chrome-headless-shell orphans left over from a previous crashed session
try {
  execSync("pkill -f chrome-headless-shell", { stdio: "ignore" });
} catch { /* no orphans — ok */ }

// Ensure browser engine is installed (agent-browser needs Chromium binaries)
try {
  const { chromium } = await import("playwright-core");
  if (!existsSync(chromium.executablePath())) {
    console.log("[startup] Chromium not found, installing...");
    execSync("npx agent-browser install", { stdio: "inherit", timeout: 120_000 });
  }
} catch {
  console.warn("[startup] WARNING: Could not verify/install browser engine. Run: npx agent-browser install");
}

// Auto-register with backend if no API key is configured
await ensureRegistered();

const app = Fastify({ logger: true });

await app.register(cors, { origin: true });
await registerRateLimiter(app);
await registerRoutes(app);

const port = Number(process.env.PORT ?? 6969);
const host = process.env.HOST ?? "127.0.0.1";

async function shutdown(signal: string): Promise<void> {
  console.log(`[shutdown] ${signal} — closing browsers and server`);
  await shutdownAllBrowsers();
  await app.close();
  process.exit(0);
}

process.on("SIGTERM", () => { shutdown("SIGTERM").catch(() => process.exit(1)); });
process.on("SIGINT",  () => { shutdown("SIGINT").catch(() => process.exit(1)); });

try {
  await app.listen({ port, host });
  console.log(`unbrowse running on http://${host}:${port}`);
  schedulePeriodicVerification();
} catch (err) {
  app.log.error(err);
  process.exit(1);
}


path: src/logger.ts
content: import fs from "node:fs";
import path from "node:path";
import os from "node:os";

const LOG_DIR = path.join(os.homedir(), ".unbrowse", "logs");

function getLogFile(): string {
  const date = new Date().toISOString().slice(0, 10); // YYYY-MM-DD
  return path.join(LOG_DIR, `unbrowse-${date}.log`);
}

function ensureLogDir(): void {
  fs.mkdirSync(LOG_DIR, { recursive: true });
}

/**
 * Log a message to both stdout and ~/.unbrowse/logs/unbrowse-YYYY-MM-DD.log.
 * Format: [HH:MM:SS] [module] message
 */
export function log(module: string, message: string): void {
  const ts = new Date().toTimeString().slice(0, 8); // HH:MM:SS
  const line = `[${ts}] [${module}] ${message}`;
  console.log(line);
  try {
    ensureLogDir();
    fs.appendFileSync(getLogFile(), line + "\n");
  } catch {
    // Never crash the main process if logging fails
  }
}


path: src/marketplace/index.ts
content: import { nanoid } from "nanoid";
import * as client from "../client/index.js";
import type { EndpointDescriptor, SkillManifest, VerificationStatus } from "../types/index.js";

export async function listSkills(): Promise<SkillManifest[]> {
  return client.listSkills();
}

export async function getSkill(skillId: string): Promise<SkillManifest | null> {
  return client.getSkill(skillId);
}

export async function publishSkill(
  draft: Omit<SkillManifest, "skill_id" | "created_at" | "updated_at" | "version"> & {
    skill_id?: string;
    version?: string;
  }
): Promise<SkillManifest> {
  // Pre-cache locally so the skill is immediately available even if the remote publish
  // fails or EmergentDB hasn't indexed it yet (eventual consistency).
  const now = new Date().toISOString();
  const preCache = {
    ...draft,
    skill_id: draft.skill_id ?? nanoid(),
    created_at: now,
    updated_at: now,
    version: draft.version ?? "1.0.0",
  } as SkillManifest;
  client.cachePublishedSkill(preCache);

  try {
    const { warnings: _, ...backendFields } = await client.publishSkill(draft);
    // Merge draft with backend response — avoids read-after-write race
    const skill = { ...draft, ...backendFields } as SkillManifest;
    client.cachePublishedSkill(skill);
    return skill;
  } catch (err) {
    console.error("[publish] remote publish failed, using local cache:", (err as Error).message);
    return preCache;
  }
}

export async function updateEndpointScore(
  skillId: string,
  endpointId: string,
  score: number,
  status?: VerificationStatus
): Promise<void> {
  await client.updateEndpointScore(skillId, endpointId, score, status);
}

// --- Pure local helpers (no backend call) ---

export function mergeEndpoints(
  existing: EndpointDescriptor[],
  incoming: EndpointDescriptor[]
): EndpointDescriptor[] {
  const merged = [...existing];
  for (const ep of incoming) {
    const dupe = merged.find(
      (e) =>
        e.method === ep.method &&
        normalizeTemplate(e.url_template) === normalizeTemplate(ep.url_template)
    );
    if (!dupe) merged.push(ep);
  }
  return merged;
}

export function normalizeTemplate(t: string): string {
  return t.replace(/\{[^}]+\}/g, "{}").toLowerCase();
}


path: src/ratelimit/index.ts
content: import type { FastifyInstance } from "fastify";
import rateLimit from "@fastify/rate-limit";

export async function registerRateLimiter(app: FastifyInstance): Promise<void> {
  await app.register(rateLimit, {
    max: 100,
    timeWindow: "1 minute",
  });
}

/** Per-route rate limit configs. Apply via route options in Fastify. */
export const ROUTE_LIMITS = {
  "/v1/intent/resolve": { max: 20, timeWindow: "1 minute" },
  "/v1/skills/:skill_id/execute": { max: 30, timeWindow: "1 minute" },
  "/v1/skills": { max: 5, timeWindow: "1 minute" }, // POST only
  "/v1/auth/login": { max: 3, timeWindow: "5 minutes" },
  "/v1/feedback": { max: 60, timeWindow: "1 minute" },
} as const;

export function routeRateLimit(path: keyof typeof ROUTE_LIMITS) {
  const cfg = ROUTE_LIMITS[path];
  return { config: { rateLimit: { max: cfg.max, timeWindow: cfg.timeWindow } } };
}

path: src/transform/drift.ts
content: import type { DriftResult, ResponseSchema } from "../types/index.js";
import { inferSchema } from "./index.js";

/**
 * Collect all field paths from a schema (dot-notation).
 */
function collectPaths(schema: ResponseSchema, prefix = ""): Map<string, string> {
  const paths = new Map<string, string>();
  if (schema.type === "object" && schema.properties) {
    for (const [key, val] of Object.entries(schema.properties)) {
      const fullPath = prefix ? `${prefix}.${key}` : key;
      paths.set(fullPath, val.type);
      if (val.type === "object" && val.properties) {
        for (const [p, t] of collectPaths(val, fullPath)) {
          paths.set(p, t);
        }
      }
      if (val.type === "array" && val.items?.type === "object") {
        for (const [p, t] of collectPaths(val.items, `${fullPath}[]`)) {
          paths.set(p, t);
        }
      }
    }
  }
  return paths;
}

/**
 * Compare stored schema against a fresh response sample.
 * Returns added/removed fields and type changes.
 */
export function detectSchemaDrift(existing: ResponseSchema, newSample: unknown): DriftResult {
  const newSchema = inferSchema([newSample]);
  const oldPaths = collectPaths(existing);
  const newPaths = collectPaths(newSchema);

  const added_fields: string[] = [];
  const removed_fields: string[] = [];
  const type_changes: Array<{ path: string; was: string; now: string }> = [];

  for (const [path, type] of newPaths) {
    if (!oldPaths.has(path)) {
      added_fields.push(path);
    } else if (oldPaths.get(path) !== type) {
      type_changes.push({ path, was: oldPaths.get(path)!, now: type });
    }
  }
  for (const path of oldPaths.keys()) {
    if (!newPaths.has(path)) {
      removed_fields.push(path);
    }
  }

  return {
    drifted: added_fields.length > 0 || removed_fields.length > 0 || type_changes.length > 0,
    added_fields,
    removed_fields,
    type_changes,
  };
}


path: src/transform/index.ts
content: import type { ProjectionOptions, ResponseSchema } from "../types/index.js";

// --- Field Projection ---

/**
 * Walk a dot-notation path with [] array expansion.
 * e.g. "elements[].actor.name" expands across array items.
 */
export function resolvePath(obj: unknown, path: string): unknown[] {
  const parts = path.split(".");
  let current: unknown[] = [obj];

  for (const part of parts) {
    const next: unknown[] = [];
    const isArray = part.endsWith("[]");
    const key = isArray ? part.slice(0, -2) : part;

    for (const item of current) {
      if (item == null || typeof item !== "object") continue;
      const val = (item as Record<string, unknown>)[key];
      if (val === undefined) continue;
      if (isArray && Array.isArray(val)) {
        next.push(...val);
      } else {
        next.push(val);
      }
    }
    current = next;
  }
  return current;
}

/**
 * Select only specified fields, rebuilding nested structure.
 */
export function project(data: unknown, fields: string[]): unknown {
  if (data == null) return data;
  if (Array.isArray(data)) return data.map((item) => project(item, fields));

  const result: Record<string, unknown> = {};
  for (const field of fields) {
    const resolved = resolvePath(data, field);
    if (resolved.length === 0) continue;
    setNestedValue(result, field, resolved.length === 1 ? resolved[0] : resolved);
  }
  return result;
}

function setNestedValue(obj: Record<string, unknown>, path: string, value: unknown): void {
  const parts = path.split(".").map((p) => (p.endsWith("[]") ? p.slice(0, -2) : p));
  let current: Record<string, unknown> = obj;
  for (let i = 0; i < parts.length - 1; i++) {
    if (!(parts[i] in current) || typeof current[parts[i]] !== "object") {
      current[parts[i]] = {};
    }
    current = current[parts[i]] as Record<string, unknown>;
  }
  current[parts[parts.length - 1]] = value;
}

/**
 * Strip nulls, empty strings, empty arrays, and ephemeral keys.
 */
const EPHEMERAL_KEYS = new Set(["trackingId", "$type", "recipeType"]);
const EPHEMERAL_SUFFIXES = ["Urn"];

function isEphemeral(key: string): boolean {
  if (EPHEMERAL_KEYS.has(key)) return true;
  return EPHEMERAL_SUFFIXES.some((s) => key.endsWith(s));
}

export function compact(data: unknown, _opts?: { max_depth?: number }): unknown {
  if (data == null) return undefined;
  if (Array.isArray(data)) {
    const filtered = data.map((item) => compact(item, _opts)).filter((v) => v !== undefined);
    return filtered.length > 0 ? filtered : undefined;
  }
  if (typeof data === "object") {
    const result: Record<string, unknown> = {};
    for (const [key, val] of Object.entries(data as Record<string, unknown>)) {
      if (isEphemeral(key)) continue;
      const compacted = compact(val, _opts);
      if (compacted !== undefined) result[key] = compacted;
    }
    return Object.keys(result).length > 0 ? result : undefined;
  }
  if (typeof data === "string" && data === "") return undefined;
  return data;
}

function truncateDepth(data: unknown, maxDepth: number, currentDepth = 0): unknown {
  if (currentDepth >= maxDepth) return typeof data === "object" && data !== null ? "[truncated]" : data;
  if (Array.isArray(data)) return data.map((item) => truncateDepth(item, maxDepth, currentDepth + 1));
  if (typeof data === "object" && data !== null) {
    const result: Record<string, unknown> = {};
    for (const [key, val] of Object.entries(data as Record<string, unknown>)) {
      result[key] = truncateDepth(val, maxDepth, currentDepth + 1);
    }
    return result;
  }
  return data;
}

/**
 * Orchestrate project + compact + max_depth truncation.
 */
export function applyProjection(data: unknown, projection: ProjectionOptions): unknown {
  let result = data;
  if (projection.fields && projection.fields.length > 0) {
    result = project(result, projection.fields);
  }
  if (projection.compact) {
    result = compact(result);
  }
  if (projection.max_depth != null) {
    result = truncateDepth(result, projection.max_depth);
  }
  return result;
}

// --- Schema Inference ---

function typeOf(value: unknown): string {
  if (value === null) return "null";
  if (Array.isArray(value)) return "array";
  if (typeof value === "number") return Number.isInteger(value) ? "integer" : "number";
  return typeof value; // string, boolean, object
}

function inferSingle(value: unknown): ResponseSchema {
  const t = typeOf(value);
  if (t === "null") return { type: "null", inferred_from_samples: 1 };
  if (t === "array") {
    const arr = value as unknown[];
    if (arr.length === 0) return { type: "array", inferred_from_samples: 1 };
    const itemSchemas = arr.map(inferSingle);
    return { type: "array", items: mergeSchemas(itemSchemas), inferred_from_samples: 1 };
  }
  if (t === "object") {
    const obj = value as Record<string, unknown>;
    const properties: Record<string, ResponseSchema> = {};
    for (const [key, val] of Object.entries(obj)) {
      properties[key] = inferSingle(val);
    }
    return {
      type: "object",
      properties,
      required: Object.keys(obj),
      inferred_from_samples: 1,
    };
  }
  return { type: t, inferred_from_samples: 1 };
}

function mergeSchemas(schemas: ResponseSchema[]): ResponseSchema {
  if (schemas.length === 0) return { type: "object", inferred_from_samples: 0 };
  if (schemas.length === 1) return schemas[0];

  const types = new Set(schemas.map((s) => s.type));
  if (types.size === 1 && types.has("object")) {
    return mergeObjectSchemas(schemas);
  }
  if (types.size === 1 && types.has("array")) {
    const itemSchemas = schemas.filter((s) => s.items).map((s) => s.items!);
    return {
      type: "array",
      items: itemSchemas.length > 0 ? mergeSchemas(itemSchemas) : undefined,
      inferred_from_samples: schemas.length,
    };
  }
  if (types.size === 1) {
    return { type: schemas[0].type, inferred_from_samples: schemas.length };
  }
  // Union type
  const unique = [...types].map((t) => ({ type: t, inferred_from_samples: schemas.length }));
  return { type: unique[0].type, anyOf: unique, inferred_from_samples: schemas.length };
}

function mergeObjectSchemas(schemas: ResponseSchema[]): ResponseSchema {
  const allKeys = new Set<string>();
  for (const s of schemas) {
    if (s.properties) Object.keys(s.properties).forEach((k) => allKeys.add(k));
  }
  const properties: Record<string, ResponseSchema> = {};
  const required: string[] = [];

  for (const key of allKeys) {
    const propSchemas = schemas
      .filter((s) => s.properties?.[key])
      .map((s) => s.properties![key]);
    properties[key] = mergeSchemas(propSchemas);
    // Required only if present in ALL samples
    if (propSchemas.length === schemas.length) required.push(key);
  }

  return {
    type: "object",
    properties,
    required: required.length > 0 ? required : undefined,
    inferred_from_samples: schemas.length,
  };
}

/**
 * Infer JSON Schema draft-07 subset from 1+ response samples.
 */
export function inferSchema(samples: unknown[]): ResponseSchema {
  if (samples.length === 0) return { type: "object", inferred_from_samples: 0 };
  const schemas = samples.map(inferSingle);
  const merged = mergeSchemas(schemas);
  merged.inferred_from_samples = samples.length;
  return merged;
}


path: src/types/index.ts
content: export * from "./skill.js";


path: src/version.ts
content: import { createHash } from "crypto";
import { readFileSync, readdirSync } from "fs";
import { join } from "path";
import { execSync } from "child_process";

// Deterministic version hash of all src/*.ts files.
// Computed once at startup. Same code = same hash.
// Used to stamp every trace so real user sessions become versioned evals.

function collectTsFiles(dir: string): string[] {
  const results: string[] = [];
  for (const entry of readdirSync(dir, { withFileTypes: true })) {
    const full = join(dir, entry.name);
    if (entry.isDirectory() && entry.name !== "node_modules") {
      results.push(...collectTsFiles(full));
    } else if (entry.name.endsWith(".ts")) {
      results.push(full);
    }
  }
  return results;
}

function computeCodeHash(): string {
  const srcDir = join((import.meta as any).dir ?? __dirname, ".");
  const files = collectTsFiles(srcDir).sort();
  const hash = createHash("sha256");
  for (const file of files) {
    hash.update(file.slice(srcDir.length)); // relative path for determinism
    hash.update(readFileSync(file, "utf-8"));
  }
  return hash.digest("hex").slice(0, 12);
}

function getGitSha(): string {
  try {
    return execSync("git rev-parse --short HEAD", { encoding: "utf-8", cwd: (import.meta as any).dir ?? __dirname }).trim();
  } catch {
    return "unknown";
  }
}

/** 12-char hex hash of all source file contents */
export const CODE_HASH: string = computeCodeHash();

/** Short git commit SHA */
export const GIT_SHA: string = getGitSha();

/** Combined version: "{code_hash}@{git_sha}" — stamped on every trace */
export const TRACE_VERSION: string = `${CODE_HASH}@${GIT_SHA}`;


path: traces/0SiahkhbMjRVtI8YnDyrn.json
content: {
  "trace_id": "0SiahkhbMjRVtI8YnDyrn",
  "skill_id": "are_jhxclgDznaAmVH-iV",
  "endpoint_id": "QwQTfoNCpbhElIEKUmD27",
  "started_at": "2026-02-23T14:44:02.283Z",
  "completed_at": "2026-02-23T14:44:03.671Z",
  "success": true,
  "status_code": 200,
  "result": {}
}

path: traces/5H6LXkBWZ2Tb717Ni8x08.json
content: {
  "trace_id": "5H6LXkBWZ2Tb717Ni8x08",
  "skill_id": "3rvgINtjR025_jSzitEp_",
  "endpoint_id": "RkSzIWXufsggLY_wcIaXC",
  "started_at": "2026-02-25T02:21:05.810Z",
  "completed_at": "2026-02-25T02:21:05.936Z",
  "success": true,
  "status_code": 200,
  "result": {
    "data": {
      "metadata": {
        "numUnseen": 0,
        "latestPublishedAt": 1,
        "$type": "com.linkedin.voyager.identity.me.NotificationsMetadata"
      },
      "entityUrn": "urn:li:collectionResponse:sEpR//eyUZldHde37d6xIHYWLz7b3giD6Mjo+o3FS8g=",
      "elements": [],
      "paging": {
        "count": 10,
        "start": 0,
        "links": []
      },
      "$type": "com.linkedin.restli.common.CollectionResponse"
    },
    "included": []
  }
}

path: traces/aFmEeZYOIAXrANJwFzCjK.json
content: trace_id: aFmEeZYOIAXrANJwFzCjK
skill_id: pb_IIHwqXD0KsGcAXyWpb
endpoint_id: nyrXnDTNOAExMpeOAeXvN
started_at: 2026-02-25T01:52:10.470Z
completed_at: 2026-02-25T01:52:11.400Z
success: no
status_code: 400
error: HTTP 400

path: traces/bFSRDjPpDpkXSKFUfD12E.json
content: trace_id: bFSRDjPpDpkXSKFUfD12E
skill_id: linkedin-voyager-profile-v1
endpoint_id: li-full-profile
started_at: 2026-02-25T02:15:50.122Z
completed_at: 2026-02-25T02:15:50.295Z
success: no
status_code: 400
error: HTTP 400

path: tsconfig.json
content: extends: ../../tsconfig.json

Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)