Skill: imoonkey/openweb (version 0.1.6)

--- name: openweb description: "Typed JSON access to 90+ real websites — Google, Amazon, Reddit, YouTube, GitHub, Instagram, Bloomberg, Zillow, and more — across search, shopping, travel, finance, social, news, and dev tools. Use to read, search, post, comment, message, or otherwise interact with these sites (prices, products, articles, stock quotes, flights, posts, comments, DMs, etc.). Run `openweb sites` for the full list." license: MIT compatibility: "Requires the `openweb` CLI (npm: @openweb-org/openweb). Some sites need a managed Chrome session. See references/cli.md for runtime details." metadata: version: "0.1.5" openclaw: links: homepage: "https://getopenweb.com" repository: "https://github.com/openweb-org/openweb" requires: bins: [openweb] env: [OPENWEB_HOME] config: ["$OPENWEB_HOME/config.json"] install: - kind: node package: "@openweb-org/openweb" bins: [openweb] --- # OpenWeb Agent-native way to access any website. Bridging agent CLI and web GUI through API. ## Load Discipline - Do not read every file in this folder. - Start with SKILL.md. Follow one route at a time. ## Use Existing Site ### 1. Find the site ```bash openweb sites # list all available sites ``` If the site has no package, do NOT say "unsupported." Route to add-site/guide.md. ### 2. Check readiness ```bash openweb <site> # transport, auth, operations list ``` - `Requires browser: yes` — browser auto-starts when needed; no manual setup required - `Requires login: yes` — user must be logged in via their browser session ### 3. Read site notes Prefer site docs when they exist in your workspace: - **Source repo:** read `src/sites/<site>/SKILL.md` for workflows and intent mapping, then `src/sites/<site>/DOC.md` for internals. - **Shipped/runtime package:** only `DOC.md`, specs, manifest, examples, and adapters are guaranteed. Per-site `SKILL.md` and `PROGRESS.md` are source-tree docs and are not shipped into `$OPENWEB_HOME/sites/<site>/`. If only the shipped package is available, rely on `openweb <site>`, `openweb <site> <op>`, examples, and `DOC.md`. ### 4. Inspect the operation ```bash openweb <site> <op> # params, response shape, permission tier openweb <site> <op> --example # real example params from fixtures ``` Check the operation's permission tier before executing: | Tier | Default | Behavior | |---|---|---| | `read` | allow | GET-like operations — execute freely | | `write` | prompt | Creates/updates — ask user before executing | | `delete` | prompt | Destructive — ask user before executing | | `transact` | deny | Financial/irreversible — always skip | ### 5. Execute ```bash openweb <site> <op> '{"key":"value"}' # stdout=JSON, stderr=JSON error ``` Auto-spill: responses over 4096 bytes write to a temp file. `openweb <site> exec <op> '{...}'` is equivalent — `exec` can be omitted when the third arg is JSON. ### 6. On failure Errors on stderr include `failureClass`: | failureClass | Action | |---|---| | `needs_browser` | Browser auto-starts; if it fails, check Chrome installation. Fallback: `openweb browser start` | | `needs_login` | `openweb login <site>` then `openweb browser restart` | | `needs_page` | Open a tab to the site URL | | `bot_blocked` | `openweb browser restart --no-headless`, user solves CAPTCHA in visible browser, then retry. For persistent sites, set `"browser": {"headless": false}` in config | | `permission_denied` | Update `permissions` in `$OPENWEB_HOME/config.json` | | `permission_required` | Ask user for confirmation, then retry | | `retriable` | Wait a few seconds, retry (max 2) | | `fatal` | Don't retry — fix params or check site name | If the table above doesn't resolve it, read references/troubleshooting.md. ### 7. Missing site or coverage Site doesn't exist or lacks needed operations? Read add-site/guide.md. ## Add / Expand / Upgrade Site Read add-site/guide.md ## Fix a Problem Read references/troubleshooting.md ## File Map All paths relative to `skills/openweb/`. ### add-site/ (workflow — load in sequence) | File | Load when | |---|---| | `add-site/guide.md` | Entry point for add/expand workflow | | `add-site/probe.md` | Probe step: CDP browser-side discovery | | `add-site/capture.md` | Capture step: recording browser traffic | | `add-site/review.md` | Review step: reading analysis-summary.json | | `add-site/curate-operations.md` | Curate: naming, noise, params, permissions | | `add-site/curate-runtime.md` | Curate: auth, transport, extraction | | `add-site/curate-schemas.md` | Curate: response schemas, examples, PII | | `add-site/verify.md` | Verify: runtime + spec + doc loop | | `add-site/document.md` | Document: per-site SKILL.md + DOC.md + PROGRESS.md, knowledge updates | ### references/ (lookup — load independently) | File | Load when | |---|---| | `references/cli.md` | CLI command syntax, flags, stdout/stderr | | `references/x-openweb.md` | Full x-openweb field schema | | `references/troubleshooting.md` | Something broke — classify, diagnose, fix | ### knowledge/ (patterns — load at decision points) | File | Load when | |---|---| | `knowledge/archetypes.md` | Expected operations by site category | | `knowledge/auth-routing.md` | Auth type unknown — signal-to-family lookup | | `knowledge/auth-primitives.md` | Configuring auth — config and gotchas | | `knowledge/bot-detection.md` | Transport/capture decisions | | `knowledge/extraction.md` | Extraction signals — SSR/DOM patterns | | `knowledge/graphql.md` | GraphQL — persisted queries, batching | | `knowledge/ws.md` | WebSocket — message/connection patterns | | `knowledge/adapter-recipes.md` | Adapter patterns, code templates, pitfalls | | `knowledge/transport-upgrade.md` | Transport tier decisions, node feasibility, API discovery |

path: add-site/capture.md
content: # Capture Techniques

How to capture site traffic for the compiler. Interactive browsing, direct API
calls, scripted capture, auth injection, and troubleshooting.

**Load when:** guide.md Step 4 (Capture Evidence), or when debugging capture scripts.

---

## Capture Modes

Both modes produce HAR traffic. Use them together — UI browsing discovers
endpoints you don't know about; direct calls fill known coverage gaps.

### Capture Granularity

Probe determines how much capture each intent family needs. Capture is not a
repeat of probe — it is compile's input. When compile adds no value for a
family, capture adds no value either.

| Mode | When to use | Typical scope |
|------|-------------|---------------|
| **None** | Pure extraction or direct adapter/intercept; compile adds no value | Skip capture entirely |
| **Micro** | Endpoint already known; compile saves manual schema/example/auth work | 1-3 representative requests |
| **Targeted** | Family needs param variation, GraphQL clustering, path normalization | Focused family capture with varied actions |
| **Broad** | Surface still unknown after probe, WS discovery, many unknown ops | Large-surface capture |

Broad capture is not the default. Use it only when micro or targeted did not
produce enough evidence, the surface is still unknown, or WS discovery needs
longer observation.

**Probe drives capture scope:** Use the family's capture mode from the probe
matrix to select granularity. Avoid blind browsing — capture only the traffic
needed for the decision being answered.

### UI Navigation

Browse the managed browser systematically to trigger each target intent:

- **Search:** Type in the on-page search box, not the URL bar. `page.goto()`
  to a search URL delivers SSR HTML; the SPA search widget triggers the JSON API.
- **Vary inputs** — 2-3 different search terms for better schema inference.
- **Click through results** — detail pages, tabs, pagination.
- **Wait for content to load** before navigating away.
- **Click UI tabs** on profiles/feeds — each triggers a different endpoint.

**SPA navigation rule:** Use in-app navigation (click links), not address-bar
navigation or `window.location.href`. Full-page reloads deliver SSR data — JSON
API calls only fire during SPA client-side routing. For programmatic browsing:
`element.click()` on links, not `Page.navigate`.

#### Login

If login required, authenticate in the managed browser. For net-new sites,
use the target URL directly (`openweb login <site>` won't work).

**Auth-dependent capture requirements:**

| Auth type | Capture requirement |
|---|---|
| `exchange_chain` (Reddit-like) | Cold page load (clear cookies or incognito) so the token exchange request appears in HAR |
| `sapisidhash` (Google/YouTube) | Must be logged into Google account; SAPISID cookie + SAPISIDHASH header must appear |
| `cookie_session` with CSRF | Perform at least one mutation (like, follow) so CSRF token appears in POST headers |

#### Write Operations

Trigger writes after read flows. Prefer `page.evaluate(fetch(...))` (below)
over UI clicks. Find write endpoints from prior-round DOC.md or openapi.yaml.
After each write, trigger the reverse (like/unlike, follow/unfollow).

**Avoid:** logout, delete account, billing, irreversible actions.

### Direct API Calls via `page.evaluate(fetch)`

More reliable than hoping UI clicks trigger the right requests:

```javascript
await page.evaluate(() => fetch('/api/endpoint?q=value', {
  credentials: 'same-origin'
}))
```

**Prefer direct fetch for:** POST-based APIs (Innertube, GraphQL), known API
patterns without UI buttons, varied-parameter samples for schema inference.
**Same-origin only** — blocked by CORS cross-origin. Navigate to the target
subdomain first.

---

## Non-Cookie Auth Injection

`credentials: 'same-origin'` only carries cookies. For non-cookie auth
(`localStorage_jwt`, `page_global`, `webpack_module_walk`), extract and inject:

```javascript
// 1. Extract (method depends on auth type)
const token = await page.evaluate(() =>
  localStorage.getItem('auth_token')       // localStorage_jwt
  // OR: window.__AUTH_TOKEN__             // page_global
)

// 2. Inject into fetch
await page.evaluate((t) => fetch('/api/endpoint', {
  headers: { 'Authorization': `Bearer ${t}` },
}), token)
```

Check the site's `openapi.yaml` for extraction method and header.

---

## Capture Target Binding

Capture attaches to `pages()[0]` on start, auto-attaches to new tabs.

**Rules:** (1) Start capture FIRST, then open new tabs. (2) Pre-existing
tabs are blind spots. (3) A separate `connectOverCDP()` creates unmonitored
pages — use the existing connection's context.

**Verification:** Check `summary.byCategory.api` — if 0 despite browsing,
traffic came from a pre-existing tab or separate connection.

---

## Scripted Capture

For complex programmatic capture, use a two-phase approach:

1. `openweb capture start` (auto-starts browser) -> run script -> `openweb capture stop`
2. `openweb compile <site-url> --capture-dir ./capture`

This separates capture from compilation for fast iteration.

### Two-Phase Script Skeleton

```typescript
import { chromium } from 'patchright'

const browser = await chromium.connectOverCDP('http://localhost:9222')
const page = browser.contexts()[0]!.pages()[0]!  // reuse monitored page
const wait = (ms: number) => new Promise(r => setTimeout(r, ms))

await page.goto('https://example.com', { waitUntil: 'load', timeout: 30_000 })
await wait(3000)

await page.evaluate(() => fetch('/api/v1/feed?limit=20', {
  credentials: 'same-origin'
}))
await wait(800)

// Cleanup — bounded timeouts, always exit
const withTimeout = <T>(p: Promise<T>, ms: number): Promise<T | void> =>
  Promise.race([p.catch(() => {}), new Promise<void>(r => setTimeout(r, ms))])
await withTimeout(browser.close(), 5_000)
process.exit(0)
```

**Key:** Reuse `pages()[0]` from the same connection — a second
`connectOverCDP()` creates unmonitored pages.

### `compile --script` Alternative

`openweb compile <url> --script ./record.ts` — manages its own capture
session via `createCaptureSession()`. Killed after 120s.

```typescript
import { parseArgs } from 'node:util'
import { chromium } from 'patchright'
import { createCaptureSession } from '../src/capture/session.js'

const { values } = parseArgs({ options: { out: { type: 'string' } }, strict: false })
const outputDir = values.out!

const browser = await chromium.connectOverCDP('http://localhost:9222')
const page = await browser.contexts()[0]!.newPage()

const session = createCaptureSession({
  cdpEndpoint: 'http://localhost:9222',
  outputDir, targetPage: page, isolateToTargetPage: true,
  onLog: (msg) => process.stderr.write(`${msg}\n`),
})
await session.ready

// ... navigate, call APIs ...
session.stop(); await session.done
await Promise.race([page.close().catch(()=>{}), new Promise<void>(r=>setTimeout(r,5000))])
process.exit(0)
```

See real adapters for examples (e.g., `src/sites/hackernews/adapters/`,
`src/sites/leetcode/adapters/`).

### Multi-Worker Browser Sharing

Multiple workers share one Chrome via `--isolate`, which scopes each session
to its own tab. Each worker runs `openweb capture start --isolate --url <url>`,
captures traffic, then `openweb capture stop --session $SESSION_ID`.

---

## Timeout Discipline

Every async operation needs a bounded timeout — CDP operations can block
indefinitely.

| Operation | Pattern | Why |
|---|---|---|
| `page.goto()` | `{ waitUntil: 'load', timeout: 30_000 }` | `'networkidle'` never fires on SPAs (persistent WS, polling). Use `'load'` + `await wait(3000)`. |
| `page.evaluate(fetch)` | `AbortController` with 15s timeout | Node.js timeouts don't reach into the browser |
| `page.close()` / `browser.close()` | `Promise.race` with 5s timeout | CDP session issues can hang |
| `session.done` | Safe as-is (3s drain timeout) | Already bounded |

**Fetch timeout pattern** (inside `page.evaluate`):

```typescript
await page.evaluate(async (args) => {
  const ctrl = new AbortController()
  const timer = setTimeout(() => ctrl.abort(), 15_000)
  try {
    const r = await fetch(args.url, {
      headers: args.headers, signal: ctrl.signal,
    })
    return { status: r.status, body: await r.text() }
  } finally { clearTimeout(timer) }
}, { url, headers })
```

Always end scripts with `process.exit(0)` — `setTimeout` in `withTimeout`
keeps the event loop alive.

---

## Capture Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| HAR has 0 API entries | Traffic from pre-existing tab | Start capture first, then open a NEW tab |
| `page.evaluate(fetch)` not in HAR | Separate Playwright connection | Use CDP on the capture's browser context |
| `No active capture session` on stop | Stale PID / process killed | `pkill -f "capture start"`, delete PID file, restart |
| HAR empty / truncated | Process killed before flush | Stop with `openweb capture stop`, never `kill -9` |
| `networkidle` hangs forever | SPA persistent connections | Use `waitUntil: 'load'` + fixed wait |
| Auth tokens missing from HAR | Didn't meet auth capture requirement | See auth-dependent capture table above |
| Cross-origin fetch blocked | CORS on `page.evaluate(fetch)` | Navigate to target subdomain first, use relative paths |


path: add-site/curate-operations.md
content: # Curate Operations

Operation-level quality: naming, noise removal, parameter descriptions, permissions,
and write operation curation. Loaded during the Curate step of `add-site/guide.md`.

**Edit target:** `$OPENWEB_HOME/sites/<site>/openapi.yaml`. Read it before making changes.

**Companion files:** auth/transport/extraction: `curate-runtime.md`.
Response schemas and examples: `curate-schemas.md`.

---

## 1. Remove Noise Operations

Delete any operation matching these criteria:

- **Analytics/tracking** — names containing: `collector`, `log`, `batch`,
  `telemetry`, `apm`, `tracking`, `beacon`, `zlab`, `commercial`, `impression`,
  `feedback`, `popup`
- **CDN/static assets** — `/static/`, `/_next/`, `/assets/`
- **Polling/heartbeat** — ping or keep-alive only
- **4xx-only clusters** — no successful responses captured
- **POST with empty/204 response** — likely telemetry
- **Bare operationId** — just `get` or `create` with no noun
- **Internal framework** — `rsc-action`, `flagship-web`, `_next`
- **SaaS dashboard internals** — internal namespace traffic; manual filtering needed

> `src/compiler/analyzer/labeler.ts` pre-filters many noise samples (via domain and path blocklists)
> (analytics/tracking/static) during analysis; the rules above cover what slips through.

### Recover Write Ops from Noisy Names

Compiler generates names like `createAnswersVoters` (upvote),
`createMembersFollowers` (follow), `createStatusesSetlike` (like). Before deleting
any POST, cross-reference against discovery write intents. Check URL path:

- `/answers/{id}/voters` or `/like` → upvote/like
- `/members/{id}/followers` or `/friendships/create` → follow
- `/{resource}/{id}/favorites` → bookmark/save

### Keep Bar

Keep a non-target operation only if it meets ALL of:

1. Returns structured JSON (not HTML, not empty)
2. On the primary domain (not off-domain CDN/analytics)
3. Has clear user value (e.g., `getMe`, `getHotSearch`, `getRecommendFeed`)
4. Can be given a meaningful name — if you cannot name it, it is noise

**Write op test:** "would a user knowingly trigger this action?" If no, exclude.
Real write ops: like, follow, bookmark, post, comment, vote, repost.
NOT: `createCollectorApm`, `createZaLogsBatch`, `createCommercialEcommerce`.

Every kept operation must have a meaningful `operationId` and a useful `summary`.
If you cannot determine what an operation does, exclude it.

---

## 2. Rename Operations and Summaries

### operationId Convention

camelCase, semantic verb + noun:

| Auto-generated | Curated |
|---|---|
| `getApiV1SearchResults` | `searchProducts` |
| `listGraphql` | `searchUsers` |
| `getUsersUser` | `getUserProfile` |
| `createAnswersVoters` | `upvoteAnswer` |

- Name by user action, not URL path or endpoint technology
- GraphQL: name by query/mutation intent, not endpoint
- Write ops: verb matching user action (`likePost`, `followUser`, `addToCart`)

### Summary Convention

Describe the user action with 3-5 key response fields so an agent can decide
whether the operation returns what it needs.

| Bad | Good |
|---|---|
| "Get video detail info" | "Get video detail — title, play count, likes, favorites, uploader info" |
| "Search content" | "Search Q&A by keyword — title, excerpt, vote count, author" |

---

## 3. Remove Anti-Bot and Fingerprint Parameters

Browser-generated tokens handled by adapter/page transport do not belong in
the operation's parameter list. Remove them.

> **Scrubbing vs parameter removal:** `src/compiler/curation/scrub.ts` automatically
> strips PII from example *values*. Removing anti-bot *parameters* (e.g.
> `dm_img_*`, `w_rid`) from the schema itself is a MANUAL curation step.

| Parameter(s) | Source |
|---|---|
| `dm_cover_img_str`, `dm_img_inter`, `dm_img_list`, `dm_img_str` | Bilibili |
| `w_rid`, `wts` | Bilibili wbi signing |
| `__a`, `__d`, `__s`, `__req`, `__rev` | Meta/Instagram/Facebook |
| Any param with `<REDACTED_TOKEN>` or fragmented JSON example | Various |

**Rule:** If a param's purpose is unclear and its example value is random/opaque,
it is anti-bot infrastructure. Remove it.

---

## 4. Improve Parameter Descriptions

Auto-generated descriptions cover only common names (`q`, `page`, `limit`).
All site-specific parameters need manual descriptions.

| Issue | Fix |
|---|---|
| Generic `id` | Add context: "Subreddit name (without r/ prefix)" |
| `queryId` | Note the GraphQL operation it references |
| Opaque IDs | "Post ID (base-36, e.g. '1jqk8w')" |
| Missing constraints | "Number of results (max 100)" |
| Enum-like params | "Sort order: 'hot', 'new', 'top'" |

Every parameter must be self-explanatory without reading the site's source code.

---

## 5. Set Permissions and Curate Write Ops

### Permission Table

| Method | Permission |
|---|---|
| GET / HEAD | `read` |
| GraphQL queries (even via POST) | `read` |
| POST / PUT / PATCH (mutations) | `write` |
| DELETE | `delete` |
| GraphQL mutations | `write` |
| Financial transactions | `transact` |

Auto-curation defaults are usually correct. Verify edge cases:
- GraphQL queries via POST should be `read`, not `write`
- Idempotent POST endpoints that only fetch data should be `read`

> **Two-stage permission derivation:** `apply-curation.ts` `defaultPermission()`
> returns `write` for any POST/PUT/PATCH (no `transact` detection). The `transact`
> permission is auto-assigned later at GENERATION time by
> `src/lib/permission-derive.ts` via `TRANSACT_PATTERNS`
> (checkout/purchase/payment/order/subscribe in the path).

### Write Op Curation

Write ops need extra attention:

1. **Naming:** verb matching user action (`likePost`, not `createStatusesSetlike`)
2. **Permission:** `write`, `delete` (destructive), or `transact` (financial)
3. **Safety level:** document SAFE / CAUTION / NEVER in DOC.md (see `add-site/document.md`)
4. **Manual verify:** test in a safe context, then undo:
   ```bash
   openweb <site> exec likePost '{"postId": "..."}'
   openweb <site> exec unlikePost '{"postId": "..."}'
   ```
   Or use `openweb verify <site> --write` with a disposable test context.

> Replay safety details: `curate-schemas.md` (Replay Safety section).

---

## 6. Merge with Existing Package

> Load this section only if the site already exists in `src/sites/<site>/`.
> For new sites, skip to the checklist.

Merge decisions are curation work — install is a dumb overwrite of the final
curated package.

### Merge Process

**Step 1 — Read existing package.** Note in the existing `openapi.yaml`:

- Write operations (`permission: write`) — manually authored
- Adapter references (`x-openweb.adapter`) — custom code
- Complex auth (`exchange_chain`, `page_global`, `sapisidhash`, `webpack_module_walk`)
- Custom `$ref` schemas in `components/`

**Step 2 — Copy new spec to temp location.** Do not overwrite the existing spec.

**Step 3 — Merge operations:**

- Add genuinely new operations (new paths not in existing spec)
- For ops in both: keep existing if it has better schemas/params or was manually
  curated; take new if existing was a stub
- **NEVER delete existing write operations**
- **NEVER delete existing adapter references**

> Permissions on existing operations are preserved as-is during merge. Only
> NEW operations receive auto-derived defaults from `apply-curation.ts`.

**Step 4 — Merge auth:** Keep existing complex auth. If existing has no auth and
new detected `cookie_session` + CSRF, take the new config.

**Step 5 — Preserve adapters:** Never copy adapter files from new package.
Existing adapter directory is always authoritative.

**Step 6 — Output:** The merged spec is the final curated artifact. Subsequent
steps (doc, verify, install) operate on this result.

---

## Curation Checklist

Before moving to schema review (`curate-schemas.md`) or runtime config
(`curate-runtime.md`), confirm:

- [ ] All noise operations removed (analytics, tracking, CDN, framework internals)
- [ ] Real write ops recovered from noisy compiler names
- [ ] Non-target ops pass keep bar (structured JSON, primary domain, user value, nameable)
- [ ] Every `operationId` is semantic verb+noun in camelCase
- [ ] Every `summary` describes the user action with 3-5 key response fields
- [ ] Anti-bot and fingerprint parameters removed
- [ ] All site-specific parameters have clear descriptions
- [ ] Permissions set correctly (read/write/delete/transact)
- [ ] GraphQL queries via POST marked as `read`, not `write`
- [ ] Write ops named by user action, tested manually or flagged for `--write` verify
- [ ] If recompiling: merge completed per merge process above


path: add-site/curate-runtime.md
content: # Curate: Runtime Configuration

Auth, transport, extraction, and adapter configuration in `openapi.yaml`. Load
during curation when the site needs auth, bot detection workarounds, SSR
extraction, or adapter-based operations.

> Full x-openweb field schema: `references/x-openweb.md`

## x-openweb Quick Reference

**Server-level** (`servers[0].x-openweb`) — applies to all operations:

| Field | Type | Required |
|-------|------|----------|
| `transport` | `node` \| `page` | Yes |
| `auth` | AuthPrimitive | If site has auth |
| `csrf` | CsrfPrimitive + `scope` | If site has CSRF |
| `signing` | SigningPrimitive | If site has signing |
| `auth_check` | AuthCheckPrimitive | When site signals "unauthenticated despite HTTP 200" via body shape |
| `headers` | `Record<string, string>` | If site needs constant headers (e.g., User-Agent override) |

**Operation-level** (per-operation `x-openweb`) — overrides server config:

| Field | Type |
|-------|------|
| `transport` | `node` \| `page` — override server transport |
| `auth` | AuthPrimitive \| `false` — override or disable |
| `csrf` | CsrfPrimitive \| `false` — override or disable |
| `signing` | SigningPrimitive \| `false` — override or disable |
| `extraction` | ExtractionPrimitive — SSR/DOM extraction |
| `adapter` | AdapterRef — delegates to TypeScript adapter |
| `pagination` | PaginationPrimitive — cursor or link-header |
| `build` | object — compiler metadata, **do not edit** |

---

## Transport Selection

```
Does the endpoint return JSON directly (XHR/fetch)?
  ├─ Yes → Does node work (no bot detection)?
  │     ├─ Yes → node (fastest)
  │     └─ No → page (browser-fetch)
  └─ No → Is data in SSR HTML?
       ├─ Yes, simple (single expression) → extraction
       └─ No, or complex → adapter
```

> **Probe-first rule:** Transport is now an INPUT from the probe step, not
> discovered during curation. The probe matrix records a transport hypothesis
> (`node_candidate`, `page_required`, `adapter_required`, `intercept_required`,
> or `extraction`). Curation configures the spec to match that hypothesis.
> `node_candidate` is provisional — only trusted after verify passes.

- **`node`** — Direct HTTP from Node.js. Auth tokens extracted from browser once,
  cached. Fast path — default unless bot detection blocks it.
- **`page`** — Executes `fetch()` inside the browser tab. Required when Akamai,
  PerimeterX, or DataDome block direct HTTP. Slower but bypasses client-side checks.
- **Mixed sites:** Set server-level to `page`, override node-friendly ops per-operation:

```yaml
servers:
  - url: https://api.example.com
    x-openweb:
      transport: page
paths:
  /api/v1/categories:
    get:
      operationId: getCategories
      x-openweb:
        transport: node   # this endpoint works without browser
```

> Bot detection details and transport impact: `knowledge/bot-detection.md`

---

## Auth Configuration

Auth is **site-level** — applies to all operations. Never remove auth to make
read-only verify pass. Write operations depend on it.

### Auth Types

| Type | Quick signal |
|------|-------------|
| `cookie_session` | Session cookies across requests |
| `localStorage_jwt` | JWT (`eyJ…`) in localStorage |
| `exchange_chain` | POST to token endpoint → bearer |
| `webpack_module_walk` | Token in webpack chunk cache |
| `page_global` | Auth data in `window.*` global |
| `sessionStorage_msal` | MSAL tokens in sessionStorage |

> Full primitive config: `knowledge/auth-primitives.md` —
> Quick routing (signals → family): `knowledge/auth-routing.md`

### Example: Cookie Session + CSRF

```yaml
servers:
  - url: https://api.example.com
    x-openweb:
      transport: node
      auth:
        type: cookie_session
      csrf:
        type: cookie_to_header
        cookie: ct0
        header: x-csrf-token
        scope: [POST, PUT, DELETE]
```

### Cross-Domain Auth

When the token lives on a different domain than the API, use `app_path` with an
absolute URL. The resolver opens a temporary page to read the token storage.
Applies to `localStorage_jwt` and `webpack_module_walk`.

```yaml
auth:
  type: localStorage_jwt
  key: BSKY_STORAGE
  path: session.currentAccount.accessJwt
  app_path: https://bsky.app   # token domain ≠ API domain
  inject:
    header: Authorization
    prefix: "Bearer "
```

### CSRF Types

| Type | Mechanism | Key fields |
|------|-----------|------------|
| `cookie_to_header` | Cookie value → request header | `cookie`, `header` |
| `meta_tag` | `<meta>` tag content → header | `name`, `header` |
| `api_response` | Endpoint → extract → inject | `endpoint`, `extract`, `inject` |

`scope` controls which methods require CSRF — typically `["POST", "PUT", "DELETE"]`.
Some sites require CSRF on GET too (X/Twitter, LinkedIn) — check captured traffic.
For `api_response`, make `endpoint` an **absolute HTTPS URL**. The resolver SSRF-validates that URL directly, then fetches it in page context so rotated cookies stay aligned with the browser jar.

### Signing

One runtime primitive: `sapisidhash` (YouTube/Google). Custom signing
(X-Bogus, `x-client-transaction-id`) requires adapter + page transport — not a
runtime primitive.

---

## Per-Operation Overrides

Disable server-level auth/CSRF/signing on genuinely public operations:

```yaml
paths:
  /api/v1/public/categories:
    get:
      operationId: getCategories
      x-openweb:
        auth: false
        csrf: false
        signing: false
```

Never remove site-level auth — override with `false` at the operation level.

---

## Extraction vs Adapter

### Extraction (Declarative YAML)

Use when a single `page_url` + expression reaches the data, including
parameterized URLs (e.g., `/dp/{asin}`). The executor resolves path params,
navigates, and evaluates.

| Type | Use when | Key fields |
|------|----------|------------|
| `ssr_next_data` | Next.js `__NEXT_DATA__` | `page_url`, `path` |
| `page_global_data` | `window.*` global | `page_url`, `expression`, `path` |
| `html_selector` | Data in DOM elements | `page_url`, `selectors`, `attribute` |
| `script_json` | `<script type="application/json">` | `selector`, `path` |

**Extraction priority** (high to low):
1. **API** (JSON response) — cleanest, most stable, clearest schema
2. **SSR JSON** (`__NEXT_DATA__`, `__INITIAL_STATE__`, LD+JSON) — structured data, no render dependency
3. **In-page JSON** (script tags, window globals) — structured but less predictable sources
4. **DOM** — last fallback, slowest and most fragile. If DOM extraction exceeds ~5 lines, escalate to adapter.

**Complexity rule:** If expression exceeds ~5 lines, move to an adapter.
Inline OK for simple `ssr_next_data`, `page_global`, short `html_selector`.

> Pattern catalog: `knowledge/extraction.md`

> **Adapter is a first-class lane.** Adapter/intercept is a normal routing
> outcome from the probe step, not a late escalation from failed replay.
> If probe evidence says adapter is the right path, route there directly.

### Adapter (TypeScript Code)

Use when extraction isn't enough:
- Multi-step navigation, dynamic waits, click/scroll interactions
- Multiple operations sharing the same real URL (differentiated by params or DOM region)
- Complex request signing that only runs in browser context
- DOM interaction beyond a single `page.evaluate()` expression

```yaml
x-openweb:
  adapter:
    name: site-name
    operation: operationName
```

The runner contract is a single `run(ctx: PreparedContext)` entry. The runtime injects
shared helpers on `ctx.helpers`:

- `helpers.pageFetch(page, { url, method?, body?, headers?, timeout? })` — browser-context fetch, returns `{ status, text }`
- `helpers.graphqlFetch(page, { url, operationName, variables, hash?, query?, batched? })` — GraphQL fetch, returns unwrapped `data`
- `helpers.ssrExtract(page, source, path?)`, `helpers.jsonLdExtract(page, typeFilter?)`, `helpers.domExtract(page, spec)` — extraction helpers
- `helpers.errors` — error factories: `unknownOp`, `missingParam`, `httpError`, `apiError`, `needsLogin`, `botBlocked`, `fatal`, `retriable`, `wrap`

`nodeFetch` and `interceptResponse` are **not** injected on `ctx.helpers`. Import them directly from `../../../lib/adapter-helpers.js` when needed; the build step bundles those imports into the emitted adapter `.js`.

### Intercept Pattern

When the site's own JS must trigger the API call (e.g., client-side signing
like JD's `h5st`, Akamai sensor-blocked `page.evaluate(fetch)`), use the
shared `interceptResponse()` helper from `src/lib/adapter-helpers.ts`:

```typescript
import { interceptResponse } from '../../../lib/adapter-helpers.js'

const data = await interceptResponse(page, {
  urlMatch: '/api/v1/items/',
  navigateUrl: `https://example.com/items/${slug}`,
  waitUntil: 'domcontentloaded',
  timeout: 20_000,
})
```

See `knowledge/adapter-recipes.md` §Response Interception for the full pattern
and decision boundary vs `pageFetch`.

Real examples:
- `src/sites/homedepot/adapters/homedepot-web.ts` — generic `interceptGraphQL()` helper
- `src/sites/jd/adapters/jd-global-api.ts` — multi-API interception from one navigation
- `src/sites/instacart/adapters/instacart-graphql.ts` — dual request+response interception

Adapter `.ts` files are authored directly in `src/sites/<site>/adapters/`.
`pnpm build` compiles `.ts` → `.js` and syncs to `$OPENWEB_HOME` and `dist/`.

### Adapter Path Semantics

For non-adapter operations, the OpenAPI path **is** the real URL path.

For adapter operations, the path is a **logical namespace** — the runtime does
NOT navigate to it. It uses `operationId` to route to the adapter. The adapter
must use `params` to navigate.

```yaml
# Logical paths — NOT real URLs:
/search/images:     # real: /search?q=...&udm=2
/search/news:       # real: /search?q=...&tbm=nws
/search/knowledge:  # real: /search?q=... (sidebar)
```

**Consequences:**
1. The adapter must call `page.goto()` — runtime only opens the server origin
2. Use `params` to build URLs — never write `_params` (unused)
3. Path naming is free — choose names clear to API consumers

#### CustomRunner Patterns

**No more `init()` / `isAuthenticated()`.** PagePlan handles navigation and
readiness at the runtime layer; auth is pre-resolved from the declared
primitive into `ctx.auth`. The runner only owns site-specific acquisition.

Declare the page lifecycle in the spec, not the runner:
```yaml
servers:
  - url: https://www.example.com
    x-openweb:
      transport: page
      page_plan:
        entry_url: /app
        ready: '.content'
        warm: true
```

> **`warm_origin` default:** when `entry_url.origin !== serverUrl.origin`, warm
> uses the entry_url origin; otherwise warm uses serverUrl. Override with
> explicit `warm_origin: 'page' | 'server' | <URL string>`.

The runner becomes a single `run(ctx)` entry with a switch on `ctx.operation`:
```typescript
import type { CustomRunner, PreparedContext } from '../../../types/adapter.js'

const runner: CustomRunner = {
  name: 'example-web',
  description: 'Example site runner',
  async run(ctx: PreparedContext) {
    const page = ctx.page!   // null only when transport: node
    switch (ctx.operation) {
      case 'getItems': {
        const result = await ctx.helpers.pageFetch(page, { url: `/api/items?q=${ctx.params.q}`, method: 'GET' })
        if (result.status >= 400) throw ctx.helpers.errors.httpError(result.status)
        return JSON.parse(result.text)
      }
      default:
        throw ctx.helpers.errors.unknownOp(ctx.operation)
    }
  },
}
export default runner
```

**Do not** hand-roll `page.goto` / `waitForSelector` / cookie warmup in the
runner — PagePlan already did that. **Do not** add a runner just to chain
two calls or to rename wire fields — expose ops separately and document in
SKILL.md.

`ctx.auth` is the only request material pre-resolved into `PreparedContext`. Server-level CSRF/signing config is not injected into adapter ops automatically.

---

## Common Mistakes

1. **Removing site-level auth for verify.** Auth exists for write ops. Use
   `auth: false` per-operation for public endpoints.

2. **"Needs browser for each request" in DOC.md.** With `node` + `cookie_session`,
   cookies are extracted once and cached. Say "cookies extracted automatically."

3. **Hand-rolling navigation in a runner.** PagePlan's `entry_url` + `ready`
   handle navigation + readiness before `run(ctx)` is invoked. Don't write
   `page.goto` / `waitForSelector` inside the runner — declare them in
   `x-openweb.page_plan` on the operation or server.

4. **Editing `build` fields.** Compiler-managed — never change manually.

5. **Extraction when an API exists.** If the site makes XHR/fetch calls, use
   the API directly. Extraction is for SSR-only data.

6. **`replay_safety` placed in spec `x-openweb` instead of `example.json`.** The spec
   uses `safety` (values: `safe`, `caution`). The `example.json` uses `replay_safety`
   (values: `safe_read`, `unsafe_mutation`).
7. **Assuming `auth: false` also disables signing.** It does not. Public ops that must bypass signing or CSRF need their own `signing: false` / `csrf: false`.


path: add-site/curate-schemas.md
content: # Curate Schemas

Response schema enrichment, example review, and replay safety. Loaded during
the Curate step when response shapes or examples need work.

> Naming, noise removal, permissions: `add-site/curate-operations.md`
> Auth, transport, extraction config: `add-site/curate-runtime.md`

## When to Load

Load this file when any of:
- A read operation has `type: object` with no `properties` in its 200 response
- Captured responses contained empty arrays for list endpoints
- The site uses SSR extraction (LD+JSON, `__NEXT_DATA__`, DOM) where schemas
  are hand-written
- You need to review examples for PII before install

---

## Bare Object Schemas

`type: object` with no `properties` in a 200 response means the schema carries
zero information — agents cannot know what fields the response contains. The
compiler infers schemas from captured responses, but under-sampled or
hand-crafted specs often produce bare schemas.

### How to Enrich

1. For each read op with a bare `type: object` response, execute it:
   ```bash
   openweb <site> exec <op> '{"param":"value"}'
   ```
2. From the actual JSON response, infer a schema (properties, types, nested
   objects).
3. Replace the bare schema in `openapi.yaml`.

### Schema Depth Rules

- **Recommended max 2-3 levels deep** (curation guideline; the compiler enforces
  `maxDepth: 10` in `schema-v2.ts`). Use `type: object` for deeply nested
  sub-objects that agents rarely need to inspect.
- **Arrays:** Describe the item schema. Don't leave `items: { type: object }`
  bare — at minimum include the key fields.
- **Nullable fields:** Mark with `nullable: true` when the field may be absent
  or null. Common nullable fields: user bio, profile image, optional metadata,
  pagination cursors (null on last page).

### Acceptable Bare Schemas

Not every bare schema needs enrichment:
- Operations that return truly opaque objects (e.g., raw API proxy responses)
- Empty 204 responses (no body)

---

## Empty-Array Fallback

When a captured response contains an empty array (e.g., `{"items": []}`), the
compiler infers `items: { type: object }` with no properties — the same bare
schema problem. There is no compiler warning for this case.

### Fix

Execute the operation with parameters that return a non-empty list, then enrich
the item schema from the actual response:

```bash
# Use a popular/common query to get non-empty results
openweb <site> exec <op> '{"q":"popular-term","limit":5}'
```

If no non-empty response is available (the list is genuinely always empty for
the operation), document the bare schema in `DOC.md` Known Issues and move on.

---

## Nullable Fields in Adapter Packages

Adapter-only packages (no compiled spec) have hand-written schemas. Mark fields
as nullable when the adapter may return `null` for optional data:

```yaml
properties:
  bio:
    type: string
    nullable: true
  avatar_url:
    type: string
    nullable: true
```

> Note: OpenAPI 3.1 prefers `type: [T, 'null']` (or `oneOf: [{type: T}, {type: 'null'}]`)
> over the legacy 3.0 `nullable: true`. Both parse correctly. This repo uses both:
> 3.0-style `nullable: true` (e.g. GitHub) and 3.1-style `type: [T, 'null']`
> (e.g. Yahoo Finance). Choose consistently within a site.

If the adapter extracts from DOM and an element is missing, the field should be
`nullable: true` rather than omitted — this gives agents a stable schema
contract. Common nullable fields: user bio, profile image, optional metadata,
pagination cursors (null on last page).

---

## LD+JSON Schema Caveat

Sites using LD+JSON extraction (`<script type="application/ld+json">`) embed
schema.org-vocabulary data (Hotel, Product, Recipe, etc.). This affects schema
curation in two ways:

**Schema shape follows schema.org, not the site's internal API.** The response
schema must match what the LD+JSON block actually contains — `@type`, `name`,
`aggregateRating`, etc. — not what the site's internal REST/GraphQL API would
return. Do not copy field names from the site's API docs if the extraction
source is LD+JSON.

> Filter LD+JSON for data-only fields; exclude `@context`, `@id`, `@type`,
> `@graph` metadata from the response schema unless meaningful to users.

**Multiple LD+JSON blocks per page.** A single page may embed several blocks
with different `@type` values. The adapter or extraction expression must filter
by `@type` to get the right block. The response schema should reflect only the
fields from the target `@type`, not a union of all blocks.

**Official docs are misleading here.** Sites with public APIs document their
*official* API shapes, which use the site's own field names and auth. LD+JSON
extraction bypasses the API entirely — the schema follows schema.org vocabulary.
Do not mix the two.

---

## Official-Doc Fallback Caveat

When enriching schemas (whether from API responses or extraction), you may find
official API documentation for the site. Be cautious:

- Official docs describe the *official* API with its own auth (API keys, OAuth
  apps), which differs from openweb's browser-session auth.
- Schema shapes may differ between official and internal APIs — different field
  names, nesting, or available fields.
- Use official docs as a *hint* for field names and types, but always verify
  against actual `exec` output.
- Do not copy official schemas verbatim — the internal API openweb targets may
  return different fields or structures.

---

## Review Examples for PII

Check parameter examples in the spec and example fixtures
(`examples/*.example.json`):

- **Real usernames, emails, phone numbers, addresses?** Replace with generic
  values (`"user123"`, `"[REDACTED_SECRET]"`).
- **Auth tokens or session IDs in examples?** Remove entirely.
- **Location data, IP addresses, device IDs?** Scrub or generalize.

The auto-scrubber catches common patterns (emails, phone numbers, SSNs) and
generic long IDs (>20 chars via `LONG_ID_RE` in `src/compiler/curation/scrub.ts`),
but manually flag anything it might miss — especially:
- Site-specific user IDs that could identify real users
- Domain-specific slug/ID formats the generic regex won't catch (e.g.,
  Starbucks `53646-283069` storeNumber)
- Internal URLs or hostnames
- Timestamps that pinpoint exact user activity

---

## Replay Safety

The compiler sets `replaySafety` per operation and writes it to
`examples/*.example.json` as `replay_safety`. This controls verify behavior.

| Replay safety | Permission | Verify behavior |
|---|---|---|
| `safe_read` | `read` | Included in `openweb verify` by default |
| `unsafe_mutation` | `write` / `delete` | Skipped by default; include with `--write` flag |

### What to Check

- Auto-curation defaults are usually correct. The compiler sets `safe_read` for
  GET/HEAD and GraphQL queries, `unsafe_mutation` for POST/PUT/PATCH/DELETE and
  GraphQL mutations.
- **GraphQL queries via POST:** Verify these have `permission: read` and
  `replay_safety: safe_read`, not `write`/`unsafe_mutation`. Auto-curation
  handles this (checks `graphql.operationType === 'query'`), but double-check.
- **Idempotent writes:** Some POST endpoints are actually safe reads (e.g.,
  search via POST body). If confirmed safe, override to `safe_read` so verify
  covers them.

### Testing Write Operations

Write ops are skipped during standard verify. To test them:
```bash
# Include write/delete ops in verify
openweb verify <site> --write

# Or test manually in a safe context
openweb <site> exec likePost '{"id":"your-own-post"}'
# Then undo: openweb <site> exec unlikePost '{"id":"your-own-post"}'
```

---

## Checklist

Before moving to verify, confirm:

- [ ] No bare `type: object` responses remain (except acceptable cases)
- [ ] Empty-array item schemas enriched with actual response data
- [ ] Adapter schemas use `nullable: true` for optional DOM-extracted fields
- [ ] LD+JSON extraction schemas follow schema.org vocabulary, not API docs
- [ ] Examples scrubbed of PII (usernames, emails, tokens, addresses)
- [ ] Replay safety correct for all operations (especially GraphQL-over-POST)
- [ ] Schema depth max 2-3 levels; deeper nesting uses `type: object`


path: add-site/document.md
content: # Document

Write per-site documentation and update knowledge after any add-site workflow.

Three outputs per site:

| File | Audience | Purpose | Update mode |
|------|----------|---------|-------------|
| **SKILL.md** | Users / agents | How to use — overview, operations, workflows, quick start, known limitations | In-place, follows /skill-creator format |
| **DOC.md** | Developers / maintainers | How it works internally — architecture, transport decisions, probe evidence, adapter patterns, pitfalls | In-place, always reflects current implementation |
| **PROGRESS.md** | Developers / maintainers | What happened — chronological entries, discovery journey, verification results | Append-only |

Optionally update **knowledge/** if you learned something general.

---

## SKILL.md — Per-Site User Doc

SKILL.md is for the agent (or human) *using* the site. It answers: "what can I
do here, and how do operations connect?"

### Layering with openapi.yaml

SKILL.md and `openapi.yaml` serve different roles — avoid duplication:

| Layer | Carries |
|---|---|
| **openapi.yaml** | Structural truth — params, types, schemas, endpoints (machine-readable) |
| **SKILL.md** | Semantic truth — intent, workflows, cross-op data flow, gotchas (agent-readable) |

SKILL.md covers only what `openapi.yaml` cannot express:

1. **Cross-operation data flow** — `channelId <- listGuildChannels` (OpenAPI has no concept of this)
2. **Intent mapping** — what each operation is for, when to use it
3. **Workflows** — multi-step sequences connecting operations
4. **Non-obvious behavior** — rate limits, required login, response quirks

For full param lists, types, and response schemas: `openweb <site> <op>`.

### Template

```markdown
# <Site Name>

## Overview
One-liner: what this site is, what archetype (e-commerce, social, etc.)

## Workflows

Common multi-step flows showing cross-operation data flow:

### [Workflow name, e.g., "Find and read messages"]
1. `listGuilds` -> pick guild -> `guildId`
2. `listGuildChannels(guildId)` -> pick channel -> `channelId`
3. `getChannelMessages(channelId, limit)` -> messages

### [Another workflow]
1. `searchProducts(query)` -> results with `productId`
2. `getProductDetail(productId)` -> full product info

## Operations

| Operation | Intent | Key Input | Key Output | Notes |
|-----------|--------|-----------|------------|-------|
| listGuilds | list my servers | — | id, name, icon | entry point |
| listGuildChannels | channels in server | guildId <- listGuilds | id, name, type | |
| getChannelMessages | read messages | channelId <- listGuildChannels | id, content, author | paginated |

## Quick Start

\```bash
# [Intent description]
openweb <site> exec <op> '<full JSON params>'

# [Another intent]
openweb <site> exec <op> '<full JSON params>'
\```

## Known Limitations
- [User-facing limitations: rate limits, missing operations, auth requirements]
```

### Workflow Completeness Standard

**Every write/mutation operation must have a documented read→write chain** showing
which read operation provides the required IDs and params, with explicit
`→ fieldName` arrows tracing each value to its source.

A workflow is **incomplete** if a write op appears without a preceding read op
that supplies its required parameters.

**Canonical example — Uber Eats `addToCart`:**

```markdown
### Add item with customizations
1. `getRestaurantMenu(storeUuid)` → find item → `itemUuid`, `sectionUuid`, `subsectionUuid`
2. `getItemDetails(storeUuid, sectionUuid, subsectionUuid, menuItemUuid)` → `customizationsList` with groups
3. `addToCart(storeUuid, itemUuid, customizations)` → `customizations = {groupUuid: [{uuid: optionUuid, quantity: 1}]}`
```

Key properties of a complete write workflow:
- Every param in the write op traces back to a specific read op's output field
- Field names are explicit (`→ itemUuid`, not just "get the item")
- Multi-hop chains show each intermediate step (read → read → write)
- Constraints are noted inline (`minPermitted`, `maxPermitted`)

**Anti-pattern** — vague workflow missing data flow:
```markdown
### Add comment
1. Get the item ID
2. `addComment(parent, text)`
```

**Correct** — explicit field tracing:
```markdown
### Add comment
1. `getStoryDetail(id)` → `item.id`
2. `addComment(parent=item.id, text)`
```

#### Self-creating chained fixtures

When a write op needs an ID that isn't externally stable (e.g. a tweet you
own, a cart-item generated server-side), don't hand-pin a fixture — chain
the op behind a create op in the same example file using
`${prev.<opId>.<jsonpath>}`. For text fields that the upstream rejects on
duplicates, use the `${now}` template helper for per-run uniqueness. See
`src/lib/template-resolver.ts` for both helpers.

Canonical example — **x write-op cascade**:

1. `createTweet` (order:1) — text uses `${now}` so Twitter's
   duplicate-status (187) check never fires across runs.
2. `likeTweet`, `createBookmark`, `createRetweet`, `reply` (order:2-9) —
   each `tweetId` is `${prev.createTweet.create_tweet.tweet_results.result.rest_id}`.
3. `deleteTweet` (order:20) — cleans up the parent.

Result: every run is hermetic. No fixture rot, no manual ID refresh, no
shared-state pollution. Apply this pattern whenever the create op's
permission is `write` and the inverse op exists in the same site.

### Column guide for Operations table

- **Operation**: operationId from openapi.yaml
- **Intent**: what this achieves (short phrase)
- **Key Input**: main params + source (`<- source_operation`). Omit trivial params
- **Key Output**: key response fields the user cares about (not full schema)
- **Notes**: pagination, rate limits, gotchas. Mark entry points (no input dependencies)

The `<- source` annotations are the soul of this table — they turn a flat list
into a directed graph so the agent knows where to get each param.

### Required vs optional sections

**Required:** Overview, Workflows, Operations, Quick Start. If a required
section has nothing interesting (e.g., single-operation site), a one-liner is
enough.

**Optional:** Known Limitations — include when there are user-visible
constraints worth documenting.

### Documenting skipped ops

If an op ships with `examples/<op>.example.json.skip` (renamed from
`.example.json`), it is intentionally excluded from `verify` and won't
appear in PASS/FAIL counts. Document each one in **Known Limitations**
with the reason and the unblocker. Common reasons:

- Upstream renamed an internal mutation/queryId — needs a fresh capture
  from the live page (e.g. x `deleteDM`: `DMMessageDeleteMutation` no
  longer in Twitter's bundles).
- Op needs externally-produced state we can't synthesize in a chain
  (e.g. x `hideReply`/`unhideReply`: Twitter blocks hiding self-replies,
  so the op needs a 2nd account replying to our test tweet).
- Op produces side-effects on shared state with no rollback path
  (e.g. ops that consume one-shot tokens or post to public channels).

Format:

```markdown
## Known Limitations
- `deleteDM` — skipped (`.example.json.skip`); Twitter renamed
  `DMMessageDeleteMutation`. Unblock: capture new queryId + opName from
  a live DM-delete in managed Chrome, update the adapter map.
- `hideReply` / `unhideReply` — skipped; Twitter blocks hiding own
  replies on own thread. Unblock: have a 2nd account reply to a test
  tweet, then point fixture at that reply ID.
```

---

## DOC.md — Per-Site Developer Doc

DOC.md is for the developer or maintainer working on the site package. It
answers: "how does this site work internally, and what should I know before
changing it?"

This file consolidates what was previously split across old DOC.md internals
and summary.md.

### Template

```markdown
# <Site Name> — Internals

## API Architecture
- REST / GraphQL / SSR / hybrid?
- What domain(s) does the API live on?
- Unusual patterns (persisted queries, protobuf, JSONP, etc.)

## Auth
- Auth type (cookie_session, page_global, etc.)
- How tokens are obtained
- CSRF/signing requirements

## Transport
- node, page, or adapter? Why?
- If mixed: which ops use which transport and why?
- If adapter: name the file (`adapters/<site>.ts`)
- Key decisions and evidence (from probe)

## Extraction
- How data is extracted (direct JSON, ssr_next_data, html_selector, page_global, adapter)
- Parsing quirks

## Adapter Patterns
- [If applicable] Key patterns used in adapter code
- Helper functions, data transformations

## Known Issues
- Bot detection? (DataDome, PerimeterX, etc.)
- Rate limiting?
- Dynamic fields causing verify DRIFT?
- Pitfalls encountered during implementation

## Probe Results
- [Settled conclusions only — not the raw probe matrix]
- Transport hypothesis and final outcome per family
```

### Required vs optional sections

**Required:** Auth, Transport, Known Issues. If a section has nothing
interesting, a one-liner is enough: `No auth required.`

**Optional:** API Architecture, Extraction, Adapter Patterns, Probe Results —
include when non-obvious.

---

## PROGRESS.md — History Trace

Append-only log. Each entry records what happened, why, and what was verified.

### Entry template

```markdown
## YYYY-MM-DD — [Activity Type]

**Context:** [Why this work was done]
**Changes:** [What changed]
**Verification:** [Results]
**Key discovery:** [Optional — non-obvious finding worth preserving]
**Pitfalls encountered:** [Optional — mistakes or dead ends worth documenting]
```

Example:

```markdown
## 2026-04-11 — Transport Upgrade

**Context:** Probe discovered open GraphQL API at api.graphql.imdb.com
**Changes:** 3/4 ops migrated from SSR __NEXT_DATA__ to node GraphQL
**Verification:** 4/4 PASS
**Key discovery:** Introspection disabled, but error messages expose schema
**Pitfalls encountered:** ratings histogram only in __NEXT_DATA__, not GraphQL
```

Fixture-refresh example (chained write ops):

```markdown
## 2026-04-19 — Fixture Chain Rewrite

**Context:** 10 of 19 write ops failing in aggregate verify due to dead
fixture IDs (deleted upstream tweets, renamed mutations, duplicate-status
on createTweet).
**Changes:** Rewrote 9 example.json files to chain via
`${prev.createTweet.create_tweet.tweet_results.result.rest_id}`. Each run
creates a fresh parent tweet (text uses new `${now}` template helper for
uniqueness), then like/bookmark/retweet/reply reuse that ID; deleteTweet
cleans up at order:20. 3 ops moved to `.example.json.skip` (see
SKILL.md Known Limitations).
**Verification:** 16/16 PASS in scope (was 9/19).
**Key discovery:** Twitter's duplicate-status (187) check has a multi-day
rolling window — static text in createTweet eventually collides; `${now}`
sidesteps it cleanly.
```

### When to write

- After initial add-site workflow -> first entry
- After any site update (new operations, auth fix, transport change)
- After knowledge learned during troubleshooting

---

## Migration from Existing Files

New sites use the three-file model directly. Old sites migrate on next touch —
no bulk migration.

### Migration mapping

| Existing content | New home |
|------------------|----------|
| DOC.md: Overview, Workflows, Operations table, Quick Start | -> SKILL.md |
| DOC.md: Site Internals, Probe Results, Known Issues (dev) | -> DOC.md (keep) |
| summary.md: Architecture, Decision, Key Patterns, Pitfalls | -> DOC.md |
| summary.md: Discovery Journey, Verification results | -> PROGRESS.md |

### Steps

1. Create `SKILL.md` from DOC.md above-the-divider content
2. Keep DOC.md below-the-divider content, merge in summary.md architecture/patterns
3. Create or extend `PROGRESS.md` with summary.md discovery/verification entries
4. Delete `summary.md` after migration
5. Remove the `---` divider convention from DOC.md (no longer needed — the split is file-level)

---

## Knowledge Update

After completing any workflow that taught you something new, decide whether to
save it and where.

### Scope decision

Is this site-specific or general?

| Scope | Where | Example |
|---|---|---|
| **Site-specific** | Site's `DOC.md` | "LinkedIn uses Voyager API with CSRF on all mutations" |
| **General pattern** | `knowledge/` files | "Next.js sites use `__NEXT_DATA__` for SSR extraction" |
| **Both** | Both, with the general version abstracted | Site DOC.md gets the specific detail; knowledge/ gets the pattern |

**Rule of thumb:** if you'd tell the next agent working on a *different* site,
it's general. If it only matters for *this* site, it's site-specific.

### Persistence test

Before saving to knowledge/, verify the learning is durable:

1. **Will this still be true in 6 months?** If no -> site-specific DOC.md only
2. **Is this already captured by the code?** If yes -> don't save
3. **Does this change how an agent should behave?** If no -> not worth saving

### Principles

- **Novel only** — don't duplicate what's in the code or existing docs
- **Patterns, not instances** — "Next.js sites use `__NEXT_DATA__`" not "Walmart uses `__NEXT_DATA__`"
- **Deduplicate** — search existing knowledge files before adding; refine rather than restate
- **Keep files < 200 lines** — split when a file grows too large

### Classify

Which knowledge file does it belong to? Search existing files first.

| File | Scope |
|------|-------|
| `knowledge/auth-routing.md` | Auth family identification signals |
| `knowledge/auth-primitives.md` | Auth, CSRF, signing config and gotchas |
| `knowledge/archetypes.md` | Expected operations by site category |
| `knowledge/bot-detection.md` | Detection systems, transport impact, capture strategy |
| `knowledge/extraction.md` | SSR data, DOM, page globals, adapter extraction |
| `knowledge/graphql.md` | Persisted queries, batching, introspection, schema |
| `knowledge/ws.md` | WebSocket message types, connection patterns |

Create a new `knowledge/<topic>.md` only if the pattern doesn't fit any existing
file.

### Write format

Follow the normalized entry format used in knowledge files:

```markdown
### Pattern Name

Description — what it is and when it occurs.

- **Detection signals:** how to recognize this pattern
- **Impact:** what this means for transport, auth, or site modeling
- **Action:** what to do when you encounter it
- **Example:** (optional) concrete example, generalized
```

Not every field is required — skip what's not relevant. The goal: the next agent
encountering this pattern can recognize it and know what to do.

### Size management

After writing, check file size. If a knowledge file exceeds 200 lines:

1. Identify a coherent subtopic that can split out
2. Create a new `knowledge/<subtopic>.md`
3. Move the relevant entries

---

## Checklist

Before marking the Document step complete:

- [ ] SKILL.md written with all required sections (Overview, Workflows, Operations, Quick Start)
- [ ] Operations table has `<- source` annotations for cross-op data flow
- [ ] Workflows section covers common multi-step intents
- [ ] Every write/mutation op has a read→write chain with explicit `→ fieldName` arrows (see Workflow Completeness Standard)
- [ ] Quick Start has copy-paste commands
- [ ] DOC.md written with all required sections (Auth, Transport, Known Issues)
- [ ] PROGRESS.md has at least one entry using the entry template
- [ ] Skipped ops (any `examples/*.example.json.skip`) are documented in SKILL.md Known Limitations with reason + unblocker
- [ ] Knowledge update: scope decision made (site-specific / general / both / nothing new)
- [ ] If general knowledge: persistence test passed, written to correct knowledge file


path: add-site/guide.md
content: # Add or Expand a Site

> **Audience: site authors** adding or expanding OpenWeb's site coverage. This workflow describes how to capture, probe, and curate a site's API surface (analogous to using browser DevTools to understand a site). End users running `openweb <site> <op>` do not enter this workflow.

Complete workflow for adding a new site package or expanding an existing one.

```mermaid
flowchart TD
    PRE["Pre-flight"] --> FRAME["1. Frame"]
    FRAME --> PROBE["2. Probe"]
    PROBE --> ROUTE{"3. Route intent families"}

    ROUTE -->|replay or ws family| CAP["4. Capture Evidence"]
    CAP --> COMP["5. Compile"]
    COMP --> REV["6. Review"]
    REV -->|missing evidence| CAP
    REV -->|wrong lane / wrong transport| PROBE
    REV -->|usable draft| BUILD["7. Build Package"]

    ROUTE -->|extraction family| BUILD
    ROUTE -->|adapter/intercept family| BUILD

    BUILD --> VER["8. Verify"]
    VER -->|wrong transport / signing / route| PROBE
    VER -->|missing traffic / missing ops| CAP
    VER -->|spec or doc issue| BUILD
    VER -->|all pass| INS["9. Install"]
    INS --> DOC["10. Document"]
```

## Pre-flight Decisions

### Mode

Your scope determines which steps to skip:

- **New site:** Run all steps.
- **Adding ops to existing site:** Skip Capture/Compile for families that already have working ops. Read existing SKILL.md and DOC.md first.
- **Upgrading transport/stability:** Skip Framing (intents already exist). Focus on Probe with fresh eyes — see knowledge/transport-upgrade.md.

When in doubt, don't skip — run the step. Skipping a probe is more expensive than running an unnecessary one.

### Incremental Mode (Existing Site)

Read the runtime package first: `DOC.md`, `openapi.yaml`, and `examples/` from
`src/sites/<site>/` or `$OPENWEB_HOME/sites/<site>/`. If you are in the source
repo, also read `src/sites/<site>/SKILL.md` and `PROGRESS.md` — those source
docs are not shipped into `$OPENWEB_HOME/sites/<site>/`. Identify gaps against
the target intents.
Only the intent families with gaps re-enter Frame and Probe; everything
else stays as-is.

### Net-New Mode

Read these knowledge files in order. Each produces a decision or a
hypothesis to confirm during Probe.

1. **`knowledge/archetypes.md`** — identify archetype, read expected operations.
   **Decision:** Target operations for discovery.

2. **`knowledge/bot-detection.md`** — check for site or vertical.
   **Hypothesis:** Expected bot-detection level (none / soft / hard). Confirm
   in Probe; runtime knobs (real Chrome profile, short sessions) are decided
   from Probe evidence, not pre-flight.

3. **`knowledge/auth-routing.md`** — scan routing table.
   **Hypothesis:** Likely auth family for the archetype. Probe confirms
   whether target endpoints actually require auth. Default: if any target
   intent plausibly needs auth, log in before Capture — unauthenticated
   capture silently misses auth-required endpoints.

### Pre-flight Exit Criteria

- Target site archetype understood
- Likely auth family noted
- Likely bot-detection level noted
- Scope reduced to the families that actually need work

## Critical Rules

### Browser First, No Direct HTTP

**NEVER use curl, fetch, wget, or direct HTTP to probe a site.** Bot
detection tracks IP reputation. A single non-browser request raises the risk
score, poisoning subsequent browser sessions.

### Write Operation Safety

| Level | Examples | Rule |
|-------|----------|------|
| SAFE | like, bookmark, follow, add-to-cart | Capture freely, reversible |
| CAUTION | send message (to self), post (then delete) | Only in safe contexts |
| NEVER | purchase, delete account, send to others | Do not trigger |

Verify skips write ops by default (`replaySafety: unsafe_mutation`).

---

## Step 1: Frame

Define 3-5 target intents as **user actions**, not API names.

**Group into intent families.** An intent family is a group of target intents
that share the same surface and likely the same runtime lane. Example:
- `searchProducts` -> search surface
- `getProductDetail`, `getProductReviews`, `getProductPrice` -> detail surface

For each family:
- mark as read, write, or realtime
- identify one representative page or flow
- note whether safe writes must be exercised during later capture

Create or update `src/sites/<site>/SKILL.md` with initial overview and
target-intent checklist, and `DOC.md` with internal notes.
Read `add-site/document.md` for templates.

**Write intents** — add writes for core interactions (social: like/follow/
bookmark/repost; commerce: add-to-cart/wishlist). Perform ALL safe writes
during capture — missing one means missing that operation.

**Exit criteria:** Every target intent belongs to a family. Each family has
one representative page or flow. Draft SKILL.md and DOC.md exist or are updated.

---

## Step 2: Probe

Discover data source, transport constraints, auth/signing signals, and the
runtime lane for each family — **before** committing to capture or compile.

**Rules:** browser only, no direct Node HTTP probes, one family at a time,
keep sessions short, watch for block pages / redirect loops / rate limits as
early stop signals.

### Probe Protocol Summary

Run the probe in one managed-browser CDP session. Full protocol with CDP code,
troubleshooting, and edge cases: `add-site/probe.md`.

1. **Connect and navigate** — CDP attach, go to representative URL, wait for hydration
2. **Health check** — detect challenge/block patterns in title/URL
3. **Inventory SSR sources** — `__NEXT_DATA__`, LD+JSON, `__INITIAL_STATE__`, DOM selectors
4. **Intercept network/WS** — new page, listener before navigation, filter static/tracking
5. **Test browser-side fetch** — replay one API via `page.evaluate(fetch)`, check shape match
6. **Inspect auth/CSRF/signing/bot-detection** — cookies, headers, rotation patterns
7. **Record transport hypothesis** — `node_candidate` | `page_required` | `adapter_required` | `intercept_required` | `extraction`
8. **Persist probe matrix** — write to DOC.md `## Internal: Probe Results`

### Probe Matrix

One row per family in DOC.md. See `add-site/probe.md` for full format and
column definitions. Common values (not enums — use natural language for novel
situations): evidence kind (`api`, `graphql`, `ssr`, `dom`, `mixed`), lane
(`replay`, `extraction`, `adapter/intercept`, `ws`), capture mode (`none`,
`micro`, `targeted`, `broad`), compile role (`required`, `helpful`, `skip`).

If the family is write-oriented, note login/CSRF requirements. If GraphQL
persisted queries are present, record explicitly. If rate-limited, stop and
route conservatively.

**Exit criteria:** For every family — evidence kind known, transport hypothesis
recorded, auth/CSRF/signing hypothesis recorded, lane chosen, capture mode
chosen, compile role chosen.

---

## Step 3: Route Intent Families

Convert probe evidence into a family lane and artifact plan.

### Routing Table

| Probe evidence | Lane | Capture mode | Compile role | Build direction |
|---|---|---|---|---|
| Rich SSR/JSON/DOM data, no useful API | extraction | none | skip | declarative extraction if simple, adapter if complex |
| Stable replayable REST/GraphQL, browser replay works, compile will help | replay | micro or targeted | required | capture-backed compile flow |
| Stable replayable REST/GraphQL, browser replay works, compile may help little | replay | none or micro | helpful | conservative default is still micro or targeted |
| Site JS succeeds but browser replay fails, or per-request signing rotates | adapter/intercept | none or micro | helpful | adapter or passive intercept path |
| Multi-step UI, DOM interaction, logical ops not 1:1 with URLs | adapter | none or micro | optional | adapter path |
| WS is primary data channel | ws | targeted or broad | required | capture-backed WS compile flow |

### Routing Rules

- Route by intent family, not whole site. Mixed sites are expected.
- Adapter/intercept is a normal route outcome, not a late escalation.
- Do not claim final `node` transport from probe alone — probe can only
  produce `node_candidate`.

### Skip Rules

**Skip Capture** when: the family has a clean extraction path; the family has
a direct adapter/intercept path and compile would add little; the family is
unchanged and existing verified evidence still holds.

**Skip Compile** when: no HAR was collected for the family; the family is
extraction-only; the family is adapter/intercept-only and compile would not
materially improve schema/examples/auth understanding.

**Skip Review** when: compile was skipped for that family.

**Skip Broad Capture** unless: the surface is still unknown after probe; WS
discovery needs it; micro or targeted capture did not produce enough evidence.

**Exit criteria:** Every family has a lane, a capture mode, and a compile role.

---

## Step 4: Capture Evidence

Collect only the traffic needed for families that Route assigned a capture
mode other than `none`.

```bash
openweb capture start --isolate --url https://<site-domain>
# browse to trigger each target intent
openweb capture stop --session <session-id>
```

`--isolate` records only the dedicated tab's traffic. Output goes to
`./capture/` by default, or `./capture-<session-id>/` with `--isolate`
(unless `--output <dir>` overrides it).

Read `add-site/capture.md` for capture modes, auth injection, scripted
capture, and troubleshooting.

### Capture Mode Guidance

| Mode | When to use |
|---|---|
| **micro** (1-3 requests) | Endpoint already known; compile could save manual schema/example/auth work; broad capture would mostly add noise |
| **targeted** | Family needs param variation; GraphQL clustering matters; path normalization or pagination discovery matters; multiple sibling intents share the family |
| **broad** | Surface still not understood after probe; WS discovery needs longer observation; replay family genuinely contains many unknown operations |

### Capture Rules

- Capture follows probe findings, not replaces them.
- If auth or CSRF depends on a mutation, include at least one safe reversible
  write in the capture.
- Use in-app navigation and browser-context fetches where appropriate.
- Avoid blind browsing.

**Exit criteria:** `traffic.har` exists. Every capture-backed family has the
evidence it asked for. The HAR is intentionally scoped to the decision being
answered.

---

## Step 5: Compile

Run only for families whose capture mode is not `none`.

```bash
openweb compile <site-url> --capture-dir <capture-dir>
```

Runs: analyze -> auto-curate -> generate -> verify.

| Output | Location |
|--------|----------|
| `analysis-summary.json`, `analysis.json`, `analysis-full.json`, `verify-report.json`, `summary.txt` | `$OPENWEB_HOME/compile/<site>/` |
| `openapi.yaml`, `asyncapi.yaml` (if WebSocket operations exist), `manifest.json`, `examples/` | `$OPENWEB_HOME/sites/<site>/` |

Auto-curation accepts all clusters, picks top auth candidate, uses
suggested camelCase operation names.

### Compile's Role Under Probe-First

Compile materializes draft artifacts — it does not override probe conclusions.
Transport and extraction signals defer to probe. Compile is retained for:

- Schema inference from traffic
- Example generation
- authCandidates and csrfOptions ranking
- GraphQL sub-clustering
- Path normalization
- OpenAPI / AsyncAPI emission

**Exit criteria:** Compile-backed families have draft artifacts.
`analysis-summary.json` exists for review.

---

## Step 6: Review

Check `summary.txt`, then `verify-report.json`. Cross-check compile output
against the probe matrix.

Read `add-site/review.md` for detailed analysis reading (start with
`analysis-summary.json`).

| `driftType` | Action |
|-------------|--------|
| `auth_drift` | Auth expired or no browser for cookies |
| `schema_drift` | Response shape changed |
| `endpoint_removed` | Wrong path, network error, or site down |
| `error` | Check `detail` (e.g., "no browser tab open") |

### Failure-Based Loop Targets

| Situation | Return to |
|-----------|-----------|
| Missing family coverage, missing evidence | Capture (Step 4) |
| Wrong target domain or evidence scope | Capture (Step 4) |
| Compile output contradicts family lane or transport hypothesis | Probe (Step 2) |
| Intents mapped to operations, usable draft | Continue to Build Package (Step 7) |
| Auth contamination (off-domain cookies) | Re-capture with `--isolate` |
| Site blocked | Document in DOC.md, inform user |

**Adapter escalation** — write an adapter when: per-request headers change
every call (signing), same op returns different hashes (query rotation),
or op works in browser but 404s from `page.evaluate(fetch)`.

**SSR-heavy:** Many noise ops, zero data ops = SSR-delivered data. Write
an adapter. **GraphQL persisted queries:** Deployment-scoped hashes; on
"PersistedQueryNotFound", re-capture or adapter. See `knowledge/graphql.md`.

**Exit criteria:** Every compile-backed family is either accepted for build or
explicitly routed back to Probe or Capture. All target intents have operations.

---

## Step 7: Build Package

Assemble all families into one site package. This is the merge step for mixed
sites — replay-backed operations from compile output, extraction operations
from probe evidence, and adapter/intercept operations from direct authoring
all converge here.

Edit the generated spec and docs. Artifacts stay in
`$OPENWEB_HOME/sites/<site>/`.

1. **Merge** (if existing) — read `add-site/curate-operations.md`
2. **Operations** — `add-site/curate-operations.md` (noise, naming, params)
3. **Runtime** — `add-site/curate-runtime.md` (auth/CSRF, transport, extraction)
4. **Schemas** — `add-site/curate-schemas.md` (schemas, examples/PII)
5. **DOC.md** — if you can't write a clear workflow, revisit naming/grouping
6. **PROGRESS.md** — append entry

### Adapter/Intercept Build Guidance

Read `add-site/curate-runtime.md` § Intercept Pattern for the canonical
`interceptApi()` template and real examples (Home Depot, JD, Instacart).

Adapter openapi.yaml paths are logical namespaces (not real URLs). Each
adapter operation needs:

```yaml
x-openweb:
  adapter:
    name: site-name
    operation: operationName
  transport: page
```

Adapter `.ts` files are authored directly in `src/sites/<site>/adapters/`
(see Package Lifecycle in Install step).

### Build Rules

- Transport starts from the probe hypothesis.
- `node_candidate` must still survive verify before it is trusted.
- Keep auth site-level unless an operation is genuinely public.
- Use compile artifacts when present; add manual examples when compile was
  skipped.
- Keep intercept families on passive interception during implementation
  instead of forcing blind recapture.
- After direct `src/sites/` edits for adapter-only or extraction-only
  families, run `pnpm build` before runtime verify when cache precedence
  would otherwise hide the new files.

**Auth preservation:** Auth is site-level. Even if read ops pass without it,
removing auth/csrf from `servers[0].x-openweb` breaks all write ops.

### Param Traceability

Every operation parameter must trace to a concrete source. Before finalizing
the package, walk each write/mutation op and verify:

1. **For each required param** — which read op's response field provides it?
2. **Document the chain** in SKILL.md with `→ fieldName` arrows
3. **Multi-hop chains** (read → read → write) must show every intermediate step

If a write op param cannot be traced to any read op's output, either:
- A read op is missing from the spec (add it), or
- The param source is unclear (document it explicitly — user-provided, constant, etc.)

**Canonical example** — Uber Eats `addToCart` param traceability:

| Param | Source |
|-------|--------|
| `storeUuid` | `searchRestaurants` → `storeUuid` |
| `itemUuid` | `getRestaurantMenu(storeUuid)` → `catalogItems[].uuid` |
| `customizations` | `getItemDetails(storeUuid, sectionUuid, subsectionUuid, menuItemUuid)` → `customizationsList[].options[].uuid` |

Every `←` in the Operations table and every `→` in a Workflow must have a
matching field name — no anonymous arrows.

**Exit criteria:** `openapi.yaml` and/or `asyncapi.yaml` represent all target
families. Extraction and adapter configuration exist where needed. Examples
exist for runtime verify. DOC.md and PROGRESS.md updated enough for
verification.

---

## Step 8: Verify

**Must be performed by an independent agent** — not the agent that built
the package.

Read `add-site/verify.md` for the full process covering three dimensions:
- **Runtime** — do operations return data?
- **Spec** — does the spec follow curation standards?
- **Doc** — does DOC.md follow the template?

### Failure-Based Loop Table

| Failure | Return to |
|---|---|
| 403 / 999 / bot block / redirect loop / wrong signing | Probe (Step 2) |
| Missing operation / missing evidence / missing write-time token | Capture (Step 4) |
| Schema, naming, doc, or merge-quality issue | Build Package (Step 7) |
| Auth expired | Login and rerun verify |

### Verify Rules

- Runtime must cover every target intent with at least one working operation.
- Spec must meet curation standards.
- Doc must match the spec and the real workflows.
- Node transport is only trusted after verify passes under runtime conditions.

**Exit criteria:** Runtime, spec, and doc all pass per `add-site/verify.md`.

---

## Step 9: Install

Copy the verified package to the source tree.

```bash
mkdir -p src/sites/<site>
cp -r $OPENWEB_HOME/sites/<site>/* src/sites/<site>/
pnpm build && pnpm test
```

Verify: `ls src/sites/<site>/openapi.yaml`, `openweb sites`, `openweb <site>`.

### Package Lifecycle

Three paths: `$OPENWEB_HOME/sites/` (compile cache),
`src/sites/` (source — edit here), `dist/sites/` (build).

Install behavior depends on how the package was built:
- **Compile-backed families:** copy staged non-adapter artifacts from
  `$OPENWEB_HOME/sites/<site>/` into `src/sites/<site>/`.
- **Adapter/extraction families with no compile:** the authoritative files
  are already in `src/sites/<site>/`; Install is primarily a
  confirmation/build step, not a copy step.
- **Mixed packages:** install only the staged non-adapter artifacts and
  preserve source-authored adapter files.
- Source-only docs (`SKILL.md`, `PROGRESS.md`) do not come from the staged
  runtime package; keep editing them in `src/sites/<site>/`.

Do not overwrite `adapters/`. Run `pnpm build` after source edits —
it syncs FROM `src/sites/` TO cache/dist, overwriting cache edits.

**Exit criteria:** Build and tests pass. Source-tree files confirmed.

---

## Step 10: Document

Read `add-site/document.md` for the three-file model, templates, and knowledge
update criteria.

Update all three per-site doc files:

1. **SKILL.md** — Finalize user-facing doc: overview, workflows with cross-op
   data flow, operations table, quick start commands, known limitations.
2. **DOC.md** — Finalize developer doc: auth, transport decisions and evidence,
   adapter patterns, known issues. Clean up probe results to settled
   conclusions only.
3. **PROGRESS.md** — Append entry with context, changes, verification results,
   key discoveries, and pitfalls encountered.
4. **manifest.json** — Verify site metadata is current.

- Document family lanes used by the site, transport limits, bot-detection
  constraints, and why compile was skipped for any family (in DOC.md).
- If you learned something that generalizes across sites, write to
  `knowledge/`.
- If you hit pipeline friction (not site-specific), write
  `src/sites/<site>/pipeline-gaps.md`.

**Exit criteria:** SKILL.md, DOC.md, and PROGRESS.md finalized per templates.
Knowledge updated if applicable. Pipeline gaps documented if any.


path: add-site/probe.md
content: # Probe Protocol

Full CDP protocol for browser-side discovery of data sources, transport constraints, and auth signals per intent family.

**Load when:** guide.md Step 2 (Probe), or when re-probing after verify failure.

---

## Rules

- Browser only -- no direct Node HTTP probes
- One family at a time
- Keep sessions short (single CDP connection, minimal page loads)
- Watch for block pages, redirect loops, and rate limits as early stop signals
- If blocked, note the block and stop probing that family until login/challenge resolution

## Pre-Probe Checklist

Before running CDP code, do a quick manual sanity check:

1. Open DevTools Network/XHR -- if zero JSON responses appear, it is SSR/DOM-only.
2. Check `view-source:` for `__NEXT_DATA__`, `__INITIAL_STATE__`, LD+JSON -- these are extraction targets (use `x-openweb.extraction`), not adapter-only signals.
3. Try the on-page search box -- some sites serve SSR for direct nav but use JSON APIs for in-app search/pagination.
4. Check if the mobile app calls a hidden public API the web version does not use.

---

## Probe Steps

### 2a. Connect and Navigate

Open a managed-browser CDP session and navigate to the representative page for this intent family.

```typescript
const browser = await chromium.connectOverCDP('http://localhost:9222')
const context = browser.contexts()[0]!
const page = context.pages()[0]!
await page.goto(representativeUrl, { waitUntil: 'load', timeout: 30_000 })
await page.waitForTimeout(3000) // hydration
```

### 2b. Health Check

Read `page.title()` and inspect the URL and title for challenge or block patterns:

- `challenges.cloudflare.com` in URL
- `captcha-delivery.com` in URL
- Title contains `access denied` or `just a moment`

If blocked, note the block and stop probing that family until login or challenge resolution.

### 2c. Inventory SSR and Extraction Sources

Check for structured data already present in the page -- these are extraction candidates that avoid API replay entirely.

```typescript
const ssr = await page.evaluate(() => ({
  nextData: !!document.querySelector('#__NEXT_DATA__'),
  ldJsonCount: document.querySelectorAll('script[type="application/ld+json"]').length,
  initialState: typeof (window as any).__INITIAL_STATE__ !== 'undefined',
  nuxt: typeof (window as any).__NUXT__ !== 'undefined',
  preloadedState: typeof (window as any).__PRELOADED_STATE__ !== 'undefined',
}))
```

Also check obvious DOM selectors for target data (e.g., `.product-title`, `.price`).

**Extraction priority** (prefer higher, use lower only when higher is unavailable):

1. **API** (JSON response) -- cleanest, most stable, clearest schema
2. **SSR JSON** (`__NEXT_DATA__`, `__INITIAL_STATE__`, LD+JSON) -- structured data, no render dependency
3. **In-page JSON** (script tags, window globals) -- structured but less predictable sources
4. **DOM** -- last fallback, slowest and most fragile; if DOM extraction exceeds ~5 lines, escalate to adapter

### 2d. Passively Intercept Network and WS Traffic

Open a **new page** in the same context. Set listeners **before** navigation so nothing is missed.

```typescript
const apis: Array<{url: string, method: string, status: number, contentType: string}> = []
const probePage = await context.newPage()
probePage.on('response', async (resp) => {
  const ct = resp.headers()['content-type'] ?? ''
  if (ct.includes('json') && resp.status() < 400) {
    apis.push({
      url: resp.url(), method: resp.request().method(),
      status: resp.status(), contentType: ct,
    })
  }
})
await probePage.goto(targetUrl, { waitUntil: 'load', timeout: 30_000 })
await probePage.waitForTimeout(5000) // let XHR/fetch calls complete
```

Filter out static and tracking requests by URL pattern. Note:
- API domains and path patterns
- Query params
- GraphQL operation names
- Persisted-query hashes

### 2e. Test Browser-Side Fetch Feasibility

When APIs are found in 2d, pick one representative API URL and replay it via `page.evaluate(fetch)`:

```typescript
const testUrl = apis[0].url
const fetched = await page.evaluate(async (url) => {
  try {
    const r = await fetch(url, { credentials: 'include' })
    const body = await r.json()
    return { status: r.status, topKeys: Object.keys(body) }
  } catch (e) { return { status: 0, topKeys: [] } }
}, testUrl)
// Compare fetched.topKeys to the intercepted response shape.
// Same shape -> browser replay works.
// Empty/error/different -> signing required -> adapter/intercept lane.
```

If no APIs found in 2d, skip this step.

**CORS caveat for cross-origin APIs:**
- This replay test is only meaningful for same-origin requests.
- If the discovered API lives on a different origin, browser fetch may fail because of CORS rather than signing or bot detection.
- For cross-origin APIs, first navigate to the API origin or another same-origin page on that subdomain before using browser-context fetch as a feasibility test.
- Do not classify a family as `intercept_required` from a cross-origin CORS failure alone.

**Interpretation:**

| Outcome | Meaning |
|---|---|
| Same shape as intercepted response | Browser replay works -- `node_candidate` or `page_required` depending on bot-detection evidence |
| Empty / error / different shape | Signing or session binding required -- `adapter_required` or `intercept_required` |
| CORS error on cross-origin URL | Inconclusive -- re-test from same origin before concluding |

### 2f. Inspect Auth, CSRF, Signing, Bot-Detection Signals

```typescript
const cookies = await context.cookies()
const botCookies = cookies.filter(c =>
  ['_abck', '_px3', '_pxhd', 'datadome', 'cf_clearance'].includes(c.name) ||
  c.name.startsWith('ry_ry-')
)
```

Inspect:
- **Cookies** for session auth signals (names like `session`, `sid`, `token`, `auth`)
- **Intercepted request headers** for `Authorization`, CSRF-like cookie-to-header patterns
- **Per-request headers or params** for rotation between intercepted requests (signing signal)
- **Bot-detection cookies:** `botCookies.length > 0` means `page_required` even if browser-side fetch works (Node will likely fail)
- **Monkey-patched fetch (signing signal):** Run `await page.evaluate(() => window.fetch.toString().length)`. Native `fetch.toString()` is short (~40 chars). If length > ~100, `fetch` is monkey-patched — the site injects signing/header logic into every request. This means node transport cannot bypass the signing layer; route to `adapter_required` or `intercept_required`.

### 2g. Record Transport Hypothesis

Based on all evidence from 2a-2f, record one hypothesis per family:

| Hypothesis | Condition |
|---|---|
| `node_candidate` | Browser-side replay works AND no strong bot-detection evidence |
| `page_required` | Browser replay works BUT bot-detection evidence suggests Node may still fail |
| `adapter_required` | Family needs browser interaction or logic beyond direct replay |
| `intercept_required` | Site's own JS can fetch data but programmatic replay cannot |
| `extraction` | SSR/JSON/DOM is the simplest stable path |

> **Transport decision guidance:** Before committing to a transport hypothesis, run the **Node Feasibility Quick-Check** in [`knowledge/transport-upgrade.md`](../knowledge/transport-upgrade.md) §Node Feasibility Quick-Check — a 3-step curl-based test that validates whether node transport is actually viable for each family.

### 2h. Persist the Probe Matrix

Write the family decision to `DOC.md` under `## Internal: Probe Results`. One row per family.

**Format:**

```markdown
## Internal: Probe Results

| Family | Representative page/action | Evidence kind | Replay test | Auth/CSRF/signing | Transport hypothesis | Lane | Capture mode | Compile role |
|---|---|---|---|---|---|---|---|---|
| detail | product page load | api + intercept | browser fetch blocked | custom signing | intercept_required | adapter/intercept | none or micro | helpful |
| search | in-page search | ssr + dom | n/a | none | extraction | extraction | none | skip |
```

**Column definitions** (common values, not enums -- agent may use natural language for novel situations):

| Column | Common values |
|---|---|
| Evidence kind | `api`, `graphql`, `ws`, `ssr`, `dom`, `mixed` |
| Transport hypothesis | `node_candidate`, `page_required`, `adapter_required`, `intercept_required`, `extraction` |
| Lane | replay, extraction, adapter/intercept, ws |
| Capture mode | `none`, `micro`, `targeted`, `broad` |
| Compile role | `required`, `helpful`, `skip` |

These values are consumed by agents reading the guide to make routing decisions, not by code. Do not hardcode as enums -- the routing table matches on evidence semantics, not literal values.

---

## Additional Probe Guidance

- **Write-oriented families:** Ensure the probe notes whether login and CSRF evidence will require a safe write during later capture.
- **GraphQL persisted queries:** If present, record explicitly -- this affects compile usefulness and replay stability.
- **Rate limiting:** If rate limiting appears during probe, stop escalating requests and route conservatively.

---

## Probe Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| Challenge/captcha page (Cloudflare, DataDome, PerimeterX) | Bot detection triggered before probe completes | Stop probe, resolve challenge or login first, then re-probe |
| Zero API responses in 2d | Site is SSR-only or APIs fire only on user interaction | Check pre-probe checklist items 1-3; try triggering in-app search or pagination before concluding extraction-only |
| Browser fetch returns different shape than intercepted response | Per-request signing, token rotation, or server-side session binding | Route to adapter/intercept lane; do not force replay |
| CORS error on browser fetch | API is cross-origin; failure is CORS policy, not signing | Navigate to API origin first and re-test; do not classify as `intercept_required` from CORS alone |
| Rate limit during probe (429, throttle page) | Too many probe requests in short session | Stop escalating, record what you have, route conservatively based on partial evidence |
| DOC.md claims site needs browser, but probe suggests node is viable | Stale transport assessment in existing docs | Do not trust old docs — verify against [`knowledge/transport-upgrade.md`](../knowledge/transport-upgrade.md) §Disproved Assumptions Pattern. Probe evidence overrides historical DOC.md claims. |

---

## Related Files

- `add-site/guide.md` -- main workflow (probe is Step 2)
- `add-site/capture.md` -- capture modes and evidence collection
- `knowledge/auth-routing.md` -- auth family routing patterns
- `knowledge/bot-detection.md` -- bot-detection signals and mitigation
- `knowledge/extraction.md` -- extraction lane details
- `knowledge/transport-upgrade.md` -- node feasibility quick-check, stability ladder, disproved assumptions


path: add-site/review.md
content: # Reviewing the Analysis Report

How to review the analysis output and decide: continue to curation, re-capture, or stop.

## Conditional Step

Review only applies to families where compile ran. If compile was skipped
(extraction-only or adapter-only families), there is nothing to review — those
families proceed directly to Build Package (guide.md Step 7).

## Probe Matrix Cross-Check

Before diving into analysis output, re-read the probe matrix from
`DOC.md ## Internal: Probe Results`. Review compares compile output to probe
findings:
- Treat probe findings as the **classifier** (lane, transport hypothesis)
- Treat compile findings as **artifact evidence** (schemas, auth ranking, clusters)
- If compile output contradicts the probe-determined lane or transport, the
  family routes back to Probe (guide.md Step 2), not forward to Build

## Which File to Read

**Read `analysis-summary.json` first.** It is a compact (<100 KB) subset of the
full report containing `summary`, `clusters`, `authCandidates`, and
`extractionSignals` — everything needed for the review steps below. It omits
`samples` and `navigation` arrays, which are large and rarely needed.

Only fall back to `analysis.json` (the full stripped report) when you need
per-sample detail — e.g., inspecting individual request/response bodies,
debugging a specific cluster's member requests, or verifying raw header values.

## What to Read and What to Skip

| Section | Where | Action |
|---|---|---|
| `summary` | `analysis-summary.json` | **Read first** — health check (~10 lines) |
| `authCandidates` | `analysis-summary.json` | **Read** — auth review (~10-30 lines) |
| `clusters` | `analysis-summary.json` | **Read** — coverage mapping (~20-100 lines/cluster) |
| `extractionSignals` | `analysis-summary.json` | **Read if SSR suspected** (~5-20 lines) |
| `ws` | `analysis-summary.json` | **Read if WS expected** |
| `samples` | `analysis.json` only | **Skip unless needed** — every labeled request, huge |
| `navigation` | `analysis.json` only | **Skip** — page-level groups, debugging only |

## Review Order

1. Summary signals
2. Auth candidates
3. Clusters
4. Extraction signals
5. WebSocket analysis (if expected)
6. Coverage decision

---

## 1. Summary Signals

Read the first ~30 lines (covers `version` through `summary`):

- **`summary.byCategory.api`** — API request count. Zero or very low = capture
  missed target traffic.
- **`summary.byCategory.off_domain`** — if high, the API lives on a different
  domain. Re-compile using the API domain as the site URL.
- **`summary.clusterCount`** — candidate operations found.

## 2. Auth Candidates

> Before reading: check `knowledge/auth-routing.md` for expected auth type.

Search for `"authCandidates"` in `analysis-summary.json`. For the rank-1 candidate:

- **`auth.type`** — matches expectation from `knowledge/auth-routing.md`?
- **`confidence`** — >0.7 reliable; 0.3-0.7 suspect (cross-check with routing
  table); <0.3 no meaningful auth (expected for public APIs, investigate if
  login required).
- **`evidence.matchedCookies`** — real auth cookies or tracking cookies?
  Tracking cookies (`__cf_bm`, `_ga`, `__gads`, `datadome`) should NOT appear.
  If they're the only matched cookies, the candidate is a false positive —
  check lower-ranked candidates or confirm the site is public.
- **`csrf`** — social sites with write ops usually need it. Check `csrf.type`
  (`cookie_to_header` or `meta_tag`) and cookie/header names.
- **`evidence.notes`** — human-readable detection rationale.

Empty or all-zero-confidence = no auth detected. Expected for public APIs;
red flag for sites requiring login.

### CSRF Verification

Auto-detected CSRF may be wrong. Check the top-level `csrfOptions` array in
analysis-summary.json — each entry is a CsrfPrimitive scored by
`scoreCsrfMatch()` (auth-candidates.ts:228).

**False positives:** locale cookies (`lc-main=en_US` -> `x-li-lang`), preference
cookies — short values, not tokens.

**Real CSRF signals:** headers named `csrf-token`, `x-csrf-token`; cookies
named `csrftoken`, `_csrf`; values are long random strings (>10 chars).

**Override:** edit `openapi.yaml` directly after compile:
- Set `x-openweb-csrf` with the correct type, cookie, and header values

> Full auth/CSRF primitive details: `knowledge/auth-primitives.md`

## 3. Clusters

Search for `"clusters"` in `analysis-summary.json`.

### Mapping Target Intents

Map each target intent to a cluster. Missing cluster = not captured, return
to capture. Per cluster check:

- **`suggestedOperationId` / `suggestedSummary`** — detected operation
- **`method` + `pathTemplate`** — HTTP shape
- **`sampleCount`** — request count

### Path Normalization

`/users/123` and `/users/456` should normalize to `/users/{id}`. If paths
got incorrectly merged, `normalization.originalPaths` shows what was collapsed.

### GraphQL Clusters

> For details: `knowledge/graphql.md` (persisted queries, batching)

- `graphql.operationName` — each query should get its own cluster
- `graphql.discriminator` — split method (`operationName`, `queryId`,
  `persistedQueryHash`, or `queryShape`)
- High `sampleCount` (100+) on `/graphql` with NO sub-cluster metadata =
  all operations collapsed. Fix: capture with more varied UI interactions.

### Cluster Red Flags

- **4xx-only clusters** — auth-required endpoints or stale URLs; cross-ref
  with `authCandidates`
- **Very high `sampleCount`** (100+) — GraphQL collapse, polling endpoint,
  or unfiltered analytics (exclude during curation)
- **`parameters`** — check required flags and example values
- **`responseVariants`** — observed status codes and content types

Note clusters to exclude and names to change — applied during curation.

## 4. Extraction Signals

> Decision flow and pattern details: `knowledge/extraction.md`

Search for `"extractionSignals"`. Auto-detected types:

- **`ssr_next_data`** — Next.js `__NEXT_DATA__`. `estimatedSize` shows if
  real data or skeleton.
- **`script_json`** — `<script type="application/json">` blocks. `selector`
  and `id` locate the element.

Not auto-detected (check manually if suspected):
- `page_global` (`window.__INITIAL_STATE__`), `__NUXT__`, `html_selector`

**Extraction vs API replay:** weak API clusters + rich SSR data = prefer
extraction. Note for curation.

## 5. WebSocket Analysis

> Message types and curation signals: `knowledge/ws.md`

Search for `"ws"` at top level:

- `connections[].url` — data channel or telemetry?
- `connections[].executableOperationCount` — meaningful operations?
- `connections[].heartbeatCandidates` — interval and payload
- `connections[].operations[]` — patterns (`subscribe`, `stream`,
  `request_reply`, `publish`)

Heartbeat-only and presence/typing channels = noise, exclude.

## 6. Coverage Decision

### Gap Diagnosis

If target operations are missing:

| Symptom | Cause | Fix |
|---|---|---|
| No API calls for feature | SSR data | Check page source for `__NEXT_DATA__`, `window.__INITIAL_STATE__` |
| High `off_domain` count | Cross-origin API | Re-compile with API domain |
| All endpoints 401/403 | Not logged in | Log in via managed browser, re-capture |
| GraphQL mega-cluster | Ops collapsed | More varied queries during capture |
| Content after scroll only | Lazy loading | Scroll, click "load more" |
| Auth `confidence: 0`, `rejectedSignals` mentions no overlap | Capture was unauthenticated | Re-capture with login |
| CSRF missing | Token in JS, not cookie/meta | Find manually in dev tools, add to spec |

### Failure-Based Loop Targets

| Failure type | Route to | Rationale |
|---|---|---|
| Missing family coverage (target intent has no cluster) | Capture (Step 4) | Need more traffic evidence |
| Wrong target domain or evidence scope | Capture (Step 4) | Scoping problem |
| Compile output contradicts family lane or transport | Probe (Step 2) | Lane was wrong, re-discover |
| 2 capture iterations with no new clusters | Stop | Likely infeasible with current pipeline |
| Site flagged BLOCKED | Stop | Document in DOC.md Known Issues |
| Bot detection blocks all transports | Stop | Document and inform user |
| Usable draft with matched intents | Build Package (Step 7) | Proceed |

---

## Related Files

- `add-site/guide.md` — loads this at Review step
- `add-site/curate-operations.md` — used during Build Package (Step 7)
- `knowledge/auth-routing.md` — expected auth by site signal
- `knowledge/auth-primitives.md` — auth primitive config
- `knowledge/graphql.md` — GraphQL patterns
- `knowledge/extraction.md` — extraction decision flow
- `knowledge/ws.md` — WS patterns


path: add-site/verify.md
content: # Verify

Verify a site package across runtime, spec, and documentation before install.

## When to Load

- After building the package and writing DOC.md (guide.md Step 8)
- Standalone re-verification of an existing site package
- After any site update (new operations, auth fix, transport change)

**Independent verifier:** Verification should be performed by a separate agent
from the one that curated the spec. This separation catches blind spots.

## Example File Format

Example fixtures at `examples/*.example.json` are used by verify and runtime exec:

```json
{
  "operation_id": "searchProducts",
  "order": 1,
  "cases": [{
    "input": { "k": "laptop" },
    "assertions": { "status": 200, "response_schema_valid": true }
  }]
}
```

Optional fields:
- **`order`** — execution order for verify (lower = earlier). Use when operations have dependencies (e.g., addToCart before removeFromCart). Files without `order` run after ordered files, sorted alphabetically.
- **`replay_safety`** — `"safe_read"` (default) or `"unsafe_mutation"`. `unsafe_mutation` ops are skipped by default and only run when verify is invoked with `--write`.

Files without a `cases` array are treated as malformed example files and fail verify. For adapter-only
packages where compile doesn't generate examples, create them manually.

---

## Three Dimensions

All three must pass before install.

```
Verify
├── Runtime  — does the operation execute and return data?
├── Spec     — does the spec follow curation standards?
└── Doc      — does DOC.md match the spec and follow template?
```

---

## Runtime Verify

### Batch Verify

```bash
openweb verify <site>
openweb verify <site> --browser              # pre-start/keep alive the managed browser
openweb verify <site> --ops op1,op2          # only verify specific operations
openweb verify <site> --ops op1 --browser    # combine filters
```

Browser-backed ops already auto-start Chrome on demand. Use `--browser` when you
want the managed browser up front and kept alive across the verify run (useful
for auth cascades, CAPTCHA/manual-login prep, or avoiding browser idle shutdown).
Use `--ops` to debug individual operations without running the full suite.
The `--ops` value is a comma-separated list of operation IDs with **no spaces**
(e.g., `--ops searchStores,getMenu`, not `--ops searchStores, getMenu`).

#### Status Interpretation

| Status | Meaning | Action |
|--------|---------|--------|
| `PASS` | Works | Continue to runtime exec |
| `DRIFT` | Works but type change or required field missing detected | Advisory warning (exit 0). Review schema if persistent |
| `auth_expired` | Session expired | `openweb login <site>`, `openweb browser restart`, rerun |
| `FAIL` | Execution failed | Read detail line, fix spec or environment, rerun |
| `FAIL` (403 + cookies) | CSRF misconfiguration | Check `authCandidates[0].csrfOptions` in analysis.json |

Single-site verify exits non-zero for `FAIL`, `auth_expired`, or `bot_blocked`. `DRIFT` is advisory and still exits 0.

### Runtime Exec

Batch verify checks HTTP sanity. Runtime exec proves an agent can get usable
data against the target intents identified during discovery.

Read the site's DOC.md to find target intents, then exec the best operation for
each:

```bash
openweb <site> exec <operation> '{"param": "value"}'
```

**Exit criterion:** Each target intent has at least one operation returning
real data — HTTP 2xx, valid JSON, non-empty response with expected fields.

If all pass → continue to Spec Verify.
If any fail → diagnose below.

### Diagnosis Table

| Response | Likely Cause | Fix |
|----------|-------------|-----|
| 403 | Wrong CSRF config, missing headers, expired session | Check CSRF cookie/header names. Check if CSRF scope excludes GET. If cookies missing: `openweb login <site>` |
| 401 | Session expired | `openweb login <site>`, restart browser |
| 999 / bot block | Node transport hitting bot detection | Switch to `transport: page` |
| 200 HTML (not JSON) | SSR page endpoint, not API | Remove op and use API equivalent, or add extraction config |
| 404 | Wrong path template | Fix path parameter normalization in spec |
| 400 | Bad param examples or missing required params | Update `exampleValue` fields |
| 200 empty/wrong data | Wrong query variables or response schema | Compare captured request params vs what you're sending |
| Timeout / hang | Stale token cache, browser not running | `openweb browser restart`, clear token cache |
| 429 | Rate limited by site | Verify uses 1.5s inter-op delay; if still hitting limits, add `headers` with custom UA or reduce batch size |
| Redirect loop | Auth-gated endpoint, not logged in | Log in, or remove endpoint |

After fixing the spec, return to batch verify.

### Failure-Based Loop Targets

| Failure | Return to |
|---|---|
| 403 / 999 / bot block / redirect loop / wrong signing | Probe (Step 2) — re-discover transport |
| Missing operation / missing evidence / missing write-time token | Capture (Step 4) — gather more evidence |
| Schema, naming, doc, or merge-quality issue | Build Package (Step 7) — fix spec/doc |
| Auth expired | Login and rerun verify |
| 2 cycles with no progress on same failure | Stop — document in DOC.md Known Issues |
| Bot detection blocks all transports | Stop — document blocker, inform user |
| Only non-target bonus ops fail | Proceed to Install — document in Known Issues |

> **Node transport trust:** `node_candidate` from probe is provisional.
> Node transport is only trusted after verify passes under runtime conditions.
> If node-transport operations fail with 403/999/bot-block, switch to
> `transport: page` and re-verify.

### WS Verification

If AsyncAPI operations are present, verify:

| Check | What to verify |
|-------|---------------|
| **Connection** | WebSocket connects with the detected auth |
| **Heartbeat** | Interval matches spec config |
| **Subscribe** | Subscribe operations receive expected event types |
| **Message shape** | Incoming messages are intelligible and match the expected envelope well enough for the site workflow. Current runtime verify does **not** perform WS schema diff yet |

WS failures usually stem from auth or URL issues. Check the connection URL
template and auth header injection before re-capturing.

---

## Spec Verify

Review the curated spec fresh — the verifier should not have seen curation
decisions.

Curation standards are defined in `curate-operations.md`, `curate-runtime.md`,
and `curate-schemas.md`. This checklist verifies compliance without restating
those standards.

### Checklist

| Check | Pass criterion |
|-------|---------------|
| **No noise** | No analytics, tracking, CDN, telemetry, or heartbeat-only operations |
| **No anti-bot params** | No `dm_*`, `w_rid`, `x-bogus`, `__a/__d/__s`, `msToken` in param lists |
| **Naming** | camelCase verb+noun (`searchProducts`, not `getApiV1Search`) |
| **Summaries** | Each operation has a summary listing 3-5 key response fields |
| **Auth** | `x-openweb.auth` matches site's actual pattern; CSRF present if needed |
| **Transport** | Correct per bot-detection level; browser-backed ops have the needed `page_plan`, and extraction ops have `page_url` when required |
| **Permissions** | GET → read, mutations → write/delete, GraphQL queries via POST → read |
| **Schemas** | No bare `type: object` for ops returning structured JSON |
| **No PII** | No real user data in parameter examples or fixtures |
| **Write ops** | Permission set, safety level documented, `replay_safety = unsafe_mutation` |
| **replay_safety** | `replay_safety` in `x-openweb` → spec MUST NOT contain; field belongs in `examples/*.example.json` only. Use `safety` (`safe`/`caution`) in the spec's `x-openweb` block |
| **Extraction** | Complex expressions (>5 lines) extracted to adapter files |

### Merge Integrity (existing packages only)

If curation merged with an existing package:

- Existing write operations preserved
- Existing adapter references preserved
- Existing auth config preserved (unless explicitly replaced)
- No duplicate operations (same path + method)

### Write/Undo Pair Asymmetry — Silent-Success Canary

For sites where writes return opaque-200 envelopes (YouTube InnerTube, many GraphQL mutations, IG mobile-API), HTTP status alone can't distinguish "succeeded" from "silently no-op'd". The reliable signal is **pairing the write with its inverse**:

- A `do` op (`subscribeChannel`, `followUser`, `savePin`) running unauthenticated may HTTP-200 with empty body — looks fine in isolation.
- The paired `undo` op (`unsubscribeChannel`, `unfollowUser`, `unsavePin`) correctly 401s, because there's no state to undo.
- The asymmetry — do passes, undo fails — is the **only** evidence the do-side never wrote anything.

**Action when verifying write ops:**
1. Always include both directions of a pair in `--ops` lists.
2. If one side passes and the other fails, suspect missing auth on the passing side, not a fixture problem on the failing side.
3. For an op without a natural inverse (`createPost`, `sendMessage`), follow the do-op with a confirming read (`getPost`, `listMessages`) and assert the new id appears.

This pattern surfaced YouTube's missing `sapisidhash` binding on `subscribeChannel` in the 2026-04-18 sweep (`8cfebff`). The `do` side had been silently broken for an unknown duration.

### Cross-Op Response Templating — A Closed-Input Limitation

Verify treats each example's `input` as a **closed input**: there is currently no syntax to feed a server-generated value (cart-item-id, message-id, saved-pin-id, subscription-id) from one op's response into a later op's input. This blocks any create→destroy paired write where the destroy op needs an id returned by the create op.

- **Detection signal:** the destroy op (`removeFromCart`, `unsavePin`, `deleteComment`) requires an opaque server-generated id whose only source is the create op's response body.
- **Affected sites observed:** doordash `removeFromCart`, costco `removeFromCart` + `updateCartQuantity`, target `removeFromCart`, pinterest `unsavePin`, x several pair-creates — likely 8–12 ops total across 5+ sites.
- **Current workarounds** (pick one when documenting a site):
  1. **Pre-seed a stable fixture** in the user account (e.g., trello's [REDACTED_SECRET] card in `cbfa285`) so the destroy op always has a known-real id to operate on.
  2. **Document the workflow** for agent-level use — agents can chain `addToCart` → read response → `removeFromCart` manually even though static verify cannot. Mark the destroy op in SKILL.md Operations notes with "Static `verify --write` cannot replay this — agents chain manually".
  3. **Wait for cross-op response templating** — proposed `${prev.<opId>.<json-path>}` syntax in example.json `input` would let verify substitute previous op responses (sequenced by `order`). One architectural change unlocks ~8–12 ops.
- **Action:** when adding a new write op whose required id is server-generated, do NOT claim "verified" if you can only run the create side. Document the gap in the site's DOC.md Known Issues + SKILL.md Known Limitations and use a pre-seeded fixture if one is feasible.

---

## Doc Verify

Review DOC.md against the template defined in `document.md`. The verifier
checks completeness and accuracy, not style.

### Checklist

| Check | Pass criterion |
|-------|---------------|
| **Overview** | One-liner with site archetype |
| **Workflows** | At least one multi-step workflow showing cross-operation data flow |
| **Operations table** | All operations listed with Intent, Key Input (← source), Key Output |
| **Data flow** | Every non-trivial param has a `← source` annotation |
| **Entry points** | Operations with no input dependencies marked |
| **Quick Start** | Copy-paste commands for common intents |
| **No spec duplication** | No full param lists or response schemas repeated from openapi.yaml |
| **Known Issues** | Verify failures, bot detection, rate limits documented |
| **PROGRESS.md** | File exists with at least one dated entry recording the session |

### Cross-Check with Spec

- Every operation in openapi.yaml appears in the Operations table
- No phantom operations in DOC.md that don't exist in the spec
- Workflow steps reference real operationIds

---

## Exit Criteria

All three dimensions must pass:

1. **Runtime** — Every target intent has at least one operation returning real
   data. Non-target failures documented in Known Issues.
2. **Spec** — Checklist passes. No noise, correct auth/transport/permissions.
3. **Doc** — DOC.md covers all operations, has workflows with data flow.
   PROGRESS.md exists with at least one dated entry.
   annotations, cross-checks clean against spec.

When all pass → proceed to install (guide.md Install step).

## Related Files

- `guide.md` — workflow that invokes this at Step 8 (Verify)
- `curate-operations.md` — spec standards: naming, noise, permissions
- `curate-runtime.md` — spec standards: auth, transport, extraction
- `curate-schemas.md` — spec standards: schemas, examples, PII
- `document.md` — DOC.md template and knowledge-update guidance
- `../references/troubleshooting.md` — deep debugging when verify loops fail


path: knowledge/archetypes.md
content: # Site Archetypes — Expected Operations

Quick-reference for typical operations by site category. Use during Step 1 (Frame) to set target operations before discovery.

> Archetypes are heuristic starting points, not limiting checklists. Define targets based on user needs and actual site capabilities.

## Archetype Matrix

| Archetype | Typical Examples |
|-----------|-----------------|
| **Social Media** | Short-form feed + social graph platforms |
| **Messaging** | Real-time chat + group communication |
| **Content Platforms** | Video, wiki, AI chat, community forums |
| **E-commerce** | Product search + cart + checkout |
| **Travel** | Listings + availability + booking |
| **Food Delivery** | Restaurant search + menu + ordering |
| **Job Boards** | Job search + company profiles + salary data |
| **Productivity / Enterprise** | Task/project management + collaboration |
| **Developer Tools** | Repos, issues, CI/CD, container registries |
| **Finance / Banking** | Portfolios, quotes, transactions |
| **Email** | Inbox, messages, contacts |
| **Cloud / Storage** | File management, sharing, sync |
| **Public Data APIs** | Weather, reference, fun facts |
| **News** | Headlines, articles, search |
| **Chinese Web** | Chinese-language equivalents of above categories |

---

## Social Media

- Read — core:
  - feed / timeline (paginated)
  - user profile
  - search (posts, users)
  - explore / discover feed (algorithmic recommendations)
  - trending / popular
  - notifications
  - followers list / following list
  - thread / conversation (full reply chain)
- Read — media:
  - reels / short-video feed
  - article / long-form content
  - stories (ephemeral, 24h expiry)
  - notes / status (lightweight text updates)
  - download media (images, videos)
- Write — content (reversible pairs):
  - createPost / deletePost
  - reply / deleteReply
  - comment / deleteComment
  - quote-post / deleteQuote
- Write — engagement (reversible pairs):
  - like / unlike
  - repost (retweet, reblog) / unrepost
  - bookmark / unbookmark
  - save / unsave (collections)
  - upvote / downvote / removeVote (for platforms with voting)
- Write — social graph (reversible pairs):
  - follow / unfollow
  - block / unblock
  - mute / unmute
  - acceptFollowRequest (one-way, for private accounts)
  - hideReply / unhideReply (platform-specific)
- Write — messaging (reversible pairs):
  - sendDM / deleteDM (private messages within social platforms)

## Messaging

- Read:
  - list conversations (paginated)
  - read messages (paginated)
  - list contacts
  - search messages
  - notifications
- Write (reversible pairs):
  - sendMessage / deleteMessage
  - addReaction / removeReaction
  - pinMessage / unpinMessage
- Write (one-way):
  - markAsRead
  - editMessage
- Real-time: WebSocket gateway for events

## Content Platforms

- Read — core:
  - feed / homepage (paginated)
  - content detail (by ID)
  - search
  - user / channel profile
  - comments (paginated)
  - notifications
  - explore / discover (recommendations)
- Read — media:
  - playlist / collection listing
  - transcript / captions
- Write — engagement (reversible pairs):
  - like / unlike
  - subscribe / unsubscribe
  - save / unsave
  - comment / deleteComment
- Write — library (reversible pairs):
  - addToPlaylist / removeFromPlaylist (music/video)
  - createPlaylist / deletePlaylist

> Note: some bundled content sites (e.g. youtube-music) are currently read-only — engagement/library writes are not yet implemented.

---

## E-commerce

- Read: search products, product detail (by ID), product reviews (paginated), price comparison
- Write (reversible pairs):
  - addToCart / removeFromCart
  - updateCartQuantity (set to 0 = remove)
  - saveItem (wishlist) / unsaveItem
- Read: view cart
- Transact (not yet implemented in any bundled site; placeholder for future): checkout, placeOrder

## Travel

- Read: search listings (paginated), listing detail (by ID), price/availability, reviews
- Write (reversible pairs, not yet implemented in any bundled site; placeholder for future):
  - saveProperty (wishlist) / unsaveProperty
- Transact (deny by default): book/reserve

## Food Delivery

- Read: search restaurants (paginated), restaurant menu (by ID), delivery estimate
- Write (reversible pairs):
  - addToCart / removeFromCart
- Transact (deny by default): placeOrder

## Job Boards

- Read: search jobs, job detail, company profile, salary data, reviews
- Write (reversible pairs, not yet implemented in any bundled site; placeholder for future):
  - saveJob / unsaveJob

---

## Productivity / Enterprise

- Read: list documents/items (paginated), document/item detail, search, dashboard/overview, notifications, activity feed
- Write (reversible pairs):
  - createItem / deleteItem (or archiveItem / unarchiveItem)
  - updateItem (reversible via re-update)
  - completeTask / uncompleteTask
  - assignItem / unassignItem
- Write (one-way): moveItem, addLabel, removeLabel

## Developer Tools

- Read: list repos/projects (paginated), repo detail, list issues (paginated), search, user/org profile, notifications, activity feed
- Write (reversible pairs):
  - starRepo / unstarRepo
  - createIssue / closeIssue (or reopenIssue)
  - createComment / deleteComment
  - watchRepo / unwatchRepo
- Write (one-way): forkRepo (creates new repo, no undo needed)

## Finance / Banking

- Read: account/portfolio overview, stock/asset quote (by symbol), market data, transaction history (paginated), search securities, watchlist
- Write & Transact (not yet implemented in any bundled site; bundled finance sites — robinhood, fidelity, xueqiu, yahoo-finance — are read-only; placeholder for future): addToWatchlist / removeFromWatchlist, placeOrder, transfer

## Email & Cloud / Storage

- Read: list inbox/files (paginated), read message/file metadata, search
- Write (reversible pairs):
  - starMessage / unstarMessage
  - moveToFolder / moveBack
  - archiveMessage / unarchiveMessage
- Write (one-way): sendMessage (email), uploadFile, deleteFile

---

## Public Data APIs

**Weather / Data:** Current data (read, by location/params), forecast/historical (read, by range), lookup by coordinates or ID (read)

**Prediction / Fun:** Query/predict (read, single call), random result (read)

**Reference / Lookup:** Search/list (read, paginated or filtered), detail by ID or name (read), random entry (read, if supported)

**News:** Headlines/feed (read, paginated), article detail (read, by ID/URL), search articles (read)

---

## Chinese Web

Operations vary by category — use the relevant archetype section (social, commerce, etc.) as a starting point. Common across Chinese sites:

- Search (read) — every site has some form of search
- Content detail (read, by ID) — article, video, product, stock quote
- User profile (read) — author, seller, company
- Feed / trending (read, paginated) — hot lists, rankings, recommendations
- Comments (read, paginated) — nested or flat


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector:  ()