Skill: bill492/cf-crawl (version 1.0.0)

--- name: cf-crawl description: Crawl websites using Cloudflare Browser Rendering /crawl API. Async multi-page crawl with markdown/HTML/JSON output, link following, pattern filtering, and AI-powered structured data extraction. Use when crawling entire sites or multiple pages, building knowledge bases, extracting structured data from websites, or when web_fetch is insufficient (JS rendering, multi-page, authenticated crawls). --- # Cloudflare /crawl Async site crawler via CF Browser Rendering API. Start a job → poll for results → get markdown/HTML/JSON per page. ## Quick Start ```bash # Crawl a site (5 pages, markdown, no JS rendering = fast + free) bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com" --limit 5 --format markdown # With JS rendering (for SPAs, dynamic content) bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com" --render --limit 10 # Start only (get job ID, poll later) bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com" --limit 100 --start-only # Poll existing job bash ~/clawd/skills/cf-crawl/scripts/poll.sh <job-id> ``` ## Credentials Stored at `~/.clawdbot/secrets/cloudflare-crawl.env`: ``` CF_ACCOUNT_ID=<account_id> CF_CRAWL_API_TOKEN=<token_with_read_and_edit> ``` ## Key Options | Option | Description | |--------|-------------| | `--limit N` | Max pages (default 10) | | `--depth N` | Max link depth (default 10) | | `--format markdown\|html\|json` | Output format (default markdown) | | `--render` | Enable headless browser (default: off = fast fetch, free during beta) | | `--include PAT` | Wildcard URL pattern to include (repeatable) | | `--exclude PAT` | Wildcard URL pattern to exclude (repeatable) | | `--external` | Follow external domain links | | `--subdomains` | Follow subdomain links | | `--source all\|sitemaps\|links` | URL discovery method | | `--json-prompt "..."` | AI extraction prompt (with `--format json`) | | `--json-schema file.json` | JSON schema for structured extraction | | `--timeout SEC` | Max poll wait (default 300s) | | `--output FILE` | Write full results to file | | `--raw` | Output raw API response | | `--start-only` | Print job ID without polling | ## Common Patterns ### Crawl docs site for knowledge base ```bash bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://docs.example.com/" \ --limit 50 --depth 3 --format markdown --output docs.json ``` ### Crawl with URL filtering ```bash bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com/" \ --include "/docs/**" --exclude "/docs/archive/**" --limit 20 ``` ### AI-powered structured extraction ```bash bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com/products" \ --format json --render \ --json-prompt "Extract product name, price, and description" \ --json-schema schema.json ``` ### Long-running crawl (background) ```bash JOB_ID=$(bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://big-site.com" \ --limit 1000 --start-only) # Check later: bash ~/clawd/skills/cf-crawl/scripts/poll.sh "$JOB_ID" ``` ## Cost Notes - `render: false` (default) — fast HTML fetch, **free during beta** - `render: true` — uses Browser Rendering minutes (paid) - `format json` — uses Workers AI tokens for extraction (paid) - Results cached in R2 with `--max-age` (default 24hr) ## API Details See `references/api-reference.md` for full parameter documentation, response schema, and lifecycle details.

path: references/api-reference.md
content: # Cloudflare /crawl API Reference

## Endpoint
```
POST https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl
GET  https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}
DELETE https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}
```

## Auth
`Authorization: Bearer <API_TOKEN>` — token needs Browser Rendering Read + Edit.

## POST body (start crawl)

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `url` | string | *required* | Starting URL |
| `limit` | number | 10 | Max pages (max 100,000) |
| `depth` | number | 100,000 | Max link depth from start URL |
| `source` | string | `"all"` | `all`, `sitemaps`, `links` |
| `formats` | string[] | `["html"]` | `html`, `markdown`, `json` (can combine) |
| `render` | boolean | `true` | `false` = fast HTML fetch, no JS. Free during beta. |
| `maxAge` | number | 86400 | R2 cache seconds (max 604800 = 7 days) |
| `modifiedSince` | number | — | Unix timestamp; only crawl pages modified after |
| `options.includeExternalLinks` | boolean | false | Follow external domain links |
| `options.includeSubdomains` | boolean | false | Follow subdomain links |
| `options.includePatterns` | string[] | — | Wildcard: `*` (not `/`), `**` (any) |
| `options.excludePatterns` | string[] | — | Higher priority than include |
| `jsonOptions.prompt` | string | — | AI extraction prompt (requires `json` format) |
| `jsonOptions.response_format` | object | — | JSON schema for structured extraction |
| `jsonOptions.custom_ai` | object | — | BYO model instead of Workers AI |

### Render-related options (when render=true)
- `rejectResourceTypes` — block images/fonts/stylesheets
- `waitForSelector` — CSS selector to wait for (SPA support)
- `gotoOptions` — Puppeteer goto options
- `setExtraHTTPHeaders` — custom headers for authenticated crawls
- HTTP basic auth and cookies also supported

## GET response (poll/retrieve)

```json
{
  "success": true,
  "result": {
    "id": "job-uuid",
    "status": "completed|running|errored|cancelled_due_to_timeout|cancelled_due_to_limits|cancelled_by_user",
    "browserSecondsUsed": 1.23,
    "total": 50,
    "finished": 50,
    "skipped": 0,
    "records": [
      {
        "url": "https://...",
        "status": "completed|queued|disallowed|skipped|errored|cancelled",
        "markdown": "...",
        "html": "...",
        "json": { ... },
        "metadata": {
          "status": 200,
          "title": "Page Title",
          "url": "https://...",
          "lastModified": "..."
        }
      }
    ],
    "cursor": 10
  }
}
```

### GET query params
- `limit` — max records to return
- `cursor` — pagination cursor (returned when >10MB)
- `status` — filter: `queued`, `completed`, `disallowed`, `skipped`, `errored`, `cancelled`

## Lifecycle
- Max runtime: 7 days
- Results available: 14 days after completion
- Polling tip: use `?limit=1` for lightweight status checks

## Cost
- `render: true` — uses Browser Rendering minutes (paid)
- `render: false` — fast HTML fetch, currently **free during beta**
- `formats: ["json"]` — uses Workers AI tokens (paid)


path: scripts/crawl.sh
content: #!/usr/bin/env bash
# cf-crawl: Start a Cloudflare crawl job and poll for results
# Usage: crawl.sh <url> [options]
#   --limit N        Max pages (default 10, max 100000)
#   --depth N        Max link depth (default 10)
#   --format FMT     markdown|html|json (default markdown)
#   --render         Enable JS rendering (default: false/fast fetch)
#   --include PAT    Include URL pattern (repeatable)
#   --exclude PAT    Exclude URL pattern (repeatable)
#   --external       Follow external links
#   --subdomains     Follow subdomain links
#   --source SRC     all|sitemaps|links (default all)
#   --max-age SEC    Cache max age in seconds (default 86400)
#   --json-prompt P  Prompt for JSON AI extraction (requires --format json)
#   --json-schema F  Path to JSON schema file (requires --format json)
#   --poll-interval  Seconds between polls (default 5)
#   --timeout SEC    Max seconds to wait (default 300)
#   --output FILE    Write results to file (default stdout)
#   --raw            Output raw API response (no formatting)
#   --start-only     Start job, print ID, don't poll

set -euo pipefail

# Load credentials
if [[ -f ~/.clawdbot/secrets/cloudflare-crawl.env ]]; then
  source ~/.clawdbot/secrets/cloudflare-crawl.env
fi
: "${CF_ACCOUNT_ID:?Set CF_ACCOUNT_ID}"
: "${CF_CRAWL_API_TOKEN:?Set CF_CRAWL_API_TOKEN}"

BASE="https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/browser-rendering/crawl"

# Defaults
LIMIT=10
DEPTH=10
FORMAT="markdown"
RENDER=false
INCLUDES=()
EXCLUDES=()
EXTERNAL=false
SUBDOMAINS=false
SOURCE="all"
MAX_AGE=86400
JSON_PROMPT=""
JSON_SCHEMA=""
POLL_INTERVAL=5
TIMEOUT=300
OUTPUT=""
RAW=false
START_ONLY=false
URL=""

# Parse args
while [[ $# -gt 0 ]]; do
  case "$1" in
    --limit) LIMIT="$2"; shift 2;;
    --depth) DEPTH="$2"; shift 2;;
    --format) FORMAT="$2"; shift 2;;
    --render) RENDER=true; shift;;
    --include) INCLUDES+=("$2"); shift 2;;
    --exclude) EXCLUDES+=("$2"); shift 2;;
    --external) EXTERNAL=true; shift;;
    --subdomains) SUBDOMAINS=true; shift;;
    --source) SOURCE="$2"; shift 2;;
    --max-age) MAX_AGE="$2"; shift 2;;
    --json-prompt) JSON_PROMPT="$2"; shift 2;;
    --json-schema) JSON_SCHEMA="$2"; shift 2;;
    --poll-interval) POLL_INTERVAL="$2"; shift 2;;
    --timeout) TIMEOUT="$2"; shift 2;;
    --output) OUTPUT="$2"; shift 2;;
    --raw) RAW=true; shift;;
    --start-only) START_ONLY=true; shift;;
    -*) echo "Unknown option: $1" >&2; exit 1;;
    *) URL="$1"; shift;;
  esac
done

[[ -z "$URL" ]] && { echo "Usage: crawl.sh <url> [options]" >&2; exit 1; }

# Build JSON body
BODY=$(jq -n \
  --arg url "$URL" \
  --argjson limit "$LIMIT" \
  --argjson depth "$DEPTH" \
  --argjson render "$RENDER" \
  --arg source "$SOURCE" \
  --argjson maxAge "$MAX_AGE" \
  --argjson external "$EXTERNAL" \
  --argjson subdomains "$SUBDOMAINS" \
  '{
    url: $url,
    limit: $limit,
    depth: $depth,
    render: $render,
    source: $source,
    maxAge: $maxAge,
    formats: [],
    options: {
      includeExternalLinks: $external,
      includeSubdomains: $subdomains
    }
  }')

# Set format
BODY=$(echo "$BODY" | jq --arg fmt "$FORMAT" '.formats = [$fmt]')

# Add include/exclude patterns
for p in "${INCLUDES[@]+"${INCLUDES[@]}"}"; do
  BODY=$(echo "$BODY" | jq --arg p "$p" '.options.includePatterns += [$p]')
done
for p in "${EXCLUDES[@]+"${EXCLUDES[@]}"}"; do
  BODY=$(echo "$BODY" | jq --arg p "$p" '.options.excludePatterns += [$p]')
done

# Add JSON options if format is json
if [[ "$FORMAT" == "json" ]]; then
  if [[ -n "$JSON_PROMPT" ]]; then
    JSON_OPTS=$(jq -n --arg prompt "$JSON_PROMPT" '{prompt: $prompt}')
    if [[ -n "$JSON_SCHEMA" && -f "$JSON_SCHEMA" ]]; then
      JSON_OPTS=$(echo "$JSON_OPTS" | jq --slurpfile schema "$JSON_SCHEMA" '.response_format = {type: "json_schema", json_schema: $schema[0]}')
    fi
    BODY=$(echo "$BODY" | jq --argjson jo "$JSON_OPTS" '.jsonOptions = $jo')
  fi
fi

# Start crawl
RESPONSE=$(curl -s -X POST "$BASE" \
  -H "Authorization: Bearer $CF_CRAWL_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$BODY")

SUCCESS=$(echo "$RESPONSE" | jq -r '.success')
if [[ "$SUCCESS" != "true" ]]; then
  echo "Error starting crawl:" >&2
  echo "$RESPONSE" | jq . >&2
  exit 1
fi

JOB_ID=$(echo "$RESPONSE" | jq -r '.result')
echo "Crawl started: $JOB_ID" >&2

if [[ "$START_ONLY" == "true" ]]; then
  echo "$JOB_ID"
  exit 0
fi

# Poll for results
ELAPSED=0
while [[ $ELAPSED -lt $TIMEOUT ]]; do
  sleep "$POLL_INTERVAL"
  ELAPSED=$((ELAPSED + POLL_INTERVAL))

  RESULT=$(curl -s "$BASE/$JOB_ID" \
    -H "Authorization: Bearer $CF_CRAWL_API_TOKEN")

  STATUS=$(echo "$RESULT" | jq -r '.result.status // "unknown"')
  FINISHED=$(echo "$RESULT" | jq -r '.result.finished // 0')
  TOTAL=$(echo "$RESULT" | jq -r '.result.total // 0')

  echo "[$ELAPSED/${TIMEOUT}s] Status: $STATUS ($FINISHED/$TOTAL pages)" >&2

  if [[ "$STATUS" != "running" ]]; then
    break
  fi
done

if [[ "$STATUS" == "running" ]]; then
  echo "Timeout after ${TIMEOUT}s. Job still running: $JOB_ID" >&2
  echo "Poll later: curl -s '$BASE/$JOB_ID' -H 'Authorization: Bearer \$CF_CRAWL_API_TOKEN'" >&2
  exit 2
fi

# Fetch all results (handle pagination)
ALL_RECORDS="[]"
CURSOR=""
while true; do
  QUERY="?status=completed"
  [[ -n "$CURSOR" ]] && QUERY="${QUERY}&cursor=${CURSOR}"

  PAGE=$(curl -s "$BASE/$JOB_ID$QUERY" \
    -H "Authorization: Bearer $CF_CRAWL_API_TOKEN")

  RECORDS=$(echo "$PAGE" | jq '.result.records // []')
  ALL_RECORDS=$(echo "$ALL_RECORDS" "$RECORDS" | jq -s '.[0] + .[1]')

  NEXT_CURSOR=$(echo "$PAGE" | jq -r '.result.cursor // empty')
  if [[ -z "$NEXT_CURSOR" || "$NEXT_CURSOR" == "null" ]]; then
    break
  fi
  CURSOR="$NEXT_CURSOR"
done

# Build final output
FINAL=$(echo "$RESULT" | jq --argjson records "$ALL_RECORDS" '.result.records = $records')

if [[ "$RAW" == "true" ]]; then
  OUT="$FINAL"
else
  # Formatted summary
  OUT=$(echo "$FINAL" | jq '{
    job_id: .result.id,
    status: .result.status,
    browser_seconds: .result.browserSecondsUsed,
    total_pages: .result.total,
    completed_pages: .result.finished,
    pages: [.result.records[] | {
      url: .url,
      title: .metadata.title,
      status: .status,
      content_length: ((.markdown // .html // "") | length),
      json: .json
    }]
  }')
fi

if [[ -n "$OUTPUT" ]]; then
  echo "$FINAL" > "$OUTPUT"
  echo "Results written to $OUTPUT" >&2
  echo "$OUT"
else
  echo "$OUT"
fi


path: scripts/poll.sh
content: #!/usr/bin/env bash
# Poll/retrieve results for an existing crawl job
# Usage: poll.sh <job-id> [--raw] [--status completed|queued|errored]
set -euo pipefail

if [[ -f ~/.clawdbot/secrets/cloudflare-crawl.env ]]; then
  source ~/.clawdbot/secrets/cloudflare-crawl.env
fi
: "${CF_ACCOUNT_ID:?Set CF_ACCOUNT_ID}"
: "${CF_CRAWL_API_TOKEN:?Set CF_CRAWL_API_TOKEN}"

BASE="https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/browser-rendering/crawl"
JOB_ID="${1:?Usage: poll.sh <job-id>}"
shift
RAW=false
STATUS_FILTER=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --raw) RAW=true; shift;;
    --status) STATUS_FILTER="$2"; shift 2;;
    *) shift;;
  esac
done

QUERY=""
[[ -n "$STATUS_FILTER" ]] && QUERY="?status=$STATUS_FILTER"

RESULT=$(curl -s "$BASE/$JOB_ID$QUERY" \
  -H "Authorization: Bearer $CF_CRAWL_API_TOKEN")

if [[ "$RAW" == "true" ]]; then
  echo "$RESULT" | jq .
else
  echo "$RESULT" | jq '{
    job_id: .result.id,
    status: .result.status,
    browser_seconds: .result.browserSecondsUsed,
    total: .result.total,
    finished: .result.finished,
    pages: [.result.records[]? | {
      url: .url,
      title: .metadata.title,
      status: .status,
      content_length: ((.markdown // .html // "") | length)
    }]
  }'
fi


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)