Skill: meirk-brd/brightdata-scrape (version 1.0.0)

--- name: scrape description: Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing. --- # Bright Data — Scrape Get clean content (markdown, HTML, JSON, screenshot) from one or more URLs via the Bright Data CLI. This skill owns the "fetch raw or lightly-structured content" job. For platform-specific structured data (Amazon, LinkedIn, TikTok, etc.), **stop and use `data-feeds` instead** — you'll get clean JSON without selector logic. ## Setup gate (run first) Before any scrape, verify the CLI is installed and authenticated: ```bash if ! command -v bdata >/dev/null 2>&1; then echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md" elif ! bdata zones >/dev/null 2>&1; then echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)" fi ``` If either check fails, halt and route the user to `skills/bright-data-best-practices/references/cli-setup.md`. Do not attempt the legacy `curl` fallback silently — ask the user first. ## Pick your path | Situation | Action | |---|---| | Single URL | `bdata scrape <url> -f markdown` | | Small list (≤ ~20 URLs) | shell loop, 1 at a time (see `references/patterns.md`) | | Larger list (dozens+) | `xargs -P 4` with parallelism cap (see `references/patterns.md`) | | Paginated listing | scrape page 1 → extract next-page URL → append → repeat (see `references/examples.md`) | | JS-heavy / login-gated / interaction-required | escalate to `bdata browser` (see `brightdata-cli` skill) | | Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, … | **stop — hand off to `data-feeds`** | | No URL yet, just a topic | **hand off to `search`** | ## Action Core commands: ```bash # Clean markdown (default) bdata scrape "https://example.com/article" -f markdown -o article.md # Raw HTML (when you need the DOM) bdata scrape "https://example.com" -f html -o page.html # Structured JSON (when the Unlocker returns parsed fields) bdata scrape "https://example.com" -f json --pretty -o page.json # Visual snapshot (saves PNG) bdata scrape "https://example.com" -f screenshot -o page.png # Geo-targeted (override the exit country) bdata scrape "https://example.com" --country de -f markdown ``` Full flag reference: [`references/flags.md`](references/flags.md). ## Verification gate (run before claiming success) 1. **Non-empty output:** `test -s "$out_path"` — or, for stdout, at least 200 bytes of content. 2. **Not a block page** — grep the output for any of these signatures (case-insensitive): - `Access Denied` - `Just a moment` - `Attention Required` - `Checking your browser` - `captcha` - `cf-browser-verification` - `cloudflare` *(with < 2KB total body)* 3. **Expected markers present** for the task: e.g., a product page should contain a price pattern (`\$\d`); an article should contain at least one `<h1>` or `# ` heading. 4. **On failure, escalation ladder:** - Retry with a different `--country` (e.g., `--country de` if the origin site is US) - Escalate to `bdata browser` for full JS rendering (hand off to `brightdata-cli` skill) Do not report success until all checks above pass. ## Red flags - Claiming success without inspecting the output. - Silencing errors with `2>/dev/null` — you'll miss auth failures and rate-limit errors. - Running `bdata scrape` on Amazon/LinkedIn/TikTok/Instagram/YouTube/Reddit URLs — these are supported by `data-feeds` and return structured data directly. Scraping loses the structure. - Scraping the same URL repeatedly in the same task — cache the first result. - Looping `bdata scrape` sequentially for large lists instead of using `xargs -P 4` (or similar) with a parallelism cap. - Using `curl` against `api.brightdata.com` directly — legacy path; only when the CLI isn't available. ## References - [`references/flags.md`](references/flags.md) — every flag with when-to-use notes. - [`references/patterns.md`](references/patterns.md) — shell-loop batching, `xargs` parallelism, pagination recipe, retry/backoff, block-page recovery chain, legacy `curl` fallback. - [`references/examples.md`](references/examples.md) — (1) single page → markdown, (2) batch a list of URLs with parallelism cap, (3) paginated listing, (4) block-page recovery.

path: references/examples.md
content: # Scrape — worked examples

## Example 1 — single page to markdown

Fetch a blog post as clean markdown and verify the output.

```bash
bdata scrape "https://engineering.example.com/posts/scaling-bigquery" \
    -f markdown -o post.md

# Verify
test -s post.md || { echo "empty output"; exit 1; }
grep -qiE 'access denied|just a moment|captcha|cloudflare' post.md \
    && { echo "block page"; exit 1; }
head -n 1 post.md   # should show the title as an h1
```

## Example 2 — batch a list of URLs with parallelism cap

Given `urls.txt` (one URL per line), scrape all to `out/` with max 4 parallel requests:

```bash
mkdir -p out
xargs -a urls.txt -n 1 -P 4 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata scrape "$url" -f markdown -o "out/${hash}.md" \
        || echo "FAIL: $url" >&2
' _ {}

# Count successes / failures
ok=$(find out -name "*.md" -size +0 | wc -l)
total=$(wc -l < urls.txt)
echo "Scraped $ok of $total URLs"
```

## Example 3 — paginated listing

Scrape every page of a paginated article index until an empty page is hit:

```bash
mkdir -p out
page=1
while :; do
    url="https://blog.example.com/archive?page=$page"
    out="out/archive-${page}.md"
    bdata scrape "$url" -f markdown -o "$out"

    if [[ ! -s "$out" ]] || ! grep -qE '\[.+\]\(/posts/' "$out"; then
        rm -f "$out"
        echo "Done after $((page - 1)) pages"
        break
    fi
    page=$((page + 1))
done
```

## Example 4 — block-page recovery chain

Scrape a URL that's intermittently Cloudflare-gated. Try a set of exit countries; if all return block pages, hand off to `bdata browser`:

```bash
URL="https://protected.example.com/catalog"
OUT="catalog.md"

for args in "" "--country de" "--country jp" "--country gb"; do
    bdata scrape "$URL" $args -f markdown -o "$OUT" || continue
    [[ -s "$OUT" ]] || continue
    if ! grep -qiE 'access denied|just a moment|captcha|cloudflare' "$OUT"; then
        echo "Succeeded with: $args"
        exit 0
    fi
done

echo "All country rotations returned block pages — escalating to bdata browser" >&2
# (caller hands off to the bdata browser command, documented in the brightdata-cli skill)
exit 1
```


path: references/flags.md
content: # `bdata scrape` — flag reference

Verified against `@brightdata/cli` v0.1.8 on 2026-04-19.

Usage: `bdata scrape [options] <url>`

| Flag | Values | Default | When to use |
|---|---|---|---|
| `-f, --format <format>` | `markdown`, `html`, `screenshot`, `json` | `markdown` | `markdown` for readable content; `html` when you need DOM fidelity; `screenshot` to save a PNG; `json` when the Unlocker has a structured extractor for the URL. |
| `--country <code>` | ISO code (`us`, `de`, `jp`, …) | — | Force a geo-targeted exit. Use when the target site geoblocks, personalizes by country, or returns different content by region. |
| `--zone <name>` | Unlocker zone name | account default | Override the default zone — e.g., when you have a dedicated zone with different residential/mobile settings. |
| `--mobile` | (flag) | off | **Not functional in v0.1.8** — flag is declared but `build_request` in the CLI does not forward it. Do not rely on it. |
| `--async` | (flag) | off | **Not functional in v0.1.8** — the Web Unlocker API currently rejects the `async` field with a validation error. Do not use until the CLI ships a fix. |
| `-o, --output <path>` | file path | stdout | Write result to a file. Required for binary formats (`screenshot`). Recommended for anything > 1KB. |
| `--json` | (flag) | off | Force JSON output envelope (metadata + content). Useful in scripts. |
| `--pretty` | (flag) | off | Pretty-print JSON. Combine with `--json` or `-f json`. |
| `--timing` | (flag) | off | Print request timing breakdown to stderr. Debugging only. |
| `-k, --api-key <key>` | API key | saved credentials or `BRIGHTDATA_API_KEY` | Per-command override. Rarely needed — prefer `bdata login`. |

## Format decision matrix

| Goal | Format |
|---|---|
| Feed content to an LLM | `markdown` |
| Extract via selectors / regex | `html` |
| Visual regression / proof-of-view | `screenshot` (writes PNG — use `-o` required) |
| URL has a structured extractor (Unlocker auto-parses) | `json` |


path: references/patterns.md
content: # Scrape — patterns

## Verification checklist (shared across all Bright Data CLI skills)

Before claiming a scrape succeeded:

1. **Output is non-empty.** `test -s "$out"` for files; for stdout, min 200 bytes.
2. **Not a block page.** Grep (case-insensitive) for any of:
   - `Access Denied`
   - `Just a moment`
   - `Attention Required`
   - `Checking your browser`
   - `captcha`
   - `cf-browser-verification`
   - `cloudflare` *(with < 2KB total body)*
3. **Expected markers present** for the task (e.g., price pattern on a product page, at least one heading in an article).

Failing any check → retry with `--country`, then escalate to `bdata browser`.

## Small-batch shell loop (≤ ~20 URLs)

```bash
mkdir -p out
while IFS= read -r url; do
    hash=$(printf '%s' "$url" | md5sum | cut -c1-8)
    bdata scrape "$url" -f markdown -o "out/${hash}.md" \
        || echo "FAIL: $url" >&2
done < urls.txt
```

The `|| echo` prevents one failure from aborting the loop; failures are visible on stderr.

## Large batch with parallelism cap (`xargs -P`)

```bash
mkdir -p out
xargs -a urls.txt -n 1 -P 4 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata scrape "$url" -f markdown -o "out/${hash}.md" || echo "FAIL: $url" >&2
' _ {}
```

`-P 4` caps concurrency at 4 parallel `bdata scrape` invocations. Raise cautiously — each scrape consumes bandwidth and counts against zone budget.

## Pagination recipe (listing pages)

For a listing with `?page=N` pagination:

```bash
page=1
while :; do
    url="https://example.com/list?page=$page"
    out="out/page-${page}.md"
    bdata scrape "$url" -f markdown -o "$out"

    # Stop when the page is empty or doesn't contain an item marker
    if [[ ! -s "$out" ]] || ! grep -q '\[.*\](/item/' "$out"; then
        rm -f "$out"
        break
    fi
    page=$((page + 1))
done
```

Adapt the "contains an item marker" grep to the actual site's output.

## Retry / backoff

`bdata scrape` returns non-zero on failure. Wrap with a simple retry:

```bash
scrape_with_retry() {
    local url=$1 out=$2 attempt=1 max=3
    while (( attempt <= max )); do
        if bdata scrape "$url" -f markdown -o "$out"; then
            return 0
        fi
        sleep $((2 ** attempt))   # 2s, 4s, 8s
        attempt=$((attempt + 1))
    done
    return 1
}
```

## Block-page recovery chain

When a scrape returns a block-page signature:

1. Retry same URL with a different `--country` (e.g. `de` if origin is US).
2. If still blocked, escalate to `bdata browser` (real-browser with JS execution).

Example:

```bash
try_scrape() {
    local url=$1 out=$2
    for args in "" "--country de" "--country jp" "--country gb"; do
        bdata scrape "$url" $args -f markdown -o "$out" || continue
        [[ -s "$out" ]] || continue
        if ! grep -qiE 'access denied|just a moment|captcha|cloudflare' "$out"; then
            return 0
        fi
    done
    return 1
}
```

If all country rotations return block pages, hand off to the `bdata browser` command.

## Legacy `curl` fallback (deprecated)

Only when the CLI cannot be installed. Requires env vars `BRIGHTDATA_API_KEY` and `BRIGHTDATA_UNLOCKER_ZONE`:

```bash
curl -sS "https://api.brightdata.com/request" \
    -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
        \"zone\": \"$BRIGHTDATA_UNLOCKER_ZONE\",
        \"url\": \"$URL\",
        \"format\": \"raw\",
        \"data_format\": \"markdown\"
    }"
```

Prefer the CLI path. This block exists only for environments without Node.js.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)