Skill: meirk-brd/brightdata-search (version 1.0.0)

--- name: search description: Search the web via the Bright Data CLI — `bdata search` for Google/Bing/Yandex SERP, `bdata discover` for intent-ranked semantic results. Use when the user wants SERP results, needs URLs to feed into scraping, or wants semantic web discovery with optional page content. Hands off to `scrape` once target URLs are chosen, and to `data-feeds` when the user wants structured data from a known platform. Requires the Bright Data CLI; proactively guides install + login if missing. --- # Bright Data — Search Find things on the web. Two commands live in this skill: - **`bdata search`** — classic keyword SERP (Google/Bing/Yandex). Best when you want "what ranks for keyword X." - **`bdata discover`** — AI intent-ranked discovery with optional page content. Best when you want "pages about topic Y that match intent Z." For structured data from a known platform (Amazon, LinkedIn, TikTok, …), **stop and use `data-feeds` instead**. ## Setup gate (run first) ```bash if ! command -v bdata >/dev/null 2>&1; then echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md" elif ! bdata zones >/dev/null 2>&1; then echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)" fi ``` Halt and route to `skills/bright-data-best-practices/references/cli-setup.md` if either check fails. ## Pick your path | Situation | Action | |---|---| | Single keyword query, just SERP | `bdata search "<query>" --engine google --json --pretty` | | Paginated SERP (more results) | loop `--page 0`, `--page 1`, … (0-indexed) | | Multiple queries | shell loop over a queries file | | Intent-ranked / semantic (not keyword) | `bdata discover "<query>" --intent "<intent>" --num-results 20` | | Want page bodies along with results, one pass | `bdata discover ... --include-content` | | News / images / shopping SERP | `bdata search "<query>" --type news` (or `images`, `shopping`) | | Want Amazon/LinkedIn/TikTok/… structured data | **stop — hand off to `data-feeds`** | | Have URLs, want content | **hand off to `scrape`** | ## Action Core commands: ```bash # Google SERP, structured JSON bdata search "site:example.com privacy policy" --engine google --json --pretty # Localized Bing (German results, German language) bdata search "datenschutz" --engine bing --country de --language de --json # Second page of results (0-indexed) bdata search "machine learning papers" --page 1 --json # Mobile SERP (rankings differ from desktop) bdata search "best coffee shops" --device mobile --json # News vertical bdata search "openai" --type news --json --pretty # Intent-ranked discovery bdata discover "enterprise LLM platforms" \ --intent "vendor pages with pricing" \ --num-results 15 --json # Discovery with page content in markdown bdata discover "webhook best practices" \ --include-content --num-results 10 -o results.json # Date-filtered discovery bdata discover "react server components" \ --start-date 2025-01-01 --end-date 2025-12-31 --num-results 20 ``` Full flag reference: [`references/flags.md`](references/flags.md). ### `search` vs `discover` — pick the right one | You want | Use | |---|---| | "What Google ranks for this exact keyword" | `search` | | "Pages that match this meaning/intent" | `discover` | | "News / images / shopping vertical SERP" | `search --type <vertical>` | | "Results + page bodies in one call" | `discover --include-content` | | "Dedup / semantic ranking across queries" | `discover` | ## Verification gate 1. **JSON parses cleanly:** `jq . <output>` returns 0. 2. **Result array non-empty** — if empty, the query is legitimately zero-result; relax the query and re-run. Don't claim success on empty results without telling the user. 3. **Required fields present:** - `search`: results live at `.organic[]`; each has `title` + `link` - `discover`: results live at `.results[]`; each has `title` + `link`; if `--include-content`, also `content` 4. **For `discover --include-content`:** no block-page signatures in the `content` field (same list as scrape, case-insensitive): - `Access Denied` - `Just a moment` - `Attention Required` - `Checking your browser` - `captcha` - `cf-browser-verification` - `cloudflare` *(with < 2KB total body)* 5. **Geo sanity:** if the user expected country-specific results, inspect TLDs / languages of top results. If mis-localized, re-run with explicit `--country` and `--language`. ## Red flags - Using `search` to *fetch content* from Amazon, LinkedIn, TikTok, etc. when `data-feeds` returns clean structured data in one call. - Scraping every SERP result blindly — filter first (domain allowlist, keyword in title, relevance heuristic). - Confusing `search` (keyword) with `discover` (semantic). They answer different questions. - Running multiple queries without deduping URLs across result sets before scraping. - Assuming SERP order is universal — it's personalized by geo + device. Always set `--country` and `--device` explicitly for reproducibility. - Using `--page` as a result count — it's a page index, not a limit. Each page returns ~10 results. - Assuming SERP results are at `.results[]` — for `bdata search` they live at `.organic[]`. (Discover uses `.results[]`.) - Hardcoding `--num-results 100` on `discover` without realizing the pipeline polls until that many are found; can be slow. ## References - [`references/flags.md`](references/flags.md) — full flags for `search` and `discover` with when-to-use notes. - [`references/patterns.md`](references/patterns.md) — multi-query dedup, SERP → filter → scrape pipeline, `search` vs `discover` decision, legacy `curl` fallback, shared verification checklist. - [`references/examples.md`](references/examples.md) — (1) single Google query, (2) localized Bing, (3) batch queries + dedup into URL list, (4) `discover --include-content` end-to-end.

path: references/examples.md
content: # Search — worked examples

## Example 1 — single Google query

```bash
bdata search "postgres jsonb index performance" \
    --engine google --country us --json --pretty -o serp.json

# Verify
jq '.organic | length' serp.json        # > 0
jq -r '.organic[0] | "\(.title)\n\(.link)"' serp.json
```

## Example 2 — localized Bing (German)

```bash
bdata search "datenschutz-grundverordnung leitfaden" \
    --engine bing --country de --language de --json -o serp-de.json

# Sanity-check locale: top results should have .de TLDs or German text
jq -r '.organic[0:5][].link' serp-de.json
```

## Example 3 — batch queries + dedup into URL list

Given `queries.txt` with multiple related queries:

```bash
mkdir -p out
: > out/all-urls.txt

while IFS= read -r q; do
    hash=$(printf '%s' "$q" | md5sum | cut -c1-8)
    bdata search "$q" --engine google --country us --json \
        -o "out/serp-${hash}.json"
    jq -r '.organic[].link' "out/serp-${hash}.json" >> out/all-urls.txt
done < queries.txt

sort -u out/all-urls.txt > out/unique-urls.txt
echo "Total: $(wc -l < out/all-urls.txt)  Unique: $(wc -l < out/unique-urls.txt)"
```

## Example 4 — `discover --include-content` end-to-end

Find recent articles on a topic, with body content in one call, ready to feed into an LLM:

```bash
bdata discover "post-quantum cryptography deployment" \
    --intent "practical deployment case studies from 2025" \
    --num-results 15 \
    --start-date 2025-01-01 \
    --include-content \
    --country us --language en \
    --timeout 900 \
    --json --pretty -o pqc.json

# Verify count
jq '.results | length' pqc.json              # should be ~15
jq -r '.results[0] | .title, .link' pqc.json
jq -r '.results[0].content' pqc.json | head  # body as markdown

# Block-page sanity (no content should match)
jq -r '.results[].content // empty' pqc.json \
    | grep -iE 'access denied|just a moment|captcha|cloudflare' \
    && echo "WARN: one or more results returned a block page"
```


path: references/flags.md
content: # `bdata search` / `bdata discover` — flag reference

Verified against `@brightdata/cli` v0.1.8 on 2026-04-19.

## `bdata search` — classic keyword SERP

Usage: `bdata search [options] <query>`

| Flag | Values | Default | When to use |
|---|---|---|---|
| `--engine <name>` | `google`, `bing`, `yandex` | `google` | Pick the SERP source. `google` for most; `yandex` for RU-centric queries; `bing` as a cross-check. |
| `--country <code>` | ISO (`us`, `de`, `jp`, …) | — | Localized SERP. Required for reproducibility across runs. |
| `--language <code>` | ISO (`en`, `fr`, …) | — | Result language. Often paired with `--country`. |
| `--page <n>` | integer, **0-indexed** | `0` | Result page. Loop `0, 1, 2, …` for more results. ~10 results/page. |
| `--type <type>` | `web`, `news`, `images`, `shopping` | `web` | Search vertical. |
| `--zone <name>` | SERP zone name | account default | Override zone. Rarely needed. |
| `--device <type>` | `desktop`, `mobile` | `desktop` | Mobile rankings differ. Pick explicitly for reproducibility. |
| `-o, --output <path>` | file path | stdout | Write to file. |
| `--json` | (flag) | off | Force JSON envelope. |
| `--pretty` | (flag) | off | Pretty-print JSON. |
| `--timing` | (flag) | off | Print request timing breakdown to stderr. Debugging only. |
| `-k, --api-key <key>` | API key | saved / env | Per-command override. |

### SERP JSON output shape

When invoked with `--json`, `bdata search` returns an envelope with these top-level keys (from the `Search_response` type):

- `general` — query metadata (search engine, query, results count, language, device)
- `organic` — main result array. Each entry has `rank`, `title`, `link`, `description` (some also have `global_rank`)
- Vertical result arrays (present depending on the query and `--type`): `news`, `images`, `shopping`, `videos`, `recipes`, `maps`, `paid`, `product_listing_ads`
- Semantic blocks: `knowledge_graph`, `people_also_ask`, `related_searches`, `perspectives`

Each vertical has its own record shape:
- `news` → `title`, `link`, `source`, `date`, `image`
- `images` → `title`, `link`, `source`, `original_image`, `image`
- `shopping` → `title`, `link`, `price`, `shop`, `rating`, `reviews_cnt`

To extract just the links from the main organic results: `jq -r '.organic[].link'`. For news: `jq -r '.news[].link'`.

## `bdata discover` — AI intent-ranked discovery

Usage: `bdata discover [options] <query>`

| Flag | Values | Default | When to use |
|---|---|---|---|
| `--intent <text>` | free text | — | Semantic intent used to re-rank results. E.g., `"product pricing pages"`, `"academic papers"`. |
| `--country <code>` | ISO | `US` | Localization. |
| `--city <name>` | city string | — | City-level localization (e.g., `"New York"`). |
| `--language <code>` | ISO | `en` | Language. |
| `--num-results <n>` | integer | — | Target result count. Discovery polls until reached or timeout. |
| `--filter-keywords <csv>` | comma-separated | — | Require these keywords in result pages. |
| `--include-content` | (flag) | off | Fetch and include page body (markdown) for each result. Big payload; slower. |
| `--no-remove-duplicates` | (flag) | off | Keep dup URLs. Default dedups. |
| `--start-date <YYYY-MM-DD>` | ISO date | — | Only content updated from this date. |
| `--end-date <YYYY-MM-DD>` | ISO date | — | Only content updated through this date. |
| `--timeout <sec>` | integer | `600` | Max seconds to wait for the target `--num-results`. |
| `-o, --output <path>` | file path | stdout | Write to file. Required for larger result sets. |
| `--timing` | (flag) | off | Print request timing breakdown to stderr. Debugging only. |
| `--json` / `--pretty` | flags | — | JSON formatting. |

### Discover JSON output shape

- `results` — array of result objects, each with `title`, `link`, `description`, `relevance_score` (and `content` when `--include-content` is set)
- `status`, `timestamp`, `duration_seconds` — job metadata

To extract links: `jq -r '.results[].link'`. To pull `content` bodies: `jq -r '.results[].content'`.

## When to use `search` vs `discover`

| You want | Use |
|---|---|
| "What Google ranks right now for keyword X" | `search` |
| "Pages that match this meaning/intent, ranked by relevance" | `discover` |
| Vertical SERP (news/images/shopping) | `search --type` |
| Dedup + semantic ranking across many queries | `discover` |
| Time-bounded content discovery | `discover --start-date --end-date` |
| Get result list AND page bodies in one call | `discover --include-content` |


path: references/patterns.md
content: # Search — patterns

## Verification checklist (shared across all Bright Data CLI skills)

Before claiming a search succeeded:

1. **JSON parses cleanly** (`jq . <output>` returns 0).
2. **Result array non-empty** — if empty, re-check; don't claim success on zero results silently.
3. **Expected fields present:** `title` + `link` at `.organic[]` (`search`) or `.results[]` (`discover`). If `discover --include-content`, results also have `content`.
4. **No block-page signatures** in `discover --include-content` bodies:
   - `Access Denied`
   - `Just a moment`
   - `Attention Required`
   - `Checking your browser`
   - `captcha`
   - `cf-browser-verification`
   - `cloudflare` *(with < 2KB total body)*
5. **Geo sanity:** result TLDs / languages match the requested `--country` / `--language`.

## Multi-query batch with dedup

Given `queries.txt` (one query per line), collect deduped result URLs:

```bash
mkdir -p out
: > out/all-urls.txt

while IFS= read -r q; do
    hash=$(printf '%s' "$q" | md5sum | cut -c1-8)
    bdata search "$q" --engine google --country us --json \
        -o "out/serp-${hash}.json"
    jq -r '.organic[].link' "out/serp-${hash}.json" >> out/all-urls.txt
done < queries.txt

sort -u out/all-urls.txt > out/urls.txt
wc -l out/urls.txt
```

`bdata search`'s JSON envelope holds the main results at `.organic[]`. Confirm once with `jq 'keys'` on any SERP output if the shape ever surprises you.

## SERP → filter → scrape pipeline

```bash
# 1. Search
bdata search "enterprise monitoring tools" --engine google --country us \
    --json -o serp.json

# 2. Filter (domain allowlist; relevance heuristic)
jq -r '.organic[]
    | select(.link | test("^https://(?!.*(reddit|pinterest))"))
    | select(.title | test("monitoring"; "i"))
    | .link' serp.json > urls.txt

# 3. Scrape (hands off to the scrape skill's patterns)
mkdir -p out
xargs -a urls.txt -n 1 -P 4 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata scrape "$url" -f markdown -o "out/${hash}.md" || echo "FAIL: $url" >&2
' _ {}
```

## `search` vs `discover` decision

Default to `search` for keyword-exactness tasks (SEO research, "what ranks for X"). Default to `discover` when the user's description is a topic, intent, or concept rather than a keyword ("pages about how companies adopt LLMs", "recent articles on post-quantum crypto").

Rule of thumb: if the user's phrasing is a complete sentence or describes intent, `discover`. If it's a short keyword string, `search`.

## Pagination with `search`

`--page` is 0-indexed. Loop pages until empty or duplicate:

```bash
prev_hash=""
for page in 0 1 2 3 4; do
    bdata search "long tail query" --page "$page" --json -o "p${page}.json"
    hash=$(sha1sum "p${page}.json" | awk '{print $1}')
    # Same hash twice → we're looping; break
    [[ "$hash" == "$prev_hash" ]] && { rm "p${page}.json"; break; }
    # Empty results → break
    count=$(jq '.organic | length' "p${page}.json")
    [[ "$count" == "0" ]] && { rm "p${page}.json"; break; }
    prev_hash=$hash
done
```

## Discover: extracting links and content

```bash
# Just links (discover's array is at .results[])
bdata discover "enterprise LLM vendors" --num-results 20 --json -o disc.json
jq -r '.results[].link' disc.json > vendor-urls.txt

# Links + markdown bodies in one call
bdata discover "incident postmortems" \
    --intent "public post-mortem write-ups from 2025" \
    --num-results 15 --include-content \
    --json -o postmortems.json

# Pull just the bodies as separate markdown files
jq -r '.results[] | .link + "\n" + (.content // "")' postmortems.json
# Or write each to a file:
jq -c '.results[]' postmortems.json | while IFS= read -r row; do
    link=$(jq -r '.link' <<<"$row")
    hash=$(printf '%s' "$link" | md5sum | cut -c1-8)
    jq -r '.content // ""' <<<"$row" > "out/${hash}.md"
done
```

## Legacy `curl` fallback (deprecated)

Only when CLI cannot be installed. SERP API endpoint via Web Unlocker. The CLI's `bdata search` prefers `BRIGHTDATA_SERP_ZONE` and falls back to `BRIGHTDATA_UNLOCKER_ZONE`; pick whichever is set in your environment:

```bash
curl -sS "https://api.brightdata.com/request" \
    -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
        \"zone\": \"${BRIGHTDATA_SERP_ZONE:-$BRIGHTDATA_UNLOCKER_ZONE}\",
        \"url\": \"https://www.google.com/search?q=$(printf '%s' "$QUERY" | jq -sRr @uri)&brd_json=1\",
        \"format\": \"raw\"
    }"
```

Prefer the CLI path. This block exists only for environments without Node.js.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)