Skill: meirk-brd/brightdata-data-feeds (version 1.0.0)

--- name: data-feeds description: Extract structured data from 40+ supported platforms (Amazon, LinkedIn, Instagram, TikTok, Facebook, YouTube, Reddit, and more) via the Bright Data CLI (`bdata pipelines`). Use when the user wants clean JSON from a known platform URL rather than raw HTML. Hands off to `scrape` for unsupported URLs and to `search` when target URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing. --- # Bright Data — Data Feeds (Pipelines) Extract structured data from supported platforms via `bdata pipelines`. One call, clean JSON, no scraping logic. For unsupported URLs, hand off to `scrape`. To find target URLs first, hand off to `search`. ## Setup gate (run first) ```bash if ! command -v bdata >/dev/null 2>&1; then echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md" elif ! bdata zones >/dev/null 2>&1; then echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)" fi ``` Halt and route to `skills/bright-data-best-practices/references/cli-setup.md` if either check fails. ## Supported pipeline types (verified 2026-04-19) **Always verify with `bdata pipelines list` before hardcoding names** — they change. Current 43 types: `amazon_product`, `amazon_product_reviews`, `amazon_product_search`, `apple_app_store`, `bestbuy_products`, `booking_hotel_listings`, `crunchbase_company`, `ebay_product`, `etsy_products`, `facebook_company_reviews`, `facebook_events`, `facebook_marketplace_listings`, `facebook_posts`, `github_repository_file`, `google_maps_reviews`, `google_play_store`, `google_shopping`, `homedepot_products`, `instagram_comments`, `instagram_posts`, `instagram_profiles`, `instagram_reels`, `linkedin_company_profile`, `linkedin_job_listings`, `linkedin_people_search`, `linkedin_person_profile`, `linkedin_posts`, `reddit_posts`, `reuter_news`, `tiktok_comments`, `tiktok_posts`, `tiktok_profiles`, `tiktok_shop`, `walmart_product`, `walmart_seller`, `x_posts`, `yahoo_finance_business`, `youtube_comments`, `youtube_profiles`, `youtube_videos`, `zara_products`, `zillow_properties_listing`, `zoominfo_company_profile` **Naming note:** inconsistent across platforms. `amazon_product` (singular), `tiktok_profiles` (plural), `linkedin_person_profile` (not `linkedin_profile`). Always copy from `bdata pipelines list`. ## Pick your path | Situation | Action | |---|---| | Know the platform + have URL(s) | `bdata pipelines <type> <url>` | | Don't know which pipeline fits | `bdata pipelines list` first | | Pipeline takes keyword or multi-arg input | See "Keyword- and multi-arg pipelines" below | | Multiple URLs on the same pipeline type | shell loop with parallelism cap (see `references/patterns.md`) | | Long job (reviews, company employees, big post feeds) | raise `--timeout 1800` | | URL is on an unsupported platform | **stop — hand off to `scrape`** | | Need to find URLs first | **hand off to `search`** | ## Keyword- and multi-arg pipelines (do NOT take a single URL) A few pipelines take non-URL or multi-positional inputs. Invoke with no args to see the exact usage line from the CLI: | Pipeline | Args | |---|---| | `amazon_product_search` | `<keyword> <domain_url>` — e.g., `"running shoes" https://www.amazon.com` | | `linkedin_people_search` | `<url> <first_name> <last_name>` — search a company/school/URL for a named person | | `facebook_company_reviews` | `<url> [num_reviews]` — optional num_reviews defaults to `10` | | `google_maps_reviews` | `<url> [days_limit]` — optional days_limit defaults to `3` | | `youtube_comments` | `<url> [num_comments]` — optional num_comments defaults to `10` | All other 37 pipelines take a single URL. ## Action Core commands: ```bash # List available pipeline types (source of truth) bdata pipelines list # Amazon product bdata pipelines amazon_product \ "https://www.amazon.com/dp/B08N5WRWNW" \ --format json --pretty -o product.json # Amazon product reviews (slower — reviews can be hundreds) bdata pipelines amazon_product_reviews \ "https://www.amazon.com/dp/B08N5WRWNW" \ --timeout 1200 -o reviews.json # Amazon product search (keyword + domain URL) bdata pipelines amazon_product_search \ "noise cancelling headphones" "https://www.amazon.com" \ --format json --pretty -o search.json # LinkedIn person profile bdata pipelines linkedin_person_profile \ "https://www.linkedin.com/in/example" -o person.json # LinkedIn company bdata pipelines linkedin_company_profile \ "https://www.linkedin.com/company/example" -o company.json # LinkedIn people search (url + first + last name) bdata pipelines linkedin_people_search \ "https://www.linkedin.com/company/example" "Jane" "Doe" \ -o people.json # Instagram posts bdata pipelines instagram_posts \ "https://www.instagram.com/example/" -o posts.json # Google Maps reviews (url + days_limit, default 3) bdata pipelines google_maps_reviews \ "https://maps.google.com/?cid=1234567890" 90 -o reviews.json # YouTube comments (url + num_comments, default 10) bdata pipelines youtube_comments \ "https://www.youtube.com/watch?v=abc123" 100 -o yt-comments.json # NDJSON for big feeds (one record per line) bdata pipelines linkedin_posts "https://www.linkedin.com/in/example" \ --format ndjson -o posts.ndjson # Raise polling timeout for long jobs bdata pipelines amazon_product_reviews "<url>" --timeout 1800 -o out.json ``` Full flag reference + full type table: [`references/flags.md`](references/flags.md). ## Verification gate 1. **JSON parses cleanly:** `jq . <output>` returns 0 (or for `--format ndjson`, each line parses). 2. **Record count matches expected.** One URL usually = one record, *but* reviews/posts/comments pipelines return arrays sized by what the platform shows. Always check: ```bash jq 'length' out.json # top-level array count # OR jq 'if type == "array" then length else 1 end' out.json ``` 3. **No top-level error:** ```bash jq -e 'if type == "object" then has("error") | not else true end' out.json \ || { echo "pipeline reported error"; exit 1; } ``` 4. **No per-record error:** for array results, ensure no record has an `error` field: ```bash jq -e 'if type == "array" then map(has("error")) | any | not else true end' out.json \ || echo "WARN: one or more records have error fields" ``` Partial failures are silent — this check is non-optional. 5. **Core fields present** for the pipeline type (examples): - `amazon_product` → `.title` + `.price` (or `.final_price`) - `linkedin_person_profile` → `.name` + `.headline` (or `.position`) - `instagram_posts` → `.caption` or `.description` + `.url` or `.post_id` - `youtube_videos` → `.title` + `.video_id` or `.url` Spot-check with `jq keys` on the first record to learn the exact schema. 6. **On failure:** double `--timeout` and retry once. If still failing, `bdata pipelines list` to confirm the type name hasn't changed. ## Red flags - Using `bdata scrape` on Amazon/LinkedIn/TikTok/etc. when `bdata pipelines <type>` returns structured fields in one call. Loses structure and costs more time. - Looping `bdata pipelines` for large jobs without rate-limiting — each call can trigger a long-running pipeline on the server. Cap parallelism at 2–3. - Claiming success without the record-count + per-record error check. Partial failures are silent in pipeline output. - Hardcoding pipeline type names (`amazon_products` with an `s`, `linkedin_profile` without `_person_`, etc.) — they're inconsistent across platforms. Always copy from `bdata pipelines list`. - Using a tight `--timeout` on pipelines that legitimately take 5–15 minutes (reviews, company employees, big post feeds). Default 600s is a floor for small inputs; raise for long ones. - Calling a keyword- or multi-arg pipeline (`amazon_product_search`, `linkedin_people_search`, `google_maps_reviews`, `facebook_company_reviews`, `youtube_comments`) with URL-only args — will fail with `"Usage: ..."`. Always check `bdata pipelines <type>` error output when in doubt. - Passing a `pages_to_search` third arg to `amazon_product_search` — it's hardcoded to `1` by the CLI and extra args are ignored. ## References - [`references/flags.md`](references/flags.md) — full `pipelines` flags + complete table of all 43 types with input shapes. - [`references/patterns.md`](references/patterns.md) — sync timeout tuning, shell-loop batching with parallelism cap, partial-failure detection, keyword-shaped pipeline cheatsheet, legacy `curl` fallback, shared verification checklist. - [`references/examples.md`](references/examples.md) — (1) single Amazon product, (2) batch LinkedIn companies, (3) long reviews job with raised timeout, (4) mixed-platform workflow calling `pipelines list` first, (5) keyword-shaped `amazon_product_search`.

path: references/examples.md
content: # Data-feeds — worked examples

## Example 1 — single Amazon product

```bash
bdata pipelines amazon_product \
    "https://www.amazon.com/dp/B08N5WRWNW" \
    --format json --pretty -o product.json

# Verify
jq . product.json >/dev/null || { echo "parse fail"; exit 1; }
jq -e 'has("error") | not' product.json >/dev/null \
    || { echo "pipeline error: $(jq -r .error product.json)"; exit 1; }
jq -r '.title, .price // .final_price' product.json
```

## Example 2 — batch LinkedIn companies via shell loop

Given `companies.txt` (one LinkedIn company URL per line):

```bash
mkdir -p out
xargs -a companies.txt -n 1 -P 2 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata pipelines linkedin_company_profile "$url" \
        --timeout 900 -o "out/${hash}.json" || echo "FAIL: $url" >&2
' _ {}

# Summarize results
total=$(wc -l < companies.txt)
ok=$(find out -name "*.json" -size +0 | wc -l)
echo "Fetched $ok of $total companies"

# Partial-failure sweep
for f in out/*.json; do
    jq -e 'has("error") | not' "$f" >/dev/null \
        || echo "ERROR in $f: $(jq -r .error "$f")"
done
```

## Example 3 — long reviews job with raised timeout

Amazon product reviews can take 10+ minutes for products with many reviews:

```bash
bdata pipelines amazon_product_reviews \
    "https://www.amazon.com/dp/B08N5WRWNW" \
    --timeout 1800 \
    --format json --pretty -o reviews.json

# Verify count
count=$(jq 'if type == "array" then length else 1 end' reviews.json)
echo "Got $count reviews"

# Check first record schema
jq '.[0] | keys' reviews.json
```

## Example 4 — mixed platform workflow (discover types first)

User says: "Give me all the data on this TikTok profile and their recent posts."

```bash
# 1. Find the right pipelines
bdata pipelines list | grep '^tiktok'
# → tiktok_comments, tiktok_posts, tiktok_profiles, tiktok_shop

# 2. Profile info
bdata pipelines tiktok_profiles \
    "https://www.tiktok.com/@example" -o profile.json

# 3. Recent posts
bdata pipelines tiktok_posts \
    "https://www.tiktok.com/@example" --timeout 1200 -o posts.json

# Verify both
for f in profile.json posts.json; do
    jq . "$f" >/dev/null || { echo "$f: parse fail"; exit 1; }
    jq -e 'if type == "object" then has("error") | not else true end' "$f" \
        >/dev/null || { echo "$f: pipeline error"; exit 1; }
done

echo "profile: $(jq 'keys' profile.json)"
echo "posts: $(jq 'if type == "array" then length else 1 end' posts.json) records"
```

## Example 5 — keyword-shaped `amazon_product_search`

Search Amazon by keyword on a given domain (no URL needed, just the domain root):

```bash
bdata pipelines amazon_product_search \
    "mechanical keyboard" "https://www.amazon.com" \
    --format json --pretty -o search.json

jq 'length' search.json                                     # number of results (output is a top-level array)
jq -r '.[0:5][] | "\(.title) — \(.price)"' search.json
```

Two positional args: `<keyword> <domain_url>`. The CLI hardcodes `pages_to_search` to `1` internally — pass only those two args. Calling with fewer args prints the expected usage.


path: references/flags.md
content: # `bdata pipelines` — flag reference

Verified against `@brightdata/cli` v0.1.8 on 2026-04-19.

Usage: `bdata pipelines [options] <type> [params...]`

## Flags

| Flag | Values | Default | When to use |
|---|---|---|---|
| `--format <fmt>` | `json`, `csv`, `ndjson`, `jsonl` | `json` | `json` for most; `ndjson` or `jsonl` for big result sets you'll stream-process line-by-line. `csv` for spreadsheet consumers. |
| `--timeout <seconds>` | integer | `600` (or `$BRIGHTDATA_POLLING_TIMEOUT`) | Max seconds to poll for completion. Raise to `1200`–`1800` for reviews, company employees, or long post feeds. |
| `-o, --output <path>` | file path | stdout | Write to file. Recommended for any result > 1KB. |
| `--json` | (flag) | off | Force JSON envelope. |
| `--pretty` | (flag) | off | Pretty-print JSON. |
| `--timing` | (flag) | off | Stderr timing breakdown. Debug only. |
| `-k, --api-key <key>` | API key | saved / env | Per-command override. |

## Pipeline types — input shape (verified 2026-04-19)

Always cross-check with `bdata pipelines list` before hardcoding names. All 43 types:

### URL-input pipelines (single `<url>`)

| Pipeline | Typical URL shape |
|---|---|
| `amazon_product` | `https://www.amazon.<tld>/dp/<ASIN>` |
| `amazon_product_reviews` | `https://www.amazon.<tld>/dp/<ASIN>` |
| `apple_app_store` | app page URL |
| `bestbuy_products` | product page URL |
| `booking_hotel_listings` | listing URL |
| `crunchbase_company` | company page URL |
| `ebay_product` | product URL |
| `etsy_products` | product URL |
| `facebook_events` | event URL |
| `facebook_marketplace_listings` | listing URL |
| `facebook_posts` | page or post URL |
| `github_repository_file` | file URL |
| `google_play_store` | app URL |
| `google_shopping` | product URL |
| `homedepot_products` | product URL |
| `instagram_comments` | post URL |
| `instagram_posts` | profile or post URL |
| `instagram_profiles` | profile URL |
| `instagram_reels` | profile or reel URL |
| `linkedin_company_profile` | `https://www.linkedin.com/company/<slug>` |
| `linkedin_job_listings` | job URL |
| `linkedin_person_profile` | `https://www.linkedin.com/in/<slug>` |
| `linkedin_posts` | profile URL |
| `reddit_posts` | subreddit or post URL |
| `reuter_news` | article URL |
| `tiktok_comments` | video URL |
| `tiktok_posts` | profile or post URL |
| `tiktok_profiles` | `https://www.tiktok.com/@<handle>` |
| `tiktok_shop` | product URL |
| `walmart_product` | product URL |
| `walmart_seller` | seller URL |
| `x_posts` | profile or post URL |
| `yahoo_finance_business` | company page URL |
| `youtube_profiles` | channel URL |
| `youtube_videos` | video URL |
| `zara_products` | product URL |
| `zillow_properties_listing` | listing URL |
| `zoominfo_company_profile` | company URL |

### Keyword / multi-arg pipelines (NOT a single URL)

| Pipeline | Args | Notes |
|---|---|---|
| `amazon_product_search` | `<keyword> <domain_url>` | Two positional args. `pages_to_search` is hardcoded to `1` by the CLI; extra args are ignored. |
| `linkedin_people_search` | `<url> <first_name> <last_name>` | Three positional args — search the given company/school/URL for a named person. |
| `facebook_company_reviews` | `<url> [num_reviews]` | Optional second arg; defaults to `10`. |
| `google_maps_reviews` | `<url> [days_limit]` | Optional second arg; defaults to `3`. |
| `youtube_comments` | `<url> [num_comments]` | Optional second arg; defaults to `10`. |

When in doubt about a pipeline's args, invoke it with no params — the CLI prints the expected usage line.

## Timeout guidance

| Pipeline category | Suggested `--timeout` |
|---|---|
| Single-item products / profiles | `600` (default) |
| Post / video feeds | `900`–`1200` |
| Reviews, comments, large employee lists | `1200`–`1800` |
| Company crawls (Crunchbase, ZoomInfo) | `1200`+ |

`BRIGHTDATA_POLLING_TIMEOUT=1800 bdata pipelines …` also works as an env-var default.


path: references/patterns.md
content: # Data-feeds — patterns

## Verification checklist (shared across all Bright Data CLI skills)

Before claiming a pipeline succeeded:

1. **JSON parses cleanly** (`jq . <output>` for json; line-by-line for ndjson).
2. **Record count matches expected.** Partial failures are silent — count explicitly.
3. **No top-level `error` key** on object-shaped outputs.
4. **No per-record `error` key** on array-shaped outputs.
5. **Core fields present** for the pipeline type — spot-check with `jq keys` on record 0.

## Sync timeout tuning

```bash
# Default — good for products / profiles
bdata pipelines amazon_product "<url>" -o out.json

# Long jobs — reviews, comment feeds, company employees
bdata pipelines amazon_product_reviews "<url>" --timeout 1800 -o out.json

# Or set a session default
export BRIGHTDATA_POLLING_TIMEOUT=1800
bdata pipelines linkedin_posts "<url>" -o out.json
```

## Batching same pipeline across many URLs (parallelism-capped)

```bash
mkdir -p out
xargs -a urls.txt -n 1 -P 2 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata pipelines linkedin_company_profile "$url" \
        --timeout 900 -o "out/${hash}.json" || echo "FAIL: $url" >&2
' _ {}
```

`-P 2` — keep parallelism LOW for pipelines. Each call can trigger a long server-side job; over-provisioning wastes quota and triggers rate limits.

## NDJSON verification (line-by-line parsing)

For `--format ndjson` output, each line must independently parse as JSON:

```bash
bad=0
while IFS= read -r line; do
    jq -e . <<<"$line" >/dev/null || { echo "BAD LINE"; bad=$((bad+1)); }
done < out.ndjson
echo "$bad malformed lines"
```

## Partial-failure detection

Partial failures are silent. Run this check after any batch:

```bash
for f in out/*.json; do
    # 1. Parseable?
    if ! jq . "$f" >/dev/null 2>&1; then
        echo "PARSE FAIL: $f"; continue
    fi
    # 2. Top-level error?
    if jq -e 'if type == "object" then has("error") else false end' "$f" >/dev/null; then
        echo "ERROR: $f → $(jq -r .error "$f")"; continue
    fi
    # 3. Per-record error in arrays?
    if jq -e 'if type == "array" then map(has("error")) | any else false end' "$f" >/dev/null; then
        echo "PARTIAL FAIL: $f"
    fi
done
```

## Pipeline-type resolution workflow

When the user names a platform but not a pipeline type:

```bash
# 1. Get the source of truth
bdata pipelines list > types.txt

# 2. Grep by platform
grep '^linkedin' types.txt
# → linkedin_company_profile
# → linkedin_job_listings
# → linkedin_people_search
# → linkedin_person_profile
# → linkedin_posts

# 3. Pick by intent:
#    - Profile of a person → linkedin_person_profile
#    - Profile of a company → linkedin_company_profile
#    - Job postings for a company → linkedin_job_listings
#    - Posts from a profile → linkedin_posts
#    - Search for people by attributes → linkedin_people_search
```

## Keyword- and multi-arg pipeline cheatsheet

These pipelines take non-URL or multi-positional inputs. Verify with the CLI error message if unsure — invoke with no args:

```bash
# amazon_product_search — search Amazon by keyword on a given domain
bdata pipelines amazon_product_search \
    "running shoes" "https://www.amazon.com" \
    -o search.json

# linkedin_people_search — find a named person on a company/school URL
bdata pipelines linkedin_people_search \
    "https://www.linkedin.com/company/example" "Jane" "Doe" \
    -o people.json

# google_maps_reviews — reviews for a place; optional days_limit defaults to 3
bdata pipelines google_maps_reviews \
    "https://maps.google.com/?cid=1234567890" 90 \
    -o gmaps-reviews.json

# facebook_company_reviews — optional num_reviews defaults to 10
bdata pipelines facebook_company_reviews \
    "https://www.facebook.com/example" 50 \
    -o fb-reviews.json

# youtube_comments — optional num_comments defaults to 10
bdata pipelines youtube_comments \
    "https://www.youtube.com/watch?v=abc123" 100 \
    -o yt-comments.json
```

When unsure about args, invoke with none — CLI prints the expected usage.

## Legacy `curl` fallback (deprecated)

Only when CLI cannot be installed. Pipelines map to the `/datasets/v3/trigger` + `/datasets/v3/snapshot` endpoints; the polling loop is non-trivial. If this path is truly needed, check the git history for the old `scripts/datasets.sh` (removed in this revision) as a starting point. Prefer the CLI.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)