Skill: meirk-brd/brightdata-scraper-builder (version 1.0.0)

--- name: scraper-builder description: "Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create a crawler, extract data from a website, scrape product pages, handle pagination, build a data pipeline from a web source, or automate data collection from any site — even if they don't explicitly say 'scraper'. Triggers on phrases like 'build a scraper for', 'scrape data from', 'extract products from', 'crawl pages on', 'get data from [website]', or 'I need to pull data from'." --- # Scraper Builder You are building a production-ready web scraper for the user. Your job is to guide them from "I want data from site X" to a working, robust scraper that handles real-world challenges like pagination, dynamic content, anti-bot protection, and data parsing. ## Critical: Always Validate Your Output After building the scraper, **always run it** on a small sample (1-3 pages) and show the extracted data to the user before scaling up. If the output is empty, malformed, or missing fields, iterate — fix selectors, switch APIs, or adjust the parsing logic. A scraper that doesn't produce clean data is not done. Take your time with the reconnaissance phase. Spending 2 minutes analyzing the HTML upfront prevents hours of debugging later. Quality is more important than speed here. ## How This Skill Works This skill orchestrates Bright Data's four APIs to build scrapers intelligently. Rather than writing fragile custom scraping code, you analyze the target site first, then pick the most reliable and cost-effective extraction method. The decision tree is: 1. **Does a pre-built scraper already exist?** → Use Web Scraper API (zero parsing code needed) 2. **Is the page static / no interaction needed?** → Use Web Unlocker API (cheapest, simplest) 3. **Does the page need clicks, scrolls, or JS interaction?** → Use Browser API (full automation) 4. **Need search engine results?** → Use SERP API The skill produces complete, runnable code — not pseudocode or outlines. --- ## Phase 1: Understand the Target Before writing any code, you need to understand what the user wants and what the site looks like. Ask these questions (skip any the user already answered): 1. **What site?** — The target URL or domain 2. **What data?** — Which fields they need (product names, prices, reviews, etc.) 3. **What scope?** — Single page, category pages, search results, entire site section? 4. **Pagination?** — Do they need to scrape across multiple pages? 5. **Volume?** — Roughly how many items/pages? (affects sync vs async choice and concurrency strategy — see [references/concurrency-guide.md](references/concurrency-guide.md)) 6. **Output format?** — JSON, CSV, database? (default to JSON if unspecified) 7. **Language preference?** — Python or Node.js? (default to Python if unspecified) Don't over-interview. If the user says "build a scraper for Amazon product pages", you already know: site=Amazon, data=product details, scope=product pages. Jump ahead. --- ## Phase 2: Check for Pre-Built Scrapers Before doing any custom work, check if Bright Data already has a scraper for this domain. This is the fastest, cheapest, and most reliable path. Read [references/supported-domains.md](references/supported-domains.md) for the curated list of common pre-built scrapers. But the curated list may not be complete — Bright Data supports 100+ domains and adds new scrapers regularly. If you don't see the target domain in the curated list, **query the live Dataset List API** to check: ```bash curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \ https://api.brightdata.com/datasets/list ``` This returns every available scraper with its `dataset_id` and name. Search the results for the target domain. You can also browse the full documentation index at `https://docs.brightdata.com/llms.txt` to discover scraper-specific docs and supported parameters. ### If a pre-built scraper exists Use the Web Scraper API or Python SDK platform-specific scrapers. This gives you structured JSON with no parsing code needed. **Python SDK approach (preferred):** ```python from brightdata import BrightDataClient async with BrightDataClient() as client: result = await client.scrape.amazon.products(url="https://amazon.com/dp/B0CRMZHDG8") if result.success: print(result.data) # Structured product data ``` **REST API approach (shell/curl):** ```bash bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB" ``` For bulk scraping with pre-built scrapers, use the async trigger/poll/fetch pattern: ```python async with BrightDataClient() as client: # Trigger without waiting job = await client.scrape.amazon.products_trigger(url=url) # Poll until ready await job.wait(timeout=180, poll_interval=10, verbose=True) # Fetch results data = await job.fetch() ``` Skip to Phase 5 (pagination/orchestration) if the user needs multi-page scraping with a pre-built scraper. ### If no pre-built scraper exists Continue to Phase 3 — you need to analyze the site and build a custom scraper. --- ## Phase 3: Site Reconnaissance This is the critical step that separates reliable scrapers from brittle ones. You need to understand the site's structure before writing extraction code. ### Step 3a: Fetch the page HTML Use Web Unlocker to get the raw HTML. This tells you whether the content is server-rendered or client-rendered, and gives you the actual DOM to analyze. ```python import requests import os [REDACTED_SECRET]"BRIGHTDATA_API_KEY"] ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"] response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={ "zone": ZONE, "url": "https://target-site.com/page", "format": "raw" } ) html = response.text ``` Or use the scrape skill's shell script: ```bash bash skills/scrape/scripts/scrape.sh "https://target-site.com/page" ``` ### Step 3b: Analyze the HTML structure Read [references/site-analysis-guide.md](references/site-analysis-guide.md) for the detailed analysis playbook. Look at the fetched HTML and determine: 1. **Is the content in the HTML?** If the data you need is present in the raw HTML, Web Unlocker is sufficient. If the HTML is mostly empty shells with JS framework markers (`<div id="root"></div>`, `<div id="__next"></div>`, `ng-app`), the content is client-rendered and you need Browser API. 2. **Identify reliable selectors.** Find the CSS selectors or data attributes that target the data fields. Prefer selectors in this order (most reliable → least): - `data-*` attributes (e.g., `[data-testid="product-price"]`) — survive redesigns - Semantic HTML with specific classes (e.g., `.product-card .price`) - `id` attributes — unique but may change - Structural selectors (e.g., `div > span:nth-child(2)`) — fragile, avoid 3. **Identify the data pattern.** Is it: - **List page** — multiple items in a repeating structure (product grid, search results) - **Detail page** — single item with many fields (product page, profile) - **Paginated** — multiple pages of results with next/prev controls - **Infinite scroll** — content loads on scroll (needs Browser API) - **API-backed** — check the Network tab pattern; some sites fetch data from JSON APIs 4. **Check for hidden APIs.** Many modern sites load data via XHR/fetch calls to internal APIs. If you see structured JSON endpoints in the page source or network activity, hitting those directly through Web Unlocker is often cleaner than parsing HTML. ### Step 3c: Decide the extraction approach Based on your analysis: | Finding | Approach | |---------|----------| | Content in HTML, no interaction needed | **Web Unlocker** — fetch HTML, parse with BeautifulSoup/Cheerio | | Content loaded via JSON API | **Web Unlocker** — hit the API endpoint directly | | Content requires JS rendering | **Browser API** — render then extract | | Content needs click/scroll/interaction | **Browser API** — automate the interaction | | Infinite scroll pagination | **Browser API** — scroll and collect | | Standard URL-based pagination | **Web Unlocker** — iterate page URLs | | CAPTCHA-heavy site | **Browser API** — auto-solves CAPTCHAs | --- ## Phase 4: Build the Extractor Now write the actual extraction code. The approach depends on Phase 3's decision. ### Approach A: Web Unlocker + HTML Parsing Best for static sites or sites with server-rendered HTML. This is the cheapest and fastest approach. ```python import requests import os from bs4 import BeautifulSoup [REDACTED_SECRET]"BRIGHTDATA_API_KEY"] ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"] def fetch_page(url: str) -> str: """Fetch a page through Bright Data Web Unlocker.""" response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={"zone": ZONE, "url": url, "format": "raw"} ) response.raise_for_status() return response.text def parse_products(html: str) -> list[dict]: """Extract product data from HTML. Customize selectors per site.""" soup = BeautifulSoup(html, "html.parser") products = [] for card in soup.select(".product-card"): # Adjust selector product = { "name": card.select_one(".product-title").get_text(strip=True), "price": card.select_one(".product-price").get_text(strip=True), "url": card.select_one("a")["href"], # Add more fields as needed } products.append(product) return products # Usage html = fetch_page("https://example.com/products") products = parse_products(html) ``` **Key patterns for robust parsing:** - Always use `.get_text(strip=True)` to clean whitespace - Use `.get("href", "")` instead of `["href"]` to avoid KeyError on missing attributes - Wrap individual field extraction in try/except so one bad item doesn't kill the whole scrape - Normalize prices (strip currency symbols, convert to float) in a separate step ### Approach B: Web Unlocker + Direct API Extraction When you discover the site loads data from a JSON API endpoint, hit it directly. This is the cleanest approach — no HTML parsing needed. ```python import requests import json import os [REDACTED_SECRET]"BRIGHTDATA_API_KEY"] ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"] def fetch_api(api_url: str) -> dict: """Fetch a JSON API endpoint through Web Unlocker.""" response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={"zone": ZONE, "url": api_url, "format": "raw"} ) return json.loads(response.text) # Example: site with internal API data = fetch_api("https://example.com/api/products?page=1&limit=50") products = data["results"] # Already structured! ``` ### Approach C: Browser API + Playwright Use when the site requires JavaScript rendering, interaction (clicks, scrolls, form fills), or has aggressive anti-bot measures. ```python import asyncio from playwright.async_api import async_playwright AUTH = os.environ.get("BROWSER_AUTH", "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD") async def scrape_with_browser(url: str) -> str: """Scrape a page using Bright Data Browser API.""" async with async_playwright() as p: browser = await p.chromium.connect_over_cdp( f"wss://{AUTH}@brd.superproxy.io:9222" ) page = await browser.new_page() page.set_default_navigation_timeout(120_000) # 2 minutes — required # Block unnecessary resources to reduce bandwidth costs await page.route("**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2}", lambda route: route.abort()) await page.goto(url, wait_until="domcontentloaded") # Wait for the content you need to appear await page.wait_for_selector(".product-card", timeout=30_000) # Extract data using page.evaluate for performance products = await page.evaluate(""" () => Array.from(document.querySelectorAll('.product-card')).map(card => ({ name: card.querySelector('.product-title')?.textContent?.trim(), price: card.querySelector('.product-price')?.textContent?.trim(), url: card.querySelector('a')?.href, })) """) await browser.close() return products ``` **Browser API rules you must follow:** - Always set navigation timeout to 120 seconds (`set_default_navigation_timeout(120_000)`) - One `page.goto()` per session — for a new URL, create a new browser connection - Use `wait_until="domcontentloaded"` not `networkidle` (SPAs never reach networkidle) - Wait for specific selectors rather than arbitrary delays - Block images, CSS, fonts to reduce bandwidth costs - Use `page.evaluate()` for bulk extraction — it's faster than individual selector calls ### Approach D: Browser API for Infinite Scroll For sites that load more content when you scroll down. ```python async def scrape_infinite_scroll(url: str, max_items: int = 100) -> list: """Scrape a page with infinite scroll.""" async with async_playwright() as p: browser = await p.chromium.connect_over_cdp( f"wss://{AUTH}@brd.superproxy.io:9222" ) page = await browser.new_page() page.set_default_navigation_timeout(120_000) await page.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2}", lambda route: route.abort()) await page.goto(url, wait_until="domcontentloaded") all_items = [] previous_count = 0 while len(all_items) < max_items: # Scroll to bottom await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") await page.wait_for_timeout(2000) # Wait for content to load # Extract all currently visible items items = await page.evaluate(""" () => Array.from(document.querySelectorAll('.item-selector')).map(el => ({ // ... extract fields })) """) all_items = items if len(all_items) == previous_count: break # No new content loaded — we've reached the end previous_count = len(all_items) await browser.close() return all_items[:max_items] ``` --- ## Phase 5: Handle Pagination Most scraping tasks involve multiple pages. The approach depends on the pagination type. Read [references/pagination-patterns.md](references/pagination-patterns.md) for detailed pagination strategies. ### Pattern 1: URL-Based Pagination (most common) Pages are accessed via URL parameters like `?page=2` or `?offset=20`. ```python import time def scrape_all_pages(base_url: str, max_pages: int = 50) -> list[dict]: """Scrape all pages of a paginated listing.""" all_items = [] for page_num in range(1, max_pages + 1): url = f"{base_url}?page={page_num}" html = fetch_page(url) items = parse_products(html) if not items: break # No more results all_items.extend(items) print(f"Page {page_num}: {len(items)} items (total: {len(all_items)})") time.sleep(1) # Be respectful — don't hammer the site return all_items ``` ### Pattern 2: Next-Page Link Pagination Follow "next" links found in the HTML. ```python def scrape_with_next_links(start_url: str) -> list[dict]: """Follow next-page links to scrape all pages.""" all_items = [] url = start_url while url: html = fetch_page(url) items = parse_products(html) all_items.extend(items) # Find next page link soup = BeautifulSoup(html, "html.parser") next_link = soup.select_one("a.next-page, a[rel='next'], .pagination .next a") url = next_link["href"] if next_link else None # Handle relative URLs if url and not url.startswith("http"): from urllib.parse import urljoin url = urljoin(start_url, url) time.sleep(1) return all_items ``` ### Pattern 3: Concurrent Bulk Scraping When you have many URLs (50+), **always use concurrent requests with a semaphore** — never fetch them one-by-one in a sequential loop. Read [references/concurrency-guide.md](references/concurrency-guide.md) for the full concurrency playbook including per-site tuning, multi-site parallelism, and retry strategies. ```python import asyncio import aiohttp CONCURRENCY = 20 # Start here, tune per site — see concurrency guide async def scrape_pages_concurrent(urls: list[str]) -> list[dict]: """Scrape multiple pages with controlled concurrency.""" sem = asyncio.Semaphore(CONCURRENCY) async def fetch_one(session, url): async with sem: async with session.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={"zone": ZONE, "url": url, "format": "raw"}, timeout=aiohttp.ClientTimeout(total=60), ) as resp: return {"url": url, "html": await resp.text()} async with aiohttp.ClientSession() as session: tasks = [fetch_one(session, url) for url in urls] results = await asyncio.gather(*tasks, return_exceptions=True) all_items = [] for r in results: if not isinstance(r, Exception): all_items.extend(parse_products(r["html"])) return all_items # Generate all page URLs urls = [f"https://example.com/products?page={i}" for i in range(1, 51)] items = asyncio.run(scrape_pages_concurrent(urls)) ``` ### Pattern 4: Cursor/Token-Based Pagination (APIs) Some APIs use cursor tokens instead of page numbers. ```python def scrape_with_cursor(api_base: str) -> list[dict]: """Handle cursor-based API pagination.""" all_items = [] cursor = None while True: url = f"{api_base}?limit=100" if cursor: url += f"&cursor={cursor}" data = fetch_api(url) all_items.extend(data["results"]) cursor = data.get("next_cursor") if not cursor: break return all_items ``` --- ## Phase 6: Assemble the Complete Scraper Now put it all together into a clean, runnable script. Every scraper you build should have: 1. **Configuration** — environment variables, target URLs, output settings 2. **Fetcher** — the function that retrieves pages (Web Unlocker or Browser API) 3. **Parser** — the function that extracts structured data from HTML/JSON 4. **Paginator** — the logic that handles multiple pages 5. **Concurrency** — parallel fetching with semaphore control (see [references/concurrency-guide.md](references/concurrency-guide.md)) 6. **Output** — saving results to the requested format 7. **Error handling** — retries, logging, graceful failures **Important:** If the user has more than ~50 URLs to scrape, the scraper **must** use concurrent requests — not a sequential loop. See [references/concurrency-guide.md](references/concurrency-guide.md) for the complete concurrent scraper template and tuning guidelines. ### Template: Complete Scraper Script ```python #!/usr/bin/env python3 """ Scraper for [SITE NAME] - [DESCRIPTION] Built with Bright Data [API NAME] Usage: export BRIGHTDATA_API_KEY="your-api-key" export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name" python scraper.py """ import json import os import sys import time import logging import requests from bs4 import BeautifulSoup from urllib.parse import urljoin # --- Configuration --- [REDACTED_SECRET]"BRIGHTDATA_API_KEY"] ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"] TARGET_URL = "https://example.com/products" OUTPUT_FILE = "results.json" MAX_PAGES = 50 logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s") log = logging.getLogger(__name__) # --- Fetcher --- def fetch_page(url: str, retries: int = 3) -> str: for attempt in range(retries): try: response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={"zone": ZONE, "url": url, "format": "raw"}, timeout=60, ) response.raise_for_status() return response.text except requests.RequestException as e: log.warning(f"Attempt {attempt + 1} failed for {url}: {e}") if attempt < retries - 1: time.sleep(2 ** attempt) raise RuntimeError(f"Failed to fetch {url} after {retries} attempts") # --- Parser --- def parse_items(html: str) -> list[dict]: soup = BeautifulSoup(html, "html.parser") items = [] for card in soup.select("ITEM_SELECTOR"): try: item = { "name": card.select_one("NAME_SELECTOR").get_text(strip=True), "price": card.select_one("PRICE_SELECTOR").get_text(strip=True), "url": card.select_one("a").get("href", ""), } items.append(item) except (AttributeError, TypeError) as e: log.warning(f"Failed to parse item: {e}") continue return items # --- Paginator --- def scrape_all(base_url: str) -> list[dict]: all_items = [] for page in range(1, MAX_PAGES + 1): url = f"{base_url}?page={page}" log.info(f"Scraping page {page}...") html = fetch_page(url) items = parse_items(html) if not items: log.info(f"No items on page {page} — stopping") break all_items.extend(items) log.info(f"Got {len(items)} items (total: {len(all_items)})") time.sleep(1) return all_items # --- Output --- def save_results(items: list[dict], path: str): with open(path, "w") as f: json.dump(items, f, indent=2, ensure_ascii=False) log.info(f"Saved {len(items)} items to {path}") # --- Main --- if __name__ == "__main__": items = scrape_all(TARGET_URL) save_results(items, OUTPUT_FILE) ``` When building the scraper for the user, customize this template: - Replace `ITEM_SELECTOR`, `NAME_SELECTOR`, `PRICE_SELECTOR` with real selectors from Phase 3 - Choose the right fetcher (Web Unlocker vs Browser API) - Choose the right pagination pattern from Phase 5 - Add the specific fields the user needs - Adjust the output format if they want CSV or database insertion --- ## Decision Reference: API Selection Quick Guide | Scenario | API | Cost | Speed | |----------|-----|------|-------| | Site has pre-built scraper (Amazon, LinkedIn, etc.) | Web Scraper API | Per record | Fast | | Static HTML pages, no JS needed | Web Unlocker | Per request (success only) | Fast | | Site exposes JSON API | Web Unlocker → API endpoint | Per request (success only) | Fastest | | JS-rendered content (React, Vue, Angular) | Browser API | Per bandwidth | Medium | | Infinite scroll | Browser API | Per bandwidth | Slow | | Form submission / login required | Browser API | Per bandwidth | Medium | | CAPTCHA-heavy sites | Browser API | Per bandwidth | Medium | | Search engine results | SERP API | Per request | Fast | --- ## Common Pitfalls to Avoid 1. **Don't default to Browser API when Web Unlocker suffices.** Browser API costs more (bandwidth-based) and is slower. Always try Web Unlocker first. 2. **Don't use structural CSS selectors** like `div:nth-child(3) > span`. They break when the site adds a banner or rearranges elements. Use data attributes or semantic selectors. 3. **Don't hardcode pagination limits.** Always check if the page returned actual items. An empty page means you've reached the end. 4. **Don't skip the reconnaissance phase.** Spending 2 minutes analyzing the HTML saves hours of debugging brittle selectors. 5. **Don't forget error handling per item.** One malformed product card shouldn't crash the entire scrape. Wrap individual item parsing in try/except. 6. **Don't use `networkidle` with Browser API.** SPAs never truly reach network idle. Use `domcontentloaded` + `wait_for_selector` instead. 7. **Don't create a new browser session per page when scraping a list.** If you're on a list page and clicking "next", you can stay in the same session. Only create new sessions for different base URLs. 8. **Don't scrape URLs sequentially when you have many of them.** Fetching 1,000+ URLs one-by-one with `time.sleep(1)` between each is unacceptably slow. Use concurrent requests with a semaphore. See [references/concurrency-guide.md](references/concurrency-guide.md). --- ## Examples ### Example 1: E-commerce product scraper (pre-built exists) User says: "Build a scraper for Amazon product pages, I have a list of 200 ASINs" Actions: 1. Check supported-domains.md → Amazon has pre-built scrapers 2. 200 URLs → use async trigger/poll/fetch (over 20 URL sync limit) 3. Use `client.scrape.amazon.products_trigger()` with batch of URLs 4. Poll until ready, download structured JSON 5. No HTML parsing needed — data comes pre-structured Result: Complete Python script with async batch scraping, progress logging, JSON output. ### Example 2: Custom site scraper (no pre-built) User says: "I need to scrape all job listings from jobs.customsite.com including pagination" Actions: 1. Check supported-domains.md → not listed → query Dataset List API → not found 2. Fetch page HTML via Web Unlocker to analyze structure 3. Content is in the HTML (SSR) → Web Unlocker approach 4. Identify selectors: `.job-card`, `.job-title`, `.company-name`, `.salary` 5. Pagination via `?page=N` URL parameter 6. Build complete scraper with fetcher + parser + paginator Result: Complete Python script using Web Unlocker + BeautifulSoup with URL-based pagination. ### Example 3: JS-heavy site (Browser API needed) User says: "Scrape product prices from a React SPA that loads data on scroll" Actions: 1. Fetch HTML via Web Unlocker → body is empty `div#root` → client-rendered 2. Check for hidden API in page source → no API found 3. Escalate to Browser API with Playwright 4. Implement infinite scroll pattern with resource blocking 5. Extract data via `page.evaluate()` after content loads Result: Async Playwright script with Browser API, infinite scroll handling, bandwidth optimization. --- ## Troubleshooting ### Web Unlocker returns empty or blocked page **Cause:** Site requires JavaScript rendering or has aggressive bot detection. **Solution:** Escalate to Browser API. Also try adding `data_format: "markdown"` to see if the content is there but in a different format. ### Selectors work locally but fail in production **Cause:** Site serves different HTML to different regions or user agents. **Solution:** Add `country` parameter to Web Unlocker request to target the same region. Verify selectors on the actual HTML returned by the API, not browser DevTools. ### Scraper returns duplicate items across pages **Cause:** Pagination logic is wrapping around or site uses inconsistent pagination. **Solution:** Track seen item IDs in a set. Break when duplicates appear. Verify the pagination URL pattern is correct. ### Browser API session times out **Cause:** Navigation timeout too short or site is slow to unblock. **Solution:** Always set `set_default_navigation_timeout(120_000)`. Use `wait_until="domcontentloaded"` not `networkidle`. Check if the site requires premium domains enabled on your zone. ### API returns 401 Unauthorized **Cause:** Missing or invalid `BRIGHTDATA_API_KEY` environment variable. **Solution:** Verify the key is set: `echo $BRIGHTDATA_API_KEY`. Get a fresh key from `https://brightdata.com/cp/setting/users`. --- ## Reference Files - **[references/supported-domains.md](references/supported-domains.md)** — Complete list of pre-built scrapers with dataset IDs and Python SDK methods. Check this FIRST before writing custom scraping code. - **[references/site-analysis-guide.md](references/site-analysis-guide.md)** — Step-by-step playbook for analyzing a site's HTML structure and choosing reliable selectors. - **[references/pagination-patterns.md](references/pagination-patterns.md)** — Detailed pagination strategies with code examples for every common pattern. - **[references/concurrency-guide.md](references/concurrency-guide.md)** — How to scrape URLs concurrently with semaphore control, per-site tuning, multi-site parallelism, retries, and progress tracking. **Read this whenever the user has 50+ URLs to scrape.**

path: evals/evals.json
content: {
  "skill_name": "scraper-builder",
  "evals": [
    {
      "id": 1,
      "prompt": "Build a scraper for Amazon product pages, I have about 50 ASINs I need to get pricing data for",
      "expected_output": "A Python script that uses Bright Data's pre-built Amazon scraper (Web Scraper API or Python SDK) with async batch processing for 50 URLs, structured JSON output with pricing fields",
      "files": []
    },
    {
      "id": 2,
      "prompt": "I need to scrape job listings from https://jobs.lever.co/openai including title, location, team, and link for each job. Need all pages.",
      "expected_output": "A Python script that first checks for pre-built scrapers, then does site reconnaissance with Web Unlocker, analyzes HTML structure, identifies selectors, handles pagination, and produces a complete working scraper",
      "files": []
    },
    {
      "id": 3,
      "prompt": "build a scraper for a React-based e-commerce site at https://shop.example.com that loads products on infinite scroll, I need product name, price, and image URL",
      "expected_output": "A Python script using Bright Data Browser API with Playwright, infinite scroll handling, resource blocking for bandwidth optimization, and data extraction via page.evaluate()",
      "files": []
    }
  ]
}


path: references/concurrency-guide.md
content: # Concurrency & Parallelism Guide for Scrapers

## The Problem

By default, coding agents write scrapers that fetch URLs **one at a time** — sequential loops with `time.sleep()` between requests. For small jobs (10-20 pages) this is fine, but for thousands of URLs it's painfully slow. A scraper fetching 9,000 URLs sequentially at 1 request/second takes **2.5 hours**. With 20 concurrent requests, it takes **7.5 minutes**.

**Always build scrapers with concurrency when the user has more than ~50 URLs to scrape.** This is not an optimization — it's a baseline requirement for production scrapers.

---

## Core Principle: Semaphore-Controlled Concurrency

Never fire all requests at once (`asyncio.gather(*all_tasks)`) — this overwhelms both the target site and the proxy infrastructure. Instead, use a **semaphore** to cap the number of concurrent in-flight requests.

```python
import asyncio
import aiohttp

CONCURRENCY = 20  # Max concurrent requests — tune per site (see below)

async def fetch_page(session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str) -> dict:
    """Fetch a single URL with concurrency control."""
    async with sem:
        async with session.post(
            "https://api.brightdata.com/request",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"zone": ZONE, "url": url, "format": "raw"},
            timeout=aiohttp.ClientTimeout(total=60),
        ) as response:
            html = await response.text()
            return {"url": url, "html": html, "status": response.status}

async def scrape_concurrent(urls: list[str]) -> list[dict]:
    """Scrape all URLs with controlled concurrency."""
    sem = asyncio.Semaphore(CONCURRENCY)
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_page(session, sem, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)

    successes = []
    for r in results:
        if isinstance(r, Exception):
            log.warning(f"Failed: {r}")
        else:
            successes.append(r)
    return successes
```

---

## How to Choose the Right Concurrency Level

The optimal concurrency depends on which Bright Data API you're using and the target site. Here are recommended starting points:

| API | Recommended Concurrency | Notes |
|-----|------------------------|-------|
| Web Unlocker | 20-50 concurrent requests | Bright Data handles rate limiting on their side |
| Web Scraper API (async) | 5-10 batch triggers | Each trigger handles batching internally |
| Browser API | 5-10 concurrent sessions | Each session is a real browser — resource-heavy |
| SERP API | 20-50 concurrent requests | Similar to Web Unlocker |

### Per-Site Tuning

Start with the recommended default, then adjust:

1. **Start conservative** — begin at 10 concurrent requests
2. **Watch for errors** — if you see 429 (rate limited) or 502/503 errors, reduce concurrency
3. **Ramp up gradually** — if no errors after 100 requests, increase to 20, then 30, etc.
4. **Different sites, different limits** — if scraping 3 sites, tune each independently

```python
# Per-site concurrency configuration
SITE_CONCURRENCY = {
    "amazon.com": 10,       # Aggressive anti-bot — keep low
    "example-blog.com": 30, # Lenient site — can push higher
    "news-site.com": 20,    # Middle ground
}
```

---

## Multi-Site Concurrent Scraping

When scraping multiple sites simultaneously (e.g., enriching a database from 3 different sources), run each site's scraper concurrently with **independent concurrency controls per site**:

```python
import asyncio
import aiohttp

async def scrape_site(site_name: str, urls: list[str], concurrency: int) -> list[dict]:
    """Scrape one site with its own concurrency limit."""
    sem = asyncio.Semaphore(concurrency)
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_page(session, sem, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
    log.info(f"{site_name}: scraped {len([r for r in results if not isinstance(r, Exception)])} / {len(urls)}")
    return [r for r in results if not isinstance(r, Exception)]

async def scrape_all_sites(site_configs: dict[str, dict]) -> dict[str, list[dict]]:
    """Scrape multiple sites in parallel, each with its own concurrency limit.

    site_configs = {
        "surfline": {"urls": [...], "concurrency": 20},
        "magicseaweed": {"urls": [...], "concurrency": 15},
        "wannasurf": {"urls": [...], "concurrency": 10},
    }
    """
    tasks = {
        name: scrape_site(name, cfg["urls"], cfg["concurrency"])
        for name, cfg in site_configs.items()
    }
    results = await asyncio.gather(*tasks.values())
    return dict(zip(tasks.keys(), results))
```

---

## Adding Retries with Backoff

Concurrent scrapers need smarter retry logic than sequential ones. Use exponential backoff:

```python
async def fetch_with_retry(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    url: str,
    max_retries: int = 3,
) -> dict | None:
    """Fetch a URL with concurrency control and exponential backoff."""
    async with sem:
        for attempt in range(max_retries):
            try:
                async with session.post(
                    "https://api.brightdata.com/request",
                    headers={"Authorization": f"Bearer {API_KEY}"},
                    json={"zone": ZONE, "url": url, "format": "raw"},
                    timeout=aiohttp.ClientTimeout(total=60),
                ) as response:
                    if response.status == 429:
                        wait = 2 ** attempt
                        log.warning(f"Rate limited on {url}, waiting {wait}s")
                        await asyncio.sleep(wait)
                        continue
                    response.raise_for_status()
                    return {"url": url, "html": await response.text()}
            except Exception as e:
                if attempt < max_retries - 1:
                    await asyncio.sleep(2 ** attempt)
                else:
                    log.error(f"Failed after {max_retries} attempts: {url} — {e}")
                    return None
```

---

## Progress Tracking

For large jobs, show progress so the user knows it's working:

```python
import asyncio

async def scrape_with_progress(urls: list[str], concurrency: int = 20) -> list[dict]:
    """Scrape with a live progress counter."""
    sem = asyncio.Semaphore(concurrency)
    completed = 0
    total = len(urls)
    results = []

    async def fetch_and_track(session, url):
        nonlocal completed
        result = await fetch_with_retry(session, sem, url)
        completed += 1
        if completed % 50 == 0 or completed == total:
            log.info(f"Progress: {completed}/{total} ({100*completed//total}%)")
        return result

    async with aiohttp.ClientSession() as session:
        tasks = [fetch_and_track(session, url) for url in urls]
        results = await asyncio.gather(*tasks)

    return [r for r in results if r is not None]
```

---

## Complete Template: Concurrent Scraper

```python
#!/usr/bin/env python3
"""
Concurrent scraper template.
Usage:
    export BRIGHTDATA_API_KEY="your-key"
    export BRIGHTDATA_UNLOCKER_ZONE="your-zone"
    python scraper.py
"""

import asyncio
import json
import logging
import os

import aiohttp
from bs4 import BeautifulSoup

[REDACTED_SECRET]"BRIGHTDATA_API_KEY"]
ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"]
CONCURRENCY = 20
OUTPUT_FILE = "results.json"

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
log = logging.getLogger(__name__)


async def fetch_page(session, sem, url, max_retries=3):
    async with sem:
        for attempt in range(max_retries):
            try:
                async with session.post(
                    "https://api.brightdata.com/request",
                    headers={"Authorization": f"Bearer {API_KEY}"},
                    json={"zone": ZONE, "url": url, "format": "raw"},
                    timeout=aiohttp.ClientTimeout(total=60),
                ) as resp:
                    if resp.status == 429:
                        await asyncio.sleep(2 ** attempt)
                        continue
                    resp.raise_for_status()
                    return {"url": url, "html": await resp.text()}
            except Exception as e:
                if attempt < max_retries - 1:
                    await asyncio.sleep(2 ** attempt)
                else:
                    log.error(f"Failed: {url} — {e}")
                    return None


def parse_items(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    items = []
    for card in soup.select("ITEM_SELECTOR"):
        try:
            items.append({
                "name": card.select_one("NAME_SELECTOR").get_text(strip=True),
                "url": card.select_one("a").get("href", ""),
            })
        except (AttributeError, TypeError):
            continue
    return items


async def main():
    # Generate your URLs
    urls = [f"https://example.com/page/{i}" for i in range(1, 501)]

    sem = asyncio.Semaphore(CONCURRENCY)
    completed = 0
    total = len(urls)
    all_items = []

    async with aiohttp.ClientSession() as session:
        tasks = [fetch_page(session, sem, url) for url in urls]
        for coro in asyncio.as_completed(tasks):
            result = await coro
            completed += 1
            if result:
                items = parse_items(result["html"])
                all_items.extend(items)
            if completed % 50 == 0 or completed == total:
                log.info(f"Progress: {completed}/{total} — {len(all_items)} items extracted")

    with open(OUTPUT_FILE, "w") as f:
        json.dump(all_items, f, indent=2, ensure_ascii=False)
    log.info(f"Done. Saved {len(all_items)} items to {OUTPUT_FILE}")


if __name__ == "__main__":
    asyncio.run(main())
```

---

## Quick Decision Guide

| Situation | Approach |
|-----------|----------|
| < 50 URLs | Sequential is fine — keep it simple |
| 50-500 URLs | Concurrent with semaphore (concurrency=10-20) |
| 500-10,000 URLs | Concurrent with semaphore (concurrency=20-50), add progress tracking |
| 10,000+ URLs | Concurrent + batch into chunks of 1,000, save intermediate results |
| Multiple sites | Independent concurrent scrapers per site, run in parallel |
| Pre-built scraper (Web Scraper API) | Use async trigger/poll/fetch — batching is built in |


path: references/pagination-patterns.md
content: # Pagination Patterns

Complete reference for handling every common pagination pattern when building scrapers with Bright Data.

---

## Table of Contents

- [Pattern 1: URL Parameter Pagination](#pattern-1-url-parameter-pagination)
- [Pattern 2: Next-Link Pagination](#pattern-2-next-link-pagination)
- [Pattern 3: Offset-Based Pagination](#pattern-3-offset-based-pagination)
- [Pattern 4: Cursor/Token Pagination](#pattern-4-cursortoken-pagination)
- [Pattern 5: Infinite Scroll](#pattern-5-infinite-scroll)
- [Pattern 6: Load More Button](#pattern-6-load-more-button)
- [Pattern 7: Category/Sitemap Crawling](#pattern-7-categorysitemap-crawling)
- [Async Bulk Pagination](#async-bulk-pagination)
- [Rate Limiting and Politeness](#rate-limiting-and-politeness)
- [Detecting End of Results](#detecting-end-of-results)

---

## Pattern 1: URL Parameter Pagination

The most common pattern. Pages are accessed via a parameter like `?page=2`.

### Basic Implementation

```python
import time
import requests
import os

[REDACTED_SECRET]"BRIGHTDATA_API_KEY"]
ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"]

def fetch_page(url: str) -> str:
    response = requests.post(
        "https://api.brightdata.com/request",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"zone": ZONE, "url": url, "format": "raw"},
        timeout=60,
    )
    response.raise_for_status()
    return response.text

def scrape_paginated(base_url: str, param: str = "page", start: int = 1, max_pages: int = 100) -> list:
    """
    Scrape pages where pagination is controlled by a URL parameter.

    Args:
        base_url: The URL without pagination parameter (e.g., "https://site.com/products")
        param: The pagination parameter name (e.g., "page", "p", "pg")
        start: Starting page number (usually 1, sometimes 0)
        max_pages: Safety limit to prevent infinite loops
    """
    all_items = []
    separator = "&" if "?" in base_url else "?"

    for page_num in range(start, start + max_pages):
        url = f"{base_url}{separator}{param}={page_num}"
        html = fetch_page(url)
        items = parse_items(html)  # Your parsing function

        if not items:
            break

        all_items.extend(items)
        time.sleep(1)

    return all_items
```

### Common URL Patterns

| Site Pattern | URL Format | Parameter |
|-------------|-----------|-----------|
| `?page=2` | Most common | `page` |
| `?p=2` | Shortened | `p` |
| `?pg=2` | Alternate | `pg` |
| `/page/2` | Path-based | Build URL differently |
| `?page=2&sort=price` | With other params | Preserve existing params |

### Path-Based Pagination

Some sites use `/page/2/` instead of `?page=2`:

```python
def scrape_path_paginated(base_url: str, max_pages: int = 100) -> list:
    all_items = []

    for page_num in range(1, max_pages + 1):
        if page_num == 1:
            url = base_url
        else:
            url = f"{base_url.rstrip('/')}/page/{page_num}/"

        html = fetch_page(url)
        items = parse_items(html)

        if not items:
            break

        all_items.extend(items)
        time.sleep(1)

    return all_items
```

---

## Pattern 2: Next-Link Pagination

Follow "next page" links found in the HTML. Useful when page numbers aren't predictable or URLs are complex.

```python
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def scrape_with_next_links(start_url: str, max_pages: int = 100) -> list:
    all_items = []
    url = start_url
    pages_scraped = 0

    while url and pages_scraped < max_pages:
        html = fetch_page(url)
        items = parse_items(html)
        all_items.extend(items)
        pages_scraped += 1

        # Find the next page link — try multiple common selectors
        soup = BeautifulSoup(html, "html.parser")
        next_link = (
            soup.select_one('a[rel="next"]') or
            soup.select_one('.pagination .next a') or
            soup.select_one('a.next-page') or
            soup.select_one('li.next a') or
            soup.select_one('[aria-label="Next page"]') or
            soup.select_one('[aria-label="Next"]')
        )

        if next_link and next_link.get("href"):
            url = urljoin(url, next_link["href"])
        else:
            url = None

        time.sleep(1)

    return all_items
```

---

## Pattern 3: Offset-Based Pagination

Instead of page numbers, the API uses an offset (how many items to skip).

```python
def scrape_offset_based(base_url: str, limit: int = 20, max_items: int = 1000) -> list:
    """
    Scrape with offset-based pagination.
    Common URL pattern: ?offset=0&limit=20, ?offset=20&limit=20, etc.
    """
    all_items = []
    offset = 0

    while offset < max_items:
        separator = "&" if "?" in base_url else "?"
        url = f"{base_url}{separator}offset={offset}&limit={limit}"

        html_or_json = fetch_page(url)
        items = parse_items(html_or_json)

        if not items:
            break

        all_items.extend(items)
        offset += limit
        time.sleep(1)

    return all_items
```

---

## Pattern 4: Cursor/Token Pagination

APIs that return a `next_cursor` or `next_token` for fetching the next batch. Common in modern REST and GraphQL APIs.

```python
import json

def scrape_cursor_based(api_url: str, limit: int = 100) -> list:
    """
    Scrape API with cursor-based pagination.
    Each response includes a cursor for the next page.
    """
    all_items = []
    cursor = None

    while True:
        url = f"{api_url}?limit={limit}"
        if cursor:
            url += f"&cursor={cursor}"

        raw = fetch_page(url)
        data = json.loads(raw)

        items = data.get("results", data.get("data", data.get("items", [])))
        if not items:
            break

        all_items.extend(items)

        # Look for next cursor in common locations
        cursor = (
            data.get("next_cursor") or
            data.get("cursor") or
            data.get("pagination", {}).get("next_cursor") or
            data.get("meta", {}).get("next_cursor")
        )

        if not cursor:
            break

        time.sleep(1)

    return all_items
```

---

## Pattern 5: Infinite Scroll

Content loads when the user scrolls to the bottom. Requires Browser API.

```python
import asyncio
from playwright.async_api import async_playwright

AUTH = os.environ.get("BROWSER_AUTH")

async def scrape_infinite_scroll(
    url: str,
    item_selector: str,
    max_items: int = 500,
    scroll_pause: float = 2.0,
    max_scroll_attempts: int = 20,
) -> list:
    """
    Scrape a page with infinite scroll.

    Args:
        url: The page URL
        item_selector: CSS selector for individual items
        max_items: Stop after this many items
        scroll_pause: Seconds to wait after each scroll for content to load
        max_scroll_attempts: Stop if no new content after this many scrolls
    """
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(
            f"wss://{AUTH}@brd.superproxy.io:9222"
        )
        page = await browser.new_page()
        page.set_default_navigation_timeout(120_000)

        # Block unnecessary resources
        await page.route("**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2}",
                         lambda route: route.abort())

        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector(item_selector, timeout=30_000)

        previous_count = 0
        no_new_content_count = 0

        while no_new_content_count < max_scroll_attempts:
            # Count current items
            current_count = await page.evaluate(
                f"document.querySelectorAll('{item_selector}').length"
            )

            if current_count >= max_items:
                break

            if current_count == previous_count:
                no_new_content_count += 1
            else:
                no_new_content_count = 0

            previous_count = current_count

            # Scroll to bottom
            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(int(scroll_pause * 1000))

        # Extract all items at once
        items = await page.evaluate(f"""
            () => Array.from(document.querySelectorAll('{item_selector}')).map(el => ({{
                // Customize extraction per site:
                text: el.textContent.trim(),
                // name: el.querySelector('.name')?.textContent?.trim(),
                // price: el.querySelector('.price')?.textContent?.trim(),
            }}))
        """)

        await browser.close()
        return items[:max_items]
```

### Detecting Infinite Scroll API Endpoints

Many infinite scroll pages actually fetch data from an API when you scroll. If you can find this API endpoint, you can skip the browser entirely and use Web Unlocker to hit the API directly — this is faster and cheaper.

Look for these patterns in the page's JavaScript:
- Fetch/XHR calls triggered on scroll
- URLs with `page`, `offset`, `after`, or `cursor` parameters
- GraphQL queries with pagination variables

If you find the API, use Pattern 4 (Cursor) or Pattern 3 (Offset) with Web Unlocker instead.

---

## Pattern 6: Load More Button

A "Load More" or "Show More" button that fetches additional items. Two approaches:

### If the button triggers an API call (preferred)

Find the API endpoint and hit it directly with Web Unlocker — no browser needed.

### If no API is discoverable — use Browser API

```python
async def scrape_load_more(
    url: str,
    item_selector: str,
    button_selector: str,
    max_clicks: int = 50,
) -> list:
    """Click 'Load More' button repeatedly to load all content."""
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(
            f"wss://{AUTH}@brd.superproxy.io:9222"
        )
        page = await browser.new_page()
        page.set_default_navigation_timeout(120_000)
        await page.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2}",
                         lambda route: route.abort())

        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector(item_selector, timeout=30_000)

        for _ in range(max_clicks):
            try:
                button = await page.wait_for_selector(
                    button_selector, timeout=5_000
                )
                if not button:
                    break
                await button.click()
                await page.wait_for_timeout(2000)
            except Exception:
                break  # Button gone — all content loaded

        # Extract all items
        items = await page.evaluate(f"""
            () => Array.from(document.querySelectorAll('{item_selector}')).map(el => ({{
                text: el.textContent.trim(),
            }}))
        """)

        await browser.close()
        return items
```

---

## Pattern 7: Category/Sitemap Crawling

When you need to scrape across categories, sections, or the entire site.

### Approach A: Scrape sitemap.xml

Many sites publish a sitemap that lists all pages:

```python
from bs4 import BeautifulSoup

def get_urls_from_sitemap(sitemap_url: str) -> list[str]:
    """Extract all URLs from a sitemap.xml file."""
    xml = fetch_page(sitemap_url)
    soup = BeautifulSoup(xml, "xml")

    urls = []
    # Handle sitemap index (links to other sitemaps)
    for sitemap in soup.select("sitemap loc"):
        sub_xml = fetch_page(sitemap.text.strip())
        sub_soup = BeautifulSoup(sub_xml, "xml")
        urls.extend(loc.text.strip() for loc in sub_soup.select("url loc"))

    # Handle direct URL listings
    urls.extend(loc.text.strip() for loc in soup.select("url loc"))

    return urls

# Usage
all_urls = get_urls_from_sitemap("https://example.com/sitemap.xml")
product_urls = [u for u in all_urls if "/product/" in u]
```

### Approach B: Crawl category pages

```python
def discover_and_scrape(category_urls: list[str]) -> list:
    """Scrape category listing pages to find individual item URLs, then scrape each."""
    all_items = []

    for cat_url in category_urls:
        # Scrape the category listing (may be paginated)
        item_urls = scrape_paginated_listing(cat_url)

        # Scrape each individual item page
        for item_url in item_urls:
            item_data = scrape_item_page(item_url)
            all_items.append(item_data)
            time.sleep(1)

    return all_items
```

---

## Async Bulk Pagination

When you know all page URLs upfront, fetch them concurrently for maximum speed.

### Using Python SDK

```python
import asyncio
from brightdata import BrightDataClient

async def scrape_all_pages_async(urls: list[str]) -> list:
    """Fetch multiple pages concurrently."""
    async with BrightDataClient() as client:
        # Scrape all URLs concurrently
        tasks = [client.scrape_url(url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)

        all_items = []
        for i, result in enumerate(results):
            if isinstance(result, Exception):
                print(f"Failed: {urls[i]} — {result}")
                continue
            if result.success:
                items = parse_items(result.data)
                all_items.extend(items)

        return all_items

# Generate all page URLs upfront
urls = [f"https://example.com/products?page={i}" for i in range(1, 51)]
items = asyncio.run(scrape_all_pages_async(urls))
```

### Using Web Scraper API Trigger (for supported domains)

For sites with pre-built scrapers, submit all URLs at once:

```python
import requests
import time

def bulk_scrape_with_trigger(urls: list[str], dataset_id: str) -> list:
    """Submit all URLs to Web Scraper API at once."""
    # Trigger
    response = requests.post(
        "https://api.brightdata.com/datasets/v3/trigger",
        params={"dataset_id": dataset_id, "format": "json"},
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        json={"input": [{"url": u} for u in urls]}
    )
    snapshot_id = response.json()["snapshot_id"]

    # Poll
    while True:
        progress = requests.get(
            f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
            headers={"Authorization": f"Bearer {API_KEY}"}
        )
        status = progress.json()["status"]
        if status == "ready":
            break
        if status == "failed":
            raise RuntimeError("Bulk scrape failed")
        time.sleep(10)

    # Download
    data = requests.get(
        f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
        params={"format": "json"},
        headers={"Authorization": f"Bearer {API_KEY}"}
    ).json()

    return data
```

---

## Rate Limiting and Politeness

Even with Bright Data handling proxy rotation and anti-bot bypass, be respectful of target sites:

```python
import time
import random

def polite_delay(min_seconds: float = 0.5, max_seconds: float = 2.0):
    """Random delay between requests to avoid hammering the site."""
    time.sleep(random.uniform(min_seconds, max_seconds))
```

**Guidelines:**
- Add 1-2 second delays between requests for sequential scraping
- For concurrent/async scraping, Bright Data's infrastructure handles rate limiting
- If you get consistent failures, increase the delay
- For Browser API sessions, the delay is less important since each session is independent

---

## Detecting End of Results

Knowing when to stop is just as important as knowing how to paginate.

### Strategies

1. **Empty results:** If `parse_items()` returns an empty list, stop.

2. **Duplicate detection:** If you see items you've already scraped, you've wrapped around.
   ```python
   seen_ids = set()
   for item in items:
       if item["id"] in seen_ids:
           return all_items  # Duplicates — stop
       seen_ids.add(item["id"])
   ```

3. **Total count comparison:** If the page shows "X of Y results", check when you've reached Y.
   ```python
   total_match = re.search(r'of (\d+) results', html)
   if total_match:
       total = int(total_match.group(1))
       if len(all_items) >= total:
           break
   ```

4. **Same page detection:** For next-link pagination, check you're not looping back.
   ```python
   visited_urls = set()
   if url in visited_urls:
       break  # Already visited this page
   visited_urls.add(url)
   ```

5. **Fewer items than expected:** If a page returns fewer items than the page size, it's the last page.
   ```python
   if len(items) < expected_page_size:
       all_items.extend(items)
       break  # Partial page = last page
   ```


path: references/site-analysis-guide.md
content: # Site Analysis Guide

A step-by-step playbook for analyzing any website before building a scraper. This phase is critical — spending time here prevents hours of debugging fragile selectors later.

---

## Table of Contents

- [Step 1: Initial Page Fetch](#step-1-initial-page-fetch)
- [Step 2: Content Rendering Detection](#step-2-content-rendering-detection)
- [Step 3: Hidden API Discovery](#step-3-hidden-api-discovery)
- [Step 4: Selector Strategy](#step-4-selector-strategy)
- [Step 5: Pagination Analysis](#step-5-pagination-analysis)
- [Step 6: Anti-Bot Assessment](#step-6-anti-bot-assessment)
- [Step 7: Final Decision Matrix](#step-7-final-decision-matrix)

---

## Step 1: Initial Page Fetch

Start by fetching the target page through Web Unlocker to get the raw HTML the server returns.

```python
import requests
import os

[REDACTED_SECRET]"BRIGHTDATA_API_KEY"]
ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"]

# Fetch raw HTML
response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"zone": ZONE, "url": TARGET_URL, "format": "raw"}
)
html = response.text

# Also fetch as markdown for a quick readable overview
md_response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"zone": ZONE, "url": TARGET_URL, "format": "raw", "data_format": "markdown"}
)
markdown = md_response.text
```

**What to look for in the raw HTML:**
- Is the actual content (product names, prices, text) present in the HTML?
- Or is the HTML mostly empty containers waiting for JavaScript to fill them?

---

## Step 2: Content Rendering Detection

Determine if the site renders content server-side (SSR) or client-side (CSR).

### Signs of Server-Side Rendering (Web Unlocker is sufficient)
- The data you need appears directly in the raw HTML
- Product names, prices, descriptions are visible in the source
- The HTML contains `<meta>` tags with structured data (good SEO sites do this)
- JSON-LD blocks (`<script type="application/ld+json">`) contain structured product/article data

### Signs of Client-Side Rendering (Browser API needed)
- The HTML body is mostly empty: `<div id="root"></div>` or `<div id="__next"></div>`
- Large JavaScript bundles are referenced but content isn't in the HTML
- Framework markers: `ng-app`, `data-reactroot`, `data-v-` (Vue), `__NUXT__`
- The HTML contains loading skeletons or placeholder text

### Hybrid / SSR with Hydration
Many modern sites (Next.js, Nuxt) do SSR + client hydration. The initial HTML contains the content (so Web Unlocker works), but the page also loads JS for interactivity. For these sites, Web Unlocker is the right choice — you get the data without the browser overhead.

**How to confirm:** If you see the data in the raw HTML AND you see framework scripts, it's SSR with hydration. Web Unlocker is fine.

### JSON-LD Extraction (The Hidden Goldmine)

Many e-commerce and content sites embed structured data in JSON-LD format for SEO. This is often the cleanest data source — already structured, no selector parsing needed.

```python
from bs4 import BeautifulSoup
import json

soup = BeautifulSoup(html, "html.parser")
for script in soup.select('script[type="application/ld+json"]'):
    try:
        data = json.loads(script.string)
        print(json.dumps(data, indent=2))
    except (json.JSONDecodeError, TypeError):
        continue
```

Common JSON-LD types you'll find:
- `Product` — name, price, description, image, brand, availability, reviews
- `Article` / `NewsArticle` — headline, author, datePublished, description
- `Organization` — name, address, phone, social profiles
- `BreadcrumbList` — navigation path (useful for category/hierarchy)
- `ItemList` — collections of items on listing pages

---

## Step 3: Hidden API Discovery

Many modern websites fetch their data from internal APIs. Finding and using these endpoints directly is often the cleanest approach — structured JSON with no HTML parsing.

### Where to look in the HTML

1. **`__NEXT_DATA__` (Next.js sites):**
   ```python
   next_data_script = soup.select_one('script#__NEXT_DATA__')
   if next_data_script:
       data = json.loads(next_data_script.string)
       # The page's props are in data["props"]["pageProps"]
       page_data = data["props"]["pageProps"]
   ```

2. **`__NUXT__` or `window.__NUXT__` (Nuxt.js sites):**
   Look for `<script>window.__NUXT__=` in the HTML.

3. **Embedded state objects:**
   Search the HTML for patterns like:
   - `window.__INITIAL_STATE__`
   - `window.__PRELOADED_STATE__`
   - `window.__APP_STATE__`
   - `window.__data`

4. **API URLs in script bundles:**
   Search the HTML for patterns like `"/api/"`, `"graphql"`, or the site's API subdomain.

### Using Web Unlocker to Hit APIs Directly

Once you find an API endpoint, fetch it directly — this is faster and cheaper than parsing HTML:

```python
# Example: hitting a discovered API endpoint
api_data = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "zone": ZONE,
        "url": "https://target-site.com/api/products?category=electronics&page=1",
        "format": "raw"
    }
)
products = json.loads(api_data.text)
```

---

## Step 4: Selector Strategy

When you need to parse HTML, choosing the right selectors is the most important decision for scraper longevity.

### Selector Reliability Ranking (Best → Worst)

1. **Data attributes** — `[data-testid="product-price"]`, `[data-product-id]`
   - Created specifically for testing/automation
   - Survive visual redesigns
   - Rarely change in refactors

2. **Semantic data attributes** — `[data-price]`, `[data-sku]`, `[itemprop="price"]`
   - Carry semantic meaning tied to the business data
   - `itemprop` attributes are Schema.org microdata — very stable

3. **Aria/role attributes** — `[role="listitem"]`, `[aria-label="Price"]`
   - Accessibility attributes are stable (removing them breaks screen readers)

4. **Specific class names** — `.product-card`, `.price-amount`
   - Good when class names are semantic (describe content, not style)
   - Less reliable with CSS-in-JS (e.g., `.css-1a2b3c` changes every build)

5. **ID selectors** — `#product-details`, `#price-section`
   - Unique, but many modern frameworks don't use IDs
   - Often auto-generated and unstable

6. **Structural selectors** — `div > span:nth-child(2)`, `table tr td:last-child`
   - Most fragile — breaks if any element is added/removed
   - Use only as last resort and add defensive checks

### Identifying CSS-in-JS (Unreliable Class Names)

If you see class names like these, do NOT rely on them:
- `css-1a2b3c`, `sc-bZQynM`, `jss123` (CSS Modules / styled-components / JSS)
- `_1Fk2e`, `_3xM2T` (CSS Modules with hash)
- Single-character or random-looking names

Instead, look for:
- Parent elements with semantic classes
- Data attributes on the same or nearby elements
- Text content matching (find elements by their visible text)

### Building Robust Selectors

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

# GOOD: Data attribute
price = soup.select_one('[data-testid="product-price"]')

# GOOD: Schema.org microdata
price = soup.select_one('[itemprop="price"]')

# GOOD: Semantic class with parent context
price = soup.select_one('.product-details .price')

# OK: Class name that looks semantic
price = soup.select_one('.product-price')

# BAD: Structural selector (fragile)
price = soup.select_one('.product-card > div:nth-child(3) > span')

# BAD: CSS-in-JS class (will break on next deploy)
price = soup.select_one('.css-1xk2y3z')
```

### Defensive Extraction Pattern

Always wrap individual field extraction so one missing field doesn't crash the scraper:

```python
def safe_text(element, selector: str, default: str = "") -> str:
    """Safely extract text from a child element."""
    el = element.select_one(selector)
    return el.get_text(strip=True) if el else default

def safe_attr(element, selector: str, attr: str, default: str = "") -> str:
    """Safely extract an attribute from a child element."""
    el = element.select_one(selector)
    return el.get(attr, default) if el else default

# Usage
for card in soup.select(".product-card"):
    product = {
        "name": safe_text(card, ".product-title"),
        "price": safe_text(card, ".product-price"),
        "image": safe_attr(card, "img", "src"),
        "url": safe_attr(card, "a", "href"),
    }
```

---

## Step 5: Pagination Analysis

Look at the page to understand how pagination works.

### What to Look For

1. **URL parameters:** Check if the URL changes with pages:
   - `?page=2`, `?p=2` — numbered pagination
   - `?offset=20`, `?skip=20` — offset-based
   - `?cursor=abc123` — cursor-based (common in APIs)
   - `?start=20` — start-index based

2. **HTML pagination controls:**
   ```python
   # Look for pagination elements
   pagination = soup.select_one('.pagination, .pager, nav[aria-label*="pagination"]')
   if pagination:
       links = pagination.select('a')
       for link in links:
           print(link.get_text(strip=True), link.get('href'))
   ```

3. **"Load More" buttons:** The page has a button that fetches more items — this usually means an API call you can replicate directly, or you need Browser API to click it.

4. **Infinite scroll:** No pagination controls at all; content loads on scroll. Check for:
   - `IntersectionObserver` patterns in scripts
   - Event listeners on scroll events
   - The page only showing a subset of items initially

5. **Total count indicators:** Look for text like "Showing 1-20 of 1,234 results" — this tells you total pages and current position.

### Pagination Decision

| What You See | Strategy |
|-------------|----------|
| URL changes with `?page=N` | Iterate URLs with Web Unlocker |
| "Next" link in HTML | Follow links with Web Unlocker |
| "Load More" button triggering API call | Hit the API directly with Web Unlocker |
| "Load More" button with no visible API | Browser API — click the button |
| Infinite scroll | Browser API — scroll and collect |
| API with cursor/token pagination | Hit API directly, pass cursor |

---

## Step 6: Anti-Bot Assessment

Determine how aggressively the site blocks scrapers.

### Low Protection (Web Unlocker handles easily)
- Standard websites, blogs, news sites
- Sites with basic rate limiting
- Sites that serve content to search engine crawlers

### Medium Protection (Web Unlocker usually works)
- E-commerce sites with bot detection
- Sites behind Cloudflare, Akamai, PerimeterX
- Web Unlocker's auto-bypass handles most of these

### High Protection (May need Browser API)
- Sites requiring JavaScript execution for content
- Sites with behavior-based bot detection (mouse movement, scroll patterns)
- Sites with aggressive fingerprinting
- Sites that require solving multiple CAPTCHAs

**How to tell:** If Web Unlocker returns the full content, it's sufficient. If it returns empty pages, blocked messages, or CAPTCHA pages, escalate to Browser API.

---

## Step 7: Final Decision Matrix

After completing your analysis, use this matrix:

| Content in HTML? | Interaction needed? | API endpoint found? | → Use |
|:-:|:-:|:-:|---|
| Yes | No | No | **Web Unlocker** + HTML parsing |
| Yes | No | Yes | **Web Unlocker** → API endpoint (preferred) |
| No | No | Yes | **Web Unlocker** → API endpoint |
| No | No | No | **Browser API** (JS rendering needed) |
| — | Yes (click/scroll) | No | **Browser API** |
| — | Yes (infinite scroll) | Yes | **Web Unlocker** → API (replicate scroll endpoint) |
| — | Yes (infinite scroll) | No | **Browser API** + scroll automation |


path: references/supported-domains.md
content: # Pre-Built Scrapers — Supported Domains

Check this list before writing any custom scraping code. If the target domain is listed here, use the Web Scraper API or Python SDK platform-specific scraper instead of building a custom solution.

**This list is a curated subset.** Bright Data supports 100+ domains with pre-built scrapers. The tables below cover the most common ones, but the full list is dynamic and grows regularly. If you don't see a domain here, **always check the live dataset list** before falling back to a custom scraper.

---

## Discovering ALL Available Scrapers (Dynamic Lookup)

The tables below are a quick reference, but they may not include every scraper Bright Data offers. Before building a custom scraper for any domain, use these two methods to check for a pre-built solution:

### Method 1: Query the Dataset List API

This returns every available pre-built scraper with its `dataset_id` and name:

```bash
curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
     https://api.brightdata.com/datasets/list
```

```python
import requests
import os

response = requests.get(
    "https://api.brightdata.com/datasets/list",
    headers={"Authorization": f"Bearer {os.environ['BRIGHTDATA_API_KEY']}"}
)
datasets = response.json()

# Search for a specific domain
domain = "shopify"  # whatever site you're looking for
matches = [d for d in datasets if domain.lower() in d["name"].lower()]
for d in matches:
    print(f"  {d['id']}: {d['name']} ({d['size']:,} records)")
```

**Response format:**
```json
[
  {"id": "gd_l1vijqt9jfj7olije", "name": "Crunchbase companies information", "size": 2300000},
  {"id": "gd_l1vikfch901nx3by4", "name": "Instagram - Profiles", "size": 620000000}
]
```

Each item has:
- `id` — the `dataset_id` to use with the Web Scraper API (`/datasets/v3/scrape` or `/datasets/v3/trigger`)
- `name` — human-readable name describing the scraper
- `size` — number of records in the dataset

When you find a matching dataset, use its `id` as the `dataset_id` parameter in Web Scraper API calls.

### Method 2: Browse the Full Documentation

Fetch the complete Bright Data documentation index at:

```
https://docs.brightdata.com/llms.txt
```

This file lists all available documentation pages. Use it to discover scraper-specific docs, API references, and guides for any supported domain. Particularly useful for finding input/output schemas and supported parameters for specific scrapers.

### When to Use Dynamic Lookup

- **Always** when the target domain isn't in the Quick Lookup table below
- When you're unsure if a scraper exists for a specific data type on a known domain (e.g., "does Bright Data have an Amazon seller reviews scraper?")
- When the user asks about a niche or recently-added platform
- When building scrapers for sites in categories like real estate, travel, food delivery, job boards, or classifieds — Bright Data frequently adds new scrapers for these verticals

---

## Quick Lookup

| Domain | Has Pre-Built Scraper? | Approach |
|--------|----------------------|----------|
| Amazon | Yes | SDK: `client.scrape.amazon.*` or Shell: `datasets.sh amazon_*` |
| LinkedIn | Yes | SDK: `client.scrape.linkedin.*` or Shell: `datasets.sh linkedin_*` |
| Instagram | Yes | SDK: `client.scrape.instagram.*` or Shell: `datasets.sh instagram_*` |
| Facebook | Yes | SDK: `client.scrape.facebook.*` or Shell: `datasets.sh facebook_*` |
| TikTok | Yes | SDK: `client.scrape.tiktok.*` or Shell: `datasets.sh tiktok_*` |
| YouTube | Yes | SDK: `client.scrape.youtube.*` or Shell: `datasets.sh youtube_*` |
| Twitter/X | Yes | Shell: `datasets.sh x_posts` |
| Reddit | Yes | SDK: `client.scrape.reddit.*` or Shell: `datasets.sh reddit_posts` |
| Walmart | Yes | Shell: `datasets.sh walmart_*` |
| eBay | Yes | Shell: `datasets.sh ebay_product` |
| Best Buy | Yes | Shell: `datasets.sh bestbuy_products` |
| Home Depot | Yes | Shell: `datasets.sh homedepot_products` |
| Etsy | Yes | Shell: `datasets.sh etsy_products` |
| Zara | Yes | Shell: `datasets.sh zara_products` |
| Google Maps | Yes | Shell: `datasets.sh google_maps_reviews` |
| Google Shopping | Yes | Shell: `datasets.sh google_shopping` |
| Google Play Store | Yes | Shell: `datasets.sh google_play_store` |
| Apple App Store | Yes | Shell: `datasets.sh apple_app_store` |
| Crunchbase | Yes | Shell: `datasets.sh crunchbase_company` |
| ZoomInfo | Yes | Shell: `datasets.sh zoominfo_company_profile` |
| Zillow | Yes | Shell: `datasets.sh zillow_properties_listing` |
| Booking.com | Yes | Shell: `datasets.sh booking_hotel_listings` |
| Yahoo Finance | Yes | Shell: `datasets.sh yahoo_finance_business` |
| Reuters | Yes | Shell: `datasets.sh reuter_news` |
| GitHub | Yes | Shell: `datasets.sh github_repository_file` |
| Any other site | No | Use Web Unlocker or Browser API (custom scraper) |

---

## E-Commerce Scrapers

### Amazon

| Data Type | Shell Command | Python SDK | Dataset ID |
|-----------|--------------|------------|------------|
| Product details | `datasets.sh amazon_product <url>` | `client.scrape.amazon.products(url=...)` | `gd_l7q7dkf244hwjntr0` |
| Product reviews | `datasets.sh amazon_product_reviews <url>` | `client.scrape.amazon.reviews(url=...)` | — |
| Seller info | — | `client.scrape.amazon.sellers(url=...)` | — |
| Product search | `datasets.sh amazon_product_search <keyword> <domain>` | `client.scrape.amazon.products_search(keyword=...)` | — |

**Amazon-specific tips:**
- Use `x-unblock-zipcode` header with Web Unlocker to get location-specific pricing
- Product search accepts keywords + domain URL (e.g., `https://www.amazon.com`)
- Reviews scraper returns all reviews for a given product URL

### Walmart

| Data Type | Shell Command | Dataset ID |
|-----------|--------------|------------|
| Product details | `datasets.sh walmart_product <url>` | — |
| Seller info | `datasets.sh walmart_seller <url>` | — |

### eBay

| Data Type | Shell Command | Dataset ID |
|-----------|--------------|------------|
| Listing details | `datasets.sh ebay_product <url>` | — |

### Other E-Commerce

| Platform | Shell Command |
|----------|--------------|
| Best Buy | `datasets.sh bestbuy_products <url>` |
| Home Depot | `datasets.sh homedepot_products <url>` |
| Etsy | `datasets.sh etsy_products <url>` |
| Zara | `datasets.sh zara_products <url>` |
| Google Shopping | `datasets.sh google_shopping <url>` |

---

## Social Media Scrapers

### Instagram

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Profile | `datasets.sh instagram_profiles <url>` | `client.scrape.instagram.profiles(url=...)` |
| Posts | `datasets.sh instagram_posts <url>` | `client.scrape.instagram.posts(url=...)` |
| Reels | `datasets.sh instagram_reels <url>` | `client.scrape.instagram.reels(url=...)` |
| Comments | `datasets.sh instagram_comments <url>` | `client.scrape.instagram.comments(url=...)` |
| Profile search | — | `client.scrape.instagram.profiles_search(user_name=...)` |
| Posts by user | — | `client.scrape.instagram.posts_search(url=..., num_of_posts=20)` |

### Facebook

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Posts | `datasets.sh facebook_posts <url>` | `client.scrape.facebook.posts_by_profile(url=..., num_of_posts=10)` |
| Group posts | — | `client.scrape.facebook.posts_by_group(url=..., num_of_posts=10)` |
| Marketplace | `datasets.sh facebook_marketplace_listings <url>` | — |
| Reviews | `datasets.sh facebook_company_reviews <url> [num]` | — |
| Events | `datasets.sh facebook_events <url>` | — |
| Reels | — | `client.scrape.facebook.reels(url=...)` |
| Comments | — | `client.scrape.facebook.comments(url=..., num_of_comments=20)` |

### TikTok

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Profile | `datasets.sh tiktok_profiles <url>` | `client.scrape.tiktok.profiles(url=...)` |
| Posts/Videos | `datasets.sh tiktok_posts <url>` | — |
| Shop products | `datasets.sh tiktok_shop <url>` | — |
| Comments | `datasets.sh tiktok_comments <url>` | — |

### YouTube

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Channel | `datasets.sh youtube_profiles <url>` | `client.scrape.youtube.profiles(url=...)` |
| Videos | `datasets.sh youtube_videos <url>` | `client.scrape.youtube.videos(url=...)` |
| Comments | `datasets.sh youtube_comments <url> [num]` | `client.scrape.youtube.comments(url=...)` |
| Video search | — | `client.scrape.youtube.videos_search(keyword=..., num_of_videos=10)` |

### Twitter/X

| Data Type | Shell Command |
|-----------|--------------|
| Posts/Tweets | `datasets.sh x_posts <url>` |

### Reddit

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Posts | `datasets.sh reddit_posts <url>` | `client.scrape.reddit.posts(url=...)` |

---

## Professional / Business Scrapers

### LinkedIn

| Data Type | Shell Command | Python SDK |
|-----------|--------------|------------|
| Person profile | `datasets.sh linkedin_person_profile <url>` | `client.scrape.linkedin.profiles(url=...)` |
| Company profile | `datasets.sh linkedin_company_profile <url>` | `client.scrape.linkedin.companies(url=...)` |
| Job listings | `datasets.sh linkedin_job_listings <url>` | — |
| Posts | `datasets.sh linkedin_posts <url>` | `client.scrape.linkedin.posts(url=...)` |
| People search | `datasets.sh linkedin_people_search <url> <first> <last>` | `client.scrape.linkedin.profiles_search(keyword=..., location=...)` |
| Job search | — | `client.scrape.linkedin.jobs_search(keyword=..., location=...)` |
| Company search | — | `client.scrape.linkedin.companies_search(keyword=...)` |

### Other Business

| Platform | Shell Command |
|----------|--------------|
| Crunchbase | `datasets.sh crunchbase_company <url>` |
| ZoomInfo | `datasets.sh zoominfo_company_profile <url>` |

---

## Other Scrapers

| Platform | Data Type | Shell Command |
|----------|-----------|--------------|
| Zillow | Property listings | `datasets.sh zillow_properties_listing <url>` |
| Booking.com | Hotel listings | `datasets.sh booking_hotel_listings <url>` |
| Yahoo Finance | Company/stock data | `datasets.sh yahoo_finance_business <url>` |
| Reuters | News articles | `datasets.sh reuter_news <url>` |
| GitHub | Repository files | `datasets.sh github_repository_file <url>` |
| Google Maps | Business reviews | `datasets.sh google_maps_reviews <url> [days]` |
| Google Play | App details | `datasets.sh google_play_store <url>` |
| Apple App Store | App details | `datasets.sh apple_app_store <url>` |

---

## Using Pre-Built Scrapers for Bulk Operations

### Python SDK — Concurrent Batch

```python
import asyncio
from brightdata import BrightDataClient

async def bulk_scrape(urls: list[str]):
    async with BrightDataClient() as client:
        tasks = [client.scrape.amazon.products(url=u) for u in urls]
        results = await asyncio.gather(*tasks)
        return [r.data for r in results if r.success]
```

### Shell — Multiple URLs with Web Scraper API

```bash
# Direct API call with multiple inputs
bash scripts/fetch.sh gd_l7q7dkf244hwjntr0 '[{"url":"https://amazon.com/dp/B001"},{"url":"https://amazon.com/dp/B002"}]'
```

### REST API — Async Trigger for Large Batches

For 20+ URLs, use the async trigger endpoint:

```python
import requests

# Trigger
response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={"dataset_id": "gd_l7q7dkf244hwjntr0", "format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
    json={"input": [{"url": u} for u in urls]}
)
snapshot_id = response.json()["snapshot_id"]

# Poll + Download (see Web Scraper API reference for full pattern)
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)