Skill: yumiu8103-hue/web-markdown-scraper (version 1.0.0)

--- name: web_markdown_scraper description: > Fetch one or more public webpages with Scrapling, extract the main content, and convert HTML into Markdown using html2text. Supports static HTTP, concurrent async, stealth anti-bot (Camoufox/Firefox), and dynamic Playwright Chromium fetching modes with production-grade automatch. metadata: {"openclaw":{"emoji":"🕸️","requires":{"bins":["python3"]}}} --- # Web Markdown Scraper Use this skill when the user wants to: - Scrape one or more public webpages (static or JavaScript-rendered) - Convert HTML pages into clean Markdown - Extract article/body text for summarization, analysis, or indexing - Bypass anti-bot protections (Cloudflare, Datadome, etc.) via stealth mode - Scrape many URLs concurrently (async mode) - Track page elements reliably across website redesigns (automatch) - Save the extracted results as `.md` files ## Fetcher Mode Selection Guide | Mode | Fetcher Class | Best For | |------|--------------|----------| | `http` (default) | `Fetcher` | Fast static pages, RSS, APIs | | `async` | `AsyncFetcher` | Batch of 5+ static URLs in parallel | | `stealth` | `StealthyFetcher` | Anti-bot sites, Cloudflare, fingerprint checks | | `dynamic` | `PlayWrightFetcher` | Heavy SPAs, React/Vue/Angular apps | **Decision rule**: Start with `http`. If you get a 403 / CAPTCHA / empty body, switch to `stealth`. If the content is rendered client-side (empty on first load), use `dynamic`. Use `async` when scraping many static URLs at once to save time. ## Inputs ### URL sources - `--url URL` — one target URL (repeat flag for multiple: `--url A --url B`) - `--url-file FILE` — plain text file with one URL per line ### Fetcher - `--mode http|async|stealth|dynamic` — fetcher backend (default: `http`) ### Content extraction - `--selector CSS` — CSS selector for the main content area (omit = full page) - `--preserve-links` — keep hyperlinks in the Markdown output - `--output-dir DIR` — save per-page `.md` files and a master `index.json` here ### AutoMatch — production resilience - `--auto-save` — fingerprint & persist selected elements to the local DB on first run - `--auto-match` — on subsequent runs, find elements by fingerprint even if the site layout has changed (do NOT need to update the CSS selector) ### Browser options (stealth / dynamic only) - `--headless true|false|virtual` — headless mode; `virtual` uses Xvfb (default: `true`) - `--network-idle` — wait until no network activity for ≥500 ms before capturing - `--block-images` — block image loading (saves bandwidth and proxy quota) - `--disable-resources` — drop fonts/images/media/stylesheets for ~25% faster loads - `--wait-selector CSS` — pause until this element appears in the DOM - `--wait-selector-state attached|visible|detached|hidden` — element state (default: `attached`) - `--timeout MS` — global timeout in ms (default: 30 000) - `--wait MS` — extra idle wait after page load in ms ### StealthyFetcher extras (stealth mode only) - `--humanize SECONDS` — simulate human-like cursor movement (max duration in seconds) - `--geoip` — spoof browser timezone, locale, language, and WebRTC IP from proxy geolocation - `--block-webrtc` — prevent real-IP leaks via WebRTC - `--disable-ads` — install uBlock Origin in the browser session - `--proxy URL` — HTTP/SOCKS proxy as a URL string, or JSON: `'{"server":"host:port","username":"u","password":"p"}'` ### Reliability - `--retry N` — retry failed requests up to N times with exponential backoff (max 30 s) ## Rules 1. Only process public `http://` or `https://` pages. 2. Never bypass login walls, CAPTCHAs, paywalls, or access controls. 3. Prefer the main article or body content; avoid polluting the output with navigation, headers, footers, or cookie banners — use `--selector` to target the content area. 4. When `--auto-save` is used, always also pass `--selector` so Scrapling knows which element fingerprint to record. 5. On subsequent runs for layout-changed pages, use `--auto-match` instead of `--auto-save`. Do not use both flags at once. 6. Use `--mode async` for batch jobs with 5+ static URLs for parallel execution. 7. Combine `--disable-resources` with `--block-images` in stealth/dynamic mode when you only need text content — this can cut load times by up to 40%. 8. Always inspect the top-level `ok` field and per-result `ok` fields before using content. 9. If `ok` is `false`, report the exact `error` string — do not invent or guess content. 10. When `--network-idle` is insufficient, use `--wait-selector` for a specific DOM element to guarantee the content has loaded before capture. ## Command Patterns ### Basic static page ```bash python3 "{baseDir}/scrape_to_markdown.py" --url "<URL>" ``` ### Static page — target specific content area ```bash python3 "{baseDir}/scrape_to_markdown.py" --url "<URL>" --selector "article.main-content" ``` ### Stealth mode — bypass anti-bot protection ```bash python3 "{baseDir}/scrape_to_markdown.py" --url "<URL>" --mode stealth --network-idle ``` ### Stealth + proxy + human fingerprint (maximum stealth) ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url "<URL>" \ --mode stealth \ --proxy "[REDACTED_SECRET]" \ --humanize 2.0 \ --geoip \ --block-webrtc \ --network-idle ``` ### Dynamic SPA page (Playwright Chromium) ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url "<URL>" \ --mode dynamic \ --wait-selector ".product-list" \ --network-idle \ --disable-resources ``` ### Async concurrent batch (multiple URLs) ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --mode async \ --url "<URL1>" --url "<URL2>" --url "<URL3>" ``` ### Batch from file + stealth + save to disk ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url-file urls.txt \ --mode stealth \ --disable-resources \ --output-dir outputs ``` ### First-run automatch setup (save fingerprint) ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url "<URL>" \ --selector ".article-body" \ --auto-save \ --output-dir outputs ``` ### Subsequent run after site layout change (adaptive match) ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url "<URL>" \ --selector ".article-body" \ --auto-match \ --output-dir outputs ``` ### Full production scrape ```bash python3 "{baseDir}/scrape_to_markdown.py" \ --url "<URL>" \ --mode stealth \ --selector "main article" \ --auto-match \ --preserve-links \ --network-idle \ --disable-resources \ --timeout 60000 \ --retry 3 \ --output-dir outputs ``` ## Output Handling JSON is printed to stdout. Always check `ok` before using content. **Top-level fields:** - `ok` — `true` only if every URL succeeded - `total` / `succeeded` / `failed` — count summary - `results` — array of per-URL result objects - `output_index_file` — path to saved `index.json` (if `--output-dir` used) **Per-URL result fields (when `ok: true`):** - `url` — the requested URL - `status` — HTTP status code (e.g. `200`) - `title` — page `<title>` text - `markdown` — extracted content as Markdown ← **use this as main content** - `markdown_length` — character count (useful for quality checks) - `output_markdown_file` — path to saved `.md` file (if `--output-dir` used) **On failure (`ok: false` in a result):** - `error` — exact error message; report this verbatim, do not invent content


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)