Skill: damirikys/scrapling-fetcher (version 1.0.3)

--- name: scrapling description: "Web scraping using Scrapling — a Python framework with anti-bot bypass (Cloudflare Turnstile, fingerprint spoofing), adaptive element tracking, stealth headless browser, and full CSS/XPath extraction. Use when web_fetch fails (Cloudflare, JS-rendered pages), or when extracting structured data from websites (prices, articles, lists). Supports HTTP, stealth, and full browser modes. Source: github.com/D4Vinci/Scrapling (PyPI: scrapling). Only use on sites you have permission to scrape." license: MIT metadata: source: https://github.com/D4Vinci/Scrapling pypi: https://pypi.org/project/scrapling/ --- # Scrapling Skill **Source:** https://github.com/D4Vinci/Scrapling (open source, MIT-like license) **PyPI:** `scrapling` — install before first use (see below) > ⚠️ Only scrape sites you have permission to access. Respect `robots.txt` and Terms of Service. Do not use stealth modes to bypass paywalls or access restricted content without authorization. ## Installation (one-time, confirm with user before running) ```bash pip install scrapling[all] patchright install chromium # required for stealth/dynamic modes ``` - `scrapling[all]` installs `patchright` (a stealth fork of Playwright, bundled as a PyPI package — not a typo), `curl_cffi`, MCP server deps, and IPython shell. - `patchright install chromium` downloads Chromium (~100 MB) via patchright's own installer (same mechanism as `playwright install chromium`). - Confirm with user before running — installs ~200 MB of dependencies and browser binaries. ## Script `scripts/scrape.py` — CLI wrapper for all three fetcher modes. ```bash # Basic fetch (text output) python3 ~/skills/scrapling/scripts/scrape.py <url> -q # CSS selector extraction python3 ~/skills/scrapling/scripts/scrape.py <url> --selector ".class" -q # Stealth mode (Cloudflare bypass) — only on sites you're authorized to access python3 ~/skills/scrapling/scripts/scrape.py <url> --mode stealth -q # JSON output python3 ~/skills/scrapling/scripts/scrape.py <url> --selector "h2" --json -q ``` ## Fetcher Modes - **http** (default) — Fast HTTP with browser TLS fingerprint spoofing. Most sites. - **stealth** — Headless Chrome with anti-detect. For Cloudflare/anti-bot. - **dynamic** — Full Playwright browser. For heavy JS SPAs. ## When to Use Each Mode - `web_fetch` returns 403/429/Cloudflare challenge → use `--mode stealth` - Page content requires JS execution → use `--mode dynamic` - Regular site, just need text/data → use `--mode http` (default) ## Python Inline Usage For custom logic beyond the CLI, write inline Python. See `references/patterns.md` for: - Adaptive scraping (`auto_save` / `adaptive` — saves element fingerprints locally) - Session/cookie handling - Async usage - XPath, find_similar, attribute extraction ## Notes - **MCP server** (`scrapling mcp`): starts a local network service for AI-native scraping. Only start if explicitly needed and trusted — it exposes a local HTTP server. - **`auto_save=True`**: persists element fingerprints to disk for adaptive re-scraping. Creates local state in working directory. - Stealth/dynamic modes use Chromium headless — no `xvfb-run` needed. - For large-scale crawls, use the Spider API (see Scrapling docs).

path: scripts/scrape.py
content: #!/usr/bin/env python3
"""
Scrapling fetch script - fetches a URL with stealth capabilities.

Usage:
  python3 scrape.py <url> [options]

Options:
  --mode     http | stealth | dynamic (default: http)
  --selector CSS selector to extract specific elements
  --attr     Attribute to extract from elements (default: text)
  --json     Output as JSON array
  --text     Extract all visible text (default if no selector)
  -q         Quiet mode (suppress INFO logs)
"""

import argparse
import sys
import logging


def main():
    parser = argparse.ArgumentParser(description='Scrapling fetch')
    parser.add_argument('url', help='URL to fetch')
    parser.add_argument('--mode', choices=['http', 'stealth', 'dynamic'], default='http',
                        help='Fetcher mode: http (fast), stealth (anti-bot), dynamic (JS rendering)')
    parser.add_argument('--selector', '-s', help='CSS selector to extract elements')
    parser.add_argument('--attr', default='text', help='Attribute to extract (text, href, src, etc.)')
    parser.add_argument('--json', action='store_true', help='Output as JSON array')
    parser.add_argument('-q', '--quiet', action='store_true', help='Suppress INFO logs')
    args = parser.parse_args()

    if args.quiet:
        logging.disable(logging.INFO)

    try:
        page = _fetch(args.url, args.mode)
    except Exception as e:
        print(f"ERROR: {e}", file=sys.stderr)
        sys.exit(1)

    if not args.selector:
        # Default: return all visible text
        text = page.get_all_text(' ', ignore_tags=('script', 'style', 'noscript'))
        print(text)
        return

    elements = page.css(args.selector)
    if not elements:
        print(f"No elements matched: {args.selector}", file=sys.stderr)
        sys.exit(1)

    results = []
    for el in elements:
        if args.attr == 'text':
            val = el.text or el.get_all_text()
        elif args.attr == 'html':
            val = el.html_content
        else:
            val = el.attrib.get(args.attr, '')
        results.append(val.strip())

    if args.json:
        import json
        print(json.dumps(results, ensure_ascii=False, indent=2))
    else:
        for r in results:
            if r:
                print(r)


def _fetch(url, mode):
    if mode == 'http':
        from scrapling.fetchers import Fetcher
        return Fetcher.get(url)
    elif mode == 'stealth':
        from scrapling.fetchers import StealthyFetcher
        return StealthyFetcher.fetch(url, headless=True, network_idle=True)
    elif mode == 'dynamic':
        from scrapling.fetchers import DynamicFetcher
        return DynamicFetcher.fetch(url, headless=True, network_idle=True)


if __name__ == '__main__':
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (MEDIUM)