Skill: zhengxinjipai/scrapling-web-scraper (version 1.0.0)

--- name: Scrapling Web Scraping description: Zero-bot-detection web scraping for OpenClaw. Bypass Cloudflare, handle JavaScript-heavy sites, and adapt to website changes automatically. Use when you need to scrape protected websites, extract data from dynamic JavaScript SPAs, or bypass anti-bot detection systems. Supports three modes - basic (fast HTTP), stealth (undetectable), dynamic (browser automation). identifier: scrapling-web-scraping version: 1.0.0 author: 老二 category: web-scraping --- # Scrapling Web Scraping Zero-bot-detection web scraping for OpenClaw. Bypass Cloudflare, handle JavaScript-heavy sites, and adapt to website changes automatically. ## Quick Start ```bash # Install Scrapling pip install "scrapling[all]" scrapling install # Basic usage python3 /root/.openclaw/skills/scrapling-web-scraping/scrapling_tool.py https://example.com # Bypass Cloudflare python3 /root/.openclaw/skills/scrapling-web-scraping/scrapling_tool.py https://protected-site.com --mode stealth --cloudflare # Extract specific data python3 /root/.openclaw/skills/scrapling-web-scraping/scrapling_tool.py https://example.com --selector ".product-title" # JavaScript-heavy sites python3 /root/.openclaw/skills/scrapling-web-scraping/scrapling_tool.py https://spa-app.com --mode dynamic --wait ".content-loaded" ``` ## Usage with OpenClaw ### Natural Language Commands **Basic scraping:** > "用Scrapling抓取 https://example.com 的标题和所有链接" **Bypass protection:** > "用隐身模式抓取 https://protected-site.com，绕过Cloudflare" **Extract data:** > "抓取 https://shop.com 的商品名称和价格，CSS选择器是 .product" **Dynamic content:** > "抓取 https://spa-app.com，等待 .data-loaded 元素加载完成" ### Python Code ```python # Basic scraping from scrapling.fetchers import Fetcher page = Fetcher.get('https://example.com') title = page.css('title::text').get() # Bypass Cloudflare from scrapling.fetchers import StealthyFetcher page = StealthyFetcher.fetch('https://protected.com', headless=True, solve_cloudflare=True) # JavaScript sites from scrapling.fetchers import DynamicFetcher page = DynamicFetcher.fetch('https://spa-app.com', headless=True, network_idle=True) ``` ## Features | Feature | Command | Description | |---------|---------|-------------| | Basic Scrape | `--mode basic` | Fast HTTP requests | | Stealth Mode | `--mode stealth` | Bypass Cloudflare/anti-bot | | Dynamic Mode | `--mode dynamic` | Handle JavaScript sites | | CSS Selectors | `--selector ".class"` | Extract specific elements | | JSON Output | `--json` | Machine-readable output | ## Examples ### 1. Scrape with CSS Selector ```bash python3 scrapling_tool.py https://quotes.toscrape.com --selector ".quote .text" --json ``` ### 2. Bypass Cloudflare ```bash python3 scrapling_tool.py https://nopecha.com/demo/cloudflare --mode stealth --cloudflare ``` ### 3. Wait for Dynamic Content ```bash python3 scrapling_tool.py https://spa-app.com --mode dynamic --wait ".loaded" --json ``` ## CLI Reference ``` python3 scrapling_tool.py URL [options] Options: --mode {basic,stealth,dynamic} Scraping mode (default: basic) --selector, -s CSS_SELECTOR Extract specific elements --cloudflare Solve Cloudflare (stealth mode only) --wait SELECTOR Wait for element (dynamic mode only) --json, -j Output as JSON ``` ## Advanced: Custom Scripts Create custom scraping scripts in `/root/.openclaw/skills/scrapling-web-scraping/`: ```python from scrapling.fetchers import StealthyFetcher # Your custom scraper def scrape_products(url): page = StealthyFetcher.fetch(url, headless=True) products = [] for item in page.css('.product'): products.append({ 'name': item.css('.name::text').get(), 'price': item.css('.price::text').get(), 'link': item.css('a::attr(href)').get() }) return products ``` ## Notes - Requires Python 3.10+ - First run: `scrapling install` to download browsers - Respect website Terms of Service - Use responsibly --- **Created**: 2026-03-05 by 老二 **Source**: https://github.com/D4Vinci/Scrapling

path: scrapling_tool.py
content: #!/usr/bin/env python3
"""
Scrapling Web Scraping Tool for OpenClaw
简化版封装，方便命令行使用
"""

import sys
import json
import argparse

# 尝试导入scrapling
try:
    from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
    from scrapling.fetchers import FetcherSession, StealthySession, DynamicSession
    SCRAPLING_AVAILABLE = True
except ImportError:
    SCRAPLING_AVAILABLE = False
    print("❌ Scrapling未安装。请先运行: pip install 'scrapling[all]'")
    sys.exit(1)

def scrape_basic(url, selector=None):
    """基础抓取"""
    try:
        page = Fetcher.get(url)
        result = {
            "url": url,
            "title": page.css('title::text').get(),
            "status": "success"
        }
        if selector:
            result["data"] = page.css(selector).getall()
        else:
            # 提取主要文本内容
            result["text"] = page.css('body').get()
        return result
    except Exception as e:
        return {"error": str(e), "url": url}

def scrape_stealth(url, selector=None, solve_cloudflare=False):
    """隐身模式抓取（绕过Cloudflare）"""
    try:
        page = StealthyFetcher.fetch(url, 
                                      headless=True,
                                      solve_cloudflare=solve_cloudflare)
        result = {
            "url": url,
            "title": page.css('title::text').get(),
            "mode": "stealth",
            "status": "success"
        }
        if selector:
            result["data"] = page.css(selector).getall()
        return result
    except Exception as e:
        return {"error": str(e), "url": url}

def scrape_dynamic(url, selector=None, wait_for=None):
    """动态内容抓取（JavaScript渲染）"""
    try:
        page = DynamicFetcher.fetch(url, headless=True, network_idle=True)
        
        if wait_for:
            page.wait_for_selector(wait_for, timeout=10000)
        
        result = {
            "url": url,
            "title": page.css('title::text').get(),
            "mode": "dynamic",
            "status": "success"
        }
        if selector:
            result["data"] = page.css(selector).getall()
        return result
    except Exception as e:
        return {"error": str(e), "url": url}

def main():
    parser = argparse.ArgumentParser(description='Scrapling Web Scraping Tool')
    parser.add_argument('url', help='Target URL to scrape')
    parser.add_argument('--mode', choices=['basic', 'stealth', 'dynamic'], 
                       default='basic', help='Scraping mode')
    parser.add_argument('--selector', '-s', help='CSS selector to extract')
    parser.add_argument('--cloudflare', action='store_true', 
                       help='Solve Cloudflare (stealth mode)')
    parser.add_argument('--wait', help='Wait for selector (dynamic mode)')
    parser.add_argument('--json', '-j', action='store_true',
                       help='Output as JSON')
    
    args = parser.parse_args()
    
    # 根据模式选择抓取方式
    if args.mode == 'basic':
        result = scrape_basic(args.url, args.selector)
    elif args.mode == 'stealth':
        result = scrape_stealth(args.url, args.selector, args.cloudflare)
    elif args.mode == 'dynamic':
        result = scrape_dynamic(args.url, args.selector, args.wait)
    
    # 输出结果
    if args.json:
        print(json.dumps(result, indent=2, ensure_ascii=False))
    else:
        print(f"URL: {result.get('url')}")
        print(f"Title: {result.get('title')}")
        if 'data' in result:
            print(f"\nExtracted Data ({len(result['data'])} items):")
            for i, item in enumerate(result['data'][:5], 1):
                print(f"  {i}. {item[:100]}...")
        if 'error' in result:
            print(f"\n❌ Error: {result['error']}")

if __name__ == '__main__':
    main()


path: skill.json
content: name: Scrapling Web Scraping
description: Zero-bot-detection web scraping for OpenClaw. Bypass Cloudflare, handle JavaScript-heavy sites, and adapt to website changes automatically. Use when you need to scrape protected websites, extract data from dynamic JavaScript SPAs, or bypass anti-bot detection systems. Supports three modes - basic (fast HTTP), stealth (undetectable), dynamic (browser automation).
version: 1.0.0
identifier: scrapling-web-scraping
author: 老二
category: web-scraping

Security scan verdicts — ClawScan: suspicious | static analysis:  | VirusTotal: clean | SkillSpector:  ()