Skill: moochmaniac/moochmaniac-firecrawl-scraper (version 1.0.0)

--- name: firecrawl-scraper description: Web scraping, crawling, and search via Firecrawl API. Converts web pages to clean markdown/JSON optimized for AI consumption. Use when you need to extract content from websites, crawl entire sites, or search and scrape web results. Handles JavaScript-heavy sites, dynamic content, and provides structured output. 96% web coverage including modern SPAs. --- # Firecrawl Scraper Professional web scraping powered by Firecrawl API. Converts websites to clean, AI-ready markdown or structured JSON. ## When to Use - Extract content from web pages for analysis - Scrape documentation sites or knowledge bases - Crawl entire websites systematically - Search the web and get scraped content - Parse JavaScript-heavy or dynamic sites - Convert HTML to clean markdown for LLM processing - Competitive research or content aggregation ## Quick Start **Scrape a single page:** ```bash python3 scripts/scrape.py https://example.com ``` **Crawl a website:** ```bash python3 scripts/scrape.py --crawl https://docs.example.com --depth 2 --limit 10 ``` **Search and scrape:** ```bash python3 scripts/scrape.py --search "AI agent frameworks" --limit 5 ``` **Check crawl status:** ```bash python3 scripts/scrape.py --crawl-status abc123 ``` ## Commands ### Scrape (Single Page) Extract content from a single URL: ```bash python3 scripts/scrape.py <url> [options] ``` **Options:** - `--formats markdown,html,screenshot` — Output formats (default: markdown) - `--full` — Include full page (no main content extraction) - `--json` — Output raw JSON response **Examples:** ```bash # Basic scrape python3 scripts/scrape.py https://docs.example.com # Get HTML and markdown python3 scripts/scrape.py https://site.com --formats markdown,html # Full page (no content filtering) python3 scripts/scrape.py https://site.com --full # JSON output python3 scripts/scrape.py https://site.com --json ``` ### Crawl (Entire Website) Systematically crawl and scrape multiple pages: ```bash python3 scripts/scrape.py --crawl <url> [options] ``` **Options:** - `--depth N` — Maximum crawl depth (default: 2) - `--limit N` — Maximum pages to crawl (default: 10) - `--json` — Output raw JSON response **Examples:** ```bash # Basic crawl python3 scripts/scrape.py --crawl https://docs.site.com # Deep crawl with limit python3 scripts/scrape.py --crawl https://blog.com --depth 3 --limit 50 # Shallow crawl python3 scripts/scrape.py --crawl https://site.com --depth 1 --limit 5 ``` **Note:** Crawl returns a job ID. Use `--crawl-status` to check progress and retrieve results. ### Search (Web Search + Scrape) Search the web and get scraped content from results: ```bash python3 scripts/scrape.py --search <query> [options] ``` **Options:** - `--limit N` — Number of results (default: 5) - `--json` — Output raw JSON response **Examples:** ```bash # Search and scrape python3 scripts/scrape.py --search "WordPress security best practices" # More results python3 scripts/scrape.py --search "AI agents 2026" --limit 10 # JSON output python3 scripts/scrape.py --search "casino bonuses" --json ``` ### Crawl Status Check status of a crawl job: ```bash python3 scripts/scrape.py --crawl-status <job-id> ``` Returns JSON with: - Status: `scraping`, `completed`, `failed` - Progress: Pages scraped - Data: Scraped content (when completed) - Credits used ## Output Formats **Markdown** (default): Clean, LLM-ready text with preserved structure - Headings, links, lists, code blocks maintained - No HTML noise or styling artifacts - Perfect for RAG, summarization, analysis **HTML**: Full HTML source (useful for parsing specific elements) **Screenshot**: Base64-encoded PNG of rendered page **JSON**: Structured data extraction (custom schemas supported) ## Features **Smart Content Extraction:** - Automatically identifies main content - Removes navigation, ads, footers - Preserves document structure **JavaScript Support:** - Handles SPAs (React, Vue, Angular) - Waits for dynamic content to load - 96% web coverage **Anti-Bot Handling:** - Proxy management built-in - Rate limiting handled automatically - CAPTCHA avoidance **Caching:** - Smart caching reduces credits - Configurable cache behavior ## API Key Setup The script looks for the Firecrawl API key in: 1. [REDACTED_SECRET] (OpenClaw workspace) 2. `secrets/firecrawl_api_key` (relative to current directory) 3. `FIRECRAWL_API_KEY` environment variable Current key is stored at: [REDACTED_SECRET] ## Credits & Pricing - **Scrape**: 1 credit per page - **Crawl**: 1 credit per page crawled - **Search**: 1 credit per result scraped - **Screenshot**: +1 credit - **Advanced features**: May use additional credits Free tier: 500 credits Paid plans: Starting at $16/month (3,000 credits) ## Use Cases **Documentation Extraction:** ```bash python3 scripts/scrape.py --crawl https://docs.framework.com --depth 2 --limit 50 ``` **Competitive Research:** ```bash python3 scripts/scrape.py --search "top casino affiliate sites" --limit 10 ``` **Content Migration:** ```bash python3 scripts/scrape.py https://old-site.com/page1 --formats markdown ``` **News Monitoring:** ```bash python3 scripts/scrape.py --search "WordPress security updates" --limit 5 ``` **Blog Scraping:** ```bash python3 scripts/scrape.py --crawl https://blog.site.com --depth 1 --limit 20 ``` ## Tips - Start with low `--limit` values to test - Use `--depth 1` for blog homepages (gets all posts) - `--depth 2-3` for documentation sites - `--search` is faster than manual crawling for research - Check `--crawl-status` regularly for long crawls - Use `--json` for programmatic processing - Markdown format is best for LLM consumption ## Comparison to Other Tools **vs web_fetch tool:** - Firecrawl: Better JS support, cleaner output, handles complex sites - web_fetch: Faster, simpler, no API credits needed - Use Firecrawl for: Modern sites, heavy JS, need high-quality markdown - Use web_fetch for: Simple pages, quick checks, no credit usage **vs browser tool:** - Firecrawl: Optimized for scraping, structured output, no browser management - browser: Full control, visual interaction, debugging - Use Firecrawl for: Content extraction at scale - Use browser for: Interactive tasks, testing, visual verification

path: scripts/scrape.py
content: #!/usr/bin/env python3
"""
Firecrawl API wrapper for OpenClaw
Scrape, crawl, and search web content with AI-optimized output
"""

import os
import sys
import json
import requests
import time
from typing import Optional, Dict, Any, List

def get_api_key() -> Optional[str]:
    """Get Firecrawl API key from secrets or environment"""
    possible_paths = [
        "workspace/secrets/firecrawl_api_key",
        "secrets/firecrawl_api_key",
        os.path.expanduser("~/.openclaw/workspace/workspace/secrets/firecrawl_api_key")
    ]
    
    for path in possible_paths:
        if os.path.exists(path):
            with open(path) as f:
                return f.read().strip()
    
    return os.getenv("FIRECRAWL_API_KEY")


def scrape_url(
    url: str,
    formats: List[str] = ["markdown"],
    include_tags: Optional[List[str]] = None,
    exclude_tags: Optional[List[str]] = None,
    only_main_content: bool = True,
    wait_for: Optional[int] = None,
    timeout: int = 30000
) -> Dict[str, Any]:
    """
    Scrape a single URL
    
    Args:
        url: URL to scrape
        formats: Output formats - ["markdown"], ["html"], ["markdown", "html"], ["screenshot"]
        include_tags: HTML tags to include (e.g., ["article", "main"])
        exclude_tags: HTML tags to exclude (e.g., ["nav", "footer"])
        only_main_content: Extract only main content (default True)
        wait_for: Milliseconds to wait before scraping
        timeout: Request timeout in milliseconds
    
    Returns:
        dict: Scrape results with markdown, html, screenshot, metadata
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"error": "Firecrawl API key not found"}
    
    payload = {
        "url": url,
        "formats": formats,
        "onlyMainContent": only_main_content,
        "timeout": timeout
    }
    
    if include_tags:
        payload["includeTags"] = include_tags
    if exclude_tags:
        payload["excludeTags"] = exclude_tags
    if wait_for:
        payload["waitFor"] = wait_for
    
    try:
        response = requests.post(
            "https://api.firecrawl.dev/v1/scrape",
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {api_key}"
            },
            json=payload,
            timeout=60
        )
        response.raise_for_status()
        return response.json()
    
    except requests.exceptions.RequestException as e:
        return {"error": f"Scrape failed: {str(e)}"}


def crawl_website(
    url: str,
    max_depth: int = 2,
    limit: int = 10,
    formats: List[str] = ["markdown"],
    exclude_paths: Optional[List[str]] = None,
    include_paths: Optional[List[str]] = None,
    ignore_sitemap: bool = False
) -> Dict[str, Any]:
    """
    Crawl an entire website
    
    Args:
        url: Base URL to crawl
        max_depth: Maximum crawl depth (default 2)
        limit: Maximum pages to crawl (default 10)
        formats: Output formats
        exclude_paths: URL patterns to exclude
        include_paths: URL patterns to include
        ignore_sitemap: Ignore sitemap.xml
    
    Returns:
        dict: Crawl job ID and status URL
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"error": "Firecrawl API key not found"}
    
    payload = {
        "url": url,
        "maxDepth": max_depth,
        "limit": limit,
        "scrapeOptions": {
            "formats": formats
        },
        "ignoreSitemap": ignore_sitemap
    }
    
    if exclude_paths:
        payload["excludePaths"] = exclude_paths
    if include_paths:
        payload["includePaths"] = include_paths
    
    try:
        response = requests.post(
            "https://api.firecrawl.dev/v1/crawl",
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {api_key}"
            },
            json=payload,
            timeout=60
        )
        response.raise_for_status()
        return response.json()
    
    except requests.exceptions.RequestException as e:
        return {"error": f"Crawl failed: {str(e)}"}


def get_crawl_status(crawl_id: str) -> Dict[str, Any]:
    """Get status of crawl job"""
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"error": "Firecrawl API key not found"}
    
    try:
        response = requests.get(
            f"https://api.firecrawl.dev/v1/crawl/{crawl_id}",
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=30
        )
        response.raise_for_status()
        return response.json()
    
    except requests.exceptions.RequestException as e:
        return {"error": f"Status check failed: {str(e)}"}


def search_web(
    query: str,
    limit: int = 5,
    formats: List[str] = ["markdown"]
) -> Dict[str, Any]:
    """
    Search the web and scrape results
    
    Args:
        query: Search query
        limit: Number of results (default 5)
        formats: Output formats for scraped pages
    
    Returns:
        dict: Search results with scraped content
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"error": "Firecrawl API key not found"}
    
    payload = {
        "query": query,
        "limit": limit,
        "scrapeOptions": {
            "formats": formats
        }
    }
    
    try:
        response = requests.post(
            "https://api.firecrawl.dev/v1/search",
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {api_key}"
            },
            json=payload,
            timeout=60
        )
        response.raise_for_status()
        return response.json()
    
    except requests.exceptions.RequestException as e:
        return {"error": f"Search failed: {str(e)}"}


def format_scrape_output(data: Dict[str, Any]) -> str:
    """Format scrape output for display"""
    if "error" in data:
        return f"Error: {data['error']}"
    
    if not data.get("success"):
        return f"Scrape failed: {data.get('error', 'Unknown error')}"
    
    result_data = data.get("data", {})
    output = []
    
    # Metadata
    metadata = result_data.get("metadata", {})
    output.append(f"=== {metadata.get('title', 'Untitled')} ===")
    output.append(f"URL: {result_data.get('url', 'N/A')}")
    output.append(f"Status: {metadata.get('statusCode', 'N/A')}")
    output.append(f"Credits used: {data.get('creditsUsed', 0)}")
    output.append("")
    
    # Content
    if "markdown" in result_data:
        output.append("=== Markdown Content ===")
        markdown = result_data["markdown"]
        # Truncate if too long
        if len(markdown) > 2000:
            output.append(markdown[:2000])
            output.append(f"\n... (truncated, {len(markdown)} total chars)")
        else:
            output.append(markdown)
    
    return "\n".join(output)


def format_crawl_output(data: Dict[str, Any]) -> str:
    """Format crawl output for display"""
    if "error" in data:
        return f"Error: {data['error']}"
    
    if not data.get("success"):
        return f"Crawl failed: {data.get('error', 'Unknown error')}"
    
    output = []
    output.append("=== Crawl Job Started ===")
    output.append(f"Job ID: {data.get('id', 'N/A')}")
    output.append(f"Status URL: {data.get('url', 'N/A')}")
    output.append("")
    output.append("Check status with:")
    output.append(f"  python3 scripts/scrape.py --crawl-status {data.get('id', 'N/A')}")
    
    return "\n".join(output)


def format_search_output(data: Dict[str, Any]) -> str:
    """Format search output for display"""
    if "error" in data:
        return f"Error: {data['error']}"
    
    if not data.get("success"):
        return f"Search failed: {data.get('error', 'Unknown error')}"
    
    results = data.get("data", [])
    output = []
    output.append(f"=== Search Results ({len(results)}) ===")
    output.append("")
    
    for i, result in enumerate(results, 1):
        output.append(f"[{i}] {result.get('title', 'Untitled')}")
        output.append(f"    URL: {result.get('url', 'N/A')}")
        
        markdown = result.get("markdown", "")
        if markdown:
            preview = markdown[:200]
            if len(markdown) > 200:
                preview += "..."
            output.append(f"    {preview}")
        
        output.append("")
    
    return "\n".join(output)


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Firecrawl Scraper - Web scraping to clean markdown/JSON")
        print("\nUsage:")
        print("  Scrape: scrape.py <url> [--formats markdown,html] [--main-only]")
        print("  Crawl:  scrape.py --crawl <url> [--depth 2] [--limit 10]")
        print("  Search: scrape.py --search <query> [--limit 5]")
        print("  Status: scrape.py --crawl-status <job-id>")
        print("\nExamples:")
        print("  scrape.py https://example.com")
        print("  scrape.py https://docs.site.com --formats markdown")
        print("  scrape.py --crawl https://example.com --depth 3 --limit 20")
        print("  scrape.py --search 'AI agent frameworks' --limit 5")
        print("  scrape.py --crawl-status abc123")
        sys.exit(1)
    
    # Parse arguments
    if sys.argv[1] == "--crawl":
        # Crawl mode
        if len(sys.argv) < 3:
            print("Error: --crawl requires a URL")
            sys.exit(1)
        
        url = sys.argv[2]
        max_depth = 2
        limit = 10
        
        i = 3
        while i < len(sys.argv):
            if sys.argv[i] == "--depth" and i + 1 < len(sys.argv):
                max_depth = int(sys.argv[i + 1])
                i += 2
            elif sys.argv[i] == "--limit" and i + 1 < len(sys.argv):
                limit = int(sys.argv[i + 1])
                i += 2
            else:
                i += 1
        
        result = crawl_website(url, max_depth=max_depth, limit=limit)
        if "--json" in sys.argv:
            print(json.dumps(result, indent=2))
        else:
            print(format_crawl_output(result))
    
    elif sys.argv[1] == "--search":
        # Search mode
        if len(sys.argv) < 3:
            print("Error: --search requires a query")
            sys.exit(1)
        
        query = sys.argv[2]
        limit = 5
        
        if "--limit" in sys.argv:
            idx = sys.argv.index("--limit")
            if idx + 1 < len(sys.argv):
                limit = int(sys.argv[idx + 1])
        
        result = search_web(query, limit=limit)
        if "--json" in sys.argv:
            print(json.dumps(result, indent=2))
        else:
            print(format_search_output(result))
    
    elif sys.argv[1] == "--crawl-status":
        # Status check
        if len(sys.argv) < 3:
            print("Error: --crawl-status requires a job ID")
            sys.exit(1)
        
        job_id = sys.argv[2]
        result = get_crawl_status(job_id)
        print(json.dumps(result, indent=2))
    
    else:
        # Scrape mode
        url = sys.argv[1]
        formats = ["markdown"]
        only_main = True
        
        if "--formats" in sys.argv:
            idx = sys.argv.index("--formats")
            if idx + 1 < len(sys.argv):
                formats = sys.argv[idx + 1].split(",")
        
        if "--full" in sys.argv:
            only_main = False
        
        result = scrape_url(url, formats=formats, only_main_content=only_main)
        
        if "--json" in sys.argv:
            print(json.dumps(result, indent=2))
        else:
            print(format_scrape_output(result))


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)