Skill: kirkraman/kirk-scrape (version 1.0.0)

--- --- name: Scrape name: Scrape description: Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via SkillBoss API Hub. requires_env: [SKILLBOSS_API_KEY] --- ## Pre-Scrape Compliance Checklist Before writing any scraping code: 1. **robots.txt** — Fetch `{domain}/robots.txt`, check if target path is disallowed. If yes, stop. 2. **Terms of Service** — Check `/terms`, `/tos`, `/legal`. Explicit scraping prohibition = need permission. 3. **Data type** — Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA. 4. **Authentication** — Data behind login is off-limits without authorization. Never scrape protected content. 5. **API available?** — If site offers an API, use it. Always. Scraping when API exists often violates ToS. ## Legal Boundaries - **Public data, no login** — Generally legal (hiQ v. LinkedIn 2022) - **Bypassing barriers** — CFAA violation risk (Van Buren v. US 2021) - **Ignoring robots.txt** — Gray area, often breaches ToS (Meta v. Bright Data 2024) - **Personal data without consent** — GDPR/CCPA violation - **Republishing copyrighted content** — Copyright infringement ## Request Discipline - **Rate limit**: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure. - **User-Agent**: Real browser string + contact email: `Mozilla/5.0 ... (contact: [REDACTED_SECRET])` - **Respect 429**: Exponential backoff. Ignoring 429s shows intent to harm. - **Session reuse**: Keep connections open to reduce server load. ## Data Handling - **Strip PII immediately** — Don't collect names, emails, phones unless legally justified. - **No fingerprinting** — Don't combine data to identify individuals indirectly. - **Minimize storage** — Cache only what you need, delete what you don't. - **Audit trail** — Log what, when, where. Evidence of good faith if challenged. ## SkillBoss API Hub Scraping For managed content extraction with automatic anti-bot handling, use SkillBoss API Hub's scraping capability (`type: "scraper"`). Authenticate with `SKILLBOSS_API_KEY` and call `https://api.skillbossai.com/v1/pilot`. Result is returned at `result.data.markdown`. For code patterns, robots.txt parser, and SkillBoss API Hub scraping integration, see `code.md`

path: README.md
content: # scrape - OpenClaw Skill

## Installation

```bash
openclaw skill install scrape
```

> 📖 **Complete setup guide**: https://skillboss.co/skill.md

## Usage
See SKILL.md for details.

## License
MIT


path: code.md
content: # Scrape — Code Patterns

## robots.txt Check (Do First)

```python
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

def can_scrape(url: str, user_agent: str = "*") -> bool:
    parsed = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"

    rp = RobotFileParser()
    rp.set_url(robots_url)
    try:
        rp.read()
    except Exception:
        return True  # No robots.txt = allowed
    return rp.can_fetch(user_agent, url)
```

## Session Setup

```python
import requests

def create_session(contact_email: str) -> requests.Session:
    session = requests.Session()
    session.headers.update({
        "User-Agent": f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (contact: {contact_email})",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    })
    return session
```

## Rate-Limited Fetcher

```python
import time
import random
import logging

logger = logging.getLogger(__name__)

def fetch_politely(session, url, min_delay=2.0, max_retries=5):
    """Fetch with rate limiting, backoff, and audit logging."""

    for attempt in range(max_retries):
        # Polite delay with jitter
        delay = min_delay + random.uniform(0, 0.5)
        time.sleep(delay)

        response = session.get(url)

        # Audit trail
        logger.info(f"SCRAPE url={url} status={response.status_code}")

        # Check rate limit headers
        remaining = response.headers.get("X-RateLimit-Remaining")
        if remaining and int(remaining) < 5:
            logger.warning(f"Rate limit low: {remaining} remaining")
            time.sleep(10)  # Proactive slowdown

        # Handle 429
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After", 60)
            wait = int(retry_after) if str(retry_after).isdigit() else 60
            logger.warning(f"429 received, waiting {wait}s")
            time.sleep(wait)
            continue

        # Success or client error (don't retry 4xx except 429)
        if response.status_code < 500:
            return response

        # Server error: exponential backoff
        wait = min(2 ** attempt + random.uniform(0, 1), 60)
        logger.warning(f"5xx error, retry in {wait:.1f}s")
        time.sleep(wait)

    raise Exception(f"Failed after {max_retries} retries: {url}")
```

## Full Example (Direct HTTP)

```python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s"
)

contact = "[REDACTED_SECRET]"
target = "https://example.com/products"

# 1. Check robots.txt
if not can_scrape(target):
    raise SystemExit("Blocked by robots.txt")

# 2. Create session
session = create_session(contact)

# 3. Fetch politely
response = fetch_politely(session, target)

# 4. Process (your logic here)
print(f"Got {len(response.text)} bytes")
```

## SkillBoss API Hub — Managed Scraping

Use SkillBoss API Hub (`type: "scraping"`) for managed content extraction with automatic anti-bot handling, JavaScript rendering, and structured output. Replaces services like Firecrawl, Jina AI, Apify, ScrapingDog.

```python
import requests, os

SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]
API_BASE = "https://api.heybossai.com/v1"

def pilot(body: dict) -> dict:
    r = requests.post(
        f"{API_BASE}/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json=body,
        timeout=60,
    )
    return r.json()

# Scrape a URL via SkillBoss API Hub
result = pilot({"type": "scraper", "inputs": {"url": "https://example.com/products"}})
content = result["result"]["data"]["markdown"]
print(content)
```

## SkillBoss API Hub — Scraping + LLM Analysis (Chain)

Combine scraping with LLM analysis in a single call via SkillBoss API Hub:

```python
# 1. Scrape the page
scrape_result = pilot({"type": "scraper", "inputs": {"url": "https://example.com/products"}})
page_content = scrape_result["result"]["data"]["markdown"]

# 2. Analyze with LLM via /v1/pilot
analysis_result = pilot({
    "type": "chat",
    "inputs": {
        "messages": [
            {"role": "user", "content": f"Extract all product names and prices from this page:\n\n{page_content}"}
        ]
    },
    "prefer": "balanced"
})
extracted = analysis_result["data"]["result"]["choices"][0]["message"]["content"]
print(extracted)
```


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)