Skill: quincygunter/qui-anycrawl (version 1.0.0)

# AnyCrawl Skill Web scraping, crawling, and search via SkillBoss API Hub — powered by high-performance multi-threaded crawling backends (Firecrawl, Jina AI, Apify, ScrapingDog, and more). ## Setup ### Method 1: Environment variable (Recommended) ```bash export SKILLBOSS_API_KEY="your-api-key" ``` Make it permanent by adding to `~/.bashrc` or `~/.zshrc`: ```bash echo 'export SKILLBOSS_API_KEY="your-api-key"' >> ~/.bashrc source ~/.bashrc ``` Get your API key at: https://heybossai.com ### Method 2: OpenClaw gateway config ```bash openclaw config.patch --set SKILLBOSS_API_KEY="your-api-key" ``` ## requires.env ``` SKILLBOSS_API_KEY ``` ## Functions ### 1. anycrawl_scrape Scrape a single URL and convert to LLM-ready structured data via SkillBoss API Hub (`type: "scraping"`). **Parameters:** - `url` (string, required): URL to scrape - `engine` (string, optional): Scraping engine - `"cheerio"` (default), `"playwright"`, `"puppeteer"` - `formats` (array, optional): Output formats - `["markdown"]`, `["html"]`, `["text"]`, `["json"]`, `["screenshot"]` - `timeout` (number, optional): Timeout in milliseconds (default: 30000) - `wait_for` (number, optional): Delay before extraction in ms (browser engines only) - `wait_for_selector` (string/object/array, optional): Wait for CSS selectors - `include_tags` (array, optional): Include only these HTML tags (e.g., `["h1", "p", "article"]`) - `exclude_tags` (array, optional): Exclude these HTML tags - `proxy` (string, optional): Proxy URL (e.g., `"http://proxy:port"`) - `json_options` (object, optional): JSON extraction with schema/prompt - `extract_source` (string, optional): `"markdown"` (default) or `"html"` **Examples:** ```javascript // Basic scrape with default cheerio anycrawl_scrape({ url: "https://example.com" }) // Scrape SPA with Playwright anycrawl_scrape({ url: "https://spa-example.com", engine: "playwright", formats: ["markdown", "screenshot"] }) // Extract structured JSON anycrawl_scrape({ url: "https://product-page.com", engine: "cheerio", json_options: { schema: { type: "object", properties: { product_name: { type: "string" }, price: { type: "number" }, description: { type: "string" } }, required: ["product_name", "price"] }, user_prompt: "Extract product details from this page" } }) ``` ### 2. anycrawl_search Search Google and return structured results via SkillBoss API Hub (`type: "search"`). **Parameters:** - `query` (string, required): Search query - `engine` (string, optional): Search engine - `"google"` (default) - `limit` (number, optional): Max results per page (default: 10) - `offset` (number, optional): Number of results to skip (default: 0) - `pages` (number, optional): Number of pages to retrieve (default: 1, max: 20) - `lang` (string, optional): Language locale (e.g., `"en"`, `"zh"`, `"vi"`) - `safe_search` (number, optional): 0 (off), 1 (medium), 2 (high) - `scrape_options` (object, optional): Scrape each result URL with these options **Examples:** ```javascript // Basic search anycrawl_search({ query: "OpenAI ChatGPT" }) // Multi-page search in Vietnamese anycrawl_search({ query: "hướng dẫn Node.js", pages: 3, lang: "vi" }) // Search and auto-scrape results anycrawl_search({ query: "best AI tools 2026", limit: 5, scrape_options: { engine: "cheerio", formats: ["markdown"] } }) ``` ### 3. anycrawl_crawl_start Start crawling an entire website (async job) via SkillBoss API Hub (`type: "scraping"` with crawl mode). **Parameters:** - `url` (string, required): Seed URL to start crawling - `engine` (string, optional): `"cheerio"` (default), `"playwright"`, `"puppeteer"` - `strategy` (string, optional): `"all"`, `"same-domain"` (default), `"same-hostname"`, `"same-origin"` - `max_depth` (number, optional): Max depth from seed URL (default: 10) - `limit` (number, optional): Max pages to crawl (default: 100) - `include_paths` (array, optional): Path patterns to include (e.g., `["/blog/*"]`) - `exclude_paths` (array, optional): Path patterns to exclude (e.g., `["/admin/*"]`) - `scrape_paths` (array, optional): Only scrape URLs matching these patterns - `scrape_options` (object, optional): Per-page scrape options **Examples:** ```javascript // Crawl entire website anycrawl_crawl_start({ url: "https://docs.example.com", engine: "cheerio", max_depth: 5, limit: 50 }) // Crawl only blog posts anycrawl_crawl_start({ url: "https://example.com", strategy: "same-domain", include_paths: ["/blog/*"], exclude_paths: ["/blog/tags/*"], scrape_options: { formats: ["markdown"] } }) // Crawl product pages only anycrawl_crawl_start({ url: "https://shop.example.com", strategy: "same-domain", scrape_paths: ["/products/*"], limit: 200 }) ``` ### 4. anycrawl_crawl_status Check crawl job status. **Parameters:** - `job_id` (string, required): Crawl job ID **Example:** ```javascript anycrawl_crawl_status({ job_id: [REDACTED_SECRET] }) ``` ### 5. anycrawl_crawl_results Get crawl results (paginated). **Parameters:** - `job_id` (string, required): Crawl job ID - `skip` (number, optional): Number of results to skip (default: 0) **Example:** ```javascript // Get first 100 results anycrawl_crawl_results({ job_id: "xxx", skip: 0 }) // Get next 100 results anycrawl_crawl_results({ job_id: "xxx", skip: 100 }) ``` ### 6. anycrawl_crawl_cancel Cancel a running crawl job. **Parameters:** - `job_id` (string, required): Crawl job ID ### 7. anycrawl_search_and_scrape Quick helper: Search Google then scrape top results. **Parameters:** - `query` (string, required): Search query - `max_results` (number, optional): Max results to scrape (default: 3) - `scrape_engine` (string, optional): Engine for scraping (default: `"cheerio"`) - `formats` (array, optional): Output formats (default: `["markdown"]`) - `lang` (string, optional): Search language **Example:** ```javascript anycrawl_search_and_scrape({ query: "latest AI news", max_results: 5, formats: ["markdown"] }) ``` ## Engine Selection Guide | Engine | Best For | Speed | JS Rendering | |--------|----------|-------|--------------| | `cheerio` | Static HTML, news, blogs | Fastest | No | | `playwright` | SPAs, complex web apps | Slower | Yes | | `puppeteer` | Chrome-specific, metrics | Slower | Yes | ## Response Format All scraping responses follow this structure (result path: `data.result`): ```json { "data": { "result": { ... } } } ``` All search responses follow this structure (result path: `data.result`): ```json { "data": { "result": [ ... ] } } ``` Error response: ```json { "success": false, "error": "Error type", "message": "Human-readable message" } ``` ## Common Error Codes - `400` - Bad Request (validation errors) - `401` - Unauthorized (invalid API key) - `402` - Payment Required (insufficient credits) - `404` - Not Found - `429` - Rate limit exceeded - `500` - Internal server error ## API Limits - Rate limits apply based on your plan - Crawl jobs expire after 24 hours - Max crawl limit: depends on credits ## Links - API Docs: https://heybossai.com/docs - Website: https://heybossai.com

path: index.js
content: // AnyCrawl Skill - powered by SkillBoss API Hub
// Scrape, Crawl, and Search web content via https://api.heybossai.com/v1/pilot

const [REDACTED_SECRET];
const API_BASE = "https://api.heybossai.com/v1";

async function pilot(body) {
  if (!API_KEY) {
    throw new Error(
      "SKILLBOSS_API_KEY not found!\n\n" +
      "Please set your API key:\n" +
      "export SKILLBOSS_API_KEY=\"your-key\"\n\n" +
      "Get your API key at: https://heybossai.com"
    );
  }

  const r = await fetch(`${API_BASE}/pilot`, {
    method: "POST",
    headers: { "Authorization": `Bearer ${API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(body)
  });

  const data = await r.json();

  if (!r.ok) {
    throw new Error(`SkillBoss API Error: ${data.error || data.message || r.statusText}`);
  }

  return data;
}

/**
 * Scrape a single URL and convert to LLM-ready structured data
 *
 * @param {Object} params
 * @param {string} params.url - URL to scrape (required)
 * @param {string} params.engine - Scraping engine: 'cheerio' (default), 'playwright', 'puppeteer'
 * @param {Array} params.formats - Output formats: ['markdown'], ['html'], ['text'], ['json'], ['screenshot'], etc.
 * @param {number} params.timeout - Timeout in milliseconds (default: 30000)
 * @param {number} params.wait_for - Delay before extraction in ms (browser engines only)
 * @param {string|Object|Array} params.wait_for_selector - Wait for CSS selectors
 * @param {Array} params.include_tags - Include only these HTML tags
 * @param {Array} params.exclude_tags - Exclude these HTML tags
 * @param {string} params.proxy - Proxy URL (http://proxy:port)
 * @param {Object} params.json_options - JSON extraction options with schema and prompt
 * @param {string} params.extract_source - Source for JSON extraction: 'markdown' (default) or 'html'
 *
 * @returns {Promise<Object>} Scraped content in requested formats
 */
export async function anycrawl_scrape({
  url,
  engine = "cheerio",
  formats = ["markdown"],
  timeout,
  wait_for,
  wait_for_selector,
  include_tags,
  exclude_tags,
  proxy,
  json_options,
  extract_source
}) {
  if (!url) {
    throw new Error("URL is required");
  }

  const inputs = { url, engine, formats };

  if (timeout !== undefined) inputs.timeout = timeout;
  if (wait_for !== undefined) inputs.wait_for = wait_for;
  if (wait_for_selector !== undefined) inputs.wait_for_selector = wait_for_selector;
  if (include_tags !== undefined) inputs.include_tags = include_tags;
  if (exclude_tags !== undefined) inputs.exclude_tags = exclude_tags;
  if (proxy !== undefined) inputs.proxy = proxy;
  if (json_options !== undefined) inputs.json_options = json_options;
  if (extract_source !== undefined) inputs.extract_source = extract_source;

  const result = await pilot({ type: "scraping", inputs, prefer: "balanced" });
  return result.result;
}

/**
 * Search Google and return structured results
 *
 * @param {Object} params
 * @param {string} params.query - Search query string (required)
 * @param {string} params.engine - Search engine: 'google' (default)
 * @param {number} params.limit - Max results per page (default: 10)
 * @param {number} params.offset - Number of results to skip (default: 0)
 * @param {number} params.pages - Number of pages to retrieve (default: 1, max: 20)
 * @param {string} params.lang - Language locale (e.g., 'en', 'zh', 'vi', default: 'en')
 * @param {number} params.safe_search - Safe search: 0 (off), 1 (medium), 2 (high)
 * @param {Object} params.scrape_options - Optional: scrape each result URL with these options
 *
 * @returns {Promise<Object>} Search results and suggestions
 */
export async function anycrawl_search({
  query,
  engine = "google",
  limit,
  offset,
  pages,
  lang,
  safe_search,
  scrape_options
}) {
  if (!query) {
    throw new Error("Query is required");
  }

  const inputs = { query };

  if (engine !== undefined) inputs.engine = engine;
  if (limit !== undefined) inputs.limit = limit;
  if (offset !== undefined) inputs.offset = offset;
  if (pages !== undefined) inputs.pages = pages;
  if (lang !== undefined) inputs.lang = lang;
  if (safe_search !== undefined) inputs.safe_search = safe_search;
  if (scrape_options !== undefined) inputs.scrape_options = scrape_options;

  const result = await pilot({ type: "search", inputs, prefer: "balanced" });
  return result.result;
}

/**
 * Start a crawl job for an entire website
 *
 * @param {Object} params
 * @param {string} params.url - Seed URL to start crawling (required)
 * @param {string} params.engine - Scraping engine: 'cheerio' (default), 'playwright', 'puppeteer'
 * @param {string} params.strategy - Crawl scope: 'all', 'same-domain' (default), 'same-hostname', 'same-origin'
 * @param {number} params.max_depth - Max depth from seed URL (default: 10)
 * @param {number} params.limit - Max pages to crawl (default: 100)
 * @param {Array} params.include_paths - Path patterns to include (glob-like)
 * @param {Array} params.exclude_paths - Path patterns to exclude (glob-like)
 * @param {Array} params.scrape_paths - Only scrape URLs matching these patterns
 * @param {Object} params.scrape_options - Per-page scrape options
 *
 * @returns {Promise<Object>} Job ID and status
 */
export async function anycrawl_crawl_start({
  url,
  engine = "cheerio",
  strategy = "same-domain",
  max_depth,
  limit,
  include_paths,
  exclude_paths,
  scrape_paths,
  scrape_options
}) {
  if (!url) {
    throw new Error("URL is required");
  }

  const inputs = { url, engine, strategy, crawl: true };

  if (max_depth !== undefined) inputs.max_depth = max_depth;
  if (limit !== undefined) inputs.limit = limit;
  if (include_paths !== undefined) inputs.include_paths = include_paths;
  if (exclude_paths !== undefined) inputs.exclude_paths = exclude_paths;
  if (scrape_paths !== undefined) inputs.scrape_paths = scrape_paths;
  if (scrape_options !== undefined) inputs.scrape_options = scrape_options;

  const result = await pilot({ type: "scraping", inputs, prefer: "balanced" });
  return result.result;
}

/**
 * Get crawl job status
 *
 * @param {Object} params
 * @param {string} params.job_id - Crawl job ID (required)
 *
 * @returns {Promise<Object>} Job status, progress, and stats
 */
export async function anycrawl_crawl_status({ job_id }) {
  if (!job_id) {
    throw new Error("Job ID is required");
  }

  const result = await pilot({ type: "scraping", inputs: { job_id, action: "status" }, prefer: "balanced" });
  return result.result;
}

/**
 * Get crawl results (paginated)
 *
 * @param {Object} params
 * @param {string} params.job_id - Crawl job ID (required)
 * @param {number} params.skip - Number of results to skip (default: 0)
 *
 * @returns {Promise<Object>} Crawled pages with content
 */
export async function anycrawl_crawl_results({ job_id, skip = 0 }) {
  if (!job_id) {
    throw new Error("Job ID is required");
  }

  const result = await pilot({ type: "scraping", inputs: { job_id, action: "results", skip }, prefer: "balanced" });
  return result.result;
}

/**
 * Cancel a crawl job
 *
 * @param {Object} params
 * @param {string} params.job_id - Crawl job ID (required)
 *
 * @returns {Promise<Object>} Cancellation confirmation
 */
export async function anycrawl_crawl_cancel({ job_id }) {
  if (!job_id) {
    throw new Error("Job ID is required");
  }

  const result = await pilot({ type: "scraping", inputs: { job_id, action: "cancel" }, prefer: "balanced" });
  return result.result;
}

/**
 * Quick search and scrape - Search Google then scrape top results
 *
 * @param {Object} params
 * @param {string} params.query - Search query (required)
 * @param {number} params.max_results - Max results to scrape (default: 3)
 * @param {string} params.scrape_engine - Engine for scraping: 'cheerio' (default) or 'playwright'
 * @param {Array} params.formats - Output formats for scraped content
 * @param {string} params.lang - Search language
 *
 * @returns {Promise<Object>} Search results with scraped content
 */
export async function anycrawl_search_and_scrape({
  query,
  max_results = 3,
  scrape_engine = "cheerio",
  formats = ["markdown"],
  lang
}) {
  if (!query) {
    throw new Error("Query is required");
  }

  // First, search
  const searchInputs = { query, limit: max_results };
  if (lang) searchInputs.lang = lang;

  const searchResult = await pilot({ type: "search", inputs: searchInputs, prefer: "balanced" });
  const searchData = searchResult.result;

  if (!searchData) {
    return searchResult.data;
  }

  // Filter only web results (not suggestions)
  const webResults = Array.isArray(searchData)
    ? searchData.filter(r => r.url && r.category === "web")
    : [];

  // Scrape top results
  const scrapedResults = [];
  for (const result of webResults.slice(0, max_results)) {
    try {
      const scrapeResult = await pilot({
        type: "scraping",
        inputs: { url: result.url, engine: scrape_engine, formats },
        prefer: "balanced"
      });

      scrapedResults.push({
        ...result,
        scraped: scrapeResult.result,
        scrape_error: null
      });
    } catch (error) {
      scrapedResults.push({
        ...result,
        scraped: null,
        scrape_error: error.message
      });
    }
  }

  return {
    success: true,
    query,
    search_results: searchData,
    scraped_results: scrapedResults
  };
}

// Export all functions for OpenClaw
export default {
  anycrawl_scrape,
  anycrawl_search,
  anycrawl_crawl_start,
  anycrawl_crawl_status,
  anycrawl_crawl_results,
  anycrawl_crawl_cancel,
  anycrawl_search_and_scrape
};


path: package.json
content: {
  "name": "anycrawl",
  "version": "1.0.0",
  "description": "AnyCrawl API integration for OpenClaw - Scrape, Crawl, and Search web content",
  "author": "techlaai",
  "license": "MIT",
  "keywords": ["anycrawl", "scrape", "crawl", "search", "web", "llm"],
  "main": "index.js",
  "type": "module",
  "engines": {
    "node": ">=18"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/techlaai/anycrawl-skill"
  },
  "dependencies": {}
}


path: README.md
content: # AnyCrawl

High-performance web scraping, crawling, and search via SkillBoss API Hub.

## Features

- **Google Search** - Structured SERP results
- **Single Page Scrape** - Convert any URL to markdown/html/json
- **Full Site Crawl** - Async crawling with depth control
- **Multi-engine** - Cheerio (fast), Playwright/Puppeteer (JS rendering)

## Quick Start

```bash
# 1. Set API key
export SKILLBOSS_API_KEY="your-api-key"

# 2. Use it!
anycrawl_search({ query: "AI news", limit: 5 })
anycrawl_scrape({ url: "https://example.com" })
```

Or add to `~/.bashrc` to make it permanent:
```bash
echo 'export SKILLBOSS_API_KEY="your-api-key"' >> ~/.bashrc
source ~/.bashrc
```

## API Functions

| Function | Description |
|----------|-------------|
| `anycrawl_search` | Google search with structured results |
| `anycrawl_scrape` | Scrape single URL to markdown/html |
| `anycrawl_crawl_start` | Start async website crawl |
| `anycrawl_crawl_status` | Check crawl progress |
| `anycrawl_crawl_results` | Get crawled pages |
| `anycrawl_search_and_scrape` | Search + auto-scrape top results |

## Docs

Full documentation in [SKILL.md](./SKILL.md)


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)