Skill: yash-kavaiya/crawlee (version 1.0.0)

--- name: crawlee description: > Expert guide for building web scrapers and crawlers using Crawlee (JavaScript/TypeScript and Python). Use this skill whenever the user wants to: scrape a website, build a web crawler, extract data from web pages, automate browser navigation, handle anti-bot blocking, manage proxies or sessions for scraping, use Playwright/Puppeteer/Cheerio/BeautifulSoup for web data extraction, crawl sitemaps, download files from URLs, or deploy a scraper to the cloud. Trigger even for loosely related phrases like "get data from a website", "automate browser", "scrape prices", "extract links", "crawl URLs", or "bypass bot detection". Covers CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler, HttpCrawler, JSDOMCrawler (JS), and BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler (Python). --- # Crawlee Skill Crawlee is a production-grade web scraping and browser automation library for **JavaScript/TypeScript** (Node.js 16+) and **Python** (3.10+). It handles anti-blocking, proxies, session management, storage, and concurrency out of the box. > **Docs**: https://crawlee.dev/js/docs | https://crawlee.dev/python/docs > **GitHub**: https://github.com/apify/crawlee --- ## 1. Choose Your Crawler ### JavaScript / TypeScript | Crawler | When to Use | JS Required | |---|---|---| | `CheerioCrawler` | Fast HTML parsing, no JS rendering needed | ❌ | | `HttpCrawler` | Raw HTTP responses, custom parsing | ❌ | | `JSDOMCrawler` | DOM manipulation without full browser | ❌ | | `PlaywrightCrawler` | Modern headless browser (Chromium/Firefox/WebKit) | ✅ | | `PuppeteerCrawler` | Chromium/Chrome headless automation | ✅ | | `AdaptivePlaywrightCrawler` | Auto-detects if JS rendering is needed | Auto | | `BasicCrawler` | Custom HTTP logic from scratch | ❌ | **Rule of thumb**: Start with `CheerioCrawler`. Upgrade to `PlaywrightCrawler` only when JS rendering is required. ### Python | Crawler | When to Use | |---|---| | `BeautifulSoupCrawler` | HTML parsing with BeautifulSoup (fast, no JS) | | `ParselCrawler` | CSS/XPath selectors, Scrapy-style (fast, no JS) | | `PlaywrightCrawler` | Full browser automation (Chromium/Firefox/WebKit) | | `AdaptivePlaywrightCrawler` | Auto HTTP vs browser decision | --- ## 2. Installation ### JavaScript ```bash # Recommended: use the CLI npx crawlee create my-crawler cd my-crawler && npm install # Or manually: npm install crawlee # For Playwright: npm install crawlee playwright npx playwright install # For Puppeteer: npm install crawlee puppeteer ``` Add to `package.json`: ```json { "type": "module" } ``` ### Python ```bash pip install crawlee # With BeautifulSoup: pip install 'crawlee[beautifulsoup]' # With Playwright: pip install 'crawlee[playwright]' playwright install ``` --- ## 3. Core Concepts ### The Two Questions Every Crawler Answers 1. **Where to go?** → `Request` objects in a `RequestQueue` 2. **What to do there?** → `requestHandler` function (JS) / decorated handler (Python) ### Key Classes (JS) - `Request` — A single URL + metadata to crawl - `RequestQueue` — Dynamic, deduplicated queue of URLs - `Dataset` — Append-only structured result storage (like a table) - `KeyValueStore` — Blob storage for screenshots, PDFs, state - `ProxyConfiguration` — Manages proxy rotation - `SessionPool` — Manages browser sessions + cookies --- ## 4. Quick Start Examples ### JavaScript — CheerioCrawler (Recommended Start) ```javascript import { CheerioCrawler, Dataset } from 'crawlee'; const crawler = new CheerioCrawler({ async requestHandler({ $, request, enqueueLinks, log }) { const title = $('title').text(); log.info(`Title of ${request.loadedUrl}: ${title}`); await Dataset.pushData({ url: request.loadedUrl, title }); // Enqueue all links found on this page await enqueueLinks(); }, maxRequestsPerCrawl: 100, // Safety limit }); await crawler.run(['https://example.com']); ``` ### JavaScript — PlaywrightCrawler ```javascript import { PlaywrightCrawler, Dataset } from 'crawlee'; const crawler = new PlaywrightCrawler({ // headless: false, // Uncomment to see the browser async requestHandler({ page, request, enqueueLinks, log }) { const title = await page.title(); log.info(`${request.loadedUrl}: ${title}`); await Dataset.pushData({ url: request.loadedUrl, title }); await enqueueLinks(); }, }); await crawler.run(['https://example.com']); ``` ### Python — BeautifulSoupCrawler ```python import asyncio from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext async def main() -> None: crawler = BeautifulSoupCrawler(max_requests_per_crawl=50) @crawler.router.default_handler async def handler(context: BeautifulSoupCrawlingContext) -> None: title = context.soup.title.string if context.soup.title else None context.log.info(f'Processing {context.request.url}: {title}') await context.push_data({'url': context.request.url, 'title': title}) await context.enqueue_links() await crawler.run(['https://example.com']) if __name__ == '__main__': asyncio.run(main()) ``` ### Python — PlaywrightCrawler ```python import asyncio from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext async def main() -> None: crawler = PlaywrightCrawler(headless=True, browser_type='chromium') @crawler.router.default_handler async def handler(context: PlaywrightCrawlingContext) -> None: title = await context.page.title() await context.push_data({'url': context.request.url, 'title': title}) await context.enqueue_links() await crawler.run(['https://example.com']) if __name__ == '__main__': asyncio.run(main()) ``` --- ## 5. Routing — Handling Multiple Page Types Use labels + router to handle different kinds of pages (list pages, detail pages, etc.). ### JavaScript ```javascript import { PlaywrightCrawler, Dataset } from 'crawlee'; import { router } from './routes.js'; const crawler = new PlaywrightCrawler({ requestHandler: router }); await crawler.run([{ url: 'https://shop.example.com', label: 'START' }]); ``` ```javascript // routes.js import { createPlaywrightRouter } from 'crawlee'; export const router = createPlaywrightRouter(); router.addHandler('START', async ({ page, enqueueLinks }) => { await enqueueLinks({ selector: 'a.category', label: 'CATEGORY' }); }); router.addHandler('CATEGORY', async ({ page, enqueueLinks }) => { await enqueueLinks({ selector: 'a.product', label: 'DETAIL' }); // Enqueue next page const next = await page.$('a.next-page'); if (next) await enqueueLinks({ selector: 'a.next-page', label: 'CATEGORY' }); }); router.addDefaultHandler(async ({ page, request, pushData }) => { // DETAIL pages const title = await page.title(); const price = await page.$eval('.price', el => el.textContent); await pushData({ url: request.url, title, price }); }); ``` ### Python ```python from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext crawler = BeautifulSoupCrawler() @crawler.router.handler('CATEGORY') async def category_handler(context: BeautifulSoupCrawlingContext) -> None: await context.enqueue_links(selector='a.product', label='DETAIL') @crawler.router.default_handler async def detail_handler(context: BeautifulSoupCrawlingContext) -> None: title = context.soup.title.string await context.push_data({'url': context.request.url, 'title': title}) ``` --- ## 6. Enqueuing Links ### JavaScript — `enqueueLinks()` ```javascript // Enqueue all links on page await enqueueLinks(); // Filter by glob pattern await enqueueLinks({ globs: ['https://example.com/products/**'] }); // Filter by regex await enqueueLinks({ regexps: [/\/product\/\d+/] }); // Enqueue only specific selector await enqueueLinks({ selector: 'a.pagination', label: 'LIST' }); // Enqueue with custom label and transformations await enqueueLinks({ selector: 'a.item', label: 'DETAIL', transformRequestFunction: (req) => { req.userData.scrapedAt = new Date().toISOString(); return req; }, }); ``` ### Python ```python await context.enqueue_links() await context.enqueue_links(selector='a.product', label='DETAIL') await context.enqueue_links(include=[re.compile(r'/products/\d+')]) ``` --- ## 7. Storage ### Dataset (structured results) ```javascript // JS — Write await Dataset.pushData({ url, title, price }); await Dataset.pushData([item1, item2, item3]); // batch write // JS — Read / Export const dataset = await Dataset.open(); await dataset.exportToCSV('results'); // saves to KV store await dataset.exportToJSON('results'); for await (const item of dataset) { console.log(item); } ``` ```python # Python — Write await context.push_data({'url': url, 'title': title}) # Python — Read / Export from crawlee.storages import Dataset dataset = await Dataset.open() await dataset.export_to(key='results', content_type='csv') ``` Data is saved to `./storage/datasets/default/*.json` by default. ### KeyValueStore (blobs, screenshots, state) ```javascript // JS await KeyValueStore.setValue('OUTPUT', { results: [...] }); const value = await KeyValueStore.getValue('OUTPUT'); // Save a screenshot const store = await KeyValueStore.open(); await store.setValue('screenshot', await page.screenshot(), { contentType: 'image/png' }); ``` ```python # Python from crawlee.storages import KeyValueStore kvs = await KeyValueStore.open() await kvs.set_value('result', {'data': 'value'}) value = await kvs.get_value('result') ``` ### Storage location ``` ./storage/ datasets/default/ # Dataset rows as JSON files key_value_stores/default/ # KV store entries request_queues/default/ # Request queue state ``` Override with env var: [REDACTED_SECRET] --- ## 8. Proxy Management ```javascript // JS — Basic proxy rotation import { ProxyConfiguration } from 'crawlee'; const proxyConfiguration = new ProxyConfiguration({ proxyUrls: [ 'http://user:[REDACTED_SECRET]:8000', 'http://user:[REDACTED_SECRET]:8000', ], }); const crawler = new CheerioCrawler({ proxyConfiguration, useSessionPool: true, persistCookiesPerSession: true, async requestHandler({ proxyInfo, request }) { console.log('Using proxy:', proxyInfo?.url); }, }); ``` ```javascript // JS — Tiered proxies (smart cost/reliability balancing) const proxyConfiguration = new ProxyConfiguration({ tieredProxyUrls: [ [null], // Tier 0: no proxy (cheapest) ['http://cheap-datacenter-proxy'], // Tier 1: datacenter ['http://expensive-residential'], // Tier 2: residential (most reliable) ], }); // Crawlee auto-escalates tiers when blocking is detected, then drops back when clear ``` ```python # Python from crawlee.proxy_configuration import ProxyConfiguration proxy_configuration = ProxyConfiguration( proxy_urls=['http://proxy1.com/', 'http://proxy2.com/'], ) crawler = BeautifulSoupCrawler( proxy_configuration=proxy_configuration, use_session_pool=True, ) ``` --- ## 9. Session Management Sessions tie together cookies, proxy IPs, and headers to simulate a consistent user identity. ```javascript // JS const crawler = new CheerioCrawler({ useSessionPool: true, // Enable (default: true) persistCookiesPerSession: true, sessionPoolOptions: { maxPoolSize: 100 }, async requestHandler({ session, $ }) { const title = $('title').text(); if (title === 'Access Denied') { session?.retire(); // Mark this IP+cookie combo as blocked } else if (title === 'Slow') { session?.markBad(); // Penalize but don't retire } // session.markGood() is called automatically on success }, }); ``` ```python # Python from crawlee.sessions import SessionPool crawler = BeautifulSoupCrawler( use_session_pool=True, session_pool=SessionPool(max_pool_size=100), ) @crawler.router.default_handler async def handler(context: BeautifulSoupCrawlingContext) -> None: title = context.soup.title.string if context.soup.title else '' if title == 'Access Denied': context.session.retire() ``` --- ## 10. Avoiding Blocks ```javascript // JS — Playwright with fingerprint rotation (built-in, zero config needed) const crawler = new PlaywrightCrawler({ // Fingerprints automatically randomized by default in Playwright/Puppeteer crawlers // headless: false, // Use headful for harder targets async requestHandler({ page }) { // Add realistic delays await page.waitForTimeout(1000 + Math.random() * 2000); }, }); // Use got-scraping for HTTP (built into CheerioCrawler/HttpCrawler) // It automatically sets realistic headers and TLS fingerprints ``` **Anti-blocking checklist:** - ✅ Use `CheerioCrawler` — it uses `got-scraping` which mimics real browser HTTP - ✅ Enable `useSessionPool: true` with a `proxyConfiguration` - ✅ Use tiered proxies for automatic failover - ✅ Set `maxRequestsPerMinute` to avoid rate limits - ✅ For browser crawlers — fingerprints are rotated automatically - ✅ Use `persistCookiesPerSession: true` - ✅ Retire sessions on blocks: `session.retire()` --- ## 11. Concurrency & Scaling ```javascript // JS const crawler = new CheerioCrawler({ maxConcurrency: 50, // Max parallel requests (default: 200) minConcurrency: 1, // Don't set too high! maxRequestsPerMinute: 120, // Rate limit maxRequestsPerCrawl: 1000, // Total request cap (safety) requestHandlerTimeoutSecs: 30, }); ``` ```python # Python from crawlee import ConcurrencySettings crawler = BeautifulSoupCrawler( concurrency_settings=ConcurrencySettings( max_concurrency=50, max_tasks_per_minute=120, ), max_requests_per_crawl=1000, ) ``` **Scaling notes:** - Crawlee auto-scales concurrency based on CPU/memory - Don't set `minConcurrency` high — it can crash under load - `maxRequestsPerMinute` is smoother than raw concurrency throttling --- ## 12. Configuration & Environment Variables | Env Variable | Default | Purpose | |---|---|---| | `CRAWLEE_STORAGE_DIR` | `./storage` | Storage root directory | | `CRAWLEE_DEFAULT_DATASET_ID` | `default` | Override default dataset ID | | [REDACTED_SECRET] | `default` | Override default KVS ID | | [REDACTED_SECRET] | `default` | Override default queue ID | | `CRAWLEE_PURGE_ON_START` | `true` | Clear storage before each run | ```javascript // JS — Programmatic configuration import { Configuration } from 'crawlee'; const config = new Configuration({ storageDir: '/data/crawlee', persistStateIntervalMillis: 30_000, }); const crawler = new CheerioCrawler({ /* ... */ }, config); ``` --- ## 13. Docker Deployment ```dockerfile FROM apify/actor-node-playwright-chrome:20 COPY package*.json ./ RUN npm ci --only=prod COPY . ./ CMD ["node", "src/main.js"] ``` For Cheerio (smaller image): ```dockerfile FROM apify/actor-node:20 ``` --- ## 14. Common Patterns ### Pagination ```javascript // JS — Enqueue next page router.addHandler('LIST', async ({ page, enqueueLinks }) => { await enqueueLinks({ selector: '.product', label: 'DETAIL' }); const hasNext = await page.$('a.next'); if (hasNext) await enqueueLinks({ selector: 'a.next', label: 'LIST' }); }); ``` ### Downloading Files ```javascript // JS — Save to KeyValueStore const { body } = await sendRequest({ responseType: 'buffer' }); await KeyValueStore.setValue('file.pdf', body, { contentType: 'application/pdf' }); ``` ### Taking Screenshots ```javascript // JS — Playwright async requestHandler({ page, request }) { const screenshot = await page.screenshot({ fullPage: true }); await KeyValueStore.setValue( `screenshot-${Date.now()}`, screenshot, { contentType: 'image/png' } ); } ``` ### Shared State Across Handlers ```javascript // JS — useState() async requestHandler({ useState }) { const state = await useState({ count: 0 }); state.count++; console.log('Total processed:', state.count); } ``` ### Error Handling & Retries ```javascript // JS const crawler = new CheerioCrawler({ maxRequestRetries: 3, // Retry failed requests up to 3 times failedRequestHandler: async ({ request, error }) => { console.error(`Failed: ${request.url}`, error.message); await Dataset.pushData({ url: request.url, error: error.message }); }, }); ``` ```python # Python crawler = BeautifulSoupCrawler(max_request_retries=3) @crawler.failed_request_handler async def on_failed(context: BasicCrawlingContext, error: Exception) -> None: context.log.error(f'Failed {context.request.url}: {error}') ``` ### Sitemap Crawling ```javascript import { CheerioCrawler } from 'crawlee'; import { Sitemap } from '@crawlee/utils'; const { urls } = await Sitemap.load('https://example.com/sitemap.xml'); const crawler = new CheerioCrawler({ /* ... */ }); await crawler.run(urls); ``` ### Run as Web Server ```javascript import { CheerioCrawler } from 'crawlee'; import { createServer } from 'http'; const server = createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost').searchParams.get('url'); const crawler = new CheerioCrawler({ maxRequestsPerCrawl: 1, async requestHandler({ $ }) { res.end(JSON.stringify({ title: $('title').text() })); }, }); await crawler.run([url]); }); server.listen(3000); ``` --- ## 15. TypeScript Support ```typescript import { CheerioCrawler, CheerioCrawlingContext, Dataset } from 'crawlee'; interface Product { url: string; title: string; price: number; } const crawler = new CheerioCrawler({ async requestHandler({ $, request }: CheerioCrawlingContext) { const title = $('h1').text(); const price = parseFloat($('.price').text().replace('$', '')); await Dataset.pushData<Product>({ url: request.url, title, price }); }, }); ``` --- ## 16. Cloud Deployment (Apify Platform) ```javascript import { Actor } from 'apify'; import { CheerioCrawler } from 'crawlee'; await Actor.init(); const input = await Actor.getInput(); const { startUrls } = input; const crawler = new CheerioCrawler({ async requestHandler({ $, request }) { await Actor.pushData({ url: request.url, title: $('title').text() }); }, }); await crawler.run(startUrls); await Actor.exit(); ``` Deploy with: `apify push` --- ## 17. Debugging Tips ```javascript // Enable verbose logging import { Log } from 'crawlee'; Log.setLevel(Log.LEVELS.DEBUG); // Run headful (browser crawlers only) const crawler = new PlaywrightCrawler({ headless: false, // ... }); // Limit requests while developing const crawler = new CheerioCrawler({ maxRequestsPerCrawl: 10, // ... }); ``` --- ## 18. Reference Files For advanced topics, see: - `references/js-api.md` — Full JS API quick reference - `references/python-api.md` — Full Python API quick reference Both language docs: https://crawlee.dev


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)