Skill: jiafar/amazon-scraper (version 3.4.1)

--- name: amazon-scraper description: > High-performance containerized Amazon scraper (Docker + playwright-extra + Stealth plugin). Bypasses Amazon headless detection. Supports Amazon BSR, search results, and product detail pages. Also includes a generic mode for other dynamic web pages. Use when user mentions any of these: 爬虫, 爬取, 抓取, 采集, 数据采集, 爬数据, 抓数据, 获取数据, scrape, crawl, extract, fetch data, pull data, 亚马逊, Amazon, ASIN, BSR, Best Sellers, 畅销榜, 热销榜, 新品榜, 飙升榜, 排行榜, 选品, 竞品分析, 竞品调研, 市场调研, 品类分析, 类目分析, 产品调研, 月销量, bought in past month, 销量, 评论数, 价格对比, 网页内容, 网站数据, 页面抓取, 动态页面, 关键词搜索, 搜索结果, search results, 产品详情, 产品信息, listing数据, listing分析, top 100, top sellers, 热门产品, 爆款, 跑量款, 价格带, 评分分布, review分析, 评论分析 --- # Amazon Scraper Docker容器化爬虫，基于 playwright-extra + Stealth 插件，专为绕过亚马逊反爬检测优化，同时支持通用动态网页爬取。 ## ⚙️ 系统要求 - **Docker Engine 20.10+**（必须已安装并运行） - **磁盘空间**：~2GB（镜像 + Playwright 浏览器二进制文件） - **内存**：建议 2GB+（Playwright 运行时需要） ## 快速开始 首次使用：在 skill 目录下执行一键构建脚本： ```bash bash scripts/setup.sh ``` 脚本会自动完成：构建 `amazon-scraper` 镜像 + 创建 `~/scrapes` 输出目录。 ## 模式选择规则 ### 1. Amazon模式 (`amazon_handler.js`) **自动触发条件:** URL包含 `amazon.com`，或用户提到亚马逊/Amazon/ASIN/BSR/选品/竞品/畅销榜/类目分析等关键词 根据URL自动识别页面类型： | URL特征 | 页面类型 | 可获取字段 | |---|---|---| | `/gp/bestsellers/` | 畅销榜 | rank, title, asin, price, rating, reviews, image, url | | `/zg/new-releases/` | 新品榜 | 同上 | | `/zg/movers-and-shakers/` | 飙升榜 | 同上 | | `/s?k=` 或 `/s/` | 搜索结果 | title, asin, price, rating, reviews, image, url, **boughtPastMonth**, sponsored | | `/dp/` 或 `/gp/product/` | 产品详情 | title, asin, price, rating, reviews, brand, bsr, **boughtPastMonth**, dateFirstAvailable, category, bullets, details, image | **⚠️ 重要规则:** - **Best Sellers页面没有月销量(boughtPastMonth)数据** — 亚马逊不在榜单页显示此信息 - **要获取月销量，必须用搜索页(`/s?k=关键词`)或产品详情页(`/dp/ASIN`)** - 如果用户同时需要排名+月销量，建议：先爬Best Sellers拿排名，再用搜索页补月销 - **BSR URL 必须使用 `/gp/bestsellers/`**，`/zgbs/` 会返回 Page Not Found ```bash # 畅销榜（有排名，无月销） docker run -t --rm amazon-scraper node assets/amazon_handler.js "https://www.amazon.com/gp/bestsellers/electronics" # 搜索结果（有月销，无排名） docker run -t --rm amazon-scraper node assets/amazon_handler.js "https://www.amazon.com/s?k=feather+duster" # 产品详情（最全字段：BSR、品牌、卖点、月销） docker run -t --rm amazon-scraper node assets/amazon_handler.js "https://www.amazon.com/dp/B001TQ6IHS" # 多页爬取 docker run -t --rm amazon-scraper node assets/amazon_handler.js "URL" --pages 2 # 保存结果到文件 docker run -t --rm -v ~/scrapes:/data amazon-scraper node assets/amazon_handler.js "URL" --output result.json # 用自己的代理覆盖内置配置 docker run -t --rm -e AMAZON_PROXIES="[REDACTED_SECRET]" amazon-scraper node assets/amazon_handler.js "URL" ``` **输出格式:** JSON ```json { "status": "SUCCESS", "type": "bestsellers|search|product-detail", "category": "品类名", "totalProducts": 30, "scrapedAt": "ISO时间", "products": [ { "rank": 1, "title": "产品名", "asin": "B001TQ6IHS", "price": 9.94, "priceStr": "$9.94", "rating": 4.6, "reviews": 20547, "boughtPastMonth": "1K+", "image": "https://...", "url": "https://..." } ] } ``` ### 2. 通用模式 (`main_handler.js`) **触发条件:** 非Amazon的URL，或用户提到爬取/抓取任意网页内容 - 基于和 Amazon 模式相同的 `playwright-extra` + Stealth 架构 - 内置代理已预配置，无需额外设置 - 支持 `--output` 文件保存 - 可通过环境变量覆盖内置代理 - Playwright打开页面，等待JS加载完成 - 提取 `document.body.innerText`（纯文本，去广告噪音） - 输出上限10000字符 - 输出: `{status:"SUCCESS", type:"GENERIC", title, data}` ```bash # 通用爬取（代理已内置） docker run -t --rm amazon-scraper node assets/main_handler.js "https://任意网址" # 保存文件 docker run -t --rm -v ~/scrapes:/data \ amazon-scraper node assets/main_handler.js "https://任意网址" --output page.json ``` ## Agent调用决策树 ``` 用户给了URL? ├─ 包含 amazon.com → 用 amazon_handler.js │ ├─ 需要月销量? → 建议用搜索URL(/s?k=) 或详情页(/dp/) │ └─ 需要排名? → 用畅销榜URL(/gp/bestsellers/) └─ 其他网站 → 用 main_handler.js (通用模式) 用户没给URL，只说了需求? ├─ "爬亚马逊XX品类Top" / "XX类目排行" / "XX畅销榜" → 构造 https://www.amazon.com/gp/bestsellers/品类 ├─ "搜亚马逊XX" / "XX关键词搜索" / "找XX产品" → 构造 https://www.amazon.com/s?k=关键词 ├─ "分析某个ASIN" / "看看这个产品" / "XX的详情" → 构造 https://www.amazon.com/dp/ASIN ├─ "XX的月销量" / "XX卖了多少" / "XX销量怎么样" → 用搜索页或详情页（有boughtPastMonth） ├─ "竞品分析" / "竞品调研" / "对手在卖什么" → 先搜索再逐个爬详情 ├─ "选品" / "什么好卖" / "品类机会" / "市场调研" → Best Sellers + 搜索结合 └─ 其他网页 → 先web_search找到URL，再用通用模式爬 ``` ## 常见用户意图 → 操作映射 | 用户说 | 操作 | |---|---| | "帮我看看亚马逊XX品类" | 爬 /gp/bestsellers/品类 畅销榜 | | "XX在亚马逊卖得怎么样" | 搜索 /s?k=XX 看月销 | | "分析一下这个ASIN: BXXXXXXXXX" | 爬 /dp/ASIN 详情页 | | "XX品类有什么机会" | 畅销榜 + 搜索 综合分析 | | "帮我爬这个链接" | 判断URL类型，选对应handler | | "帮我抓XX网站的内容" | 通用模式 | | "搜一下XX的竞品" | 搜索页爬取 + 分析 | | "XX月销多少" / "XX一个月卖多少" | 搜索页或详情页 | | "帮我看看top 100" / "热门产品" | Best Sellers畅销榜 | | "新品有哪些" / "最近上了什么新品" | /zg/new-releases/ | | "什么产品涨得快" / "飙升榜" | /zg/movers-and-shakers/ | ## 代理配置 **本 skill 已内置 5 个轮询代理，无需额外配置即可直接使用。** 如需覆盖内置代理，可通过环境变量注入自己的代理： | 变量 | 用途 | 格式 | |---|---|---| | `AMAZON_PROXY` | 单代理 | `[REDACTED_SECRET]` | | `AMAZON_PROXIES` | 多代理轮询 | `[REDACTED_SECRET]` | - **轮询**：多页爬取时每页自动切换下一个代理 - **故障切换**：单页失败时自动重试列表中下一个代理 - 代理配置存放于 `config/proxies.json`，可直接修改文件更新代理列表 ## 反爬能力 - **playwright-extra + puppeteer-extra-plugin-stealth** — 自动修改 navigator、WebGL、Canvas 等 headless 特征 - **Chrome 123 UserAgent** — 模拟真实 Mac Chrome 浏览器 - **完整浏览器指纹 headers** — Accept-Encoding: identity, Sec-Ch-Ua, Sec-Fetch-* 等 - **1920x1080 viewport** — 避免移动端/小屏检测 - 自动滚动加载懒加载内容 - Docker沙箱隔离，每次启动全新浏览器上下文 - 代理轮询分散请求源 IP ## 局限 - 通用模式输出上限10000字符 - Amazon单页最多约30-50个产品 - 不支持需要登录的页面 - Docker容器启动有~15秒冷启动时间（含 stealth 插件初始化）

path: Dockerfile.sh
content: FROM mcr.microsoft.com/playwright:v1.40.0-jammy

WORKDIR /app

COPY package.json ./
RUN npm install
RUN npx playwright install chromium

COPY assets/ ./assets/
COPY scripts/ ./scripts/
COPY config/ ./config/

RUN mkdir -p /data

CMD ["node", "assets/amazon_handler.js"]


path: assets/amazon_handler.js
content: const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();
chromium.use(stealth);

const fs = require('fs');
const path = require('path');

const args = process.argv.slice(2);
const targetUrl = args.find(a => a.startsWith('http'));
const maxPages = parseInt(args.find((a, i) => args[i-1] === '--pages') || '1');
const outputFile = args.find((a, i) => args[i-1] === '--output');

if (!targetUrl) {
    console.error('Usage: node assets/amazon_handler.js <AMAZON_URL> [--pages N] [--output path.json]');
    process.exit(1);
}

function saveResult(data) {
    if (outputFile) {
        const outPath = path.isAbsolute(outputFile) ? outputFile : path.join('/data', outputFile);
        const dir = path.dirname(outPath);
        if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
        fs.writeFileSync(outPath, JSON.stringify(data, null, 2));
        console.error(`Result saved to: ${outPath}`);
    }
}

function detectPageType(url) {
    if (url.includes('/zgbs/') || url.includes('/bestsellers/')) return 'bestsellers';
    if (url.includes('/zg/new-releases/')) return 'new-releases';
    if (url.includes('/zg/movers-and-shakers/')) return 'movers-shakers';
    if (url.includes('/dp/') || url.includes('/gp/product/')) return 'product-detail';
    if (url.includes('/s?') || url.includes('/s/')) return 'search';
    return 'generic';
}

const pageType = detectPageType(targetUrl);

// ---- Proxy rotation setup ----
// Load built-in proxy config, allow env vars to override
function loadProxies() {
    try {
        const configPath = path.join(__dirname, '..', 'config', 'proxies.json');
        if (fs.existsSync(configPath)) {
            const cfg = JSON.parse(fs.readFileSync(configPath, 'utf8'));
            if (cfg.proxies && cfg.proxies.length > 0) {
                return cfg.proxies;
            }
        }
    } catch (e) {
        console.error('Failed to load config/proxies.json:', e.message);
    }
    const single = process.env.AMAZON_PROXY;
    const multi = process.env.AMAZON_PROXIES;
    if (multi) return multi.split(',').map(s => s.trim()).filter(Boolean);
    if (single) return [single];
    return [];
}

const PROXY_LIST = loadProxies();

let proxyIdx = 0;
function getNextProxy() {
    if (PROXY_LIST.length === 0) return null;
    const p = PROXY_LIST[proxyIdx % PROXY_LIST.length];
    proxyIdx++;
    return p;
}

function parseProxy(proxyUrl) {
    try {
        const url = new URL(proxyUrl);
        return {
            server: `${url.protocol}//${url.host}`,
            username: decodeURIComponent(url.username),
            [REDACTED_SECRET])
        };
    } catch (e) {
        console.error('Invalid proxy URL format:', proxyUrl);
        return null;
    }
}

async function createContext(browser) {
    const proxyUrl = getNextProxy();
    const contextOptions = {
        viewport: { width: 1920, height: 1080 },
        userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36',
        extraHTTPHeaders: {
            'Accept-Language': 'en-US,en;q=0.9',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
            'Accept-Encoding': 'identity',
            'Sec-Ch-Ua': '"Google Chrome";v="123", "Not:A-Brand";v="8"',
            'Sec-Ch-Ua-Mobile': '?0',
            'Sec-Ch-Ua-Platform': '"macOS"',
            'Upgrade-Insecure-Requests': '1',
            'Sec-Fetch-Site': 'none',
            'Sec-Fetch-Mode': 'navigate',
            'Sec-Fetch-User': '?1',
            'Sec-Fetch-Dest': 'document'
        }
    };
    if (proxyUrl) {
        const parsed = parseProxy(proxyUrl);
        if (parsed) {
            contextOptions.proxy = parsed;
            console.error(`Using proxy: ${parsed.server} (user: ${parsed.username})`);
        }
    }
    const context = await browser.newContext(contextOptions);
    await context.setExtraHTTPHeaders({
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'identity',
        'Cache-Control': 'max-age=0',
        'Sec-Ch-Ua': '"Google Chrome";v="123", "Not:A-Brand";v="8"',
        'Sec-Ch-Ua-Mobile': '?0',
        'Sec-Ch-Ua-Platform': '"macOS"',
        'Sec-Fetch-Dest': 'document',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-User': '?1',
        'Upgrade-Insecure-Requests': '1'
    });
    return context;
}

async function runCrawler() {
    const browser = await chromium.launch({
        headless: true,
        args: ['--no-sandbox', '--disable-setuid-sandbox']
    });

    let context = null;
    let page = null;

    try {
        let allProducts = [];

        for (let pg = 1; pg <= maxPages; pg++) {
            let url = targetUrl;
            if (pg > 1) url = targetUrl.includes('?') ? `${targetUrl}&pg=${pg}` : `${targetUrl}?pg=${pg}`;

            // Retry with proxy rotation on failure
            let success = false;
            let lastErr = null;
            for (let attempt = 0; attempt < Math.max(PROXY_LIST.length, 1); attempt++) {
                try {
                    if (context) await context.close();
                    context = await createContext(browser);
                    page = await context.newPage();

                    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });
                    await page.waitForTimeout(3000);
                    await page.evaluate(async () => {
                        for (let i = 0; i < 5; i++) { window.scrollBy(0, window.innerHeight); await new Promise(r => setTimeout(r, 500)); }
                        window.scrollTo(0, 0);
                    });
                    success = true;
                    break;
                } catch (err) {
                    lastErr = err;
                    console.error(`Attempt ${attempt + 1} failed: ${err.message}`);
                    if (page) { try { await page.close(); } catch(e) {} }
                    if (context) { try { await context.close(); } catch(e) {} }
                }
            }
            if (!success) {
                throw new Error(`All proxies failed. Last error: ${lastErr.message}`);
            }

            let products = [];

            if (pageType === 'bestsellers' || pageType === 'new-releases' || pageType === 'movers-shakers') {
                products = await page.evaluate(() => {
                    const items = [];
                    const cards = document.querySelectorAll('[data-asin]');

                    if (cards.length > 0) {
                        cards.forEach(card => {
                            try {
                                const rankEl = card.querySelector('.zg-bdg-text, [class*="zg-badge"]');
                                const titleEl = card.querySelector('a span, ._cDEzb_p13n-sc-css-line-clamp-1_1Fn1y, .p13n-sc-truncate');
                                const ratingEl = card.querySelector('[class*="a-icon-alt"]');
                                const reviewEl = card.querySelector('[class*="a-size-small"]');
                                const priceEl = card.querySelector('.p13n-sc-price, ._cDEzb_p13n-sc-price_3mJ9Z, .a-price .a-offscreen');
                                const imgEl = card.querySelector('img');
                                const linkEl = card.querySelector('a[href*="/dp/"]');
                                const asin = card.getAttribute('data-asin') || (linkEl && linkEl.href && linkEl.href.match(/\/dp\/([A-Z0-9]{10})/) ? linkEl.href.match(/\/dp\/([A-Z0-9]{10})/)[1] : null);

                                let boughtPastMonth = null;
                                card.querySelectorAll('span').forEach(s => {
                                    const t = s.textContent.trim();
                                    if (t.match(/bought in past month/i)) {
                                        const m = t.match(/([\d,.]+[KkMm]?\+?)\s*bought/i);
                                        boughtPastMonth = m ? m[1] : t;
                                    }
                                });

                                const rank = rankEl ? parseInt(rankEl.textContent.replace('#', '')) : null;
                                const title = titleEl ? titleEl.textContent.trim() : null;
                                const rating = ratingEl ? parseFloat(ratingEl.textContent) : null;
                                const reviews = reviewEl ? parseInt(reviewEl.textContent.replace(/[^0-9]/g, '')) : null;
                                const priceText = priceEl ? priceEl.textContent.trim() : null;
                                const price = priceText ? parseFloat(priceText.replace(/[^0-9.]/g, '')) : null;
                                const image = imgEl ? imgEl.src : null;
                                const url = linkEl ? linkEl.href : null;

                                if (title) {
                                    items.push({ rank, title, rating, reviews, price, priceStr: priceText, asin, image, url, boughtPastMonth });
                                }
                            } catch (e) {}
                        });
                    }

                    if (items.length === 0) {
                        const text = document.body.innerText;
                        const lines = text.split('\n').map(l => l.trim()).filter(Boolean);
                        let currentRank = null;
                        let currentProduct = {};

                        for (const line of lines) {
                            const rankMatch = line.match(/^#(\d+)$/);
                            if (rankMatch) {
                                if (currentRank && currentProduct.title) items.push(currentProduct);
                                currentRank = parseInt(rankMatch[1]);
                                currentProduct = { rank: currentRank };
                                continue;
                            }
                            if (currentRank) {
                                if (line.match(/^([\d.]+) out of 5 stars$/)) {
                                    currentProduct.rating = parseFloat(line);
                                } else if (line.match(/^\$([\d,.]+)$/)) {
                                    currentProduct.price = parseFloat(line.replace(/[$,]/g, ''));
                                    currentProduct.priceStr = line;
                                } else if (line.match(/^\s*[\d,]+\s*$/) && !currentProduct.reviews && currentProduct.rating) {
                                    currentProduct.reviews = parseInt(line.replace(/,/g, ''));
                                } else if (line.match(/bought in past month/i)) {
                                    const m = line.match(/([\d,.]+[KkMm]?\+?)\s*bought/i);
                                    currentProduct.boughtPastMonth = m ? m[1] : line;
                                } else if (!currentProduct.title && line.length > 10
                                    && !line.includes('out of 5') && !line.includes('Best Seller')
                                    && !line.includes('Previous page') && !line.includes('Next page')) {
                                    currentProduct.title = line;
                                }
                            }
                        }
                        if (currentRank && currentProduct.title) items.push(currentProduct);
                    }
                    return items;
                });

            } else if (pageType === 'product-detail') {
                products = await page.evaluate(() => {
                    const title = (document.querySelector('#productTitle') || {}).textContent?.trim();
                    const priceEl = document.querySelector('.a-price .a-offscreen, #priceblock_ourprice, #priceblock_dealprice');
                    const priceStr = priceEl ? priceEl.textContent.trim() : null;
                    const price = priceStr ? parseFloat(priceStr.replace(/[^0-9.]/g, '')) : null;
                    const ratingEl = document.querySelector('#acrPopover .a-icon-alt');
                    const rating = ratingEl ? parseFloat(ratingEl.textContent) : null;
                    const reviewsEl = document.querySelector('#acrCustomerReviewText');
                    const reviews = reviewsEl ? parseInt(reviewsEl.textContent.replace(/[^0-9]/g, '')) : null;
                    const asin = (document.querySelector('[data-asin]') || {}).getAttribute?.('data-asin') || (window.location.pathname.match(/\/dp\/([A-Z0-9]{10})/) || [])[1];
                    const brand = (document.querySelector('#bylineInfo') || {}).textContent?.trim();
                    const image = (document.querySelector('#landingImage, #imgBlkFront') || {}).src;
                    const bsrMatch = document.body.innerText.match(/Best Sellers Rank.*?#([\d,]+)/);
                    const bsr = bsrMatch ? parseInt(bsrMatch[1].replace(/,/g, '')) : null;
                    const breadcrumbs = Array.from(document.querySelectorAll('#wayfinding-breadcrumbs_feature_div a')).map(a => a.textContent.trim());
                    const bullets = Array.from(document.querySelectorAll('#feature-bullets li span')).map(s => s.textContent.trim()).filter(Boolean);

                    let boughtPastMonth = null;
                    const boughtMatch = document.body.innerText.match(/([\d,.]+[KkMm]?\+?)\s*bought in past month/i);
                    if (boughtMatch) boughtPastMonth = boughtMatch[1];

                    const dateMatch = document.body.innerText.match(/Date First Available\s*[:\n]\s*([A-Za-z]+ \d+,? \d{4})/);
                    const dateFirstAvailable = dateMatch ? dateMatch[1] : null;

                    const details = {};
                    document.querySelectorAll('#productDetails_techSpec_section_1 tr, #detailBullets_feature_div li').forEach(row => {
                        const key = (row.querySelector('th, .a-text-bold') || {}).textContent?.trim()?.replace(/[:\s]+$/, '');
                        const val = (row.querySelector('td, span:not(.a-text-bold)') || {}).textContent?.trim();
                        if (key && val) details[key] = val;
                    });

                    return [{ title, price, priceStr, rating, reviews, asin, brand, image, bsr, boughtPastMonth, dateFirstAvailable, category: breadcrumbs, bullets, details }];
                });

            } else if (pageType === 'search') {
                products = await page.evaluate(() => {
                    const items = [];
                    document.querySelectorAll('[data-component-type="s-search-result"]').forEach(card => {
                        try {
                            const asin = card.getAttribute('data-asin');
                            const titleEl = card.querySelector('h2 a span');
                            const priceEl = card.querySelector('.a-price .a-offscreen');
                            const ratingEl = card.querySelector('.a-icon-alt');
                            const reviewEl = card.querySelector('[class*="s-link-style"] .a-size-base');
                            const imgEl = card.querySelector('.s-image');
                            const linkEl = card.querySelector('h2 a');
                            const sponsoredEl = card.querySelector('.s-label-popover-default');

                            let boughtPastMonth = null;
                            card.querySelectorAll('span').forEach(s => {
                                const t = s.textContent.trim();
                                if (t.match(/bought in past month/i)) {
                                    const m = t.match(/([\d,.]+[KkMm]?\+?)\s*bought/i);
                                    boughtPastMonth = m ? m[1] : t;
                                }
                            });

                            items.push({
                                asin,
                                title: titleEl ? titleEl.textContent.trim() : null,
                                price: priceEl ? parseFloat(priceEl.textContent.replace(/[^0-9.]/g, '')) : null,
                                priceStr: priceEl ? priceEl.textContent.trim() : null,
                                rating: ratingEl ? parseFloat(ratingEl.textContent) : null,
                                reviews: reviewEl ? parseInt(reviewEl.textContent.replace(/[^0-9]/g, '')) : null,
                                image: imgEl ? imgEl.src : null,
                                url: linkEl ? 'https://www.amazon.com' + linkEl.getAttribute('href') : null,
                                boughtPastMonth,
                                sponsored: !!sponsoredEl
                            });
                        } catch (e) {}
                    });
                    return items;
                });

            } else {
                // Generic fallback
                const content = await page.evaluate(() => document.body.innerText);
                const result = { status: 'SUCCESS', type: 'GENERIC', title, data: content.substring(0, 10000) };
                console.log(JSON.stringify(result));
                saveResult(result);
                await browser.close();
                return;
            }

            // Deduplicate by ASIN
            products = products.filter((p, i, arr) => {
                if (!p.asin) return true;
                const firstIdx = arr.findIndex(x => x.asin === p.asin);
                if (firstIdx !== i) {
                    if (p.rank && !arr[firstIdx].rank) arr[firstIdx].rank = p.rank;
                    if (p.boughtPastMonth && !arr[firstIdx].boughtPastMonth) arr[firstIdx].boughtPastMonth = p.boughtPastMonth;
                    return false;
                }
                return true;
            });

            allProducts.push(...products);
            if (pg < maxPages) await page.waitForTimeout(2000);
        }

        const metadata = await page.evaluate(() => {
            const title = document.title;
            const breadcrumbs = Array.from(document.querySelectorAll('#zg_browseRoot a, .zg-breadcrumb a')).map(a => a.textContent.trim());
            return { title, breadcrumbs };
        });

        const result = {
            status: 'SUCCESS',
            type: pageType,
            url: targetUrl,
            category: metadata.title?.replace('Amazon Best Sellers: Best ', '').replace('Amazon.com : ', ''),
            breadcrumbs: metadata.breadcrumbs,
            totalProducts: allProducts.length,
            pages: maxPages,
            scrapedAt: new Date().toISOString(),
            products: allProducts
        };
        console.log(JSON.stringify(result));
        saveResult(result);

    } catch (err) {
        const errorResult = { status: 'ERROR', message: err.message };
        console.error(JSON.stringify(errorResult));
        saveResult(errorResult);
        process.exit(1);
    } finally {
        if (context) { try { await context.close(); } catch(e) {} }
        await browser.close();
    }
}

runCrawler();


path: assets/main_handler.js
content: const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();
chromium.use(stealth);

const fs = require('fs');
const path = require('path');

const args = process.argv.slice(2);
const targetUrl = args.find(a => a.startsWith('http'));
const outputFile = args.find((a, i) => args[i-1] === '--output');

if (!targetUrl) {
    console.error('Usage: node assets/main_handler.js <URL> [--output path.json]');
    process.exit(1);
}

function saveResult(data) {
    if (outputFile) {
        const outPath = path.isAbsolute(outputFile) ? outputFile : path.join('/data', outputFile);
        const dir = path.dirname(outPath);
        if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
        fs.writeFileSync(outPath, JSON.stringify(data, null, 2));
        console.error(`Result saved to: ${outPath}`);
    }
}

// ---- Proxy rotation setup ----
// Load built-in proxy config, allow env vars to override
function loadProxies() {
    try {
        const configPath = path.join(__dirname, '..', 'config', 'proxies.json');
        if (fs.existsSync(configPath)) {
            const cfg = JSON.parse(fs.readFileSync(configPath, 'utf8'));
            if (cfg.proxies && cfg.proxies.length > 0) {
                return cfg.proxies;
            }
        }
    } catch (e) {
        console.error('Failed to load config/proxies.json:', e.message);
    }
    const single = process.env.AMAZON_PROXY;
    const multi = process.env.AMAZON_PROXIES;
    if (multi) return multi.split(',').map(s => s.trim()).filter(Boolean);
    if (single) return [single];
    return [];
}

const PROXY_LIST = loadProxies();

let proxyIdx = 0;
function getNextProxy() {
    if (PROXY_LIST.length === 0) return null;
    const p = PROXY_LIST[proxyIdx % PROXY_LIST.length];
    proxyIdx++;
    return p;
}

function parseProxy(proxyUrl) {
    try {
        const url = new URL(proxyUrl);
        return {
            server: `${url.protocol}//${url.host}`,
            username: decodeURIComponent(url.username),
            [REDACTED_SECRET])
        };
    } catch (e) {
        console.error('Invalid proxy URL format:', proxyUrl);
        return null;
    }
}

async function createContext(browser) {
    const proxyUrl = getNextProxy();
    const contextOptions = {
        viewport: { width: 1920, height: 1080 },
        userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36',
        extraHTTPHeaders: {
            'Accept-Language': 'en-US,en;q=0.9',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
            'Accept-Encoding': 'identity',
            'Sec-Ch-Ua': '"Google Chrome";v="123", "Not:A-Brand";v="8"',
            'Sec-Ch-Ua-Mobile': '?0',
            'Sec-Ch-Ua-Platform': '"macOS"',
            'Upgrade-Insecure-Requests': '1',
            'Sec-Fetch-Site': 'none',
            'Sec-Fetch-Mode': 'navigate',
            'Sec-Fetch-User': '?1',
            'Sec-Fetch-Dest': 'document'
        }
    };
    if (proxyUrl) {
        const parsed = parseProxy(proxyUrl);
        if (parsed) {
            contextOptions.proxy = parsed;
            console.error(`Using proxy: ${parsed.server} (user: ${parsed.username})`);
        }
    }
    const context = await browser.newContext(contextOptions);
    await context.setExtraHTTPHeaders({
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'identity',
        'Cache-Control': 'max-age=0',
        'Sec-Ch-Ua': '"Google Chrome";v="123", "Not:A-Brand";v="8"',
        'Sec-Ch-Ua-Mobile': '?0',
        'Sec-Ch-Ua-Platform': '"macOS"',
        'Sec-Fetch-Dest': 'document',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-User': '?1',
        'Upgrade-Insecure-Requests': '1'
    });
    return context;
}

(async () => {
    const browser = await chromium.launch({
        headless: true,
        args: ['--no-sandbox', '--disable-setuid-sandbox']
    });

    let context = null;
    let page = null;

    try {
        let success = false;
        let lastErr = null;
        for (let attempt = 0; attempt < Math.max(PROXY_LIST.length, 1); attempt++) {
            try {
                if (context) await context.close();
                context = await createContext(browser);
                page = await context.newPage();

                await page.goto(targetUrl, { waitUntil: 'networkidle', timeout: 60000 });
                await page.waitForTimeout(2000);

                success = true;
                break;
            } catch (err) {
                lastErr = err;
                console.error(`Attempt ${attempt + 1} failed: ${err.message}`);
                if (page) { try { await page.close(); } catch(e) {} }
                if (context) { try { await context.close(); } catch(e) {} }
            }
        }
        if (!success) {
            throw new Error(`All proxies failed. Last error: ${lastErr.message}`);
        }

        const title = await page.title();
        const content = await page.evaluate(() => document.body.innerText);
        const result = {
            status: 'SUCCESS',
            type: 'GENERIC',
            url: targetUrl,
            title,
            data: content.substring(0, 10000),
            scrapedAt: new Date().toISOString()
        };
        console.log(JSON.stringify(result));
        saveResult(result);

    } catch (err) {
        const errorResult = { status: 'ERROR', message: err.message };
        console.error(JSON.stringify(errorResult));
        saveResult(errorResult);
        process.exit(1);
    } finally {
        if (context) { try { await context.close(); } catch(e) {} }
        await browser.close();
    }
})();


path: config/proxies.json
content: {
  "proxies": [
    "[REDACTED_SECRET]",
    "[REDACTED_SECRET]",
    "[REDACTED_SECRET]",
    "[REDACTED_SECRET]",
    "[REDACTED_SECRET]"
  ]
}


path: package.json
content: {
  "name": "amazon-scraper",
  "version": "3.4.1",
  "description": "High-performance containerized Amazon scraper with stealth mode. Bypasses headless detection. Supports BSR, search, product detail, and generic pages. Built on Docker + playwright-extra + Stealth.",
  "main": "assets/amazon_handler.js",
  "scripts": {
    "test": "node assets/main_handler.js https://www.amazon.com/gp/bestsellers/electronics",
    "setup": "bash scripts/setup.sh"
  },
  "keywords": [
    "amazon",
    "scraper",
    "asin",
    "bsr",
    "playwright",
    "openclaw",
    "docker"
  ],
  "author": "Joseph",
  "license": "MIT",
  "dependencies": {
    "playwright": "^1.40.0",
    "playwright-extra": "^4.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2"
  },
  "openclaw": {
    "requires": {
      "docker": true
    },
    "requiredBinaries": ["docker"]
  }
}


path: scripts/setup.sh
content: #!/bin/bash
set -e

SCRIPT_DIR="$(cd "$(dirname "$0")/.." && pwd)"

# Dockerfile.sh is the actual Dockerfile (renamed for clawhub bundling compatibility)
# clawhub only bundles files with known text extensions; Dockerfile has no extension
if [ ! -f "$SCRIPT_DIR/Dockerfile" ]; then
  cp "$SCRIPT_DIR/Dockerfile.sh" "$SCRIPT_DIR/Dockerfile"
fi

echo "🔧 Building Docker image 'amazon-scraper'..."
docker build -t amazon-scraper "$SCRIPT_DIR"

mkdir -p ~/scrapes
echo "✅ Done. Image 'amazon-scraper' is ready."
echo "   Scrape output directory: ~/scrapes"
echo ""
echo "Quick test (proxy pre-configured):"
echo "  docker run -t --rm amazon-scraper node assets/amazon_handler.js \"https://www.amazon.com/gp/bestsellers/electronics\""
echo ""
echo "Save results to file (mapped to ~/scrapes):"
echo "  docker run -t --rm -v ~/scrapes:/data amazon-scraper node assets/amazon_handler.js \"URL\" --output result.json"
echo ""
echo "Multi-page search with output:"
echo "  docker run -t --rm -v ~/scrapes:/data amazon-scraper node assets/amazon_handler.js \"https://www.amazon.com/s?k=phone\" --pages 2 --output phone_search.json"
echo ""
echo "Override built-in proxies with your own:"
echo "  docker run -t --rm -e AMAZON_PROXIES=\"[REDACTED_SECRET]" amazon-scraper node assets/amazon_handler.js \"URL\""
echo ""
echo "Tips:"
echo "  - 5 rotating proxies are pre-configured in config/proxies.json"
echo "  - BSR URLs use /gp/bestsellers/ (not /zgbs/) for correct page routing"
echo "  - Output path is relative to /data inside container; map -v ~/scrapes:/data to persist locally"
echo "  - AMAZON_PROXIES env var overrides built-in config; AMAZON_PROXY uses a single proxy"


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)