Skill: sounderliu/gpu-cluster-monitor (version 1.0.2)

# Skill: deep-scraper ## Overview A high-performance engineering tool for deep web scraping. It uses a containerized Docker + Crawlee (Playwright) environment to penetrate protections on complex websites like YouTube and X/Twitter, providing "interception-level" raw data. ## Requirements 1. **Docker**: Must be installed and running on the host machine. 2. **Image**: Build the environment with the tag `clawd-crawlee`. * Build command: `docker build -t clawd-crawlee skills/deep-scraper/` ## Integration Guide Simply copy the `skills/deep-scraper` directory into your `skills/` folder. Ensure the Dockerfile remains within the skill directory for self-contained deployment. ## Standard Interface (CLI) ```bash docker run -t --rm -v $(pwd)/skills/deep-scraper/assets:/usr/src/app/assets clawd-crawlee node assets/main_handler.js [TARGET_URL] ``` ## Output Specification (JSON) The scraping results are printed to stdout as a JSON string: - `status`: SUCCESS | PARTIAL | ERROR - `type`: TRANSCRIPT | DESCRIPTION | GENERIC - `videoId`: (For YouTube) The validated Video ID. - `data`: The core text content or transcript. ## Core Rules 1. **ID Validation**: All YouTube tasks MUST verify the Video ID to prevent cache contamination. 2. **Privacy**: Strictly forbidden from scraping password-protected or non-public personal information. 3. **Alpha-Focused**: Automatically strips ads and noise, delivering pure data optimized for LLM processing.

path: package.json
content: {
  "name": "deep-scraper",
  "version": "1.0.0",
  "description": "High-performance containerized scraper for YouTube, X (Twitter), and dynamic SPAs. Features raw API interception and ID validation.",
  "main": "assets/main_handler.js",
  "scripts": {
    "test": "node assets/main_handler.js https://www.youtube.com/watch?v=dQw4w9WgXcQ"
  },
  "keywords": [
    "scraper",
    "crawlee",
    "youtube",
    "alpha",
    "openclaw"
  ],
  "author": "Joseph",
  "license": "MIT",
  "dependencies": {
    "crawlee": "^3.0.0",
    "playwright": "^1.40.0"
  },
  "openclaw": {
    "requires": {
      "docker": true
    }
  }
}


path: assets/main_handler.js
content: const { PlaywrightCrawler } = require('crawlee');

/**
 * Deep-Scraper: main_handler.js
 * Optimized for containerized execution in OpenClaw.
 * 
 * Usage: node assets/main_handler.js [TARGET_URL]
 */

const targetUrl = process.argv[2];
const videoId = targetUrl?.split('v=')[1]?.split('&')[0];
const mode = targetUrl?.includes('youtube.com') ? 'YOUTUBE' : 'GENERIC';

if (!targetUrl) {
    console.error('Error: No target URL provided.');
    process.exit(1);
}

const crawler = new PlaywrightCrawler({
    launchContext: {
        launchOptions: {
            headless: true,
            args: ['--no-sandbox', '--disable-setuid-sandbox'], // Required for Docker
        },
    },
    maxRequestRetries: 1,
    requestHandlerTimeoutSecs: 300,
    async requestHandler({ page, log }) {
        log.info(`Deep-Scraper starting in ${mode} mode for: ${targetUrl}`);
        
        // Clear context to ensure a fresh session (avoid cache leakage)
        const context = page.context();
        await context.clearCookies();

        if (mode === 'YOUTUBE') {
            let transcriptUrl = null;
            
            // Network Interception for raw transcript API
            page.on('request', request => {
                const reqUrl = request.url();
                // Filter by Video ID to ensure matching data
                if (reqUrl.includes('youtube.com/api/timedtext') && reqUrl.includes(videoId)) {
                    transcriptUrl = reqUrl;
                }
            });

            await page.goto(targetUrl, { waitUntil: 'networkidle' });
            
            // Induce UI interaction to trigger hidden API requests
            try {
                const moreBtn = await page.waitForSelector('#expand, tp-yt-paper-button#expand', { timeout: 5000 });
                if (moreBtn) await moreBtn.click();
                await page.waitForTimeout(2000);
            } catch (e) {
                log.info('UI trigger failed, proceeding with network capture check...');
            }

            if (transcriptUrl) {
                // Fetch directly from intercepted API for 100% accuracy
                const xmlData = await page.evaluate(async (url) => {
                    const resp = await fetch(url);
                    return await resp.text();
                }, transcriptUrl);
                
                // Strip XML tags and clean whitespace
                const cleanText = xmlData.replace(/<[^>]*>/g, ' ').replace(/\s+/g, ' ').trim();
                console.log(JSON.stringify({ 
                    status: 'SUCCESS', 
                    type: 'TRANSCRIPT', 
                    videoId, 
                    data: cleanText.substring(0, 15000) 
                }));
            } else {
                // Fallback to video description if transcript is unavailable
                const desc = await page.evaluate(() => document.querySelector('#description-inline-expander')?.innerText || '');
                console.log(JSON.stringify({ 
                    status: 'PARTIAL', 
                    type: 'DESCRIPTION', 
                    videoId, 
                    data: desc 
                }));
            }
        } else {
            // Generic dynamic page scraping
            await page.goto(targetUrl, { waitUntil: 'networkidle' });
            const title = await page.title();
            const content = await page.evaluate(() => document.body.innerText);
            console.log(JSON.stringify({ 
                status: 'SUCCESS', 
                type: 'GENERIC', 
                title, 
                data: content.substring(0, 10000) 
            }));
        }
    },
});

crawler.run([targetUrl]);


path: assets/youtube_handler.js
content: const { PlaywrightCrawler } = require('crawlee');

const targetUrl = process.argv[2];
const videoId = targetUrl.split('v=')[1]?.split('&')[0];

const crawler = new PlaywrightCrawler({
    launchContext: {
        launchOptions: {
            headless: true,
            args: ['--no-sandbox', '--disable-setuid-sandbox'],
        },
    },
    maxRequestRetries: 1,
    requestHandlerTimeoutSecs: 300,
    async requestHandler({ page, log }) {
        log.info(`正在尝试强制抓取视频内容 (ID: ${videoId}): ${targetUrl}`);
        
        let transcriptUrl = null;

        // 监听所有网络请求
        page.on('request', request => {
            const reqUrl = request.url();
            // 只要包含 timedtext 且包含当前 videoId 的请求都捕获
            if (reqUrl.includes('youtube.com/api/timedtext') && reqUrl.includes(videoId)) {
                log.info(`截获成功: ${reqUrl}`);
                transcriptUrl = reqUrl;
            }
        });

        // 1. 彻底清除缓存和 Cookie 确保全新加载
        const context = page.context();
        await context.clearCookies();

        // 2. 访问页面并等待
        await page.goto(targetUrl, { waitUntil: 'networkidle' });
        await page.waitForTimeout(5000);

        // 3. 强制触发：展开描述栏并寻找“转录稿”按钮
        try {
            const moreBtn = await page.waitForSelector('#expand, tp-yt-paper-button#expand', { timeout: 10000 });
            await moreBtn.click();
            await page.waitForTimeout(2000);

            // 在描述栏里暴力寻找按钮
            await page.evaluate(() => {
                const btns = Array.from(document.querySelectorAll('button, ytd-button-renderer'));
                const target = btns.find(b => b.innerText.includes('转录') || b.innerText.includes('Transcript'));
                if (target) {
                    target.scrollIntoView();
                    target.click();
                }
            });
            log.info('已尝试点击触发按钮');
            await page.waitForTimeout(5000);
        } catch (e) {
            log.info('UI 触发失败，继续观察网络流量...');
        }

        // 4. 如果还没拿到，尝试开启 CC
        if (!transcriptUrl) {
            try {
                await page.click('.ytp-subtitles-button');
                log.info('已尝试点击 CC 按钮');
                await page.waitForTimeout(5000);
            } catch (e) {}
        }

        if (transcriptUrl) {
            const xmlData = await page.evaluate(async (apiUrl) => {
                const resp = await fetch(apiUrl);
                return await resp.text();
            }, transcriptUrl);

            // 增强清理逻辑
            const cleanText = xmlData.replace(/<[^>]*>/g, ' ').replace(/\s+/g, ' ').trim();
            console.log('--- 深度抓取成功 (ID 校验通过) ---');
            console.log(JSON.stringify({ 
                videoId,
                transcript: cleanText.substring(0, 15000) 
            }, null, 2));
        } else {
            // 终极备选方案：直接抓取页面可见的描述文字
            const description = await page.evaluate(() => document.querySelector('#description-inline-expander')?.innerText || '');
            console.log('--- 字幕抓取失败，返回描述信息 ---');
            console.log(JSON.stringify({ videoId, description }, null, 2));
        }
    },
});

crawler.run([targetUrl]);


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (MEDIUM)