Skill: seckho1985/xhs-search-summarizer-seckho (version 1.0.3)

--- name: xiaohongshu-search-summarizer description: Searches Xiaohongshu(小红书) for a given keyword, extracts the top N posts (including texts, images, and user comments), and then synthesizes a comprehensive final analytical report. Use this skill whenever the user wants to search Xiaohongshu for a specific topic, compile research seamlessly combining text and images, or needs an aggregated thematic summary of social media posts, comments, and visual data on a given subject. compatibility: Requires playwright-cli and python3, and the Python 'requests' package installed. --- # Xiaohongshu Search and Summarize This skill automates the process of extracting high-quality multi-modal content (text + images) from Xiaohongshu (小红书) and actively assists you in generating a deeply integrated, analytical final report for the user. Due to Xiaohongshu's aggressive anti-scraping mechanisms, direct HTTP requests or naive scraping often result in 404s or blocks. This skill natively bypasses these by simulating a real user through the `playwright-cli` in a headed browser window. It operates in two distinct phases: ### Phase 1: Subagent Data Collection 1. Simulate a search for the keyword on Xiaohongshu in a headed browser. 2. Advance through image sliders to fully load all lazy pictures from the top N posts. 3. Extract titles, descriptions, top comments, and all high-resolution images. 4. Download those images to a local directory and generate a raw data document (`[keyword]_raw_data.md`). ### Phase 2: AI Multi-Modal Synthesis (Your Job) 5. **You** MUST use your file reading capabilities to read the `[keyword]_raw_data.md` file. 6. Inside the raw data markdown, you will find paths to image files. **You MUST use your file reading / vision capabilities on these image file paths** to actually ingest and "see" their visual content. If you skip this step, you are only reading file names, not the images themselves! 7. **You** analyze the texts, summarize the genuinely useful comments (discarding noise like "pm me"), and interpret the semantic content of the images you just viewed (e.g. diagrams, guidelines, step-by-step UI flows). 8. **You** compile everything into a beautifully synthesized, single comprehensive report rather than just a linear list of posts. ### Dependencies - `playwright-cli` (Must be available on the path) - `python3` (Required to download images and stitch the raw data markdown) - `requests` Python package (`pip install requests`) — used by `parse.py` to download images --- ## Usage Instructions ### Step 1: Run the Extraction Script Execute the wrapper script in `scripts/run.sh`. It accepts the following arguments: ```bash /bin/bash <skill_dir>/scripts/run.sh "YOUR KEYWORD" <MAX_POSTS> <OUTPUT_DIRECTORY> ``` - **`YOUR KEYWORD`**: The search term to look up on Xiaohongshu. - **`<MAX_POSTS>`**: (Optional, default = 10) The number of top posts to scan. - **`<OUTPUT_DIRECTORY>`**: (Optional, default = `./`) Directory where the raw data and images will be saved. **Example execution**: ```bash /bin/bash ~/.claude/skills/xiaohongshu-search-summarizer/scripts/run.sh "openclaw使用场景" 10 "./xhs_report_openclaw_scenarios" ``` ### Step 2: Read Raw Data & Images Once the bash script finishes successfully, navigate to the `OUTPUT_DIRECTORY` and use your file reading capabilities to ingest the generated `[keyword]_raw_data.md` file. Inside this file, you will find descriptions, comments, and file paths pointing to `post_X_img_Y.webp` or `post_X_img_Y.jpg`. ### Step 3: Synthesis & Summarization **This is the most critical step.** Do not just return the raw markdown file to the user. Instead, write a polished comprehensive markdown report that reorganizes the information logically, while retaining a high level of detail. Follow these strict compilation rules: - **Do not list posts individually** (e.g. avoid "Post 1: ... Post 2: ..."). - **Read the Images:** You MUST use your file reading and vision capabilities on the `.webp` or `.jpg` image files found in the raw data directory to interpret their contents. - **Detailed & Comprehensive Synthesis:** Provide a highly detailed summary that includes diverse viewpoints, nuances, and specific examples found across different posts. Avoid over-summarizing or losing important context; preserve the richness and diversity of the information. - **Extract and merge themes:** Group ideas by concepts, steps, recurring themes, or pros/cons. - **Evaluate comments:** Merge insights from valuable comments directly into the core narrative. Skip useless or repetitive comments, but preserve diverse opinions or helpful counter-arguments from the comments section. - **Integrate images contextually:** Embed the most relevant and high-quality images directly into the flow of your final report to support the analytical points being made. Describe their visual meaning based on what you saw with your vision capabilities. - **Save to OUTPUT_DIRECTORY:** Save your beautifully compiled final Markdown report using your file writing capabilities directly into the same `<OUTPUT_DIRECTORY>` as the raw data (e.g., `<OUTPUT_DIRECTORY>/[keyword]_synthesis.md`), and give the user the path to it. ### Error Handling If you encounter `404 Not Found` or "element not visible" errors during the browser invocation: - Keep in mind that Xiaohongshu may demand a login challenge. If the site pauses waiting for a login, instruct the user to verify the `playwright-cli` browser window and perform necessary authentication manually, then try the script again.

path: _meta.json
content: ownerId: kn741vcs55cd5vbj3hbm7wktzx82hz1j
slug: xiaohongshu-search-summarizer
version: 1.0.3
publishedAt: 1773129163624

path: scripts/parse.py
content: import json
import os
import sys
import requests
import re
from urllib.parse import urlparse


ALLOWED_IMAGE_DOMAINS = (".xhscdn.com", ".xiaohongshu.com")


def is_safe_image_url(url):
    """Validate that an image URL is from a known Xiaohongshu domain and uses HTTPS."""
    try:
        parsed = urlparse(url)
        if parsed.scheme != "https":
            return False
        hostname = parsed.hostname or ""
        return any(hostname.endswith(domain) for domain in ALLOWED_IMAGE_DOMAINS)
    except Exception:
        return False

if len(sys.argv) < 4:
    print("Usage: parse.py <keyword> <json_file_path> <output_dir>")
    sys.exit(1)

keyword = sys.argv[1]
json_path = sys.argv[2]
output_dir = sys.argv[3]

try:
    with open(json_path, 'r', encoding='utf-8') as f:
        text = f.read()
except FileNotFoundError:
    print(f"File {json_path} not found.")
    sys.exit(1)

if "### Ran Playwright code" in text:
    text = text.split("### Ran Playwright code")[0]
if "### Result" in text:
    text = text.split("### Result")[1].strip()

# remove leading/trailing quotes if stringified
text = text.strip()
if text.startswith('"') and text.endswith('"'):
    # unescape the string first
    import ast
    try:
        text = ast.literal_eval(text)
    except:
        pass

try:
    data = json.loads(text)
except json.JSONDecodeError as e:
    print(f"JSON decode failed: {e}")
    sys.exit(1)

os.makedirs(output_dir, exist_ok=True)

md_filename = f"{keyword.replace(' ', '_')}_raw_data.md"
md_path = os.path.join(output_dir, md_filename)

output_md = f"# 小红书「{keyword}」搜索原始数据提取\n\n"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)"
}

count = 0
for i, post in enumerate(data):
    if not isinstance(post, dict):
        continue
    
    count += 1
    title = post.get("title", f"帖子 {i+1}")
    desc = post.get("desc", "")
    images = post.get("images", [])
    
    output_md += f"## {count}. {title}\n\n"
    
    desc_lines = desc.split('\\n') if '\\n' in desc else desc.split('\n')
    for line in desc_lines:
        line = line.strip()
        if not line: continue
        if line.startswith('#'): continue
        output_md += f"> {line}\n"
    output_md += "\n"
    
    comments = post.get("comments", [])
    if comments:
        output_md += "**💬 Top 评论：**\n"
        for c in comments:
            user = c.get("user", "User")
            content = c.get("content", "").replace('\n', ' ')
            output_md += f"- **{user}**: {content}\n"
        output_md += "\n"
    
    for j, img_url in enumerate(images):
        if not is_safe_image_url(img_url):
            print(f"Skipping untrusted image URL: {img_url}")
            continue
        try:
            res = requests.get(img_url, headers=headers, timeout=10, allow_redirects=False)
            if res.status_code == 200:
                img_ext = "jpg"
                if "webp" in img_url: img_ext = "webp"
                
                img_filename = f"post_{count}_img_{j+1}.{img_ext}"
                img_path = os.path.join(output_dir, img_filename)
                
                with open(img_path, 'wb') as img_f:
                    img_f.write(res.content)
                
                output_md += f"![图 {j+1}]({img_path})\n"
        except Exception as e:
            print(f"Error fetching {img_url}: {e}")
            
    output_md += "\n---\n\n"

with open(md_path, 'w', encoding='utf-8') as f:
    f.write(output_md)
    
print(f"Report generated successfully at: {md_path}")


path: scripts/run.sh
content: #!/bin/bash
KEYWORD="$1"
MAX_POSTS="${2:-10}"
OUTPUT_DIR="${3:-.}"

if [ -z "$KEYWORD" ]; then
    echo "Usage: ./run.sh <keyword> [max_posts] [output_dir]"
    exit 1
fi

# Validate MAX_POSTS is a positive integer within bounds
if ! [[ "$MAX_POSTS" =~ ^[0-9]+$ ]] || [ "$MAX_POSTS" -lt 1 ] || [ "$MAX_POSTS" -gt 50 ]; then
    echo "Error: max_posts must be a positive integer between 1 and 50."
    exit 1
fi

mkdir -p "$OUTPUT_DIR"

SCRIPT_PATH="/tmp/xhs_scrape_$$.js"

export KEYWORD MAX_POSTS
cat << 'EOF' > "$SCRIPT_PATH"
async page => {
    const keyword = encodeURIComponent(process.env.KEYWORD || "");
    const maxPosts = parseInt(process.env.MAX_POSTS || "10", 10);
    await page.goto(\`https://www.xiaohongshu.com/search_result?keyword=\${keyword}&source=web_explore_feed\`, { waitUntil: 'domcontentloaded' });
    await page.waitForTimeout(4000); 

    const maxPostsLimit = 10;
    const results = [];
    
    // get the note items covers
    const items = page.locator('.note-item a.cover').filter({ hasNotText: '' }); 
    const count = await items.count();
    
    for (let i = 0; i < Math.min(count, maxPosts); i++) {
        try {
            await page.evaluate((index) => {
                const links = document.querySelectorAll('.note-item a.cover');
                if (links[index]) links[index].removeAttribute('target');
            }, i);
            await items.nth(i).click({ force: true, timeout: 5000 });
        } catch(e) { continue; }
        
        // Wait for modal detail to appear
        try {
            await page.waitForSelector('#detail-title', { timeout: 3000 });
        } catch(e) {}
        await page.waitForTimeout(1000);
        
        // Click through sliders
        try {
            let nextBtn = page.locator('.swiper-button-next');
            while(await nextBtn.isVisible() && !await nextBtn.isDisabled()) {
                await nextBtn.click({ force: true });
                await page.waitForTimeout(300);
            }
        } catch(e) {}

        const data = await page.evaluate(() => {
            const elTitle = document.querySelector('#detail-title');
            const title = elTitle ? elTitle.innerText : '';
            const elDesc = document.querySelector('#detail-desc');
            const desc = elDesc ? elDesc.innerText : '';
            
            let imgUrls = [];
            const swiperImgs = document.querySelectorAll('.note-slider .swiper-slide:not(.swiper-slide-duplicate) img');
            swiperImgs.forEach(img => {
                if (img.src && !img.src.includes('avatar') && !imgUrls.includes(img.src)) {
                    imgUrls.push(img.src);
                }
            });
            
            if (imgUrls.length === 0) {
                 const singleImg = document.querySelector('.media-container img');
                 if (singleImg && singleImg.src && !singleImg.src.includes('avatar')) {
                     imgUrls.push(singleImg.src);
                 }
            }
            
            // Extract comments
            let commentsRes = [];
            let commentEls = document.querySelectorAll('.comment-item');
            if (commentEls.length === 0) {
                commentEls = document.querySelectorAll('[class*="commentItem"]');
            }
            if (commentEls.length === 0) {
                commentEls = document.querySelectorAll('.parent-comment');
            }
            
            commentEls.forEach(el => {
                let userEl = el.querySelector('.name');
                let user = userEl ? userEl.innerText : 'User';
                let contentEl = el.querySelector('.content');
                let content = contentEl ? contentEl.innerText.trim() : '';
                
                if(!content){
                    // Fallback cleanup
                    content = el.innerText.replace(user, '').trim();
                    content = content.replace(/^[:：]/, '').trim().substring(0, 150);
                }
                
                if (content && commentsRes.length < 5) {
                    commentsRes.push({ user, content });
                }
            });
            
            return { title, desc, images: imgUrls, comments: commentsRes };
        });
        
        if (data.title || data.desc || data.images.length > 0) {
            results.push(data);
        }
        
        try {
            const closeBtn = page.locator('.close-box');
            if (await closeBtn.count() > 0) {
                await closeBtn.first().click({ force: true, timeout: 2000 });
            } else {
                 throw new Error("No close box");
            }
        } catch (e) {
            await page.keyboard.press('Escape');
            await page.waitForTimeout(1000); 
            if (page.url().includes("/explore/")) {
                await page.goBack();
            }
        }
        await page.waitForTimeout(1500);
    }
    
    return JSON.stringify(results, null, 2);
}
EOF

echo "Running Xiaohongshu scraper for keyword: $KEYWORD"
npx @playwright/cli run-code "$(< "$SCRIPT_PATH")" > "$OUTPUT_DIR/xhs_data.json"

echo "Parsing results and downloading images..."
python3 "$(dirname "$0")/parse.py" "$KEYWORD" "$OUTPUT_DIR/xhs_data.json" "$OUTPUT_DIR"
rm -f "$SCRIPT_PATH"
echo "Raw markdown and artifacts saved to $OUTPUT_DIR"


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)