Skill: nowhitestar/web-crawl (version 1.0.0)

--- name: web-crawl version: "1.0.0" description: Advanced web crawling and content extraction tool with multiple extraction modes activation: keywords: ["crawl", "抓取", "提取网页", "研究", "深度研究", "research", "analyze website"] tags: ["web", "research", "crawling"] --- # Web Crawl Skill Advanced web content extraction with multiple modes and intelligent content detection. ## When to Use Use this skill when: - User asks to "研究" / "深度研究" a topic - User wants to "抓取" / "提取" content from websites - Need to analyze multiple web pages systematically - Current `web_fetch` output is insufficient ## Extraction Modes | Mode | Use Case | |------|----------| | `text` | Clean plain text | | `markdown` | Formatted Markdown (recommended) | | `links` | Extract all links | | `structured` | JSON metadata + content | | `full` | Markdown + links combined | ## Tools Available - `web_crawl` - Extract content from a single URL - `parallel_crawl` - Extract from multiple URLs in parallel - `research_topic` - Multi-step research with search + crawl ## Example Usage ``` User: "研究一下 OpenManus-Max 项目" → Use research_topic tool with query="OpenManus-Max GitHub features" ```

path: EXAMPLES.md
content: # Web Crawl Skill - Usage Examples

## Basic Usage

### 1. Crawl Single URL

```python
# In Python
import asyncio
from web_crawl import crawl_url

result = asyncio.run(crawl_url(
    url="https://example.com",
    mode="markdown",
    max_length=10000
))
print(result)
```

### 2. Parallel Crawl Multiple URLs

```python
from web_crawl import parallel_crawl

urls = [
    "https://site1.com/article",
    "https://site2.com/guide",
    "https://site3.com/docs",
]

result = asyncio.run(parallel_crawl(
    urls=urls,
    mode="markdown",
    max_length=8000
))
print(result)
```

## Deep Research Workflow

### Step 1: Search for Sources

```python
web_search:0 {
  "query": "OpenManus-Max features comparison",
  "count": 8
}
```

### Step 2: Crawl Top Results

```python
exec:1 {
  "command": "cd ~/.openclaw/workspace-main/skills/web-crawl && python3 -c \"import asyncio; from web_crawl import parallel_crawl; print(asyncio.run(parallel_crawl(['https://url1.com', 'https://url2.com'], mode='markdown', max_length=8000)))\""
}
```

### Step 3: Analyze and Synthesize

Use the crawled content to:
- Extract key findings
- Compare sources
- Identify unique insights
- Cite sources

## Extraction Modes Explained

| Mode | Best For | Output Format |
|------|----------|---------------|
| `text` | Quick reading | Plain text |
| `markdown` | Content creation | Formatted Markdown |
| `links` | Discovery | List of links |
| `structured` | Data extraction | JSON metadata |
| `full` | Complete analysis | Markdown + Links |

## Advanced: CSS Selector Targeting

```python
# Extract only the article content
result = asyncio.run(crawl_url(
    url="https://example.com/page",
    mode="markdown",
    selector="article.main-content"  # CSS selector
))
```

## Research Templates

### Product Research

```python
# Research a product
queries = [
    "ProductName review 2024",
    "ProductName vs competitor",
    "ProductName pricing features",
]

# 1. Search each query
# 2. Crawl top 3 results per query
# 3. Synthesize pros/cons, pricing, features
```

### Company Research

```python
# Research a company
queries = [
    "CompanyName funding valuation",
    "CompanyName leadership team",
    "CompanyName recent news",
]

# 1. Search for company info
# 2. Crawl official site + news sources
# 3. Extract funding, team, products
```

### Technology Research

```python
# Research a technology
queries = [
    "TechnologyName documentation",
    "TechnologyName getting started",
    "TechnologyName github examples",
]

# 1. Search for docs and tutorials
# 2. Crawl official docs + GitHub
# 3. Extract key concepts and examples
```

## Integration with OpenClaw

When user asks for research:

1. **Understand intent**: What type of research? (product, company, tech, topic)
2. **Generate search queries**: Use research.py templates
3. **Execute search**: Use web_search tool
4. **Crawl sources**: Use web_crawl tool
5. **Synthesize**: Analyze and present findings

### Example Session

**User**: "研究一下 OpenManus-Max"

**Assistant**:
```
我来为你深度研究 OpenManus-Max。首先搜索相关信息...

[Use web_search for multiple queries]

找到以下关键来源：
1. GitHub 官方仓库
2. 技术博客分析
3. 社区讨论

现在抓取详细内容...

[Use parallel_crawl on top URLs]

## 研究发现

### 核心特性
- ...

### 架构设计
- ...

### 与 OpenManus 对比
- ...

### 优势与局限
- ...

---
来源：
- [1] GitHub: OpenDemon/OpenManus-Max
- [2] 技术博客分析
- [3] 社区讨论
```


path: README.md
content: # 🕷️ Web Crawl Skill - 深度研究增强工具

## 已实现功能

✅ **多模式内容提取**
- `text` - 纯文本提取
- `markdown` - 完整 Markdown 转换（保留格式）
- `links` - 链接提取
- `structured` - JSON 结构化数据
- `full` - 综合提取

✅ **智能内容识别**
- 自动识别 `main`, `article`, `content` 等主要内容区域
- 自动清理脚本、样式、广告等元素

✅ **完整 HTML → Markdown 转换**
- 标题、段落、链接、图片、列表、表格
- 代码块、引用、分隔线

✅ **并行抓取**
- 同时抓取多个 URL
- 可配置并发数

## 使用方式

### 方法 1: 直接调用 Python 函数

```python
from skills.web-crawl.web_crawl import crawl_url, parallel_crawl

# 抓取单个页面
result = crawl_url(
    url="https://example.com",
    mode="markdown",
    max_length=10000
)
print(result)

# 并行抓取多个页面
result = parallel_crawl(
    urls=["https://site1.com", "https://site2.com"],
    mode="markdown",
    max_length=8000
)
print(result)
```

### 方法 2: 命令行

```bash
cd ~/.openclaw/workspace-main/skills/web-crawl

# 抓取单个 URL
python3 web_crawl.py https://example.com markdown 5000

# 可用模式: text, markdown, links, structured, full
```

### 方法 3: 作为 Agent 工具使用

在对话中，当你需要深度研究时，我会自动使用此工具：

**你**: "研究一下 OpenManus-Max"

**我**:
1. 使用 `web_search` 搜索相关来源
2. 使用 `parallel_crawl` 抓取前 5 个结果
3. 分析、整合、输出研究报告

## 深度研究工作流

```
用户提问
    ↓
[web_search] 搜索多个相关查询
    ↓
[parallel_crawl] 并行抓取 Top URLs
    ↓
[分析] 提取关键信息、对比、整合
    ↓
[输出] 结构化研究报告 + 来源引用
```

## 与 web_fetch 的对比

| 功能 | web_fetch (旧) | web_crawl (新) |
|------|----------------|----------------|
| 提取模式 | 2种 (text/markdown) | 5种 |
| CSS 选择器 | ❌ | ✅ |
| 智能正文识别 | 基础 | 高级 |
| 链接提取 | ❌ | ✅ |
| 结构化数据 | ❌ | JSON 输出 |
| 并行抓取 | ❌ | ✅ |
| 表格转换 | ⚠️ | ✅ 完整支持 |

## 示例输出

### Markdown 模式

```markdown
# Page Title

## Section Heading

This is **bold** and *italic* text.

- List item 1
- List item 2

[Link text](https://example.com)

| Column 1 | Column 2 |
|----------|----------|
| Data 1   | Data 2   |
```

### Structured 模式

```json
{
  "url": "https://example.com",
  "title": "Page Title",
  "description": "Meta description",
  "headings": [
    {"level": "h1", "text": "Main Title"},
    {"level": "h2", "text": "Section"}
  ],
  "main_text": "Content preview...",
  "links_count": 42,
  "images_count": 5
}
```

## 安装依赖

```bash
pip3 install requests beautifulsoup4
```

## 文件结构

```
skills/web-crawl/
├── SKILL.md          # Skill 定义文件
├── web_crawl.py      # 核心爬虫实现
├── research.py       # 深度研究工具
├── EXAMPLES.md       # 使用示例
└── README.md         # 本文档
```

---

**现在你可以要求我进行深度研究了！** 🚀


path: research.py
content: #!/usr/bin/env python3
"""
Deep Research Tool - Multi-step research with search + crawl

This tool combines Brave search with advanced crawling for comprehensive research.
"""

import asyncio
import json
from typing import Any, Dict, List, Optional

# Import the crawler
from web_crawl import WebCrawler, parallel_crawl


async def brave_search(query: str, count: int = 8) -> List[Dict[str, str]]:
    """
    Perform Brave search using OpenClaw's web_search tool
    
    Returns list of results with title, url, snippet
    """
    # This is a wrapper that will be called via OpenClaw's tool system
    # For now, return mock structure
    return []


async def research_topic(
    query: str,
    max_sources: int = 5,
    extract_mode: str = "markdown",
    max_content_length: int = 8000,
) -> str:
    """
    Perform deep research on a topic
    
    Workflow:
    1. Search for relevant URLs
    2. Crawl top results in parallel
    3. Synthesize findings
    
    Args:
        query: Research topic/question
        max_sources: Maximum number of sources to analyze
        extract_mode: Content extraction mode
        max_content_length: Max content length per source
    
    Returns:
        Comprehensive research report
    """
    # Step 1: Search (will be done via OpenClaw's web_search tool)
    # For now, we return instructions on how to use this
    
    instructions = f"""# Deep Research Instructions

To perform deep research on "{query}", follow these steps:

## Step 1: Search
Use OpenClaw's web_search tool to find relevant sources:
```
web_search:0 {{
  "query": "{query}",
  "count": {max_sources + 3}
}}
```

## Step 2: Crawl Sources
Use the parallel_crawl function on the top {max_sources} URLs:
```python
parallel_crawl(urls=[url1, url2, ...], mode="{extract_mode}", max_length={max_content_length})
```

## Step 3: Synthesize
Analyze the crawled content and provide:
- Key findings summary
- Important quotes/data points
- Source citations
- Any contradictions or gaps

---

**Tip**: For academic research, use mode="structured" to get metadata.
For content creation, use mode="markdown" for clean text.
"""
    
    return instructions


async def analyze_website(
    url: str,
    depth: int = 1,
    max_pages: int = 10,
) -> str:
    """
    Comprehensive website analysis
    
    Analyzes a website's structure, content, and linked pages
    """
    crawler = WebCrawler()
    
    # First, get the main page and extract links
    result = await crawler.crawl(url, mode="structured", max_length=50000)
    
    if not result["success"]:
        return f"❌ Failed to analyze {url}: {result.get('error')}"
    
    # Parse structured data to get links
    try:
        data = json.loads(result["content"])
        links_count = data.get("links_count", 0)
        title = data.get("title", "")
    except:
        links_count = 0
        title = ""
    
    report = f"""# Website Analysis: {url}

## Overview
- **Title**: {title}
- **Total Links**: {links_count}

## Main Page Content
{result["content"][:3000]}

---

**Note**: For deeper analysis (following linked pages), use parallel_crawl on specific URLs.
"""
    
    return report


# Research planning templates
RESEARCH_TEMPLATES = {
    "product": {
        "description": "Research a product or service",
        "search_queries": [
            "{topic} review",
            "{topic} features",
            "{topic} pricing",
            "{topic} vs competitors",
        ],
    },
    "company": {
        "description": "Research a company or organization",
        "search_queries": [
            "{topic} company profile",
            "{topic} funding",
            "{topic} leadership",
            "{topic} news",
        ],
    },
    "technology": {
        "description": "Research a technology or framework",
        "search_queries": [
            "{topic} documentation",
            "{topic} tutorial",
            "{topic} github",
            "{topic} best practices",
        ],
    },
    "person": {
        "description": "Research a person",
        "search_queries": [
            "{topic} biography",
            "{topic} achievements",
            "{topic} interviews",
        ],
    },
    "topic": {
        "description": "General topic research",
        "search_queries": [
            "{topic} guide",
            "{topic} explained",
            "{topic} latest developments",
        ],
    },
}


def get_research_plan(topic: str, research_type: str = "topic") -> str:
    """
    Generate a research plan for a given topic
    
    Args:
        topic: Research topic
        research_type: Type of research (product, company, technology, person, topic)
    
    Returns:
        Research plan with suggested search queries
    """
    template = RESEARCH_TEMPLATES.get(research_type, RESEARCH_TEMPLATES["topic"])
    
    queries = [q.format(topic=topic) for q in template["search_queries"]]
    
    plan = f"""# Research Plan: {topic}

**Type**: {template['description']}

## Suggested Search Queries
"""
    for i, query in enumerate(queries, 1):
        plan += f"{i}. `{query}`\n"
    
    plan += f"""
## Execution Steps

1. **Search Phase**: Run each query, collect top 3-5 URLs per query
2. **Crawl Phase**: Use `parallel_crawl` to extract content from all URLs
3. **Analysis Phase**: Synthesize findings, identify key insights
4. **Verification Phase**: Cross-reference important claims across sources

## Extraction Mode Recommendations
- **For data/numbers**: Use `structured` mode
- **For content/ideas**: Use `markdown` mode  
- **For link discovery**: Use `links` mode

---

Ready to start research? Run the search queries above!
"""
    
    return plan


if __name__ == "__main__":
    import sys
    
    if len(sys.argv) < 2:
        print("Usage: python research.py <command> [args]")
        print("Commands:")
        print("  plan <topic> [type]  - Generate research plan")
        print("  crawl <url> [mode]   - Crawl single URL")
        print("  analyze <url>        - Analyze website")
        sys.exit(1)
    
    command = sys.argv[1]
    
    if command == "plan":
        topic = sys.argv[2]
        rtype = sys.argv[3] if len(sys.argv) > 3 else "topic"
        print(get_research_plan(topic, rtype))
        
    elif command == "crawl":
        url = sys.argv[2]
        mode = sys.argv[3] if len(sys.argv) > 3 else "markdown"
        result = asyncio.run(crawl_url(url, mode))
        print(result)
        
    elif command == "analyze":
        url = sys.argv[2]
        result = asyncio.run(analyze_website(url))
        print(result)


path: web_crawl.py
content: #!/usr/bin/env python3
"""
OpenClaw Web Crawl Tool - Advanced Content Extraction
Enhanced version of web_fetch with multiple extraction modes

Features:
- Multiple extraction modes: text, markdown, links, structured, full
- CSS selector support for targeted extraction
- Intelligent main content detection
- Full HTML to Markdown conversion
- Parallel crawling capability
"""

import re
import json
import concurrent.futures
from typing import Any, Dict, List, Optional
from urllib.parse import urljoin, urlparse

try:
    import requests
    from bs4 import BeautifulSoup, NavigableString, Tag
except ImportError:
    raise ImportError("Required packages: requests, beautifulsoup4. Install: pip3 install requests beautifulsoup4")


class WebCrawler:
    """Advanced web content crawler and extractor"""
    
    def __init__(self, timeout: float = 30.0):
        self.timeout = timeout
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                         "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9,zh-CN;q=0.8",
            "Accept-Encoding": "gzip, deflate, br",
            "DNT": "1",
            "Connection": "keep-alive",
        }
    
    def crawl(
        self,
        url: str,
        mode: str = "markdown",
        max_length: int = 10000,
        selector: str = "",
    ) -> Dict[str, Any]:
        """
        Crawl and extract content from a URL
        
        Args:
            url: Target URL
            mode: Extraction mode (text, markdown, links, structured, full)
            max_length: Maximum content length
            selector: Optional CSS selector for targeted extraction
        
        Returns:
            Dict with success status and extracted content
        """
        try:
            resp = requests.get(url, headers=self.headers, timeout=self.timeout, allow_redirects=True)
            resp.raise_for_status()
            html = resp.text
            soup = BeautifulSoup(html, "html.parser")
            
            # Apply CSS selector if provided
            if selector:
                elements = soup.select(selector)
                if not elements:
                    return {
                        "success": False,
                        "error": f"No elements found for selector: {selector}",
                    }
                # Create new soup with selected elements
                new_soup = BeautifulSoup("<div></div>", "html.parser")
                container = new_soup.div
                for el in elements:
                    container.append(el.copy() if hasattr(el, 'copy') else el)
                soup = new_soup
            
            # Extract based on mode
            if mode == "text":
                result = self._extract_text(soup, max_length)
            elif mode == "markdown":
                result = self._extract_markdown(soup, url, max_length)
            elif mode == "links":
                result = self._extract_links(soup, url, max_length)
            elif mode == "structured":
                result = self._extract_structured(soup, url, max_length)
            elif mode == "full":
                result = self._extract_full(soup, url, max_length)
            else:
                result = self._extract_markdown(soup, url, max_length)
            
            return {
                "success": True,
                "url": url,
                "mode": mode,
                "content": result,
            }
                
        except requests.HTTPError as e:
            return {
                "success": False,
                "error": f"HTTP {e.response.status_code}: {url}",
            }
        except Exception as e:
            return {
                "success": False,
                "error": f"Crawl failed: {str(e)}",
            }
    
    def _clean_soup(self, soup: BeautifulSoup) -> BeautifulSoup:
        """Remove useless elements like scripts, styles, ads"""
        for tag in soup.find_all([
            "script", "style", "nav", "footer", "header", 
            "aside", "noscript", "iframe", "svg", "canvas"
        ]):
            tag.decompose()
        return soup
    
    def _extract_text(self, soup: BeautifulSoup, max_length: int) -> str:
        """Extract clean plain text"""
        soup = self._clean_soup(soup)
        text = soup.get_text(separator="\n", strip=True)
        # Clean excessive newlines
        text = re.sub(r"\n{3,}", "\n\n", text)
        return text[:max_length]
    
    def _extract_markdown(self, soup: BeautifulSoup, base_url: str, max_length: int) -> str:
        """Extract content as formatted Markdown"""
        soup = self._clean_soup(soup)
        lines = []
        
        # Extract title
        title = soup.find("title")
        if title:
            title_text = title.get_text(strip=True)
            if title_text:
                lines.append(f"# {title_text}\n")
        
        # Find main content area (smart detection)
        main = (
            soup.find("main")
            or soup.find("article")
            or soup.find(attrs={"role": "main"})
            or soup.find("div", class_=re.compile(r"content|article|post|entry|body", re.I))
            or soup.body
            or soup
        )
        
        if main:
            lines.append(self._tag_to_markdown(main, base_url))
        
        result = "\n".join(lines)
        result = re.sub(r"\n{3,}", "\n\n", result)
        return result[:max_length]
    
    def _tag_to_markdown(self, tag: Tag, base_url: str) -> str:
        """Recursively convert HTML tags to Markdown"""
        parts = []
        
        for child in tag.children:
            if isinstance(child, NavigableString):
                text = str(child).strip()
                if text:
                    parts.append(text)
                    
            elif isinstance(child, Tag):
                name = child.name
                
                if name in ("h1", "h2", "h3", "h4", "h5", "h6"):
                    level = int(name[1])
                    text = child.get_text(strip=True)
                    if text:
                        parts.append(f"\n{'#' * level} {text}\n")
                        
                elif name == "p":
                    text = child.get_text(strip=True)
                    if text:
                        parts.append(f"\n{text}\n")
                        
                elif name == "a":
                    text = child.get_text(strip=True)
                    href = child.get("href", "")
                    if href and text and not href.startswith(("#", "javascript:")):
                        full_url = urljoin(base_url, href)
                        parts.append(f"[{text}]({full_url})")
                    elif text:
                        parts.append(text)
                        
                elif name == "img":
                    alt = child.get("alt", "image")
                    src = child.get("src", "")
                    if src:
                        full_url = urljoin(base_url, src)
                        parts.append(f"![{alt}]({full_url})")
                        
                elif name in ("ul", "ol"):
                    items = child.find_all("li", recursive=False)
                    for i, li in enumerate(items):
                        prefix = f"{i+1}." if name == "ol" else "-"
                        text = li.get_text(strip=True)
                        if text:
                            parts.append(f"\n{prefix} {text}")
                    parts.append("")
                    
                elif name in ("pre", "code"):
                    code = child.get_text()
                    if "\n" in code:
                        parts.append(f"\n```\n{code}\n```\n")
                    else:
                        parts.append(f"`{code.strip()}`")
                        
                elif name == "blockquote":
                    text = child.get_text(strip=True)
                    if text:
                        parts.append(f"\n> {text}\n")
                        
                elif name == "table":
                    parts.append(self._table_to_markdown(child))
                    
                elif name in ("strong", "b"):
                    text = child.get_text(strip=True)
                    if text:
                        parts.append(f"**{text}**")
                        
                elif name in ("em", "i"):
                    text = child.get_text(strip=True)
                    if text:
                        parts.append(f"*{text}*")
                        
                elif name == "br":
                    parts.append("\n")
                    
                elif name == "hr":
                    parts.append("\n---\n")
                    
                else:
                    # Recursively process other tags
                    inner = self._tag_to_markdown(child, base_url)
                    if inner.strip():
                        parts.append(inner)
        
        return " ".join(parts)
    
    def _table_to_markdown(self, table: Tag) -> str:
        """Convert HTML table to Markdown table"""
        rows = table.find_all("tr")
        if not rows:
            return ""
        
        md_rows = []
        for row in rows:
            cells = row.find_all(["th", "td"])
            md_cells = [cell.get_text(strip=True).replace("|", "\\|") for cell in cells]
            if md_cells:
                md_rows.append("| " + " | ".join(md_cells) + " |")
        
        if len(md_rows) > 0:
            # Add separator row
            num_cols = md_rows[0].count("|") - 1
            separator = "| " + " | ".join(["---"] * max(num_cols, 1)) + " |"
            md_rows.insert(1, separator)
        
        return "\n" + "\n".join(md_rows) + "\n"
    
    def _extract_links(self, soup: BeautifulSoup, base_url: str, max_length: int) -> str:
        """Extract all links from the page"""
        links = []
        
        for a in soup.find_all("a", href=True):
            href = a["href"]
            text = a.get_text(strip=True)
            
            if href.startswith(("#", "javascript:", "mailto:", "tel:")):
                continue
                
            full_url = urljoin(base_url, href)
            links.append(f"- [{text or 'link'}]({full_url})")
        
        result = f"Found {len(links)} links:\n\n" + "\n".join(links)
        return result[:max_length]
    
    def _extract_structured(self, soup: BeautifulSoup, base_url: str, max_length: int) -> str:
        """Extract structured data as JSON"""
        data = {
            "url": base_url,
            "title": "",
            "description": "",
            "headings": [],
            "main_text": "",
            "links_count": 0,
            "images_count": 0,
        }
        
        # Title
        title = soup.find("title")
        if title:
            data["title"] = title.get_text(strip=True)
        
        # Meta description
        meta_desc = soup.find("meta", attrs={"name": "description"})
        if meta_desc:
            data["description"] = meta_desc.get("content", "")
        
        # Headings
        for h in soup.find_all(["h1", "h2", "h3"]):
            text = h.get_text(strip=True)
            if text:
                data["headings"].append({"level": h.name, "text": text})
        
        # Counts
        data["links_count"] = len(soup.find_all("a", href=True))
        data["images_count"] = len(soup.find_all("img"))
        
        # Main text preview
        soup_clean = self._clean_soup(soup)
        data["main_text"] = soup_clean.get_text(separator=" ", strip=True)[:3000]
        
        return json.dumps(data, indent=2, ensure_ascii=False)[:max_length]
    
    def _extract_full(self, soup: BeautifulSoup, base_url: str, max_length: int) -> str:
        """Full extraction: Markdown + Links"""
        md = self._extract_markdown(soup, base_url, max_length // 2)
        links = self._extract_links(soup, base_url, max_length // 4)
        return f"{md}\n\n---\n\n{links}"[:max_length]


def crawl_url(
    url: str,
    mode: str = "markdown",
    max_length: int = 10000,
    selector: str = "",
) -> str:
    """
    Crawl a single URL and return formatted result
    
    Use this for extracting content from web pages.
    """
    crawler = WebCrawler()
    result = crawler.crawl(url, mode, max_length, selector)
    
    if result["success"]:
        return f"✅ Crawled: {result['url']}\n\n{result['content']}"
    else:
        return f"❌ Failed: {result['error']}"


def parallel_crawl(
    urls: List[str],
    mode: str = "markdown",
    max_length: int = 10000,
    max_workers: int = 5,
) -> str:
    """
    Crawl multiple URLs in parallel
    
    Use this for researching multiple sources at once.
    """
    crawler = WebCrawler()
    
    def crawl_one(url: str) -> Dict[str, Any]:
        return crawler.crawl(url, mode, max_length)
    
    results = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        future_to_url = {executor.submit(crawl_one, url): url for url in urls}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                results.append(future.result())
            except Exception as e:
                results.append({"success": False, "error": str(e), "url": url})
    
    # Format results
    output = [f"# Parallel Crawl Results ({len(urls)} URLs)\n"]
    
    for i, result in enumerate(results, 1):
        url = result.get("url", "Unknown")
        output.append(f"\n---\n\n## Source {i}: {url}\n")
        
        if result.get("success"):
            content = result.get("content", "")
            # Truncate if too long
            if len(content) > max_length:
                content = content[:max_length] + "\n\n[Content truncated...]"
            output.append(content)
        else:
            output.append(f"❌ Failed: {result.get('error', 'Unknown error')}")
    
    return "\n".join(output)


# CLI interface
if __name__ == "__main__":
    import sys
    
    if len(sys.argv) < 2:
        print("Usage: python web_crawl.py <url> [mode] [max_length]")
        print("Modes: text, markdown, links, structured, full")
        sys.exit(1)
    
    url = sys.argv[1]
    mode = sys.argv[2] if len(sys.argv) > 2 else "markdown"
    max_length = int(sys.argv[3]) if len(sys.argv) > 3 else 10000
    
    result = crawl_url(url, mode, max_length)
    print(result)


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)