Skill: 24k-handsomer/webpage-reader-skill (version 0.0.2)

# OpenClaw 网页下载器技能 ## 技能描述 网页读取器技能是一个强大的工具，允许您使用Google Chrome的无头浏览器读取和分析网页内容。此技能可以： - 检查系统是否安装了Google Chrome - 如果未找到Chrome，自动尝试安装（在支持的平台上） - 使用Chrome的无头模式和优化参数下载网页内容 - 读取和处理下载的HTML内容 - 生成网页内容摘要 - 安全处理临时文件以保护您的隐私 ## 安装指南 ### 先决条件 - Python 3.8或更高版本 - Google Chrome浏览器（将被自动检测，如果缺少将提供安装协助） ### 安装步骤 1. **在OpenClaw中安装技能**： - 打开OpenClaw - 进入技能管理器 - 点击"添加技能" - 选择您下载此技能的目录 - 点击"安装" ### 平台特定说明 - **Windows**：Chrome安装需要从[Google Chrome](https://www.google.com/chrome/)手动下载 - **macOS**：自动安装需要Homebrew。如果未安装Homebrew，需要手动安装。 - **Linux**：支持在Ubuntu/Debian和Fedora/CentOS/RHEL发行版上自动安装。对于其他发行版，需要手动安装。 ## 使用示例 ### 基本用法 ```python from webpage_reader import main result = main("https://example.com") if result['success']: print("网页下载成功！") print("摘要：") print(result['summary']) print("\n内容预览：") print(result['content'][:500] + "..." if len(result['content']) > 500 else result['content']) else: print(f"错误：{result['message']}") ``` ### 命令行用法 ```bash python webpage_reader.py https://example.com ``` ### OpenClaw界面用法 1. 打开OpenClaw 2. 选择网页读取器技能 3. 在输入字段中输入URL 4. 点击"运行" 5. 在输出面板中查看结果 ## 技术详情 ### Chrome命令参数 技能使用以下Chrome命令参数以获得最佳性能： ```bash google-chrome --headless=new --no-sandbox --disable-gpu --disable-dev-shm-usage --virtual-time-budget=8000 --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" --hide-scrollbars --blink-settings=imagesEnabled=true --dump-dom <url> ``` ### 输出格式 技能返回具有以下结构的字典： ```python { "success": bool, # 操作是否成功 "message": str, # 状态消息 "content": str, # 网页的完整HTML内容 "summary": str # 网页内容摘要 } ``` ## 故障排除 ### 常见问题 1. **未找到Chrome** - **解决方案**：从[https://www.google.com/chrome/](https://www.google.com/chrome/)手动安装Google Chrome 2. **权限错误** - **解决方案**：以适当的权限运行技能，尤其是在Linux上安装Chrome时 3. **超时错误** - **解决方案**：技能有60秒的超时。对于大型网页，这可能不够。您可以在`download_webpage`函数中修改超时时间。 4. **内容为空** - **解决方案**：检查URL是否可访问，且未被CAPTCHA或其他反爬措施阻止 5. **编码错误** - **解决方案**：技能使用UTF-8编码。对于使用不同编码的网页，您可能需要修改`read_webpage_content`函数中的编码处理。 ### 日志记录 技能生成详细的日志以帮助诊断问题。日志默认输出到控制台，但可以配置为写入文件（如果需要）。 ## 贡献 欢迎贡献！请随时提交Pull Request。 ## 许可证 此技能以MIT许可证发布。有关详细信息，请参阅LICENSE文件。 ## 支持 如果您遇到任何问题或有疑问，请在GitHub存储库上打开一个issue。

path: README.md
content: # OpenClaw 网页下载器技能

## 技能描述

网页读取器技能是一个强大的工具，允许您使用Google Chrome的无头浏览器读取和分析网页内容。此技能可以：

- 检查系统是否安装了Google Chrome
- 如果未找到Chrome，自动尝试安装（在支持的平台上）
- 使用Chrome的无头模式和优化参数下载网页内容
- 读取和处理下载的HTML内容
- 生成网页内容摘要
- 安全处理临时文件以保护您的隐私

## 安装指南

### 先决条件

- Python 3.8或更高版本
- Google Chrome浏览器（将被自动检测，如果缺少将提供安装协助）

### 安装步骤

1. **在OpenClaw中安装技能**：
   - 打开OpenClaw
   - 进入技能管理器
   - 点击"添加技能"
   - 选择您下载此技能的目录
   - 点击"安装"

### 平台特定说明

- **Windows**：Chrome安装需要从[Google Chrome](https://www.google.com/chrome/)手动下载
- **macOS**：自动安装需要Homebrew。如果未安装Homebrew，需要手动安装。
- **Linux**：支持在Ubuntu/Debian和Fedora/CentOS/RHEL发行版上自动安装。对于其他发行版，需要手动安装。

## 使用示例

### 基本用法

```python
from webpage_reader import main

result = main("https://example.com")

if result['success']:
    print("网页下载成功！")
    print("摘要：")
    print(result['summary'])
    print("\n内容预览：")
    print(result['content'][:500] + "..." if len(result['content']) > 500 else result['content'])
else:
    print(f"错误：{result['message']}")
```

### 命令行用法

```bash
python webpage_reader.py https://example.com
```

### OpenClaw界面用法

1. 打开OpenClaw
2. 选择网页读取器技能
3. 在输入字段中输入URL
4. 点击"运行"
5. 在输出面板中查看结果

## 技术详情

### Chrome命令参数

技能使用以下Chrome命令参数以获得最佳性能：

```bash
google-chrome --headless=new --no-sandbox --disable-gpu --disable-dev-shm-usage --virtual-time-budget=8000 --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" --hide-scrollbars --blink-settings=imagesEnabled=true --dump-dom <url>
```

### 输出格式

技能返回具有以下结构的字典：

```python
{
    "success": bool,          # 操作是否成功
    "message": str,           # 状态消息
    "content": str,           # 网页的完整HTML内容
    "summary": str            # 网页内容摘要
}
```

## 故障排除

### 常见问题

1. **未找到Chrome**
   - **解决方案**：从[https://www.google.com/chrome/](https://www.google.com/chrome/)手动安装Google Chrome

2. **权限错误**
   - **解决方案**：以适当的权限运行技能，尤其是在Linux上安装Chrome时

3. **超时错误**
   - **解决方案**：技能有60秒的超时。对于大型网页，这可能不够。您可以在`download_webpage`函数中修改超时时间。

4. **内容为空**
   - **解决方案**：检查URL是否可访问，且未被CAPTCHA或其他反爬措施阻止

5. **编码错误**
   - **解决方案**：技能使用UTF-8编码。对于使用不同编码的网页，您可能需要修改`read_webpage_content`函数中的编码处理。

### 日志记录

技能生成详细的日志以帮助诊断问题。日志默认输出到控制台，但可以配置为写入文件（如果需要）。

## 贡献

欢迎贡献！请随时提交Pull Request。

## 许可证

此技能以MIT许可证发布。有关详细信息，请参阅LICENSE文件。

## 支持

如果您遇到任何问题或有疑问，请在GitHub存储库上打开一个issue。

path: README_en.md
content: # Webpage Downloader Skill for OpenClaw

## Skill Description

The Webpage Deader skill is a powerful tool that allows you to download and analyze webpage content using Google Chrome's headless browser. This skill can:

- Check if Google Chrome is installed on your system
- Automatically attempt to install Chrome if it's not found (on supported platforms)
- Download webpage content using Chrome's headless mode with optimized parameters
- Read and process the downloaded HTML content
- Generate a summary of the webpage content
- Safely handle temporary files to protect your privacy

## Installation Guide

### Prerequisites

- Python 3.8 or higher
- Google Chrome browser (will be automatically detected, with installation assistance provided if missing)

### Installation Steps

1. **Install the skill** in OpenClaw:
   - Open OpenClaw
   - Go to Skills Manager
   - Click "Add Skill"
   - Select the directory where you downloaded this skill
   - Click "Install"

### Platform-Specific Notes

- **Windows**: Chrome installation requires manual download from [Google Chrome](https://www.google.com/chrome/)
- **macOS**: Requires Homebrew for automatic installation. If Homebrew is not installed, manual installation is required.
- **Linux**: Supports automatic installation on Ubuntu/Debian and Fedora/CentOS/RHEL distributions. For other distributions, manual installation is required.

## Usage Examples

### Basic Usage

```python
from webpage_reader import main

result = main("https://example.com")

if result['success']:
    print("Webpage downloaded successfully!")
    print("Summary:")
    print(result['summary'])
    print("\nContent preview:")
    print(result['content'][:500] + "..." if len(result['content']) > 500 else result['content'])
else:
    print(f"Error: {result['message']}")
```

### Command Line Usage

```bash
python webpage_reader.py https://example.com
```

### OpenClaw Interface Usage

1. Open OpenClaw
2. Select the Webpage Downloader skill
3. Enter the URL in the input field
4. Click "Run"
5. View the results in the output panel

## Technical Details

### Chrome Command Parameters

The skill uses the following Chrome command parameters for optimal performance:

```bash
google-chrome --headless=new --no-sandbox --disable-gpu --disable-dev-shm-usage --virtual-time-budget=8000 --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" --hide-scrollbars --blink-settings=imagesEnabled=true --dump-dom <url>
```

### Output Format

The skill returns a dictionary with the following structure:

```python
{
    "success": bool,          # Whether the operation succeeded
    "message": str,           # Status message
    "content": str,           # Full HTML content of the webpage
    "summary": str            # Summary of the webpage content
}
```

## Troubleshooting

### Common Issues

1. **Chrome not found**
   - **Solution**: Install Google Chrome manually from [https://www.google.com/chrome/](https://www.google.com/chrome/)

2. **Permission errors**
   - **Solution**: Run the skill with appropriate permissions, especially when installing Chrome on Linux

3. **Timeout errors**
   - **Solution**: The skill has a 60-second timeout. For large webpages, this may not be sufficient. You can modify the timeout in the `download_webpage` function.

4. **Empty content**
   - **Solution**: Check if the URL is accessible and not blocked by CAPTCHA or other anti-scraping measures

5. **Encoding errors**
   - **Solution**: The skill uses UTF-8 encoding. For webpages with different encodings, you may need to modify the encoding handling in the `read_webpage_content` function.

### Logging

The skill generates detailed logs to help diagnose issues. Logs are output to the console by default, but can be configured to write to a file if needed.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This skill is released under the MIT License. See the LICENSE file for details.

## Support

If you encounter any issues or have questions, please open an issue on the GitHub repository.

path: skill.json
content: {
  "name": "webpage_reader",
  "version": "1.0.0",
  "description": "Downloads webpage content using Google Chrome headless browser and returns the content with a summary",
  "author": "OpenClaw Team",
  "license": "MIT",
  "main": "webpage_reader.py",
  "entry_point": "main",
  "args": [
    {
      "name": "url",
      "type": "string",
      "required": true,
      "description": "The URL of the webpage to download"
    }
  ],
  "returns": {
    "type": "object",
    "properties": {
      "success": {
        "type": "boolean",
        "description": "Whether the operation was successful"
      },
      "message": {
        "type": "string",
        "description": "Status message"
      },
      "content": {
        "type": "string",
        "description": "The downloaded webpage content"
      },
      "summary": {
        "type": "string",
        "description": "Summary of the webpage content"
      }
    }
  },
  "dependencies": {
    "python": ">=3.8"
  },
  "platforms": [
    "Windows",
    "macOS",
    "Linux"
  ],
  "keywords": [
    "webpage",
    "download",
    "chrome",
    "headless",
    "scraping"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/openclaw/webpage-reader-skill"
  }
}

path: webpage_reader.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
OpenClaw Skill: Webpage Downloader

This skill downloads webpage content using Google Chrome headless browser.
"""

import os
import sys
import subprocess
import platform
import logging
import tempfile
import shutil
from typing import Optional, Dict, Any

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger('webpage_downloader')

def check_chrome_installed() -> bool:
    """
    Check if Google Chrome is installed on the system.
    
    Returns:
        bool: True if Chrome is installed, False otherwise
    """
    try:
        if platform.system() == 'Windows':
            # Check common Chrome paths on Windows
            chrome_paths = [
                os.path.join(os.environ.get('PROGRAMFILES', ''), 'Google', 'Chrome', 'Application', 'chrome.exe'),
                os.path.join(os.environ.get('PROGRAMFILES(X86)', ''), 'Google', 'Chrome', 'Application', 'chrome.exe'),
                os.path.join(os.environ.get('LOCALAPPDATA', ''), 'Google', 'Chrome', 'Application', 'chrome.exe')
            ]
            for path in chrome_paths:
                if os.path.exists(path):
                    logger.info(f"Chrome found at: {path}")
                    return True
            return False
        else:
            # For macOS and Linux, use which command
            result = subprocess.run(
                ['which', 'google-chrome'] if platform.system() != 'Darwin' else ['which', 'chrome'],
                capture_output=True,
                text=True
            )
            if result.returncode == 0:
                logger.info(f"Chrome found at: {result.stdout.strip()}")
                return True
            return False
    except Exception as e:
        logger.error(f"Error checking Chrome installation: {e}")
        return False

def install_chrome() -> bool:
    """
    Install Google Chrome based on the operating system.
    
    Returns:
        bool: True if installation successful, False otherwise
    """
    try:
        system = platform.system()
        logger.info(f"Installing Chrome on {system}")
        
        if system == 'Windows':
            # Windows installation logic
            logger.info("Please manually download and install Chrome from: https://www.google.com/chrome/")
            return False
        elif system == 'Darwin':  # macOS
            # macOS installation using Homebrew
            try:
                subprocess.run(['brew', 'install', 'google-chrome'], check=True)
                return True
            except subprocess.CalledProcessError:
                logger.error("Homebrew not found. Please install Homebrew first or manually install Chrome.")
                return False
        elif system == 'Linux':
            # Linux installation
            distro = platform.dist()[0].lower() if hasattr(platform, 'dist') else 'unknown'
            
            if 'ubuntu' in distro or 'debian' in distro:
                subprocess.run(['sudo', 'apt-get', 'update'], check=True)
                subprocess.run(['sudo', 'apt-get', 'install', '-y', 'google-chrome-stable'], check=True)
                return True
            elif 'fedora' in distro or 'centos' in distro or 'rhel' in distro:
                subprocess.run(['sudo', 'dnf', 'install', '-y', 'google-chrome-stable'], check=True)
                return True
            else:
                logger.error("Unsupported Linux distribution. Please manually install Chrome.")
                return False
        else:
            logger.error(f"Unsupported operating system: {system}")
            return False
    except Exception as e:
        logger.error(f"Error installing Chrome: {e}")
        return False

def download_webpage(url: str, output_file: str) -> bool:
    """
    Download webpage content using Chrome headless browser.
    
    Args:
        url: The URL to download
        output_file: The file to save the output
    
    Returns:
        bool: True if download successful, False otherwise
    """
    try:
        # Build the Chrome command
        chrome_cmd = [
            'google-chrome' if platform.system() != 'Windows' else 'chrome',
            '--headless=new',
            '--no-sandbox',
            '--disable-gpu',
            '--disable-dev-shm-usage',
            '--virtual-time-budget=8000',
            '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36',
            '--hide-scrollbars',
            '--blink-settings=imagesEnabled=true',
            '--dump-dom',
            url
        ]
        
        logger.info(f"Running Chrome command: {' '.join(chrome_cmd)}")
        
        # Execute the command and capture output
        result = subprocess.run(
            chrome_cmd,
            capture_output=True,
            text=True,
            timeout=60
        )
        
        if result.returncode != 0:
            logger.error(f"Chrome command failed with return code {result.returncode}")
            logger.error(f"Error output: {result.stderr}")
            return False
        
        # Write the output to file
        with open(output_file, 'w', encoding='utf-8') as f:
            f.write(result.stdout)
        
        logger.info(f"Webpage downloaded successfully to {output_file}")
        return True
    except subprocess.TimeoutExpired:
        logger.error("Chrome command timed out")
        return False
    except Exception as e:
        logger.error(f"Error downloading webpage: {e}")
        return False

def read_webpage_content(file_path: str) -> Optional[str]:
    """
    Read the content of the downloaded webpage.
    
    Args:
        file_path: Path to the HTML file
    
    Returns:
        Optional[str]: The content of the file, or None if error
    """
    try:
        with open(file_path, 'r', encoding='utf-8') as f:
            content = f.read()
        logger.info(f"Read {len(content)} characters from {file_path}")
        return content
    except Exception as e:
        logger.error(f"Error reading file: {e}")
        return None

def summarize_content(content: str) -> str:
    """
    Generate a simple summary of the webpage content.
    
    Args:
        content: The HTML content
    
    Returns:
        str: Summary of the content
    """
    # Simple summary - count characters and find title if present
    import re
    
    # Extract title
    title_match = re.search(r'<title>(.*?)</title>', content, re.IGNORECASE | re.DOTALL)
    title = title_match.group(1).strip() if title_match else "No title found"
    
    # Count content length
    content_length = len(content)
    
    # Count paragraphs
    paragraph_count = len(re.findall(r'<p[^>]*>', content, re.IGNORECASE))
    
    summary = f"Webpage Summary:\n"
    summary += f"Title: {title}\n"
    summary += f"Content length: {content_length} characters\n"
    summary += f"Paragraph count: {paragraph_count}\n"
    summary += "\nNote: This is a basic summary. For detailed analysis, please review the full content."
    
    return summary

def main(url: str) -> Dict[str, Any]:
    """
    Main function to download and process webpage.
    
    Args:
        url: The URL to process
    
    Returns:
        Dict[str, Any]: Result containing status and content
    """
    result = {
        'success': False,
        'message': '',
        'content': None,
        'summary': None
    }
    
    try:
        # Check if Chrome is installed
        if not check_chrome_installed():
            logger.info("Chrome not found, attempting to install...")
            if not install_chrome():
                result['message'] = "Chrome installation failed. Please install Chrome manually."
                return result
        
        # Create temporary directory for output
        with tempfile.TemporaryDirectory() as temp_dir:
            output_file = os.path.join(temp_dir, 'url.html')
            
            # Download webpage
            if not download_webpage(url, output_file):
                result['message'] = "Failed to download webpage"
                return result
            
            # Read content
            content = read_webpage_content(output_file)
            if not content:
                result['message'] = "Failed to read webpage content"
                return result
            
            # Generate summary
            summary = summarize_content(content)
            
            # Set result
            result['success'] = True
            result['message'] = "Webpage downloaded and processed successfully"
            result['content'] = content
            result['summary'] = summary
            
    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        result['message'] = f"An unexpected error occurred: {str(e)}"
    
    return result

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python webpage_reader.py <url>")
        sys.exit(1)
    
    url = sys.argv[1]
    result = main(url)
    
    if result['success']:
        print("Success!")
        print(result['summary'])
        print("\nFull content saved to url.html (temporary file)")
    else:
        print(f"Error: {result['message']}")
        sys.exit(1)

Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)