Skill: mengbin92/duckduckgo-search-skill (version 1.0.0)

--- name: duckduckgo-search description: Search the web and fetch URL content using DuckDuckGo. Use when the user wants to search for information online without requiring API keys or paid services. Supports text search with results including title URL and snippet. Also supports URL fetching to extract readable content from web pages. Triggers on phrases like "search for" "look up" "find information about" "fetch url" "get page content" or when web_search is unavailable. --- # DuckDuckGo Search & Fetch Search the web and fetch URL content using DuckDuckGo (no API key required). ## Prerequisites 需要安装依赖: ```bash pip3 install duckduckgo-search ``` ## 功能 ### 1. 网页搜索 (ddg_search.py) ```bash python3 scripts/ddg_search.py "your search query" [--max-results 10] ``` ### 2. 网页抓取 (ddg_fetch.py) ```bash python3 scripts/ddg_fetch.py "https://example.com" [--timeout 30] ``` ## Usage Examples ### 搜索 ```bash # Basic search python3 scripts/ddg_search.py "OpenClaw AI agent" # Search with more results python3 scripts/ddg_search.py "Python best practices" --max-results 15 ``` ### 抓取网页 ```bash # Fetch a webpage python3 scripts/ddg_fetch.py "https://openclaw.ai" # With custom timeout python3 scripts/ddg_fetch.py "https://example.com" --timeout 15 # Plain text output python3 scripts/ddg_fetch.py "https://example.com" --format text ``` ## Output Format ### 搜索结果 (JSON) ```json { "query": "search query", "count": 10, "results": [ { "title": "Result title", "url": "https://example.com", "snippet": "Description snippet" } ] } ``` ### 抓取结果 (JSON) ```json { "url": "https://example.com", "title": "Page Title", "text": "Extracted readable content...", "description": "Meta description", "status_code": 200, "error": null } ``` ## Integration with OpenClaw Example workflow ```python # Search result = exec({ "command": "python3 /path/to/skills/duckduckgo-search/scripts/ddg_search.py query" }) # Parse: json.loads(result.stdout) # Fetch URL result = exec({ "command": "python3 /path/to/skills/duckduckgo-search/scripts/ddg_fetch.py https://example.com" }) # Parse: json.loads(result.stdout) ```

path: README.md
content: # OpenClaw Skills

A collection of useful skills for OpenClaw.

## Skills

### skill-linter

Analyze and validate SKILL.md files for best practices, common issues, and improvement suggestions.

**Usage:**
```bash
/skill-linter /path/to/SKILL.md
```

### duckduckgo-search

Search the web and fetch URL content using DuckDuckGo (no API key required).

**Prerequisites:**
```bash
pip3 install duckduckgo-search
```

**Usage:**
```bash
# Search
python3 duckduckgo-search/scripts/ddg_search.py "your query" --max-results 10

# Fetch URL
python3 duckduckgo-search/scripts/ddg_fetch.py "https://example.com"
```

## Installation

Install via ClawHub:

```bash
clawhub install skill-linter
clawhub install duckduckgo-search
```

## License

MIT


path: scripts/ddg_fetch.py
content: #!/usr/bin/env python3
"""
DuckDuckGo Web Fetcher
Fetch and extract readable content from URLs
"""

import argparse
import json
import sys
import urllib.request
import urllib.parse
import re
from html.parser import HTMLParser


class MLStripper(HTMLParser):
    """Strip HTML tags and decode entities"""
    def __init__(self):
        super().__init__()
        self.reset()
        self.fed = []
        
    def handle_data(self, d):
        self.fed.append(d)
        
    def handle_entityref(self, name):
        entities = {
            'amp': '&', 'lt': '<', 'gt': '>', 'quot': '"',
            'apos': "'", 'nbsp': ' ', 'copy': '©', 'reg': '®',
            'trade': '™', 'hellip': '…', 'mdash': '—', 'ndash': '–'
        }
        self.fed.append(entities.get(name, f'&{name};'))
        
    def get_data(self):
        return ''.join(self.fed)


def strip_tags(html):
    """Remove HTML tags from string"""
    s = MLStripper()
    try:
        s.feed(html)
    except:
        return html
    return s.get_data()


def clean_text(text):
    """Clean and normalize text"""
    # Remove extra whitespace
    text = re.sub(r'\s+', ' ', text)
    # Remove special characters but keep common ones
    text = re.sub(r'[^\w\s.,!?;:\-\'"()\[\]中文中文\-]', '', text)
    return text.strip()


def fetch_url(url: str, timeout: int = 30):
    """
    Fetch a URL and extract readable content.
    
    Args:
        url: URL to fetch
        timeout: Request timeout in seconds
    
    Returns:
        dict: Fetched content with title, text, and metadata
    """
    result = {
        "url": url,
        "title": "",
        "text": "",
        "content": "",
        "error": None,
        "status_code": 0
    }
    
    try:
        # Validate URL
        parsed = urllib.parse.urlparse(url)
        if not parsed.scheme:
            # Add https if missing
            url = 'https://' + url
            parsed = urllib.parse.urlparse(url)
        
        if not parsed.netloc:
            result["error"] = "Invalid URL"
            return result
        
        # Build request
        req = urllib.request.Request(
            url,
            headers={
                'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.0 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.0',
                'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
                'Accept-Encoding': 'gzip, deflate',
                'DNT': '1',
                'Connection': 'keep-alive',
            }
        )
        
        # Make request
        with urllib.request.urlopen(req, timeout=timeout) as response:
            result["status_code"] = response.status
            
            # Get content encoding
            content_encoding = response.headers.get('Content-Encoding', '')
            
            # Read content
            content = response.read()
            
            # Decode based on encoding
            if content_encoding == 'gzip':
                import gzip
                content = gzip.decompress(content)
            
            html = content.decode('utf-8', errors='ignore')
        
        # Extract title
        title_match = re.search(r'<title[^>]*>([^<]+)</title>', html, re.IGNORECASE)
        if title_match:
            result["title"] = strip_tags(title_match.group(1)).strip()
        
        # Try to find main content
        contentSelectors = [
            # Common article content selectors
            (r'<article[^>]*>(.*?)</article>', 'article'),
            (r'<main[^>]*>(.*?)</main>', 'main'),
            (r'<div[^>]*class="[^"]*content[^"]*"[^>]*>(.*?)</div>', 'content div'),
            (r'<div[^>]*class="[^"]*article[^"]*"[^>]*>(.*?)</div>', 'article div'),
            (r'<div[^>]*class="[^"]*post[^"]*"[^>]*>(.*?)</div>', 'post div'),
            (r'<div[^>]*id="[^"]*content[^"]*"[^>]*>(.*?)</div>', 'content div'),
            (r'<div[^>]*id="[^"]*main[^"]*"[^>]*>(.*?)</div>', 'main div'),
        ]
        
        main_content = ""
        for pattern, name in contentSelectors:
            match = re.search(pattern, html, re.DOTALL | re.IGNORECASE)
            if match:
                main_content = match.group(1)
                break
        
        # If no content found, use body
        if not main_content:
            body_match = re.search(r'<body[^>]*>(.*?)</body>', html, re.DOTALL | re.IGNORECASE)
            if body_match:
                main_content = body_match.group(1)
        
        # Extract text from content
        if main_content:
            # Remove script and style elements
            main_content = re.sub(r'<script[^>]*>.*?</script>', '', main_content, flags=re.DOTALL | re.IGNORECASE)
            main_content = re.sub(r'<style[^>]*>.*?</style>', '', main_content, flags=re.DOTALL | re.IGNORECASE)
            main_content = re.sub(r'<noscript[^>]*>.*?</noscript>', '', main_content, flags=re.DOTALL | re.IGNORECASE)
            main_content = re.sub(r'<!--.*?-->', '', main_content, flags=re.DOTALL)
            
            # Strip tags and clean
            text = strip_tags(main_content)
            text = clean_text(text)
            
            # Limit text length
            max_length = 10000
            if len(text) > max_length:
                text = text[:max_length] + "..."
            
            result["text"] = text
            result["content"] = main_content[:5000]  # Keep some HTML for reference
        
        # Extract meta description
        desc_match = re.search(r'<meta[^>]*name="description"[^>]*content="([^"]*)"', html, re.IGNORECASE)
        if not desc_match:
            desc_match = re.search(r'<meta[^>]*property="og:description"[^>]*content="([^"]*)"', html, re.IGNORECASE)
        
        if desc_match:
            result["description"] = desc_match.group(1)
        
        return result
        
    except urllib.error.HTTPError as e:
        result["error"] = f"HTTP Error {e.code}: {e.reason}"
        return result
    except urllib.error.URLError as e:
        result["error"] = f"URL Error: {str(e.reason)}"
        return result
    except Exception as e:
        result["error"] = f"Error: {type(e).__name__}: {str(e)}"
        return result


def main():
    parser = argparse.ArgumentParser(description="Fetch URL content (no API key required)")
    parser.add_argument("url", help="URL to fetch")
    parser.add_argument("--timeout", "-t", type=int, default=30,
                        help="Request timeout in seconds (default: 30)")
    parser.add_argument("--format", "-f", choices=["text", "json"], default="json",
                        help="Output format (default: json)")
    
    args = parser.parse_args()
    
    # Fetch content
    result = fetch_url(args.url, args.timeout)
    
    # Output
    if args.format == "json":
        print(json.dumps(result, ensure_ascii=False, indent=2))
    else:
        if result.get("error"):
            print(f"Error: {result['error']}", file=sys.stderr)
            sys.exit(1)
        
        if result.get("title"):
            print(f"Title: {result['title']}\n")
        print(result.get("text", ""))
    
    # Return error code if failed
    if result.get("error") and not result.get("text"):
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/ddg_search.py
content: #!/usr/bin/env python3
"""
DuckDuckGo Search Script
Search the web using DuckDuckGo (no API key required)
Fallback to alternative search methods if DDG blocks
"""

import argparse
import json
import sys
import subprocess
import urllib.request
import urllib.parse
import urllib.error
import re


def search_with_curl(query: str, max_results: int = 10):
    """Use curl to fetch DuckDuckGo results"""
    try:
        encoded_query = urllib.parse.quote_plus(query)
        url = f"https://lite.duckduckgo.com/lite/?q={encoded_query}"
        
        cmd = [
            'curl', '-s', '-L', 
            '-A', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.0',
            '--connect-timeout', '10',
            '--max-time', '20',
            url
        ]
        
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=25)
        html = result.stdout
        
        if result.returncode != 0:
            return {"error": f"curl failed: {result.stderr}", "results": []}
        
        # Parse results
        results = []
        
        # Pattern 1: Standard result format
        result_pattern = r'<tr>.*?class="result-link"[^\u003e]*href="([^"]*)"[^\u003e]*>([^\u003c]*)</a>.*?class="result-snippet"[^\u003e]*>([^\u003c]*)<'
        matches = re.findall(result_pattern, html, re.DOTALL | re.IGNORECASE)
        
        for url, title, snippet in matches[:max_results]:
            title = title.strip()
            snippet = snippet.strip()
            
            # Clean up URL
            if url.startswith('/'):
                url = 'https://duckduckgo.com' + url
            
            if title:
                results.append({
                    "title": title,
                    "url": url,
                    "snippet": snippet[:200] if snippet else ""
                })
        
        return {"results": results}
        
    except Exception as e:
        return {"error": str(e), "results": []}


def search_ddg(query: str, max_results: int = 10):
    """
    Search DuckDuckGo
    """
    try:
        encoded_query = urllib.parse.quote_plus(query)
        url = f"https://lite.duckduckgo.com/lite/?q={encoded_query}"
        
        headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.0',
            'Accept': 'text/html,application/xhtml+xml',
            'Accept-Language': 'en-US,en;q=0.9',
        }
        
        req = urllib.request.Request(url, headers=headers)
        
        with urllib.request.urlopen(req, timeout=15) as response:
            html = response.read().decode('utf-8', errors='ignore')
        
        # Parse results
        results = []
        
        # Find result rows
        import re
        rows = re.findall(r'<tr[^\u003e]*>(.*?)</tr>', html, re.DOTALL)
        
        for row in rows:
            # Extract link
            link_match = re.search(r'<a[^\u003e]*href="([^"]*)"[^\u003e]*>([^\u003c]*)</a>', row)
            if link_match:
                url = link_match.group(1)
                title = link_match.group(2).strip()
                
                # Extract snippet
                snippet_match = re.search(r'<td[^\u003e]*class="[^"]*snippet[^"]*"[^\u003e]*>([^\u003c]*)<', row)
                snippet = snippet_match.group(1).strip() if snippet_match else ""
                
                # Clean URL - decode DuckDuckGo redirect URLs
                if url.startswith('/'):
                    url = 'https://duckduckgo.com' + url
                
                # Decode DuckDuckGo redirect URLs
                if 'uddg=' in url:
                    match = re.search(r'uddg=([^&]+)', url)
                    if match:
                        url = urllib.parse.unquote(match.group(1))
                
                if title and len(title) > 3:
                    results.append({
                        "title": title,
                        "url": url,
                        "snippet": snippet[:200]
                    })
        
        return {
            "query": query,
            "count": len(results),
            "results": results[:max_results]
        }
        
    except Exception as e:
        # Try curl fallback
        fallback = search_with_curl(query, max_results)
        if fallback.get("results"):
            return {
                "query": query,
                "count": len(fallback["results"]),
                "results": fallback["results"]
            }
        return {
            "query": query,
            "error": str(e),
            "results": []
        }


def main():
    parser = argparse.ArgumentParser(description="Search DuckDuckGo (no API key required)")
    parser.add_argument("query", help="Search query")
    parser.add_argument("--max-results", "-n", type=int, default=10,
                        help="Maximum results (1-20, default: 10)")
    
    args = parser.parse_args()
    
    # Validate max_results
    max_results = max(1, min(20, args.max_results))
    
    # Perform search
    result = search_ddg(args.query, max_results)
    
    # Output JSON
    print(json.dumps(result, ensure_ascii=False, indent=2))
    
    # Return error code if search failed
    if "error" in result and not result.get("results"):
        sys.exit(1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)