Skill: johstracke/url-fetcher (version 1.0.0)

--- name: url-fetcher description: Simple web content fetching without API keys or external dependencies. Uses Python stdlib (urllib) only. Features: fetch HTML/text from URLs, basic HTML to markdown conversion, path-validated file writes (blocks system paths), URL validation (blocks localhost/internal). Security: File writes use is_safe_path() to prevent malicious writes. Perfect for content aggregation, research collection, and web scraping without API costs or dependencies. --- # URL Fetcher Fetch web content without API keys or external dependencies. Uses Python standard library only. ## Quick Start ```bash url_fetcher.py fetch <url> url_fetcher.py fetch --markdown <url> [output_file] ``` **Examples:** ```bash # Fetch and preview url_fetcher.py fetch https://example.com # Fetch and save HTML url_fetcher.py fetch https://example.com ~/workspace/page.html # Fetch and convert to basic markdown url_fetcher.py fetch --markdown https://example.com ~/workspace/page.md ``` ## Features - **No dependencies** - Uses Python stdlib (urllib) only - **No API keys** - Completely free to use - **URL validation** - Blocks localhost/internal networks - **Basic markdown conversion** - Extract content from HTML - **Path validation** - Safe file writes only (workspace, home, /tmp) - **Error handling** - Timeout and network error handling ## When to Use - **Content aggregation** - Collect pages for processing - **Research collection** - Save articles/pages locally - **Simple scraping** - Extract text from web pages - **Markdown conversion** - Basic HTML to text/markdown - **No-API alternatives** - When you can't use paid APIs ## Limitations - **Basic markdown** - Simple regex-based conversion (not a full parser) - **No JavaScript** - Only fetches static HTML - **Rate limiting** - No built-in rate limiting (add your own if needed) - **Bot detection** - Some sites may block the default User-Agent ## Security Features ### URL Validation - ✅ Allows: http/https URLs - ❌ Blocks: file://, data://, javascript: URLs - ❌ Blocks: localhost, 127.0.0.1, ::1 (internal networks) ### File Path Validation - ✅ Allows: workspace, home directory, /tmp - ❌ Blocks: system paths (/etc, /usr, /var, etc.) - ❌ Blocks: sensitive dotfiles (~/.ssh, ~/.bashrc, etc.) ### Error Handling - Timeout after 10 seconds - HTTP error handling - Network error handling - Character encoding handling ## Usage Patterns ### Collecting Research ```bash # Fetch multiple articles url_fetcher.py fetch https://example.com/article1.md ~/workspace/research/article1.md url_fetcher.py fetch https://example.com/article2.md ~/workspace/research/article2.md # Convert to markdown for reading url_fetcher.py fetch --markdown https://example.com/article.md ~/workspace/research/article.md ``` ### Content Aggregation ```bash # Fetch pages for processing url_fetcher.py fetch https://news.example.com ~/workspace/content/latest.html # Extract text url_fetcher.py fetch --markdown https://blog.example.com ~/workspace/content/post.md ``` ### Quick Preview ```bash # Just preview content (no file save) url_fetcher.py fetch https://example.com ``` ## Advanced Usage ### Batch Fetching ```bash #!/bin/bash # batch_fetch.sh URLS=( "https://example.com/page1" "https://example.com/page2" "https://example.com/page3" ) OUTPUT_DIR="$HOME/workspace/fetched" mkdir -p "$OUTPUT_DIR" for url in "${URLS[@]}"; do filename=$(echo $url | sed 's|/||g') url_fetcher.py fetch --markdown "$url" "$OUTPUT_DIR/$filename.md" sleep 1 # Be nice to servers done ``` ### Integration with Other Skills **Combine with research-assistant:** ```bash # Fetch article url_fetcher.py fetch --markdown https://example.com/article.md ~/workspace/article.md # Extract key points # Then use research-assistant to organize findings ``` **Combine with task-runner:** ```bash # Add task to fetch content task_runner.py add "Fetch article on topic X" "research" # Fetch when ready url_fetcher.py fetch https://example.com/topic-x.md ~/workspace/research/topic-x.md ``` ## Troubleshooting ### Connection Timeout ``` Error: Request timeout after 10s ``` **Solution:** The server is slow or unreachable. Try again later or check the URL. ### HTTP 403/429 Errors ``` Error: HTTP 403: Forbidden ``` **Solution:** The site blocks automated requests. Try: - Add delay between requests - Use a different User-Agent (modify source) - Respect robots.txt - Consider using an API if available ### Encoding Issues ``` Error with special characters ``` **Solution:** The tool uses UTF-8 with error-ignore. Some characters may be lost. ### Markdown Quality ``` Note: Basic markdown extraction ``` **Solution:** This tool uses simple regex for HTML→MD conversion. For better results: - Use dedicated markdown parsers - Or post-process the output - Or use a paid API with better parsing ## Best Practices 1. **Be respectful** - Add delays between requests (don't hammer servers) 2. **Check robots.txt** - Respect site's crawling policies 3. **Rate limit yourself** - Don't fetch too fast 4. **Validate URLs** - Only fetch from trusted sources 5. **Save safely** - Always use path-validated outputs 6. **Preview first** - Use preview mode before saving ## Integration Examples ### Python Integration ```python from pathlib import Path import subprocess def fetch_and_process(url): """Fetch URL and process""" output = Path.home() / "workspace" / "fetched" / "page.md" output.parent.mkdir(parents=True, exist_ok=True) # Fetch subprocess.run([ "python3", "/path/to/url_fetcher.py", "fetch", "--markdown", url, str(output) ]) # Process content content = output.read_text() return content ``` ### Bash Integration ```bash # Function for fetching fetch_content() { local url="$1" local output="$2" python3 ~/workspace/skills/url-fetcher/scripts/url_fetcher.py \ fetch --markdown "$url" "$output" } # Usage fetch_content "https://example.com" ~/workspace/example.md ``` ## Alternatives ### When You Need More Features **For full-featured scraping:** - Use `requests` + `beautifulsoup4` (requires pip install) - Or use `scrapy` framework (requires pip install) - Or use paid APIs (Firecrawl, Apify) **For better markdown:** - `markdownify` library (requires pip install) - Or use AI-based parsing (OpenAI, Anthropic APIs) **For complex workflows:** - Browser automation (OpenClaw browser tool) - Headless Chrome (Puppeteer, Playwright) - Or use scraping APIs (Zyte, ScraperAPI) ## Zero-Cost Advantage This skill requires: - ✅ Python 3 (included with OpenClaw) - ✅ No API keys - ✅ No external packages - ✅ No paid services - ✅ No rate limiting (other than what you add) Perfect for autonomous agents with budget constraints. ## Contributing If you improve this skill, please: 1. Test with security-checker 2. Document new features 3. Publish to ClawHub with credit ## License Use freely in your OpenClaw skills and workflows.

path: scripts/url_fetcher.py
content: #!/usr/bin/env python3
"""
URL Fetcher - Simple web content fetching without external dependencies
Uses only Python stdlib (urllib) - no API keys, no pip install needed

Usage: python3 url_fetcher.py fetch <url> [output_file]
       python3 url_fetcher.py fetch --markdown <url> [output_file]
"""

import sys
import json
from pathlib import Path
from urllib.parse import urlparse, urlunparse
from urllib.request import urlopen, Request
from urllib.error import URLError, HTTPError
from datetime import datetime
import re

# Safe file locations
SAFE_PATHS = [
    Path.home() / ".openclaw" / "workspace",
    Path.home(),
    Path("/tmp")
]

def is_safe_path(filepath):
    """Check if file path is in safe locations"""
    try:
        path = Path(filepath).expanduser().resolve()
        path_str = str(path)
        
        for safe in SAFE_PATHS:
            if path_str.startswith(str(safe.resolve())):
                blocked = ["/etc", "/usr", "/var", "/root", ".ssh", ".bashrc", ".zshrc"]
                if not any(b in path_str for b in blocked):
                    return True
        return False
    except:
        return False

def fetch_url(url, timeout=10):
    """Fetch URL content safely using urllib (stdlib only)"""
    try:
        # Validate URL
        parsed = urlparse(url)
        if not parsed.scheme or parsed.scheme not in ['http', 'https']:
            return {"error": "Invalid URL scheme (only http/https allowed)"}
        
        # Block localhost/internal networks
        blocked_hosts = ['localhost', '127.0.0.1', '::1', '0.0.0.0', '0.0.0.0']
        if parsed.hostname in blocked_hosts or parsed.hostname == '0.0.0.0':
            return {"error": "Internal/localhost URLs blocked"}
        
        # Create request with headers
        req = Request(
            url,
            headers={
                'User-Agent': 'Mozilla/5.0 (compatible; URLFetcher/1.0)'
            }
        )
        
        # Fetch
        response = urlopen(req, timeout=timeout)
        
        # Get content type
        content_type = response.headers.get('Content-Type', 'text/plain')
        
        # Read content
        content = response.read().decode('utf-8', errors='ignore')
        
        return {
            "url": url,
            "status_code": response.status,
            "content_type": content_type,
            "content_length": len(content),
            "text": content,
            "fetched_at": datetime.now().isoformat()
        }
        
    except HTTPError as e:
        return {"error": f"HTTP {e.code}: {e.reason}"}
    except URLError as e:
        return {"error": f"URL error: {e.reason}"}
    except TimeoutError:
        return {"error": f"Request timeout after {timeout}s"}
    except Exception as e:
        return {"error": f"Unexpected error: {str(e)}"}

def simple_markdown_extract(html_content, url):
    """Very basic HTML to markdown conversion"""
    # Remove script and style tags
    html = re.sub(r'<script[^>]*>.*?</script>', '', html_content, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<style[^>]*>.*?</style>', '', html, flags=re.DOTALL | re.IGNORECASE)
    
    # Convert common HTML tags to markdown
    html = re.sub(r'<h1[^>]*>(.*?)</h1>', r'# \1', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<h2[^>]*>(.*?)</h2>', r'## \1', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<h3[^>]*>(.*?)</h3>', r'### \1', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<p[^>]*>(.*?)</p>', r'\1\n\n', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<strong[^>]*>(.*?)</strong>', r'**\1**', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<b[^>]*>(.*?)</b>', r'**\1**', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<em[^>]*>(.*?)</em>', r'*\1*', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<i[^>]*>(.*?)</i>', r'*\1*', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<a[^>]*href="([^"]*)"[^>]*>(.*?)</a>', r'[\2](\1)', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<li[^>]*>(.*?)</li>', r'- \1', html, flags=re.DOTALL | re.IGNORECASE)
    html = re.sub(r'<ul[^>]*>|</ul>|<ol[^>]*>|</ol>', '', html, flags=re.IGNORECASE)
    html = re.sub(r'<br[^>]*>', '\n', html, flags=re.IGNORECASE)
    
    # Remove remaining tags
    html = re.sub(r'<[^>]+>', ' ', html)
    
    # Clean up whitespace
    html = re.sub(r'\s+', ' ', html)
    html = html.strip()
    
    # Add header
    title = url.split('/')[-1] or 'Untitled'
    md = f"# {title}\n\n"
    md += f"**Source:** {url}\n"
    md += f"**Fetched:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n"
    md += f"**Note:** Basic markdown extraction. For better results, use a proper parser.\n\n"
    md += "---\n\n"
    md += html
    
    return md

def fetch_command(args):
    """Handle fetch command"""
    if len(args) < 1:
        print("Usage: url_fetcher.py fetch [--markdown] <url> [output_file]")
        return
    
    url = None
    output_file = None
    as_markdown = False
    
    for arg in args:
        if arg == '--markdown':
            as_markdown = True
        elif arg.startswith('http'):
            url = arg
        else:
            output_file = arg
    
    if not url:
        print("Error: URL is required")
        return
    
    print(f"🌐 Fetching: {url}")
    result = fetch_url(url)
    
    if 'error' in result:
        print(f"❌ Error: {result['error']}")
        return
    
    print(f"✅ Status: {result['status_code']}")
    print(f"   Content-Type: {result['content_type']}")
    print(f"   Size: {result['content_length']} bytes")
    
    content = result['text']
    
    if as_markdown:
        content = simple_markdown_extract(content, url)
        print("   Format: Markdown (basic extraction)")
    
    if output_file:
        if not is_safe_path(output_file):
            print(f"❌ Security error: Unsafe output path: {output_file}")
            return
        
        Path(output_file).parent.mkdir(parents=True, exist_ok=True)
        Path(output_file).write_text(content)
        print(f"   Saved to: {output_file}")
    else:
        preview = content[:500] + "..." if len(content) > 500 else content
        print(f"\n📄 Preview:\n{preview}")

def main():
    if len(sys.argv) < 2:
        print("URL Fetcher - Simple web content fetching (no API keys, no pip install)")
        print("\nCommands:")
        print("  fetch [--markdown] <url> [output_file]  - Fetch URL content")
        print("\nExamples:")
        print("  url_fetcher.py fetch https://example.com")
        print("  url_fetcher.py fetch --markdown https://example.com output.md")
        print("\nFeatures:")
        print("  • Uses Python stdlib (urllib) - no external dependencies")
        print("  • Validates URLs (blocks localhost/internal)")
        print("  • Path validation for file writes")
        print("  • Basic HTML to markdown conversion")
        return
    
    command = sys.argv[1]
    args = sys.argv[2:]
    
    if command == "fetch":
        fetch_command(args)
    else:
        print(f"Unknown command: {command}")

if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)