Skill: smile-xuc/haibo-jina-reader (version 1.0.0)

--- name: jina-reader description: Extract clean, readable markdown content from any URL using Jina Reader API. Use when you need to fetch and parse web pages without dealing with HTML, JavaScript rendering, or paywalls. Ideal for research, article summarization, content analysis, and working with search results from tavily-search, web_search, or searxng skills. --- # Jina Reader ## Overview Jina Reader provides clean markdown extraction from any URL, bypassing HTML complexity, JavaScript rendering, and many paywalls. It returns structured text content with metadata (title, URL, published time) that's perfect for AI analysis. ## Quick Start ### Extract markdown content ```bash scripts/jina-reader.py <url> ``` ### Extract with JSON metadata ```bash scripts/jina-reader.py <url> --format json ``` ### Save to file ```bash scripts/jina-reader.py <url> -o output.md ``` ## Core Operations ### 1. Basic Extraction Extract clean markdown from any URL: ```bash scripts/jina-reader.py https://example.com/article ``` **Returns:** Full markdown content including title, metadata headers, and structured text. **Use when:** You need readable text from a webpage for summarization, analysis, or content processing. ### 2. JSON Format Get structured data with metadata: ```bash scripts/jina-reader.py https://example.com/article --format json ``` **Returns:** ```json { "status": "success", "metadata": { "title": "Article Title", "url": "https://example.com/article", "published": "Mon, 10 Feb 2026 12:00:00 GMT" }, "content": "Markdown content..." } ``` **Use when:** You need programmatic access to metadata or want to integrate with other tools. ### 3. Shell Script Quick Access For simple one-liners: ```bash scripts/jina-reader.sh https://example.com/article ``` **Returns:** Raw markdown content directly to stdout. **Use when:** Quick extraction without arguments or when piping to other commands. ## Usage Patterns ### Combining with Search When using tavily-search, web_search, or searxng skills: 1. Get search results with relevant URLs 2. Extract content from top results using jina-reader 3. Process and summarize the extracted content ```bash # Example workflow URL="https://example.com/article" scripts/jina-reader.py "$URL" --format json | jq -r '.content' ``` ### Batch Processing Extract from multiple URLs: ```bash for url in $(cat urls.txt); do scripts/jina-reader.py "$url" -o "output/$(basename $url).md" done ``` ### Content Analysis Pipe extracted content to analysis tools: ```bash scripts/jina-reader.py https://example.com/article | wc -w scripts/jina-reader.py https://example.com/article | grep -i "keyword" ``` ## Options ### Python Script (`jina-reader.py`) - `url` (required): The URL to extract content from - `-f, --format`: Output format - `markdown` or `json` (default: markdown) - `-t, --timeout`: Request timeout in seconds (default: 30) - `-o, --output`: Save output to file instead of stdout ### Shell Script (`jina-reader.sh`) - `url` (required): The URL to extract content from ## Limitations - **Timeout:** Default 30 seconds. Increase with `-t` for slow-loading pages - **Rate limits:** Jina Reader API has rate limits. Use batching strategically - **Dynamic content:** Can't extract content generated by client-side JavaScript after page load - **Authentication:** Can't access pages requiring login or special headers ## Troubleshooting ### Timeout errors ```bash scripts/jina-reader.py <url> -t 60 # Increase timeout ``` ### Invalid URLs The tool auto-prepends `https://` if missing. Use fully qualified URLs for reliability. ### Empty content Some pages may block scraping. Try the shell script as fallback, or verify the URL is accessible. ## Resources ### scripts/jina-reader.py Full-featured Python tool with JSON output, metadata extraction, and file saving. ### scripts/jina-reader.sh Lightweight shell script for quick markdown extraction.

path: scripts/jina-reader.py
content: #!/usr/bin/env python3
"""
Jina Reader - Web content extraction tool
Extracts clean markdown from any URL using Jina Reader API
"""

import sys
import json
import argparse
from urllib.parse import urlparse
from pathlib import Path

try:
    import requests
except ImportError:
    print("Error: requests module not found. Install with: pip install requests")
    sys.exit(1)


def extract_content(url: str, output_format: str = "markdown", timeout: int = 30) -> dict:
    """
    Extract content from URL using Jina Reader API

    Args:
        url: The URL to extract content from
        output_format: Output format - 'markdown' or 'json'
        timeout: Request timeout in seconds

    Returns:
        Dictionary with content and metadata
    """
    # Normalize URL
    if not url.startswith(('http://', 'https://')):
        url = f'https://{url}'

    jina_url = f"https://r.jina.ai/http://{url.replace('http://', '').replace('https://', '')}"

    headers = {
        'User-Agent': 'Mozilla/5.0 (compatible; JinaReader/1.0)'
    }

    try:
        response = requests.get(jina_url, headers=headers, timeout=timeout)
        response.raise_for_status()
        content = response.text

        if output_format == "json":
            # Try to extract metadata from response
            lines = content.split('\n')
            metadata = {}
            markdown_content = []

            current_field = None
            for line in lines:
                if line.startswith('Title:'):
                    metadata['title'] = line.replace('Title:', '').strip()
                elif line.startswith('URL Source:'):
                    metadata['url'] = line.replace('URL Source:', '').strip()
                elif line.startswith('Published Time:'):
                    metadata['published'] = line.replace('Published Time:', '').strip()
                elif line.startswith('Markdown Content:'):
                    current_field = 'content'
                elif current_field == 'content':
                    markdown_content.append(line)

            return {
                'status': 'success',
                'metadata': metadata,
                'content': '\n'.join(markdown_content)
            }
        else:
            return {
                'status': 'success',
                'content': content
            }

    except requests.exceptions.Timeout:
        return {
            'status': 'error',
            'error': f'Request timeout after {timeout} seconds'
        }
    except requests.exceptions.RequestException as e:
        return {
            'status': 'error',
            'error': str(e)
        }


def main():
    parser = argparse.ArgumentParser(
        description='Extract clean markdown content from any URL using Jina Reader API'
    )
    parser.add_argument('url', help='URL to extract content from')
    parser.add_argument(
        '-f', '--format',
        choices=['markdown', 'json'],
        default='markdown',
        help='Output format (default: markdown)'
    )
    parser.add_argument(
        '-t', '--timeout',
        type=int,
        default=30,
        help='Request timeout in seconds (default: 30)'
    )
    parser.add_argument(
        '-o', '--output',
        help='Save output to file'
    )

    args = parser.parse_args()

    result = extract_content(args.url, args.format, args.timeout)

    if result['status'] == 'error':
        print(f"Error: {result['error']}", file=sys.stderr)
        sys.exit(1)

    if args.format == 'json':
        output = json.dumps(result, indent=2, ensure_ascii=False)
    else:
        output = result['content']

    if args.output:
        Path(args.output).write_text(output, encoding='utf-8')
        print(f"Saved to {args.output}")
    else:
        print(output)


if __name__ == '__main__':
    main()


path: scripts/jina-reader.sh
content: #!/usr/bin/env bash
# Jina Reader - 网页内容提取工具
# Usage: jina-reader.sh <url>

set -e

if [ -z "$1" ]; then
    echo "Usage: jina-reader.sh <url>"
    exit 1
fi

URL="$1"
JINA_API="https://r.jina.ai/http://${URL#http://}"

curl -s "$JINA_API"


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (HIGH)