Skill: lb1121/chinese-ebook-downloader (version 2.0.0)

--- name: chinese-ebook-downloader description: > Download Chinese-language ebooks from multiple sources with automatic A→B→C fallback. Primary source: online book library with ~100% coverage, no daily limit. Fallback sources: secondary library, Anna's Archive. Supports EPUB→PDF auto-conversion via weasyprint. Handles file host decryption, countdown wait, JS API extraction, GBK-encoded ZIP. Formats: PDF, EPUB, MOBI, AZW3. Use when: "下载电子书", "下载这本书", "找一下某本书的电子版", "帮我下个epub/mobi/azw3/pdf". --- # Chinese Ebook Downloader Download Chinese ebooks from multiple sources with automatic fallback and format conversion. ## Quick Start ```bash # Single book download (multi-source fallback) python scripts/download_book.py --title "超越百岁" --author "彼得·阿提亚" # Multi-source batch download (A→B→C fallback + EPUB→PDF conversion) python scripts/multi_source_download.py ~/Books/ # Search Anna's Archive directly python scripts/search_source_c.py "书名" "作者" # Convert EPUB to PDF python scripts/epub_to_pdf.py book.epub book.pdf ``` ## Download Sources (Priority Order) | Source | Coverage | Limit | Notes | |--------|----------|-------|-------| | **Source A** (online book library) | ~100% | None | Primary — high coverage for popular Chinese books | | **Source B** (secondary library) | ~8% | None | Fallback for missing titles | | **Source C** (Anna's Archive) | Wide | Rate-limited | Last resort — uses libgen.li mirrors | > **Note:** Z-Library has been deprecated due to 10/day download limit. ## Multi-Source Fallback The `multi_source_download.py` script automatically tries sources in order: ``` Source A → Source B → Source C → EPUB→PDF Conversion ``` **Workflow per book:** 1. Try Source A (ZIP → extract PDF/EPUB) 2. If failed, try Source B (file host download) 3. If failed, try Source C (Anna's Archive via libgen.li) 4. If only EPUB found, auto-convert to PDF using weasyprint **Usage:** ```bash # Edit BOOKS list in script, then run: python scripts/multi_source_download.py ~/Books/ ``` ## EPUB → PDF Conversion When only EPUB format is available, auto-convert using weasyprint: ```bash # Single file python scripts/epub_to_pdf.py input.epub output.pdf # Batch convert directory python scripts/epub_to_pdf.py --batch ~/Books/ ``` **Requirements:** `ebooklib`, `weasyprint`, CJK fonts installed. ## Scripts Reference | Script | Purpose | |--------|---------| | `download_book.py` | Primary download from Source A | | `search_secondary_source.py` | Source B search & download | | `search_source_c.py` | Anna's Archive search & download | | `batch_download.py` | Batch download from JSON list | | `multi_source_download.py` | Multi-source A→B→C fallback | | `epub_to_pdf.py` | EPUB/MOBI to PDF conversion | | `anna_iso_batch.sh` | Anna's Archive isolated batch (one process per book) | ## Source A Workflow (Primary) ``` Search → Get file host link → Decrypt → Wait countdown → API fetch → curl download → Extract ZIP ``` ### Step 1: Search Search the primary library for the book title. Navigate to download page, extract file host URL and password. ### Step 2: Decrypt Navigate to file host URL, enter password, click decrypt. ### Step 3: Wait for countdown File hosting service requires countdown before download. **Do not skip.** ### Step 4: Fetch real download URL **Get page variables:** ```javascript JSON.stringify({api_server, userid, file_id, share_id, file_chk, start_time, wait_seconds, verifycode}) ``` **Call API:** ```javascript (async () => { var url = api_server + '/get_file_url.php?uid=' + userid + '&fid=' + file_id + '&folder_id=0&share_id=' + share_id + '&file_chk=' + file_chk + '&start_time=' + start_time + '&wait_seconds=' + wait_seconds + '&mb=0&app=0&acheck=0' + '&verifycode=' + verifycode + '&rd=' + Math.random(); var headers = typeof getAjaxHeaders === 'function' ? getAjaxHeaders() : {}; var resp = await fetch(url, {headers: headers}); return JSON.stringify(await resp.json()); })() ``` Response `code: 200` → `downurl` is real URL. ### Step 5: Download ```bash curl -L -o "book.zip" "DOWNURL" \ -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" \ --max-time 1200 ``` ### Step 6: Extract ZIP (GBK encoding) ```python import zipfile with zipfile.ZipFile('book.zip', 'r') as z: for info in z.infolist(): try: name = info.filename.encode('cp437').decode('gbk') except: name = info.filename ext = os.path.splitext(name)[1].lower() if ext in ('.epub', '.azw3', '.mobi', '.pdf', '.txt'): data = z.read(info.filename) with open(os.path.basename(name), 'wb') as f: f.write(data) ``` ## Book Name Matching Strategy When a book title is long or contains multiple names (e.g. box sets): - Removes subtitles (after "：" or ":") - Removes parenthetical content ("（...）", "(...)") - Removes "套装共X册" bundle descriptions - Splits "+"-connected titles into individual books - Tries each keyword until match found - Falls back to full title + author **Examples:** - "杨定一全部生命系列：真原医+静坐+好睡（套装3册）" → tries "真原医", "静坐", "好睡" - "超越百岁：长寿的科学与艺术" → tries "超越百岁", then "超越百岁 彼得·阿提亚" ## Format Selection | Flag | Description | |------|-------------| | `--format pdf` | PDF only (default, preferred for NotebookLM) | | `--format epub` | EPUB only | | `--format mobi` | MOBI only | | `--format azw3` | AZW3 only | | `--format any` | Accept any available format | ## Batch Download ```bash python scripts/batch_download.py --book-list books.json --output-dir ~/Books/ ``` JSON format: ```json [ {"title": "超越百岁", "file_url": "<file_host_url>", "password": "<password>"} ] ``` Features: resume via `_progress.json`, skip existing, rate limiting. ## Troubleshooting | Problem | Solution | |---------|----------| | IP blocking | Use browser tool, not web_fetch | | Link 404 | Link expired, re-search | | API non-200 | Re-navigate and re-decrypt | | Download is HTML | URL expired, fresh API call needed | | ZIP filenames garbled | Use Python cp437→gbk, not unzip | | Timeout on large files | Increase `--max-time` to 1200 | | Anna's Archive blocked | Try different mirror, use `anna_iso_batch.sh` |

path: README.md
content: # chinese-ebook-downloader

An OpenClaw AgentSkill for downloading Chinese-language ebooks from multiple online sources.

## Features

- **Multi-source**: Primary (online book library, ~100% coverage), Secondary (another library), Fallback (Z-Library)
- **No daily limits**: Uses file hosting service which has no download quotas
- **Full automation**: Search → Decrypt → API → Download → Extract ZIP
- **GBK encoding support**: Properly handles Chinese filenames in ZIP archives
- **Batch download**: Resume-capable with progress tracking
- **Multiple formats**: PDF, EPUB, MOBI, AZW3

## Download Sources

| Source | Coverage | Daily Limit | Notes |
|--------|----------|-------------|-------|
| Source A (primary) | ~100% | None | Primary source |
| Source B (secondary) | ~8% | None | Fallback |
| Source C (Z-Library) | Medium | 10/day | Last resort |

## Installation

Install via [ClawHub](https://clawhub.com):
```bash
clawhub install chinese-ebook-downloader
```

Or clone from GitHub:
```bash
git clone https://github.com/yourusername/chinese-ebook-downloader.git
```

## Prerequisites

- Python 3.9+
- [Playwright](https://playwright.dev/): `pip install playwright && playwright install chromium`
- [OpenClaw](https://openclaw.ai) AgentSkill runtime (for SKILL.md to work as a skill)

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `SOURCE_A_BASE_URL` | *(built-in)* | Primary book source base URL |
| `SOURCE_B_BASE_URL` | *(built-in)* | Secondary book source base URL |
| `FILE_HOST_BASE_URL` | *(built-in)* | File hosting service base URL |
| `EBOOK_DEFAULT_PASSWORD` | *(built-in)* | Default file host password |

## Usage

### As an Agent Skill

Once installed, the skill auto-triggers when you ask to download a book:
> "下载《超越百岁》"
> "帮我找一下《癌症传》的电子版"
> "Download 《谷物大脑》 ebook"

### As Scripts

```bash
# Single book
python scripts/download_book.py --title "超越百岁" --author "彼得·阿提亚"

# Known file host URL
python scripts/download_book.py --file-url "<file_host_url>" --title "超越百岁"

# Batch download
python scripts/batch_download.py --book-list books.json --output-dir ~/Books/
```

### Batch Download JSON Format

```json
[
  {"title": "超越百岁", "file_url": "<file_host_url>", "password": "<password>"},
  {"title": "谷物大脑", "file_url": "<file_host_url>", "password": "<password>"}
]
```

## Workflow

```
Search primary source → Extract file host link → Browser decrypt
→ Wait countdown → JS API → curl download ZIP → Python extract (GBK)
```

## Troubleshooting

| Issue | Solution |
|-------|----------|
| `web_fetch` blocked | Use browser tool instead |
| Link 404 | Link expired, re-search |
| API returns non-200 | Re-navigate to file host, re-decrypt |
| Downloaded HTML not ZIP | Download URL expired, get fresh one |
| ZIP filenames garbled | Script handles this (cp437→gbk decode) |
| Slow downloads | Normal for free-tier file hosting |

## License

MIT


path: _meta.json
content: {
  "name": "chinese-ebook-downloader",
  "version": "2.0.0",
  "description": "Download Chinese ebooks from multiple sources with automatic fallback and EPUB→PDF conversion",
  "author": "TonyStack",
  "license": "MIT",
  "keywords": ["ebook", "download", "chinese", "pdf", "epub", "anna-archive"],
  "repository": ""
}


path: scripts/anna_async_download.py
content: #!/usr/bin/env python3
"""Batch download from Anna's Archive using async download_book_from_annas_archive."""
import sys, os, asyncio, time, glob, shutil

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from search_source_c import download_book_from_annas_archive

BOOKS = [
    {"title": "生命密码（套装共2册）", "author": "尹烨", "search": "生命密码 尹烨"},
    {"title": "吃出自愈力", "author": "[美]威廉·李", "search": "吃出自愈力 威廉·李"},
    {"title": "谷物大脑", "author": "戴维·珀尔马特 克里斯廷·洛伯格", "search": "谷物大脑 戴维·珀尔马特"},
    {"title": "不老时代", "author": "迈克尔·罗伊森", "search": "不老时代 年轻 长寿 罗伊森"},
    {"title": "疗愈的饮食与断食", "author": "杨定一", "search": "疗愈的饮食与断食 杨定一"},
    {"title": "菌群大脑", "author": "克里斯廷·洛伯格", "search": "菌群大脑 克里斯廷·洛伯格"},
    {"title": "疯狂的尿酸", "author": "戴维·珀尔马特", "search": "疯狂的尿酸 痛风 戴维·珀尔马特"},
    {"title": "逆龄饮食", "author": "安德烈亚斯·乔普", "search": "逆龄饮食 再生医学 乔普"},
]

OUTPUT_DIR = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/Documents/Books/微信读书/医学&健康/")

async def download_one(book, output_dir):
    """Download a single book, return True if successful."""
    parts = book["search"].split()
    title = parts[0] if len(parts) == 1 else " ".join(parts[:-1])
    author = parts[-1]
    
    print(f"  🔍 Searching: {title} - {author}")
    
    result = await download_book_from_annas_archive(
        title, author, output_dir=output_dir, headless=True
    )
    
    if result and result.get("status") == "done" and result.get("files"):
        for f in result["files"]:
            if os.path.exists(f):
                size = os.path.getsize(f)
                if size > 1000:
                    ext = os.path.splitext(f)[1].lower()
                    if ext != ".pdf":
                        print(f"  📥 Got {ext.upper()}: {os.path.basename(f)} ({size/1024/1024:.1f} MB) - needs PDF conversion")
                    else:
                        print(f"  ✅ PDF: {os.path.basename(f)} ({size/1024/1024:.1f} MB)")
                    return True
                else:
                    print(f"  ⚠️ Fake file ({size}B), removing")
                    os.remove(f)
    
    print(f"  ❌ Failed: {book['title']}")
    return False

async def main():
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    done, failed, skipped = 0, 0, 0
    
    for i, book in enumerate(BOOKS, 1):
        # Check if already exists
        existing_pdf = glob.glob(os.path.join(OUTPUT_DIR, f"*{book['title']}*.pdf"))
        existing_epub = glob.glob(os.path.join(OUTPUT_DIR, f"*{book['title']}*.epub"))
        if existing_pdf:
            print(f"\n⏭️ [{i}/{len(BOOKS)}] {book['title']} - PDF already exists")
            skipped += 1
            continue
        
        print(f"\n{'='*60}")
        print(f"📚 [{i}/{len(BOOKS)}] {book['title']}")
        print(f"{'='*60}")
        
        if await download_one(book, OUTPUT_DIR):
            done += 1
        else:
            failed += 1
        
        time.sleep(3)
    
    print(f"\n{'='*60}")
    print(f"📊 Results: {done} done, {failed} failed, {skipped} skipped (total {len(BOOKS)})")
    print(f"{'='*60}")

if __name__ == "__main__":
    asyncio.run(main())


path: scripts/anna_batch_download.py
content: #!/usr/bin/env python3
"""Download books from Anna's Archive using the existing search_source_c.py infrastructure."""
import sys, os, json, subprocess, time, glob, shutil

BOOKS = [
    {"title": "你是你吃出来的2", "alt_titles": ["你是你吃出来的 / 2", "你是你吃出来的2：家常便饭里的营养秘密"], "author": "夏萌"},
    {"title": "生命密码", "alt_titles": ["生命密码（套装共2册）"], "author": "尹烨"},
    {"title": "吃出自愈力", "alt_titles": [], "author": "威廉·李"},
    {"title": "谷物大脑", "alt_titles": [], "author": "戴维·珀尔马特"},
    {"title": "不老时代", "alt_titles": ["不老时代：年轻又长寿的科学和方法"], "author": "迈克尔·罗伊森"},
    {"title": "疗愈的饮食与断食", "alt_titles": ["疗愈的饮食与断食：新时代的个人营养学"], "author": "杨定一"},
    {"title": "菌群大脑", "alt_titles": [], "author": "克里斯廷·洛伯格"},
    {"title": "疯狂的尿酸", "alt_titles": ["疯狂的尿酸：不止是痛风"], "author": "戴维·珀尔马特"},
    {"title": "逆龄饮食", "alt_titles": ["逆龄饮食：逆转慢性疾病与衰老的再生医学新成果"], "author": "安德烈亚斯·乔普"},
]

OUTPUT_DIR = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/Documents/Books/微信读书/医学&健康/")
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
PYTHON = [REDACTED_SECRET]

def search_and_download(book, output_dir):
    """Try each title variant until we find a PDF."""
    titles = [book["title"]] + book.get("alt_titles", [])
    for title in titles:
        query = f"{title} {book['author']}"
        print(f"\n🔍 Trying: {query}")
        
        tmp_dir = f"/tmp/ebook_dl_{int(time.time())}"
        os.makedirs(tmp_dir, exist_ok=True)
        
        try:
            result = subprocess.run(
                [PYTHON, f"{SCRIPT_DIR}/search_source_c.py", title, book["author"], tmp_dir],
                capture_output=True, text=True, timeout=120
            )
            
            output = result.stdout + result.stderr
            
            # Check if any PDF was downloaded
            pdfs = glob.glob(os.path.join(tmp_dir, "*.pdf"))
            if pdfs:
                for pdf in pdfs:
                    size = os.path.getsize(pdf)
                    if size > 1000:  # Not a fake/error page
                        # Move to output dir
                        dest = os.path.join(output_dir, os.path.basename(pdf))
                        if os.path.exists(dest):
                            os.remove(dest)
                        shutil.move(pdf, dest)
                        print(f"  ✅ Downloaded: {os.path.basename(dest)} ({size/1024/1024:.1f} MB)")
                        shutil.rmtree(tmp_dir)
                        return True
                    else:
                        print(f"  ⚠️ Fake PDF ({size}B), skipping")
            
            # Also check epub for conversion
            epubs = glob.glob(os.path.join(tmp_dir, "*.epub"))
            if epubs:
                for epub in epubs:
                    size = os.path.getsize(epub)
                    if size > 1000:
                        dest = os.path.join(output_dir, os.path.basename(epub))
                        if os.path.exists(dest):
                            os.remove(dest)
                        shutil.move(epub, dest)
                        print(f"  📥 Got EPUB: {os.path.basename(dest)} ({size/1024/1024:.1f} MB) - will convert to PDF")
                        shutil.rmtree(tmp_dir)
                        return True
            
            print(f"  ❌ No valid download found")
            shutil.rmtree(tmp_dir)
            
        except subprocess.TimeoutExpired:
            print(f"  ⏱️ Timeout")
            shutil.rmtree(tmp_dir, ignore_errors=True)
        except Exception as e:
            print(f"  💥 Error: {e}")
            shutil.rmtree(tmp_dir, ignore_errors=True)
        
        time.sleep(3)  # Rate limiting
    
    return False

if __name__ == "__main__":
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    
    done = 0
    failed = 0
    
    for i, book in enumerate(BOOKS, 1):
        # Skip if already downloaded as PDF
        existing = glob.glob(os.path.join(OUTPUT_DIR, f"*{book['title']}*.pdf"))
        if existing:
            print(f"\n⏭️ [{i}/{len(BOOKS)}] {book['title']} - already exists, skipping")
            done += 1
            continue
        
        print(f"\n{'='*60}")
        print(f"📚 [{i}/{len(BOOKS)}] {book['title']} - {book['author']}")
        print(f"{'='*60}")
        
        if search_and_download(book, OUTPUT_DIR):
            done += 1
        else:
            failed += 1
        
        time.sleep(5)
    
    print(f"\n{'='*60}")
    print(f"📊 Results: {done} done, {failed} failed out of {len(BOOKS)}")
    print(f"{'='*60}")


path: scripts/anna_iso_batch.sh
content: #!/bin/bash
# Batch download - each book in its own process to avoid Playwright context issues
set -e

OUTPUT_DIR="$HOME/Documents/Books/微信读书/医学&健康/"
PYTHON=[REDACTED_SECRET]
SCRIPT_DIR="$HOME/.openclaw/workspace/skills/chinese-ebook-downloader/scripts"

declare -a BOOKS=(
  "生命密码 尹烨"
  "吃出自愈力 威廉·李"
  "谷物大脑 戴维·珀尔马特"
  "不老时代 年轻 长寿 罗伊森"
  "疗愈的饮食与断食 杨定一"
  "菌群大脑 克里斯廷·洛伯格"
  "疯狂的尿酸 痛风 戴维·珀尔马特"
  "逆龄饮食 再生医学 乔普"
)

DONE=0
FAIL=0
SKIP=0

for entry in "${BOOKS[@]}"; do
  read -r title author <<< "$entry"
  
  # Check if already downloaded
  if ls "$OUTPUT_DIR"/*"$title"*.pdf 1>/dev/null 2>&1; then
    echo "⏭️ $title - already exists, skipping"
    SKIP=$((SKIP+1))
    continue
  fi
  
  echo ""
  echo "=============================="
  echo "📚 $title - $author"
  echo "=============================="
  
  if timeout 180 "$PYTHON" -c "
import sys, asyncio, os
sys.path.insert(0, '$SCRIPT_DIR')
from search_source_c import download_book_from_annas_archive
async def go():
    result = await download_book_from_annas_archive('$title', '$author', output_dir='$OUTPUT_DIR', headless=True)
    if result and result.get('status') == 'done' and result.get('files'):
        for f in result['files']:
            if os.path.exists(f) and os.path.getsize(f) > 1000:
                ext = os.path.splitext(f)[1].lower()
                sz = os.path.getsize(f) / (1024*1024)
                print(f'✅ {ext.upper()}: {os.path.basename(f)} ({sz:.1f} MB)')
                sys.exit(0)
            elif os.path.exists(f):
                os.remove(f)
    print('❌ Failed')
    sys.exit(1)
asyncio.run(go())
" 2>&1; then
    DONE=$((DONE+1))
  else
    echo "❌ Failed: $title"
    FAIL=$((FAIL+1))
  fi
  
  sleep 5
done

echo ""
echo "=============================="
echo "📊 Results: $DONE done, $FAIL failed, $SKIP skipped"
echo "=============================="


path: scripts/batch_download.py
content: #!/usr/bin/env python3
"""
Batch download Chinese ebooks from online sources via file hosting service.
Usage: python batch_download.py --book-list books.json --output-dir ~/Books/
Supports resume via _progress.json.
"""
import sys
import os
import json
import asyncio
import time
import re
import subprocess
from pathlib import Path

# Add parent dir to path for imports
sys.path.insert(0, str(Path(__file__).parent))
from search_secondary_source import search_yabook as search_secondary_source, decrypt_ctfile as decrypt_file_host, get_download_url
from search_source_c import search_annas_archive as search_source_c, download_book_from_annas_archive as download_from_source_c

def _import_download_book():
    from download_book import download_book as download_from_primary, download_with_curl, verify_file, sanitize_filename, extract_zip
    return download_from_primary, download_with_curl, verify_file, sanitize_filename, extract_zip


def sanitize_filename(name: str) -> str:
    """Clean filename for filesystem."""
    name = re.sub(r'[\[\]【】（）()《》]', '', name)
    name = re.sub(r'[：:]', ' -', name)
    name = re.sub(r'[\\/:*?"<>|]', ' ', name)
    name = re.sub(r'\s+', ' ', name).strip()
    return name[:200]


def detect_format_from_files(files: list) -> str:
    """Detect the format(s) of extracted ebook files. Returns 'pdf', 'epub', etc. or 'unknown'."""
    formats = set()
    for f in files:
        ext = Path(f).suffix.lower().lstrip('.')
        if ext in ('pdf', 'epub', 'mobi', 'azw3'):
            formats.add(ext.upper())
    return ', '.join(sorted(formats)) if formats else 'unknown'


def sanitize_filename(name: str) -> str:
    """Clean filename for filesystem."""
    name = re.sub(r'[\[\]【】（）()《》]', '', name)
    name = re.sub(r'[：:]', ' -', name)
    name = re.sub(r'[\\/:*?"<>|]', ' ', name)
    name = re.sub(r'\s+', ' ', name).strip()
    return name[:200]


def load_progress(output_dir: str) -> dict:
    """Load progress file."""
    progress_file = os.path.join(output_dir, '_progress.json')
    if os.path.exists(progress_file):
        with open(progress_file, 'r') as f:
            return json.load(f)
    return {}


def save_progress(output_dir: str, progress: dict):
    """Save progress file."""
    progress_file = os.path.join(output_dir, '_progress.json')
    with open(progress_file, 'w') as f:
        json.dump(progress, f, ensure_ascii=False, indent=2)


def download_with_curl(url: str, output_path: str) -> bool:
    """Download file with curl."""
    cmd = [
        'curl', '-L', '-o', output_path, url,
        '-H', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
        '-H', 'Referer: https://placeholder.example.com/',
        '--max-time', '600',
        '--connect-timeout', '30'
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.returncode == 0


def verify_file(filepath: str) -> bool:
    """Verify downloaded file is valid."""
    if not os.path.exists(filepath):
        return False
    size = os.path.getsize(filepath)
    if size < 1024:  # Too small, probably an error page
        return False
    
    result = subprocess.run(['file', filepath], capture_output=True, text=True)
    output = result.stdout.lower()
    
    # Check for valid document types
    valid_types = ['pdf document', 'epub', 'mobi', 'zip archive']
    return any(t in output for t in valid_types) or size > 1024 * 1024  # >1MB is likely valid


async def download_from_secondary(title: str, author: str, output_dir: str, preferred_format: str = "pdf", browser=None) -> dict:
    """Try secondary book source. Returns {"status": "done"|"failed", ...}"""
    try:
        search_fn = search_secondary_source
    except ImportError:
        return {"status": "failed", "error": "Secondary source module not available"}

    results = await search_fn(title, author, browser=browser)
    if not results:
        return {"status": "failed", "error": "No results on secondary source"}

    best = results[0]
    file_url = best.get('ctfile_url', best.get('file_url', ''))
    [REDACTED_SECRET]'password', '')
    if not file_url:
        return {"status": "failed", "error": "No download link found"}

    api_vars = await decrypt_file_host(file_url, password, browser=browser)
    if not api_vars:
        return {"status": "failed", "error": "Could not decrypt file link (secondary)"}

    downurl = await get_download_url(api_vars, browser=browser)
    if not downurl:
        return {"status": "failed", "error": "Could not get download URL (secondary)"}

    print(f"  ⬇️  Downloading from secondary source...")
    clean = sanitize_filename(f"{title} - {author}")
    output_path = os.path.join(output_dir, f"{clean}.zip")
    success = download_with_curl(downurl, output_path)
    if not success:
        return {"status": "failed", "error": "curl download failed (secondary)"}

    # Try to extract ZIP with format filter
    try:
        extracted = extract_zip(output_path, output_dir, preferred_ext=preferred_format)
        os.remove(output_path)
        ebook_exts = ('.pdf', '.epub', '.mobi', '.azw3')
        if preferred_format != 'any':
            ebook_exts = (f'.{preferred_format}',)
        ebook_files = [f for f in extracted if f.lower().endswith(ebook_exts)]
        if ebook_files:
            return {"status": "done", "files": ebook_files, "source": "secondary"}
        # Files extracted but none match preferred format
        fmt = detect_format_from_files(extracted)
        return {"status": "format_mismatch", "available_formats": fmt, "source": "secondary", "files": extracted}
    except Exception:
        os.remove(output_path) if os.path.exists(output_path) else None
        return {"status": "failed", "error": "ZIP extraction failed (secondary)"}


async def download_from_tertiary(title: str, author: str, output_dir: str, preferred_format: str = "pdf", browser=None) -> dict:
    """Source C: Anna's Archive (search works, download may be blocked by DDoS-Guard)."""
    try:
        return await download_from_source_c(
            title=title, author=author, output_dir=output_dir,
            preferred_format=preferred_format, browser=browser,
        )
    except Exception as e:
        return {"status": "failed", "error": f"Source C error: {e}"}


async def download_book(title: str, author: str, output_dir: str, preferred_format: str = "pdf", browser=None) -> dict:
    """Download a single book with multi-source fallback and format awareness."""
    clean_title = sanitize_filename(f"{title} - {author}" if author else title)

    # Check if already downloaded (only for the preferred format)
    if preferred_format != 'any':
        p = os.path.join(output_dir, f"{clean_title}.{preferred_format}")
        if os.path.exists(p) and os.path.getsize(p) > 1024:
            return {"status": "done", "message": "Already exists", "source": "cache"}
    else:
        for ext in ('.pdf', '.epub', '.mobi', '.azw3'):
            p = os.path.join(output_dir, f"{clean_title}{ext}")
            if os.path.exists(p) and os.path.getsize(p) > 1024:
                return {"status": "done", "message": "Already exists", "source": "cache"}

    print(f"\n{'='*60}")
    print(f"📚 {title} - {author}")
    print(f"{'='*60}")

    available_formats = []  # Track what formats each source has

    # Source 1: Primary online library
    try:
        _dl = _import_download_book
        download_from_primary = _dl()[0]
        print(f"  🔍 Trying primary source (preferred: {preferred_format})...")
        result = await download_from_primary(title=title, author=author, output_dir=output_dir, preferred_format=preferred_format, browser=browser)
        if result.get('status') == 'done':
            result["source"] = "primary"
            return result
        if result.get('status') == 'format_mismatch':
            fmt = result.get('available_formats', 'unknown')
            available_formats.append(f"{fmt} (primary)")
            print(f"  ⚠️  Primary has wrong format: {fmt}")
        else:
            print(f"  ❌ Primary failed: {result.get('error', '?')}")
    except Exception as e:
        print(f"  ❌ Primary source error: {e}")

    # Source 2: Secondary source
    print(f"  🔍 Trying secondary source (preferred: {preferred_format})...")
    result = await download_from_secondary(title, author, output_dir, preferred_format=preferred_format, browser=browser)
    if result.get('status') == 'done':
        return result
    if result.get('status') == 'format_mismatch':
        fmt = result.get('available_formats', 'unknown')
        available_formats.append(f"{fmt} (secondary)")
        print(f"  ⚠️  Secondary has wrong format: {fmt}")
    else:
        print(f"  ❌ Secondary failed: {result.get('error', '?')}")

    # Source 3: Tertiary source
    result = await download_from_tertiary(title, author, output_dir, preferred_format=preferred_format, browser=browser)
    if result.get('status') == 'done':
        return result

    # Build failure message
    if available_formats and preferred_format != 'any':
        msg = f"No {preferred_format.upper()} available for this book. Sources returned: {', '.join(available_formats)}"
    elif result.get('error'):
        msg = result['error']
    else:
        msg = "All sources failed"
    return {"status": "failed", "error": msg}


async def main():
    import argparse
    
    parser = argparse.ArgumentParser(description="Batch download Chinese ebooks")
    parser.add_argument('--book-list', required=True, help='JSON file with book list')
    parser.add_argument('--output-dir', required=True, help='Output directory')
    parser.add_argument('--start', type=int, default=0, help='Start index (0-based)')
    parser.add_argument('--limit', type=int, default=0, help='Max books to download (0=all)')
    parser.add_argument('--format', default='pdf', choices=['pdf', 'epub', 'mobi', 'azw3', 'any'],
                        help='Preferred format (default: pdf)')
    args = parser.parse_args()
    
    # Load book list
    with open(args.book_list, 'r') as f:
        books = json.load(f)
    
    os.makedirs(args.output_dir, exist_ok=True)
    progress = load_progress(args.output_dir)
    
    # Apply start/limit
    books = books[args.start:]
    if args.limit > 0:
        books = books[:args.limit]
    
    stats = {"done": 0, "failed": 0, "skipped": 0}

    from playwright.async_api import async_playwright
    pw = await async_playwright().start()
    browser = await pw.chromium.launch(headless=True)

    try:
        for i, book in enumerate(books):
            title = book.get('title', '')
            author = book.get('author', '')
            key = f"{title}|{author}"
            
            # Skip already done
            if progress.get(key) == 'done':
                print(f"⏭️  Skipping (already done): {title}")
                stats['skipped'] += 1
                continue
            
            # Skip previously failed (will retry)
            # Allow retry for 'failed' and 'retry' status
            
            result = await download_book(title, author, args.output_dir, preferred_format=args.format, browser=browser)
            
            # Save progress
            progress[key] = result['status']
            save_progress(args.output_dir, progress)
            
            if result['status'] == 'done':
                stats['done'] += 1
            else:
                stats['failed'] += 1
                print(f"❌ Failed: {result.get('error', 'Unknown')}")
            
            # Rate limiting - wait between books
            if i < len(books) - 1:
                wait = 5
                print(f"⏳ Waiting {wait}s before next book...")
                await asyncio.sleep(wait)
            
            # Report every 5 books
            if (i + 1) % 5 == 0:
                print(f"\n📊 Progress: {stats['done']} done, {stats['failed']} failed, {stats['skipped']} skipped ({i+1}/{len(books)})")
    finally:
        await browser.close()
        await pw.stop()
    
    # Final summary
    print(f"\n{'='*60}")
    print(f"📊 FINAL SUMMARY")
    print(f"  ✅ Downloaded: {stats['done']}")
    print(f"  ❌ Failed:     {stats['failed']}")
    print(f"  ⏭️  Skipped:    {stats['skipped']}")
    print(f"  📁 Output:     {args.output_dir}")
    print(f"{'='*60}")


if __name__ == '__main__':
    asyncio.run(main())


path: scripts/download_book.py
content: #!/usr/bin/env python3
"""
Download Chinese ebooks from online book library via file hosting service.

Primary source: Online book library (high coverage)
Secondary sources: other online book libraries, Z-Library

Usage:
    python download_book.py --title "书名" [--author "作者"] [--output-dir ~/Downloads]
    python download_book.py --ctfile-url "<file_host_url>" --title "书名"

Workflow:
    1. Search online book library for the title
    2. Navigate to content / download page
    3. Extract file host link + password
    4. Browser decrypt (input password, click decrypt)
    5. Wait for countdown
    6. Get API variables and call download URL API
    7. Download with curl
    8. Extract ZIP with GBK encoding fix (cp437→gbk)
"""
import sys
import os
import re
import json
import asyncio
import subprocess
import zipfile
import tempfile
import shutil
from pathlib import Path
from urllib.parse import quote, urljoin
from typing import Optional, List, Tuple, Dict

try:
    from playwright.async_api import async_playwright
except ImportError:
    print("Error: Install playwright: pip install playwright && playwright install chromium")
    sys.exit(1)

# Constants
SOURCE_A_BASE_URL = os.environ.get("SOURCE_A_BASE_URL", "https://dushupai.com")
FILE_HOST_BASE_URL = os.environ.get("FILE_HOST_BASE_URL", "https://z701.com")
DEFAULT_PASSWORD = os.environ.get("EBOOK_DEFAULT_PASSWORD", "")
DOWNLOAD_TIMEOUT = 1200  # 20 minutes for large files
WAIT_SECONDS = 60  # File host requires 60 second wait


def extract_search_queries(title: str, author: str = "") -> List[str]:
    """Extract multiple search keyword combinations from a long book title."""
    queries = []

    # Clean: remove parenthetical content
    clean = re.sub(r'[（(].+?[）)]', '', title).strip()
    # Remove trailing 套装 descriptions
    clean = re.sub(r'（套装.*$', '', clean).strip()
    clean = re.sub(r'套装共.*$', '', clean).strip()

    # Main title (before colon)
    main = clean.split('：')[0].split(':')[0].strip()

    # Split bundled books by '+' — these are likely individual searchable titles
    if '+' in clean:
        parts = clean.split('+')
        for idx, p in enumerate(parts):
            p = p.strip()
            p = re.sub(r'[（(].+?[）)]', '', p).strip()
            # For first part, also try content after colon (the actual book name in series)
            if idx == 0 and '：' in p:
                after_colon = p.split('：')[-1].strip()
                if after_colon and len(after_colon) > 1:
                    queries.append(after_colon)
            p = p.split('：')[0].split(':')[0].strip()
            # Remove series prefixes for better matching
            short = re.sub(r'(系列|丛书|集)$', '', p).strip()
            if short and short != p:
                queries.append(short)
            if len(p) > 1:
                queries.append(p)

    # Add main title if different from the individual parts
    if main and main not in queries:
        queries.append(main)

    # Author + main title
    if author and main:
        clean_author = author.split('[')[0].split('（')[0].strip()
        if clean_author:
            combined = f"{main} {clean_author}"
            if combined not in queries:
                queries.append(combined)

    return queries or [title]


def sanitize_filename(name: str) -> str:
    """Clean filename for filesystem."""
    name = re.sub(r'[\[\]【】（）()《》]', '', name)
    name = re.sub(r'[：:]', ' -', name)
    name = re.sub(r'[\\/:*?"<>|]', ' ', name)
    name = re.sub(r'\s+', ' ', name).strip()
    return name[:200]


async def search_primary_source(title: str, author: str = "", headless: bool = True, browser=None) -> List[dict]:
    """
    Search primary book source and return results with file host links.

    For long titles, automatically extracts multiple search queries.
    Returns list of dicts with: title, id, file_url, pwd

    Args:
        browser: Shared Playwright browser instance. If None, creates a new one.
    """
    queries = extract_search_queries(title, author)
    # Always include full title+author as fallback
    full_query = f"{title} {author}".strip()
    if full_query not in queries:
        queries.append(full_query)

    results = []
    _owns_browser = browser is None

    for qi, query in enumerate(queries):
        tag = f"(query {qi+1}/{len(queries)})" if len(queries) > 1 else ""
        print(f"  🔍 Searching primary source {tag}: {query}")

        if _owns_browser:
            p = await async_playwright().start()
            browser = await p.chromium.launch(headless=headless)
        page = await browser.new_page()

        try:
            search_url = f"{SOURCE_A_BASE_URL}/?s={quote(query)}"
            await page.goto(search_url, wait_until='domcontentloaded', timeout=30000)
            await asyncio.sleep(2)

            # Get search results
            items = await page.query_selector_all('article a, .post-title a, h2 a, .entry-title a')

            if not items:
                print(f"    No results for: {query[:30]}...")
                continue

            # Check first 5 results
            found = False
            for item in items[:5]:
                try:
                    item_title = (await item.inner_text()).strip()
                    item_href = await item.get_attribute('href')

                    if not item_href or not item_title:
                        continue

                    # Check if it's a book content page
                    if '/book-content-' not in item_href and '/post/' not in item_href:
                        continue

                    full_url = urljoin(SOURCE_A_BASE_URL, item_href)

                    # Extract book ID from URL
                    id_match = re.search(r'book-content-(\d+)', full_url)
                    if not id_match:
                        id_match = re.search(r'/post/(\d+)', full_url)
                    book_id = id_match.group(1) if id_match else ""

                    print(f"    Checking: {item_title[:50]}...")
                    await page.goto(full_url, wait_until='domcontentloaded', timeout=20000)
                    await asyncio.sleep(2)

                    # Find download link
                    ctfile_url, pwd = await _extract_file_host_link(page, book_id)

                    if ctfile_url:
                        print(f"    ✅ Found file host link (password: {pwd})")
                        results.append({
                            "title": item_title,
                            "id": book_id,
                            "ctfile_url": ctfile_url,
                            "pwd": pwd
                        })
                        found = True
                        break  # Use first match
                except Exception as e:
                    print(f"    Warning: {e}")
                    continue

            if found:
                break  # Stop trying more queries
        finally:
            await page.close()

    if _owns_browser:
        await browser.close()
        await p.stop()

    if not results:
        print(f"  ❌ No results found across all queries")

    return results


async def _extract_file_host_link(page, book_id: str) -> Tuple[str, str]:
    """
    Extract file host link from book page.
    May need to navigate to download-book-{id}.html first.
    """
    # # Check for direct file host link
    links = await page.query_selector_all('a[href*="ctfile.com"]')
    for link in links:
        href = await link.get_attribute('href')
        if href and '/f/' in href:
            return href, DEFAULT_PASSWORD

    # Check for download-book-{id}.html link
    download_links = await page.query_selector_all(f'a[href*="download-book-{book_id}"], a[href*="download"]')
    for link in download_links:
        href = await link.get_attribute('href')
        if href:
            full_url = urljoin(SOURCE_A_BASE_URL, href)
            print(f"    Navigating to download page: {href}")
            await page.goto(full_url, wait_until='domcontentloaded', timeout=20000)
            await asyncio.sleep(2)

            # # Extract file host link from download page
            ctfile_links = await page.query_selector_all('a[href*="ctfile.com"]')
            for ct_link in ctfile_links:
                ct_href = await ct_link.get_attribute('href')
                if ct_href and '/f/' in ct_href:
                    return ct_href, DEFAULT_PASSWORD

            # Try to find in page text
            body = await page.query_selector('body')
            if body:
                text = await body.inner_text()
                # # Look for file host URL pattern
                url_match = re.search(r'https://url\d+\.ctfile\.com/f/\d+-\d+-\d+[^\s]*', text)
                if url_match:
                    return url_match.group(0), DEFAULT_PASSWORD

    return "", DEFAULT_PASSWORD


async def decrypt_file_host(file_url: str, password: str = DEFAULT_PASSWORD, headless: bool = True, browser=None) -> Optional[dict]:
    """
    Decrypt file host link and get download API variables.

    Args:
        browser: Shared Playwright browser instance. If None, creates a new one.
    """
    _owns_browser = browser is None
    if _owns_browser:
        p = await async_playwright().start()
        browser = await p.chromium.launch(headless=headless)
    page = await browser.new_page()

    try:
        print(f"Decrypting file host link...")
        await page.goto(file_url, wait_until='domcontentloaded', timeout=30000)
        await asyncio.sleep(3)

        # Check if password page
        body_text = await (await page.query_selector('body')).inner_text()

        if '输入文件' in body_text or '访问密码' in body_text or '解密' in body_text:
            input_box = await page.query_selector('input[type="text"], input[type="password"], input.layui-input')
            if input_box and password:
                await input_box.type(password, delay=50)

                btn = await page.query_selector('button:has-text("解密"), input[type="submit"][value*="解密"], .layui-btn')
                if not btn:
                    btn = await page.query_selector('button, input[type="submit"]')
                if btn:
                    await btn.click()
                    print(f"  Entered [REDACTED_SECRET]")
                    await asyncio.sleep(5)

        # Wait for the countdown (60 seconds required by file host)
        print(f"  Waiting {WAIT_SECONDS} seconds for countdown...")

        try:
            countdown = await page.query_selector('#down_interval, .countdown, #wait_time')
            if countdown:
                remaining = WAIT_SECONDS
                while remaining > 0:
                    try:
                        text = await countdown.inner_text()
                        nums = re.findall(r'\d+', text)
                        if nums:
                            remaining = int(nums[0])
                    except:
                        pass
                    print(f"    Countdown: {remaining}s remaining...", end='\r')
                    await asyncio.sleep(5)
                    remaining -= 5
        except:
            for i in range(WAIT_SECONDS // 10):
                print(f"    Waiting... {WAIT_SECONDS - i * 10}s remaining", end='\r')
                await asyncio.sleep(10)

        print()

        try:
            result = await page.evaluate("""() => {
                return JSON.stringify({
                    api_server: typeof api_server !== 'undefined' ? api_server : null,
                    userid: typeof userid !== 'undefined' ? userid : null,
                    file_id: typeof file_id !== 'undefined' ? file_id : null,
                    share_id: typeof share_id !== 'undefined' ? share_id : '',
                    file_chk: typeof file_chk !== 'undefined' ? file_chk : null,
                    start_time: typeof start_time !== 'undefined' ? start_time : null,
                    wait_seconds: typeof wait_seconds !== 'undefined' ? wait_seconds : 0,
                    verifycode: typeof verifycode !== 'undefined' ? verifycode : null,
                    filename: typeof filename !== 'undefined' ? filename : null,
                    file_size: typeof file_size !== 'undefined' ? file_size : null
                });
            }""")
            data = json.loads(result)

            if data.get('api_server') and data.get('file_id'):
                print(f"  Got API variables (file_id: {data['file_id']})")
                return data
            else:
                print("  Could not extract API variables")
                return None
        except Exception as e:
            print(f"  Error extracting variables: {e}")
            return None
    finally:
        await page.close()
        if _owns_browser:
            await browser.close()
            await p.stop()


async def get_download_url(api_vars: dict, browser=None) -> Optional[str]:
    """Call file host API to get real download URL."""
    _owns_browser = browser is None
    if _owns_browser:
        p = await async_playwright().start()
        browser = await p.chromium.launch(headless=True)
    page = await browser.new_page()

    try:
        ct_url = f"{FILE_HOST_BASE_URL}/f/{api_vars['userid']}-{api_vars['file_id']}"
        await page.goto(ct_url, wait_until='domcontentloaded', timeout=15000)
        await asyncio.sleep(2)

        try:
            js_code = f"""(async () => {{
                try {{
                    var url = '{api_vars['api_server']}/get_file_url.php'
                        + '?uid={api_vars["userid"]}'
                        + '&fid={api_vars["file_id"]}'
                        + '&folder_id=0'
                        + '&share_id={api_vars.get("share_id", "")}'
                        + '&file_chk={api_vars["file_chk"]}'
                        + '&start_time={api_vars["start_time"]}'
                        + '&wait_seconds={api_vars.get("wait_seconds", 0)}'
                        + '&mb=0&app=0&acheck=0'
                        + '&verifycode={api_vars["verifycode"]}'
                        + '&rd=' + Math.random();
                    var headers = typeof getAjaxHeaders === 'function' ? getAjaxHeaders() : {{}};
                    var resp = await fetch(url, {{headers: headers}});
                    var data = await resp.json();
                    return JSON.stringify(data);
                }} catch(e) {{ return JSON.stringify({{error: e.message}}); }}
            }})()"""

            result = await page.evaluate(js_code)
            data = json.loads(result)

            if data.get('code') == 200 and data.get('downurl'):
                file_size = data.get('file_size', 0)
                size_mb = int(file_size) / 1024 / 1024 if file_size else 0
                print(f"  Got download URL ({size_mb:.1f} MB)")
                return data['downurl']
            else:
                print(f"  API error: {data}")
                return None
        except Exception as e:
            print(f"  Error: {e}")
            return None
    finally:
        await page.close()
        if _owns_browser:
            await browser.close()
            await p.stop()


def download_with_curl(url: str, output_path: str) -> bool:
    """Download file with curl."""
    cmd = [
        'curl', '-L', '-o', output_path, url,
        '-H', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
        '-H', 'Referer: ' + FILE_HOST_BASE_URL + '/',
        '--max-time', str(DOWNLOAD_TIMEOUT),
        '--connect-timeout', '30',
        '--retry', '3'
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.returncode == 0


def extract_zip(zip_path: str, output_dir: str, preferred_ext: str = "any") -> list[str]:
    """
    Extract ZIP file with GBK encoding fix for Chinese filenames.

    Python's zipfile uses CP437 by default, but Chinese filenames are GBK encoded.
    We need to decode from CP437 back to bytes, then decode as GBK.

    Args:
        preferred_ext: Preferred format extension (without dot). "any" extracts all files.
    Returns:
        List of extracted file paths.
    """
    extracted_files = []
    ext_dot = f".{preferred_ext}" if preferred_ext != "any" else None

    with zipfile.ZipFile(zip_path, 'r') as zf:
        for info in zf.infolist():
            if info.is_dir():
                continue

            # Fix GBK encoding: cp437 -> bytes -> gbk
            try:
                fixed_name = info.filename.encode('cp437').decode('gbk')
            except (UnicodeDecodeError, UnicodeEncodeError):
                try:
                    fixed_name = info.filename.encode('cp437').decode('utf-8')
                except:
                    fixed_name = info.filename

            # Sanitize the filename
            fixed_name = sanitize_filename(fixed_name)

            # Format filter
            if ext_dot and not fixed_name.lower().endswith(ext_dot):
                print(f"  Skipped (not {preferred_ext}): {fixed_name}")
                continue

            info.filename = fixed_name
            zf.extract(info, output_dir)
            extracted_path = os.path.join(output_dir, fixed_name)
            extracted_files.append(extracted_path)
            print(f"  Extracted: {fixed_name}")

    return extracted_files


def verify_file(filepath: str) -> bool:
    """Verify downloaded file is valid."""
    if not os.path.exists(filepath):
        return False
    size = os.path.getsize(filepath)
    if size < 1024:  # Too small, probably an error page
        return False

    result = subprocess.run(['file', filepath], capture_output=True, text=True)
    output = result.stdout.lower()

    # Check for valid document types
    valid_types = ['pdf document', 'epub', 'mobi', 'zip archive', 'azw']
    return any(t in output for t in valid_types) or size > 1024 * 1024  # >1MB is likely valid


async def download_book(
    title: str,
    author: str = "",
    output_dir: str = ".",
    ctfile_url: str = "",
    password: str = DEFAULT_PASSWORD,
    headless: bool = True,
    preferred_format: str = "pdf",
    browser=None
) -> dict:
    """
    Download a single book.

    Args:
        title: Book title
        author: Book author (optional)
        output_dir: Output directory
        ctfile_url: Direct file host URL (optional, skip search if provided)
        password: file host password
        headless: Run browser in headless mode

    Returns:
        {"status": "done"|"failed", "error": "...", "files": [...]}
    """
    clean_title = sanitize_filename(f"{title} - {author}" if author else title)

    # Step 1: Search or use provided URL
    print(f"\n{'='*60}")
    print(f"Book: {title}" + (f" - {author}" if author else ""))
    print(f"{'='*60}")

    # --- Try primary source (Source A: dushupai) ---
    primary_result = None
    if not ctfile_url:
        print(f"  🔍 Searching primary source...")
        results = await search_primary_source(title, author, headless, browser=browser)
        if results:
            best = results[0]
            ctfile_url = best['ctfile_url']
            [REDACTED_SECRET]'pwd', DEFAULT_PASSWORD)
            print(f"  ✓ Found on primary source")
        else:
            print(f"  ❌ No results on primary source")

    if ctfile_url:
        # Step 2: Decrypt
        api_vars = await decrypt_file_host(ctfile_url, password, headless, browser=browser)
        if api_vars:
            # Step 3: Get download URL
            downurl = await get_download_url(api_vars, browser=browser)
            if downurl:
                primary_result = await _download_and_extract(
                    downurl, clean_title, output_dir, preferred_format
                )
        if primary_result and primary_result.get('status') == 'done':
            return primary_result
        print(f"  ⚠️ Primary source failed: {primary_result.get('error', 'unknown') if primary_result else 'decrypt failed'}")

    # --- Fallback: Source C (Anna's Archive) ---
    print(f"  🔍 Searching fallback source (Source C)...")
    try:
        from search_source_c import (
            search_annas_archive,
            download_book_from_annas_archive,
        )
        fallback_result = await download_book_from_annas_archive(
            title=title,
            author=author,
            output_dir=output_dir,
            preferred_format=preferred_format,
            headless=headless,
            browser=browser,
        )
        if fallback_result.get('status') == 'done':
            return fallback_result
        fallback_err = fallback_result.get('error', 'unknown')
        if fallback_result.get('manual_url'):
            print(f"  ⚠️ Source C download blocked, manual URL: {fallback_result['manual_url']}")
        else:
            print(f"  ⚠️ Source C failed: {fallback_err}")
    except Exception as e:
        print(f"  ⚠️ Source C error: {e}")

    return {"status": "failed", "error": "All sources failed"}


async def _download_and_extract(downurl, clean_title, output_dir, preferred_format):
    """Download a file and extract if it's a ZIP archive."""
    with tempfile.TemporaryDirectory() as tmpdir:
        zip_path = os.path.join(tmpdir, f"{clean_title}.zip")

        print(f"Downloading...")
        success = download_with_curl(downurl, zip_path)
        if not success:
            return {"status": "failed", "error": "curl download failed"}

        # Verify
        if not verify_file(zip_path):
            return {"status": "failed", "error": "Downloaded file is invalid"}

        # Check if it's a ZIP file
        result = subprocess.run(['file', zip_path], capture_output=True, text=True)
        if 'zip archive' in result.stdout.lower():
            # Extract ZIP
            print(f"Extracting ZIP (preferred: {preferred_format})...")
            extracted_files = extract_zip(zip_path, output_dir, preferred_ext=preferred_format)

            # Filter to ebook formats only
            ebook_exts = ('.pdf', '.epub', '.mobi', '.azw', '.azw3')
            if preferred_format != 'any':
                ebook_exts = (f'.{preferred_format}',)
            ebook_files = [f for f in extracted_files if f.lower().endswith(ebook_exts)]

            if ebook_files:
                total_size = sum(os.path.getsize(f) for f in ebook_files) / 1024 / 1024
                print(f"Done: {len(ebook_files)} file(s), {total_size:.1f} MB")
                for f in ebook_files:
                    print(f"  {os.path.basename(f)}")
                return {"status": "done", "files": ebook_files}
            else:
                return {"status": "done", "files": extracted_files, "warning": "No ebook files found in archive"}
        else:
            # Direct file (not ZIP)
            final_path = os.path.join(output_dir, f"{clean_title}.pdf")
            shutil.move(zip_path, final_path)
            size_mb = os.path.getsize(final_path) / 1024 / 1024
            print(f"Done: {final_path} ({size_mb:.1f} MB)")
            return {"status": "done", "files": [final_path]}


async def main():
    import argparse

    parser = argparse.ArgumentParser(description="Download Chinese ebooks from online sources")
    parser.add_argument('--title', required=True, help='Book title')
    parser.add_argument('--author', default='', help='Book author')
    parser.add_argument('--output-dir', default='.', help='Output directory')
    parser.add_argument('--ctfile-url', default='', help='Direct file host URL (skip search)')
    parser.add_argument('--password', default=DEFAULT_PASSWORD, help='File host password')
    parser.add_argument('--no-headless', action='store_true', help='Show browser window')
    parser.add_argument('--format', default='pdf', choices=['pdf', 'epub', 'mobi', 'azw3', 'any'],
                        help='Preferred format (default: pdf)')
    args = parser.parse_args()

    os.makedirs(args.output_dir, exist_ok=True)

    result = await download_book(
        title=args.title,
        author=args.author,
        output_dir=args.output_dir,
        ctfile_url=args.ctfile_url,
        [REDACTED_SECRET],
        headless=not args.no_headless,
        preferred_format=args.format
    )

    if result['status'] == 'done':
        print(f"\nSuccess!")
    else:
        print(f"\nFailed: {result.get('error', 'Unknown error')}")
        sys.exit(1)


if __name__ == '__main__':
    asyncio.run(main())


path: scripts/epub_to_pdf.py
content: #!/usr/bin/env python3
"""Convert EPUB/MOBI files to PDF using ebooklib + weasyprint."""
import sys, os, glob
from ebooklib import epub
from weasyprint import HTML

def epub_to_pdf(epub_path, pdf_path=None):
    """Convert an EPUB file to PDF."""
    if pdf_path is None:
        pdf_path = epub_path.rsplit('.', 1)[0] + '.pdf'
    
    print(f"Converting: {epub_path} -> {pdf_path}")
    
    book = epub.read_epub(epub_path)
    
    html_parts = []
    for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
        content = item.get_content().decode('utf-8', errors='ignore')
        html_parts.append(content)
    
    full_html = f"""
    <html>
    <head>
        <meta charset="utf-8">
        <style>
            body {{ font-family: "Noto Sans CJK SC", "PingFang SC", "Microsoft YaHei", sans-serif; 
                   font-size: 11pt; line-height: 1.8; max-width: 700px; margin: 0 auto; padding: 40px; }}
            h1, h2, h3 {{ font-weight: bold; margin-top: 1.5em; }}
            p {{ text-indent: 2em; margin: 0.5em 0; }}
            img {{ max-width: 100%; height: auto; }}
        </style>
    </head>
    <body>{''.join(html_parts)}</body>
    </html>
    """
    
    HTML(string=full_html).write_pdf(pdf_path)
    size_mb = os.path.getsize(pdf_path) / (1024*1024)
    print(f"Done: {pdf_path} ({size_mb:.1f} MB)")
    return pdf_path

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: epub_to_pdf.py <file.epub|file.mobi> [output.pdf]")
        print("       epub_to_pdf.py --batch <directory>")
        sys.exit(1)
    
    if sys.argv[1] == '--batch':
        directory = sys.argv[2] if len(sys.argv) > 2 else '.'
        for ext in ('*.epub', '*.mobi'):
            for f in glob.glob(os.path.join(directory, ext)):
                try:
                    epub_to_pdf(f)
                except Exception as e:
                    print(f"Failed: {f} - {e}")
    else:
        epub_to_pdf(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None)


path: scripts/multi_source_download.py
content: #!/usr/bin/env python3
"""Multi-source download with fallback: Source A -> Source B -> Source C (Anna's) -> epub->pdf"""
import sys, os, asyncio, json, time, glob, shutil, subprocess

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, SCRIPT_DIR)
PYTHON = [REDACTED_SECRET]

BOOKS = [
    {"title": "菌群大脑", "author": "克里斯廷·洛伯格", "search": "菌群大脑 克里斯廷·洛伯格"},
    {"title": "疯狂的尿酸", "author": "戴维·珀尔马特", "search": "疯狂的尿酸 痛风 戴维·珀尔马特"},
    {"title": "逆龄饮食", "author": "安德烈亚斯·乔普", "search": "逆龄饮食 再生医学 乔普"},
]

OUTPUT_DIR = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/Documents/Books/微信读书/医学&健康/")


def run_source_a(title, author, output_dir):
    """Source A: dushupai.com (primary, ZIP with PDF inside)"""
    import importlib.util
    spec = importlib.util.spec_from_file_location("download_book", f"{SCRIPT_DIR}/download_book.py")
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    
    async def _dl():
        try:
            result = await mod.download_book(title=title, author=author, output_dir=output_dir, headless=True)
            return result
        except Exception as e:
            print(f"  Source A error: {e}")
            return None
    
    return asyncio.run(_dl())


def run_source_b(title, author, output_dir):
    """Source B: yabook.org -> file host download"""
    from search_secondary_source import search_yabook, decrypt_ctfile, get_download_url
    
    async def _dl():
        try:
            results = await search_yabook(title, author, headless=True)
            if not results:
                print("  Source B: no results")
                return None
            
            # Try first result
            r = results[0]
            print(f"  Source B found: {r.get('title', '?')}")
            
            # Get download link from file host
            if not r.get('file_host_url'):
                print("  Source B: no file host URL")
                return None
            
            # Download via curl
            url = r['file_host_url']
            [REDACTED_SECRET]'password', '')
            out_path = os.path.join(output_dir, f"{title} - {author}.pdf")
            
            # Try direct curl first
            cmd = ["curl", "-L", "-o", out_path, "--max-time", "300", url]
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=310)
            if os.path.exists(out_path) and os.path.getsize(out_path) > 5000:
                return out_path
            
            # If password-protected, need browser decrypt
            print(f"  Source B: direct download failed, trying browser decrypt...")
            dl_url = await get_download_url(r['file_host_url'], pwd, headless=True)
            if dl_url:
                cmd = ["curl", "-L", "-o", out_path, "--max-time", "600", dl_url]
                subprocess.run(cmd, capture_output=True, text=True, timeout=610)
                if os.path.exists(out_path) and os.path.getsize(out_path) > 5000:
                    return out_path
            
            return None
        except Exception as e:
            print(f"  Source B error: {e}")
            return None
    
    return asyncio.run(_dl())


def run_source_c(title, author, output_dir):
    """Source C: Anna's Archive"""
    from search_source_c import download_book_from_annas_archive
    
    async def _dl():
        try:
            result = await download_book_from_annas_archive(title, author, output_dir=output_dir, headless=True)
            if result and result.get('status') == 'done' and result.get('files'):
                for f in result['files']:
                    if os.path.exists(f) and os.path.getsize(f) > 5000:
                        return f
            print("  Source C: no valid download")
            return None
        except Exception as e:
            print(f"  Source C error: {e}")
            return None
    
    return asyncio.run(_dl())


def convert_epub_to_pdf(epub_path, output_dir):
    """Convert epub to PDF using weasyprint"""
    pdf_path = epub_path.rsplit('.', 1)[0] + '.pdf'
    try:
        cmd = [PYTHON, f"{SCRIPT_DIR}/epub_to_pdf.py", epub_path, pdf_path]
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
        if os.path.exists(pdf_path) and os.path.getsize(pdf_path) > 5000:
            return pdf_path
    except Exception as e:
        print(f"  Convert error: {e}")
    return None


def download_book_all_sources(book, output_dir):
    """Try all sources in order, fallback to epub->pdf conversion"""
    title, author = book["title"], book["author"]
    
    # Check if PDF already exists
    existing = glob.glob(os.path.join(output_dir, f"*{title}*.pdf"))
    for e in existing:
        if os.path.getsize(e) > 1000:
            print(f"  ⏭️ {title} already exists: {os.path.basename(e)}")
            return True
    
    sources = [
        ("Source A (dushupai)", lambda: run_source_a(title, author, output_dir)),
        ("Source B (yabook)", lambda: run_source_b(title, author, output_dir)),
        ("Source C (Anna's)", lambda: run_source_c(title, author, output_dir)),
    ]
    
    last_epub = None
    
    for src_name, src_fn in sources:
        print(f"\n  🔄 Trying {src_name}...")
        try:
            result = src_fn()
        except Exception as e:
            print(f"  ❌ {src_name} crashed: {e}")
            result = None
            continue
        
        if result:
            if isinstance(result, list):
                for f in result:
                    if os.path.exists(f):
                        size = os.path.getsize(f)
                        if size > 5000:
                            ext = os.path.splitext(f)[1].lower()
                            if ext == '.pdf':
                                print(f"  ✅ {src_name} → PDF: {os.path.basename(f)} ({size/1024/1024:.1f} MB)")
                                return True
                            elif ext == '.epub':
                                last_epub = f
                                print(f"  📥 {src_name} → EPUB: {os.path.basename(f)} ({size/1024/1024:.1f} MB)")
                            elif ext == '.zip':
                                # Source A returns ZIP, extract and check for PDF
                                print(f"  📦 {src_name} → ZIP: {os.path.basename(f)} ({size/1024/1024:.1f} MB)")
                                # Extract
                                extract_dir = f"/tmp/ebook_extract_{int(time.time())}"
                                os.makedirs(extract_dir, exist_ok=True)
                                try:
                                    subprocess.run(["unzip", "-o", "-d", extract_dir, f], capture_output=True, timeout=60)
                                    pdfs = glob.glob(os.path.join(extract_dir, "**/*.pdf"), recursive=True)
                                    for pdf in pdfs:
                                        if os.path.getsize(pdf) > 5000:
                                            dest = os.path.join(output_dir, f"{title} - {author}.pdf")
                                            if os.path.exists(dest):
                                                os.remove(dest)
                                            shutil.copy2(pdf, dest)
                                            print(f"  ✅ Extracted PDF: {os.path.basename(pdf)} ({os.path.getsize(pdf)/1024/1024:.1f} MB)")
                                            shutil.rmtree(extract_dir)
                                            return True
                                    # Check for epub inside zip
                                    epubs = glob.glob(os.path.join(extract_dir, "**/*.epub"), recursive=True)
                                    for epub in epubs:
                                        if os.path.getsize(epub) > 5000:
                                            last_epub = epub
                                            print(f"  📥 Found EPUB in ZIP")
                                    shutil.rmtree(extract_dir)
                                except Exception as e:
                                    print(f"  ❌ Extract failed: {e}")
                                    shutil.rmtree(extract_dir, ignore_errors=True)
            elif isinstance(result, str) and os.path.exists(result):
                size = os.path.getsize(result)
                if size > 5000:
                    ext = os.path.splitext(result)[1].lower()
                    if ext == '.pdf':
                        print(f"  ✅ {src_name} → PDF ({size/1024/1024:.1f} MB)")
                        return True
                    elif ext == '.epub':
                        last_epub = result
                        print(f"  📥 {src_name} → EPUB ({size/1024/1024:.1f} MB)")
        time.sleep(3)
    
    # Fallback: convert epub to pdf
    if last_epub:
        print(f"\n  🔄 Converting EPUB → PDF with weasyprint...")
        pdf = convert_epub_to_pdf(last_epub, output_dir)
        if pdf:
            print(f"  ✅ Converted: {os.path.basename(pdf)} ({os.path.getsize(pdf)/1024/1024:.1f} MB)")
            return True
    
    return False


if __name__ == "__main__":
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    done, failed = 0, 0
    
    for i, book in enumerate(BOOKS, 1):
        print(f"\n{'='*60}")
        print(f"📚 [{i}/{len(BOOKS)}] {book['title']} - {book['author']}")
        print(f"{'='*60}")
        
        if download_book_all_sources(book, OUTPUT_DIR):
            done += 1
        else:
            failed += 1
        
        time.sleep(5)
    
    print(f"\n{'='*60}")
    print(f"📊 Results: {done} done, {failed} failed")
    print(f"{'='*60}")


path: scripts/search_secondary_source.py
content: #!/usr/bin/env python3
"""
Search secondary online book source for a Chinese ebook and extract file host download link + password.
Usage: python search_secondary_source.py "书名" ["作者"]
"""
import sys
import os
import re
import json
import asyncio
from typing import Optional, List
from urllib.parse import quote, urljoin

try:
    from playwright.async_api import async_playwright
except ImportError:
    print("Install playwright: pip install playwright && playwright install chromium")
    sys.exit(1)

SOURCE_B_BASE_URL = os.environ.get("SOURCE_B_BASE_URL", "https://yabook.org")
FILE_HOST_BASE_URL = os.environ.get("FILE_HOST_BASE_URL", "https://z701.com")


async def search_yabook(title: str, author: str = "", headless: bool = True, browser=None) -> List[dict]:
    """Search secondary book source and return results with file host links."""
    query = f"{title} {author}".strip()
    results = []
    _owns_browser = browser is None
    if _owns_browser:
        p = await async_playwright().start()
        browser = await p.chromium.launch(headless=headless)
    page = await browser.new_page()

    try:
        search_url = f"{SOURCE_B_BASE_URL}/?s={quote(query)}"
        print(f"Searching secondary source for: {query}")
        await page.goto(search_url, wait_until='domcontentloaded', timeout=30000)
        await asyncio.sleep(2)

        items = await page.query_selector_all('article .entry-title a')
        
        if not items:
            items = await page.query_selector_all('.excerpt a[title]')
        
        if not items:
            print("No search results found")
            return []

        for item in items[:5]:
            try:
                item_title = (await item.inner_text()).strip()
                item_href = await item.get_attribute('href')
                
                if not item_href:
                    continue
                
                full_url = urljoin(SOURCE_B_BASE_URL, item_href)
                
                print(f"  Checking: {item_title[:40]}...")
                await page.goto(full_url, wait_until='domcontentloaded', timeout=20000)
                await asyncio.sleep(2)

                file_link = await _extract_file_host_link(page)
                password = await _extract_password(page)
                
                if file_link:
                    print(f"  Found file host link + [REDACTED_SECRET]")
                    results.append({
                        "title": item_title,
                        "url": full_url,
                        "ctfile_url": file_link,
                        "password": password
                    })
                    break
            except Exception as e:
                print(f"  Error: {e}")
                continue
    finally:
        await page.close()
        if _owns_browser:
            await browser.close()
            await p.stop()
    
    return results


async def _extract_file_host_link(page) -> Optional[str]:
    """Extract file host download link from page."""
    # Method 1: Direct link in page
    links = await page.query_selector_all('a[href*="ctfile.com"]')
    for link in links:
        href = await link.get_attribute('href')
        if href and '/f/' in href:
            return href
    
    # Method 2: In /go/ redirect links
    links = await page.query_selector_all('a[href*="/go/"]')
    for link in links:
        href = await link.get_attribute('href')
        if href:
            # Click to follow redirect
            try:
                resp = await link.evaluate('el => fetch(el.href).then(r => r.url)', link)
                if 'ctfile.com' in resp:
                    return resp
            except:
                pass
    
    return None


async def _extract_password(page) -> str:
    """Extract download password from page."""
    text = await (await page.query_selector('article') or await page.query_selector('main') or await page.query_selector('body')).inner_text()
    
    # Look for password pattern
    match = re.search(r'密码[：:]\s*(\d{4,8})', text)
    if match:
        return match.group(1)
    
    # Look for extraction code pattern
    match = re.search(r'提取码[：:]\s*(\w{4,6})', text)
    if match:
        return match.group(1)
    
    # Common default passwords (from env or fallback)
    default_pwd = os.environ.get("EBOOK_DEFAULT_PASSWORD", "")
    for pwd in ['202630', default_pwd, '1234', 'abcd']:
        if pwd in text:
            return pwd
    
    return ""


async def decrypt_ctfile(ctfile_url: str, password: str, headless: bool = True, browser=None) -> Optional[dict]:
    """Decrypt file host link and get download API variables."""
    _owns_browser = browser is None
    if _owns_browser:
        p = await async_playwright().start()
        browser = await p.chromium.launch(headless=headless)
    page = await browser.new_page()

    try:
        print(f"Decrypting file host link: {ctfile_url[:50]}...")
        await page.goto(ctfile_url, wait_until='domcontentloaded', timeout=30000)
        await asyncio.sleep(3)
        
        body_text = await (await page.query_selector('body')).inner_text()
        
        if '输入文件' in body_text or '访问密码' in body_text:
            input_box = await page.query_selector('input[type="text"], input[type="password"]')
            if input_box and password:
                await input_box.type(password, delay=50)
                btn = await page.query_selector('button:has-text("解密")')
                if not btn:
                    btn = await page.query_selector('input[type="submit"]')
                if btn:
                    await btn.click()
                    await asyncio.sleep(5)
        
        try:
            result = await page.evaluate("""() => {
                return JSON.stringify({
                    api_server: typeof api_server !== 'undefined' ? api_server : null,
                    userid: typeof userid !== 'undefined' ? userid : null,
                    file_id: typeof file_id !== 'undefined' ? file_id : null,
                    share_id: typeof share_id !== 'undefined' ? share_id : '',
                    file_chk: typeof file_chk !== 'undefined' ? file_chk : null,
                    start_time: typeof start_time !== 'undefined' ? start_time : null,
                    wait_seconds: typeof wait_seconds !== 'undefined' ? wait_seconds : 0,
                    verifycode: typeof verifycode !== 'undefined' ? verifycode : null
                });
            }""")
            data = json.loads(result)
            
            if data.get('api_server') and data.get('file_id'):
                print(f"  Got API variables (file_id: {data['file_id']})")
                return data
            else:
                print("  Could not extract API variables")
                return None
        except Exception as e:
            print(f"  Error extracting variables: {e}")
            return None
    finally:
        await page.close()
        if _owns_browser:
            await browser.close()
            await p.stop()


async def get_download_url(api_vars: dict, browser=None) -> Optional[str]:
    """Call file host API to get real download URL."""
    _owns_browser = browser is None
    if _owns_browser:
        p = await async_playwright().start()
        browser = await p.chromium.launch(headless=True)
    page = await browser.new_page()

    try:
        ct_url = f"{FILE_HOST_BASE_URL}/f/{api_vars['userid']}-{api_vars['file_id']}"
        await page.goto(ct_url, wait_until='domcontentloaded', timeout=15000)
        await asyncio.sleep(2)
        
        try:
            js_code = f"""(async () => {{
                try {{
                    var url = '{api_vars['api_server']}/get_file_url.php'
                        + '?uid={api_vars["userid"]}'
                        + '&fid={api_vars["file_id"]}'
                        + '&folder_id=0'
                        + '&share_id={api_vars.get("share_id", "")}'
                        + '&file_chk={api_vars["file_chk"]}'
                        + '&start_time={api_vars["start_time"]}'
                        + '&wait_seconds={api_vars.get("wait_seconds", 0)}'
                        + '&mb=0&app=0&acheck=0'
                        + '&verifycode={api_vars["verifycode"]}'
                        + '&rd=' + Math.random();
                    var headers = typeof getAjaxHeaders === 'function' ? getAjaxHeaders() : {{}};
                    var resp = await fetch(url, {{headers: headers}});
                    var data = await resp.json();
                    return JSON.stringify(data);
                }} catch(e) {{ return JSON.stringify({{error: e.message}}); }}
            }})()"""
            
            result = await page.evaluate(js_code)
            data = json.loads(result)
            
            if data.get('code') == 200 and data.get('downurl'):
                print(f"  Got download URL ({data.get('file_size', 0)} bytes)")
                return data['downurl']
            else:
                print(f"  API error: {data}")
                return None
        except Exception as e:
            print(f"  Error: {e}")
            return None
    finally:
        await page.close()
        if _owns_browser:
            await browser.close()
            await p.stop()


if __name__ == '__main__':
    if len(sys.argv) < 2:
        print('Usage: python search_secondary_source.py "书名" ["作者"]')
        sys.exit(1)
    
    title = sys.argv[1]
    author = sys.argv[2] if len(sys.argv) > 2 else ""
    
    results = asyncio.run(search_yabook(title, author))
    
    if results:
        print(f"\nFound {len(results)} result(s):")
        for i, r in enumerate(results):
            print(f"  [{i+1}] {r['title']}")
            print(f"      URL: {r['ctfile_url']}")
            print(f"      Password: {r['password']}")
    else:
        print("\nNo results found")


path: scripts/search_source_c.py
content: #!/usr/bin/env python3
"""
Search Source C: Anna's Archive (影子图书馆)
Searches for Chinese ebooks via Anna's Archive mirror domains.

Uses HTTP requests (urllib) for search - reliable and fast.
Browser (Playwright) only used for download attempts.

Usage: python search_source_c.py "书名" ["作者"]
"""
import sys
import os
import re
import asyncio
import subprocess
import urllib.parse
from typing import Optional, List, Dict
from html import unescape

try:
    from playwright.async_api import async_playwright
except ImportError:
    pass  # Only needed for download attempts

# Mirrors to try (priority order)
SOURCE_C_MIRRORS = os.environ.get(
    "SOURCE_C_MIRRORS",
    "https://annas-archive.gl,https://annas-archive.pk,https://annas-archive.gd"
).split(",")

_USER_AGENT = (
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
)

_SEARCH_TIMEOUT = 45  # seconds for search (pages can be 500KB+)


def _fetch_url(url: str, timeout: int = 20) -> Optional[str]:
    """Fetch URL content using curl (more reliable than urllib for DDoS-Guard)."""
    import subprocess
    cmd = [
        "curl", "-s", "-L",
        "--max-time", str(timeout),
        "--connect-timeout", str(min(timeout, 10)),
        "-H", f"User-Agent: {_USER_AGENT}",
        "-H", "Accept: text/html,application/xhtml+xml,*/*;q=0.8",
        "-H", "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8",
        url,
    ]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout + 5)
        if result.returncode == 0 and result.stdout:
            return result.stdout
        return None
    except Exception as e:
        print(f"  curl error for {url[:50]}: {e}")
        return None


def _find_working_mirror() -> Optional[str]:
    """Find a working Anna's Archive mirror."""
    for mirror in SOURCE_C_MIRRORS:
        html = _fetch_url(mirror, timeout=20)  # Server can be slow
        if html and "Anna" in html and len(html) > 5000:
            print(f"  Mirror OK: {mirror} ({len(html)} bytes)")
            return mirror
    return None


def _extract_search_queries(title: str, author: str = "") -> List[str]:
    """Generate multiple search queries from title/author."""
    queries = []
    clean = re.sub(r"[（(].+?[）)]", "", title).strip()
    clean = re.sub(r"[：:]", " ", clean).strip()
    if author:
        queries.append(f"{clean} {author}")
    if clean:
        queries.append(clean)
    short = clean.split()[0] if clean else title
    if len(short) >= 2:
        queries.append(short)
    return queries


def _has_chinese(text: str) -> bool:
    return any("\u4e00" <= c <= "\u9fff" for c in text)


def _title_similarity(query: str, result_title: str) -> float:
    q = query.lower().strip()
    r = result_title.lower().strip()
    if q == r:
        return 1.0
    if q in r or r in q:
        return 0.9
    q_words = set(q)
    r_words = set(r)
    if not q_words or not r_words:
        return 0.0
    return len(q_words & r_words) / max(len(q_words), len(r_words))


def _strip_tags(html: str) -> str:
    """Remove HTML tags and decode entities."""
    text = re.sub(r"<[^>]+>", " ", html)
    text = re.sub(r"&nbsp;", " ", text)
    return unescape(text).strip()


def _parse_search_results(html: str, mirror: str, query: str) -> List[Dict]:
    """Parse Anna's Archive search results HTML.

    Anna's Archive HTML structure per result:
      <a href="/md5/{md5}" class="...custom-a...">TITLE</a>
      <a href="/search?q=AUTHOR" ...>AUTHOR</a>
      <a href="/search?q=PUBLISHER" ...>PUBLISHER, YEAR</a>
      ...metadata spans for format, size, source...
    """
    results = []
    seen_md5s = set()

    # Find each result block: starts with /md5/ link, ends at the next /md5/ or end
    # Pattern: capture the md5 link + everything until the next md5 link
    blocks = re.split(r'href="/md5/', html)[1:]  # Skip text before first match

    for block in blocks:
        md5_match = re.match(r'([a-f0-9]{32})"', block)
        if not md5_match:
            continue
        md5 = md5_match.group(1)
        if md5 in seen_md5s:
            continue

        # Extract title from the first link (the md5 link itself)
        # Two patterns:
        # 1. href="/md5/{md5}" class="...font-semibold...">TITLE</a>  (title in md5 link)
        # 2. href="/md5/{md5}" class="...block...">  (md5 link is an image wrapper)
        title = ""

        # Pattern 1: Title is in the md5 link with font-semibold
        title_match = re.search(
            r'href="/md5/' + md5 + r'"[^>]*font-semibold[^>]*>(.*?)</a>',
            block, re.DOTALL
        )
        if title_match:
            title = _strip_tags(title_match.group(1)).strip()

        # Pattern 2: Title in a subsequent link with text-[#2563eb]
        if not title:
            title_match2 = re.search(
                r'text-\[#2563eb\][^>]*>(.*?)</a>',
                block, re.DOTALL
            )
            if title_match2:
                title = _strip_tags(title_match2.group(1)).strip()

        if not title or not _has_chinese(title):
            continue

        sim = _title_similarity(query, title)
        if sim < 0.15:
            continue

        # Extract author from subsequent link with user-edit icon
        author = ""
        author_match = re.search(r'icon-\[mdi--user-edit\][^>]*>\s*(.*?)</a>', block, re.DOTALL)
        if author_match:
            author = _strip_tags(author_match.group(1)).strip()

        # Extract publisher info (contains year)
        publisher_info = ""
        pub_match = re.search(r'icon-\[mdi--company\][^>]*>\s*(.*?)</a>', block, re.DOTALL)
        if pub_match:
            publisher_info = _strip_tags(pub_match.group(1)).strip()

        # Extract metadata from the block
        fmt_match = re.search(r"\b(pdf|epub|mobi|azw3)\b", block, re.I)
        size_match = re.search(r"([\d.]+\s*(?:MB|KB|GB))", block)
        year_match = re.search(r"\b((?:19|20)\d{2})\b", block)
        source_match = re.search(
            r"\b(zlib|duxiu|lgli|ia|hathi|lgrs|upload|zlibzh|nexusstc)\b",
            block, re.I
        )

        seen_md5s.add(md5)
        results.append({
            "title": title,
            "author": author,
            "md5": md5,
            "format": fmt_match.group(1).lower() if fmt_match else "",
            "size": size_match.group(1) if size_match else "",
            "year": year_match.group(1) if year_match else "",
            "source": source_match.group(1).upper() if source_match else "",
            "publisher": publisher_info,
            "mirror_url": f"{mirror}/md5/{md5}",
            "mirror_domain": mirror,
            "score": sim,
        })

    results.sort(key=lambda x: x["score"], reverse=True)
    return results


def search_annas_archive(
    title: str,
    author: str = "",
    preferred_format: str = "pdf",
) -> List[Dict]:
    """
    Search Anna's Archive for a Chinese ebook (synchronous, HTTP-based).

    Returns list of dicts with keys:
        - title, author, md5, format, size, year, source, mirror_url, mirror_domain, score
    """
    queries = _extract_search_queries(title, author)
    all_results = []

    mirror = _find_working_mirror()
    if not mirror:
        print("  ⚠️  No working mirror found")
        return []

    for qidx, query in enumerate(queries):
        if all_results:
            break

        ext_param = preferred_format if preferred_format != "any" else ""
        url = (
            f"{mirror}/search?q={urllib.parse.quote(query)}&lang=zh"
            + (f"&ext={ext_param}" if ext_param else "")
            + "&sort=&fast=1"
        )
        print(f"  Searching: {query}")

        html = _fetch_url(url, timeout=_SEARCH_TIMEOUT)
        if not html:
            continue

        # Check for DDoS-Guard or error
        if "DDoS-Guard" in html or "checking your browser" in html.lower():
            print("  DDoS-Guard challenge on search page, skipping")
            continue

        results = _parse_search_results(html, mirror, query)
        if results:
            all_results.extend(results)
            print(f"  Found {len(results)} result(s)")
            for r in results[:3]:
                print(f"    - {r['title'][:50]} [{r['format']}] {r['size']} ({r['source']})")
        else:
            # Check if the response was too small (DDoS-Guard page)
            if len(html) < 5000:
                print(f"  Response too small ({len(html)} bytes) - likely blocked")
            else:
                # Check for DDoS-Guard in content
                if "ddos-guard" in html.lower() or "checking your browser" in html.lower():
                    print("  DDoS-Guard challenge in search response")
                else:
                    print("  No matching Chinese results found")

    return all_results


async def search_annas_archive_async(
    title: str,
    author: str = "",
    headless: bool = True,
    browser=None,
    context=None,
    preferred_format: str = "pdf",
) -> List[Dict]:
    """Async wrapper for search (same as sync version, for compatibility)."""
    return search_annas_archive(title, author, preferred_format)


async def get_download_page(
    md5: str,
    mirror_domain: str,
    headless: bool = True,
    browser=None,
    context=None,
) -> Optional[Dict]:
    """
    Get book detail page and extract download options (requires Playwright).
    """
    if "playwright" not in sys.modules:
        return None

    _owns_browser = browser is None
    _owns_context = context is None
    try:
        if _owns_browser:
            pw = await async_playwright().start()
            browser, context = await _create_stealth_context(pw, headless)
        elif _owns_context:
            # Browser provided but not context - create context from browser
            context = await browser.new_context(
                user_agent=_USER_AGENT,
                viewport={"width": 1920, "height": 1080},
                locale="zh-CN",
            )
            await context.add_init_script("""
                Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
                window.chrome = { runtime: {} };
            """)

        page = await context.new_page()
        try:
            url = f"{mirror_domain}/md5/{md5}"
            await page.goto(url, timeout=15000, wait_until="commit")
            await asyncio.sleep(3)

            result = await page.evaluate("""() => {
                const data = {
                    slow_download_urls: [],
                    fast_download_urls: [],
                    external_urls: [],
                    torrent_url: '',
                    info: {}
                };
                document.querySelectorAll('a').forEach(a => {
                    const href = a.getAttribute('href') || '';
                    const text = a.textContent.trim();
                    if (href.includes('/slow_download/')) data.slow_download_urls.push(href);
                    else if (href.includes('/fast_download/')) data.fast_download_urls.push(href);
                    else if (
                        (href.includes('z-lib') || href.includes('libgen'))
                        && text.length > 3 && text.length < 80
                    ) data.external_urls.push({text, href});
                    else if (href.includes('.torrent')) data.torrent_url = href;
                });
                return data;
            }""")
            return result
        finally:
            await page.close()
    except Exception as e:
        print(f"  Detail page error: {e}")
        return None
    finally:
        if _owns_context and context:
            await context.close()
        if _owns_browser and browser:
            await browser.close()


async def _create_stealth_context(playwright, headless: bool = True):
    """Create a browser context with anti-detection measures."""
    browser = await playwright.chromium.launch(
        headless=headless,
        args=["--disable-blink-features=AutomationControlled"],
    )
    context = await browser.new_context(
        user_agent=_USER_AGENT,
        viewport={"width": 1920, "height": 1080},
        locale="zh-CN",
    )
    await context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
        window.chrome = { runtime: {} };
    """)
    return browser, context


async def try_download_libgen(
    md5: str,
    mirror_domain: str,
    output_path: str,
    headless: bool = True,
    browser=None,
    context=None,
) -> Dict:
    """Try to download via libgen.li external link from book detail page."""
    if "playwright" not in sys.modules:
        return {"status": "failed", "error": "Playwright not available"}

    _owns_browser = browser is None
    _owns_context = context is None
    try:
        if _owns_browser:
            pw = await async_playwright().start()
            browser, context = await _create_stealth_context(pw, headless)
        elif _owns_context:
            # Browser provided but not context - create context from browser
            context = await browser.new_context(
                user_agent=_USER_AGENT,
                viewport={"width": 1920, "height": 1080},
                locale="zh-CN",
            )
            await context.add_init_script("""
                Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
                window.chrome = { runtime: {} };
            """)

        page = await context.new_page()
        try:
            # Visit main page first to get DDoS-Guard cookies
            print(f"  Loading detail page for external links...")
            await page.goto(mirror_domain, wait_until="domcontentloaded", timeout=30000)
            await asyncio.sleep(4)

            # Go to book detail page
            url = f"{mirror_domain}/md5/{md5}"
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)
            await asyncio.sleep(3)

            # Extract libgen.li file link
            external_info = await page.evaluate("""() => {
                const links = document.querySelectorAll('a');
                for (const a of links) {
                    const href = a.getAttribute('href') || '';
                    if (href.includes('libgen.li/file.php')) {
                        return {libgen_url: href, text: a.textContent.trim().substring(0, 80)};
                    }
                }
                return {libgen_url: null};
            }""")
            libgen_url = external_info.get("libgen_url")
            if not libgen_url:
                return {"status": "failed", "error": "No libgen.li link found on detail page"}

            print(f"  Found libgen.li link, trying direct download...")

            # Try downloading directly via curl from libgen.li
            cmd = [
                "curl", "-L", "-o", output_path, libgen_url,
                "-H", f"User-Agent: {_USER_AGENT}",
                "-H", f"Referer: {url}",
                "--max-time", "600", "--connect-timeout", "30",
            ]
            result = subprocess.run(cmd, capture_output=True, text=True)
            if result.returncode == 0:
                size = os.path.getsize(output_path) if os.path.exists(output_path) else 0
                if size > 1024:
                    # Verify it's not an HTML error page
                    with open(output_path, 'rb') as f:
                        head = f.read(100)
                    if head[:5] in (b'%PDF-', b'PK\x03\x04', b'\x89PNG'):
                        return {"status": "done", "path": output_path, "size": size}
                    if b'<html' in head[:50].lower() or b'<!doct' in head[:50].lower():
                        os.remove(output_path)
                        return {"status": "failed", "error": "Downloaded HTML error page instead of file"}

            return {"status": "failed", "error": "libgen.li download failed"}
        finally:
            await page.close()
    except Exception as e:
        print(f"  libgen download error: {e}")
        return {"status": "failed", "error": str(e)}
    finally:
        if _owns_context and context:
            await context.close()
        if _owns_browser and browser:
            await browser.close()


async def try_download_slow(
    md5: str,
    mirror_domain: str,
    output_path: str,
    headless: bool = True,
    browser=None,
    context=None,
    max_partners: int = 3,
) -> Dict:
    """Attempt download from slow partner servers via Playwright."""
    if "playwright" not in sys.modules:
        return {"status": "failed", "error": "Playwright not available"}

    _owns_browser = browser is None
    _owns_context = context is None
    try:
        if _owns_browser:
            pw = await async_playwright().start()
            browser, context = await _create_stealth_context(pw, headless)
        elif _owns_context:
            # Browser provided but not context - create context from browser
            context = await browser.new_context(
                user_agent=_USER_AGENT,
                viewport={"width": 1920, "height": 1080},
                locale="zh-CN",
            )
            await context.add_init_script("""
                Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
                window.chrome = { runtime: {} };
            """)

        for partner_idx in range(max_partners):
            page = await context.new_page()
            try:
                url = f"{mirror_domain}/slow_download/{md5}/0/{partner_idx}"
                print(f"  Trying slow partner #{partner_idx + 1}...")
                await page.goto(url, timeout=20000, wait_until="commit")

                # Wait for countdown (up to 45 seconds) checking every 5 seconds
                dl_link = None
                for wait_sec in range(9):
                    await asyncio.sleep(5)
                    elapsed = (wait_sec + 1) * 5
                    
                    dl_link = await page.evaluate("""() => {
                        // Check for external download link that appears after countdown
                        const links = document.querySelectorAll('a');
                        for (const a of links) {
                            const href = a.getAttribute('href') || '';
                            if (href.startsWith('http') && !href.includes('annas-archive') 
                                && !href.includes('reddit') && !href.includes('matrix')
                                && !href.includes('translate') && !href.includes('software.')
                                && !href.includes('open-slum') && !href.includes('jdownloader')
                                && href.length > 20 && !href.includes('javascript')) {
                                return href;
                            }
                        }
                        return null;
                    }""")
                    
                    if dl_link:
                        print(f"  Download link found after {elapsed}s")
                        break
                    
                    # Check if we're still on the countdown page
                    text = await page.evaluate('() => document.body.innerText')
                    if 'Please wait' not in text and 'countdown' not in text.lower():
                        # Page might have loaded or been redirected
                        url_now = page.url
                        if 'slow_download' not in url_now:
                            print(f"  Page redirected away from slow_download after {elapsed}s")
                            break
                
                if dl_link:
                    cmd = ["curl", "-L", "-o", output_path, dl_link,
                           "-H", f"User-Agent: {_USER_AGENT}",
                           "-H", f"Referer: {url}",
                           "--max-time", "600", "--connect-timeout", "30"]
                    result = subprocess.run(cmd, capture_output=True, text=True)
                    if result.returncode == 0:
                        size = os.path.getsize(output_path) if os.path.exists(output_path) else 0
                        if size > 1024:
                            with open(output_path, 'rb') as f:
                                head = f.read(100)
                            if head[:5] in (b'%PDF-', b'PK\x03\x04', b'\x89PNG') or b'<html' not in head[:50].lower():
                                return {"status": "done", "path": output_path, "size": size}
                            else:
                                os.remove(output_path)
                    return {"status": "failed", "error": "Download failed or got error page"}
                
                return {"status": "blocked", "error": f"No download link after countdown (partner #{partner_idx+1})"}
                print(f"  Partner #{partner_idx + 1} error: {e}")
            finally:
                await page.close()

        return {"status": "blocked", "error": "All partners blocked by DDoS-Guard"}
    except Exception as e:
        print(f"  slow download error: {e}")
        return {"status": "failed", "error": str(e)}
    finally:
        if _owns_context and context:
            await context.close()
        if _owns_browser and browser:
            await browser.close()


async def download_book_from_annas_archive(
    title: str,
    author: str,
    output_dir: str,
    preferred_format: str = "pdf",
    headless: bool = True,
    browser=None,
    context=None,
) -> Dict:
    """Full download flow: search → find best result → attempt download."""
    import re

    # Search without format filter to get all available results
    # The format preference is handled when selecting the best result
    results = search_annas_archive(title, author, "any")
    if not results:
        return {"status": "not_found", "error": "No results on Anna's Archive"}

    # Prefer the requested format, but fall back to any available
    if preferred_format != "any":
        preferred_results = [r for r in results if r.get("format", "").lower() == preferred_format.lower()]
        if preferred_results:
            results = preferred_results

    best = results[0]
    print(f"  Best match: {best['title'][:50]} [{best['format']}] ({best['source']})")

    if preferred_format != "any" and best.get("format"):
        if best["format"].lower() != preferred_format.lower():
            print(f"  ⚠️  Format mismatch: wanted {preferred_format}, got {best['format']}")

    # Try download via browser
    clean_name = re.sub(r'[\\/:*?"<>|]', ' ', f"{title} - {author}").strip()
    ext = best.get("format", preferred_format if preferred_format != "any" else "pdf")
    output_path = os.path.join(output_dir, f"{clean_name}.{ext}")

    # Strategy 1: Try libgen.li external link (most reliable for direct download)
    dl_result = await try_download_libgen(
        best["md5"], best["mirror_domain"], output_path,
        headless=headless, browser=browser, context=context,
    )
    if dl_result["status"] == "done":
        return {"status": "done", "files": [output_path], "source": f"annas-archive→libgen ({best['source']})"}

    print(f"  libgen.li failed: {dl_result.get('error', 'unknown')}")

    # Strategy 2: Try slow download partners via browser
    dl_result = await try_download_slow(
        best["md5"], best["mirror_domain"], output_path,
        headless=headless, browser=browser, context=context,
    )

    if dl_result["status"] == "done":
        return {"status": "done", "files": [output_path], "source": f"annas-archive ({best['source']})"}

    return {
        "status": "blocked",
        "error": dl_result.get("error", "Download blocked"),
        "search_results": results[:3],
        "manual_url": best["mirror_url"],
    }


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print('Usage: python search_source_c.py "书名" ["作者"]')
        sys.exit(1)

    title = sys.argv[1]
    author = sys.argv[2] if len(sys.argv) > 2 else ""

    results = search_annas_archive(title, author)

    if results:
        print(f"\nFound {len(results)} result(s):")
        for i, r in enumerate(results):
            print(f"  [{i+1}] {r['title'][:60]}")
            print(f"      Format: {r['format']} | Size: {r['size']} | Year: {r['year']}")
            print(f"      Source: {r['source']} | Score: {r['score']:.2f}")
            print(f"      URL: {r['mirror_url']}")
    else:
        print("\nNo results found")


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)