Skill: matusvojtek/tubescribe (version 1.1.8)

--- name: TubeScribe description: "YouTube video summarizer with speaker detection, formatted documents, and audio output. Works out of the box with macOS built-in TTS. Optional recommended tools (pandoc, ffmpeg, mlx-audio) enhance quality. Requires internet for YouTube access. No paid APIs or subscriptions. Use when user sends a YouTube URL or asks to summarize/transcribe a YouTube video." metadata: { "openclaw": { "emoji": "🎬", "requires": { "bins": ["summarize"] } } } --- # TubeScribe 🎬 **Turn any YouTube video into a polished document + audio summary.** Drop a YouTube link → get a beautiful transcript with speaker labels, key quotes, timestamps that link back to the video, and an audio summary you can listen to on the go. ### 💸 Free & No Paid APIs - **No subscriptions or API keys** — works out of the box - **Local processing** — transcription, speaker detection, and TTS run on your machine - **Network access** — fetching from YouTube (captions, metadata, comments) requires internet - **No data uploaded** — nothing is sent to external services; all processing stays on your machine - **Safe sub-agent** — spawned sub-agent has strict instructions: no software installation, no network calls beyond YouTube ### ✨ Features - **📄 Transcript with summary and key quotes** — Export as DOCX, HTML, or Markdown - **🎯 Smart Speaker Detection** — Automatically identifies participants - **🔊 Audio Summaries** — Listen to key points (MP3/WAV) - **📝 Clickable Timestamps** — Every quote links directly to that moment in the video - **💬 YouTube Comments** — Viewer sentiment analysis and best comments - **📋 Queue Support** — Send multiple links, they get processed in order - **🚀 Non-Blocking Workflow** — Conversation continues while video processes in background ### 🎬 Works With Any Video - Interviews & podcasts (multi-speaker detection) - Lectures & tutorials (single speaker) - Music videos (lyrics extraction) - News & documentaries - Any YouTube content with captions ## Quick Start When user sends a YouTube URL: 1. Spawn sub-agent with the full pipeline task **immediately** 2. Reply: "🎬 TubeScribe is processing — I'll let you know when it's ready!" 3. Continue conversation (don't wait!) 4. Sub-agent notification will announce completion with title and details **DO NOT BLOCK** — spawn and move on instantly. ## First-Time Setup Run setup to check dependencies and configure defaults: ```bash python skills/tubescribe/scripts/setup.py ``` This checks: `summarize` CLI, `pandoc`, `ffmpeg`, `Kokoro TTS` ## Full Workflow (Single Sub-Agent) Spawn ONE sub-agent that does the entire pipeline: ```python sessions_spawn( task=f""" ## TubeScribe: Process {youtube_url} ⚠️ CRITICAL: Do NOT install any software. No pip, brew, curl, venv, or binary downloads. If a tool is missing, STOP and report what's needed. Run the COMPLETE pipeline — do not stop until all steps are done. ### Step 1: Extract ```bash python3 skills/tubescribe/scripts/tubescribe.py "{youtube_url}" ``` Note the **Source** and **Output** paths printed by the script. Use those exact paths in subsequent steps. ### Step 2: Read source JSON Read the Source path from Step 1 output and note: - metadata.title (for filename) - metadata.video_id - metadata.channel, upload_date, duration_string ### Step 3: Create formatted markdown Write to the Output path from Step 1: 1. `# **<title>**` --- 2. Video info block — Channel, Date, Duration, URL (clickable). Empty line between each field. --- 3. `## **Participants**` — table with bold headers: ``` | **Name** | **Role** | **Description** | |----------|----------|-----------------| ``` --- 4. `## **Summary**` — 3-5 paragraphs of prose --- 5. `## **Key Quotes**` — 5 best with clickable YouTube timestamps. Format each as: ``` "Quote text here." - [12:34](https://www.youtube.com/watch?v=ID&t=754s) "Another quote." - [25:10](https://www.youtube.com/watch?v=ID&t=1510s) ``` Use regular dash `-`, NOT em dash `—`. Do NOT use blockquotes `>`. Plain paragraphs only. --- 6. `## **Viewer Sentiment**` (if comments exist) --- 7. `## **Best Comments**` (if comments exist) — Top 5, NO lines between them: ``` Comment text here. *- ▲ 123 @AuthorName* Next comment text here. *- ▲ 45 @AnotherAuthor* ``` Attribution line: dash + italic. Just blank line between comments, NO `---` separators. --- 8. `## **Full Transcript**` — merge segments, speaker labels, clickable timestamps ### Step 4: Create DOCX Clean the title for filename (remove special chars), then: ```bash pandoc <output_path> -o ~/Documents/TubeScribe/<safe_title>.docx ``` ### Step 5: Generate audio Write the summary text to a temp file, then use TubeScribe's built-in audio generation: ```bash # Write summary to temp file (use python3 to write, avoids shell escaping issues) python3 -c " text = '''YOUR SUMMARY TEXT HERE''' with open('<temp_dir>/tubescribe_<video_id>_summary.txt', 'w') as f: f.write(text) " # Generate audio (auto-detects engine, voice, format from config) python3 skills/tubescribe/scripts/tubescribe.py \ --generate-audio <temp_dir>/tubescribe_<video_id>_summary.txt \ --audio-output ~/Documents/TubeScribe/<safe_title>_summary ``` This reads `~/.tubescribe/config.json` and uses the configured TTS engine (mlx/kokoro/builtin), voice blend, and speed automatically. Output format (mp3/wav) comes from config. ### Step 6: Cleanup ```bash python3 skills/tubescribe/scripts/tubescribe.py --cleanup <video_id> ``` ### Step 7: Open folder ```bash open ~/Documents/TubeScribe/ ``` ### Report Tell what was created: DOCX name, MP3 name + duration, video stats. """, label="tubescribe", runTimeoutSeconds=900, cleanup="delete" ) ``` **After spawning, reply immediately:** > 🎬 TubeScribe is processing - I'll let you know when it's ready! Then continue the conversation. The sub-agent notification announces completion. ## Configuration Config file: `~/.tubescribe/config.json` ```json { "output": { "folder": "~/Documents/TubeScribe", "open_folder_after": true, "open_document_after": false, "open_audio_after": false }, "document": { "format": "docx", "engine": "pandoc" }, "audio": { "enabled": true, "format": "mp3", "tts_engine": "mlx" }, "mlx_audio": { "path": "~/.openclaw/tools/mlx-audio", "model": "mlx-community/Kokoro-82M-bf16", "voice": "af_heart", "lang_code": "a", "speed": 1.05 }, "kokoro": { "path": "~/.openclaw/tools/kokoro", "voice_blend": { "af_heart": 0.6, "af_sky": 0.4 }, "speed": 1.05 }, "processing": { "subagent_timeout": 600, "cleanup_temp_files": true } } ``` ### Output Options | Option | Default | Description | |--------|---------|-------------| | `output.folder` | `~/Documents/TubeScribe` | Where to save files | | `output.open_folder_after` | `true` | Open output folder when done | | `output.open_document_after` | `false` | Auto-open generated document | | `output.open_audio_after` | `false` | Auto-open generated audio summary | ### Document Options | Option | Default | Values | Description | |--------|---------|--------|-------------| | `document.format` | `docx` | `docx`, `html`, `md` | Output format | | `document.engine` | `pandoc` | `pandoc` | Converter for DOCX (falls back to HTML) | ### Audio Options | Option | Default | Values | Description | |--------|---------|--------|-------------| | `audio.enabled` | `true` | `true`, `false` | Generate audio summary | | `audio.format` | `mp3` | `mp3`, `wav` | Audio format (mp3 needs ffmpeg) | | `audio.tts_engine` | `mlx` | `mlx`, `kokoro`, `builtin` | TTS engine (mlx = fastest on Apple Silicon) | ### MLX-Audio Options (preferred on Apple Silicon) | Option | Default | Description | |--------|---------|-------------| | `mlx_audio.path` | `~/.openclaw/tools/mlx-audio` | mlx-audio venv location | | `mlx_audio.model` | `mlx-community/Kokoro-82M-bf16` | MLX model to use | | `mlx_audio.voice` | `af_heart` | Voice preset (used if no voice_blend) | | `mlx_audio.voice_blend` | `{af_heart: 0.6, af_sky: 0.4}` | Custom voice mix (weighted blend) | | `mlx_audio.lang_code` | `a` | Language code (a=US English) | | `mlx_audio.speed` | `1.05` | Playback speed (1.0 = normal, 1.05 = 5% faster) | ### Kokoro PyTorch Options (fallback) | Option | Default | Description | |--------|---------|-------------| | `kokoro.path` | `~/.openclaw/tools/kokoro` | Kokoro repo location | | `kokoro.voice_blend` | `{af_heart: 0.6, af_sky: 0.4}` | Custom voice mix | | `kokoro.speed` | `1.05` | Playback speed (1.0 = normal, 1.05 = 5% faster) | ### Processing Options | Option | Default | Description | |--------|---------|-------------| | `processing.subagent_timeout` | `600` | Seconds for sub-agent (increase for long videos) | | `processing.cleanup_temp_files` | `true` | Remove /tmp files after completion | ### Comment Options | Option | Default | Description | |--------|---------|-------------| | `comments.max_count` | `50` | Number of comments to fetch | | `comments.timeout` | `90` | Timeout for comment fetching (seconds) | ### Queue Options | Option | Default | Description | |--------|---------|-------------| | `queue.stale_minutes` | `30` | Consider a processing job stale after this many minutes | ## Output Structure ``` ~/Documents/TubeScribe/ ├── {Video Title}.html # Formatted document (or .docx / .md) └── {Video Title}_summary.mp3 # Audio summary (or .wav) ``` After generation, opens the folder (not individual files) so you can access everything. ## Dependencies **Required:** - `summarize` CLI — `brew install steipete/tap/summarize` - Python 3.8+ **Optional (better quality):** - `pandoc` — DOCX output: `brew install pandoc` - `ffmpeg` — MP3 audio: `brew install ffmpeg` - `yt-dlp` — YouTube comments: `brew install yt-dlp` - mlx-audio — Fastest TTS on Apple Silicon: `pip install mlx-audio` (uses MLX backend for Kokoro) - Kokoro TTS — PyTorch fallback: see https://github.com/hexgrad/kokoro ### yt-dlp Search Paths TubeScribe checks these locations (in order): | Priority | Path | Source | |----------|------|--------| | 1 | `which yt-dlp` | System PATH | | 2 | `/opt/homebrew/bin/yt-dlp` | Homebrew (Apple Silicon) | | 3 | `/usr/local/bin/yt-dlp` | Homebrew (Intel) / Linux | | 4 | `~/.local/bin/yt-dlp` | pip install --user | | 5 | `~/.local/pipx/venvs/yt-dlp/bin/yt-dlp` | pipx | | 6 | `~/.openclaw/tools/yt-dlp/yt-dlp` | TubeScribe auto-install | If not found, setup downloads a standalone binary to the tools directory. The tools directory version doesn't conflict with system installations. ## Queue Handling When user sends multiple YouTube URLs while one is processing: ### Check Before Starting ```bash python skills/tubescribe/scripts/tubescribe.py --queue-status ``` ### If Already Processing ```bash # Add to queue instead of starting parallel processing python skills/tubescribe/scripts/tubescribe.py --queue-add "NEW_URL" # → Replies: "📋 Added to queue (position 2)" ``` ### After Completion ```bash # Check if more in queue python skills/tubescribe/scripts/tubescribe.py --queue-next # → Automatically pops and processes next URL ``` ### Queue Commands | Command | Description | |---------|-------------| | `--queue-status` | Show what's processing + queued items | | `--queue-add URL` | Add URL to queue | | `--queue-next` | Process next item from queue | | `--queue-clear` | Clear entire queue | ### Batch Processing (multiple URLs at once) ```bash python skills/tubescribe/scripts/tubescribe.py url1 url2 url3 ``` Processes all URLs sequentially with a summary at the end. ## Error Handling The script detects and reports these errors with clear messages: | Error | Message | |-------|---------| | Invalid URL | ❌ Not a valid YouTube URL | | Private video | ❌ Video is private — can't access | | Video removed | ❌ Video not found or removed | | No captions | ❌ No captions available for this video | | Age-restricted | ❌ Age-restricted video — can't access without login | | Region-blocked | ❌ Video blocked in your region | | Live stream | ❌ Live streams not supported — wait until it ends | | Network error | ❌ Network error — check your connection | | Timeout | ❌ Request timed out — try again later | When an error occurs, report it to the user and don't proceed with that video. ## Tips - For long videos (>30 min), increase sub-agent timeout to 900s - Speaker detection works best with clear interview/podcast formats - Single-speaker videos (tutorials, lectures) skip speaker labels automatically - Timestamps link directly to YouTube at that moment - Use batch mode for multiple videos: `tubescribe url1 url2 url3`

path: CHANGELOG.md
content: # Changelog

All notable changes to TubeScribe.

## [1.1.8] - 2026-02-10

### Changed
- **Description updated** — clarifies works out of the box, optional tools enhance quality, internet required for YouTube
- Added **OpenClaw metadata** — declares `summarize` as required binary (`requires.bins`)
- Added security clarifications to SKILL.md: no data uploaded, sub-agent has strict no-install instructions
- Expanded setup.py documentation in README (what it does and doesn't do)
- Fixed display name on ClawHub: "TubeScribe" (was "Tubescribe")

## [1.1.7] - 2026-02-10

### Security
- **Code injection risk in voice blending** — `blend_name` was interpolated raw into subprocess code strings; now passed via `json.dumps` for safe escaping
- **URL validation bypass** — replaced regex-based domain extraction with `urllib.parse.urlparse()` to prevent bypass via URLs like `[REDACTED_SECRET]`
- **CSS-based XSS in HTML writer** — added `style` attribute stripping alongside existing `on*` event handler removal

### Fixed
- **`is_processing()` had write side-effects** — split into read-only `is_processing()` and separate `clear_stale_current()` for stale entry cleanup
- **Duplicate `tempfile` import** — removed redundant local import in `generate_builtin_audio` (already imported at module level)
- **Silent fallthrough for unknown doc formats** — `convert_to_document` now warns and falls back to markdown instead of silently returning raw path
- **`prompt_yn` crash in non-interactive mode** — `input()` now catches `EOFError`/`KeyboardInterrupt` and returns `False`
- **Hardcoded pandoc version** — `install_pandoc()` now fetches latest release from GitHub API, falls back to known version
- **Fragile misaki espeak patch** — replaced exact string match with regex pattern; skips if already patched; warns if target code has changed

### Added
- **Markdown list support in HTML writer** — unordered (`- item`), ordered (`1. item`) lists now render as `<ul>`/`<ol>` with proper styling
- **Fenced code block support** — ` ``` ` blocks render as `<pre><code>` with syntax-appropriate styling
- **Inline code support** — `` `code` `` renders as `<code>` with monospace background
- **Config validation** — `load_config()` now validates types and ranges (format enums, numeric bounds, booleans); invalid values silently revert to defaults
- **File locking on `save_config`** — uses `fcntl.flock` to prevent concurrent write corruption (matches queue locking pattern)
- **Missing fcntl warning** — queue operations now log a stderr warning (once) when file locking is unavailable on non-Unix platforms

### Changed
- **"100% Free & Local" → "Free & No Paid APIs"** — clarified that YouTube fetching requires internet; processing and TTS are local
- Added **Privacy & Network** section to README — explains exactly what uses network vs what runs locally
- Clarified that config paths (`~/.tubescribe/config.json`) point to local TTS installations, not remote services
- Updated SKILL.md to match new wording
- Updated author line: `Jackie 🦊 & Matus 🇸🇰`
- Updated footer: `Made by Jackie 🦊 & Matus 🇸🇰`
- **Renamed `config.get()` → `config.get_value()`** — avoids shadowing Python's built-in `get`

## [1.1.6] - 2026-02-10

### Fixed
- Wrapped long code line in SKILL.md that caused horizontal scroll on ClawHub page

## [1.1.5] - 2026-02-10

### Added
- **MLX-Audio TTS backend** — 3-4x faster audio generation on Apple Silicon via Apple's MLX framework
  - Uses `mlx-community/Kokoro-82M-bf16` — same Kokoro model, native Metal GPU backend
  - New `mlx_audio` config section for model, voice, language, and speed settings
  - Custom voice blending support (`voice_blend` config with arbitrary weights, e.g. 60% af_heart + 40% af_sky)
  - Blended voices cached as `.safetensors` — created once, reused on subsequent runs
  - Graceful fallback chain: mlx → kokoro (PyTorch) → builtin (macOS say)
- **`find_mlx_audio()` helper** — detects mlx-audio venv or system install
- **`_get_or_create_mlx_blended_voice()`** — creates and caches blended voice tensors in HF cache
- **Setup wizard with platform-aware TTS install** — detects Apple Silicon, offers mlx-audio; offers PyTorch Kokoro on other platforms
- **`install_mlx_audio()`** — creates venv, installs deps, patches misaki/espeak for homebrew, verifies
- **`is_apple_silicon()` helper** — platform detection for install decisions

### Changed
- **Default TTS engine is now `mlx`** — fastest option on Apple Silicon (M1/M2/M3/M4)
- **TTS engine priority:** `mlx` > `kokoro` > `builtin` (was: `kokoro` > `builtin`)
- **Config default updated** — `audio.tts_engine` default changed from `builtin` to `mlx`
- **Kokoro install rewritten** — clean venv + pip install (no more git clone of 2GB repo with CoreML models)
- **`find_kokoro()` simplified** — checks venv first, then system Python, then legacy locations
- **SKILL.md** — updated audio generation instructions with mlx-audio commands
- **`fcntl` import is now conditional** — prevents crash on non-Unix systems; queue locking degrades gracefully to no-op
- **`determine_config` uses `deep_merge`** — was using shallow `.update()` which could lose nested user customizations
- **Atomic queue operations** — replaced separate `load_queue()` + `save_queue()` with `_locked_queue()` context manager that holds exclusive lock across read-modify-write
- **Temp files use per-user directory** — `/tmp/tubescribe-{uid}/` with `0o700` permissions instead of predictable `/tmp/tubescribe_{video_id}_*` paths

### Fixed
- **Double JSON encoding in Kokoro TTS** — `voice_blend` was double-encoded via `json.dumps(json.dumps(...))`, now uses `!r` repr formatting
- **Builtin TTS mp3 path handling** — `generate_builtin_audio` could produce wrong filenames when `output_path` didn't end with `.mp3`; now matches the defensive checks in mlx/kokoro generators
- **HTML injection via raw passthrough** — markdown lines starting with `<` were passed through unescaped; dangerous tags (`script`, `iframe`, `object`, `embed`, `form`, etc.) are now escaped
- **Recursive fallback in `convert_to_document`** — docx-to-html fallback used a recursive call; replaced with direct HTML generation
- **Dead code in `install_mlx_audio`** — removed unused `espeak_py` variable that was immediately overwritten by glob
- **`safe_unescape` replacement order** — backslash escapes were processed in wrong order; `\\n` in input would incorrectly become a newline instead of literal `\n`. Double-backslash replacement now runs first
- **Hardcoded `max_comments` ignored config** — `prepare_source_data` passed `max_comments=50` directly, ignoring `comments.max_count` from config
- **`view_count` dropped from source data** — `get_video_metadata` returned `view_count` but `prepare_source_data` didn't include it in the output JSON
- **Missing `encoding='utf-8'` in `convert_to_document`** — file reads/writes could break on systems where default encoding isn't UTF-8
- **Unhandled exception in `--generate-audio`** — if all TTS engines failed, raw traceback was shown instead of clean error message
- **`text=True` missing in `find_mlx_audio`** — subprocess calls were inconsistent with the rest of the codebase
- **yt-dlp `max_comments` format** — was passing 4 values (undocumented format); now uses correct 5-value format `COUNT,COUNT,0,0,1` per yt-dlp docs (`max-comments,max-parents,max-replies,max-replies-per-thread,max-depth`); `max-depth=1` fetches top-level comments only

### Security
- **XSS via HTML tag blocklist bypass** — dangerous tags like `<svg onload=...>`, `<img onerror=...>` bypassed the blocklist. Replaced with strict allowlist of safe tags and strips all `on*` event handler attributes
- **Protocol-relative URL bypass** — URLs like `//evil.com` started with `/` and passed link validation. Now explicitly blocked
- **TOCTOU race in queue operations** — concurrent processes could corrupt the queue. Fixed with atomic `_locked_queue()` context manager
- **`tarfile.extractall` without filter** — added `filter='data'` on Python 3.12+ for defense-in-depth against path traversal

### Other
- **`.gitignore`** — excludes `.DS_Store`, `__pycache__/`, `*.pyc`

### Benchmark (M1 Max, 2026-02-08)

| Text | PyTorch Gen | MLX Gen | Speedup |
|------|------------|---------|---------|
| Short (192 chars) | 2.35s | 0.76s | **3.1x** |
| Long (619 chars) | 8.20s | 2.05s | **4.0x** |

---

## [1.1.4] - 2026-02-08

### Fixed
- Wrapped long line in SKILL.md that caused horizontal scroll on ClawHub page

---

## [1.1.3] - 2026-02-08

### Fixed
- **YouTube URL normalization** — `/live/`, `/shorts/` and other non-standard URL formats now handled correctly
- **False-positive error detection** — error patterns only checked when command actually fails (was triggering on successful runs)
- **Queue file locking** — proper lock-before-open semantics with `os.open()` + `os.fdopen()` (prevents corruption)
- **Timezone comparison crash** — stale job detection now ensures timezone-aware datetime comparison
- **HTML link double-encoding** — URLs with `&` were getting double-escaped; links now extracted before HTML escaping
- **Built-in TTS for long text** — uses temp file to avoid CLI argument length limits with macOS `say`
- **HTML headings/tables** — now render bold, italic, and links correctly (was only escaping, not formatting)
- **Cleanup includes TTS temp files** — removes `_tts.py` alongside source JSON and output markdown

### Security
- **Zip-slip prevention** — validates zip/tar entries during pandoc/yt-dlp install to prevent path traversal attacks
- **Single-quote escaping** — added `'` → `&#39;` in HTML escape function
- **Sub-agent install guard** — SKILL.md now instructs sub-agents to never install software

### Changed
- **Kokoro voice blend from config** — reads voice blend dict + speed from `~/.tubescribe/config.json` instead of hardcoded values
- **Config uses nested keys throughout** — `config["audio"]["format"]` instead of flat `config["audio_format"]`
- **Renamed `config.set()` → `config.set_value()`** — avoids shadowing Python builtin
- **DRY: setup.py delegates to tubescribe.py** — imports `find_ytdlp`, `find_kokoro`, `KOKORO_DEPS` instead of duplicating
- **Uses `find_ytdlp()` helper** — multi-path search instead of raw `subprocess.run(["yt-dlp", ...])`
- **SKILL.md uses relative paths** — portable across installations
- **TTS respects config** — reads `tts_engine` setting (kokoro/builtin/disabled)
- **Better error for missing `say`** — FileNotFoundError with helpful message about Kokoro alternative
- **Removed `live_stream` error pattern** — was unreliable, caused false positives
- **Default Kokoro path** — updated to `~/.openclaw/tools/kokoro`

---

## [1.1.2] - 2026-02-06

### Fixed
- **Stale queue detection crash** — timezone-aware/naive datetime comparison caused TypeError, leaving queue permanently stuck
- **Config file corruption** — `_save_kokoro_cache()` was writing internal wrapper keys (`_raw`, flat keys) back to config.json
- **HTML table rendering** — all rows rendered as `<th>` headers; body rows now correctly use `<td>`
- **XSS vulnerability in HTML output** — paragraph text was not escaped before inline formatting; `<script>` tags could pass through
- **`--no-audio` CLI flag** — was defined but never wired up
- **Empty title filenames** — videos with empty titles produced `.docx` filename; now falls back to video ID
- **URL validation** — accepted any domain with `?v=` parameter; now validates YouTube domains only
- **Comment HTML double-wrapping** — `<p align="right">` tags got wrapped in another `<p>`; raw HTML now passes through
- **`safe_unescape()` fragility** — replaced encode/decode chain with `json.loads()` approach

### Removed
- **python-docx fallback** — was a half-broken DOCX generator; fallback chain is now pandoc → HTML (cleaner, better output)
- **`set_current()` dead code** — was never called; queue uses `pop_from_queue()` instead
- **`afconvert` dependency** — `say` now outputs WAV directly via `--data-format`; no intermediate AIFF conversion

### Changed
- **Built-in TTS** — simplified from `say` → AIFF → `afconvert` → WAV to just `say` → WAV directly
- **Timestamps** — replaced `subprocess.run(["date"])` with native `datetime.now().astimezone().isoformat()`
- **Queue file locking** — added `fcntl.flock()` advisory locking to prevent corruption from concurrent access
- **Config cleanup** — removed stale flat keys from config.json

### Added
- **HTML blockquotes** — `> text` lines now render as `<blockquote>` blocks
- **`sanitize_filename()` fallback parameter** — accepts video ID as fallback for empty titles

---

## [1.1.1] - 2026-02-06

- **Non-blocking async workflow** — Sub-agent handles entire pipeline (extract → process → DOCX → audio → cleanup)
- **Queue processing** — More robust handling of multiple videos
- **Comments section** — Viewer sentiment analysis and best comments

---

## [1.1.0] - 2026-02-06

### Changed
- **Comment sections renamed** — "Comment Summary" → "Viewer Sentiment", "Best Comments" reformatted with italic attribution
- **Bold headings** — Title and section headers use explicit bold (`# **Title**`) for consistent DOCX rendering

### Fixed
- Comment text and attribution were merging into single line in DOCX output

---

## [1.0.9] - 2026-02-06

### Added
- **YouTube comments** — Fetches top 50 comments, adds Comment Summary + Best Comments sections
- **yt-dlp support** — Auto-install to `~/.openclaw/tools/yt-dlp/` if not present
- **Progress feedback** — Clear step-by-step output with stages
- **Video metadata** — Channel name, upload date, and duration in output
- **Better error messages** — Human-readable errors for common issues:
  - Private videos, removed videos, no captions
  - Age-restricted, region-blocked, live streams
  - Invalid URLs, network errors, timeouts
- **CLI batch processing** — Process multiple URLs: `tubescribe url1 url2 url3`
- **Session queue** — Queue management for processing multiple videos:
  - `--queue-add URL` — Add to queue
  - `--queue-status` — Show current + queued items
  - `--queue-next` — Process next from queue
  - `--queue-clear` — Clear queue
- **Processing time estimates** — Shows estimated time based on word count

### Fixed
- **Code injection vulnerability** — Text now properly escaped with `json.dumps()`
- **Config schema compatibility** — Setup and runtime use same config format
- **Missing import** — Added `import json` for `--quiet --check-only` mode
- **Output directory default** — Now uses config value instead of current directory
- **Comment sorting** — Uses `comment_sort=top` to get highest-liked (not newest)
- **Unicode escape crash** — `safe_unescape()` handles edge cases in video descriptions
- **YouTube Shorts/Live URLs** — Now extracts video ID from `/shorts/` and `/live/` URLs

### New Config Options
- `comments.max_count` — Number of comments to fetch (default: 50)
- `comments.timeout` — Timeout for comment fetching (default: 90s)
- `queue.stale_minutes` — Consider processing stale after N minutes (default: 30)

### Output Format
- **Clickable URL** — Video URL in header is now a markdown link
- **Bold table headers** — Participants table uses `| **Name** | **Role** |`
- **Section separators** — `---` between all major sections
- **Best Comments** — Two-line format: comment text, then `   ▲ likes @Author`
- **Viewer Sentiment** — Flat section (not nested under "Comment Highlights")

### Changed
- **Metadata extraction** — Now uses `yt-dlp` if available (better data), falls back to HTML scraping
- **Transcript timeout** — Increased from 60s to 120s for long videos
- **SKILL.md output format** — Now includes video info block (channel, date, duration, URL)

## [1.0.8] - 2026-02-05

### Fixed
- Recovered from ClawHub publish disaster via `clawhub undelete`

## [1.0.7] - 2026-02-04

### Added
- Kokoro TTS integration with dynamic path detection
- Path caching for instant Kokoro startup (2.5s → 0.1ms)
- Smart dependency detection (system pip → known locations → fallback venv)
- Transcript segment merging in SKILL.md instructions

### Fixed
- MP3 output was using macOS `say` instead of Kokoro

## [1.0.0] - 2026-02-04

### Added
- Initial release
- YouTube transcript extraction via `summarize` CLI
- Sub-agent processing for speaker detection and summarization
- Document output (HTML, DOCX, Markdown)
- Audio summary generation (Kokoro TTS or macOS built-in)
- Setup wizard with dependency checking


path: README.md
content: # TubeScribe 🎬

**Turn any YouTube video into a polished document + audio summary.**

Drop a YouTube link → get a beautiful transcript with speaker labels, key quotes, clickable timestamps, and an audio summary you can listen to on the go.

## Features

- 🎯 **Smart Speaker Detection** — Automatically identifies participants
- 🔊 **Audio Summaries** — Listen to key points (MP3/WAV)
- 📝 **Clickable Timestamps** — Every quote links directly to that moment in the video
- 💬 **YouTube Comments** — Viewer sentiment analysis and best comments
- 📄 **Transcript with summary and key quotes** — Export as DOCX, HTML, or Markdown
- 📋 **Queue Support** — Send multiple links, they get processed in order
- 🚀 **Non-Blocking Workflow** — Conversation continues while video processes in background

## Free & No Paid APIs

- **No subscriptions** — no API keys, no paid services
- **No usage limits** — process as many videos as you want
- **Local processing** — transcription, speaker detection, TTS all run on your machine
- **Network access** — YouTube fetching (video metadata, captions, comments) requires internet

## Quick Start

Just send a YouTube URL to your agent. TubeScribe handles everything automatically.

### Non-Blocking Processing

TubeScribe runs in the background:
1. You send a YouTube link
2. Agent replies: "🎬 TubeScribe is processing **[title]**..."
3. **You can keep chatting** — conversation isn't blocked
4. When done, you get notified with the results

No waiting, no freezing — just seamless async processing.

### First-Time Setup

```bash
python skills/tubescribe/scripts/setup.py
```

**What setup.py does:** Checks if recommended tools are present (`summarize`, `pandoc`, `ffmpeg`, TTS engine) and offers to install any missing ones. It only downloads from official sources (Homebrew, PyPI, GitHub releases). You can skip any install — the skill works without them using macOS built-in TTS and HTML fallback.

**What setup.py does NOT do:** It does not upload data, contact unknown hosts, or modify system files outside `~/.openclaw/tools/` and `~/.tubescribe/`.

## Output Example

```
~/Documents/TubeScribe/
├── Interview With Expert.docx    # Formatted document
└── Interview With Expert.mp3     # Audio summary
```

### Document Structure

1. **Title** + video info (channel, date, duration)
2. **Participants** — who's speaking
3. **Summary** — key points in 3-5 paragraphs
4. **Key Quotes** — 5 best moments with clickable timestamps
5. **Viewer Sentiment** — what commenters are saying
6. **Best Comments** — top 5 comments by likes
7. **Full Transcript** — merged paragraphs with speaker labels

## Batch & Queue

### Multiple videos at once
```bash
tubescribe url1 url2 url3
```

### Queue management
```bash
tubescribe --queue-add "URL"      # Add while processing
tubescribe --queue-status         # Check queue
tubescribe --queue-next           # Process next
tubescribe --queue-clear          # Clear queue
```

## Configuration

Config file: `~/.tubescribe/config.json`

| Setting | Default | Options |
|---------|---------|---------|
| `document.format` | `docx` | `docx`, `html`, `md` |
| `audio.format` | `mp3` | `mp3`, `wav` |
| `audio.tts_engine` | `mlx` | `mlx`, `kokoro`, `builtin` |
| `mlx_audio.voice_blend` | `{af_heart: 0.6, af_sky: 0.4}` | any voice mix |
| `output.folder` | `~/Documents/TubeScribe` | any path |

## Requirements

- **Required:** `summarize` CLI (`brew install steipete/tap/summarize`)
- **Optional:**
  - `pandoc` — DOCX output (`brew install pandoc`)
  - `ffmpeg` — MP3 audio (`brew install ffmpeg`)
  - `yt-dlp` — YouTube comments (`brew install yt-dlp`)
  - mlx-audio — Fastest TTS on Apple Silicon (auto-installed via setup)
  - Kokoro TTS — PyTorch fallback for non-Apple-Silicon (auto-installed via setup)

### yt-dlp Installation

TubeScribe checks these locations for yt-dlp (in order):

1. System PATH (`which yt-dlp`)
2. Homebrew: `/opt/homebrew/bin/yt-dlp` or `/usr/local/bin/yt-dlp`
3. pip/pipx: `~/.local/bin/yt-dlp`
4. TubeScribe tools: `~/.openclaw/tools/yt-dlp/yt-dlp`

If not found, setup will offer to download a standalone binary to the tools directory.

## Error Handling

Clear messages for common issues:

| Issue | Message |
|-------|---------|
| Private video | ❌ Video is private — can't access |
| No captions | ❌ No captions available for this video |
| Invalid URL | ❌ Not a valid YouTube URL |
| Age-restricted | ❌ Age-restricted video — can't access without login |

## Privacy & Network

**What uses the network:**
- `summarize` CLI / `yt-dlp` — fetches video captions, metadata, and comments from YouTube
- `setup.py` — one-time download of tools (pandoc, ffmpeg, yt-dlp, TTS models) from official sources

**What runs locally (no network):**
- Speaker detection and transcript analysis (Claude sub-agent, same as your main agent)
- TTS audio generation (MLX-Audio Kokoro, Apple `say`, or Kokoro PyTorch — all on-device)
- Document generation (pandoc)
- Audio conversion (ffmpeg)

**No data is uploaded anywhere.** Video content is fetched *from* YouTube, processed on your machine, and saved locally. Nothing is sent back.

**Config paths (`~/.tubescribe/config.json`)** point to local TTS installations on your machine (e.g., `~/.openclaw/tools/mlx-audio`). These are not remote services.

## Security

### Code Injection (Fixed in v1.1.0)
Earlier development versions had a vulnerability where video text could be injected into dynamically executed Python code. This was fixed by properly escaping all text with `json.dumps()`.

### HTML Output (Fixed in v1.1.2+)
- XSS prevention: all text escaped before inline formatting
- Single-quote escaping added in v1.1.3
- Link double-encoding fixed in v1.1.3

### Archive Extraction (Fixed in v1.1.3)
Zip-slip path traversal prevention when installing pandoc/yt-dlp via setup script.

### Shell Commands
The skill uses subprocess to call external CLI tools (`summarize`, `yt-dlp`, `pandoc`, `ffmpeg`). YouTube URLs are validated and normalized before processing, and filenames are sanitized. However, as with any tool that processes external content, review the code if you have concerns.

### External Dependencies
The setup script downloads tools from official sources:
- **pandoc** — from Homebrew or official releases
- **yt-dlp** — from GitHub releases (yt-dlp/yt-dlp)
- **mlx-audio** — pip install from PyPI (Apple Silicon only, uses MLX framework)
- **Kokoro TTS** — pip install from PyPI (PyTorch, cross-platform fallback)

All sources are well-known and widely used. Review `scripts/setup.py` if you have concerns about supply chain security.

## License

MIT

---

Made by Jackie 🦊 & Matus 🇸🇰


path: scripts/config.py
content: #!/usr/bin/env python3
"""
TubeScribe Configuration
========================

Central configuration management for TubeScribe.
All settings in one place, with defaults and validation.
"""

import os
import json
import sys
from pathlib import Path

try:
    import fcntl
    _HAS_FCNTL = True
except ImportError:
    _HAS_FCNTL = False

# Config location
CONFIG_DIR = Path.home() / ".tubescribe"
CONFIG_FILE = CONFIG_DIR / "config.json"

# Default configuration with all available options
DEFAULT_CONFIG = {
    # Output settings
    "output": {
        "folder": str(Path.home() / "Documents" / "TubeScribe"),
        "open_folder_after": True,      # Open output folder when done
        "open_document_after": False,   # Auto-open generated document (docx/html/md)
        "open_audio_after": False,      # Auto-open generated audio summary
    },
    
    # Document settings
    "document": {
        "format": "docx",               # docx, html, md
        "engine": "pandoc",             # pandoc (for docx), falls back to html
    },
    
    # Audio settings
    "audio": {
        "enabled": True,                # Generate audio summary
        "format": "mp3",                # mp3, wav
        "tts_engine": "mlx",            # mlx (mlx-audio, fastest), kokoro (PyTorch), builtin (macOS say)
    },
    
    # MLX-Audio TTS settings (preferred on Apple Silicon)
    "mlx_audio": {
        "path": str(Path.home() / ".openclaw" / "tools" / "mlx-audio"),
        "model": "mlx-community/Kokoro-82M-bf16",
        "voice": "af_heart",             # Single voice name or path to .safetensors
        "voice_blend": {                  # Custom voice mix (overrides voice if set)
            "af_heart": 0.6,
            "af_sky": 0.4,
        },
        "lang_code": "a",
        "speed": 1.05,                    # Playback speed (1.0 = normal, 1.05 = 5% faster)
    },
    
    # Kokoro TTS settings (PyTorch fallback)
    "kokoro": {
        "path": str(Path.home() / ".openclaw" / "tools" / "kokoro"),
        "voice_blend": {                # Custom voice mix
            "af_heart": 0.6,
            "af_sky": 0.4,
        },
        "speed": 1.05,                  # Playback speed (1.0 = normal, 1.05 = 5% faster)
    },
    
    # Processing settings
    "processing": {
        "subagent_timeout": 600,        # Seconds for sub-agent processing
        "cleanup_temp_files": True,     # Remove /tmp files after completion
    },
    
    # Comments settings
    "comments": {
        "max_count": 50,                # Number of comments to fetch
        "timeout": 90,                  # Timeout for comment fetching (seconds)
    },
    
    # Queue settings
    "queue": {
        "stale_minutes": 30,            # Consider processing stale after this many minutes
    },
}


_CONFIG_VALIDATORS = {
    "document.format": lambda v: v in ("docx", "html", "md"),
    "audio.enabled": lambda v: isinstance(v, bool),
    "audio.format": lambda v: v in ("mp3", "wav"),
    "audio.tts_engine": lambda v: v in ("mlx", "kokoro", "builtin"),
    "mlx_audio.speed": lambda v: isinstance(v, (int, float)) and 0.1 <= v <= 5.0,
    "kokoro.speed": lambda v: isinstance(v, (int, float)) and 0.1 <= v <= 5.0,
    "processing.subagent_timeout": lambda v: isinstance(v, (int, float)) and v > 0,
    "comments.max_count": lambda v: isinstance(v, int) and v > 0,
    "comments.timeout": lambda v: isinstance(v, (int, float)) and v > 0,
    "queue.stale_minutes": lambda v: isinstance(v, (int, float)) and v > 0,
}


def _validate_config(config: dict, defaults: dict) -> dict:
    """Validate config values, reverting invalid ones to defaults."""
    for dotpath, validator in _CONFIG_VALIDATORS.items():
        keys = dotpath.split('.')
        # Navigate to value
        value = config
        default_value = defaults
        try:
            for k in keys:
                value = value[k]
                default_value = default_value[k]
        except (KeyError, TypeError):
            continue
        if not validator(value):
            # Revert to default
            target = config
            for k in keys[:-1]:
                target = target[k]
            target[keys[-1]] = deep_copy(default_value)
    return config


def get_config_dir() -> Path:
    """Get the config directory, creating if needed."""
    CONFIG_DIR.mkdir(parents=True, exist_ok=True)
    return CONFIG_DIR


def load_config() -> dict:
    """Load config from file, merging with defaults. Invalid values are reverted."""
    config = deep_copy(DEFAULT_CONFIG)

    if CONFIG_FILE.exists():
        try:
            with open(CONFIG_FILE, 'r') as f:
                user_config = json.load(f)
            config = deep_merge(config, user_config)
        except (json.JSONDecodeError, IOError):
            pass  # Use defaults if config is corrupted

    config = _validate_config(config, DEFAULT_CONFIG)
    return config


def save_config(config: dict) -> None:
    """Save config to file with file locking to prevent concurrent write corruption."""
    get_config_dir()
    fd = os.open(str(CONFIG_FILE), os.O_RDWR | os.O_CREAT | os.O_TRUNC, 0o644)
    try:
        if _HAS_FCNTL:
            fcntl.flock(fd, fcntl.LOCK_EX)
        os.write(fd, json.dumps(config, indent=2).encode('utf-8'))
    finally:
        os.close(fd)


def get_value(key: str, default=None):
    """
    Get a config value using dot notation.

    Example:
        get_value("output.folder")
        get_value("audio.format")
    """
    config = load_config()
    keys = key.split('.')
    value = config
    
    try:
        for k in keys:
            value = value[k]
        return value
    except (KeyError, TypeError):
        return default


def set_value(key: str, value) -> None:
    """
    Set a config value using dot notation.

    Example:
        set_value("output.folder", "~/Desktop/Videos")
        set_value("audio.enabled", False)
    """
    config = load_config()
    keys = key.split('.')
    
    # Navigate to the parent
    target = config
    for k in keys[:-1]:
        if k not in target:
            target[k] = {}
        target = target[k]
    
    # Set the value
    target[keys[-1]] = value
    save_config(config)


def deep_copy(obj):
    """Deep copy a dict/list structure."""
    if isinstance(obj, dict):
        return {k: deep_copy(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [deep_copy(v) for v in obj]
    return obj


def deep_merge(base: dict, override: dict) -> dict:
    """Deep merge override into base."""
    result = deep_copy(base)
    
    for key, value in override.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = deep_merge(result[key], value)
        else:
            result[key] = deep_copy(value)
    
    return result


def reset_to_defaults() -> None:
    """Reset config to defaults."""
    save_config(DEFAULT_CONFIG)


def print_config() -> None:
    """Print current config in a readable format."""
    config = load_config()
    print(json.dumps(config, indent=2))


# CLI interface
if __name__ == "__main__":
    if len(sys.argv) == 1:
        print("TubeScribe Configuration")
        print("=" * 40)
        print(f"Config file: {CONFIG_FILE}")
        print()
        print_config()
    
    elif sys.argv[1] == "get" and len(sys.argv) == 3:
        value = get_value(sys.argv[2])
        print(value if value is not None else "(not set)")
    
    elif sys.argv[1] == "set" and len(sys.argv) == 4:
        key, value = sys.argv[2], sys.argv[3]
        # Try to parse as JSON for complex values
        try:
            value = json.loads(value)
        except json.JSONDecodeError:
            pass  # Keep as string
        set_value(key, value)
        print(f"Set {key} = {value}")
    
    elif sys.argv[1] == "reset":
        reset_to_defaults()
        print("Config reset to defaults.")
    
    elif sys.argv[1] == "path":
        print(CONFIG_FILE)
    
    else:
        print("Usage:")
        print("  python config.py              # Show current config")
        print("  python config.py get KEY      # Get a value (dot notation)")
        print("  python config.py set KEY VAL  # Set a value")
        print("  python config.py reset        # Reset to defaults")
        print("  python config.py path         # Show config file path")
        print()
        print("Examples:")
        print("  python config.py get output.folder")
        print("  python config.py set audio.format wav")
        print("  python config.py set output.folder ~/Desktop/Videos")


path: scripts/html_writer.py
content: #!/usr/bin/env python3
"""
HTML Writer for TubeScribe
==========================

Generates clean HTML documents from markdown transcript data.
Zero dependencies, opens in any browser, looks great.

Usage:
    from html_writer import create_html_from_markdown
    create_html_from_markdown(md_path, output_path)
"""

import re
from datetime import datetime


def markdown_to_html(md_text: str) -> str:
    """Convert markdown text to HTML."""
    lines = md_text.split('\n')
    html_lines = []
    in_table = False
    table_past_separator = False  # Track if we've passed the header separator
    in_blockquote = False
    blockquote_lines = []
    in_code_block = False
    code_block_lines = []
    list_stack = []  # Stack of ('ul'|'ol') for nested list tracking

    def flush_blockquote():
        """Flush accumulated blockquote lines."""
        nonlocal in_blockquote, blockquote_lines
        if blockquote_lines:
            content = '</p><p>'.join(blockquote_lines)
            html_lines.append(f'<blockquote><p>{content}</p></blockquote>')
            blockquote_lines = []
        in_blockquote = False

    def flush_list():
        """Close all open list tags."""
        nonlocal list_stack
        while list_stack:
            html_lines.append(f'</{list_stack.pop()}>')

    def close_blocks():
        """Close any open table or list."""
        nonlocal in_table, table_past_separator
        if in_table:
            html_lines.append('</table>')
            in_table = False
            table_past_separator = False
        flush_list()

    for line in lines:
        # Fenced code block handling
        if line.strip().startswith('```'):
            if in_code_block:
                html_lines.append(f'<pre><code>{escape_html(chr(10).join(code_block_lines))}</code></pre>')
                code_block_lines = []
                in_code_block = False
            else:
                close_blocks()
                if in_blockquote:
                    flush_blockquote()
                in_code_block = True
            continue
        if in_code_block:
            code_block_lines.append(line)
            continue

        # Blockquote handling: > text
        if line.startswith('> '):
            close_blocks()
            quote_text = process_inline_formatting(line[2:])
            blockquote_lines.append(quote_text)
            in_blockquote = True
            continue
        elif in_blockquote:
            flush_blockquote()

        # List handling: - item or * item (unordered), 1. item (ordered)
        ul_match = re.match(r'^(\s*)[-*]\s+(.+)', line)
        ol_match = re.match(r'^(\s*)\d+\.\s+(.+)', line)
        if ul_match or ol_match:
            if in_table:
                html_lines.append('</table>')
                in_table = False
                table_past_separator = False
            if ul_match:
                list_type = 'ul'
                item_text = ul_match.group(2)
            else:
                list_type = 'ol'
                item_text = ol_match.group(2)
            if not list_stack:
                html_lines.append(f'<{list_type}>')
                list_stack.append(list_type)
            html_lines.append(f'<li>{process_inline_formatting(item_text)}</li>')
            continue

        # If we were in a list and this line isn't a list item, close the list
        if list_stack:
            flush_list()

        # Headers
        if line.startswith('# '):
            close_blocks()
            html_lines.append(f'<h1>{process_inline_formatting(line[2:])}</h1>')
        elif line.startswith('## '):
            close_blocks()
            html_lines.append(f'<h2>{process_inline_formatting(line[3:])}</h2>')
        elif line.startswith('### '):
            close_blocks()
            html_lines.append(f'<h3>{process_inline_formatting(line[4:])}</h3>')
        elif line.startswith('---'):
            close_blocks()
            html_lines.append('<hr>')
        elif line.startswith('|'):
            # Table handling
            if not in_table:
                html_lines.append('<table>')
                in_table = True
                table_past_separator = False
            # Check for separator row (contains only |, -, :, and spaces)
            if re.match(r'^[\s|:\-]+$', line):
                table_past_separator = True
                continue  # Skip separator row
            cells = [c.strip() for c in line.split('|')[1:-1]]
            # First row before separator = <th>, all rows after = <td>
            tag = 'td' if table_past_separator else 'th'
            row = ''.join(f'<{tag}>{process_inline_formatting(c)}</{tag}>' for c in cells)
            html_lines.append(f'<tr>{row}</tr>')
        elif line.strip() == '':
            close_blocks()
            html_lines.append('')
        elif line.lstrip().startswith('<') and not line.lstrip().startswith('<http'):
            # Raw HTML line - pass through safe tags only, escape everything else
            close_blocks()
            stripped = line.lstrip()
            # Allowlist of safe HTML tags (reject unknown tags entirely)
            safe_tags = {'div', 'span', 'br', 'p', 'ul', 'ol', 'li', 'pre', 'code',
                         'sub', 'sup', 'details', 'summary', 'dl', 'dt', 'dd',
                         'abbr', 'cite', 'small', 'mark', 'ins', 'del',
                         'figcaption', 'figure'}
            tag_match = re.match(r'<\s*/?\s*(\w+)', stripped)
            if tag_match and tag_match.group(1).lower() in safe_tags:
                # Strip event handler attributes (on*="...") and style attributes
                safe_line = re.sub(r'\s+on\w+\s*=\s*["\'][^"\']*["\']', '', line, flags=re.IGNORECASE)
                safe_line = re.sub(r'\s+on\w+\s*=\s*\S+', '', safe_line, flags=re.IGNORECASE)
                safe_line = re.sub(r'\s+style\s*=\s*["\'][^"\']*["\']', '', safe_line, flags=re.IGNORECASE)
                safe_line = re.sub(r'\s+style\s*=\s*\S+', '', safe_line, flags=re.IGNORECASE)
                html_lines.append(safe_line)
            else:
                html_lines.append(f'<p>{escape_html(line)}</p>')
        else:
            close_blocks()
            # Process inline formatting
            processed = process_inline_formatting(line)
            html_lines.append(f'<p>{processed}</p>')

    # Flush any remaining open blocks
    if in_code_block:
        html_lines.append(f'<pre><code>{escape_html(chr(10).join(code_block_lines))}</code></pre>')
    if in_blockquote:
        flush_blockquote()
    close_blocks()

    return '\n'.join(html_lines)


def escape_html(text: str) -> str:
    """Escape HTML special characters."""
    return (text
            .replace('&', '&amp;')
            .replace('<', '&lt;')
            .replace('>', '&gt;')
            .replace('"', '&quot;')
            .replace("'", '&#39;'))


def process_inline_formatting(text: str) -> str:
    """Process inline markdown formatting with XSS protection."""
    # Extract inline code BEFORE escaping (content should be escaped but not formatted)
    code_placeholders = {}
    code_counter = 0

    def extract_code(match):
        nonlocal code_counter
        code_text = match.group(1)
        placeholder = f"\x00CODE{code_counter}\x00"
        code_counter += 1
        code_placeholders[placeholder] = code_text
        return placeholder

    text = re.sub(r'`([^`]+)`', extract_code, text)

    # Extract links BEFORE escaping so URLs don't get double-encoded
    link_placeholders = {}
    link_counter = 0

    def extract_link(match):
        nonlocal link_counter
        link_text, url = match.groups()
        placeholder = f"\x00LINK{link_counter}\x00"
        link_counter += 1
        link_placeholders[placeholder] = (link_text, url)
        return placeholder

    text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', extract_link, text)

    # Escape HTML to prevent XSS
    text = escape_html(text)

    # Bold: **text** or __text__
    text = re.sub(r'\*\*(.+?)\*\*', r'<strong>\1</strong>', text)
    text = re.sub(r'__(.+?)__', r'<strong>\1</strong>', text)

    # Italic: *text* or _text_
    text = re.sub(r'\*(.+?)\*', r'<em>\1</em>', text)
    text = re.sub(r'(?<!\w)_(.+?)_(?!\w)', r'<em>\1</em>', text)

    # Restore links with safe URL validation
    for placeholder, (link_text, url) in link_placeholders.items():
        safe_text = escape_html(link_text)
        if url.startswith(('http://', 'https://')) or (url.startswith('/') and not url.startswith('//')):  # block protocol-relative URLs
            safe_url = url.replace('&', '&amp;').replace('"', '&quot;')
            link_html = f'<a href="{safe_url}" target="_blank">{safe_text}</a>'
        else:
            link_html = f'{safe_text} ({escape_html(url)})'
        text = text.replace(placeholder, link_html)

    # Restore inline code
    for placeholder, code_text in code_placeholders.items():
        text = text.replace(placeholder, f'<code>{escape_html(code_text)}</code>')

    return text


def create_html_from_markdown(md_path: str, output_path: str) -> str:
    """Convert a markdown file to a styled HTML document."""
    with open(md_path, 'r', encoding='utf-8') as f:
        md_content = f.read()
    
    # Extract title from first H1
    title_match = re.search(r'^# (.+)$', md_content, re.MULTILINE)
    title = title_match.group(1) if title_match else "TubeScribe Transcript"
    
    # Convert content
    html_body = markdown_to_html(md_content)
    
    # Wrap in full HTML document with nice styling
    html_doc = f'''<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{escape_html(title)}</title>
    <style>
        :root {{
            --bg: #ffffff;
            --text: #1a1a1a;
            --accent: #0066cc;
            --border: #e0e0e0;
            --quote-bg: #f5f5f5;
        }}
        @media (prefers-color-scheme: dark) {{
            :root {{
                --bg: #1a1a1a;
                --text: #e0e0e0;
                --accent: #66b3ff;
                --border: #333;
                --quote-bg: #252525;
            }}
        }}
        * {{ box-sizing: border-box; }}
        body {{
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            line-height: 1.6;
            max-width: 800px;
            margin: 0 auto;
            padding: 2rem;
            background: var(--bg);
            color: var(--text);
        }}
        h1 {{ 
            font-size: 2rem; 
            border-bottom: 2px solid var(--accent);
            padding-bottom: 0.5rem;
        }}
        h2 {{ 
            font-size: 1.5rem; 
            margin-top: 2rem;
            color: var(--accent);
        }}
        h3 {{ font-size: 1.25rem; }}
        a {{ color: var(--accent); }}
        a:hover {{ text-decoration: none; }}
        hr {{ 
            border: none; 
            border-top: 1px solid var(--border); 
            margin: 2rem 0;
        }}
        table {{
            width: 100%;
            border-collapse: collapse;
            margin: 1rem 0;
        }}
        th, td {{
            padding: 0.5rem 1rem;
            text-align: left;
            border: 1px solid var(--border);
        }}
        th {{ background: var(--quote-bg); }}
        p {{ margin: 0.75rem 0; }}
        strong {{ font-weight: 600; }}
        ul, ol {{ margin: 0.75rem 0; padding-left: 2rem; }}
        li {{ margin: 0.25rem 0; }}
        pre {{
            background: var(--quote-bg);
            padding: 1rem;
            border-radius: 6px;
            overflow-x: auto;
            margin: 1rem 0;
        }}
        pre code {{ background: none; padding: 0; }}
        code {{
            font-family: 'SF Mono', Menlo, Consolas, monospace;
            background: var(--quote-bg);
            padding: 0.15rem 0.4rem;
            border-radius: 3px;
            font-size: 0.9em;
        }}
        .timestamp {{
            font-family: monospace;
            background: var(--quote-bg);
            padding: 0.1rem 0.3rem;
            border-radius: 3px;
        }}
        footer {{
            margin-top: 3rem;
            padding-top: 1rem;
            border-top: 1px solid var(--border);
            font-size: 0.85rem;
            color: #888;
        }}
    </style>
</head>
<body>
{html_body}
<footer>
    Generated by TubeScribe 🎬 • {datetime.now().strftime('%Y-%m-%d %H:%M')}
</footer>
</body>
</html>'''
    
    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(html_doc)
    
    return output_path


if __name__ == "__main__":
    import sys
    if len(sys.argv) > 1:
        md_path = sys.argv[1]
        out_path = sys.argv[2] if len(sys.argv) > 2 else md_path.replace('.md', '.html')
        result = create_html_from_markdown(md_path, out_path)
        print(f"Created: {result}")


path: scripts/setup.py
content: #!/usr/bin/env python3
"""
TubeScribe Setup Wizard
=======================

Checks dependencies, sets optimal defaults, and offers to install
missing components for the best experience.

Usage:
    python setup.py [--check-only] [--quiet]
"""

import subprocess
import shutil
import sys
import os
import json

# Add script directory to path for imports
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, SCRIPT_DIR)

from config import (
    CONFIG_DIR, CONFIG_FILE, DEFAULT_CONFIG,
    load_config, save_config, get_config_dir
)
from tubescribe import (
    find_ytdlp as _find_ytdlp,
    find_kokoro as _find_kokoro,
    find_mlx_audio as _find_mlx_audio,
    KOKORO_DEPS,
)


def print_header():
    print("""
🎬 TubeScribe Setup
═══════════════════
""")


def check_command(cmd: str) -> bool:
    """Check if a command exists in PATH."""
    return shutil.which(cmd) is not None


def check_python_package(package: str, import_name: str = None) -> bool:
    """Check if a Python package is installed in system Python."""
    import_name = import_name or package
    try:
        result = subprocess.run(
            [sys.executable, "-c", f"import {import_name}"],
            capture_output=True, timeout=10
        )
        return result.returncode == 0
    except (subprocess.SubprocessError, OSError, TimeoutError):
        return False


def get_kokoro_path() -> str:
    """Get the path to Kokoro repo."""
    return os.path.expanduser("~/.openclaw/tools/kokoro")


def get_ml_env_path() -> str:
    """Get the path to shared ML environment."""
    return os.path.expanduser("~/.openclaw/tools/ml-env")


def check_ml_deps() -> dict:
    """Check which ML dependencies are available in system Python."""
    results = {}
    for package, import_name in KOKORO_DEPS.items():
        results[package] = check_python_package(package, import_name)
    return results


def get_python_for_kokoro() -> tuple[str, str | None]:
    """
    Find the best Python to use for Kokoro.
    Returns (python_path, kokoro_dir or None).
    """
    kokoro_dir = get_kokoro_path()
    ml_env = get_ml_env_path()
    ml_env_python = os.path.join(ml_env, "bin", "python")
    
    # Check if kokoro repo exists
    if not os.path.exists(kokoro_dir):
        return None, None
    
    # 1. Check system Python — all deps present?
    deps = check_ml_deps()
    if all(deps.values()):
        return sys.executable, kokoro_dir
    
    # 2. Check shared ML env
    if os.path.exists(ml_env_python):
        try:
            result = subprocess.run(
                [ml_env_python, "-c", "import torch, soundfile, numpy, huggingface_hub"],
                capture_output=True, timeout=10
            )
            if result.returncode == 0:
                return ml_env_python, kokoro_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    # 3. Check old-style venv inside kokoro dir (legacy)
    venv_python = os.path.join(kokoro_dir, ".venv", "bin", "python")
    if os.path.exists(venv_python):
        try:
            result = subprocess.run(
                [venv_python, "-c", "from kokoro import KPipeline"],
                capture_output=True, timeout=10, cwd=kokoro_dir
            )
            if result.returncode == 0:
                return venv_python, kokoro_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    return None, kokoro_dir  # Repo exists but no working Python


def find_kokoro() -> tuple[bool, str | None, str | None]:
    """
    Find Kokoro TTS installation.
    Returns (found: bool, python_path: str | None, kokoro_dir: str | None).
    """
    # Delegate to tubescribe's cached finder, skip cache for setup checks
    python_path, kokoro_dir = _find_kokoro(use_cache=False)
    if python_path and kokoro_dir:
        return True, python_path, kokoro_dir
    # Fall back to get_python_for_kokoro for the case where repo exists but
    # tubescribe's finder didn't find a working Python (setup may install deps)
    python_path, kokoro_dir = get_python_for_kokoro()
    if python_path and kokoro_dir:
        try:
            result = subprocess.run(
                [python_path, "-c", "from kokoro import KPipeline; print('ok')"],
                capture_output=True, timeout=10, cwd=kokoro_dir
            )
            if result.returncode == 0:
                return True, python_path, kokoro_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    return False, None, None


def check_kokoro() -> bool:
    """Check if Kokoro TTS is installed and working."""
    found, _, _ = find_kokoro()
    return found


def check_mlx_audio() -> bool:
    """Check if mlx-audio is installed and working."""
    python_path, _ = _find_mlx_audio()
    return python_path is not None


def run_checks() -> dict:
    """Run all dependency checks and return results."""
    return {
        "required": {
            "summarize": check_command("summarize"),
            "python3": sys.version_info >= (3, 8),
        },
        "document": {
            "pandoc": check_pandoc(),
        },
        "audio": {
            "mlx_audio": check_mlx_audio(),
            "kokoro": check_kokoro(),
            "ffmpeg": check_command("ffmpeg"),
        },
        "comments": {
            "yt-dlp": check_ytdlp(),
        },
        "ml_deps": check_ml_deps(),  # Individual ML dep status
    }


def print_status(name: str, installed: bool, required: bool = False):
    """Print a status line."""
    icon = "✅" if installed else ("❌" if required else "⚠️")
    status = "installed" if installed else "not found"
    print(f"  {icon} {name:20} {status}")


def determine_config(checks: dict) -> dict:
    """Determine best config based on available dependencies.
    Starts from existing user config to preserve customizations,
    falls back to defaults for missing keys."""
    from config import deep_copy, deep_merge, DEFAULT_CONFIG

    config = deep_merge(deep_copy(DEFAULT_CONFIG), load_config())
    
    # Document format: DOCX if possible, else HTML
    if checks["document"]["pandoc"]:
        config["document"]["format"] = "docx"
        config["document"]["engine"] = "pandoc"
    else:
        config["document"]["format"] = "html"
        config["document"]["engine"] = None
    
    # Audio format: MP3 if ffmpeg available, else WAV
    if checks["audio"]["ffmpeg"]:
        config["audio"]["format"] = "mp3"
    else:
        config["audio"]["format"] = "wav"
    
    # TTS engine: mlx-audio (fastest) > Kokoro PyTorch > builtin
    if checks["audio"]["mlx_audio"]:
        config["audio"]["tts_engine"] = "mlx"
    elif checks["audio"]["kokoro"]:
        config["audio"]["tts_engine"] = "kokoro"
    else:
        config["audio"]["tts_engine"] = "builtin"
    
    return config


def prompt_yn(question: str) -> bool:
    """Ask a yes/no question. Returns False in non-interactive mode."""
    try:
        response = input(f"{question} [y/N] ").strip().lower()
        return response == 'y'
    except (EOFError, KeyboardInterrupt):
        print()
        return False


def install_with_brew(formula: str, name: str) -> bool:
    """Install something with Homebrew."""
    print(f"  → Installing {name}...")
    try:
        subprocess.run(["brew", "install", formula], check=True)
        print(f"  ✅ {name} installed!")
        return True
    except subprocess.CalledProcessError:
        print(f"  ❌ Installation failed")
        return False
    except FileNotFoundError:
        print(f"  ❌ Homebrew not found. Install manually: brew install {formula}")
        return False


def get_pandoc_path() -> str:
    """Get path to pandoc in tools."""
    return os.path.expanduser("~/.openclaw/tools/pandoc/pandoc")


def install_pandoc() -> bool:
    """Install pandoc standalone binary."""
    import platform
    import urllib.request
    import tarfile
    import zipfile
    
    tools_dir = os.path.expanduser("~/.openclaw/tools/pandoc")
    os.makedirs(tools_dir, exist_ok=True)
    
    # Determine platform
    system = platform.system().lower()
    machine = platform.machine().lower()
    
    # Try to fetch latest version from GitHub API, fall back to known version
    version = None
    try:
        req = urllib.request.Request(
            "https://api.github.com/repos/jgm/pandoc/releases/latest",
            headers={"Accept": "application/vnd.github.v3+json", "User-Agent": "TubeScribe-Setup"}
        )
        with urllib.request.urlopen(req, timeout=10) as resp:
            data = json.loads(resp.read().decode('utf-8'))
            tag = data.get("tag_name", "")
            if tag:
                version = tag.lstrip("v")
    except Exception:
        pass
    if not version:
        version = "3.6.4"  # Fallback, last checked 2026-02
    
    if system == "darwin":
        if machine == "arm64":
            archive = f"pandoc-{version}-arm64-macOS.zip"
        else:
            archive = f"pandoc-{version}-x86_64-macOS.zip"
    elif system == "linux":
        if machine == "aarch64":
            archive = f"pandoc-{version}-linux-arm64.tar.gz"
        else:
            archive = f"pandoc-{version}-linux-amd64.tar.gz"
    else:
        print(f"  ❌ Unsupported platform: {system}/{machine}")
        return False
    
    url = f"https://github.com/jgm/pandoc/releases/download/{version}/{archive}"
    download_path = os.path.join(tools_dir, archive)
    
    try:
        print(f"  → Downloading pandoc {version}...")
        urllib.request.urlretrieve(url, download_path)
        
        print("  → Extracting...")
        if archive.endswith(".zip"):
            with zipfile.ZipFile(download_path, 'r') as z:
                # Validate no path traversal in zip entries
                for info in z.infolist():
                    target = os.path.realpath(os.path.join(tools_dir, info.filename))
                    if not target.startswith(os.path.realpath(tools_dir)):
                        raise ValueError(f"Unsafe zip entry: {info.filename}")
                z.extractall(tools_dir)
        else:
            with tarfile.open(download_path, 'r:gz') as t:
                # Validate no path traversal in tar entries
                for member in t.getmembers():
                    target = os.path.realpath(os.path.join(tools_dir, member.name))
                    if not target.startswith(os.path.realpath(tools_dir)):
                        raise ValueError(f"Unsafe tar entry: {member.name}")
                extract_kwargs = {"path": tools_dir}
                if sys.version_info >= (3, 12):
                    extract_kwargs["filter"] = "data"
                t.extractall(**extract_kwargs)

        # Find and move the binary
        extracted_dir = os.path.join(tools_dir, f"pandoc-{version}")
        binary_src = os.path.join(extracted_dir, "bin", "pandoc")
        
        binary_dst = os.path.join(tools_dir, "pandoc")
        if os.path.exists(binary_src):
            shutil.move(binary_src, binary_dst)
            os.chmod(binary_dst, 0o755)
        
        # Cleanup
        os.remove(download_path)
        shutil.rmtree(extracted_dir, ignore_errors=True)
        
        print(f"  ✅ Pandoc installed to {tools_dir}")
        return True
        
    except Exception as e:
        print(f"  ❌ Failed: {e}")
        return False


def check_pandoc() -> bool:
    """Check if pandoc is available (system or tools)."""
    # Check system first
    if shutil.which("pandoc"):
        return True
    # Check our tools dir
    tools_pandoc = get_pandoc_path()
    return os.path.exists(tools_pandoc) and os.access(tools_pandoc, os.X_OK)


def get_ytdlp_path() -> str:
    """Get path to yt-dlp in our tools directory."""
    return os.path.expanduser("~/.openclaw/tools/yt-dlp/yt-dlp")


def find_ytdlp() -> str | None:
    """Find yt-dlp binary. Delegates to tubescribe.find_ytdlp."""
    return _find_ytdlp()


def check_ytdlp() -> bool:
    """Check if yt-dlp is available anywhere."""
    return find_ytdlp() is not None


def install_ytdlp() -> bool:
    """
    Install yt-dlp standalone binary.
    
    Installation location: ~/.openclaw/tools/yt-dlp/yt-dlp
    
    This is a self-contained binary that doesn't conflict with
    system installations (Homebrew, pip, etc.). If user later
    installs via Homebrew, that will take precedence (checked first).
    """
    import platform
    import urllib.request
    
    # Check if already installed somewhere
    existing = find_ytdlp()
    if existing:
        print(f"  ℹ️  yt-dlp already installed at {existing}")
        return True
    
    tools_dir = os.path.expanduser("~/.openclaw/tools/yt-dlp")
    os.makedirs(tools_dir, exist_ok=True)
    
    # Determine platform
    system = platform.system().lower()
    machine = platform.machine().lower()
    
    # yt-dlp releases - standalone binaries
    # https://github.com/yt-dlp/yt-dlp/releases
    base_url = "https://github.com/yt-dlp/yt-dlp/releases/latest/download"
    
    if system == "darwin":
        binary_name = "yt-dlp_macos"  # Universal binary (x86_64 + arm64)
    elif system == "linux":
        if machine == "aarch64":
            binary_name = "yt-dlp_linux_aarch64"
        else:
            binary_name = "yt-dlp_linux"
    else:
        print(f"  ❌ Unsupported platform: {system}/{machine}")
        print(f"     Try: pip install yt-dlp")
        return False
    
    url = f"{base_url}/{binary_name}"
    download_path = os.path.join(tools_dir, "yt-dlp")
    
    try:
        print(f"  → Downloading yt-dlp...")
        urllib.request.urlretrieve(url, download_path)
        os.chmod(download_path, 0o755)
        
        print(f"  ✅ yt-dlp installed to {download_path}")
        return True
        
    except Exception as e:
        print(f"  ❌ Failed: {e}")
        print(f"     Try manually: brew install yt-dlp")
        return False


def is_apple_silicon() -> bool:
    """Check if running on Apple Silicon (M1/M2/M3/M4)."""
    import platform
    return platform.system() == "Darwin" and platform.machine() == "arm64"


def install_mlx_audio() -> bool:
    """Install mlx-audio in a dedicated venv at ~/.openclaw/tools/mlx-audio/.
    
    Only works on Apple Silicon (requires MLX framework).
    Creates a Python venv, installs mlx-audio + dependencies.
    """
    import platform
    
    if not is_apple_silicon():
        print("  ❌ mlx-audio requires Apple Silicon (M1/M2/M3/M4)")
        print("     Use Kokoro TTS (PyTorch) instead.")
        return False
    
    mlx_dir = os.path.expanduser("~/.openclaw/tools/mlx-audio")
    venv_dir = os.path.join(mlx_dir, ".venv")
    
    try:
        os.makedirs(mlx_dir, exist_ok=True)
        
        # Find a suitable Python (3.10-3.12 work best with MLX)
        python_candidates = [
            "/usr/local/bin/python3.12",
            "/opt/homebrew/bin/python3.12",
            "/opt/homebrew/bin/python3.13",
            sys.executable,
        ]
        python_bin = None
        for p in python_candidates:
            if os.path.exists(p):
                python_bin = p
                break
        
        if not python_bin:
            print("  ❌ No suitable Python found")
            return False
        
        # Create venv
        if not os.path.exists(venv_dir):
            print(f"  → Creating venv with {os.path.basename(python_bin)}...")
            subprocess.run([python_bin, "-m", "venv", venv_dir], check=True)
        
        venv_pip = os.path.join(venv_dir, "bin", "pip")
        venv_python = os.path.join(venv_dir, "bin", "python3")
        
        # Install mlx-audio + deps
        print("  → Installing mlx-audio (this may take a minute)...")
        subprocess.run(
            [venv_pip, "install", "--quiet", "mlx-audio", "misaki", "num2words",
             "spacy", "espeakng_loader"],
            check=True
        )
        
        # Patch misaki/espeak.py if homebrew espeak-ng is available
        brew_lib = "/opt/homebrew/lib/libespeak-ng.dylib"
        if os.path.exists(brew_lib):
            import glob
            import re as _re
            espeak_files = glob.glob(os.path.join(venv_dir, "lib", "python3.*",
                                                   "site-packages", "misaki", "espeak.py"))
            for espeak_py in espeak_files:
                with open(espeak_py, 'r') as f:
                    content = f.read()
                # Skip if already patched
                if "/opt/homebrew" in content:
                    continue
                # Use regex to match the target pattern flexibly (survives minor whitespace/comment changes)
                pattern = _re.compile(
                    r'EspeakWrapper\.set_library\(espeakng_loader\.get_library_path\(\)\).*?'
                    r'EspeakWrapper\.set_data_path\(espeakng_loader\.get_data_path\(\)\)',
                    _re.DOTALL
                )
                if pattern.search(content):
                    print("  → Patching misaki for homebrew espeak-ng...")
                    replacement = (
                        "import os as _os\n"
                        "_brew_lib = '/opt/homebrew/lib/libespeak-ng.dylib'\n"
                        "_brew_data = '/opt/homebrew/share/espeak-ng-data'\n"
                        "if _os.path.exists(_brew_lib):\n"
                        "    EspeakWrapper.set_library(_brew_lib)\n"
                        "    EspeakWrapper.data_path = _brew_data\n"
                        "else:\n"
                        "    EspeakWrapper.set_library(espeakng_loader.get_library_path())\n"
                        "    EspeakWrapper.data_path = espeakng_loader.get_data_path()"
                    )
                    content = pattern.sub(replacement, content, count=1)
                    with open(espeak_py, 'w') as f:
                        f.write(content)
                elif "espeakng_loader" in content:
                    print("  ⚠️  misaki/espeak.py has changed — skipping homebrew patch (may not be needed)")
        
        # Verify it works
        print("  → Verifying mlx-audio...")
        result = subprocess.run(
            [venv_python, "-c", "from mlx_audio.tts.generate import generate_audio; print('OK')"],
            capture_output=True, text=True, timeout=30
        )
        
        if result.returncode == 0 and "OK" in result.stdout:
            print("  ✅ mlx-audio installed and working!")
            print(f"  📁 Location: {mlx_dir}")
            print("  ℹ️  Kokoro model (~345MB) will download on first use")
            return True
        else:
            print(f"  ❌ Verification failed: {result.stderr[:200]}")
            return False
        
    except subprocess.CalledProcessError as e:
        print(f"  ❌ Installation failed: {e}")
        return False
    except Exception as e:
        print(f"  ❌ Error: {e}")
        return False


def install_kokoro() -> bool:
    """Install Kokoro TTS (PyTorch) in a clean venv.
    
    Creates ~/.openclaw/tools/kokoro/.venv with kokoro + torch.
    No git clone needed — kokoro is available on PyPI.
    """
    kokoro_dir = get_kokoro_path()
    venv_dir = os.path.join(kokoro_dir, ".venv")
    
    try:
        os.makedirs(kokoro_dir, exist_ok=True)
        
        # Find a suitable Python
        python_candidates = [
            "/usr/local/bin/python3.12",
            "/opt/homebrew/bin/python3.12",
            "/opt/homebrew/bin/python3.13",
            sys.executable,
        ]
        python_bin = None
        for p in python_candidates:
            if os.path.exists(p):
                python_bin = p
                break
        
        if not python_bin:
            print("  ❌ No suitable Python found")
            return False
        
        # Create venv
        if not os.path.exists(venv_dir):
            print(f"  → Creating venv with {os.path.basename(python_bin)}...")
            subprocess.run([python_bin, "-m", "venv", venv_dir], check=True)
        
        venv_pip = os.path.join(venv_dir, "bin", "pip")
        venv_python = os.path.join(venv_dir, "bin", "python3")
        
        # Install kokoro + deps (~500MB for PyTorch)
        print("  → Installing Kokoro + PyTorch (this may take a few minutes)...")
        subprocess.run(
            [venv_pip, "install", "--quiet", "kokoro", "soundfile", "torch",
             "misaki", "num2words", "spacy"],
            check=True
        )
        
        # Verify it works
        print("  → Verifying Kokoro...")
        result = subprocess.run(
            [venv_python, "-c", "from kokoro import KPipeline; print('OK')"],
            capture_output=True, text=True, timeout=30
        )
        
        if result.returncode == 0 and "OK" in result.stdout:
            print("  ✅ Kokoro installed and working!")
            print(f"  📁 Location: {kokoro_dir}")
            print("  ℹ️  Voice model (~326MB) will download on first use")
            return True
        else:
            print(f"  ❌ Verification failed: {result.stderr[:200]}")
            return False
        
    except subprocess.CalledProcessError as e:
        print(f"  ❌ Installation failed: {e}")
        return False
    except Exception as e:
        print(f"  ❌ Error: {e}")
        return False


# save_config is imported from config.py


def main(check_only: bool = False, quiet: bool = False):
    if not quiet:
        print_header()
        print("Checking dependencies...\n")
    
    checks = run_checks()
    
    # === REQUIRED DEPENDENCIES ===
    if not quiet:
        print("Required:")
    
    all_required = True
    for name, installed in checks["required"].items():
        if not quiet:
            print_status(name, installed, required=True)
        if not installed:
            all_required = False
    
    if not all_required:
        print("\n❌ Missing required dependencies!")
        if not checks["required"]["summarize"]:
            print("\n   Install summarize CLI:")
            print("   brew install steipete/tap/summarize")
        sys.exit(1)
    
    # === OPTIONAL DEPENDENCIES ===
    if not quiet:
        print("\nDocument output:")
        print_status("pandoc", checks["document"]["pandoc"])
        print("\nAudio output:")
        print_status("mlx-audio", checks["audio"]["mlx_audio"])
        print_status("kokoro", checks["audio"]["kokoro"])
        print_status("ffmpeg", checks["audio"]["ffmpeg"])
        
        print("\nComments:")
        print_status("yt-dlp", checks["comments"]["yt-dlp"])
    
    if check_only:
        if quiet:
            print(json.dumps(checks))
        return checks
    
    # === DETERMINE BEST CONFIG ===
    config = determine_config(checks)
    
    if not quiet:
        print("\n" + "─" * 40)
        print("\n📋 Your configuration (based on available tools):\n")
        
        doc_format = config["document"]["format"]
        doc_engine = config["document"].get("engine")
        doc_note = f" (via {doc_engine})" if doc_engine else ""
        if doc_format == "html":
            doc_note = " (no dependencies needed)"
        print(f"  📄 Document: {doc_format.upper()}{doc_note}")
        
        audio_format = config["audio"]["format"]
        tts_engine = config["audio"]["tts_engine"]
        if tts_engine == "mlx":
            audio_note = " (MLX — fastest on Apple Silicon)"
        elif tts_engine == "kokoro":
            audio_note = " (Kokoro PyTorch)"
        else:
            audio_note = " (built-in macOS voice)"
        print(f"  🔊 Audio:    {audio_format.upper()}{audio_note}")
        print(f"  📁 Output:   {config['output']['folder']}")
    
    # === OFFER TO INSTALL MISSING FOR BEST EXPERIENCE ===
    missing_upgrades = []
    
    if not checks["document"]["pandoc"]:
        missing_upgrades.append({
            "name": "pandoc",
            "desc": "DOCX support",
            "why": "Better document formatting, opens in Word/Pages",
            "brew": "pandoc",
            "standalone_installer": install_pandoc,
            "config_update": lambda c: c["document"].update({"format": "docx", "engine": "pandoc"}),
        })
    
    if not checks["audio"]["ffmpeg"]:
        missing_upgrades.append({
            "name": "ffmpeg",
            "desc": "MP3 audio output",
            "why": "Smaller file sizes (MP3 vs WAV)",
            "brew": "ffmpeg",
            "config_update": lambda c: c["audio"].update({"format": "mp3"}),
        })
    
    if not checks["audio"]["mlx_audio"] and not checks["audio"]["kokoro"]:
        # Neither TTS backend available — offer the best one for this platform
        if is_apple_silicon():
            missing_upgrades.append({
                "name": "mlx-audio",
                "desc": "High-quality voices (MLX — fastest on Apple Silicon)",
                "why": "Natural-sounding speech, 3-4x faster than PyTorch on M-series chips",
                "brew": None,
                "installer": install_mlx_audio,
                "install_note": "Requires ~900MB download (one-time)",
                "config_update": lambda c: c["audio"].update({"tts_engine": "mlx"}),
            })
        else:
            missing_upgrades.append({
                "name": "Kokoro TTS",
                "desc": "High-quality voices (PyTorch)",
                "why": "Natural-sounding speech instead of robotic macOS voice",
                "brew": None,
                "installer": install_kokoro,
                "install_note": "Requires ~500MB download (one-time)",
                "config_update": lambda c: c["audio"].update({"tts_engine": "kokoro"}),
            })
    
    if not checks["comments"]["yt-dlp"]:
        missing_upgrades.append({
            "name": "yt-dlp",
            "desc": "YouTube comments",
            "why": "Extract top comments for summary & best-of section",
            "brew": "yt-dlp",
            "standalone_installer": install_ytdlp,
            "config_update": lambda c: c.setdefault("comments", {}).update({"enabled": True}),
        })
    
    if missing_upgrades and not quiet:
        print("\n" + "─" * 40)
        print("\n🚀 For the best experience, consider installing:\n")
        
        for upgrade in missing_upgrades:
            print(f"  • {upgrade['desc']} ({upgrade['name']})")
            print(f"    → {upgrade['why']}")
            
            if upgrade.get("install_note"):
                print(f"    {upgrade['install_note']}")
            
            if upgrade.get("brew") and upgrade.get("standalone_installer"):
                # Offer choice: brew or standalone
                if prompt_yn(f"\n    Install {upgrade['name']}?"):
                    print("      1. Homebrew (brew install)")
                    print("      2. Standalone binary (~/.openclaw/tools/)")
                    choice = input("      Choice [1/2]: ").strip()
                    success = False
                    if choice == "2":
                        success = upgrade["standalone_installer"]()
                    else:
                        success = install_with_brew(upgrade["brew"], upgrade["name"])
                    if success and upgrade.get("config_update"):
                        upgrade["config_update"](config)
            elif upgrade.get("brew"):
                if prompt_yn(f"\n    Install {upgrade['name']}?"):
                    if install_with_brew(upgrade["brew"], upgrade["name"]):
                        if upgrade.get("config_update"):
                            upgrade["config_update"](config)
            elif upgrade.get("installer"):
                if prompt_yn(f"\n    Install {upgrade['name']}?"):
                    if upgrade["installer"]():
                        if upgrade.get("config_update"):
                            upgrade["config_update"](config)
            print()
    
    # === SAVE CONFIG ===
    save_config(config)
    
    if not quiet:
        print("─" * 40)
        print(f"\n💾 Config saved to: {CONFIG_FILE}")
        print("\n✅ Setup complete!\n")
        
        print("Final configuration:")
        print(f"  📄 Document: {config['document']['format'].upper()}")
        print(f"  🔊 Audio:    {config['audio']['format'].upper()} via {config['audio']['tts_engine']}")
        print(f"  📁 Output:   {config['output']['folder']}")
        
        print("\n" + "─" * 40)
        print("\nUsage: Just send a YouTube URL to your OpenClaw agent!")
        print("       Or run: python tubescribe.py <youtube_url>\n")
    
    return config


if __name__ == "__main__":
    check_only = "--check-only" in sys.argv
    quiet = "--quiet" in sys.argv
    main(check_only=check_only, quiet=quiet)


path: scripts/tubescribe.py
content: #!/usr/bin/env python3
"""
TubeScribe - YouTube Video Summarizer
=====================================

Extracts transcripts from YouTube videos, generates summaries with speaker
detection, and creates document + audio output.

Usage:
    python tubescribe.py <youtube_url> [options]
    python tubescribe.py url1 url2 url3  # Batch mode
    python tubescribe.py --queue-status  # Queue management

Options:
    --output-dir DIR     Output directory (default: from config)
    --doc-format FORMAT  Document format: html, docx, md
    --audio-format FMT   Audio format: wav, mp3, none
    --no-audio           Skip audio generation
    --quiet              Minimal output
    --queue-add URL      Add URL to processing queue
    --queue-status       Show queue status
    --queue-next         Process next item from queue
    --queue-clear        Clear the queue

Author: Jackie 🦊 & Matus 🇸🇰
"""

__version__ = "1.1.8"

import subprocess
import json
import re
import sys
import os
import tempfile
import time
import urllib.request
from contextlib import contextmanager
from pathlib import Path
from datetime import datetime, timedelta

try:
    import fcntl
    _HAS_FCNTL = True
except ImportError:
    _HAS_FCNTL = False

# Add scripts dir to path for imports
SCRIPT_DIR = Path(__file__).parent
sys.path.insert(0, str(SCRIPT_DIR))

from html_writer import create_html_from_markdown
from config import load_config, save_config, CONFIG_FILE


def extract_video_id(url: str) -> str | None:
    """Extract video ID from various YouTube URL formats."""
    patterns = [
        r'(?:v=|/v/|youtu\.be/)([a-zA-Z0-9_-]{11})',
        r'(?:embed/)([a-zA-Z0-9_-]{11})',
        r'(?:shorts/)([a-zA-Z0-9_-]{11})',  # YouTube Shorts
        r'(?:live/)([a-zA-Z0-9_-]{11})',    # YouTube Live
        r'^([a-zA-Z0-9_-]{11})$'            # Just the ID
    ]
    for pattern in patterns:
        match = re.search(pattern, url)
        if match:
            return match.group(1)
    return None


def is_youtube_url(url: str) -> bool:
    """Check if URL is a valid YouTube URL with proper domain validation."""
    from urllib.parse import urlparse

    # First check if we can extract a video ID
    video_id = extract_video_id(url)
    if not video_id:
        return False

    # If it's just a bare video ID (11 chars), accept it
    if re.match(r'^[a-zA-Z0-9_-]{11}$', url):
        return True

    # Otherwise validate the domain using proper URL parsing
    valid_domains = {
        'youtube.com',
        'www.youtube.com',
        'm.youtube.com',
        'youtu.be',
        'music.youtube.com',
    }

    # Ensure URL has a scheme for proper parsing
    parse_url = url if url.startswith(('http://', 'https://')) else 'https://' + url
    parsed = urlparse(parse_url)
    hostname = (parsed.hostname or '').lower()

    return hostname in valid_domains


# Error patterns from summarize/yt-dlp
ERROR_PATTERNS = {
    "private": [
        "Private video",
        "Video is private",
        "Sign in if you've been granted access",
    ],
    "unavailable": [
        "Video unavailable",
        "This video isn't available",
        "removed by the uploader",
        "account associated with this video has been terminated",
    ],
    "no_captions": [
        "Subtitles are disabled",
        "no subtitles",
        "Could not retrieve",
        "No transcript",
    ],
    "age_restricted": [
        "Sign in to confirm your age",
        "age-restricted",
        "Age-restricted",
    ],
    "region_blocked": [
        "not available in your country",
        "geo-restricted",
        "blocked in your country",
    ],
}

ERROR_MESSAGES = {
    "private": "❌ Video is private — can't access",
    "unavailable": "❌ Video not found or removed",
    "no_captions": "❌ No captions available for this video",
    "age_restricted": "❌ Age-restricted video — can't access without login",
    "region_blocked": "❌ Video blocked in your region",
    "invalid_url": "❌ Not a valid YouTube URL",
    "network": "❌ Network error — check your connection",
    "timeout": "❌ Request timed out — try again later",
    "unknown": "❌ Failed to process video",
}


def detect_error(text: str) -> str | None:
    """Detect error type from output text."""
    text_lower = text.lower()
    for error_type, patterns in ERROR_PATTERNS.items():
        for pattern in patterns:
            if pattern.lower() in text_lower:
                return error_type
    return None


def find_ytdlp() -> str | None:
    """
    Find yt-dlp binary in common locations.
    Returns full path if found, None otherwise.
    
    Search order:
    1. System PATH (via shutil.which)
    2. Homebrew locations (/opt/homebrew, /usr/local)
    3. pip/pipx user installs (~/.local/bin)
    4. Our tools directory (~/.openclaw/tools/yt-dlp)
    """
    import shutil
    
    # Check system PATH first
    system_ytdlp = shutil.which("yt-dlp")
    if system_ytdlp:
        return system_ytdlp
    
    # Common installation paths
    paths = [
        "/opt/homebrew/bin/yt-dlp",           # Homebrew (Apple Silicon)
        "/usr/local/bin/yt-dlp",              # Homebrew (Intel) / Linux
        "/usr/bin/yt-dlp",                    # System (Linux)
        os.path.expanduser("~/.local/bin/yt-dlp"),  # pip install --user
        os.path.expanduser("~/.local/pipx/venvs/yt-dlp/bin/yt-dlp"),  # pipx
        os.path.expanduser("~/.openclaw/tools/yt-dlp/yt-dlp"),  # Our tools
    ]
    
    for path in paths:
        if os.path.exists(path) and os.access(path, os.X_OK):
            return path
    
    return None


def fetch_comments(url: str, max_comments: int = None, timeout: int = None, config: dict = None) -> tuple[list[dict] | None, str | None]:
    """
    Fetch top comments from YouTube video.
    Returns (comments_list, error_message).
    Uses config values for max_comments and timeout if not specified.
    """
    ytdlp = find_ytdlp()
    if not ytdlp:
        return None, None  # No yt-dlp, skip comments silently

    # Get from config if not specified
    if config is None:
        config = load_config()
    if max_comments is None:
        max_comments = config.get("comments", {}).get("max_count", 50)
    if timeout is None:
        timeout = config.get("comments", {}).get("timeout", 90)
    
    try:
        # comment_sort=top gets highest-liked comments (default is "new")
        # Format: max_comments=max-comments,max-parents,max-replies,max-replies-per-thread,max-depth
        # We fetch top-level parent comments only: depth=1, no replies
        result = subprocess.run(
            [ytdlp, "--dump-json", "--no-download", "--write-comments", 
             "--extractor-args", f"youtube:comment_sort=top;max_comments={max_comments},{max_comments},0,0,1",
             url],
            capture_output=True, text=True, timeout=timeout
        )
        
        if result.returncode != 0:
            return None, None  # Failed, skip comments silently
        
        data = json.loads(result.stdout)
        raw_comments = data.get("comments", [])
        
        if not raw_comments:
            return [], None
        
        # Sort by likes and extract relevant fields
        comments = []
        for c in raw_comments:
            comments.append({
                "author": c.get("author", "Unknown"),
                "text": c.get("text", ""),
                "likes": c.get("like_count", 0),
                "time": c.get("timestamp", 0),
                "time_text": c.get("time_text", ""),
            })
        
        # Sort by likes descending
        comments.sort(key=lambda x: x["likes"], reverse=True)
        
        return comments[:max_comments], None
        
    except subprocess.TimeoutExpired:
        return None, None  # Timeout, skip comments
    except json.JSONDecodeError:
        return None, None  # Invalid JSON, skip comments
    except Exception:
        return None, None  # Any error, skip comments


MAX_HTML_SIZE = 5 * 1024 * 1024  # 5MB limit for safety


def format_duration(seconds: int) -> str:
    """Format seconds as H:MM:SS or M:SS."""
    hours = seconds // 3600
    minutes = (seconds % 3600) // 60
    secs = seconds % 60
    if hours > 0:
        return f"{hours}:{minutes:02d}:{secs:02d}"
    return f"{minutes}:{secs:02d}"


def safe_unescape(text: str) -> str:
    """Safely decode unicode escapes, handling edge cases."""
    if not text:
        return text
    try:
        # Use json.loads to properly decode JSON string escapes
        return json.loads(f'"{text}"')
    except (json.JSONDecodeError, ValueError):
        # Fallback: just handle the common ones manually
        text = text.replace('\\\\', '\\')
        text = text.replace('\\n', '\n')
        text = text.replace('\\t', '\t')
        text = text.replace('\\"', '"')
        return text


def get_video_metadata(url: str) -> tuple[dict | None, str | None]:
    """
    Extract video metadata from YouTube.
    Returns (metadata_dict, error_message).
    Uses yt-dlp if available, falls back to HTML scraping.
    """
    video_id = extract_video_id(url)
    
    # Try yt-dlp first (best metadata)
    ytdlp = find_ytdlp()
    if ytdlp:
        try:
            result = subprocess.run(
                [ytdlp, "--dump-json", "--no-download", "--no-warnings", url],
                capture_output=True, text=True, timeout=30
            )
            if result.returncode == 0:
                data = json.loads(result.stdout)
                return {
                    "title": data.get("title", "Unknown"),
                    "channel": data.get("channel", data.get("uploader", "Unknown")),
                    "upload_date": data.get("upload_date", ""),  # YYYYMMDD format
                    "duration": data.get("duration", 0),
                    "duration_string": format_duration(data.get("duration", 0)),
                    "view_count": data.get("view_count", 0),
                    "description": data.get("description", ""),
                    "video_id": video_id,
                }, None
            else:
                # Check for known errors
                error_type = detect_error(result.stderr)
                if error_type:
                    return None, ERROR_MESSAGES[error_type]
        except subprocess.TimeoutExpired:
            return None, ERROR_MESSAGES["timeout"]
        except json.JSONDecodeError:
            pass  # Invalid JSON, fall back to HTML
    
    # Fallback: HTML scraping
    try:
        req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
        with urllib.request.urlopen(req, timeout=15) as response:
            html = response.read(MAX_HTML_SIZE).decode('utf-8')
        
        # Check for error pages
        error_type = detect_error(html)
        if error_type:
            return None, ERROR_MESSAGES[error_type]
        
        # Extract metadata from JSON in HTML
        title_match = re.search(r'"title":"([^"]+)"', html)
        title = safe_unescape(title_match.group(1)) if title_match else "Unknown"
        
        channel_match = re.search(r'"ownerChannelName":"([^"]+)"', html)
        channel = safe_unescape(channel_match.group(1)) if channel_match else "Unknown"
        
        # Duration in seconds
        duration_match = re.search(r'"lengthSeconds":"(\d+)"', html)
        duration = int(duration_match.group(1)) if duration_match else 0
        
        # Upload date
        date_match = re.search(r'"uploadDate":"([^"]+)"', html)
        upload_date = date_match.group(1) if date_match else ""
        
        desc_match = re.search(r'"shortDescription":"([^"]*)"', html)
        description = safe_unescape(desc_match.group(1)) if desc_match else ""
        
        return {
            "title": title,
            "channel": channel,
            "upload_date": upload_date,
            "duration": duration,
            "duration_string": format_duration(duration),
            "view_count": 0,  # Not easily available from HTML
            "description": description,
            "video_id": video_id,
        }, None
        
    except urllib.error.HTTPError as e:
        if e.code == 404:
            return None, ERROR_MESSAGES["unavailable"]
        return None, ERROR_MESSAGES["unknown"]
    except urllib.error.URLError:
        return None, ERROR_MESSAGES["network"]
    except TimeoutError:
        return None, ERROR_MESSAGES["timeout"]
    except Exception as e:
        return None, f"❌ Failed to fetch metadata: {e}"


def normalize_youtube_url(url: str) -> str:
    """Normalize any YouTube URL format to standard watch?v= format."""
    video_id = extract_video_id(url)
    if video_id:
        return f"https://www.youtube.com/watch?v={video_id}"
    return url


def download_transcript(url: str) -> tuple[list[dict] | None, str | None]:
    """
    Download transcript with timestamps using summarize CLI.
    Returns (segments, error_message).
    """
    # Normalize URL to standard format (fixes /live/, /shorts/, etc.)
    normalized_url = normalize_youtube_url(url)
    try:
        result = subprocess.run(
            ["summarize", normalized_url, "--youtube", "auto", "--extract-only", "--timestamps"],
            capture_output=True, text=True, timeout=120  # 2 min timeout for long videos
        )
        
        if result.returncode != 0:
            # Only check error patterns when the command actually failed
            combined_output = result.stdout + result.stderr
            error_type = detect_error(combined_output)
            if error_type:
                return None, ERROR_MESSAGES[error_type]
            return None, f"❌ Transcript extraction failed: {result.stderr[:200]}"
        
        lines = result.stdout.strip().split('\n')
        segments = []
        
        for line in lines:
            match = re.match(r'\[(\d+:\d+(?::\d+)?)\]\s*(.*)', line)
            if match:
                segments.append({
                    "ts": match.group(1),
                    "text": match.group(2).strip()
                })
        
        if not segments:
            return None, ERROR_MESSAGES["no_captions"]
        
        return segments, None
        
    except FileNotFoundError:
        return None, "❌ 'summarize' CLI not found. Install with: brew install steipete/tap/summarize"
    except subprocess.TimeoutExpired:
        return None, ERROR_MESSAGES["timeout"]


def clean_transcript(segments: list[dict]) -> list[dict]:
    """Clean transcript segments."""
    cleaned = []
    for seg in segments:
        text = seg["text"]
        text = re.sub(r'\s+', ' ', text).strip()
        text = re.sub(r'>>\s*', '', text)  # Remove >> artifacts
        if text:
            cleaned.append({"ts": seg["ts"], "text": text})
    return cleaned


def sanitize_filename(title: str, fallback: str = "untitled") -> str:
    """Create safe filename from title. Uses fallback if result is empty."""
    safe = re.sub(r'[<>:"/\\|?*]', '', title)
    safe = re.sub(r'\s+', ' ', safe).strip()
    safe = safe[:100]  # Limit length
    return safe if safe else fallback


def _get_temp_dir() -> str:
    """Get a secure per-user temp directory for TubeScribe."""
    try:
        uid = os.getuid()
    except AttributeError:
        uid = os.getenv("USERNAME", "default")
    temp_dir = os.path.join(tempfile.gettempdir(), f"tubescribe-{uid}")
    os.makedirs(temp_dir, mode=0o700, exist_ok=True)
    return temp_dir


def prepare_source_data(url: str, temp_dir: str = None, quiet: bool = False) -> tuple[str | None, str | None, str | None]:
    """
    Prepare source data for processing.
    Returns: (source_json_path, output_md_path, error_message)
    """
    if temp_dir is None:
        temp_dir = _get_temp_dir()

    def log(msg: str):
        if not quiet:
            print(msg)

    # Validate URL first
    if not is_youtube_url(url):
        return None, None, ERROR_MESSAGES["invalid_url"]
    
    video_id = extract_video_id(url)

    timings = {}
    
    log(f"\n🎬 TubeScribe")
    log(f"{'─' * 50}")
    
    # Check if yt-dlp available for comments (always fetch if available)
    has_ytdlp = find_ytdlp() is not None
    total_steps = 4 if has_ytdlp else 3
    step = 0
    
    # Step 1: Fetch metadata
    step += 1
    log(f"\n[{step}/{total_steps}] 🔍 Fetching video metadata...")
    t_start = time.time()
    metadata, error = get_video_metadata(url)
    timings['metadata'] = time.time() - t_start
    if error:
        return None, None, error
    
    title_display = metadata['title'][:50] + "..." if len(metadata['title']) > 50 else metadata['title']
    log(f"      → \"{title_display}\"")
    log(f"      → {metadata['channel']} • {metadata['duration_string']}")
    log(f"      ⏱️ {timings['metadata']:.1f}s")
    
    # Step 2: Download transcript
    step += 1
    log(f"\n[{step}/{total_steps}] 📝 Extracting transcript...")
    t_start = time.time()
    segments, error = download_transcript(url)
    timings['transcript'] = time.time() - t_start
    if error:
        return None, None, error
    
    log(f"      → {len(segments)} raw segments")
    log(f"      ⏱️ {timings['transcript']:.1f}s")
    
    # Step 3: Clean transcript
    step += 1
    log(f"\n[{step}/{total_steps}] 🧹 Cleaning transcript...")
    t_start = time.time()
    segments = clean_transcript(segments)
    timings['cleaning'] = time.time() - t_start
    word_count = sum(len(s["text"].split()) for s in segments)
    
    log(f"      → {len(segments)} segments")
    log(f"      → ~{word_count:,} words")
    log(f"      ⏱️ {timings['cleaning']:.1f}s")
    
    # Estimate processing time
    if word_count < 3000:
        est_time = "2-4 minutes"
    elif word_count < 10000:
        est_time = "5-8 minutes"
    else:
        est_time = "10-15 minutes"
    log(f"      → Est. processing: {est_time}")
    
    # Step 4: Fetch comments (optional)
    comments = []
    if has_ytdlp:
        step += 1
        log(f"\n[{step}/{total_steps}] 💬 Fetching comments...")
        t_start = time.time()
        cfg = load_config()
        max_comments_count = cfg.get("comments", {}).get("max_count", 50)
        comments, _ = fetch_comments(url, max_comments=max_comments_count, config=cfg)
        timings['comments'] = time.time() - t_start
        if comments:
            log(f"      → {len(comments)} top comments")
        else:
            log(f"      → No comments available")
        log(f"      ⏱️ {timings['comments']:.1f}s")
    
    # Prepare output with full metadata
    source_data = {
        "metadata": {
            "url": url,
            "video_id": video_id,
            "title": metadata["title"],
            "channel": metadata["channel"],
            "upload_date": metadata["upload_date"],
            "duration": metadata["duration"],
            "duration_string": metadata["duration_string"],
            "view_count": metadata.get("view_count", 0),
            "description": metadata["description"],
        },
        "segments": segments,
        "comments": comments if comments else [],
        "stats": {
            "segment_count": len(segments),
            "word_count": word_count,
            "comment_count": len(comments) if comments else 0,
        }
    }
    
    source_path = f"{temp_dir}/tubescribe_{video_id}_source.json"
    output_path = f"{temp_dir}/tubescribe_{video_id}_output.md"
    
    with open(source_path, 'w', encoding='utf-8') as f:
        json.dump(source_data, f, indent=2, ensure_ascii=False)
    
    log(f"\n{'─' * 50}")
    log(f"✅ Ready for processing")
    log(f"   Source: {source_path}")
    log(f"   Output: {output_path}")
    
    # Timing summary
    total_extract = sum(timings.values())
    log(f"\n⏱️ Extraction timing:")
    log(f"   Metadata:   {timings.get('metadata', 0):.1f}s")
    log(f"   Transcript: {timings.get('transcript', 0):.1f}s")
    log(f"   Cleaning:   {timings.get('cleaning', 0):.1f}s")
    if 'comments' in timings:
        log(f"   Comments:   {timings['comments']:.1f}s")
    log(f"   Total:      {total_extract:.1f}s")
    
    return source_path, output_path, None


def cleanup_temp_files(video_id: str, quiet: bool = False, temp_dir: str = None) -> bool:
    """Remove temp files for a video after successful processing."""
    if temp_dir is None:
        temp_dir = _get_temp_dir()
    source_path = f"{temp_dir}/tubescribe_{video_id}_source.json"
    output_path = f"{temp_dir}/tubescribe_{video_id}_output.md"
    tts_path = f"{temp_dir}/tubescribe_{video_id}_tts.py"

    cleaned = False
    for path in [source_path, output_path, tts_path]:
        if os.path.exists(path):
            os.remove(path)
            cleaned = True
            if not quiet:
                print(f"   🗑️  Removed {path}")
    
    return cleaned


def convert_to_document(md_path: str, output_dir: str, doc_format: str, video_id: str = None) -> str:
    """Convert markdown to final document format."""
    
    # Get title from markdown for filename
    with open(md_path, 'r', encoding='utf-8') as f:
        content = f.read()
    title_match = re.search(r'^# (.+)$', content, re.MULTILINE)
    title = title_match.group(1) if title_match else "TubeScribe Output"
    fallback = video_id if video_id else "TubeScribe_Output"
    safe_title = sanitize_filename(title, fallback=fallback)
    
    if doc_format == "md":
        # Just copy/rename markdown
        out_path = os.path.join(output_dir, f"{safe_title}.md")
        with open(out_path, 'w', encoding='utf-8') as f:
            f.write(content)
        return out_path
    
    elif doc_format == "html":
        out_path = os.path.join(output_dir, f"{safe_title}.html")
        create_html_from_markdown(md_path, out_path)
        return out_path
    
    elif doc_format == "docx":
        out_path = os.path.join(output_dir, f"{safe_title}.docx")
        # Try pandoc first (system or tools)
        pandoc_paths = [
            "pandoc",  # System PATH
            os.path.expanduser("~/.openclaw/tools/pandoc/pandoc"),  # Our tools
        ]
        for pandoc in pandoc_paths:
            try:
                subprocess.run(
                    [pandoc, md_path, "-o", out_path],
                    check=True, capture_output=True
                )
                return out_path
            except FileNotFoundError:
                continue
        
        # No pandoc available, fall back to HTML
        print("   pandoc not found, falling back to HTML")
        out_path = os.path.join(output_dir, f"{safe_title}.html")
        create_html_from_markdown(md_path, out_path)
        return out_path

    else:
        print(f"   Unknown document format '{doc_format}', falling back to markdown")
        out_path = os.path.join(output_dir, f"{safe_title}.md")
        with open(out_path, 'w', encoding='utf-8') as f:
            f.write(content)
        return out_path


def generate_audio_summary(summary_text: str, output_path: str, config: dict) -> str:
    """Generate audio summary using configured TTS engine."""

    audio_format = config.get("audio", {}).get("format", "wav")
    tts_engine = config.get("audio", {}).get("tts_engine", "builtin")

    print(f"🔊 Generating audio ({tts_engine}, {audio_format})...")

    if tts_engine == "mlx":
        return generate_mlx_audio(summary_text, output_path, audio_format, config)
    elif tts_engine == "kokoro":
        return generate_kokoro_audio(summary_text, output_path, audio_format, config)
    else:
        return generate_builtin_audio(summary_text, output_path, audio_format)


def find_mlx_audio(config: dict = None) -> tuple[str | None, str | None]:
    """
    Find mlx-audio installation.
    Returns (python_path, mlx_audio_dir).
    """
    if config is None:
        config = load_config()
    
    mlx_config = config.get("mlx_audio", {})
    mlx_dir = os.path.expanduser(mlx_config.get("path", "~/.openclaw/tools/mlx-audio"))
    
    # Check venv inside mlx-audio dir
    venv_python = os.path.join(mlx_dir, ".venv", "bin", "python3")
    if os.path.exists(venv_python):
        try:
            result = subprocess.run(
                [venv_python, "-c", "from mlx_audio.tts.generate import generate_audio; print('ok')"],
                capture_output=True, text=True, timeout=15
            )
            if result.returncode == 0:
                return venv_python, mlx_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    # Check system Python
    import shutil
    system_python = shutil.which("python3")
    if system_python:
        try:
            result = subprocess.run(
                [system_python, "-c", "from mlx_audio.tts.generate import generate_audio; print('ok')"],
                capture_output=True, text=True, timeout=15
            )
            if result.returncode == 0:
                return system_python, mlx_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    return None, None


def _get_or_create_mlx_blended_voice(mlx_python: str, voice_blend: dict, model: str) -> str | None:
    """
    Create a blended voice .safetensors file from multiple voices with custom weights.
    Caches the result so it only runs once.
    Returns path to the blended .safetensors file, or None on failure.
    """
    # Build a deterministic filename from the blend spec
    parts = sorted(voice_blend.items())
    blend_name = "_".join(f"{re.sub(r'[^a-z0-9_]', '', name)}_{int(weight*100)}" for name, weight in parts)
    if not blend_name:
        return None
    safe_blend_name = json.dumps(blend_name)
    
    # Check if blend already exists in HF cache
    code = f'''
import json
from huggingface_hub import snapshot_download
from pathlib import Path

local_dir = snapshot_download(repo_id={json.dumps(model)}, local_files_only=True)
blend_path = str(Path(local_dir) / "voices" / ({safe_blend_name} + ".safetensors"))

import os
if os.path.exists(blend_path):
    print(blend_path)
else:
    print("NEEDS_CREATE")
    print(local_dir)
'''
    try:
        result = subprocess.run(
            [mlx_python, "-c", code],
            capture_output=True, text=True, timeout=15
        )
        if result.returncode != 0:
            return None
        
        lines = result.stdout.strip().split('\n')
        if lines[0] != "NEEDS_CREATE":
            return lines[0]  # Already exists
        
        local_dir = lines[1]
    except (subprocess.SubprocessError, OSError, TimeoutError):
        return None
    
    # Create the blended voice
    blend_spec = json.dumps(voice_blend)
    code = f'''
import json
import numpy as np
import mlx.core as mx
from mlx_audio.tts.models.kokoro.voice import load_voice_tensor
from safetensors.numpy import save_file

local_dir = {json.dumps(local_dir)}
voice_blend = json.loads({json.dumps(blend_spec)})
blend_name = {safe_blend_name}

# Load and blend voices
voices = []
weights = []
for name, weight in voice_blend.items():
    v = load_voice_tensor(f"{{local_dir}}/voices/{{name}}.safetensors")
    voices.append(v)
    weights.append(weight)

blended = sum(w * v for w, v in zip(weights, voices))
blend_path = f"{{local_dir}}/voices/{{blend_name}}.safetensors"
save_file({{"voice": np.array(blended)}}, blend_path)
print(blend_path)
'''
    try:
        result = subprocess.run(
            [mlx_python, "-c", code],
            capture_output=True, text=True, timeout=30
        )
        if result.returncode == 0:
            return result.stdout.strip()
    except (subprocess.SubprocessError, OSError, TimeoutError):
        pass
    
    return None


def generate_mlx_audio(text: str, output_path: str, audio_format: str, config: dict) -> str:
    """Generate audio using mlx-audio (MLX backend for Kokoro — fastest on Apple Silicon)."""
    try:
        wav_path = output_path.replace('.mp3', '.wav') if output_path.endswith('.mp3') else output_path
        if not wav_path.endswith('.wav'):
            wav_path = output_path + '.wav'

        mlx_python, mlx_dir = find_mlx_audio(config)
        if not mlx_python:
            raise Exception("mlx-audio not found. Falling back to Kokoro PyTorch.")

        mlx_config = config.get("mlx_audio", {})
        model = mlx_config.get("model", "mlx-community/Kokoro-82M-bf16")
        voice = mlx_config.get("voice", "af_heart")
        voice_blend = mlx_config.get("voice_blend", {})
        lang_code = mlx_config.get("lang_code", "a")
        speed = mlx_config.get("speed", 1.0)

        # If voice_blend is configured, create/load blended voice
        if voice_blend and len(voice_blend) > 1:
            blended_path = _get_or_create_mlx_blended_voice(mlx_python, voice_blend, model)
            if blended_path:
                voice = blended_path

        safe_text = json.dumps(text)
        safe_wav_path = json.dumps(wav_path)

        code = f'''
import os
from mlx_audio.tts.generate import generate_audio

text = {safe_text}
wav_path = {safe_wav_path}
output_dir = os.path.dirname(wav_path)
file_prefix = os.path.splitext(os.path.basename(wav_path))[0]

generate_audio(
    text=text,
    model={json.dumps(model)},
    voice={json.dumps(voice)},
    lang_code={json.dumps(lang_code)},
    speed={json.dumps(speed)},
    output_path=output_dir,
    file_prefix=file_prefix,
    audio_format="wav",
    join_audio=True,
    verbose=False,
)
print("OK")
'''
        result = subprocess.run(
            [mlx_python, "-c", code],
            capture_output=True, text=True, timeout=300
        )

        if result.returncode == 0 and os.path.exists(wav_path):
            if audio_format == "mp3":
                base = output_path[:-4] if output_path.endswith(('.mp3', '.wav')) else output_path
                mp3_path = base + '.mp3'
                subprocess.run(
                    ["ffmpeg", "-y", "-i", wav_path, "-b:a", "128k", mp3_path],
                    capture_output=True, check=True
                )
                os.remove(wav_path)
                return mp3_path
            return wav_path
        else:
            raise Exception(f"mlx-audio failed: {result.stderr[:300]}")

    except Exception as e:
        print(f"   mlx-audio failed, falling back to Kokoro PyTorch: {e}")
        return generate_kokoro_audio(text, output_path, audio_format, config)


KOKORO_DEPS = {
    "torch": "torch",
    "soundfile": "soundfile",
    "numpy": "numpy",
    "huggingface_hub": "huggingface_hub",
}


def _check_python_has_deps(python_path: str) -> bool:
    """Check if a Python has all required ML deps."""
    import_names = ", ".join(KOKORO_DEPS.values())
    try:
        result = subprocess.run(
            [python_path, "-c", f"import {import_names}"],
            capture_output=True, timeout=10
        )
        return result.returncode == 0
    except (subprocess.SubprocessError, OSError, TimeoutError):
        return False


def find_kokoro(use_cache: bool = True, config: dict = None) -> tuple[str | None, str | None]:
    """
    Find Kokoro installation efficiently.
    Returns (python_path, kokoro_dir).
    
    Checks (in order):
    1. Config cache (instant)
    2. Venv at ~/.openclaw/tools/kokoro/.venv (pip install)
    3. System Python with kokoro importable
    4. Legacy locations (cloned repos)
    """
    if config is None:
        config = load_config()
    kokoro_dir = os.path.expanduser("~/.openclaw/tools/kokoro")
    
    # 1. Check cache first (instant)
    if use_cache:
        cached = config.get("kokoro", {})
        cached_python = cached.get("python")
        cached_dir = cached.get("path")
        
        if cached_python and cached_dir:
            if cached_python == "system":
                cached_python = sys.executable
            if os.path.exists(cached_python):
                return cached_python, cached_dir
    
    # 2. Check venv at standard location (clean pip install)
    venv_python = os.path.join(kokoro_dir, ".venv", "bin", "python3")
    if os.path.exists(venv_python):
        try:
            result = subprocess.run(
                [venv_python, "-c", "from kokoro import KPipeline; print('ok')"],
                capture_output=True, timeout=10
            )
            if result.returncode == 0:
                _save_kokoro_cache(venv_python, kokoro_dir, "venv")
                return venv_python, kokoro_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    # 3. Check system Python
    if _check_python_has_deps(sys.executable):
        try:
            result = subprocess.run(
                [sys.executable, "-c", "from kokoro import KPipeline; print('ok')"],
                capture_output=True, timeout=10
            )
            if result.returncode == 0:
                _save_kokoro_cache(sys.executable, kokoro_dir, "system")
                return sys.executable, kokoro_dir
        except (subprocess.SubprocessError, OSError, TimeoutError):
            pass
    
    # 4. Legacy: check old locations (cloned repos, shared ML env)
    legacy_locations = [
        kokoro_dir,
        "/tmp/kokoro-coreml",
        os.path.expanduser("~/kokoro-coreml"),
        os.path.expanduser("~/.local/share/kokoro"),
    ]
    ml_env_python = os.path.expanduser("~/.openclaw/tools/ml-env/bin/python")
    
    for loc in legacy_locations:
        if not os.path.exists(loc):
            continue
        for python_path, source in [(ml_env_python, "ml-env"), (sys.executable, "system")]:
            if not os.path.exists(python_path):
                continue
            try:
                result = subprocess.run(
                    [python_path, "-c", "from kokoro import KPipeline; print('ok')"],
                    capture_output=True, timeout=10, cwd=loc
                )
                if result.returncode == 0:
                    _save_kokoro_cache(python_path, loc, source)
                    return python_path, loc
            except (subprocess.SubprocessError, OSError, TimeoutError):
                pass
    
    return None, None


def _save_kokoro_cache(python_path: str, kokoro_dir: str, source: str):
    """Cache found Kokoro setup."""
    try:
        config = load_config()
        if "kokoro" not in config:
            config["kokoro"] = {}
        config["kokoro"]["python"] = "system" if source == "system" else python_path
        config["kokoro"]["path"] = kokoro_dir
        config["kokoro"]["source"] = source
        save_config(config)
    except (IOError, KeyError, TypeError):
        pass  # Non-critical: cache save failed, will retry next time


def generate_kokoro_audio(text: str, output_path: str, audio_format: str, config: dict) -> str:
    """Generate audio using Kokoro TTS."""
    try:
        wav_path = output_path.replace('.mp3', '.wav') if output_path.endswith('.mp3') else output_path
        if not wav_path.endswith('.wav'):
            wav_path = output_path + '.wav'

        # Find Kokoro installation
        kokoro_python, kokoro_dir = find_kokoro()

        if not kokoro_python:
            raise Exception("Kokoro not found. Run: python setup.py")

        # Read voice blend and speed from config
        kokoro_config = config.get("kokoro", {})
        voice_blend = kokoro_config.get("voice_blend", {"af_heart": 0.6, "af_sky": 0.4})
        speed = kokoro_config.get("speed", 1.05)

        # Safely escape values for Python string using json.dumps
        safe_text = json.dumps(text)
        safe_wav_path = json.dumps(wav_path)
        safe_voice_blend = json.dumps(voice_blend)
        safe_speed = json.dumps(speed)

        code = f'''
from kokoro import KPipeline
import soundfile as sf
import torch
import numpy as np
import os
import json

pipeline = KPipeline(lang_code='a')

voice_blend = json.loads({safe_voice_blend!r})
speed = {safe_speed}

# Try to load custom voice blend, fall back to default
try:
    cache_dir = os.path.expanduser('~/.cache/huggingface/hub/models--hexgrad--Kokoro-82M/snapshots')
    snapshot = os.listdir(cache_dir)[0]
    voices_dir = os.path.join(cache_dir, snapshot, 'voices')
    voices = {{}}
    for name in voice_blend:
        voices[name] = torch.load(os.path.join(voices_dir, f'{{name}}.pt'), weights_only=True)
    voice = sum(weight * voices[name] for name, weight in voice_blend.items())
except Exception:
    voice = list(voice_blend.keys())[0] if voice_blend else 'af_heart'

text = {safe_text}
wav_path = {safe_wav_path}
audio_chunks = []
for _, _, audio in pipeline(text, voice=voice, speed=speed):
    audio_chunks.append(audio)
full_audio = np.concatenate(audio_chunks)
sf.write(wav_path, full_audio, 24000)
print("OK")
'''
        # Run from kokoro directory so local module is found
        # cwd=kokoro_dir for legacy cloned repos where kokoro is a local module
        run_kwargs = {"capture_output": True, "text": True, "timeout": 300}
        if os.path.exists(os.path.join(kokoro_dir, "kokoro", "__init__.py")):
            run_kwargs["cwd"] = kokoro_dir
        result = subprocess.run([kokoro_python, "-c", code], **run_kwargs)

        if result.returncode == 0 and os.path.exists(wav_path):
            if audio_format == "mp3":
                base = output_path[:-4] if output_path.endswith(('.mp3', '.wav')) else output_path
                mp3_path = base + '.mp3'
                subprocess.run(["ffmpeg", "-y", "-i", wav_path, "-b:a", "128k", mp3_path],
                             capture_output=True, check=True)
                os.remove(wav_path)
                return mp3_path
            return wav_path
        else:
            raise Exception(f"Kokoro failed: {result.stderr}")

    except Exception as e:
        print(f"   Kokoro failed, falling back to built-in TTS: {e}")
        return generate_builtin_audio(text, output_path, audio_format)


def generate_builtin_audio(text: str, output_path: str, audio_format: str) -> str:
    """Generate audio using macOS say command (fallback)."""
    wav_path = output_path.replace('.mp3', '.wav') if output_path.endswith('.mp3') else output_path
    if not wav_path.endswith('.wav'):
        wav_path = output_path + '.wav'

    # Write text to temp file to avoid CLI argument length limits
    with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as tf:
        tf.write(text)
        text_file = tf.name
    try:
        subprocess.run(["say", "-o", wav_path, "--data-format=LEI16@22050", "-f", text_file],
                      check=True, capture_output=True)
    except FileNotFoundError:
        os.remove(text_file)
        raise FileNotFoundError(
            "macOS 'say' command not found. Built-in TTS is only available on macOS. "
            "Install Kokoro TTS for cross-platform support: python setup.py"
        )
    finally:
        if os.path.exists(text_file):
            os.remove(text_file)

    if audio_format == "mp3":
        base = output_path[:-4] if output_path.endswith(('.mp3', '.wav')) else output_path
        mp3_path = base + '.mp3'
        try:
            subprocess.run(["ffmpeg", "-y", "-i", wav_path, "-b:a", "128k", mp3_path],
                         capture_output=True, check=True)
            os.remove(wav_path)
            return mp3_path
        except FileNotFoundError:
            print("   ffmpeg not found, keeping wav format")
            return wav_path

    return wav_path


def process_single_url(url: str, config: dict, quiet: bool = False) -> tuple[str | None, str | None, str | None]:
    """
    Process a single URL.
    Returns: (source_path, output_path, error_message)
    Temp files go to /tmp, final output uses config folder.
    """
    source_path, output_path, error = prepare_source_data(url, _get_temp_dir(), quiet)
    
    if error:
        if not quiet:
            print(f"\n{error}")
        return None, None, error
    
    return source_path, output_path, None


def main():
    import argparse
    
    parser = argparse.ArgumentParser(
        description="TubeScribe - YouTube Video Summarizer",
        epilog="Examples:\n"
               "  tubescribe https://youtube.com/watch?v=xxx\n"
               "  tubescribe url1 url2 url3  # Batch processing\n"
               "  tubescribe --queue-status  # Check queue\n",
        formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("--version", action="version", version=f"TubeScribe {__version__}")
    parser.add_argument("urls", nargs="*", help="YouTube URL(s)")
    parser.add_argument("--output-dir", default=None, help="Output directory (default: from config)")
    parser.add_argument("--doc-format", choices=["html", "docx", "md"], help="Document format")
    parser.add_argument("--audio-format", choices=["wav", "mp3", "none"], help="Audio format")
    parser.add_argument("--no-audio", action="store_true", help="Skip audio generation")
    parser.add_argument("--quiet", action="store_true", help="Minimal output")
    # Queue management
    parser.add_argument("--queue-add", metavar="URL", help="Add URL to queue")
    parser.add_argument("--queue-status", action="store_true", help="Show queue status")
    parser.add_argument("--queue-clear", action="store_true", help="Clear the queue")
    parser.add_argument("--queue-next", action="store_true", help="Process next item in queue")
    # Cleanup
    parser.add_argument("--cleanup", metavar="VIDEO_ID", help="Remove temp files for video ID")
    # Audio generation from text
    parser.add_argument("--generate-audio", metavar="TEXT_FILE", help="Generate audio from a text file (reads file, uses TTS config)")
    parser.add_argument("--audio-output", metavar="PATH", help="Output path for generated audio (used with --generate-audio)")
    
    args = parser.parse_args()
    
    config = load_config()
    
    # Override config with args
    if args.doc_format:
        config["document"]["format"] = args.doc_format
    if args.audio_format:
        config["audio"]["format"] = args.audio_format
    if args.no_audio:
        config["audio"]["enabled"] = False

    # Use config output folder if not specified
    output_dir = args.output_dir or os.path.expanduser(config.get("output", {}).get("folder", "~/Documents/TubeScribe"))
    os.makedirs(output_dir, exist_ok=True)
    
    # Audio generation from text file
    if args.generate_audio:
        text_file = args.generate_audio
        if not os.path.exists(text_file):
            print(f"❌ File not found: {text_file}")
            sys.exit(1)
        with open(text_file, 'r') as f:
            text = f.read().strip()
        if not text:
            print(f"❌ File is empty: {text_file}")
            sys.exit(1)
        audio_output = args.audio_output or os.path.join(output_dir, "summary")
        try:
            result = generate_audio_summary(text, audio_output, config)
        except Exception as e:
            print(f"❌ Audio generation failed: {e}")
            sys.exit(1)
        if result:
            print(f"✅ Audio: {result}")
        else:
            print("❌ Audio generation failed")
            sys.exit(1)
        return

    # Cleanup operation
    if args.cleanup:
        if cleanup_temp_files(args.cleanup, args.quiet):
            print(f"✅ Cleaned up temp files for {args.cleanup}")
        else:
            print(f"ℹ️  No temp files found for {args.cleanup}")
        return
    
    # Queue operations
    if args.queue_status:
        show_queue_status()
        return
    if args.queue_clear:
        clear_queue()
        return
    if args.queue_add:
        add_to_queue(args.queue_add)
        return
    if args.queue_next:
        url = pop_from_queue()
        if url:
            args.urls = [url]
        else:
            print("📋 Queue is empty")
            return
    
    # Need at least one URL
    if not args.urls:
        parser.print_help()
        return
    
    # Process URLs
    results = []
    for i, url in enumerate(args.urls):
        if len(args.urls) > 1:
            print(f"\n{'═' * 50}")
            print(f"📹 Video {i+1}/{len(args.urls)}")
            print(f"{'═' * 50}")
        
        source_path, output_path, error = process_single_url(
            url, config, args.quiet
        )
        
        if error:
            results.append({"url": url, "error": error})
        else:
            results.append({"url": url, "source": source_path, "output": output_path})
    
    # Summary for batch
    if len(args.urls) > 1:
        print(f"\n{'═' * 50}")
        print(f"📊 Batch Summary")
        print(f"{'═' * 50}")
        success = [r for r in results if "source" in r]
        failed = [r for r in results if "error" in r]
        print(f"   ✅ Success: {len(success)}")
        print(f"   ❌ Failed: {len(failed)}")
        if failed:
            print(f"\n   Failed videos:")
            for r in failed:
                print(f"      • {r['url'][:50]}...")
                print(f"        {r['error']}")
    
    # Print next steps for successful extractions
    successful = [r for r in results if "source" in r]
    if successful:
        print(f"\n📋 Next: Process with AI agent")
        for r in successful:
            print(f"   • {r['source']}")


# ═══════════════════════════════════════════════════════════════════════════════
# Queue Management
# ═══════════════════════════════════════════════════════════════════════════════

QUEUE_FILE = os.path.expanduser("~/.tubescribe/queue.json")


@contextmanager
def _locked_queue():
    """Context manager for atomic queue read-modify-write with file locking."""
    os.makedirs(os.path.dirname(QUEUE_FILE), exist_ok=True)
    fd = os.open(QUEUE_FILE, os.O_RDWR | os.O_CREAT, 0o644)
    try:
        if _HAS_FCNTL:
            fcntl.flock(fd, fcntl.LOCK_EX)
        elif not getattr(_locked_queue, '_warned', False):
            print("Warning: File locking unavailable on this platform, queue operations may race", file=sys.stderr)
            _locked_queue._warned = True
        content = b''
        while True:
            chunk = os.read(fd, 4096)
            if not chunk:
                break
            content += chunk
        try:
            data = json.loads(content.decode('utf-8')) if content.strip() else {"current": None, "queue": []}
        except json.JSONDecodeError:
            data = {"current": None, "queue": []}

        yield data

        # Write back (only reached if no exception in with-block)
        os.ftruncate(fd, 0)
        os.lseek(fd, 0, os.SEEK_SET)
        os.write(fd, json.dumps(data, indent=2).encode('utf-8'))
    finally:
        os.close(fd)


def load_queue() -> dict:
    """Load queue from file (read-only, no locking)."""
    if os.path.exists(QUEUE_FILE):
        try:
            with open(QUEUE_FILE, 'r') as f:
                return json.load(f)
        except (json.JSONDecodeError, IOError, OSError):
            pass
    return {"current": None, "queue": []}


def add_to_queue(url: str, title: str = None) -> int:
    """Add URL to queue. Returns position (1-indexed)."""
    if not is_youtube_url(url):
        print(ERROR_MESSAGES["invalid_url"])
        return 0

    with _locked_queue() as data:
        entry = {
            "url": url,
            "title": title,
            "added": datetime.now().astimezone().isoformat()
        }
        data["queue"].append(entry)
        position = len(data["queue"])
        if data["current"]:
            position += 1

    print(f"📋 Added to queue (position {position})")
    return position


def pop_from_queue() -> str | None:
    """Get next URL from queue."""
    with _locked_queue() as data:
        if not data["queue"]:
            return None
        entry = data["queue"].pop(0)
        data["current"] = entry
        return entry["url"]


def clear_current():
    """Clear current processing item."""
    with _locked_queue() as data:
        data["current"] = None


def clear_queue():
    """Clear entire queue."""
    with _locked_queue() as data:
        data["current"] = None
        data["queue"] = []
    print("📋 Queue cleared")


def show_queue_status():
    """Show queue status."""
    data = load_queue()

    print("📋 TubeScribe Queue")
    print("─" * 40)

    if data["current"]:
        title = data["current"].get("title") or data["current"]["url"][:50]
        print(f"▶️  Processing: {title}")
    else:
        print("▶️  Processing: (none)")

    if data["queue"]:
        print(f"\n📝 Queued ({len(data['queue'])}):")
        for i, entry in enumerate(data["queue"], 1):
            title = entry.get("title") or entry["url"][:50]
            print(f"   {i}. {title}")
    else:
        print("\n📝 Queue: (empty)")


def clear_stale_current(config: dict = None) -> bool:
    """Clear the current processing item if it is stale. Returns True if cleared."""
    if config is None:
        config = load_config()
    stale_minutes = config.get("queue", {}).get("stale_minutes", 30)

    with _locked_queue() as data:
        if not data["current"]:
            return False
        try:
            added_str = data["current"].get("added", "")
            if added_str:
                added = datetime.fromisoformat(added_str.replace("Z", "+00:00"))
                now = datetime.now().astimezone()
                if added.tzinfo is None:
                    added = added.astimezone()
                if now - added > timedelta(minutes=stale_minutes):
                    data["current"] = None
                    return True
        except (ValueError, TypeError):
            pass
    return False


def is_processing(config: dict = None) -> bool:
    """Check if currently processing. Clears stale entries first."""
    clear_stale_current(config)
    data = load_queue()
    return data["current"] is not None


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)