Skill: ioridev/youtube-transcript-fetcher (version 0.1.3)

--- name: youtube-transcript-fetcher description: Fetch full YouTube transcripts with robust InnerTube fallback. Use when extracting captions/transcripts from a YouTube URL, channel, or batch config, especially when normal transcript tools fail, uploader subtitles are missing, or YouTube auto-generated captions may still be recoverable. metadata: openclaw: requires: bins: ["python3", "yt-dlp"] install: - id: apt-yt-dlp kind: apt package: yt-dlp bins: ["yt-dlp"] label: Install yt-dlp (apt) - id: brew-yt-dlp kind: brew formula: yt-dlp bins: ["yt-dlp"] label: Install yt-dlp (brew) --- Fetch transcripts from YouTube videos. Prefer this skill when the goal is to obtain source transcript text, not to summarize. Run `./youtube-transcript-fetcher --url <youtube-url>` for a single video. Run `./youtube-transcript-fetcher --channel <channel-id-or-handle> --hours <n>` for recent channel videos. Run `./youtube-transcript-fetcher --config config/channels.example.json --daily` for batch mode. The main script is `scripts/youtube_transcript_fetcher.py`. It uses watch-page scraping plus InnerTube player API fallback with multiple client profiles (`ANDROID`, `WEB`, `TVHTML5_SIMPLY_EMBEDDED_PLAYER`, `IOS`) to recover caption tracks that simpler methods miss, including cases where only YouTube auto-generated captions are available. Install Python dependencies with `pip install -r requirements.txt` before first use. Return the transcript text from the generated JSON output. Do not rely on summary fields because this skill is intentionally transcript-first.

path: README.md
content: # youtube-transcript-fetcher

`youtube-transcript-fetcher` is an OpenClaw skill and CLI for extracting full YouTube transcripts reliably.

Instead of generating a short summary first, it returns the transcript text itself. That means OpenClaw can read the full content of the video directly and decide for itself how to analyze, summarize, translate, quote, or structure it.

## Why this exists

A lot of YouTube transcript tools fail on videos where normal caption fetching breaks, or only work when the uploader has explicitly provided subtitle tracks.

`youtube-transcript-fetcher` takes a more resilient path:

watch page scrape -> `INNERTUBE_API_KEY` extraction -> InnerTube player API fallback across multiple client profiles -> caption XML download -> transcript text extraction

This makes it much better at recovering transcripts from videos where simpler libraries return `No transcript available`.

## Main selling points

It is built around three ideas.

First, it fetches the transcript itself, not just a summary. Because there is no summary layer in the middle, OpenClaw can inspect the entire spoken content and keep full context.

Second, it can often recover YouTube's automatic captions even when manually attached subtitles are not present. In practice, this is the big win. A video may look like it has no uploader-provided subtitle set, but YouTube auto-generated captions are still available through the InnerTube caption path.

Third, it outputs plain JSON with the transcript included, so it is easy to plug into other OpenClaw flows.

## What it can fetch

`youtube-transcript-fetcher` can extract transcripts from:

- a single YouTube video URL
- a recent set of videos from a channel or handle
- a batch config for repeated runs

When transcript extraction succeeds, the JSON output includes a `transcript` field containing the full extracted text.

## How transcript recovery works

The core recovery flow is:

1. Fetch the YouTube watch page HTML
2. Extract `INNERTUBE_API_KEY`
3. Call `youtubei/v1/player`
4. Try multiple client identities in order:
   - `ANDROID`
   - `WEB`
   - `TVHTML5_SIMPLY_EMBEDDED_PLAYER`
   - `IOS`
5. Read caption track metadata from the player response
6. Download the caption XML from the selected `baseUrl`
7. Parse the XML into transcript text

This fallback approach is based on the same family of techniques used to recover captions in cases where normal transcript APIs are flaky.

## Why no summary is a feature

A summary throws information away.

If the goal is to let OpenClaw understand a video well, returning only a summary is weaker than returning the transcript. With `youtube-transcript-fetcher`, OpenClaw can read the source material directly, which is better for:

- detailed analysis
- extracting exact claims or quotes
- translation
- structured output generation
- custom summaries tailored to the user's task
- reusing the transcript in other tools or agents

This is also what makes the output flexible downstream. Once you have the transcript, you can summarize it, translate it, extract quotes, or turn it into structured data later with whatever model and prompt you want.

So this project is intentionally transcript-first, not summary-first.

## Usage

Single video:

```bash
./youtube-transcript-fetcher --url "https://www.youtube.com/watch?v=VIDEO_ID"
```

It also accepts raw video IDs, `youtu.be/...`, and `youtube.com/shorts/...` URLs.

Recent videos from a channel or handle:

```bash
./youtube-transcript-fetcher --channel "@channel_handle" --hours 24
```

Batch mode with config:

```bash
./youtube-transcript-fetcher --config config/channels.example.json --daily
```

You can also write output to a specific file:

```bash
./youtube-transcript-fetcher --url "https://www.youtube.com/watch?v=VIDEO_ID" --output /tmp/youtube_transcript_fetcher.json
```

## Output format

The tool writes JSON like this:

```json
{
  "generated_at": "2026-04-09T04:00:00+00:00",
  "items": [
    {
      "video_id": "...",
      "title": "...",
      "url": "...",
      "channel": "...",
      "duration": "12:34",
      "published": "20260408",
      "has_transcript": true,
      "metadata": {
        "view_count": 12345,
        "like_count": 678
      },
      "transcript": "full transcript text here"
    }
  ],
  "stats": {
    "total_videos": 1,
    "with_transcript": 1,
    "without_transcript": 0
  }
}
```

If transcript extraction fails, `has_transcript` becomes `false` and an error message is included.

## Requirements

This project depends on:

- `python3`
- `yt-dlp`
- Python packages from `requirements.txt`

Quick setup example:

```bash
pip install -r requirements.txt
```

Install `yt-dlp` with your package manager, for example:

```bash
brew install yt-dlp
```

or:

```bash
sudo apt install yt-dlp
```

## Add it to OpenClaw

This repository is structured as an OpenClaw skill.

The simplest way is to clone or copy this repository into your OpenClaw skills directory.

Example:

```bash
cd ~/clawd/skills
git clone https://github.com/ioridev/youtube-transcript-fetcher.git
```

After that, OpenClaw can discover the skill from the `SKILL.md` metadata.

If you just want to run it directly, use the bash entrypoint:

```bash
./youtube-transcript-fetcher
```

The Python implementation lives at:

```bash
scripts/youtube_transcript_fetcher.py
```

Do not run `python youtube-transcript-fetcher`, because `youtube-transcript-fetcher` itself is a shell wrapper.

Typical OpenClaw-side use is: fetch the transcript first, then let OpenClaw read the transcript text directly for analysis, translation, extraction, or summarization.

## OpenClaw usage idea

A good pattern is:

1. run `youtube-transcript-fetcher` on the target video
2. get JSON containing `transcript`
3. pass that transcript to OpenClaw as source material

That keeps the workflow transcript-first and avoids losing information through a premature summary layer.

## Practical note on auto captions

One of the main reasons to use this project is that uploader-provided subtitles are not required.

If YouTube has generated automatic captions for the video, `youtube-transcript-fetcher` can often recover them even when ordinary transcript methods fail. That is the core value of the project.

## Security note

This project fetches caption data directly from YouTube. It does not route caption downloads through a third-party proxy.

## Git tag based ClawHub release flow

This repository is set up so Git tags can be used as the publish version.

The GitHub Actions workflow watches tags like `v0.1.1` and publishes that exact version to ClawHub.

Example:

```bash
git tag v0.1.1
git push origin v0.1.1
```

That will publish ClawHub version `0.1.1`.

Before this works, add this repository secret in GitHub:

- `CLAWDHUB_TOKEN`: your ClawHub API token used by `clawdhub login --token ...`

You can also trigger the workflow manually with `workflow_dispatch` and pass a version and changelog.

## License

See `LICENSE`.


path: config/channels.example.json
content: {
  "channels": [
    {
      "name": "All About AI",
      "id": "UCyFqFYfTW2VoIQKylJ04Rtw",
      "url": "https://www.youtube.com/@AllAboutAI"
    },
    {
      "name": "Two Minute Papers",
      "id": "UCbfYPyITQ-7l4upoX8nvctg",
      "url": "https://www.youtube.com/@TwoMinutePapers"
    },
    {
      "name": "Lex Fridman",
      "id": "UCSHZKyawb77ixDdsGog4iWA",
      "url": "https://www.youtube.com/@lexfridman"
    }
  ],
  "hours_lookback": 24,
  "min_duration_seconds": 300,
  "max_videos_per_channel": 5
}


path: requirements.txt
content: requests>=2.31.0
youtube-transcript-api>=0.6.2


path: scripts/youtube_transcript_fetcher.py
content: #!/usr/bin/env python3
"""YouTube Transcript Fetcher, transcript extraction tool."""

import re
import sys
import json
import argparse
import subprocess
from datetime import datetime, timezone
from typing import Optional, List, Dict
from pathlib import Path
from urllib.parse import urlparse, parse_qs

# Default config
DEFAULT_MIN_DURATION = 300  # 5 minutes (filter Shorts)
DEFAULT_HOURS_LOOKBACK = 24
DEFAULT_MAX_VIDEOS_PER_CHANNEL = 5
DEFAULT_OUTPUT = "/tmp/youtube_transcript_fetcher.json"


def get_channel_videos(channel_id: str, hours: int, max_videos: int) -> List[Dict]:
    """Get recent videos from a YouTube channel using yt-dlp"""
    videos = []
    
    # Build channel URL
    if channel_id.startswith("UC") and len(channel_id) == 24:
        url = f"https://www.youtube.com/channel/{channel_id}/videos"
    elif channel_id.startswith("http"):
        url = channel_id.rstrip("/") + "/videos"
    else:
        url = f"https://www.youtube.com/@{channel_id}/videos"
    
    try:
        result = subprocess.run(
            [
                "yt-dlp",
                "--flat-playlist",
                "--no-warnings",
                "-J",
                "--playlist-end", str(max_videos * 2),
                url,
            ],
            capture_output=True,
            text=True,
            timeout=45,
        )
        
        if result.returncode != 0:
            print(f"⚠️ yt-dlp error for {channel_id}: {result.stderr[:100]}", file=sys.stderr)
            return []
        
        data = json.loads(result.stdout)
        entries = data.get("entries", [])
        
        for entry in entries:
            if not entry:
                continue
            
            video_id = entry.get("id")
            if not video_id:
                continue
            
            # Filter Shorts by duration
            if entry.get("duration") and entry.get("duration") < DEFAULT_MIN_DURATION:
                continue
            
            videos.append({
                "id": video_id,
                "title": entry.get("title", "Unknown"),
                "channel": entry.get("channel", entry.get("uploader", "Unknown")),
                "url": f"https://www.youtube.com/watch?v={video_id}",
                "duration_hint": entry.get("duration"),
            })
            
            if len(videos) >= max_videos:
                break
        
    except Exception as e:
        print(f"⚠️ Error fetching channel {channel_id}: {e}", file=sys.stderr)
    
    return videos


def get_video_details(video_id: str) -> Optional[Dict]:
    """Get detailed video metadata using yt-dlp"""
    try:
        result = subprocess.run(
            [
                "yt-dlp",
                "--no-warnings",
                "-j",
                "--no-download",
                f"https://www.youtube.com/watch?v={video_id}",
            ],
            capture_output=True,
            text=True,
            timeout=20,
        )
        
        if result.returncode != 0:
            return None
        
        data = json.loads(result.stdout)
        duration = data.get("duration", 0)
        
        return {
            "duration_seconds": duration,
            "duration": f"{duration // 60}:{duration % 60:02d}",
            "description": data.get("description", "")[:1000],
            "published": data.get("upload_date", ""),
            "view_count": data.get("view_count", 0),
            "like_count": data.get("like_count", 0),
        }
        
    except Exception:
        return None


def get_transcript(video_id: str) -> Optional[str]:
    """Get video transcript using multiple methods."""
    transcript = _get_transcript_innertube_proxy(video_id)
    if transcript:
        return transcript
    
    # Method 2: youtube-transcript-api (fallback, may be rate limited)
    transcript = _get_transcript_ytapi(video_id)
    if transcript:
        return transcript
    
    return None


def _parse_caption_xml(xml_text: str) -> List[str]:
    """Parse YouTube caption XML (supports multiple formats)"""
    import xml.etree.ElementTree as ET
    import html as html_mod
    
    try:
        root = ET.fromstring(xml_text)
        texts = []
        
        # Try <p> tags first (format 3 and format 2)
        for p in root.findall('.//p'):
            # Check for 
 child tags (format 3: word-level)
            words = []
            for s in p.findall('s'):
                if s.text:
                    words.append(html_mod.unescape(s.text.strip()))
            if words:
                texts.append(' '.join(words))
            elif p.text:  # format 2: direct text
                texts.append(html_mod.unescape(p.text.strip()))
        
        # If no <p> found, try <text> tags (format 1)
        if not texts:
            for elem in root.findall('.//text'):
                if elem.text:
                    texts.append(html_mod.unescape(elem.text.strip()))
        
        return texts
    except Exception:
        return []


def _download_caption(url: str) -> Optional[str]:
    """Download caption content directly from YouTube."""
    import requests

    try:
        r = requests.get(url, timeout=15)
        if r.status_code == 200 and r.text.strip():
            return r.text
    except Exception:
        pass
    
    return None


def _extract_innertube_api_key(html_text: str) -> Optional[str]:
    match = re.search(r'"INNERTUBE_API_KEY":"([^"]+)"', html_text)
    return match.group(1) if match else None


def _fetch_video_page(video_id: str) -> Optional[str]:
    import requests

    try:
        response = requests.get(
            f"https://www.youtube.com/watch?v={video_id}",
            headers={
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Safari/537.36"
            },
            timeout=15,
        )
        if response.status_code == 200 and response.text:
            return response.text
    except Exception:
        pass
    return None


def _fetch_caption_tracks_with_clients(video_id: str, api_key: str) -> List[Dict]:
    import requests

    url = f"https://www.youtube.com/youtubei/v1/player?key={api_key}"
    client_attempts = [
        {
            "name": "ANDROID",
            "user_agent": "com.google.android.youtube/21.02.35 (Linux; U; Android 11) gzip",
            "client": {
                "hl": "en",
                "gl": "US",
                "clientName": "ANDROID",
                "clientVersion": "21.02.35",
                "androidSdkVersion": 30,
                "osName": "Android",
                "osVersion": "11",
                "platform": "MOBILE",
            },
        },
        {
            "name": "WEB",
            "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Safari/537.36",
            "client": {
                "hl": "en",
                "gl": "US",
                "clientName": "WEB",
                "clientVersion": "2.20240619.06.00",
            },
        },
        {
            "name": "TVHTML5_SIMPLY_EMBEDDED_PLAYER",
            "user_agent": "Mozilla/5.0 (Chromecast; Android TV 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.5993.99 Safari/537.36",
            "client": {
                "hl": "en",
                "gl": "US",
                "clientName": "TVHTML5_SIMPLY_EMBEDDED_PLAYER",
                "clientVersion": "2.0",
                "clientScreen": "EMBED",
                "platform": "TV",
                "deviceMake": "Google",
                "deviceModel": "Chromecast",
                "osName": "Android",
                "osVersion": "10",
            },
        },
        {
            "name": "IOS",
            "user_agent": "com.google.ios.youtube/19.09.3 (iPhone; CPU iPhone OS 17_5 like Mac OS X)",
            "client": {
                "hl": "en",
                "gl": "US",
                "clientName": "IOS",
                "clientVersion": "19.09.3",
                "deviceMake": "Apple",
                "deviceModel": "iPhone16,2",
                "osName": "iOS",
                "osVersion": "17.5",
            },
        },
    ]

    for attempt in client_attempts:
        try:
            response = requests.post(
                url,
                headers={
                    "User-Agent": attempt["user_agent"],
                    "Content-Type": "application/json",
                },
                json={
                    "videoId": video_id,
                    "context": {"client": attempt["client"]},
                    "contentCheckOk": True,
                    "racyCheckOk": True,
                },
                timeout=15,
            )
            if response.status_code != 200:
                continue
            data = response.json()
            captions = data.get("captions")
            if not captions:
                continue
            tracks = captions.get("playerCaptionsTracklistRenderer", {}).get("captionTracks") or []
            if tracks:
                return [{**track, "fetchUserAgent": attempt["user_agent"]} for track in tracks]
        except Exception:
            continue
    return []


def _pick_caption_track(tracks: List[Dict]) -> Optional[Dict]:
    if not tracks:
        return None
    for prefer in ["en", "zh-Hans", "zh-Hant", "zh"]:
        for track in tracks:
            code = track.get("languageCode")
            if code == prefer:
                return track
    return tracks[0]


def _get_transcript_innertube_proxy(video_id: str) -> Optional[str]:
    """Method 1: watch-page scrape + InnerTube multi-client fallback + caption XML"""
    try:
        video_page_html = _fetch_video_page(video_id)
        if not video_page_html:
            return None

        [REDACTED_SECRET])
        if not [REDACTED_SECRET] None

        caption_tracks = _fetch_caption_tracks_with_clients(video_id, api_key)
        selected_track = _pick_caption_track(caption_tracks)
        if not selected_track:
            return None

        cap_url = selected_track.get('baseUrl')
        if not cap_url:
            return None

        xml_text = _download_caption(cap_url)
        if not xml_text:
            return None
        
        texts = _parse_caption_xml(xml_text)
        if not texts:
            return None
        
        result = ' '.join(texts).strip()
        return result if len(result) > 50 else None
        
    except Exception:
        return None


def _get_transcript_ytapi(video_id: str) -> Optional[str]:
    """Method 2 (fallback): youtube-transcript-api direct connection"""
    try:
        from youtube_transcript_api import YouTubeTranscriptApi
        
        api = YouTubeTranscriptApi()
        fetched = api.fetch(video_id, languages=["zh-Hans", "zh-Hant", "en"])
        transcript = " ".join([item["text"] for item in fetched])
        return transcript if len(transcript) > 50 else None
        
    except Exception:
        return None


def extract_video_id(url_or_id: str) -> Optional[str]:
    """Extract a YouTube video ID from a video URL or raw ID."""
    value = url_or_id.strip()
    if re.fullmatch(r"[A-Za-z0-9_-]{11}", value):
        return value

    parsed = urlparse(value)
    host = parsed.netloc.lower()
    path = parsed.path.strip("/")

    if host in {"youtu.be", "www.youtu.be"}:
        candidate = path.split("/")[0] if path else ""
        return candidate if re.fullmatch(r"[A-Za-z0-9_-]{11}", candidate) else None

    if host in {"youtube.com", "www.youtube.com", "m.youtube.com", "music.youtube.com"}:
        query_id = parse_qs(parsed.query).get("v", [None])[0]
        if query_id and re.fullmatch(r"[A-Za-z0-9_-]{11}", query_id):
            return query_id

        parts = [part for part in path.split("/") if part]
        if len(parts) >= 2 and parts[0] in {"shorts", "embed", "live", "watch"}:
            candidate = parts[1] if parts[0] != "watch" else None
            if candidate and re.fullmatch(r"[A-Za-z0-9_-]{11}", candidate):
                return candidate

    return None


def process_video(video_id: str, title: str = None, channel: str = None) -> Dict:
    """Process a single video and return transcript-oriented JSON."""
    print(f"📹 Processing: {video_id}")
    
    # Get video details
    details = get_video_details(video_id)
    if not details:
        return {
            "video_id": video_id,
            "title": title or "Unknown",
            "url": f"https://www.youtube.com/watch?v={video_id}",
            "error": "Failed to fetch video details"
        }
    
    # Get transcript
    transcript = get_transcript(video_id)
    has_transcript = transcript is not None
    
    result = {
        "video_id": video_id,
        "title": title or "Unknown",
        "url": f"https://www.youtube.com/watch?v={video_id}",
        "channel": channel or "Unknown",
        "duration": details["duration"],
        "published": details["published"],
        "has_transcript": has_transcript,
        "metadata": {
            "view_count": details.get("view_count", 0),
            "like_count": details.get("like_count", 0),
        },
        "transcript": transcript if has_transcript else None,
    }
    
    if has_transcript:
        print(f"  ✅ Transcript: {len(transcript)} chars")
    else:
        result["error"] = "No transcript available"
        print(f"  ⚠️ No transcript available")
    
    return result


def main():
    parser = argparse.ArgumentParser(description="YouTube Transcript Fetcher")
    parser.add_argument("--url", help="Single YouTube URL or video ID")
    parser.add_argument("--channel", help="Channel ID or handle")
    parser.add_argument("--config", help="Config file path (JSON)")
    parser.add_argument("--daily", action="store_true", help="Daily batch mode (requires --config)")
    parser.add_argument("--hours", type=int, default=DEFAULT_HOURS_LOOKBACK, help="Hours to look back")
    parser.add_argument("--output", default=DEFAULT_OUTPUT, help="Output JSON file")
    
    args = parser.parse_args()
    
    results = {
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "items": [],
        "stats": {
            "total_videos": 0,
            "with_transcript": 0,
            "without_transcript": 0
        }
    }
    
    # Mode 1: Single video
    if args.url:
        video_id = extract_video_id(args.url)
        if not video_id:
            print("⚠️ Could not extract a valid YouTube video ID from --url", file=sys.stderr)
            sys.exit(2)
        result = process_video(video_id)
        results["items"].append(result)
        results["stats"]["total_videos"] = 1
        if result.get("has_transcript"):
            results["stats"]["with_transcript"] = 1
        else:
            results["stats"]["without_transcript"] = 1
    
    # Mode 2: Channel scan
    elif args.channel:
        videos = get_channel_videos(args.channel, args.hours, DEFAULT_MAX_VIDEOS_PER_CHANNEL)
        print(f"📺 Found {len(videos)} videos from channel")
        
        for video in videos:
            result = process_video(video["id"], video["title"], video["channel"])
            results["items"].append(result)
            results["stats"]["total_videos"] += 1
            if result.get("has_transcript"):
                results["stats"]["with_transcript"] += 1
            else:
                results["stats"]["without_transcript"] += 1
    
    # Mode 3: Daily batch (config file)
    elif args.daily and args.config:
        with open(args.config, "r") as f:
            config = json.load(f)
        
        channels = config.get("channels", [])
        hours = config.get("hours_lookback", args.hours)
        max_videos = config.get("max_videos_per_channel", DEFAULT_MAX_VIDEOS_PER_CHANNEL)
        
        print(f"📺 Processing {len(channels)} channels")
        
        for ch in channels:
            channel_id = ch.get("id") or ch.get("url")
            channel_name = ch.get("name", "Unknown")
            
            print(f"\n🔍 Channel: {channel_name}")
            videos = get_channel_videos(channel_id, hours, max_videos)
            print(f"  Found {len(videos)} videos")
            
            for video in videos:
                result = process_video(video["id"], video["title"], channel_name)
                results["items"].append(result)
                results["stats"]["total_videos"] += 1
                if result.get("has_transcript"):
                    results["stats"]["with_transcript"] += 1
                else:
                    results["stats"]["without_transcript"] += 1
    
    else:
        parser.print_help()
        sys.exit(1)
    
    # Write output
    output_path = Path(args.output)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    
    with open(output_path, "w") as f:
        json.dump(results, f, indent=2, ensure_ascii=False)
    
    print(f"\n✅ Output written to: {output_path}")
    print(f"📊 Stats: {results['stats']}")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)