Skill: champagne315/follow-aibuilders (version 1.0.0)

--- name: follow-builders description: AI builders digest — 监控顶级 AI builders 在 X 和 YouTube 播客上的动态，将其内容重新整理为易于消化的摘要。当用户想要 AI 行业洞察、builder 更新或调用 /ai 时使用。Twitter 使用免费 Rettiwt-API（无需付费 key），YouTube 使用 yt-dlp（免费）。 --- # Follow Builders, Not Influencers 你是一个 AI 驱动的内容策展工具，追踪 AI 领域的顶级 builders —— 那些真正在构建产品、运营公司和进行研究的人 —— 并提供他们观点的精炼摘要。 理念：追随有原创观点的 builders，而不是只会复述的 influencers。 **这个 skill 完全在你的本地机器上运行，使用免费工具。** 你对信息源和数据隐私有完全控制。 ## 首次运行 —— Onboarding **先检查 onboarding 状态：** ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js check-onboarding ``` 如果 `onboardingComplete` 为 `false`，运行以下 onboarding 流程： ### Step 1: 依赖检查 告诉用户： "Follow Builders 完全在你的本地机器上运行，不需要付费 API。 **系统要求：** - **yt-dlp**（YouTube 视频列表和字幕获取）— 必须安装 - macOS: `brew install yt-dlp` - Linux: `pip install yt-dlp` - Windows: `pip install yt-dlp` 或从 https://github.com/yt-dlp/yt-dlp/releases 下载 - **Node.js 22+**（Rettiwt-API 依赖要求） - **Twitter/X**：开箱即用，使用免费 guest 认证（无需 API key） **可选：Twitter 增强认证** 如果 Twitter guest 认证被限流（返回速率限制错误），可以配置 user 认证： 1. 安装 Chrome 扩展 [X Auth Helper](https://chromewebstore.google.com/detail/x-auth-helper/miflinhihdjikdpjjigaiafeoobkhikk) 2. 在隐身模式登录 Twitter/X 3. 点击扩展获取 API_KEY 4. 使用 CLI 保存：" ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js setup-env [rettiwt_api_key] ``` 如果用户没有 API key（大多数情况），直接运行不带参数： ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js setup-env ``` 自动检测 yt-dlp 是否已安装： ```bash yt-dlp --version ``` 如果未安装，指导用户安装后再继续。 如果 yt-dlp 报错"Sign in to confirm you're not a bot"，需要配置 cookies： 在 `.env` 中添加： ``` YT_DLP_COOKIES=chrome ``` （支持 chrome、firefox、edge 等浏览器名，或 cookies.txt 文件路径） ### Step 2: 介绍 告诉用户： "我是你的 AI Builders Digest。我追踪 AI 领域的顶级 builders —— 正在构建产品的研究人员、创始人、产品经理和工程师 —— 跨越 X/Twitter 和 YouTube 播客。我可以为你提供他们观点、想法和工作的精选摘要。" ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js get-default-sources ``` 从输出中读取源数量，然后告诉用户： "我目前默认追踪 [N] 个 X 上的 builders、[M] 个播客和 [B] 个博客。你可以随时自定义这个列表。 默认情况下，我监控过去 24 小时内的新内容。你可以随时调整这个时间窗口。" ### Step 3: 语言 询问："你偏好什么语言的摘要？" - English - Chinese（从英文源翻译） - Bilingual（英文和中文并列显示） ### Step 4: 时间窗口 询问："你想监控多长时间内的内容？默认是过去 24 小时。" 选项： - 24 小时（默认） - 48 小时 - 72 小时 - 1 周（168 小时） 解释："时间窗口越长，每次摘要可能包含更多内容，但也需要更多时间处理。" ### Step 5: 信息源 显示正在追踪的默认 builders 和播客的完整列表。 ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js list-sources ``` 从输出的 `defaultSources` 中显示为清晰的列表。 告诉用户："你当前使用的是默认源列表。 **自定义选项：** - '添加 Twitter 账号 @username' —— 添加新的信息源 - '移除 @username' —— 禁用某个信息源 - '列出我的信息源' —— 显示当前所有源 - '重置为默认源' —— 恢复默认列表 - '仅使用我的自定义源' —— 切换到完全自定义模式 - '合并默认源和自定义源' —— 合并模式（默认） 你可以随时通过对话修改信息源。" ### Step 6: 配置提醒 "你的所有设置都可以随时通过对话更改： - '让摘要更简短' - '显示我的当前设置' - '添加/移除信息源' - '调整时间窗口为 48 小时' 无需编辑任何文件 —— 只需告诉我你想要什么。" ### Step 7: 保存配置 使用 CLI 工具初始化配置： ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js init-config <language> <lookback_hours> ``` `language` 为 `en`、`zh` 或 `bilingual` `lookback_hours` 为 24、48、72 或 168 ### Step 8: 欢迎摘要 **不要跳过这一步。** 立即生成并向用户发送他们的第一个摘要，让他们看看效果。 告诉用户："让我获取今天的内容并立即生成一个示例摘要。这大约需要一分钟。" 然后立即运行下面的完整内容交付工作流（Steps 1-6），不要等待 cron job。 交付摘要后，询问反馈： "这是你的第一个 AI Builders Digest！几个问题： - 长度是否合适，还是你希望摘要更短/更长？ - 有什么你希望我更多（或更少）关注的吗？ - 你想添加或移除任何信息源吗？ 只需告诉我，我会调整。随时输入 /ai 获取下一个摘要。" 等待他们的回应并应用任何反馈。然后确认更改。 --- ## 内容交付 —— 摘要运行 ### Step 1: 加载配置 ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js get-config ``` ### Step 2: 运行 fetch-and-prepare 脚本 此脚本以确定的方式处理所有数据获取 —— API 调用、去重、prompts、配置。你不需要自己获取任何东西。 ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node fetch-and-prepare.js ``` 脚本会自动将数据保存到 `digest-data.json`。 如果脚本完全失败（返回 error status），检查： - yt-dlp 是否已安装？运行 `yt-dlp --version` 验证 - 是否有网络连接？ - 如果 yt-dlp 报错"not a bot"，需要配置 YT_DLP_COOKIES（见 Step 1） - 如果 Twitter 被限流（guest auth rate limited），考虑配置 RETTIWT_API_KEY（见 Step 1） ### Step 3: 读取数据文件 使用 Read 工具读取临时数据文件： ``` 数据文件路径：${CLAUDE_SKILL_DIR}/digest-data.json ``` JSON 结构包含： - `config` —— 用户的语言和交付偏好 - `podcasts` —— 带完整转录的播客剧集 - `x` —— builders 及其最近的 tweets（文本、URLs、bios） - `blogs` —— 带完整内容的博客文章 - `prompts` —— 要遵循的重新整理指令 - `stats` —— 剧集、tweets 和博客文章的计数 - `errors` —— 非致命问题（忽略这些） ### Step 4: 检查内容 如果 `stats.podcastEpisodes` 为 0 且 `stats.xBuilders` 为 0 且 `stats.blogPosts` 为 0， 告诉用户："今天你的 builders 没有新更新。明天再来查看！"然后停止。 ### Step 5: 重新整理内容 **你唯一的工作是从 JSON 重新整理内容。** 不要从网络获取任何东西、访问任何 URLs 或调用任何 APIs。。一切都在 JSON 中。 JSON 的 `prompts` 字段已包含所有提示词的完整内容： - `prompts.digest_intro` —— 整体框架规则 - `prompts.summarize_podcast` —— 如何重新整理播客转录 - `prompts.summarize_tweets` —— 如何重新整理 tweets - `prompts.summarize_blogs` —— 如何重新整理博客文章 - `prompts.translate` —— 如何翻译为中文 **Tweets：** `x` 数组包含带 tweets 的 builders。一次处理一个： 1. 使用他们的 `bio` 字段作为他们的角色（例如 bio 说 "ceo @box" → "Box CEO Aaron Levie"） 2. 按照 `prompts.summarize_tweets` 汇总他们的 tweets 3. 输出格式（每个 builder 一个块）： ``` **Box CEO Aaron Levie** - Summarized point 1... - Summarized point 2... https://x.com/levie/status/123 ``` **Podcasts：** `podcasts` 数组包含带转录的剧集。 1. 按照 `prompts.summarize_podcast` 汇总转录 2. 输出格式： ``` **Podcast Name — Episode Title** - Key insight 1... - Key insight 2... https://youtube.com/watch?v=xyz ``` **Blogs：** `blogs` 数组包含带内容的文章。 1. 按照 `prompts.summarize_blogs` 汇总文章 2. 输出格式： ``` **Blog Name — Article Title** - Key point 1... - Key point 2... https://example.com/article ``` **用 `prompts.digest_intro` 框架包装摘要**在顶部。 ### Step 6: 应用语言设置 从 JSON 读取 `config.language`： **如果是 "en"：** 仅输出英文（默认） **如果是 "zh"：** 按照 `prompts.translate` 将整个摘要翻译为中文 **如果是 "bilingual"：** 对每个部分，先输出英文，然后是中文翻译： ``` **Box CEO Aaron Levie** - AI agents will fundamentally reshape software procurement... https://x.com/levie/status/123 **Box CEO Aaron Levie** - Box CEO Aaron Levie 认为 AI agent 将从根本上重塑软件采购... https://x.com/levie/status/123 **Replit CEO Amjad Masad launched Agent 4...** https://x.com/amasad/status/456 **Replit CEO Amjad Masad 发布了 Agent 4...** https://x.com/amasad/status/456 ``` 不要先输出所有英文然后所有中文。交替输出它们。 **完全遵循此设置。不要混合语言。** ### Step 7: 输出摘要 直接输出重新整理后的摘要。 --- ## 信息源管理 用户可以通过对话自定义他们的信息源。 ### 查看当前源 "列出我的信息源" 或 "show my sources" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js list-sources ``` 显示所有源（合并默认+用户）和当前模式。 ### 添加源 "添加 Twitter 账号 @username" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js add-source x <name> <handle> ``` "添加播客频道 @channel" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js add-source podcast <name> <youtube_channel|youtube_playlist> <url> <id> ``` "添加博客 https://..." → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js add-source blog <name> <indexUrl> [articleBaseUrl] ``` ### 移除/禁用源 "移除 @username" 或 "disable @username" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js remove-source x <handle> ``` 或临时禁用： ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js toggle-source x <handle> false ``` ### 切换模式 "仅使用我的自定义源" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js set-mode replace ``` "合并默认源和自定义源" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js set-mode merge ``` ### 重置为默认 "重置为默认源" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js reset-sources ``` ### 用户源文件格式 所有配置统一在 `config/config.json` 中： ```json { "language": "en", "sourceMode": "merge", "lookbackHours": 24, "sources": { "x_accounts": [ { "name": "Account Name", "handle": "username", "enabled": true } ], "podcasts": [ { "name": "My Podcast", "type": "youtube_channel", "url": "https://youtube.com/@mypodcast", "channelHandle": "mypodcast", "enabled": true } ], "blogs": [ { "name": "My Blog", "type": "scrape", "indexUrl": "https://blog.example.com", "articleBaseUrl": "https://blog.example.com/posts/", "enabled": true } ] } } ``` - `sourceMode: "merge"` —— 使用 config.json 中的源列表（默认） - `sourceMode: "replace"` —— 同上（源列表即为你自定义的内容） - `enabled: false` —— 临时禁用而不删除 --- ## 配置处理 当用户说听起来像设置更改的话时，处理它： ### 语言更改 - "切换到中文/英文/双语" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js set-config language <en|zh|bilingual> ``` ### 时间窗口更改 - "调整时间窗口为 X 小时" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js set-config lookbackHours <hours> ``` - "我想看更长时间的内容" → 建议增加时间窗口（如从 24h 到 48h 或 72h） - "只看最近的内容" → 减少时间窗口（如从 72h 到 24h） 示例对话： ``` 用户: "调整时间窗口为 48 小时" 助手: "好的，已将时间窗口更新为 48 小时。现在每次摘要将包含过去 48 小时内的新内容。" ``` ### Prompt 更改 当用户想要自定义摘要的风格时，直接编辑 `prompts/` 目录下的 prompt 文件。 使用 CLI 设置新内容： ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js set-prompt <filename> <new_content> ``` 可用的 prompt 文件： - `summarize-podcast.md` — 播客摘要方式 - `summarize-tweets.md` — Twitter 推文摘要方式 - `summarize-blogs.md` — 博客文章摘要方式 - `digest-intro.md` — 整体摘要格式和语气 - `translate.md` — 英文内容如何翻译成中文 对话示例： - "让摘要更短/更长" → 编辑 `summarize-tweets.md` 或 `summarize-podcast.md` - "更多关注 [X]" → 编辑相关的 prompt 文件 - "将语气改为 [X]" → 编辑相关的 prompt 文件 - "重置为默认" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js reset-prompt <filename> ``` ### 信息请求 - "显示我的设置" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js get-config ``` - "显示我的源" / "我正在关注谁？" → ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js list-sources ``` - "显示我的 prompts" → 使用 CLI 获取所有 prompt 内容： ```bash cd ${CLAUDE_SKILL_DIR}/scripts && node cli.js get-prompt summarize-podcast ``` 任何配置更改后，确认你更改了什么。 ---

path: config/config.json
content: {
  "language": "en",
  "sourceMode": "merge",
  "lookbackHours": 24,
  "onboardingComplete": false,
  "sources": {
    "podcasts": [
      {
        "name": "Latent Space",
        "type": "youtube_channel",
        "url": "https://www.youtube.com/@LatentSpacePod",
        "channelHandle": "LatentSpacePod"
      },
      {
        "name": "Training Data",
        "type": "youtube_playlist",
        "url": "https://www.youtube.com/playlist?list=PLOhHNjZItNnMm5tdW61JpnyxeYH5NDDx8",
        "playlistId": [REDACTED_SECRET]
      },
      {
        "name": "No Priors",
        "type": "youtube_channel",
        "url": "https://www.youtube.com/@NoPriorsPodcast",
        "channelHandle": "NoPriorsPodcast"
      },
      {
        "name": "Unsupervised Learning",
        "type": "youtube_channel",
        "url": "https://www.youtube.com/@RedpointAI",
        "channelHandle": "RedpointAI"
      },
      {
        "name": "Data Driven NYC",
        "type": "youtube_channel",
        "url": "https://www.youtube.com/@DataDrivenNYC/videos",
        "channelHandle": "DataDrivenNYC"
      }
    ],
    "blogs": [
      {
        "name": "Anthropic Engineering",
        "type": "scrape",
        "indexUrl": "https://www.anthropic.com/engineering",
        "articleBaseUrl": "https://www.anthropic.com/engineering/",
        "fetchMethod": "http"
      },
      {
        "name": "Claude Blog",
        "type": "scrape",
        "indexUrl": "https://claude.com/blog",
        "articleBaseUrl": "https://claude.com/blog/",
        "fetchMethod": "http"
      }
    ],
    "x_accounts": [
      { "name": "Andrej Karpathy", "handle": "karpathy" },
      { "name": "Swyx", "handle": "swyx" },
      { "name": "Josh Woodward", "handle": "joshwoodward" },
      { "name": "Kevin Weil", "handle": "kevinweil" },
      { "name": "Peter Yang", "handle": "petergyang" },
      { "name": "Nan Yu", "handle": "thenanyu" },
      { "name": "Madhu Guru", "handle": "realmadhuguru" },
      { "name": "Amanda Askell", "handle": "AmandaAskell" },
      { "name": "Cat Wu", "handle": "_catwu" },
      { "name": "Thariq", "handle": "trq212" },
      { "name": "Google Labs", "handle": "GoogleLabs" },
      { "name": "Amjad Masad", "handle": "amasad" },
      { "name": "Guillermo Rauch", "handle": "rauchg" },
      { "name": "Alex Albert", "handle": "alexalbert__" },
      { "name": "Aaron Levie", "handle": "levie" },
      { "name": "Ryo Lu", "handle": "ryolu_" },
      { "name": "Garry Tan", "handle": "garrytan" },
      { "name": "Matt Turck", "handle": "mattturck" },
      { "name": "Zara Zhang", "handle": "zarazhangrui" },
      { "name": "Nikunj Kothari", "handle": "nikunj" },
      { "name": "Peter Steinberger", "handle": "steipete" },
      { "name": "Dan Shipper", "handle": "danshipper" },
      { "name": "Aditya Agarwal", "handle": "adityaag" },
      { "name": "Sam Altman", "handle": "sama" },
      { "name": "Claude", "handle": "claudeai" }
    ]
  }
}


path: prompts/digest-intro.md
content: # Digest Intro Prompt

You are assembling the final digest from individual source summaries.

## Format

Start with this header (replace [Date] with today's date):

AI Builders Digest — [Date]

Then organize content in this order:

1. X / TWITTER section — list each builder with new posts
2. OFFICIAL BLOGS section — list each blog post from AI company blogs (OpenAI, Anthropic, etc.)
3. PODCASTS section — list each podcast with new episodes

## Rules

- Only include sources that have new content
- Skip any source with nothing new
- Under each source, paste the individual summary you generated

### Podcast links
- After each podcast summary, include the specific video URL from the JSON `url` field
  (e.g. https://youtube.com/watch?v=Iu4gEnZFQz8)
- NEVER link to the channel page. Always link to the specific video.
- Include the exact episode title from the JSON `title` field in the heading

### Tweet author formatting
- Use the author's full name and role/company, not just their last name
  (e.g. "Box CEO Aaron Levie" not "Levie")
- NEVER write Twitter handles with @ in the digest. On Telegram, @handle becomes
  a clickable link to a Telegram user, which is wrong. Instead write handles
  without @ (e.g. "Aaron Levie (levie on X)" or just use their full name)
- Include the direct link to each tweet from the JSON `url` field

### Blog post formatting
- Use the blog name as a section header (e.g. "Anthropic Engineering", "OpenAI News", "Claude Blog")
- Under each blog, list each new post with its title and summary
- Include the author name if available
- Include the direct link to the original article

### Mandatory links
- Every single piece of content MUST have an original source link
- Blog posts: the direct article URL (e.g. https://www.anthropic.com/engineering/...)
- Podcasts: the YouTube video URL (e.g. https://youtube.com/watch?v=xxx)
- Tweets: the direct tweet URL (e.g. https://x.com/levie/status/xxx)
- If you don't have a link for something, do NOT include it in the digest.
  No link = not real = do not include.

### No fabrication
- Only include content that came from the feed JSON (blogs, podcasts, and tweets)
- NEVER make up quotes, opinions, or content you think someone might have said
- NEVER speculate about someone's silence or what they might be working on
- If you have nothing real for a builder, skip them entirely

### General
- At the very end, add a line: "Generated through the Follow Builders skill: https://github.com/champagne315/follow-aibuilders"
- Keep formatting clean and scannable — this will be read on a phone screen


path: prompts/summarize-blogs.md
content: # Blog Post Summary Prompt

You are summarizing a blog post from an AI company (OpenAI, Anthropic, etc.) for a busy
professional who wants the key announcements and insights without reading the full article.

## Instructions

- Start with the blog name and article title (e.g. "Anthropic Engineering: Harness Design for Long-Running Apps")
- Write a summary of 100-300 words depending on article length and substance
- Lead with what matters: the core announcement, finding, or insight
- If the post introduces a new product, feature, or research finding, name it clearly
- If there are specific numbers, benchmarks, or results, include them
- Include at least one direct quote from the article if available
- If the post has practical implications (e.g. new API, new capability, policy change), call them out explicitly
- Keep the tone sharp and informative — like a smart colleague forwarding you the key points
- Do NOT include filler like "In this blog post..." or "The author discusses..."
- Jump straight into the substance
- Include the direct link to the original article


path: prompts/summarize-podcast.md
content: # Podcast Remix Prompt

You are remixing a podcast episode transcript for a busy professional who wants
the key insights without watching the full episode.

## Instructions

- Write a remix of 200-400 words
- Start with a one-sentence "The Takeaway" — what's the single most important takeaway?
- Introduce the context and the speaker's information (name, role/company, background) and why the audience should care
- Prioritizes insights that are counterintuitive, contrarian, or refreshingly specific to the speaker's experience. Avoid generic wisdom
- Include at least one direct quote from the source that captures (find the most memorable quote)
- Stands alone as a complete piece — avoids references like "this interview," "this video," "in this conversation," "the host asks," or "in this episode." Write as if distilling lessons from a person's philosophy, not summarizing a specific piece of content
- Assume your audience is curious adults who are not specialized experts. If the original source contains specialized knowledge that only experts in a field would understand, translate it into language understandable to a general audience
- Keep the tone sharp and conversational — like a smart friend briefing you
- Do NOT include filler like "In this episode..." or "The host and guest discussed..."
- Jump straight into the substance


path: prompts/summarize-tweets.md
content: # X/Twitter Summary Prompt

You are summarizing recent posts from an AI builder for a busy professional who wants
to know what this person is thinking and building.

## Instructions

- Start by introducing the author with their full name AND role/company
  (e.g. "Replit CEO Amjad Masad", "Box CEO Aaron Levie", "a]6z partner Justine Moore")
  Do NOT use just their last name. Do NOT use their Twitter handle with @.
- Only include substantive content: original opinions, insights, product announcements,
  technical discussions, industry analysis, or lessons learned
- SKIP: mundane personal tweets, retweets without commentary, promotional content,
  "great event!" type posts, engagement bait
- For threads: summarize the full thread as one cohesive piece, not individual tweets
- For quote tweets: include the context of what they're responding to
- Write 2-4 sentences per builder summarizing their key points
- If they made a bold prediction or shared a contrarian take, lead with that
- If they shared a tool, demo, or resource, mention it by name with the link
- If there's nothing substantive to report, say "No notable posts" rather than
  padding with fluff


path: prompts/translate.md
content: # Translation Prompt

You are translating an AI industry digest from English to Chinese.

## Instructions

- Translate the full digest into natural, fluent Mandarin Chinese (simplified characters). The translated version must sound like it was originally written in Chinese, instead of translated
- Keep technical terms in English where Chinese professionals typically use them:
  AI, LLM, GPU, API, fine-tuning, RAG, token, prompt, agent, transformer, etc.
- Keep all proper nouns in English: names of people, companies, products, tools
- Keep all URLs unchanged
- Maintain the same structure and formatting as the English version
- The tone should be professional but conversational — 像是一位懂行的朋友在跟你聊天
- For bilingual mode: interleave English and Chinese paragraph by paragraph.
  After each builder's English summary, place the Chinese translation directly below
  (separated by a blank line), then move to the next builder. Same for podcasts.
  Do NOT output all English first then all Chinese.
- Never use em-dashes


path: scripts/cli.js
content: #!/usr/bin/env node

// ============================================================================
// Follow Builders CLI Tool
// ============================================================================
// Unified command-line interface for all Follow Builders operations.
//
// Usage: node cli.js <command> [args...]
// ============================================================================

import { readFile, writeFile, mkdir, unlink } from 'fs/promises';
import { existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';

const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
const PROJECT_DIR = join(SCRIPT_DIR, '..');
const CONFIG_PATH = join(PROJECT_DIR, 'config', 'config.json');
const STATE_PATH = join(PROJECT_DIR, 'state-feed.json');
const ENV_PATH = join(PROJECT_DIR, '.env');
const PROMPTS_DIR = join(PROJECT_DIR, 'prompts');

const CONFIG_SCHEMA = {
  language: { enum: ['en', 'zh', 'bilingual'], default: 'en' },
  sourceMode: { enum: ['merge', 'replace'], default: 'merge' },
  lookbackHours: { type: 'number', min: 1, max: 168, default: 24 },
  onboardingComplete: { type: 'boolean', default: false }
};

// ============================================================================
// Utility Functions
// ============================================================================

function success(data) {
  console.log(JSON.stringify({ status: 'ok', ...data }, null, 2));
  process.exit(0);
}

function error(message, code = 1) {
  console.error(JSON.stringify({ status: 'error', message }, null, 2));
  process.exit(code);
}

async function ensureDir(dir) {
  if (!existsSync(dir)) {
    await mkdir(dir, { recursive: true });
  }
}

async function readJSON(path) {
  try {
    const content = await readFile(path, 'utf-8');
    return JSON.parse(content);
  } catch (err) {
    if (existsSync(path)) {
      error(`Failed to parse JSON at ${path}: ${err.message}`);
    }
    return null;
  }
}

async function writeJSON(path, data) {
  await ensureDir(dirname(path));
  await writeFile(path, JSON.stringify(data, null, 2));
}

async function loadConfig() {
  const config = await readJSON(CONFIG_PATH);
  if (!config) {
    error(`Config file not found: ${CONFIG_PATH}`);
  }
  return config;
}

async function saveConfig(config) {
  await writeJSON(CONFIG_PATH, config);
}

function validateConfigKey(key, value) {
  const schema = CONFIG_SCHEMA[key];
  if (!schema) {
    error(`Invalid config key: ${key}`);
  }

  if (schema.enum && !schema.enum.includes(value)) {
    error(`Invalid value for ${key}: must be one of ${schema.enum.join(', ')}`);
  }

  if (schema.type === 'number') {
    const num = Number(value);
    if (isNaN(num)) {
      error(`Invalid number value for ${key}`);
    }
    if (schema.min && num < schema.min) {
      error(`Value for ${key} must be at least ${schema.min}`);
    }
    if (schema.max && num > schema.max) {
      error(`Value for ${key} must be at most ${schema.max}`);
    }
    return num;
  }

  if (schema.type === 'boolean') {
    return value === 'true' || value === true;
  }

  return value;
}

// ============================================================================
// Commands: Onboarding
// ============================================================================

async function cmdSetupEnv() {
  const args = process.argv.slice(3);

  const rettiwtApiKey = args[0] || '';

  if (rettiwtApiKey) {
    const envContent = `# Optional: Rettiwt API key for Twitter user auth (guest auth works without this)
RETTIWT_API_KEY=${rettiwtApiKey}
`;
    await writeFile(ENV_PATH, envContent);
    success({ message: 'Rettiwt API key saved. Twitter will use user auth.', path: ENV_PATH });
  } else {
    const envContent = `# Follow Builders - no API keys required
# Optionally add RETTIWT_API_KEY for Twitter user auth if guest auth is rate limited
# FB_PROXY=http://127.0.0.1:7897
# YT_DLP_COOKIES=chrome
`;
    await writeFile(ENV_PATH, envContent);
    success({ message: 'Environment file created. Twitter will use guest auth (no key needed). YouTube requires yt-dlp installed.', path: ENV_PATH });
  }
}

async function cmdInitConfig() {
  const args = process.argv.slice(3);

  const language = args[0] || 'en';
  const lookbackHours = args[1] ? Number(args[1]) : 24;

  if (!CONFIG_SCHEMA.language.enum.includes(language)) {
    error(`Invalid language: must be one of ${CONFIG_SCHEMA.language.enum.join(', ')}`);
  }

  if (isNaN(lookbackHours) || lookbackHours < 1 || lookbackHours > 168) {
    error('Invalid lookbackHours: must be a number between 1 and 168');
  }

  const config = await loadConfig();
  config.language = language;
  config.lookbackHours = lookbackHours;
  config.onboardingComplete = true;

  await saveConfig(config);
  success({ message: 'Config initialized successfully', config });
}

async function cmdCheckOnboarding() {
  const config = await readJSON(CONFIG_PATH);
  const envExists = existsSync(ENV_PATH);

  const configExists = config !== null;
  const isComplete = config?.onboardingComplete === true;

  success({
    onboardingComplete: isComplete,
    envExists,
    configExists,
    config: config || null
  });
}

// ============================================================================
// Commands: Config Management
// ============================================================================

async function cmdGetConfig() {
  const config = await loadConfig();
  success({ config });
}

async function cmdSetConfig() {
  const args = process.argv.slice(3);

  if (args.length < 2) {
    error('Usage: cli.js set-config <key> <value>');
  }

  const [key, value] = args;

  const config = await loadConfig();
  const validatedValue = validateConfigKey(key, value);
  config[key] = validatedValue;

  await saveConfig(config);
  success({ message: `Config updated`, key, value: validatedValue, config });
}

async function cmdUpdateConfig() {
  const args = process.argv.slice(3);

  if (args.length === 0) {
    error('Usage: cli.js update-config <json_string>');
  }

  const jsonStr = args[0];
  let updates;
  try {
    updates = JSON.parse(jsonStr);
  } catch (err) {
    error(`Invalid JSON: ${err.message}`);
  }

  const config = await loadConfig();

  for (const [key, value] of Object.entries(updates)) {
    const validatedValue = validateConfigKey(key, value);
    config[key] = validatedValue;
  }

  await saveConfig(config);
  success({ message: 'Config updated successfully', config });
}

// ============================================================================
// Commands: Source Management
// ============================================================================

async function cmdListSources() {
  const config = await loadConfig();
  success({
    mode: config.sourceMode || 'merge',
    sources: config.sources || {}
  });
}

async function cmdAddSource() {
  const args = process.argv.slice(3);

  if (args.length < 2) {
    error('Usage: cli.js add-source <type> <name> <...params>');
  }

  const [type, name, ...params] = args;
  const config = await loadConfig();
  if (!config.sources) config.sources = {};

  if (type === 'x') {
    if (params.length === 0) {
      error('Usage: cli.js add-source x <name> <handle>');
    }
    const handle = params[0];
    if (!config.sources.x_accounts) config.sources.x_accounts = [];

    // Check for duplicates
    const exists = config.sources.x_accounts.find(a => a.handle === handle);
    if (exists) {
      error(`Twitter account @${handle} already exists`);
    }

    config.sources.x_accounts.push({ name, handle, enabled: true });
  } else if (type === 'podcast') {
    if (params.length < 3) {
      error('Usage: cli.js add-source podcast <name> <channel_or_playlist> <url> <id>');
    }
    const [pType, url, id] = params;
    if (!config.sources.podcasts) config.sources.podcasts = [];

    if (!['youtube_channel', 'youtube_playlist'].includes(pType)) {
      error('Podcast type must be youtube_channel or youtube_playlist');
    }

    const podcast = { name, type: pType, url, enabled: true };
    if (pType === 'youtube_channel') {
      podcast.channelHandle = id;
    } else {
      podcast.playlistId = id;
    }

    config.sources.podcasts.push(podcast);
  } else if (type === 'blog') {
    if (params.length < 1) {
      error('Usage: cli.js add-source blog <name> <indexUrl> [articleBaseUrl]');
    }
    const [indexUrl, articleBaseUrl] = params;
    if (!config.sources.blogs) config.sources.blogs = [];

    const blog = {
      name,
      type: 'scrape',
      indexUrl,
      fetchMethod: 'http',
      enabled: true
    };

    if (articleBaseUrl) {
      blog.articleBaseUrl = articleBaseUrl;
    }

    config.sources.blogs.push(blog);
  } else {
    error(`Unknown source type: ${type}. Use x, podcast, or blog`);
  }

  await saveConfig(config);
  success({ message: 'Source added successfully', sources: config.sources });
}

async function cmdRemoveSource() {
  const args = process.argv.slice(3);

  if (args.length < 2) {
    error('Usage: cli.js remove-source <type> <id>');
  }

  const [type, id] = args;
  const config = await loadConfig();
  if (!config.sources) error('No sources configured');

  let removed = false;

  if (type === 'x') {
    if (!config.sources.x_accounts) error('No Twitter accounts configured');
    config.sources.x_accounts = config.sources.x_accounts.filter(a => {
      if (a.handle === id) {
        removed = true;
        return false;
      }
      return true;
    });
  } else if (type === 'podcast') {
    if (!config.sources.podcasts) error('No podcasts configured');
    config.sources.podcasts = config.sources.podcasts.filter(p => {
      const podcastId = p.channelHandle || p.playlistId;
      if (podcastId === id) {
        removed = true;
        return false;
      }
      return true;
    });
  } else if (type === 'blog') {
    if (!config.sources.blogs) error('No blogs configured');
    config.sources.blogs = config.sources.blogs.filter(b => {
      if (b.indexUrl === id || b.name === id) {
        removed = true;
        return false;
      }
      return true;
    });
  } else {
    error(`Unknown source type: ${type}. Use x, podcast, or blog`);
  }

  if (!removed) {
    error(`Source not found: ${type}/${id}`);
  }

  await saveConfig(config);
  success({ message: 'Source removed successfully', sources: config.sources });
}

async function cmdToggleSource() {
  const args = process.argv.slice(3);

  if (args.length < 3) {
    error('Usage: cli.js toggle-source <type> <id> <true|false>');
  }

  const [type, id, state] = args;
  const enabled = state === 'true' || state === true;

  const config = await loadConfig();
  if (!config.sources) error('No sources configured');

  let found = false;

  if (type === 'x') {
    if (!config.sources.x_accounts) error('No Twitter accounts configured');
    for (const account of config.sources.x_accounts) {
      if (account.handle === id) {
        account.enabled = enabled;
        found = true;
        break;
      }
    }
  } else if (type === 'podcast') {
    if (!config.sources.podcasts) error('No podcasts configured');
    for (const podcast of config.sources.podcasts) {
      const podcastId = podcast.channelHandle || podcast.playlistId;
      if (podcastId === id) {
        podcast.enabled = enabled;
        found = true;
        break;
      }
    }
  } else if (type === 'blog') {
    if (!config.sources.blogs) error('No blogs configured');
    for (const blog of config.sources.blogs) {
      if (blog.indexUrl === id || blog.name === id) {
        blog.enabled = enabled;
        found = true;
        break;
      }
    }
  } else {
    error(`Unknown source type: ${type}. Use x, podcast, or blog`);
  }

  if (!found) {
    error(`Source not found: ${type}/${id}`);
  }

  await saveConfig(config);
  success({ message: `Source ${enabled ? 'enabled' : 'disabled'}`, sources: config.sources });
}

async function cmdSetMode() {
  const args = process.argv.slice(3);

  if (args.length === 0) {
    error('Usage: cli.js set-mode <merge|replace>');
  }

  const mode = args[0];
  if (!['merge', 'replace'].includes(mode)) {
    error('Mode must be merge or replace');
  }

  const config = await loadConfig();
  config.sourceMode = mode;
  await saveConfig(config);
  success({ message: `Mode set to ${mode}`, config });
}

async function cmdResetSources() {
  const config = await loadConfig();
  // Reset sources to empty — user can re-add what they want
  config.sources = { x_accounts: [], podcasts: [], blogs: [] };
  await saveConfig(config);
  success({ message: 'Sources reset to empty. Add new sources or restore defaults from git.' });
}

// ============================================================================
// Commands: Prompt Management
// ============================================================================

async function cmdGetPrompt() {
  const args = process.argv.slice(3);

  if (args.length === 0) {
    error('Usage: cli.js get-prompt <name>');
  }

  const name = args[0];
  const filename = name.endsWith('.md') ? name : `${name}.md`;
  const localPath = join(PROMPTS_DIR, filename);

  if (!existsSync(localPath)) {
    error(`Prompt not found: ${name}`);
  }

  const content = await readFile(localPath, 'utf-8');
  success({ name, source: 'local', content });
}

async function cmdSetPrompt() {
  const args = process.argv.slice(3);

  if (args.length < 2) {
    error('Usage: cli.js set-prompt <name> <content>');
  }

  const [name, ...contentArgs] = args;
  const content = contentArgs.join(' ');
  const filename = name.endsWith('.md') ? name : `${name}.md`;
  const localPath = join(PROMPTS_DIR, filename);

  await writeFile(localPath, content);
  success({ message: 'Prompt saved successfully', path: localPath });
}

async function cmdResetPrompt() {
  const args = process.argv.slice(3);

  if (args.length === 0) {
    error('Usage: cli.js reset-prompt <name>');
  }

  success({ message: 'Prompt files are tracked in git. Use git checkout to restore defaults.', name: args[0] });
}

// ============================================================================
// Commands: Utilities
// ============================================================================

async function cmdGetStats() {
  const state = await readJSON(STATE_PATH);

  if (!state) {
    success({ message: 'No state file found' });
  }

  success({
    seenTweets: Object.keys(state.seenTweets || {}).length,
    seenVideos: Object.keys(state.seenVideos || {}).length,
    seenArticles: Object.keys(state.seenArticles || {}).length,
    state
  });
}

// ============================================================================
// Main
// ============================================================================

const commands = {
  // Onboarding
  'setup-env': cmdSetupEnv,
  'init-config': cmdInitConfig,
  'check-onboarding': cmdCheckOnboarding,

  // Config Management
  'get-config': cmdGetConfig,
  'set-config': cmdSetConfig,
  'update-config': cmdUpdateConfig,

  // Source Management
  'list-sources': cmdListSources,
  'add-source': cmdAddSource,
  'remove-source': cmdRemoveSource,
  'toggle-source': cmdToggleSource,
  'set-mode': cmdSetMode,
  'reset-sources': cmdResetSources,

  // Prompt Management
  'get-prompt': cmdGetPrompt,
  'set-prompt': cmdSetPrompt,
  'reset-prompt': cmdResetPrompt,

  // Utilities
  'get-stats': cmdGetStats,
  'help': () => {
    success({
      message: 'Follow Builders CLI Tool',
      commands: {
        onboarding: ['setup-env', 'init-config', 'check-onboarding'],
        config: ['get-config', 'set-config', 'update-config'],
        sources: ['list-sources', 'add-source', 'remove-source', 'toggle-source', 'set-mode', 'reset-sources'],
        prompts: ['get-prompt', 'set-prompt', 'reset-prompt'],
        utilities: ['get-stats']
      }
    });
  }
};

function main() {
  const args = process.argv.slice(2);

  if (args.length === 0) {
    error('Usage: node cli.js <command> [args...]\nRun "node cli.js help" for available commands');
  }

  const [cmdName, ...cmdArgs] = args;
  const cmd = commands[cmdName];

  if (!cmd) {
    error(`Unknown command: ${cmdName}\nRun "node cli.js help" for available commands`);
  }

  cmd().catch(err => {
    error(err.message);
  });
}

main();


path: scripts/fetch-and-prepare.js
content: #!/usr/bin/env node

// ============================================================================
// Follow Builders — Fetch and Prepare Digest
// ============================================================================
// Fetches content locally using free tools and prepares digest data:
// - Fetches from Twitter/X via Rettiwt-API (free, guest or user auth)
// - Fetches from YouTube via yt-dlp (free, no API key)
// - Scrapes blogs directly via HTTP
// - Manages user-customizable sources
// - Handles deduplication with per-user state
// - Loads prompts with priority (user > remote > local)
// - Outputs JSON for LLM processing
//
// Usage: node fetch-and-prepare.js
// Env vars (optional): RETTIWT_API_KEY (for Twitter user auth if guest is rate limited)
// System deps: yt-dlp (for YouTube video listing and transcripts)
// ============================================================================

import { readFile, writeFile, mkdir } from 'fs/promises';
import { existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import dotenv from 'dotenv';
import { Rettiwt } from 'rettiwt-api';
import { listChannelVideos, fetchTranscript, checkYtDlpAvailable } from './yt-dlp-helpers.js';

const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
const PROJECT_DIR = join(SCRIPT_DIR, '..');

// Load environment variables from project root .env
const envPath = join(PROJECT_DIR, '.env');
if (existsSync(envPath)) {
  dotenv.config({ path: envPath });
}

// -- Constants ---------------------------------------------------------------

const CONFIG_PATH = join(PROJECT_DIR, 'config', 'config.json');
const STATE_PATH = join(PROJECT_DIR, 'state-feed.json');

const DEFAULT_LOOKBACK_HOURS = 24;
const MAX_TWEETS_PER_USER = 3;
const MAX_ARTICLES_PER_BLOG = 3;

const PROMPT_FILES = [
  'summarize-podcast.md',
  'summarize-tweets.md',
  'summarize-blogs.md',
  'digest-intro.md',
  'translate.md'
];

// -- Config Loading ----------------------------------------------------------

async function loadConfig() {
  const config = JSON.parse(await readFile(CONFIG_PATH, 'utf-8'));
  return config;
}

// -- Dependency Validation ---------------------------------------------------

async function validateDependencies() {
  const errors = [];

  // Check yt-dlp
  const ytDlp = await checkYtDlpAvailable();
  if (!ytDlp.available) {
    errors.push(
      'yt-dlp 未安装。请安装后重试：\n' +
      '  macOS: brew install yt-dlp\n' +
      '  Linux/Windows: pip install yt-dlp\n' +
      '  或从 https://github.com/yt-dlp/yt-dlp/releases 下载'
    );
  } else {
    console.error(`  yt-dlp ${ytDlp.version} found`);
  }

  // Check Twitter connectivity via Rettiwt
  const rettiwtOptions = { delay: 1000 };
  if (process.env.RETTIWT_API_KEY) {
    rettiwtOptions.[REDACTED_SECRET];
    console.error('  Using Twitter user auth (RETTIWT_API_KEY)');
  } else {
    console.error('  Using Twitter guest auth (no API key)');
  }
  const proxyUrl = process.env.FB_PROXY;
  if (proxyUrl) {
    rettiwtOptions.proxyUrl = proxyUrl;
    console.error(`  Using proxy: ${proxyUrl}`);
  }

  try {
    const rettiwt = new Rettiwt(rettiwtOptions);
    await rettiwt.user.details('twitter');
    console.error('  Twitter API connectivity OK');
  } catch (err) {
    const msg = err.message || String(err);
    if (msg.includes('429') || msg.includes('rate') || msg.includes('Rate')) {
      errors.push(
        'Twitter API 速率限制中。Guest 认证配额已用完。\n' +
        '建议配置 user 认证：\n' +
        '  1. 安装 Chrome 扩展 X Auth Helper\n' +
        '  2. 在隐身模式登录 Twitter/X\n' +
        '  3. 点击扩展获取 API_KEY\n' +
        '  4. 在 .env 中添加 RETTIWT_API_KEY=<api_key>'
      );
    } else {
      errors.push(`Twitter API 连接失败: ${msg}`);
    }
  }

  return errors;
}

// -- State Management --------------------------------------------------------

async function loadState() {
  if (!existsSync(STATE_PATH)) {
    return { seenTweets: {}, seenVideos: {}, seenArticles: {} };
  }

  try {
    const state = JSON.parse(await readFile(STATE_PATH, 'utf-8'));
    if (!state.seenArticles) state.seenArticles = {};
    return state;
  } catch {
    return { seenTweets: {}, seenVideos: {}, seenArticles: {} };
  }
}

async function saveState(state) {
  // Prune entries older than 7 days
  const cutoff = Date.now() - 7 * 24 * 60 * 60 * 1000;
  for (const [id, ts] of Object.entries(state.seenTweets)) {
    if (ts < cutoff) delete state.seenTweets[id];
  }
  for (const [id, ts] of Object.entries(state.seenVideos)) {
    if (ts < cutoff) delete state.seenVideos[id];
  }
  for (const [id, ts] of Object.entries(state.seenArticles || {})) {
    if (ts < cutoff) delete state.seenArticles[id];
  }

  await writeFile(STATE_PATH, JSON.stringify(state, null, 2));
}

// -- X/Twitter Fetching (Rettiwt-API) ----------------------------------------

// Parse Twitter date format: "Mon Nov 18 06:35:11 +0000 2024"
function parseTwitterDate(dateStr) {
  return new Date(dateStr);
}

async function fetchXContent(xAccounts, state, lookbackHours, errors) {
  const results = [];
  const cutoff = new Date(Date.now() - lookbackHours * 60 * 60 * 1000);

  // Initialize Rettiwt with delay between requests
  const rettiwtOptions = { delay: 1000 };
  if (process.env.RETTIWT_API_KEY) {
    rettiwtOptions.[REDACTED_SECRET];
  }
  const proxyUrl = process.env.FB_PROXY;
  if (proxyUrl) {
    rettiwtOptions.proxyUrl = proxyUrl;
  }
  const rettiwt = new Rettiwt(rettiwtOptions);

  for (const account of xAccounts) {
    try {
      // 1. Fetch user details (accepts username)
      const userDetails = await rettiwt.user.details(account.handle);

      if (!userDetails) {
        errors.push(`X API: User @${account.handle} not found`);
        continue;
      }

      const userId = userDetails.id;
      const userName = userDetails.fullName || account.name;
      const userBio = userDetails.description || '';

      // 2. Fetch user timeline (requires user ID, not username)
      const timelineData = await rettiwt.user.timeline(userId);
      const tweets = timelineData?.list || [];

      // Filter tweets within time window, exclude retweets, limit to MAX_TWEETS_PER_USER
      const newTweets = [];

      for (const tweet of tweets) {
        // Skip retweets
        if (tweet.retweetedTweet) continue;

        // Parse and check time
        const createdAt = parseTwitterDate(tweet.createdAt);
        if (createdAt < cutoff) continue;

        // Limit to MAX_TWEETS_PER_USER
        if (newTweets.length >= MAX_TWEETS_PER_USER) break;

        // Extract tweet data
        newTweets.push({
          id: tweet.id,
          text: tweet.fullText || '',
          createdAt: tweet.createdAt,
          url: `https://x.com/${account.handle}/status/${tweet.id}`,
          likes: tweet.likeCount || 0,
          retweets: tweet.retweetCount || 0,
          replies: tweet.replyCount || 0,
          isQuote: !!tweet.quoted,
          quotedTweetId: tweet.quoted || null
        });
      }

      if (newTweets.length === 0) continue;

      results.push({
        source: 'x',
        name: userName,
        handle: account.handle,
        bio: userBio,
        tweets: newTweets
      });
    } catch (err) {
      const msg = err.message || String(err);
      if (msg.includes('429') || msg.includes('rate') || msg.includes('Rate')) {
        errors.push(`X API: Rate limited at @${account.handle}, skipping remaining accounts`);
        break;
      }
      errors.push(`X API: Error fetching @${account.handle}: ${msg}`);
    }
  }

  return results;
}

// -- YouTube Fetching (yt-dlp) -----------------------------------------------

async function fetchYouTubeContent(podcasts, state, lookbackHours, errors) {
  const cutoff = new Date(Date.now() - lookbackHours * 60 * 60 * 1000);
  const allCandidates = [];

  for (const podcast of podcasts) {
    console.error(`  Processing podcast: ${podcast.name}...`);
    const videos = await listChannelVideos(podcast, lookbackHours, errors);

    for (const video of videos) {
      // Filter by cutoff
      if (video.publishedAt && new Date(video.publishedAt) < cutoff) continue;

      allCandidates.push({
        podcast,
        videoId: video.videoId,
        title: video.title,
        publishedAt: video.publishedAt || new Date().toISOString()
      });
    }
  }

  // If no new videos within time window, return early
  if (allCandidates.length === 0) {
    console.error('  No new podcast episodes found');
    return [];
  }

  // Pick 1 unseen video (oldest first for chronological order)
  const selected = allCandidates.sort((a, b) => new Date(a.publishedAt) - new Date(b.publishedAt))[0];

  console.error(`  Fetching transcript for: ${selected.title}`);

  // Fetch transcript
  const transcript = await fetchTranscript(selected.videoId, errors);

  return [{
    source: 'podcast',
    name: selected.podcast.name,
    title: selected.title,
    videoId: selected.videoId,
    url: `https://youtube.com/watch?v=${selected.videoId}`,
    publishedAt: selected.publishedAt,
    transcript
  }];
}

// -- Blog Fetching (HTML scraping) -------------------------------------------

function parseAnthropicEngineeringIndex(html) {
  const articles = [];

  // Strategy 1: Look for article data in Next.js __NEXT_DATA__ script tag
  const nextDataMatch = html.match(/<script[^>]*id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/i);
  if (nextDataMatch) {
    try {
      const data = JSON.parse(nextDataMatch[1]);
      const pageProps = data?.props?.pageProps;
      const posts = pageProps?.posts || pageProps?.articles || pageProps?.entries || [];
      for (const post of posts) {
        const slug = post.slug?.current || post.slug || '';
        articles.push({
          title: post.title || 'Untitled',
          url: `https://www.anthropic.com/engineering/${slug}`,
          publishedAt: post.publishedOn || post.publishedAt || post.date || null,
          description: post.summary || post.description || ''
        });
      }
      if (articles.length > 0) return articles;
    } catch {
      // Fall through to regex approach
    }
  }

  // Strategy 2: Regex-based extraction
  const linkRegex = /href="\/engineering\/([a-z0-9-]+)"/gi;
  const seenSlugs = new Set();
  let linkMatch;
  while ((linkMatch = linkRegex.exec(html)) !== null) {
    const slug = linkMatch[1];
    if (seenSlugs.has(slug)) continue;
    seenSlugs.add(slug);
    articles.push({
      title: '',
      url: `https://www.anthropic.com/engineering/${slug}`,
      publishedAt: null,
      description: ''
    });
  }
  return articles;
}

function parseClaudeBlogIndex(html) {
  const articles = [];
  const seenSlugs = new Set();

  const linkRegex = /href="\/blog\/([a-z0-9-]+)"/gi;
  let linkMatch;
  while ((linkMatch = linkRegex.exec(html)) !== null) {
    const slug = linkMatch[1];
    if (seenSlugs.has(slug)) continue;
    seenSlugs.add(slug);
    articles.push({
      title: '',
      url: `https://claude.com/blog/${slug}`,
      publishedAt: null,
      description: ''
    });
  }
  return articles;
}

function extractAnthropicArticleContent(html) {
  let title = '';
  let author = '';
  let publishedAt = null;
  let content = '';

  const nextDataMatch = html.match(/<script[^>]*id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/i);
  if (nextDataMatch) {
    try {
      const data = JSON.parse(nextDataMatch[1]);
      const pageProps = data?.props?.pageProps;
      const post = pageProps?.post || pageProps?.article || pageProps?.entry || pageProps;
      title = post?.title || '';
      author = post?.author?.name || post?.authors?.[0]?.name || '';
      publishedAt = post?.publishedOn || post?.publishedAt || post?.date || null;

      const body = post?.body || post?.content || [];
      if (Array.isArray(body)) {
        const textParts = [];
        for (const block of body) {
          if (block._type === 'block' && block.children) {
            const text = block.children.map(c => c.text || '').join('');
            if (text.trim()) textParts.push(text.trim());
          }
        }
        content = textParts.join('\n\n');
      }
      if (content) return { title, author, publishedAt, content };
    } catch {
      // Fall through
    }
  }

  const h1Match = html.match(/<h1[^>]*>([\s\S]*?)<\/h1>/i);
  if (h1Match) title = h1Match[1].replace(/<[^>]+>/g, '').trim();

  const articleMatch = html.match(/<article[^>]*>([\s\S]*?)<\/article>/i);
  const bodyHtml = articleMatch ? articleMatch[1] : html;

  content = bodyHtml
    .replace(/<script[\s\S]*?<\/script>/gi, '')
    .replace(/<style[\s\S]*?<\/style>/gi, '')
    .replace(/<[^>]+>/g, ' ')
    .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>')
    .replace(/&quot;/g, '"').replace(/&#39;/g, "'").replace(/&nbsp;/g, ' ')
    .replace(/\s+/g, ' ')
    .trim();

  return { title, author, publishedAt, content };
}

function extractClaudeBlogArticleContent(html) {
  let title = '';
  let author = '';
  let publishedAt = null;
  let content = '';

  const jsonLdRegex = /<script[^>]*type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/gi;
  let jsonLdMatch;
  while ((jsonLdMatch = jsonLdRegex.exec(html)) !== null) {
    try {
      const ld = JSON.parse(jsonLdMatch[1]);
      if (ld['@type'] === 'BlogPosting' || ld['@type'] === 'Article') {
        title = ld.headline || ld.name || '';
        author = ld.author?.name || '';
        publishedAt = ld.datePublished || null;
        break;
      }
    } catch {
      // Skip
    }
  }

  const richTextMatch = html.match(/<div[^>]*class="[^"]*u-rich-text-blog[^"]*"[^>]*>([\s\S]*?)<\/div>\s*<\/div>/i)
    || html.match(/<div[^>]*class="[^"]*w-richtext[^"]*"[^>]*>([\s\S]*?)<\/div>/i);

  if (richTextMatch) {
    content = richTextMatch[1]
      .replace(/<script[\s\S]*?<\/script>/gi, '')
      .replace(/<style[\s\S]*?<\/style>/gi, '')
      .replace(/<[^>]+>/g, ' ')
      .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>')
      .replace(/&quot;/g, '"').replace(/&#39;/g, "'").replace(/&nbsp;/g, ' ')
      .replace(/\s+/g, ' ')
      .trim();
  }

  if (!content) {
    if (!title) {
      const h1Match = html.match(/<h1[^>]*>([\s\S]*?)<\/h1>/i);
      if (h1Match) title = h1Match[1].replace(/<[^>]+>/g, '').trim();
    }

    content = html
      .replace(/<script[\s\S]*?<\/script>/gi, '')
      .replace(/<style[\s\S]*?<\/style>/gi, '')
      .replace(/<nav[\s\S]*?<\/nav>/gi, '')
      .replace(/<footer[\s\S]*?<\/footer>/gi, '')
      .replace(/<header[\s\S]*?<\/header>/gi, '')
      .replace(/<[^>]+>/g, ' ')
      .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>')
      .replace(/&quot;/g, '"').replace(/&#39;/g, "'").replace(/&nbsp;/g, ' ')
      .replace(/\s+/g, ' ')
      .trim();
  }

  return { title, author, publishedAt, content };
}

async function fetchBlogContent(blogs, state, lookbackHours, errors) {
  const results = [];
  const cutoff = new Date(Date.now() - lookbackHours * 60 * 60 * 1000);

  for (const blog of blogs) {
    console.error(`  Processing blog: ${blog.name}...`);
    let candidates = [];

    try {
      const indexRes = await fetch(blog.indexUrl, {
        headers: { 'User-Agent': 'FollowBuilders/1.0 (feed aggregator)' }
      });
      if (!indexRes.ok) {
        errors.push(`Blog: Failed to fetch index for ${blog.name}: HTTP ${indexRes.status}`);
        continue;
      }
      const indexHtml = await indexRes.text();

      if (blog.indexUrl.includes('anthropic.com')) {
        candidates = parseAnthropicEngineeringIndex(indexHtml);
      } else if (blog.indexUrl.includes('claude.com')) {
        candidates = parseClaudeBlogIndex(indexHtml);
      }

      const MAX_INDEX_SCAN = MAX_ARTICLES_PER_BLOG;
      const toFetch = candidates.slice(0, MAX_INDEX_SCAN);

      if (toFetch.length === 0) {
        console.error(`    No articles found on index page`);
        continue;
      }

      console.error(`    Scanning ${toFetch.length} article(s) from index...`);

      for (const article of toFetch) {
        if (results.length >= MAX_ARTICLES_PER_BLOG) break;

        try {
          const articleRes = await fetch(article.url, {
            headers: { 'User-Agent': 'FollowBuilders/1.0 (feed aggregator)' }
          });
          if (!articleRes.ok) {
            errors.push(`Blog: Failed to fetch article ${article.url}: HTTP ${articleRes.status}`);
            continue;
          }
          const articleHtml = await articleRes.text();

          let extracted;
          if (article.url.includes('anthropic.com/engineering')) {
            extracted = extractAnthropicArticleContent(articleHtml);
          } else if (article.url.includes('claude.com/blog')) {
            extracted = extractClaudeBlogArticleContent(articleHtml);
          }

          if (!extracted || !extracted.content) {
            errors.push(`Blog: No content extracted from ${article.url}`);
            continue;
          }

          // Time filter: now we have the real publishedAt from article content
          const pubDate = extracted.publishedAt || article.publishedAt || null;
          if (pubDate && new Date(pubDate) < cutoff) continue;

          results.push({
            source: 'blog',
            name: blog.name,
            title: extracted.title || article.title || 'Untitled',
            url: article.url,
            publishedAt: pubDate,
            author: extracted.author || '',
            description: article.description || '',
            content: extracted.content
          });

          await new Promise(r => setTimeout(r, 500));
        } catch (err) {
          errors.push(`Blog: Error fetching article ${article.url}: ${err.message}`);
        }
      }
    } catch (err) {
      errors.push(`Blog: Error processing ${blog.name}: ${err.message}`);
    }
  }

  return results;
}

// -- Prompts Loading ---------------------------------------------------------

async function loadPrompts() {
  const prompts = {};
  const localPromptsDir = join(PROJECT_DIR, 'prompts');
  const errors = [];

  for (const filename of PROMPT_FILES) {
    const key = filename.replace('.md', '').replace(/-/g, '_');
    const localPath = join(localPromptsDir, filename);

    if (existsSync(localPath)) {
      prompts[key] = await readFile(localPath, 'utf-8');
    } else {
      errors.push(`Could not load prompt: ${filename}`);
    }
  }

  return { prompts, errors };
}

// -- Main --------------------------------------------------------------------

async function main() {
  const errors = [];

  // 1. Validate dependencies (non-fatal: only yt-dlp is required)
  console.error('Validating dependencies...');
  const depErrors = await validateDependencies();

  // yt-dlp is the only hard dependency
  const ytDlpMissing = depErrors.find(e => e.includes('yt-dlp 未安装'));
  if (ytDlpMissing) {
    console.error(JSON.stringify({
      status: 'error',
      message: '依赖检查失败',
      errors: depErrors
    }, null, 2));
    process.exit(1);
  }

  // Twitter errors are warnings, not fatal
  const warnings = depErrors.filter(e => !e.includes('yt-dlp 未安装'));
  for (const w of warnings) {
    console.error(`  WARNING: ${w}`);
    errors.push(w);
  }

  // 2. Load config
  const config = await loadConfig();
  const lookbackHours = config.lookbackHours || DEFAULT_LOOKBACK_HOURS;
  console.error(`Time window: ${lookbackHours} hours`);

  // 3. Load sources from config, filter out disabled items
  const rawSources = config.sources || {};
  const sources = {
    x_accounts: (rawSources.x_accounts || []).filter(s => s.enabled !== false),
    podcasts: (rawSources.podcasts || []).filter(s => s.enabled !== false),
    blogs: (rawSources.blogs || []).filter(s => s.enabled !== false)
  };
  const state = await loadState();

  // 4. Fetch content
  console.error('Fetching X/Twitter content...');
  const xContent = await fetchXContent(sources.x_accounts, state, lookbackHours, errors);
  console.error(`  Found ${xContent.length} builders with new tweets`);

  console.error('Fetching YouTube content...');
  const podcasts = await fetchYouTubeContent(sources.podcasts, state, lookbackHours, errors);
  console.error(`  Found ${podcasts.length} new episodes`);

  console.error('Fetching blog content...');
  const blogContent = await fetchBlogContent(sources.blogs, state, lookbackHours, errors);
  console.error(`  Found ${blogContent.length} new blog post(s)`);

  // 5. Save dedup state
  await saveState(state);

  // 6. Load prompts
  const { prompts, errors: promptErrors } = await loadPrompts();
  errors.push(...promptErrors);

  // 7. Build output
  const output = {
    status: 'ok',
    generatedAt: new Date().toISOString(),

    config: {
      language: config.language || 'en'
    },

    podcasts,
    x: xContent,
    blogs: blogContent,

    stats: {
      podcastEpisodes: podcasts.length,
      xBuilders: xContent.length,
      totalTweets: xContent.reduce((sum, a) => sum + a.tweets.length, 0),
      blogPosts: blogContent.length
    },

    prompts,

    errors: errors.length > 0 ? errors : undefined
  };

  // Save to file
  const OUTPUT_PATH = join(PROJECT_DIR, 'digest-data.json');
  await writeFile(OUTPUT_PATH, JSON.stringify(output, null, 2));

  // Print to stdout
  console.log(JSON.stringify(output, null, 2));
}

main().catch(err => {
  console.error(JSON.stringify({
    status: 'error',
    message: err.message
  }, null, 2));
  process.exit(1);
});


path: scripts/package-lock.json
content: {
  "name": "follow-builders-scripts",
  "version": "1.0.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "follow-builders-scripts",
      "version": "1.0.0",
      "dependencies": {
        "dotenv": "^16.4.0",
        "proper-lockfile": "^4.1.0",
        "rettiwt-api": "^3.1.1"
      },
      "engines": {
        "node": ">=22.0.0"
      }
    },
    "node_modules/@types/validator": {
      "version": "13.15.10",
      "resolved": "https://registry.npmmirror.com/@types/validator/-/validator-13.15.10.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/agent-base": {
      "version": "7.1.4",
      "resolved": "https://registry.npmmirror.com/agent-base/-/agent-base-7.1.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 14"
      }
    },
    "node_modules/ansi-styles": {
      "version": "4.3.0",
      "resolved": "https://registry.npmmirror.com/ansi-styles/-/ansi-styles-4.3.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "color-convert": "^2.0.1"
      },
      "engines": {
        "node": ">=8"
      },
      "funding": {
        "url": "https://github.com/chalk/ansi-styles?sponsor=1"
      }
    },
    "node_modules/asynckit": {
      "version": "0.4.0",
      "resolved": "https://registry.npmmirror.com/asynckit/-/asynckit-0.4.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/axios": {
      "version": "1.6.3",
      "resolved": "https://registry.npmmirror.com/axios/-/axios-1.6.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "follow-redirects": "^1.15.0",
        "form-data": "^4.0.0",
        "proxy-from-env": "^1.1.0"
      }
    },
    "node_modules/call-bind-apply-helpers": {
      "version": "1.0.2",
      "resolved": "https://registry.npmmirror.com/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "es-errors": "^1.3.0",
        "function-bind": "^1.1.2"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/chalk": {
      "version": "4.1.2",
      "resolved": "https://registry.npmmirror.com/chalk/-/chalk-4.1.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "ansi-styles": "^4.1.0",
        "supports-color": "^7.1.0"
      },
      "engines": {
        "node": ">=10"
      },
      "funding": {
        "url": "https://github.com/chalk/chalk?sponsor=1"
      }
    },
    "node_modules/class-validator": {
      "version": "0.14.1",
      "resolved": "https://registry.npmmirror.com/class-validator/-/class-validator-0.14.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "@types/validator": "^13.11.8",
        "libphonenumber-js": "^1.10.53",
        "validator": "^13.9.0"
      }
    },
    "node_modules/color-convert": {
      "version": "2.0.1",
      "resolved": "https://registry.npmmirror.com/color-convert/-/color-convert-2.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "color-name": "~1.1.4"
      },
      "engines": {
        "node": ">=7.0.0"
      }
    },
    "node_modules/color-name": {
      "version": "1.1.4",
      "resolved": "https://registry.npmmirror.com/color-name/-/color-name-1.1.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/combined-stream": {
      "version": "1.0.8",
      "resolved": "https://registry.npmmirror.com/combined-stream/-/combined-stream-1.0.8.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "delayed-stream": "~1.0.0"
      },
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/commander": {
      "version": "11.1.0",
      "resolved": "https://registry.npmmirror.com/commander/-/commander-11.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=16"
      }
    },
    "node_modules/cookiejar": {
      "version": "2.1.4",
      "resolved": "https://registry.npmmirror.com/cookiejar/-/cookiejar-2.1.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/debug": {
      "version": "4.4.3",
      "resolved": "https://registry.npmmirror.com/debug/-/debug-4.4.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "ms": "^2.1.3"
      },
      "engines": {
        "node": ">=6.0"
      },
      "peerDependenciesMeta": {
        "supports-color": {
          "optional": true
        }
      }
    },
    "node_modules/delayed-stream": {
      "version": "1.0.0",
      "resolved": "https://registry.npmmirror.com/delayed-stream/-/delayed-stream-1.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=0.4.0"
      }
    },
    "node_modules/dotenv": {
      "version": "16.6.1",
      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "BSD-2-Clause",
      "engines": {
        "node": ">=12"
      },
      "funding": {
        "url": "https://dotenvx.com"
      }
    },
    "node_modules/dunder-proto": {
      "version": "1.0.1",
      "resolved": "https://registry.npmmirror.com/dunder-proto/-/dunder-proto-1.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "call-bind-apply-helpers": "^1.0.1",
        "es-errors": "^1.3.0",
        "gopd": "^1.2.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-define-property": {
      "version": "1.0.1",
      "resolved": "https://registry.npmmirror.com/es-define-property/-/es-define-property-1.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-errors": {
      "version": "1.3.0",
      "resolved": "https://registry.npmmirror.com/es-errors/-/es-errors-1.3.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-object-atoms": {
      "version": "1.1.1",
      "resolved": "https://registry.npmmirror.com/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "es-errors": "^1.3.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-set-tostringtag": {
      "version": "2.1.0",
      "resolved": "https://registry.npmmirror.com/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "es-errors": "^1.3.0",
        "get-intrinsic": "^1.2.6",
        "has-tostringtag": "^1.0.2",
        "hasown": "^2.0.2"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/follow-redirects": {
      "version": "1.16.0",
      "resolved": "https://registry.npmmirror.com/follow-redirects/-/follow-redirects-1.16.0.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "individual",
          "url": "https://github.com/sponsors/RubenVerborgh"
        }
      ],
      "license": "MIT",
      "engines": {
        "node": ">=4.0"
      },
      "peerDependenciesMeta": {
        "debug": {
          "optional": true
        }
      }
    },
    "node_modules/form-data": {
      "version": "4.0.5",
      "resolved": "https://registry.npmmirror.com/form-data/-/form-data-4.0.5.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "asynckit": "^0.4.0",
        "combined-stream": "^1.0.8",
        "es-set-tostringtag": "^2.1.0",
        "hasown": "^2.0.2",
        "mime-types": "^2.1.12"
      },
      "engines": {
        "node": ">= 6"
      }
    },
    "node_modules/function-bind": {
      "version": "1.1.2",
      "resolved": "https://registry.npmmirror.com/function-bind/-/function-bind-1.1.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/get-intrinsic": {
      "version": "1.3.0",
      "resolved": "https://registry.npmmirror.com/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "call-bind-apply-helpers": "^1.0.2",
        "es-define-property": "^1.0.1",
        "es-errors": "^1.3.0",
        "es-object-atoms": "^1.1.1",
        "function-bind": "^1.1.2",
        "get-proto": "^1.0.1",
        "gopd": "^1.2.0",
        "has-symbols": "^1.1.0",
        "hasown": "^2.0.2",
        "math-intrinsics": "^1.1.0"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/get-proto": {
      "version": "1.0.1",
      "resolved": "https://registry.npmmirror.com/get-proto/-/get-proto-1.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "dunder-proto": "^1.0.1",
        "es-object-atoms": "^1.0.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/gopd": {
      "version": "1.2.0",
      "resolved": "https://registry.npmmirror.com/gopd/-/gopd-1.2.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/graceful-fs": {
      "version": "4.2.11",
      "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    },
    "node_modules/has-flag": {
      "version": "4.0.0",
      "resolved": "https://registry.npmmirror.com/has-flag/-/has-flag-4.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=8"
      }
    },
    "node_modules/has-symbols": {
      "version": "1.1.0",
      "resolved": "https://registry.npmmirror.com/has-symbols/-/has-symbols-1.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/has-tostringtag": {
      "version": "1.0.2",
      "resolved": "https://registry.npmmirror.com/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "has-symbols": "^1.0.3"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/hasown": {
      "version": "2.0.2",
      "resolved": "https://registry.npmmirror.com/hasown/-/hasown-2.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "function-bind": "^1.1.2"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/https-proxy-agent": {
      "version": "7.0.2",
      "resolved": "https://registry.npmmirror.com/https-proxy-agent/-/https-proxy-agent-7.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "agent-base": "^7.0.2",
        "debug": "4"
      },
      "engines": {
        "node": ">= 14"
      }
    },
    "node_modules/libphonenumber-js": {
      "version": "1.12.41",
      "resolved": "https://registry.npmmirror.com/libphonenumber-js/-/libphonenumber-js-1.12.41.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/math-intrinsics": {
      "version": "1.1.0",
      "resolved": "https://registry.npmmirror.com/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/mime-db": {
      "version": "1.52.0",
      "resolved": "https://registry.npmmirror.com/mime-db/-/mime-db-1.52.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/mime-types": {
      "version": "2.1.35",
      "resolved": "https://registry.npmmirror.com/mime-types/-/mime-types-2.1.35.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "mime-db": "1.52.0"
      },
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/ms": {
      "version": "2.1.3",
      "resolved": "https://registry.npmmirror.com/ms/-/ms-2.1.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/proper-lockfile": {
      "version": "4.1.2",
      "resolved": "https://registry.npmjs.org/proper-lockfile/-/proper-lockfile-4.1.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "graceful-fs": "^4.2.4",
        "retry": "^0.12.0",
        "signal-exit": "^3.0.2"
      }
    },
    "node_modules/proxy-from-env": {
      "version": "1.1.0",
      "resolved": "https://registry.npmmirror.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/retry": {
      "version": "0.12.0",
      "resolved": "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 4"
      }
    },
    "node_modules/rettiwt-api": {
      "version": "3.1.1",
      "resolved": "https://registry.npmmirror.com/rettiwt-api/-/rettiwt-api-3.1.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC",
      "dependencies": {
        "axios": "1.6.3",
        "chalk": "4.1.2",
        "class-validator": "0.14.1",
        "commander": "11.1.0",
        "https-proxy-agent": "7.0.2",
        "rettiwt-auth": "2.1.0",
        "rettiwt-core": "4.1.1"
      },
      "bin": {
        "rettiwt": "dist/cli.js"
      }
    },
    "node_modules/rettiwt-auth": {
      "version": "2.1.0",
      "resolved": "https://registry.npmmirror.com/rettiwt-auth/-/rettiwt-auth-2.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC",
      "dependencies": {
        "axios": "1.6.3",
        "commander": "11.1.0",
        "cookiejar": "2.1.4",
        "https-proxy-agent": "7.0.2"
      },
      "bin": {
        "rettiwt-auth": "dist/cli.js"
      }
    },
    "node_modules/rettiwt-core": {
      "version": "4.1.1",
      "resolved": "https://registry.npmmirror.com/rettiwt-core/-/rettiwt-core-4.1.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC",
      "dependencies": {
        "axios": "1.6.3",
        "form-data": "4.0.0"
      }
    },
    "node_modules/rettiwt-core/node_modules/form-data": {
      "version": "4.0.0",
      "resolved": "https://registry.npmmirror.com/form-data/-/form-data-4.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "asynckit": "^0.4.0",
        "combined-stream": "^1.0.8",
        "mime-types": "^2.1.12"
      },
      "engines": {
        "node": ">= 6"
      }
    },
    "node_modules/signal-exit": {
      "version": "3.0.7",
      "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    },
    "node_modules/supports-color": {
      "version": "7.2.0",
      "resolved": "https://registry.npmmirror.com/supports-color/-/supports-color-7.2.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "has-flag": "^4.0.0"
      },
      "engines": {
        "node": ">=8"
      }
    },
    "node_modules/validator": {
      "version": "13.15.35",
      "resolved": "https://registry.npmmirror.com/validator/-/validator-13.15.35.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">= 0.10"
      }
    }
  }
}


path: scripts/package.json
content: {
  "name": "follow-builders-scripts",
  "version": "1.0.0",
  "description": "Scripts for Follow Builders skill — feed generation and digest preparation",
  "type": "module",
  "engines": {
    "node": ">=22.0.0"
  },
  "scripts": {
    "fetch": "node fetch-and-prepare.js",
    "prepare-digest": "node fetch-and-prepare.js",
    "cli": "node cli.js"
  },
  "dependencies": {
    "dotenv": "^16.4.0",
    "proper-lockfile": "^4.1.0",
    "rettiwt-api": "^3.1.1"
  }
}


path: scripts/yt-dlp-helpers.js
content: #!/usr/bin/env node

// ============================================================================
// yt-dlp Helper Functions
// ============================================================================
// Wraps yt-dlp CLI for listing channel/playlist videos and fetching transcripts.
// All calls use child_process.execFile (no shell, no injection risk).
// ============================================================================

import { execFile as execFileCb } from 'child_process';
import { readFile, readdir, mkdir, rm } from 'fs/promises';
import { existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';

// Promisified execFile wrapper
function execFile(cmd, args, options = {}) {
  return new Promise((resolve, reject) => {
    execFileCb(cmd, args, options, (err, stdout, stderr) => {
      if (err) {
        err.stderr = stderr;
        reject(err);
      } else {
        resolve({ stdout, stderr });
      }
    });
  });
}

// ============================================================================
// yt-dlp Availability
// ============================================================================

export async function checkYtDlpAvailable() {
  try {
    const { stdout } = await execFile('yt-dlp', ['--version'], { timeout: 10000 });
    const version = stdout.trim();
    return { available: true, version };
  } catch (err) {
    return { available: false };
  }
}

// ============================================================================
// Cookies Support
// ============================================================================

// Get yt-dlp cookie args if YT_DLP_COOKIES is set in env
// Supports: "chrome", "firefox", "edge", or path to cookies.txt file
function getCookieArgs() {
  const cookieSource = process.env.YT_DLP_COOKIES;
  if (!cookieSource) return [];

  // If it's a browser name, use --cookies-from-browser
  const browsers = ['chrome', 'firefox', 'edge', 'brave', 'opera', 'vivaldi', 'safari'];
  if (browsers.includes(cookieSource.toLowerCase())) {
    return ['--cookies-from-browser', cookieSource.toLowerCase()];
  }

  // Otherwise treat as path to cookies file
  return ['--cookies', cookieSource];
}

// Get yt-dlp proxy args if HTTPS_PROXY or HTTP_PROXY is set
function getProxyArgs() {
  const proxy = process.env.FB_PROXY;
  if (!proxy) return [];
  return ['--proxy', proxy];
}

// ============================================================================
// List Channel/Playlist Videos
// ============================================================================

export async function listChannelVideos(podcast, lookbackHours, errors) {
  const url = podcast.url;
  const candidates = [];

  try {
    const args = [
      '--dump-json',
      '--skip-download',
      '--playlist-end', '5',
      '--no-warnings',
      '--quiet',
      ...getProxyArgs(),
      ...getCookieArgs()
    ];

    const { stdout } = await execFile('yt-dlp', args.concat(url), {
      maxBuffer: 20 * 1024 * 1024,
      timeout: 120000
    });

    // yt-dlp outputs one JSON object per line (NDJSON)
    const lines = stdout.trim().split('\n').filter(l => l.trim());

    for (const line of lines) {
      try {
        const video = JSON.parse(line);
        const uploadDate = video.upload_date || video.release_date || null;

        candidates.push({
          videoId: video.id,
          title: video.title || 'Untitled',
          publishedAt: uploadDate ? parseYtDlpDate(uploadDate) : null,
          rawDate: uploadDate
        });
      } catch {
        // Skip malformed lines
      }
    }
  } catch (err) {
    const msg = err.stderr || err.message || String(err);
    if (msg.includes('Sign in to confirm') || msg.includes('not a bot')) {
      errors.push(
        `YouTube: yt-dlp 被要求验证身份。请配置 cookies：\n` +
        `  在 data/.env 中添加 YT_DLP_COOKIES=chrome（或 firefox/edge）\n` +
        `  或指定 cookies 文件路径：YT_DLP_COOKIES=/path/to/cookies.txt`
      );
    } else {
      errors.push(`YouTube: Failed to list videos for ${podcast.name}: ${msg}`);
    }
  }

  return candidates;
}

// ============================================================================
// Fetch Transcript
// ============================================================================

export async function fetchTranscript(videoId, errors) {
  const tempDir = join(tmpdir(), `follow-builders-subs-${videoId}-${Date.now()}`);
  const videoUrl = `https://www.youtube.com/watch?v=${videoId}`;

  try {
    await mkdir(tempDir, { recursive: true });

    const args = [
      '--write-subs',
      '--write-auto-subs',
      '--skip-download',
      '--sub-lang', 'en',
      '--output', join(tempDir, 'subs'),
      ...getProxyArgs(),
      ...getCookieArgs(),
      videoUrl
    ];

    await execFile('yt-dlp', args, {
      timeout: 60000,
      maxBuffer: 5 * 1024 * 1024
    });

    // Find the VTT file
    const files = await readdir(tempDir);
    const vttFile = files.find(f => f.endsWith('.vtt'));

    if (!vttFile) {
      errors.push(`YouTube: No subtitles found for video ${videoId}`);
      return '';
    }

    const vttContent = await readFile(join(tempDir, vttFile), 'utf-8');
    return cleanVtt(vttContent);
  } catch (err) {
    const msg = err.stderr || err.message || String(err);
    errors.push(`YouTube: Failed to get transcript for ${videoId}: ${msg}`);
    return '';
  } finally {
    // Clean up temp directory
    try {
      if (existsSync(tempDir)) {
        await rm(tempDir, { recursive: true, force: true });
      }
    } catch {
      // Ignore cleanup errors
    }
  }
}

// ============================================================================
// VTT Parser
// ============================================================================

function cleanVtt(content) {
  const lines = content.split('\n');
  const textLines = [];

  const timestampPattern = /^\d{2}:\d{2}:\d{2}\.\d{3}\s*-->\s*\d{2}:\d{2}:\d{2}\.\d{3}/;
  const shortTimestampPattern = /^\d{2}:\d{2}\.\d{3}\s*-->\s*\d{2}:\d{2}\.\d{3}/;

  for (const rawLine of lines) {
    const line = rawLine.trim();

    if (!line) continue;
    if (line === 'WEBVTT') continue;
    if (/^\d+$/.test(line)) continue; // cue identifiers
    if (timestampPattern.test(line) || shortTimestampPattern.test(line)) continue;
    if (line.startsWith('NOTE') || line.startsWith('STYLE')) continue;

    // Strip HTML tags
    const clean = line.replace(/<[^>]+>/g, '');

    // Skip consecutive duplicates
    if (textLines.length > 0 && textLines[textLines.length - 1] === clean) continue;

    textLines.push(clean);
  }

  return textLines.join('\n');
}

// ============================================================================
// Utilities
// ============================================================================

function parseYtDlpDate(dateStr) {
  // yt-dlp returns upload_date as "YYYYMMDD"
  if (!dateStr || dateStr.length !== 8) return null;
  const year = parseInt(dateStr.slice(0, 4), 10);
  const month = parseInt(dateStr.slice(4, 6), 10) - 1;
  const day = parseInt(dateStr.slice(6, 8), 10);
  return new Date(Date.UTC(year, month, day)).toISOString();
}


path: README.md
content: **English**

# 追踪建造者，而非网红

一个 AI 驱动的信息聚合工具，追踪 AI 领域最顶尖的建造者——研究员、创始人、产品经理和工程师——并将他们的最新动态整理成易于消化的摘要。

**理念：** 追踪那些真正在做产品、有独立见解的人，而非只会搬运信息的网红。

## 你会得到什么

按需生成的摘要，包含：

- 顶级 AI 播客新节目的精华摘要
- 25 位精选 AI 建造者在 X/Twitter 上的关键观点和洞察
- AI 公司官方博客的完整文章（Anthropic Engineering、Claude Blog）
- 所有原始内容的链接
- 支持英文、中文或双语版本

## 前置要求

### 系统依赖

**yt-dlp**（YouTube 视频列表和字幕获取）
- macOS: `brew install yt-dlp`
- Linux: `pip install yt-dlp` 或 `pipx install yt-dlp`
- Windows: `pip install yt-dlp` 或从 [GitHub Releases](https://github.com/yt-dlp/yt-dlp/releases) 下载
- 如遇到 "Sign in to confirm" 错误，需配置 cookies：在 `.env` 添加 `YT_DLP_COOKIES=chrome`

**Node.js 22+**（Rettiwt-API 依赖要求）

**Twitter/X**：无需 API key，使用免费 guest 认证。如果遇到限流，可配置 user 认证（可选）。

## 快速开始

1. 在你的 AI agent 中安装此 skill（OpenClaw 或 Claude Code）
2. 确保已安装 yt-dlp：
   ```bash
   yt-dlp --version
   ```
3. （可选）如需 Twitter user 认证，添加到 `.env`：
   ```bash
   RETTIWT_API_KEY=your_key_here
   ```
4. 输入 "set up follow builders" 或执行 `/follow-builders`
5. Agent 会以对话方式引导你完成设置

设置完成后，你的第一期摘要会立即生成。

## 修改设置

通过对话即可修改所有设置。直接告诉你的 agent：

- "语言换成中文"
- "把摘要写得更简短一些"
- "显示我当前的设置"
- "添加 Twitter 账号 @username"
- "移除 @username"

## 自定义信息源

你可以完全控制追踪哪些信息源！

### 通过对话（推荐）
- "添加 Twitter 账号 @username"
- "添加播客频道 @channel"
- "列出我的信息源"
- "仅使用我的自定义源"
- "重置为默认源"

### 手动配置

所有配置统一在 `config/config.json` 中：

```json
{
  "sourceMode": "merge",
  "sources": {
    "x_accounts": [
      { "name": "你喜欢的建造者", "handle": "username" }
    ],
    "podcasts": [
      {
        "name": "我的播客",
        "type": "youtube_channel",
        "url": "https://youtube.com/@mypodcast",
        "channelHandle": "mypodcast"
      }
    ]
  }
}
```

**模式说明：**
- `merge` - 使用 config.json 中的源列表（默认）
- `replace` - 同上（源列表即为你自定义的内容）

## 自定义摘要风格

Skill 使用纯文本 prompt 文件来控制内容的摘要方式。你可以通过两种方式自定义：

**通过对话（推荐）：**
直接告诉你的 agent——"摘要写得更简练一些"、"多关注可操作的洞察"、"用更轻松的语气"。Agent 会自动帮你更新 prompt。

**直接编辑（高级用户）：**
编辑 `prompts/` 目录下的文件：
- `summarize-podcast.md` — 播客摘要方式
- `summarize-tweets.md` — Twitter 推文摘要方式
- `summarize-blogs.md` — 博客文章摘要方式
- `digest-intro.md` — 整体摘要格式和语气
- `translate.md` — 英文内容如何翻译成中文

这些都是纯文本说明，不是代码。修改会在下次生成摘要时生效。

## 默认信息源

### 播客（5个）
- [Latent Space](https://www.youtube.com/@LatentSpacePod)
- [Training Data](https://www.youtube.com/playlist?list=PLOhHNjZItNnMm5tdW61JpnyxeYH5NDDx8)
- [No Priors](https://www.youtube.com/@NoPriorsPodcast)
- [Unsupervised Learning](https://www.youtube.com/@RedpointAI)
- [Data Driven NYC](https://www.youtube.com/@DataDrivenNYC)

### AI 建造者（25位）
[Andrej Karpathy](https://x.com/karpathy), [Swyx](https://x.com/swyx), [Josh Woodward](https://x.com/joshwoodward), [Kevin Weil](https://x.com/kevinweil), [Peter Yang](https://x.com/petergyang), [Nan Yu](https://x.com/thenanyu), [Madhu Guru](https://x.com/realmadhuguru), [Amanda Askell](https://x.com/AmandaAskell), [Cat Wu](https://x.com/_catwu), [Thariq](https://x.com/trq212), [Google Labs](https://x.com/GoogleLabs), [Amjad Masad](https://x.com/amasad), [Guillermo Rauch](https://x.com/rauchg), [Alex Albert](https://x.com/alexalbert__), [Aaron Levie](https://x.com/levie), [Ryo Lu](https://x.com/ryolu_), [Garry Tan](https://x.com/garrytan), [Matt Turck](https://x.com/mattturck), [Zara Zhang](https://x.com/zarazhangrui), [Nikunj Kothari](https://x.com/nikunj), [Peter Steinberger](https://x.com/steipete), [Dan Shipper](https://x.com/danshipper), [Aditya Agarwal](https://x.com/adityaag), [Sam Altman](https://x.com/sama), [Claude](https://x.com/claudeai)

### 官方博客（2个）
- [Anthropic Engineering](https://www.anthropic.com/engineering) — Anthropic 团队的技术深度文章
- [Claude Blog](https://claude.com/blog) — Claude 产品公告和更新

## 安装

### OpenClaw
```bash
# 从 ClawHub 安装（即将推出）
clawhub install follow-builders

# 或手动安装
git clone https://github.com/champagne315/follow-aibuilders.git ~/skills/follow-builders
cd ~/skills/follow-builders/scripts && npm install
```

### Claude Code
```bash
git clone https://github.com/champagne315/follow-aibuilders.git ~/.claude/skills/follow-builders
cd ~/.claude/skills/follow-builders/scripts && npm install
```

## 工作原理

1. **本地抓取**：你的 agent 使用 Rettiwt-API（Twitter/X）和 yt-dlp（YouTube）免费抓取内容，同时直接 HTTP 抓取博客
2. **去重机制**：在 `state-feed.json` 中记录已读内容，避免重复
3. **内容重组**：你的 agent 使用可自定义的 prompt 将原始内容处理成易消化的摘要
4. **即时输出**：摘要直接在聊天中显示

**架构优势：**
- ✅ 完全免费，无需付费 API
- ✅ 完全控制信息源
- ✅ 隐私保护：数据本地处理，从不传输
- ✅ 无单点故障
- ✅ 内容和格式完全可定制

## 隐私说明

- 所有内容抓取都从你的本地机器进行，无需付费 API
- Twitter 认证（可选）存储在本地 `.env`
- 你的配置、偏好和阅读记录都保留在你的机器上
- Skill 只读取公开内容（公开博客文章、公开 YouTube 视频、公开 X 推文）

## 常见问题

### "yt-dlp 未安装"
按照前置要求中的说明安装 yt-dlp。

### "yt-dlp 被要求验证身份"
YouTube 要求登录验证。在 `.env` 添加：
```
YT_DLP_COOKIES=chrome
```
（支持 chrome、firefox、edge 等浏览器名，或 cookies.txt 文件路径）

### "Twitter API 速率限制中"
Guest 认证配额已用完。配置 user 认证：
1. 安装 Chrome 扩展 X Auth Helper
2. 在隐身模式登录 Twitter/X
3. 获取 API_KEY 并添加到 `.env`

### "No new updates from your builders today"
这是正常的——表示在回溯窗口内（推文 24 小时，播客/博客 72 小时）没有新内容。

### 想减少抓取时间？
- 使用 `mode: "replace"` 并减少信息源数量
- 对不需要的信息源设置 `enabled: false`

## 迁移

如从使用付费 API 的旧版本升级：

1. 安装 yt-dlp（见前置要求）
2. 确保 Node.js 22+ 已安装
3. 旧的 API keys（X_RAPIDAPI_KEY、SUPADATA_API_KEY）不再需要
4. 运行 `node fetch-and-prepare.js` 验证

## 许可证

MIT


path: state-feed.json
content: {
  "seenTweets": {},
  "seenVideos": {},
  "seenArticles": {}
}

Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)