Skill: frankieway/social-media-analysis (version 1.0.0)

--- name: social-media-analysis description: | 社交媒体舆情数据分析工具。从飞书多维表格读取 URL，下载媒体，分析内容，生成摘要。 entrypoint: command: "node" args: - "scripts/run_backfill.js" inputs: - name: app_id type: string required: false description: 飞书开放平台应用 APP_ID - name: app_secret type: string required: false description: 飞书开放平台应用 APP_SECRET - name: bitable_url type: string required: false description: 飞书多维表视图链接（base/...?...&table=...） - name: only_empty type: string required: false default: "1" description: 只补空值（1/0），默认只填空 - name: limit type: integer required: false default: 50 description: 本次最多回填多少条记录（测试/防爆用） - name: workers type: integer required: false default: 4 description: 并发 worker 数 - name: max_chars type: integer required: false default: 100 description: 内容解析最大长度 - name: source_channel_field type: string required: false default: "来源渠道" description: 多维表字段名（来源渠道） - name: original_url_field type: string required: false default: "原文URL" description: 多维表字段名（原文URL；脚本内也做了候选兜底） - name: xhs_cookie type: string required: false description: 小红书 Cookie（可选） permissions: network: - "https://open.feishu.cn" outputs: - name: ok type: boolean description: 执行是否成功 --- **脚本清单**： - `scripts/run_backfill.js` - Clawhub 入口（依次跑多渠道回填 + 微博回填） - `extract-douyin-images.js` - 抖音图集下载 - `parse-douyin-video.js` - 抖音视频/音频下载 - `parse-weibo.js` - 微博图集下载 - `backfill-weibo-content-analysis.js` - 微博正文回填到「内容解析」 - `backfill-content-analysis-by-source.js` - 多渠道原文URL回填「内容解析」 - `parse-xiaohongshu.js` - 小红书图集下载 - `parse-toutiao-playwright.js` - 今日头条 Playwright 渲染下载 - `parse-bilibili.js` - B 站视频下载 - `build-content-analysis.js` - 统一内容解析生成（100字内，带兜底） **使用场景**： (1) 分析飞书多维表格中的社交媒体数据 (2) 批量下载媒体文件（图片/视频） (3) 生成 100 字内容解析 (4) 用户提到"舆情分析"、"社交媒体分析" # Social Media Analysis Skill ## 🎯 功能 - URL 有效性检查 - 媒体类型判断（图片/视频） - 媒体文件下载 - 视频抽帧（每 5 秒 1 帧） - 内容解析生成（100 字以内） --- ## 📋 工作流 ``` 1. 创建字段 → 2. 链接检查 → 3. 下载媒体 → 4. 抽帧分析 → 5. 更新表格 ``` ### 步骤 1: 创建字段 ```bash # 链接是否有效（单选） # 包含多媒体类型（单选：图片/视频） # 内容解析（文本） ``` ### 步骤 2: 链接检查 | 平台 | 方法 | |------|------| | 抖音 | 脚本解析（不能用 HEAD） | | 微博/今日头条/B 站 | HEAD 请求 | | 快手 | 特殊处理 | ### 步骤 3: 下载媒体 | 平台 | 视频 | 图集 | |------|------|------| | 抖音 | `parse-douyin-video.js` | `extract-douyin-images.js` | | 微博 | `yt-dlp` | `parse-weibo.js` | | 今日头条 | `yt-dlp` | `parse-toutiao-playwright.js` | | B 站 | `yt-dlp` | - | ### 步骤 4: 抽帧分析 ```bash # 视频抽帧（每 5 秒） ffmpeg -i video.mp4 -vf "fps=1/5" frames/frame_%03d.jpg # 图片分析 image frame_001.jpg "描述画面内容" ``` ### 步骤 4.5: 统一生成内容解析（强制） ```bash node scripts/build-content-analysis.js \ --title "标题" \ --text "正文" \ --platform "微博" \ --media-type "视频" \ --visual "抽帧/图片识别结果" ``` 规则： - 必须输出 `content_analysis`，不可为空。 - 长度限制 100 字以内。 - 优先描述与“小米/小爱”相关信息；无相关信息时如实描述内容。 ### 步骤 5: 更新表格 ```bash curl -X PUT "https://open.feishu.cn/open-apis/bitable/v1/apps/APP/tables/TBL/records/REC" \ -H "Authorization: Bearer TOKEN" \ -d '{"fields": {"链接是否有效": "是", "包含多媒体类型": "视频", "内容解析": "..."}}' ``` --- ## 🔧 前置准备 ### 认证 ```bash APP_ID="cli_a93d6e8a913a5bc8" APP_SECRET="<your_secret>" [REDACTED_SECRET] -s -X POST https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal \ -H "Content-Type: application/json" \ -d "{\"app_id\": \"$APP_ID\", \"app_secret\": \"$APP_SECRET\"}" | jq -r '.tenant_access_token') ``` ### 工具 | 工具 | 用途 | |------|------| | `yt-dlp` | 微博/B 站视频下载 | | `ffmpeg` | 视频抽帧 | | `node` | 运行 JS 脚本 | | `playwright` | 今日头条渲染 | --- ## 📱 平台下载方法 ### 抖音 ```bash # 判断类型 node scripts/extract-douyin-images.js URL /tmp/check # 视频 node scripts/parse-douyin-video.js URL --download # 图集 node scripts/extract-douyin-images.js URL ./images ``` ### 微博 **判断类型**： ```bash yt-dlp --list-formats URL 2>&1 | grep -q "No video formats" # 有 "No video formats" = 图文，无 = 视频 ``` **下载方法**： | 类型 | 方法 | |------|------| | **视频** | `yt-dlp -o video.mp4 URL` | | **图文（1-9 张）** | 浏览器获取 + curl 下载 | **图文下载脚本**： ```bash # 方法 1：浏览器获取图片 URL node -e " const { chromium } = require('playwright'); (async () => { const browser = await chromium.launch({ headless: true }); const page = await browser.newPage(); await page.goto('https://weibo.com/USER/ID', { waitUntil: 'domcontentloaded' }); await page.waitForTimeout(5000); const images = await page.evaluate(() => { return Array.from(document.querySelectorAll('img[src*=\\\"sinaimg\\\"]')) .map(img => img.src.replace('orj360', 'mw690')); }); console.log(images.join('\n')); await browser.close(); })().catch(console.error); " # 方法 2：curl 下载 curl -sL "IMAGE_URL" -o "images/image-01.jpg" ``` **注意**： - parse-weibo 脚本可能误判，应以 yt-dlp 为准 - 微博图片域名：sinaimg.cn（wx1-4, tvax1-4, ww1-3） - 图片尺寸：orj360（小）→ mw690（中）→ mw2000（大） ### 今日头条 ```bash # 文章图片 node scripts/parse-toutiao-playwright.js URL ./output # 视频（yt-dlp 原生支持） yt-dlp -o video.mp4 URL ``` ### B 站 ```bash yt-dlp -o video.mp4 URL ``` ### 小红书 ```bash # 判断类型 yt-dlp --dump-json URL | jq '.duration' # 视频 yt-dlp -o video.mp4 URL # 图集（需 Cookie） node scripts/parse-xiaohongshu.js URL ./images ``` --- ## ⚠️ 注意事项 1. **抖音链接检查**：必须用脚本解析，HEAD 请求会触发反爬 2. **抖音图集**：只有音频（M4A）+ 图片（WebP），无视频流 3. **今日头条**：需要 Playwright 渲染，静态 HTML 无图片 4. **抽帧标准**：统一每 5 秒 1 帧 5. **内容解析**：基于实际画面描述，不编造 --- ## 📚 示例 ### 处理单条记录 ```bash # 1. 检查链接 node scripts/parse-douyin-video.js URL --download # 2. 抽帧 ffmpeg -i video.mp4 -vf "fps=1/5" frames/frame_%03d.jpg # 3. 分析 image frames/frame_001.jpg "描述画面" # 4. 更新表格 curl -X PUT "..." -d '{"fields": {...}}' ``` ### 批量处理 ```bash # 见 WORKFLOW_V2.md 完整脚本 ``` --- ## 🔗 相关文档 - `WORKFLOW_V2.md` - 完整工作流脚本 - `memory/2026-03-23.md` - 执行记录

path: WORKFLOW_V2.md
content: # Social Media Analysis v2 - 优化工作流


```
┌─────────────────────────────────────────────────────────────────┐
│  步骤 1: 创建字段（一次性）                                       │
│  - 链接是否有效（单选：是/否）                                    │
│  - 包含多媒体类型（单选：纯文本/图片/视频）                        │
│  - 内容解析（文本）                                              │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  步骤 2: 快速有效性检查（批量）                                   │
│                                                                 │
│  2.1 HEAD 请求探测（不下载内容）                                  │
│      curl -I -L --max-time 5 URL                                │
│      - HTTP 200/302 → 有效                                      │
│      - HTTP 404/403/超时 → 无效                                 │
│                                                                 │
│  2.2 更新「链接是否有效」字段（批量 10 条/次）                       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  步骤 3: 完整媒体处理（仅有效链接）                               │
│                                                                 │
│  3.1 平台识别 + 类型判断                                         │
│      抖音：先图集脚本探测 → 0 张=视频，N 张=图集                   │
│      微博：yt-dlp --dump-json → duration=视频，无=图集            │
│      B 站：yt-dlp --dump-json → 必有视频                          │
│      今日头条：Playwright → 文章图片                             │
│      微信：Camoufox → 文章图片                                   │
│                                                                 │
│  3.2 全部下载（保证完整性）                                      │
│      - 图集：下载全部图片 + 音频                                  │
│             node scripts/extract-douyin-images.js URL output    │
│      - 视频：下载完整视频                                        │
│             node scripts/parse-douyin-video.js URL --download   │
│             yt-dlp -o video.mp4 URL                             │
│      - 今日头条/微信：下载全部文章图片                            │
│             node scripts/parse-toutiao-playwright.js URL output │
│                                                                 │
│  3.3 视频抽帧（统一标准）                                        │
│      所有视频：每 5 秒抽 1 帧                                        │
│      命令：ffmpeg -i video.mp4 -vf "fps=1/5" frames/frame_%03d.jpg │
│                                                                 │
│  3.4 更新「包含多媒体类型」字段                                   │
│      - 图片：图集/文章图片                                       │
│      - 视频：含视频内容                                          │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  步骤 4: 大模型分析（全面分析）                                   │
│                                                                 │
│  4.1 输入内容                                                    │
│      - 必选：标题 + 正文（已有字段）                              │
│      - 必选：所有抽帧图片（视频）                                 │
│      - 必选：所有图集图片（图集）                                 │
│      - 必选：所有文章图片（今日头条/微信）                        │
│                                                                 │
│  4.2 提示词模板                                                  │
│      基于以下信息生成 100 字以内内容解析：                         │
│      - 标题：{title}                                            │
│      - 正文：{text}                                             │
│      - 平台：{platform}                                         │
│      - 图片/视频内容：{visual_content}                          │
│      要求：                                                      │
│      1. 突出与"小爱/小米/小爱同学"相关的内容（如有）               │
│      2. 无相关信息则如实描述画面                                  │
│      3. 严格控制在 100 字以内                                      │
│                                                                 │
│  4.3 分析策略                                                    │
│      - 视频：分析所有抽帧图片（每 5 秒 1 帧）                        │
│      - 图集：分析所有图片                                       │
│      - 文章：分析所有配图                                       │
│                                                                 │
│  4.4 统一解析器兜底（必须执行）                                    │
│      node scripts/build-content-analysis.js \                   │
│        --title "{title}" --text "{text}" \                      │
│        --platform "{platform}" --media-type "{media_type}" \    │
│        --visual "{visual_content}"                               │
│      - 强制保障“内容解析”非空                                      │
│      - 输出限制 100 字以内                                         │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  步骤 5: 批量更新表格                                            │
│                                                                 │
│  5.1 每 10 条记录批量更新一次（减少 API 调用）                       │
│  5.2 更新字段：链接是否有效、包含多媒体类型、内容解析             │
└─────────────────────────────────────────────────────────────────┘
```





path: scripts/backfill-content-analysis-by-source.js
content: #!/usr/bin/env node
/**
 * 多渠道回填「内容解析」（纯 Node 版本）
 *
 * 功能：
 * - 从飞书多维表读取记录
 * - 按来源渠道（来源渠道字段）筛选：抖音APP/酷安APP/今日头条/快手APP/小红书APP 等
 * - 读取原文 URL（候选字段：获取原文URL/原文 URL/原文URL/原文url/doc_url）
 * - 解析 URL 提取标题/摘要（抖音额外解析 desc）
 * - 生成 100 字以内内容解析并写回「内容解析」
 *
 * 回填策略：
 * - 默认 ONLY_EMPTY=1：仅当「内容解析」为空时写回
 *
 * 运行：
 *   set APP_ID=...; set APP_SECRET=...; set BITABLE_URL=...
 *   node scripts/backfill-content-analysis-by-source.js
 */

const { buildAnalysis } = require("./build-content-analysis");

function env(key, defaultValue = "") {
  const v = process.env[key] ?? process.env[`INPUT_${String(key).toUpperCase()}`];
  return String(v ?? defaultValue).trim();
}

const APP_ID = env("APP_ID");
const APP_SECRET = env("APP_SECRET");
const BITABLE_URL = env("BITABLE_URL");

const SOURCE_CHANNEL_FIELD = env("SOURCE_CHANNEL_FIELD", "来源渠道");
const CONTENT_ANALYSIS_FIELD = env("CONTENT_ANALYSIS_FIELD", "内容解析");
const LINK_STATUS_FIELD = env("LINK_STATUS_FIELD", "链接是否有效");
const HAS_IMAGE_FIELD = env("HAS_IMAGE_FIELD", "是否包含图片");
const HAS_VIDEO_FIELD = env("HAS_VIDEO_FIELD", "是否包含视频");

const ORIGINAL_URL_CANDIDATES = [
  // Clawhub 输入名：original_url_field（这里做映射兼容）
  env("ORIGINAL_URL_FIELD") || env("WEIBO_ORIGINAL_URL_FIELD"),
  "获取原文URL",
  "原文 URL",
  "原文URL",
  "原文url",
  "doc_url",
].filter(Boolean);

// 默认覆盖你表里常见渠道（不含“新浪微博”，微博由 backfill-weibo-content-analysis.js 专门处理）
const DEFAULT_CHANNELS = [
  "微信",
  "抖音APP",
  "小米社区",
  "快手APP",
  "今日头条",
  "什么值得买",
  "酷安APP",
  "搜狐",
  "小红书APP",
];
const SOURCE_CHANNELS = (env("SOURCE_CHANNELS", DEFAULT_CHANNELS.join(",")))
  .split(",")
  .map((s) => s.trim())
  .filter(Boolean);
function normalizeChannelValue(s) {
  return String(s || "")
    .replace(/\s+/g, "")
    .trim();
}

const SOURCE_CHANNELS_SET = new Set(SOURCE_CHANNELS.map((s) => normalizeChannelValue(s)));

const ONLY_EMPTY = ["1", "true", "yes"].includes((env("ONLY_EMPTY", "1")).toLowerCase());
const MAX_TARGETS = parseInt(env("LIMIT", "50"), 10);
const PAGE_SIZE = parseInt(env("PAGE_SIZE", "200"), 10);
const WORKERS = parseInt(env("WORKERS", "4"), 10);
const MAX_CHARS = parseInt(env("MAX_CHARS", "100"), 10);

const HTTP_TIMEOUT_MS = parseInt(env("HTTP_TIMEOUT_MS", "15000"), 10);
const HTTP_RETRY_TIMES = parseInt(env("HTTP_RETRY_TIMES", "1"), 10);

const UA_COMMON =
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";

const SOURCE_CHANNEL_VALUE_TO_PLATFORM = {
  "抖音APP": ["抖音", "视频"],
  "酷安APP": ["酷安", "图文"],
  "今日头条": ["今日头条", "图文"],
  "快手APP": ["快手", "视频"],
  "小红书APP": ["小红书", "图文"],
};

const SOURCE_CHANNEL_VALUE_TO_PLATFORM_NORM = Object.fromEntries(
  Object.entries(SOURCE_CHANNEL_VALUE_TO_PLATFORM).map(([k, v]) => [normalizeChannelValue(k), v])
);

function inferMediaFlagsFromHtml(html) {
  const s = String(html || "");
  const hasImage = /<img\b/i.test(s) || /og:image/i.test(s);
  const hasVideo =
    /<video\b/i.test(s) ||
    /\bmp4\b/i.test(s) ||
    /\bm3u8\b/i.test(s) ||
    /\bwebm\b/i.test(s);
  return { hasImage, hasVideo };
}

function inferMediaTypeFromFlags(hasImage, hasVideo) {
  if (hasVideo) return "视频";
  if (hasImage) return "图片";
  return "未知";
}

function inferDefaultFlagsFromChannel(sourceChannel) {
  // 仅对“我们已知是视频/图文”的平台给出确定结论
  const mapped = SOURCE_CHANNEL_VALUE_TO_PLATFORM_NORM[sourceChannel];
  if (!mapped) return null;
  const inferredMediaType = mapped[1];
  const hasImage = String(inferredMediaType).includes("图文");
  const hasVideo = String(inferredMediaType).includes("视频");
  if (!hasImage && !hasVideo) return null;
  return { hasImage, hasVideo };
}

function parseBitableUrl(url) {
  const parsed = new URL(url);
  let appToken = "";
  const parts = parsed.pathname.split("/").filter(Boolean);
  for (let i = 0; i < parts.length; i++) {
    if (parts[i] === "base" && i + 1 < parts.length) {
      appToken = parts[i + 1];
      break;
    }
  }
  const tableId = parsed.searchParams.get("table");
  if (!appToken || !tableId) throw new Error(`BITABLE_URL 解析失败: ${url}`);
  return [appToken, tableId];
}

async function withRetry(fn, retryTimes) {
  let lastErr = null;
  for (let attempt = 0; attempt <= retryTimes; attempt++) {
    try {
      return await fn();
    } catch (e) {
      lastErr = e;
      const backoff = 600 * Math.pow(2, attempt);
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
  throw lastErr;
}

async function httpGetText(url) {
  return withRetry(async () => {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), HTTP_TIMEOUT_MS);
    try {
      const resp = await fetch(url, {
        method: "GET",
        headers: { "User-Agent": UA_COMMON, Accept: "text/html,application/json,*/*" },
        redirect: "follow",
        signal: ctrl.signal,
      });
      if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
      return await resp.text();
    } finally {
      clearTimeout(t);
    }
  }, HTTP_RETRY_TIMES);
}

async function httpGetTextWithHeaders(url, headers) {
  return withRetry(async () => {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), HTTP_TIMEOUT_MS);
    try {
      const resp = await fetch(url, {
        method: "GET",
        headers,
        redirect: "follow",
        signal: ctrl.signal,
      });
      if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
      return await resp.text();
    } finally {
      clearTimeout(t);
    }
  }, HTTP_RETRY_TIMES);
}

function safeText(v) {
  if (v == null) return "";
  if (Array.isArray(v)) return v.length ? String(v[0]).trim() : "";
  if (typeof v === "object") return JSON.stringify(v);
  return String(v).trim();
}

async function getTenantAccessToken() {
  if (!APP_ID || !APP_SECRET) throw new Error("请设置环境变量 APP_ID / APP_SECRET");
  const url = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal";
  const resp = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ app_id: APP_ID, app_secret: APP_SECRET }),
    redirect: "follow",
  });
  if (!resp.ok) throw new Error(`tenant_access_token HTTP ${resp.status}`);
  const data = await resp.json();
  if (data.code !== 0) throw new Error(`获取 tenant_access_token 失败: ${JSON.stringify(data)}`);
  return data.tenant_access_token;
}

async function fetchTargets(token) {
  if (!BITABLE_URL) throw new Error("请设置环境变量 BITABLE_URL（飞书多维表链接）");
  const [appToken, tableId] = parseBitableUrl(BITABLE_URL);
  const url = `https://open.feishu.cn/open-apis/bitable/v1/apps/${appToken}/tables/${tableId}/records`;
  const headers = { Authorization: `Bearer ${token}` };

  const targets = [];
  let pageToken = null;
  while (targets.length < MAX_TARGETS) {
    const params = new URLSearchParams();
    params.set("page_size", String(Math.min(500, PAGE_SIZE)));
    if (pageToken) params.set("page_token", pageToken);

    const resp = await fetch(`${url}?${params.toString()}`, {
      method: "GET",
      headers,
      redirect: "follow",
    });
    if (!resp.ok) throw new Error(`获取记录失败 HTTP ${resp.status}`);
    const data = await resp.json();
    if (data.code !== 0) throw new Error(`获取记录失败: ${JSON.stringify(data)}`);

    const obj = data.data || {};
    const items = obj.items || [];
    for (const rec of items) {
      const fields = rec.fields || {};
      const ch = safeText(fields[SOURCE_CHANNEL_FIELD]);
      const chNorm = normalizeChannelValue(ch);
      if (!SOURCE_CHANNELS_SET.has(chNorm)) continue;

      const existingContent = safeText(fields[CONTENT_ANALYSIS_FIELD]);
      const existingLink = safeText(fields[LINK_STATUS_FIELD]);
      const existingImg = safeText(fields[HAS_IMAGE_FIELD]);
      const existingVid = safeText(fields[HAS_VIDEO_FIELD]);

      const needsContent = !existingContent;
      const needsLink = !existingLink;
      const needsImg = !existingImg;
      const needsVid = !existingVid;

      if (ONLY_EMPTY && !(needsContent || needsLink || needsImg || needsVid)) continue;

      let originalUrl = "";
      for (const cand of ORIGINAL_URL_CANDIDATES) {
        if (fields[cand] != null) {
          const v = safeText(fields[cand]);
          if (v) {
            originalUrl = v;
            break;
          }
        }
      }
      if (!originalUrl) continue;

      if (!rec.record_id) continue;
      targets.push({
        record_id: rec.record_id,
        source_channel: chNorm,
        original_url: originalUrl,
        need_content: needsContent,
        need_link: needsLink,
        need_img: needsImg,
        need_video: needsVid,
      });
      if (targets.length >= MAX_TARGETS) break;
    }

    if (!obj.has_more || !obj.page_token) break;
    pageToken = obj.page_token;
  }

  return targets;
}

async function batchUpdateRecords(token, updates) {
  if (!updates.length) return;
  const [appToken, tableId] = parseBitableUrl(BITABLE_URL);
  const url = `https://open.feishu.cn/open-apis/bitable/v1/apps/${appToken}/tables/${tableId}/records/batch_update`;
  const resp = await fetch(url, {
    method: "POST",
    headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
    body: JSON.stringify({ records: updates }),
  });
  if (!resp.ok) throw new Error(`批量更新失败 HTTP ${resp.status}`);
  const data = await resp.json();
  if (data.code !== 0) throw new Error(`批量更新失败: ${JSON.stringify(data)}`);
}

function stripHtmlLike(s) {
  return String(s || "")
    .replace(/<[^>]*>/g, "")
    .replace(/&quot;/g, '"')
    .replace(/&#39;/g, "'")
    .replace(/&amp;/g, "&")
    .replace(/\s+/g, " ")
    .trim();
}

function extractMetaTitleDesc(html) {
  const title =
    stripHtmlLike(
      (html.match(/<meta[^>]+property=["']og:title["'][^>]+content=["']([^"']+)["'][^>]*>/i) || [])[1]
    ) ||
    stripHtmlLike((html.match(/<title>([^<]+)<\/title>/i) || [])[1]) ||
    "";

  let desc =
    stripHtmlLike(
      (html.match(/<meta[^>]+property=["']og:description["'][^>]+content=["']([^"']+)["'][^>]*>/i) || [])[1]
    ) ||
    stripHtmlLike(
      (html.match(/<meta[^>]+name=["']description["'][^>]+content=["']([^"']+)["'][^>]*>/i) || [])[1]
    ) ||
    "";

  if (!desc) {
    // JSON-LD
    const ld = html.match(/<script[^>]+type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/i);
    if (ld && ld[1]) {
      try {
        const obj = JSON.parse(ld[1]);
        if (obj && typeof obj === "object") {
          desc =
            stripHtmlLike(
              obj.articleBody ||
                obj.description ||
                obj.abstract ||
                obj.text ||
                ""
            ) || "";
          // title fallback
        }
      } catch {}
    }
  }
  return [title, desc];
}

function deepFindFirstString(obj, keys, maxDepth = 6) {
  const seen = new Set();
  const stack = [{ v: obj, d: 0 }];

  while (stack.length) {
    const { v, d } = stack.pop();
    if (d > maxDepth) continue;
    if (v == null) continue;

    const type = typeof v;
    if (type === "string") {
      continue;
    }
    if (type !== "object") continue;

    if (seen.has(v)) continue;
    seen.add(v);

    if (Array.isArray(v)) {
      for (const item of v) stack.push({ v: item, d: d + 1 });
      continue;
    }

    for (const k of keys) {
      if (Object.prototype.hasOwnProperty.call(v, k)) {
        const sv = v[k];
        if (typeof sv === "string" && sv.trim()) return sv.trim();
      }
    }

    for (const vv of Object.values(v)) {
      if (vv && typeof vv === "object") stack.push({ v: vv, d: d + 1 });
    }
  }
  return "";
}

function extractFromNextData(html) {
  // 常见于 Next.js：<script id="__NEXT_DATA__" type="application/json">...</script>
  const next = html.match(/<script[^>]+id=["']__NEXT_DATA__["'][^>]*>([\s\S]*?)<\/script>/i);
  if (!next || !next[1]) return { title: "", text: "" };
  try {
    const obj = JSON.parse(next[1]);
    const title = deepFindFirstString(obj, ["title", "headline", "name"]);
    const text =
      deepFindFirstString(obj, [
        "description",
        "abstract",
        "articleBody",
        "text",
        "content",
        "caption",
        "noteContent",
        "feedDesc",
      ]) || "";
    return { title, text };
  } catch {
    return { title: "", text: "" };
  }
}

function extractTextByRegex(html, regex, maxLen = 6000) {
  const m = html.match(regex);
  if (!m) return "";
  const raw = m[1] || "";
  const s = stripHtmlLike(raw);
  if (!s) return "";
  return s.length > maxLen ? s.slice(0, maxLen) : s;
}

function extractEscapedJsonString(html, key) {
  // 从类似："key":"...转义字符串..." 里提取，并用 JSON.parse 解码 \\uXXXX / \\n 等转义
  // 适用于很多网页把文本塞进脚本 JSON 的情况（比普通 /"key":"([^"]*)"/ 更稳）
  try {
    const re = new RegExp(`"${key}"\\s*:\\s*"((?:\\\\.|[^"\\\\])*)"`,"m");
    const m = html.match(re);
    if (!m || !m[1]) return "";
    const decoded = JSON.parse(`"${m[1]}"`);
    if (typeof decoded !== "string") return "";
    return stripHtmlLike(decoded);
  } catch {
    return "";
  }
}

function extractEscapedJsonAnyKey(html, keys) {
  for (const k of keys) {
    const v = extractEscapedJsonString(html, k);
    if (v) return v;
  }
  return "";
}

function extractToutiaoId(url) {
  const m1 = String(url).match(/\/item\/(\d+)/);
  if (m1) return m1[1];
  const m2 = String(url).match(/\/i(\d+)/);
  if (m2) return m2[1];
  return "";
}

async function tryExtractToutiaoViaAltUrls(originalUrl) {
  const id = extractToutiaoId(originalUrl);
  if (!id) return { title: "", text: "" };

  const candidates = [
    // 移动端通常更可能返回预渲染内容（比原始站点 HTML 更可读）
    `https://m.toutiao.com/i${id}/`,
    // 常见文章路径变体
    `https://www.toutiao.com/a${id}/`,
    `https://www.toutiao.com/i${id}/`,
    `https://www.toutiao.com/item/${id}/`,
  ];

  for (const u of candidates) {
    try {
      const html = await httpGetText(u);
      let [title, desc] = extractMetaTitleDesc(html);
      if (!desc) {
        const next = extractFromNextData(html);
        title = title || next.title || "";
        desc = next.text || "";
      }
      if (!desc) {
        const keys = ["abstract", "description", "articleBody", "content", "text", "title", "headline"];
        desc = extractEscapedJsonAnyKey(html, ["abstract", "description", "articleBody", "content", "text"]);
        if (!title) title = extractEscapedJsonAnyKey(html, ["title", "headline", "name"]);
      }
      if (desc && desc.trim().length >= 10) return { title, text: desc };
    } catch {
      // ignore
    }
  }

  return { title: "", text: "" };
}

async function extractToutiaoWithPlaywright(url) {
  // 尽力：如果 playwright 未安装，就直接抛错并由上层忽略
  const mod = require("playwright");
  const { chromium } = mod;
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: "networkidle", timeout: 30000 });
    await page.waitForTimeout(5000);
    const content = await page.evaluate(() => {
      const title =
        document.querySelector("h1")?.innerText ||
        document.querySelector('meta[property="og:title"]')?.getAttribute("content") ||
        "";
      const article =
        document.querySelector("article") ||
        document.querySelector(".article") ||
        document.body;
      const text = (article?.innerText || "").trim();
      return { title, text };
    });
    return {
      title: String(content.title || "").trim(),
      text: String(content.text || "").trim(),
    };
  } finally {
    await browser.close();
  }
}

async function extractDouyinDescFromUrl(url) {
  // 复刻 scripts/parse-douyin-video.js 的解析逻辑，只拿 desc
  let awemeId = "";
  let m = url.match(/modal_id=([0-9]+)/);
  if (m) awemeId = m[1];
  if (!awemeId) {
    m = url.match(/video\/([^/?]*)/);
    if (m) awemeId = m[1];
  }
  if (!awemeId) {
    m = url.match(/note\/([^/?]*)/);
    if (m) awemeId = m[1];
  }
  if (!awemeId) throw new Error(`无法从抖音 URL 中提取视频ID: ${url}`);

  const requestUrl = `https://www.douyin.com/jingxuan?modal_id=${awemeId}`;
  const headers = {
    "User-Agent": UA_COMMON,
    Accept:
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Accept-Language": "zh-CN,zh;q=0.9",
    Referer: "https://www.douyin.com/",
    Connection: "keep-alive",
  };

  const ctrl = new AbortController();
  const t = setTimeout(() => ctrl.abort(), HTTP_TIMEOUT_MS);
  try {
    const resp = await fetch(requestUrl, { method: "GET", headers, redirect: "follow", signal: ctrl.signal });
    if (!resp.ok) throw new Error(`douyin request HTTP ${resp.status}`);
    const htmlContent = await resp.text();

    const pattern = /"([^"]*?(?:playAddr|searchProps|app)[^"]*?)"/g;
    const matches = [];
    let mm;
    while ((mm = pattern.exec(htmlContent)) !== null) {
      matches.push(mm[1]);
    }
    let targetMatch = "";
    for (const s of matches) {
      if (s.includes("playAddr") && s.includes("searchProps") && s.includes("app")) {
        targetMatch = s;
        break;
      }
    }
    if (!targetMatch) throw new Error("未找到包含视频数据的 JSON 片段");
    const decodedJson = decodeURIComponent(targetMatch);
    const videoDataJson = JSON.parse(decodedJson);
    const videoDetail = (((videoDataJson || {}).app || {}).videoDetail) || {};
    const desc = videoDetail.desc || "";
    return { title: desc || "", text: desc || "" };
  } finally {
    clearTimeout(t);
  }
}

async function extractBySourceChannel(sourceChannel, originalUrl) {
  const [platform, mediaType] =
    SOURCE_CHANNEL_VALUE_TO_PLATFORM_NORM[sourceChannel] || [sourceChannel, "未知"];

  if (sourceChannel === "抖音APP") {
    const { title, text } = await extractDouyinDescFromUrl(originalUrl);
    const hasImage = false;
    const hasVideo = true;
    return { title, text, visual: "", platform, mediaType: "视频", hasImage, hasVideo };
  }

  // 默认：普通抓取 HTML（用于酷安/今日头条/快手/小红书等尽力提取 meta/json）
  let html = await httpGetText(originalUrl);

  // 小红书：如果提供了 Cookie，尽可能拿到内容
  if (sourceChannel === "小红书APP") {
    const xhsCookie = env("XHS_COOKIE");
    if (xhsCookie.trim()) {
      try {
        html = await httpGetTextWithHeaders(originalUrl, {
          Cookie: xhsCookie,
          "User-Agent": UA_COMMON,
          Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
          "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
        });
      } catch {
        // 继续使用第一次抓到的 html
      }
    }
  }

  let [title, desc] = extractMetaTitleDesc(html);
  if (!desc) {
    // 再尝试从 Next.js 数据结构里抽取
    const next = extractFromNextData(html);
    title = title || next.title || "";
    desc = next.text || "";
  }
  if (!desc) {
    // 再尝试一些常见键名的字符串抽取（best-effort，避免过度依赖站点结构）
    const maybeKeysMap = {
      "酷安APP": ["desc", "description", "content", "feedDesc"],
      "快手APP": ["caption", "description", "text", "title", "feedDesc"],
      "今日头条": ["abstract", "description", "articleBody", "content", "text"],
      "小红书APP": ["noteContent", "desc", "description", "text", "content"],
    };
    const keys = maybeKeysMap[sourceChannel];
    if (keys) {
      desc = extractEscapedJsonAnyKey(html, keys);
      if (!title) title = extractEscapedJsonAnyKey(html, ["title", "headline", "name"]);
    }
  }

  // 今日头条：若仍然没有正文摘要，尝试 Playwright 渲染提取 innerText
  if (sourceChannel === "今日头条" && (!desc || desc.length < 10)) {
    // 1) 先尝试移动端/备用路径（不依赖 playwright）
    try {
      const alt = await tryExtractToutiaoViaAltUrls(originalUrl);
      if (alt.text && alt.text.length >= 10) {
        title = title || alt.title || "";
        desc = alt.text;
      }
    } catch {
      // ignore
    }
  }

  if (sourceChannel === "今日头条" && (!desc || desc.length < 10)) {
    // 2) 仍然没有，再尝试 Playwright 渲染提取 innerText（需要 playwright 依赖）
    try {
      const content = await extractToutiaoWithPlaywright(originalUrl);
      title = title || content.title || "";
      if (content.text && content.text.length) desc = content.text;
    } catch {
      // 忽略：playwright 可能未安装/站点反爬
    }
  }

  let visual = "";
  if (!desc && title) visual = `标题：${title}`;
  const defaults = inferDefaultFlagsFromChannel(sourceChannel);
  let hasImage;
  let hasVideo;
  let inferredMediaType;
  if (defaults) {
    hasImage = defaults.hasImage;
    hasVideo = defaults.hasVideo;
    inferredMediaType = inferMediaTypeFromFlags(hasImage, hasVideo);
  } else {
    const flags = inferMediaFlagsFromHtml(html);
    hasImage = flags.hasImage;
    hasVideo = flags.hasVideo;
    inferredMediaType = inferMediaTypeFromFlags(hasImage, hasVideo);
  }

  const mediaTypeFinal = inferredMediaType === "未知" ? mediaType : inferredMediaType;
  return {
    title: title || "",
    text: desc || "",
    visual,
    platform,
    mediaType: mediaTypeFinal,
    hasImage,
    hasVideo,
  };
}

function buildContentAnalysis({ title, text, platform, mediaType, visual }) {
  const res = buildAnalysis({
    title: title || "",
    text: text || "",
    platform: platform || "",
    mediaType: mediaType || "",
    visual: visual || "",
    maxChars: MAX_CHARS,
  });
  return res.analysis || "";
}

async function worker({ token, targets, idxStart, workerId }) {
  let idx = idxStart;
  const updates = [];
  while (idx < targets.length) {
    const t = targets[idx];
    idx += WORKERS;
    const { record_id, source_channel, original_url, need_content, need_link, need_img, need_video } = t;
    try {
      const parsed = await extractBySourceChannel(source_channel, original_url);
      const valid = true;

      const defaults = inferDefaultFlagsFromChannel(source_channel);
      const hasImage = !!parsed.hasImage;
      const hasVideo = !!parsed.hasVideo;

      const fields = {};
      if (need_link) fields[LINK_STATUS_FIELD] = valid ? "是" : "否";
      if (need_img) {
        if (defaults || hasImage) fields[HAS_IMAGE_FIELD] = hasImage ? "是" : "否";
      }
      if (need_video) {
        if (defaults || hasVideo) fields[HAS_VIDEO_FIELD] = hasVideo ? "是" : "否";
      }

      if (need_content) {
        const contentVal =
          parsed.title || parsed.text ? buildContentAnalysis(parsed) : "";
        if (contentVal) fields[CONTENT_ANALYSIS_FIELD] = contentVal;
      }

      if (Object.keys(fields).length) {
        updates.push({ record_id, fields });
      }
      console.log(`  worker#${workerId} ok record_id=${record_id}`);
    } catch (e) {
      console.log(`  worker#${workerId} err record_id=${record_id}: ${e && e.message ? e.message : String(e)}`);
      const defaults = inferDefaultFlagsFromChannel(source_channel);
      const hasImage = defaults ? defaults.hasImage : false;
      const hasVideo = defaults ? defaults.hasVideo : false;

      const fields = {};
      if (need_link) fields[LINK_STATUS_FIELD] = "否";
      // 仅当渠道已知“确定类型”时才回填图片/视频；否则避免误写“否”
      if (need_img && defaults) fields[HAS_IMAGE_FIELD] = hasImage ? "是" : "否";
      if (need_video && defaults) fields[HAS_VIDEO_FIELD] = hasVideo ? "是" : "否";
      if (Object.keys(fields).length) {
        updates.push({ record_id, fields });
      }
    }
  }
  return updates;
}

async function main() {
  const TEST_URL = (process.env.TEST_URL || "").trim();
  const TEST_CHANNEL = (process.env.TEST_CHANNEL || "").trim();
  if (TEST_URL && TEST_CHANNEL) {
    const testChannelNorm = normalizeChannelValue(TEST_CHANNEL);
    const parsed = await extractBySourceChannel(testChannelNorm, TEST_URL);
    const contentVal = buildContentAnalysis(parsed);
    const hasImage = !!parsed.hasImage;
    const hasVideo = !!parsed.hasVideo;
    console.log(
      JSON.stringify(
        {
          test_channel: TEST_CHANNEL,
          test_url: TEST_URL,
          parsed,
          content_analysis: contentVal,
          link_status: "是",
          has_image: hasImage ? "是" : "否",
          has_video: hasVideo ? "是" : "否",
        },
        null,
        2
      )
    );
    return;
  }

  if (!BITABLE_URL) throw new Error("请设置环境变量 BITABLE_URL（飞书多维表链接）");

  const token = await getTenantAccessToken();
  console.log("获取飞书 token 成功");

  const targets = await fetchTargets(token);
  console.log(`筛选到需要处理的记录：${targets.length} 条（渠道：${SOURCE_CHANNELS.join(", ")})`);
  if (!targets.length) return;

  const t0 = Date.now();
  const allUpdates = [];

  // 简单并发 worker：避免引入第三方依赖
  const workers = [];
  for (let w = 0; w < WORKERS; w++) {
    workers.push(worker({ token, targets, idxStart: w, workerId: w + 1 }));
  }
  const results = await Promise.all(workers);
  for (const arr of results) allUpdates.push(...arr);

  console.log(`解析成功需要回填：${allUpdates.length} 条（耗时 ${((Date.now() - t0) / 1000).toFixed(1)}s）`);
  if (!allUpdates.length) return;

  // 批量更新
  const chunkSize = 50;
  for (let i = 0; i < allUpdates.length; i += chunkSize) {
    await batchUpdateRecords(token, allUpdates.slice(i, i + chunkSize));
    console.log(`已回填 ${Math.min(i + chunkSize, allUpdates.length)}/${allUpdates.length} 条`);
  }

  console.log(`完成：成功回填 ${allUpdates.length} 条`);
}

main().catch((e) => {
  console.error(`执行失败: ${e && e.message ? e.message : String(e)}`);
  process.exit(1);
});



path: scripts/backfill-content-analysis-by-source.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
从飞书多维表批量读取记录，按「来源渠道」解析「原文URL」并回填「内容解析」。

覆盖渠道（默认）：
- 抖音APP
- 酷安APP
- 今日头条
- 快手APP
- 小红书APP

回填策略：
- 默认 ONLY_EMPTY=1：仅当「内容解析」为空时回填

回填内容：
- 使用与 scripts/build-content-analysis.js 基本一致的规则生成「100字以内内容解析」
- 解析来源优先：平台可直接从 URL 抓取的文字摘要（如 meta/描述/抖音 desc）
- 抓取失败不会覆盖现有「内容解析」
"""

import json
import os
import re
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from html import unescape
from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import parse_qs, urlparse

import requests


# ------------------ 飞书配置 ------------------
APP_ID = os.getenv("APP_ID", "").strip()
APP_SECRET = os.getenv("APP_SECRET", "").strip()
BITABLE_URL = os.getenv("BITABLE_URL", "").strip()

# ------------------ 业务字段（按实际多维表字段名可改） ------------------
SOURCE_CHANNEL_FIELD = os.getenv("SOURCE_CHANNEL_FIELD", "来源渠道").strip()
CONTENT_ANALYSIS_FIELD = os.getenv("CONTENT_ANALYSIS_FIELD", "内容解析").strip()

# 用户提到“获取原文URL”，但项目里通常叫“原文 URL”
ORIGINAL_URL_CANDIDATES = [
    x.strip()
    for x in [
        os.getenv("WEIBO_ORIGINAL_URL_FIELD", "").strip(),  # 用户可指定一个自定义字段名
        "获取原文URL",
        "原文 URL",
        "原文URL",
        "原文url",
        "doc_url",
    ]
    if x.strip()
]

# ------------------ 渠道过滤 ------------------
DEFAULT_CHANNELS = ["抖音APP", "酷安APP", "今日头条", "快手APP", "小红书APP"]
SOURCE_CHANNELS = [
    x.strip()
    for x in os.getenv("SOURCE_CHANNELS", ",".join(DEFAULT_CHANNELS)).split(",")
    if x.strip()
]
SOURCE_CHANNELS_SET = set(SOURCE_CHANNELS)

SOURCE_CHANNEL_VALUE_TO_PLATFORM = {
    "抖音APP": ("抖音", "视频"),
    "酷安APP": ("酷安", "未知"),
    "今日头条": ("今日头条", "图文"),
    "快手APP": ("快手", "视频"),
    "小红书APP": ("小红书", "图文"),
}

# ------------------ 行为参数 ------------------
ONLY_EMPTY = os.getenv("ONLY_EMPTY", "1").strip().lower() in ("1", "true", "yes")
MAX_TARGETS = int(os.getenv("LIMIT", "50"))
FETCH_PAGE_SIZE = int(os.getenv("PAGE_SIZE", "200"))
WORKERS = int(os.getenv("WORKERS", "4"))
MAX_CHARS = int(os.getenv("MAX_CHARS", "100"))

# 抓取相关
HTTP_TIMEOUT = int(os.getenv("HTTP_TIMEOUT", "15"))
RETRY_TIMES = int(os.getenv("HTTP_RETRY_TIMES", "1"))  # 失败重试次数（不建议太高）

SESSION: Optional[requests.Session] = None


def get_session() -> requests.Session:
    global SESSION
    if SESSION is None:
        SESSION = requests.Session()
    return SESSION


def _safe_text(v: Any) -> str:
    if v is None:
        return ""
    if isinstance(v, list):
        if not v:
            return ""
        v = v[0]
    if isinstance(v, dict):
        try:
            return json.dumps(v, ensure_ascii=False)
        except Exception:
            return str(v)
    return str(v).strip()


def parse_bitable_url(url: str) -> Tuple[str, str]:
    parsed = urlparse(url)
    app_token = ""
    parts = [p for p in parsed.path.split("/") if p]
    for i, p in enumerate(parts):
        if p == "base" and i + 1 < len(parts):
            app_token = parts[i + 1]
            break
    qs = parse_qs(parsed.query)
    table_id = qs.get("table", [None])[0]
    if not app_token or not table_id:
        raise RuntimeError(f"BITABLE_URL 解析失败: {url}")
    return app_token, table_id


def get_tenant_access_token() -> str:
    if not APP_ID or not APP_SECRET:
        raise RuntimeError("请设置环境变量 APP_ID / APP_SECRET")
    url = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal"
    payload = {"app_id": APP_ID, "app_secret": APP_SECRET}
    resp = get_session().post(url, json=payload, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    if data.get("code") != 0:
        raise RuntimeError(f"获取 tenant_access_token 失败: {data}")
    return data["tenant_access_token"]


def fetch_targets(token: str) -> List[Dict[str, Any]]:
    if not BITABLE_URL:
        raise RuntimeError("请设置环境变量 BITABLE_URL（飞书多维表链接）")

    app_token, table_id = parse_bitable_url(BITABLE_URL)
    url = f"https://open.feishu.cn/open-apis/bitable/v1/apps/{app_token}/tables/{table_id}/records"
    headers = {"Authorization": f"Bearer {token}"}
    params: Dict[str, Any] = {"page_size": min(500, FETCH_PAGE_SIZE)}

    targets: List[Dict[str, Any]] = []
    page_token: Optional[str] = None

    while True:
        if page_token:
            params["page_token"] = page_token
        resp = get_session().get(url, headers=headers, params=params, timeout=20)
        resp.raise_for_status()
        data = resp.json()
        if data.get("code") != 0:
            raise RuntimeError(f"获取记录失败: {data}")

        obj = data.get("data") or {}
        items = obj.get("items") or []
        for rec in items:
            fields = rec.get("fields") or {}
            ch = _safe_text(fields.get(SOURCE_CHANNEL_FIELD))
            if ch not in SOURCE_CHANNELS_SET:
                continue

            existing = _safe_text(fields.get(CONTENT_ANALYSIS_FIELD))
            if ONLY_EMPTY and existing:
                continue

            original_url = ""
            for cand in ORIGINAL_URL_CANDIDATES:
                if cand in fields:
                    original_url = _safe_text(fields.get(cand))
                    if original_url:
                        break
            if not original_url:
                continue

            record_id = rec.get("record_id")
            if not record_id:
                continue

            targets.append(
                {
                    "record_id": record_id,
                    "source_channel": ch,
                    "original_url": original_url,
                }
            )
            if len(targets) >= MAX_TARGETS:
                return targets

        page_token = obj.get("page_token")
        has_more = obj.get("has_more")
        if not has_more or not page_token:
            break

    return targets


def batch_update_records(token: str, updates: List[Dict[str, Any]]) -> None:
    if not updates:
        return
    if not BITABLE_URL:
        raise RuntimeError("BITABLE_URL 未初始化")
    app_token, table_id = parse_bitable_url(BITABLE_URL)
    url = f"https://open.feishu.cn/open-apis/bitable/v1/apps/{app_token}/tables/{table_id}/records/batch_update"
    headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
    payload = {"records": updates}
    resp = get_session().post(url, headers=headers, data=json.dumps(payload, ensure_ascii=False).encode("utf-8"), timeout=30)
    resp.raise_for_status()
    data = resp.json()
    if data.get("code") != 0:
        raise RuntimeError(f"批量更新失败: {data}")


# ------------------ 解析与分析：统一规则 ------------------
BRAND_KWS = ["小米", "小爱", "小爱同学", "Xiaomi", "Mi", "Redmi"]
SPLIT_RE = re.compile(r"[。！？!?；;\n]")
HTML_TAG_RE = re.compile(r"<[^>]*>")
URL_RE = re.compile(r"https?:\/\/\S+")
WS_RE = re.compile(r"\s+")


def normalize_text(input_text: Any) -> str:
    text = str(input_text or "")
    text = HTML_TAG_RE.sub(" ", text)
    text = URL_RE.sub(" ", text)
    text = WS_RE.sub(" ", text)
    return text.strip()


def split_sentences(text: str) -> List[str]:
    t = normalize_text(text)
    return [s.strip() for s in SPLIT_RE.split(t) if s.strip()]


def truncate_by_chars(text: str, max_chars: int) -> str:
    if len(text) <= max_chars:
        return text
    if max_chars <= 1:
        return "…"
    return text[: max_chars - 1] + "…"


def pick_top_sentence(sentences: List[str], keywords: List[str]) -> str:
    if not sentences:
        return ""
    best = sentences[0]
    best_score = -1.0
    for s in sentences:
        score = 0.0
        lower = s.lower()
        for kw in keywords:
            if kw.lower() in lower:
                score += 2
        if 12 <= len(s) <= 60:
            score += 1
        # 复刻 JS 的“字符类”正则行为（允许出现任一关键词字符即可加分）
        if re.search(r"[发布|体验|测评|开箱|对比|评测|升级|更新|功能]", s):
            score += 1
        if score > best_score:
            best_score = score
            best = s
    return best


def build_content_analysis(title: str, text: str, platform: str, media_type: str, visual: str, max_chars: int = 100) -> str:
    title_text = normalize_text(title)
    body_text = normalize_text(text)
    visual_text = normalize_text(visual)

    sents = split_sentences(f"{title_text}。{body_text}".strip("。"))
    core = pick_top_sentence(sents, BRAND_KWS) or title_text or body_text
    visual_core = pick_top_sentence(split_sentences(visual_text), BRAND_KWS)

    has_brand = any(kw in title_text or kw in body_text or kw in visual_text for kw in BRAND_KWS)
    has_visual = len(visual_text) > 0

    if has_brand:
        analysis = f"该内容与小米/小爱相关，核心信息：{core}"
        if has_visual:
            analysis += f"；画面显示：{visual_core or truncate_by_chars(visual_text, 28)}"
    elif has_visual:
        analysis = f"该内容未明显提及小米/小爱，核心信息：{core or '文本信息有限'}；画面显示：{visual_core or truncate_by_chars(visual_text, 28)}"
    else:
        analysis = f"该内容核心信息：{core or '文本信息有限，建议补充原文或媒体素材后复核'}"

    return truncate_by_chars(analysis, max_chars)


# ------------------ 文本抽取（按平台） ------------------
UA_COMMON = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)


def _strip_html_like(s: str) -> str:
    t = unescape(s or "")
    t = HTML_TAG_RE.sub("", t)
    t = WS_RE.sub(" ", t).strip()
    return t


def http_get_text(url: str) -> str:
    headers = {"User-Agent": UA_COMMON, "Accept": "text/html,application/json,*/*"}
    last_err: Optional[Exception] = None
    for attempt in range(RETRY_TIMES + 1):
        try:
            resp = get_session().get(url, headers=headers, timeout=HTTP_TIMEOUT, allow_redirects=True)
            resp.raise_for_status()
            return resp.text or ""
        except Exception as e:
            last_err = e
            time.sleep(0.6 * (2 ** attempt))
    if last_err:
        raise last_err
    return ""


def extract_meta_description(html: str) -> Tuple[str, str]:
    # og:title
    def _meta_content(pattern: str) -> str:
        m = re.search(pattern, html, flags=re.IGNORECASE)
        if not m:
            return ""
        return _strip_html_like(m.group(1))

    title = _meta_content(r'<meta[^>]+property=["\']og:title["\'][^>]+content=["\']([^"\']+)["\']')
    if not title:
        title = _meta_content(r"<meta[^>]+name=['\"]title['\"][^>]+content=['\"]([^'\"]+)['\"]")
    if not title:
        m = re.search(r"<title>([^<]+)</title>", html, flags=re.IGNORECASE)
        if m:
            title = _strip_html_like(m.group(1))

    desc = _meta_content(r'<meta[^>]+property=["\']og:description["\'][^>]+content=["\']([^"\']+)["\']')
    if not desc:
        desc = _meta_content(r"<meta[^>]+name=['\"]description['\"][^>]+content=['\"]([^'\"]+)['\"]")

    # JSON-LD fallback
    if not desc:
        for m in re.finditer(r"<script[^>]+type=['\"]application/ld\+json['\"][^>]*>([\s\S]*?)</script>", html, flags=re.IGNORECASE):
            raw = m.group(1).strip()
            if not raw:
                continue
            try:
                obj = json.loads(raw)
                if isinstance(obj, dict):
                    desc = _strip_html_like(obj.get("description", "") or "")
                    title = title or _strip_html_like(obj.get("name", "") or obj.get("headline", "") or "")
                    if desc:
                        break
            except Exception:
                continue

    return title, desc


def extract_douyin_desc_from_url(url: str) -> Tuple[str, str]:
    """
    复刻 scripts/parse-douyin-video.js 的逻辑（但只取 desc 文本，不下载视频）。
    返回 (title, text)
    """
    # 1) 从 URL 抽取 modal_id / awemeId
    aweme_id = ""
    m = re.search(r"modal_id=([0-9]+)", url)
    if m:
        aweme_id = m.group(1)
    else:
        m = re.search(r"video/([^/?]+)", url)
        if m:
            aweme_id = m.group(1)
        else:
            m = re.search(r"note/([^/?]+)", url)
            if m:
                aweme_id = m.group(1)

    if not aweme_id:
        raise RuntimeError(f"无法从抖音 URL 中提取视频ID: {url}")

    request_url = f"https://www.douyin.com/jingxuan?modal_id={aweme_id}"
    headers = {
        "User-Agent": UA_COMMON,
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "zh-CN,zh;q=0.9",
        "Referer": "https://www.douyin.com/",
        "Connection": "keep-alive",
    }

    resp = get_session().get(request_url, headers=headers, timeout=HTTP_TIMEOUT, allow_redirects=True)
    resp.raise_for_status()
    html_content = resp.text or ""

    # 复刻 JS 的提取：在 HTML 中找 JSON 字符串片段（playAddr/searchProps/app）
    pattern = re.compile(r'"([^"]*?(?:playAddr|searchProps|app)[^"]*?)"', flags=re.DOTALL)
    matches = pattern.findall(html_content)
    target_match = ""
    for m in matches:
        if "playAddr" in m and "searchProps" in m and "app" in m:
            target_match = m
            break
    if not target_match:
        raise RuntimeError("未找到包含视频数据的 JSON 片段")

    # decodeURIComponent
    from urllib.parse import unquote

    decoded_json = unquote(target_match)
    video_data_json = json.loads(decoded_json)
    video_detail = ((video_data_json or {}).get("app") or {}).get("videoDetail") or {}
    desc = video_detail.get("desc") or ""
    return desc, desc


def extract_weibo_body(weibo_url: str) -> Tuple[str, str]:
    # 如果你只要多渠道，不传 weibo，这个也可忽略；先保留稳定实现
    # 抽 weiboId
    m = re.search(r"weibo\.com/\d+/([a-zA-Z0-9]+)", weibo_url)
    if not m:
        m = re.search(r"m\.weibo\.cn/status/([a-zA-Z0-9]+)", weibo_url)
    if not m:
        m = re.search(r"weibo\.com/\d+/([a-zA-Z0-9]+)\?", weibo_url)
    if not m:
        # fallback: 最后一段
        parts = [p for p in weibo_url.split("/") if p]
        if parts:
            last = parts[-1]
            if "?" not in last:
                m = re.match(r"(.+)", last)
    if not m:
        raise RuntimeError(f"无法从微博 URL 提取 weiboId: {weibo_url}")
    weibo_id = m.group(1)

    api_url = f"https://m.weibo.cn/statuses/show?id={weibo_id}"
    headers = {
        "User-Agent": UA_COMMON,
        "Referer": "https://m.weibo.cn/",
        "X-Requested-With": "XMLHttpRequest",
    }
    resp = get_session().get(api_url, headers=headers, timeout=HTTP_TIMEOUT)
    resp.raise_for_status()
    data = resp.json()
    if data.get("ok") != 1:
        raise RuntimeError(f"Weibo API error: {data}")
    obj = data.get("data") or {}
    text = obj.get("text") or ""
    if not text and isinstance(obj.get("longText"), str):
        text = obj.get("longText")
    return "", _strip_html_like(text)


def extract_by_channel(source_channel: str, original_url: str) -> Tuple[str, str, str, str, str]:
    """
    返回：
    - title
    - text（正文/摘要）
    - visual（可选：用于 build-content-analysis 的画面证据）
    - platform（用于 evidence）
    - media_type（用于 evidence）
    """
    platform, media_type = SOURCE_CHANNEL_VALUE_TO_PLATFORM.get(source_channel, (source_channel, "未知"))

    # 优先按渠道抓取
    if source_channel == "抖音APP":
        title, text = extract_douyin_desc_from_url(original_url)
        visual = ""
        return title or "", text or "", visual, platform, media_type

    # 其他渠道：meta/json-ld best effort
    html = http_get_text(original_url)
    title, desc = extract_meta_description(html)
    visual = ""
    # 简单兜底：如果只拿到 title，visual 也可以提供一定信息
    if not desc and title:
        visual = f"标题：{title}"

    # 特殊：如果酷安/快手 URL 返回的是 JSON/渲染失败页面，desc 可能为空
    # 这里不强行写空内容交由上层判断
    return title or "", desc or "", visual, platform, media_type


def build_analysis_for_record(source_channel: str, original_url: str) -> str:
    title, text, visual, platform, media_type = extract_by_channel(source_channel, original_url)
    # 只要 text/title 非空就尽量产出解析；否则直接返回空（由上层决定是否跳过写回）
    if not title and not text:
        return ""
    return build_content_analysis(
        title=title,
        text=text,
        platform=platform,
        media_type=media_type,
        visual=visual,
        max_chars=MAX_CHARS,
    )


def main() -> None:
    if not BITABLE_URL:
        raise RuntimeError("请设置环境变量 BITABLE_URL（飞书多维表链接）")

    [REDACTED_SECRET])
    print(f"获取飞书 token 成功")

    targets = fetch_targets(token)
    print(f"筛选到需要处理的记录：{len(targets)} 条（渠道：{', '.join(SOURCE_CHANNELS)})")
    if not targets:
        return

    updates: List[Dict[str, Any]] = []
    errors: List[Tuple[str, str]] = []

    def _task(t: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        record_id = t["record_id"]
        ch = t["source_channel"]
        url = t["original_url"]
        content_val = build_analysis_for_record(ch, url)
        if not content_val:
            return None
        return {"record_id": record_id, "fields": {CONTENT_ANALYSIS_FIELD: content_val}}

    with ThreadPoolExecutor(max_workers=max(1, WORKERS)) as ex:
        fut_to_t = {ex.submit(_task, t): t for t in targets}
        done = 0
        total = len(fut_to_t)
        for fut in as_completed(fut_to_t):
            done += 1
            t = fut_to_t[fut]
            rid = t["record_id"]
            try:
                u = fut.result()
                if u:
                    updates.append(u)
                print(f"  [{done}/{total}] record_id={rid} ok")
            except Exception as e:
                errors.append((rid, str(e)))
                print(f"  [{done}/{total}] record_id={rid} err: {e}")

    if updates:
        chunk_size = 50
        for i in range(0, len(updates), chunk_size):
            batch_update_records(token, updates[i : i + chunk_size])
            print(f"已回填 {min(i + chunk_size, len(updates))}/{len(updates)} 条")

    if errors:
        print("\n回填失败（前 10 条）：")
        for rid, msg in errors[:10]:
            print(f"- record_id={rid}: {msg}")
        print(f"失败总数：{len(errors)}")

    print(f"\n完成：成功回填 {len(updates)} 条，失败 {len(errors)} 条")


if __name__ == "__main__":
    try:
        main()
    except Exception as e:
        print(f"执行失败: {e}")
        raise



path: scripts/backfill-weibo-content-analysis.js
content: #!/usr/bin/env node
/**
 * 微博正文回填「内容解析」（纯 Node 版本）
 *
 * - 过滤：来源渠道 == "新浪微博"
 * - 读取：原文 URL（候选字段：获取原文URL/原文 URL/原文URL/原文url/doc_url）
 * - 抓取：m.weibo.cn/statuses/show?id={weiboId}
 * - 写回：内容解析（默认只写空值）
 */

const { buildAnalysis } = require("./build-content-analysis");

function env(key, defaultValue = "") {
  const v = process.env[key] ?? process.env[`INPUT_${String(key).toUpperCase()}`];
  return String(v ?? defaultValue).trim();
}

const APP_ID = env("APP_ID");
const APP_SECRET = env("APP_SECRET");
const BITABLE_URL = env("BITABLE_URL");

const SOURCE_CHANNEL_FIELD = env("SOURCE_CHANNEL_FIELD", "来源渠道");
const CONTENT_ANALYSIS_FIELD = env("CONTENT_ANALYSIS_FIELD", "内容解析");
const LINK_STATUS_FIELD = env("LINK_STATUS_FIELD", "链接是否有效");
const HAS_IMAGE_FIELD = env("HAS_IMAGE_FIELD", "是否包含图片");
const HAS_VIDEO_FIELD = env("HAS_VIDEO_FIELD", "是否包含视频");

const ONLY_EMPTY = ["1", "true", "yes"].includes(env("ONLY_EMPTY", "1").toLowerCase());
const MAX_TARGETS = parseInt(env("LIMIT", "50"), 10);
const PAGE_SIZE = parseInt(env("PAGE_SIZE", "200"), 10);
const MAX_CHARS = parseInt(env("MAX_CHARS", "100"), 10);
const WORKERS = parseInt(env("WORKERS", "4"), 10);

const HTTP_TIMEOUT_MS = parseInt(env("HTTP_TIMEOUT_MS", "15000"), 10);
const HTTP_RETRY_TIMES = parseInt(env("HTTP_RETRY_TIMES", "1"), 10);

const UA_COMMON =
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";

const ORIGINAL_URL_CANDIDATES = [
  // 兼容 Clawhub 输入：original_url_field
  env("ORIGINAL_URL_FIELD") || env("WEIBO_ORIGINAL_URL_FIELD"),
  "获取原文URL",
  "原文 URL",
  "原文URL",
  "原文url",
  "doc_url",
].filter(Boolean);

const TARGET_SOURCE_CHANNEL = env("SOURCE_CHANNEL_VALUE", "新浪微博");

function parseBitableUrl(url) {
  const parsed = new URL(url);
  let appToken = "";
  const parts = parsed.pathname.split("/").filter(Boolean);
  for (let i = 0; i < parts.length; i++) {
    if (parts[i] === "base" && i + 1 < parts.length) {
      appToken = parts[i + 1];
      break;
    }
  }
  const tableId = parsed.searchParams.get("table");
  if (!appToken || !tableId) throw new Error(`BITABLE_URL 解析失败: ${url}`);
  return [appToken, tableId];
}

async function withRetry(fn) {
  let lastErr = null;
  for (let attempt = 0; attempt <= HTTP_RETRY_TIMES; attempt++) {
    try {
      return await fn();
    } catch (e) {
      lastErr = e;
      const backoff = 600 * Math.pow(2, attempt);
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
  throw lastErr;
}

function stripHtmlLike(s) {
  return String(s || "")
    .replace(/<[^>]*>/g, "")
    .replace(/&quot;/g, '"')
    .replace(/&#39;/g, "'")
    .replace(/&amp;/g, "&")
    .replace(/\s+/g, " ")
    .trim();
}

function extractWeiboId(url) {
  let m = url.match(/weibo\.com\/\d+\/([a-zA-Z0-9]+)/);
  if (m) return m[1];
  m = url.match(/m\.weibo\.cn\/status\/([a-zA-Z0-9]+)/);
  if (m) return m[1];
  m = url.match(/weibo\.com\/\d+\/([a-zA-Z0-9]+)\?/);
  if (m) return m[1];
  const parts = url.split("/").filter(Boolean);
  const last = parts[parts.length - 1];
  if (last && !last.includes("?")) return last;
  return null;
}

async function httpGet(url) {
  return withRetry(async () => {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), HTTP_TIMEOUT_MS);
    try {
      const resp = await fetch(url, {
        method: "GET",
        headers: {
          "User-Agent": UA_COMMON,
          Accept: "application/json,text/html,*/*",
          Referer: "https://m.weibo.cn/",
          "X-Requested-With": "XMLHttpRequest",
        },
        redirect: "follow",
        signal: ctrl.signal,
      });
      if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
      return await resp.json();
    } finally {
      clearTimeout(t);
    }
  });
}

async function getTenantAccessToken() {
  if (!APP_ID || !APP_SECRET) throw new Error("请设置环境变量 APP_ID / APP_SECRET");
  const url = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal";
  const resp = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ app_id: APP_ID, app_secret: APP_SECRET }),
    redirect: "follow",
  });
  if (!resp.ok) throw new Error(`tenant_access_token HTTP ${resp.status}`);
  const data = await resp.json();
  if (data.code !== 0) throw new Error(`获取 tenant_access_token 失败: ${JSON.stringify(data)}`);
  return data.tenant_access_token;
}

function safeText(v) {
  if (v == null) return "";
  if (Array.isArray(v)) return v.length ? String(v[0]).trim() : "";
  if (typeof v === "object") return JSON.stringify(v);
  return String(v).trim();
}

async function fetchTargets(token) {
  if (!BITABLE_URL) throw new Error("请设置环境变量 BITABLE_URL（飞书多维表链接）");
  const [appToken, tableId] = parseBitableUrl(BITABLE_URL);
  const url = `https://open.feishu.cn/open-apis/bitable/v1/apps/${appToken}/tables/${tableId}/records`;
  const headers = { Authorization: `Bearer ${token}` };

  const targets = [];
  let pageToken = null;
  while (targets.length < MAX_TARGETS) {
    const params = new URLSearchParams();
    params.set("page_size", String(Math.min(500, PAGE_SIZE)));
    if (pageToken) params.set("page_token", pageToken);
    const resp = await fetch(`${url}?${params.toString()}`, { method: "GET", headers, redirect: "follow" });
    if (!resp.ok) throw new Error(`获取记录失败 HTTP ${resp.status}`);
    const data = await resp.json();
    if (data.code !== 0) throw new Error(`获取记录失败: ${JSON.stringify(data)}`);
    const obj = data.data || {};
    const items = obj.items || [];
    for (const rec of items) {
      const fields = rec.fields || {};
      const ch = safeText(fields[SOURCE_CHANNEL_FIELD]);
      if (ch !== TARGET_SOURCE_CHANNEL) continue;
      const existingContent = safeText(fields[CONTENT_ANALYSIS_FIELD]);
      const existingLink = safeText(fields[LINK_STATUS_FIELD]);
      const existingImg = safeText(fields[HAS_IMAGE_FIELD]);
      const existingVid = safeText(fields[HAS_VIDEO_FIELD]);

      const needsContent = !existingContent;
      const needsLink = !existingLink;
      const needsImg = !existingImg;
      const needsVid = !existingVid;

      if (ONLY_EMPTY && !(needsContent || needsLink || needsImg || needsVid)) continue;

      let originalUrl = "";
      for (const cand of ORIGINAL_URL_CANDIDATES) {
        if (fields[cand] != null) {
          const v = safeText(fields[cand]);
          if (v) {
            originalUrl = v;
            break;
          }
        }
      }
      if (!originalUrl) continue;
      if (!rec.record_id) continue;

      targets.push({
        record_id: rec.record_id,
        original_url: originalUrl,
        need_content: needsContent,
        need_link: needsLink,
        need_img: needsImg,
        need_video: needsVid,
      });
      if (targets.length >= MAX_TARGETS) break;
    }

    if (!obj.has_more || !obj.page_token) break;
    pageToken = obj.page_token;
  }
  return targets;
}

async function batchUpdate(token, updates) {
  if (!updates.length) return;
  const [appToken, tableId] = parseBitableUrl(BITABLE_URL);
  const url = `https://open.feishu.cn/open-apis/bitable/v1/apps/${appToken}/tables/${tableId}/records/batch_update`;
  const resp = await fetch(url, {
    method: "POST",
    headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
    body: JSON.stringify({ records: updates }),
  });
  if (!resp.ok) throw new Error(`批量更新失败 HTTP ${resp.status}`);
  const data = await resp.json();
  if (data.code !== 0) throw new Error(`批量更新失败: ${JSON.stringify(data)}`);
}

async function extractWeiboText(originalUrl) {
  const weiboId = extractWeiboId(originalUrl);
  if (!weiboId) throw new Error(`无法从微博 URL 提取 weiboId: ${originalUrl}`);
  const apiUrl = `https://m.weibo.cn/statuses/show?id=${weiboId}`;
  const data = await httpGet(apiUrl);
  if (data.ok !== 1) throw new Error(`Weibo API error: ${JSON.stringify(data)}`);
  const obj = data.data || {};
  const text = stripHtmlLike(obj.text || obj.longText || "");
  const picNum = obj.pic_num || 0;
  return { text, picNum };
}

function buildContentAnalysis({ text, picNum }) {
  if (!text) return "";
  const mediaType = picNum > 0 ? "图片" : "视频";
  const res = buildAnalysis({
    title: "",
    text,
    platform: "微博",
    mediaType,
    visual: "",
    maxChars: MAX_CHARS,
  });
  return res.analysis || "";
}

async function main() {
  // 本地烟测模式：不依赖飞书多维表
  const TEST_URL = (env("TEST_URL", "") || "").trim();
  const TEST_SOURCE_CHANNEL = (env("TEST_SOURCE_CHANNEL", "") || "").trim();
  if (TEST_URL && TEST_SOURCE_CHANNEL) {
    const { text, picNum } = await extractWeiboText(TEST_URL);
    const content = buildContentAnalysis({ text, picNum });
    const hasImage = picNum > 0;
    const hasVideo = picNum <= 0;
    console.log(
      JSON.stringify(
        {
          test_source_channel: TEST_SOURCE_CHANNEL,
          test_url: TEST_URL,
          text_preview: String(text || "").slice(0, 80),
          content_analysis: content,
          link_status: "是",
          has_image: hasImage ? "是" : "否",
          has_video: hasVideo ? "是" : "否",
        },
        null,
        2
      )
    );
    return;
  }

  const token = await getTenantAccessToken();
  console.log("获取飞书 token 成功");

  const targets = await fetchTargets(token);
  console.log(`筛选到需要处理的微博记录：${targets.length} 条`);
  if (!targets.length) return;

  const updates = [];
  const workers = [];
  for (let w = 0; w < WORKERS; w++) {
    workers.push(
      (async () => {
        for (let i = w; i < targets.length; i += WORKERS) {
          const t = targets[i];
          try {
            const { text, picNum } = await extractWeiboText(t.original_url);
            const valid = true;
            const hasImage = picNum > 0;
            const hasVideo = picNum <= 0;

            const fields = {};
            if (t.need_link) fields[LINK_STATUS_FIELD] = valid ? "是" : "否";
            if (t.need_img) fields[HAS_IMAGE_FIELD] = hasImage ? "是" : "否";
            if (t.need_video) fields[HAS_VIDEO_FIELD] = hasVideo ? "是" : "否";

            if (t.need_content) {
              const content = buildContentAnalysis({ text, picNum });
              if (content) fields[CONTENT_ANALYSIS_FIELD] = content;
            }

            if (Object.keys(fields).length) {
              updates.push({ record_id: t.record_id, fields });
            }
            console.log(`  worker#${w + 1} ok record_id=${t.record_id}`);
          } catch (e) {
            console.log(`  worker#${w + 1} err record_id=${t.record_id}: ${e?.message || String(e)}`);
            if (t.need_link) {
              updates.push({
                record_id: t.record_id,
                fields: { [LINK_STATUS_FIELD]: "否" },
              });
            }
          }
        }
      })()
    );
  }
  await Promise.all(workers);

  console.log(`解析成功需回填：${updates.length} 条`);
  const chunkSize = 50;
  for (let i = 0; i < updates.length; i += chunkSize) {
    await batchUpdate(token, updates.slice(i, i + chunkSize));
    console.log(`已回填 ${Math.min(i + chunkSize, updates.length)}/${updates.length} 条`);
  }

  console.log(`完成：成功回填 ${updates.length} 条`);
}

main().catch((e) => {
  console.error(`执行失败: ${e?.message || String(e)}`);
  process.exit(1);
});



path: scripts/backfill-weibo-content-analysis.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
将“新浪微博”的微博正文解析结果回填到飞书多维表字段「内容解析」。

筛选逻辑：
- 来源渠道 == "新浪微博"
- 内容解析 为空时才回填（可用 ONLY_EMPTY=0 强制覆盖）

解析逻辑：
- 从记录字段中读取微博原文URL（优先字段名：获取原文URL，其次原文 URL/原文URL/原文url 等）
- 提取 weiboId
- 调用：https://m.weibo.cn/statuses/show?id={weiboId}
- 取返回数据中的 data.text，并去掉尖括号内容（近似去 HTML 标签）
- 截断到 MAX_CHARS（默认 100）
"""

import json
import os
import re
import sys
import time
from html import unescape
from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import parse_qs, urlparse

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed


APP_ID = os.getenv("APP_ID", "").strip()
APP_SECRET = os.getenv("APP_SECRET", "").strip()
BITABLE_URL = os.getenv("BITABLE_URL", "").strip()

# 业务字段名（以你们多维表实际为准；脚本也会做候选字段兜底）
SOURCE_CHANNEL_FIELD = os.getenv("SOURCE_CHANNEL_FIELD", "来源渠道").strip()  # 字段名
CONTENT_ANALYSIS_FIELD = os.getenv("CONTENT_ANALYSIS_FIELD", "内容解析").strip()  # 字段名

# 你提到的“获取原文URL”（可能也有人用“原文 URL”）
WEIBO_ORIGINAL_URL_CANDIDATES = [
    os.getenv("WEIBO_ORIGINAL_URL_FIELD", "").strip() or "",  # 用户可显式指定
    "获取原文URL",
    "原文 URL",
    "原文URL",
    "原文url",
]
WEIBO_ORIGINAL_URL_CANDIDATES = [x for x in WEIBO_ORIGINAL_URL_CANDIDATES if x]

SOURCE_CHANNEL_VALUE = os.getenv("SOURCE_CHANNEL_VALUE", "新浪微博").strip()

ONLY_EMPTY = os.getenv("ONLY_EMPTY", "1").strip().lower() in ("1", "true", "yes")
MAX_CHARS = int(os.getenv("MAX_CHARS", "100"))
LIMIT = int(os.getenv("LIMIT", "50"))  # 本次最多处理多少条匹配记录
PAGE_SIZE = int(os.getenv("PAGE_SIZE", "500"))
WORKERS = int(os.getenv("WORKERS", "3"))

WEIBO_REQUEST_TIMEOUT = int(os.getenv("WEIBO_REQUEST_TIMEOUT", "15"))
WEIBO_MAX_RETRIES = int(os.getenv("WEIBO_MAX_RETRIES", "2"))


def _safe_text(v: Any) -> str:
    if v is None:
        return ""
    if isinstance(v, list):
        if not v:
            return ""
        v = v[0]
    if isinstance(v, dict):
        return json.dumps(v, ensure_ascii=False)
    return str(v).strip()


def parse_bitable_url(url: str) -> Tuple[str, str]:
    parsed = urlparse(url)
    app_token = ""
    parts = [p for p in parsed.path.split("/") if p]
    for i, p in enumerate(parts):
        if p == "base" and i + 1 < len(parts):
            app_token = parts[i + 1]
            break
    qs = parse_qs(parsed.query)
    table_id = qs.get("table", [None])[0]
    if not app_token or not table_id:
        raise RuntimeError(f"BITABLE_URL 解析失败: {url}")
    return app_token, table_id


def get_session() -> requests.Session:
    s = requests.Session()
    return s


def get_tenant_access_token() -> str:
    if not APP_ID or not APP_SECRET:
        raise RuntimeError("请设置环境变量 APP_ID / APP_SECRET")
    url = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal"
    payload = {"app_id": APP_ID, "app_secret": APP_SECRET}
    resp = get_session().post(url, json=payload, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    if data.get("code") != 0:
        raise RuntimeError(f"获取 tenant_access_token 失败: {data}")
    return data["tenant_access_token"]


def pick_first_field(fields: Dict[str, Any], candidates: List[str]) -> str:
    for c in candidates:
        if c in fields:
            v = _safe_text(fields.get(c))
            if v:
                return v
    return ""


def extract_weibo_id(url: str) -> Optional[str]:
    if not url:
        return None

    patterns = [
        # https://weibo.com/7976115583/QwgTm2OVI
        re.compile(r"weibo\.com\/\d+\/([a-zA-Z0-9]+)"),
        # https://m.weibo.cn/status/xxxxx
        re.compile(r"m\.weibo\.cn\/status\/([a-zA-Z0-9]+)"),
        # 带参数的情况
        re.compile(r"weibo\.com\/\d+\/([a-zA-Z0-9]+)\?"),
    ]
    for pat in patterns:
        m = pat.search(url)
        if m:
            return m.group(1)

    # fallback: 最后一段（不含 '?'）
    parts = [p for p in url.split("/") if p]
    if not parts:
        return None
    last = parts[-1]
    if last and "?" not in last:
        return last
    return None


_HTML_TAG_RE = re.compile(r"<[^>]*>")


def strip_html_like(text: str) -> str:
    # 微博返回 text 往往包含尖括号包装的信息，这里用正则去掉
    t = unescape(text or "")
    t = _HTML_TAG_RE.sub("", t)
    # 收敛连续空白
    t = re.sub(r"\s+", " ", t).strip()
    return t


def fetch_weibo_text(weibo_id: str) -> str:
    api_url = f"https://m.weibo.cn/statuses/show?id={weibo_id}"

    last_err: Optional[Exception] = None
    for attempt in range(WEIBO_MAX_RETRIES + 1):
        try:
            headers = {
                "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15",
                "Referer": "https://m.weibo.cn/",
                "X-Requested-With": "XMLHttpRequest",
            }
            resp = get_session().get(api_url, headers=headers, timeout=WEIBO_REQUEST_TIMEOUT)
            resp.raise_for_status()
            data = resp.json()
            if data.get("ok") != 1:
                raise RuntimeError(f"Weibo API error: {data}")
            obj = data.get("data") or {}
            text = obj.get("text") or ""
            # 有些长文接口可能在 longText 出现（尽量兜底）
            if not text and isinstance(obj.get("longText"), str):
                text = obj.get("longText")
            return strip_html_like(text)
        except Exception as e:
            last_err = e
            # 退避一点点，避免瞬时限流
            time.sleep(0.8 * (2 ** attempt))
    if last_err:
        raise last_err
    return ""


def fetch_records(token: str) -> List[Dict[str, Any]]:
    if not BITABLE_URL:
        raise RuntimeError("请设置环境变量 BITABLE_URL（飞书多维表链接）")
    app_token, table_id = parse_bitable_url(BITABLE_URL)
    url = f"https://open.feishu.cn/open-apis/bitable/v1/apps/{app_token}/tables/{table_id}/records"
    headers = {"Authorization": f"Bearer {token}"}
    params: Dict[str, Any] = {"page_size": PAGE_SIZE}

    result: List[Dict[str, Any]] = []
    page_token: Optional[str] = None
    while True:
        if page_token:
            params["page_token"] = page_token
        resp = get_session().get(url, headers=headers, params=params, timeout=15)
        resp.raise_for_status()
        data = resp.json()
        if data.get("code") != 0:
            raise RuntimeError(f"获取记录失败: {data}")

        obj = data.get("data") or {}
        items = obj.get("items") or []
        result.extend(items)

        page_token = obj.get("page_token")
        has_more = obj.get("has_more")
        if not has_more or not page_token:
            break
        if len(result) >= LIMIT:
            # 这里直接按 LIMIT 限制，避免拉太多
            result = result[:LIMIT]
            break
    return result


def batch_update_records(token: str, updates: List[Dict[str, Any]]) -> None:
    if not updates:
        return
    if not BITABLE_URL:
        raise RuntimeError("BITABLE_URL 未初始化")
    app_token, table_id = parse_bitable_url(BITABLE_URL)
    url = f"https://open.feishu.cn/open-apis/bitable/v1/apps/{app_token}/tables/{table_id}/records/batch_update"
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json",
    }
    payload = {"records": updates}
    resp = get_session().post(
        url,
        headers=headers,
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
        timeout=30,
    )
    resp.raise_for_status()
    data = resp.json()
    if data.get("code") != 0:
        raise RuntimeError(f"批量更新失败: {data}")


def build_content_value(body: str) -> str:
    body = strip_html_like(body)
    if MAX_CHARS > 0 and len(body) > MAX_CHARS:
        return body[: MAX_CHARS - 1] + "…"
    return body


def main() -> None:
    if not BITABLE_URL:
        raise RuntimeError("请设置环境变量 BITABLE_URL（飞书多维表链接）")

    [REDACTED_SECRET])
    print(f"获取飞书 token 成功")

    records = fetch_records(token)
    print(f"共拉取 {len(records)} 条记录（含分页，已按 LIMIT 限制）")

    # 1) 先在本地筛选出需要的微博记录
    targets: List[Dict[str, Any]] = []
    for rec in records:
        fields = rec.get("fields") or {}
        channel = _safe_text(fields.get(SOURCE_CHANNEL_FIELD))
        if channel != SOURCE_CHANNEL_VALUE:
            continue

        existing = _safe_text(fields.get(CONTENT_ANALYSIS_FIELD))
        if ONLY_EMPTY and existing:
            continue

        weibo_url = pick_first_field(fields, WEIBO_ORIGINAL_URL_CANDIDATES)
        if not weibo_url:
            continue

        record_id = rec.get("record_id")
        if not record_id:
            continue

        targets.append(
            {
                "record_id": record_id,
                "weibo_url": weibo_url,
            }
        )

    print(f"筛选到需要回填的记录：{len(targets)} 条（来源渠道={SOURCE_CHANNEL_VALUE}）")
    if not targets:
        return

    # 2) 并发抓取微博正文
    updates: List[Dict[str, Any]] = []
    errors: List[Tuple[str, str]] = []  # (record_id, err)

    def _task(item: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        record_id = item["record_id"]
        weibo_url = item["weibo_url"]
        weibo_id = extract_weibo_id(weibo_url)
        if not weibo_id:
            raise RuntimeError(f"无法从微博 URL 提取 weiboId: {weibo_url}")
        body = fetch_weibo_text(weibo_id)
        if not body:
            # 微博正文为空时也不强写，避免覆盖成空
            return None
        content_val = build_content_value(body)
        return {"record_id": record_id, "fields": {CONTENT_ANALYSIS_FIELD: content_val}}

    with ThreadPoolExecutor(max_workers=max(1, WORKERS)) as ex:
        fut_to_item = {ex.submit(_task, t): t for t in targets}
        completed = 0
        total = len(fut_to_item)
        for fut in as_completed(fut_to_item):
            completed += 1
            item = fut_to_item[fut]
            rid = item["record_id"]
            try:
                u = fut.result()
                if u:
                    updates.append(u)
                print(f"  [{completed}/{total}] record_id={rid} ok")
            except Exception as e:
                errors.append((rid, str(e)))
                print(f"  [{completed}/{total}] record_id={rid} err: {e}")

    # 3) 批量回填（分片）
    if updates:
        chunk_size = 50
        for i in range(0, len(updates), chunk_size):
            batch_update_records(token, updates[i : i + chunk_size])
            print(f"已回填 {min(i + chunk_size, len(updates))}/{len(updates)} 条")

    if errors:
        print("\n以下记录回填失败（仅输出前 10 条）：")
        for rid, msg in errors[:10]:
            print(f"- record_id={rid}: {msg}")

    print(f"\n完成：成功回填 {len(updates)} 条，失败 {len(errors)} 条")


if __name__ == "__main__":
    try:
        main()
    except Exception as e:
        print(f"执行失败: {e}", file=sys.stderr)
        sys.exit(1)



path: scripts/build-content-analysis.js
content: #!/usr/bin/env node
/**
 * 内容解析生成器（统一版）
 *
 * 目标：
 * 1) 保证每条记录都有“内容解析”
 * 2) 优先输出与品牌相关且有证据的信息
 * 3) 严格控制在 100 字以内
 *
 * 用法：
 * node build-content-analysis.js \
 *   --title "标题" \
 *   --text "正文" \
 *   --platform "微博" \
 *   --media-type "视频" \
 *   --visual "画面中出现小米手机开箱，桌面有小爱音箱"
 *
 * 输出：
 * {
 *   "analysis": "100字以内内容解析",
 *   "evidence": { ... },
 *   "score": 0.86
 * }
 */

function parseArgs(argv) {
  const args = {};
  for (let i = 2; i < argv.length; i++) {
    const cur = argv[i];
    if (!cur.startsWith("--")) continue;
    const key = cur.slice(2);
    const next = argv[i + 1];
    if (next && !next.startsWith("--")) {
      args[key] = next;
      i++;
    } else {
      args[key] = "true";
    }
  }
  return args;
}

function normalizeText(input) {
  return String(input || "")
    .replace(/<[^>]*>/g, " ")
    .replace(/https?:\/\/\S+/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

function splitSentences(text) {
  return normalizeText(text)
    .split(/[。！？!?；;\n]/)
    .map((s) => s.trim())
    .filter(Boolean);
}

function truncateByChars(text, maxChars) {
  const src = String(text || "");
  if (src.length <= maxChars) return src;
  return `${src.slice(0, maxChars - 1)}…`;
}

function pickTopSentence(sentences, keywords) {
  if (!sentences.length) return "";
  let best = sentences[0];
  let bestScore = -1;

  for (const s of sentences) {
    let score = 0;
    const lower = s.toLowerCase();
    for (const kw of keywords) {
      if (lower.includes(kw.toLowerCase())) score += 2;
    }
    if (s.length >= 12 && s.length <= 60) score += 1;
    if (/[发布|体验|测评|开箱|对比|评测|升级|更新|功能]/.test(s)) score += 1;
    if (score > bestScore) {
      bestScore = score;
      best = s;
    }
  }
  return best;
}

function buildAnalysis({
  title,
  text,
  platform,
  mediaType,
  visual,
  maxChars = 100,
}) {
  const BRAND_KWS = ["小米", "小爱", "小爱同学", "Xiaomi", "Mi", "Redmi"];
  const titleText = normalizeText(title);
  const bodyText = normalizeText(text);
  const visualText = normalizeText(visual);
  const sents = splitSentences(`${titleText}。${bodyText}`);

  const core = pickTopSentence(sents, BRAND_KWS) || titleText || bodyText;
  const visualCore = pickTopSentence(splitSentences(visualText), BRAND_KWS);

  const hasBrand =
    BRAND_KWS.some((kw) => titleText.includes(kw) || bodyText.includes(kw) || visualText.includes(kw));
  const hasVisual = visualText.length > 0;

  let analysis = "";
  if (hasBrand) {
    analysis = `该内容与小米/小爱相关，核心信息：${core}`;
    if (hasVisual) {
      analysis += `；画面显示：${visualCore || truncateByChars(visualText, 28)}`;
    }
  } else if (hasVisual) {
    analysis = `该内容未明显提及小米/小爱，核心信息：${core || "文本信息有限"}；画面显示：${visualCore || truncateByChars(visualText, 28)}`;
  } else {
    analysis = `该内容核心信息：${core || "文本信息有限，建议补充原文或媒体素材后复核"}`;
  }

  const evidence = {
    title_used: !!titleText,
    text_used: !!bodyText,
    visual_used: !!visualText,
    brand_related: hasBrand,
    platform: normalizeText(platform) || "未知平台",
    media_type: normalizeText(mediaType) || "未知",
  };

  // 评分用于判定“准确性风险”，可在批处理时做复核阈值
  let score = 0.4;
  if (evidence.title_used) score += 0.2;
  if (evidence.text_used) score += 0.2;
  if (evidence.visual_used) score += 0.1;
  if (evidence.brand_related) score += 0.1;
  if (score > 0.98) score = 0.98;

  return {
    analysis: truncateByChars(analysis, maxChars),
    evidence,
    score: Number(score.toFixed(2)),
  };
}

function main() {
  const args = parseArgs(process.argv);
  const maxChars = Number(args["max-chars"] || 100);
  const result = buildAnalysis({
    title: args.title || "",
    text: args.text || "",
    platform: args.platform || "",
    mediaType: args["media-type"] || "",
    visual: args.visual || "",
    maxChars: Number.isFinite(maxChars) ? maxChars : 100,
  });
  process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
}

if (require.main === module) {
  main();
}

module.exports = {
  buildAnalysis,
  normalizeText,
  splitSentences,
};



path: scripts/download-xiaohongshu-album.js
content: #!/usr/bin/env node
/**
 * 小红书图集专用下载脚本
 * 功能：从页面提取 imageList 并下载所有图片
 * 
 * 用法：
 *   node download-xiaohongshu-album.js <URL> <输出目录> [--cookie <COOKIE_STRING>]
 * 
 * 示例：
 *   node download-xiaohongshu-album.js "https://www.xiaohongshu.com/explore/69ba4a870000000028009ac0" ./images
 *   node download-xiaohongshu-album.js "URL" ./images --cookie "unread=xxx; id_token=xxx"
 */

const https = require('https');
const fs = require('fs');
const path = require('path');

// 解析命令行参数
const args = process.argv.slice(2);
const URL = args[0];
const OUTPUT_DIR = args[1];
let COOKIE = '';

for (let i = 2; i < args.length; i++) {
  if (args[i] === '--cookie' && args[i + 1]) {
    COOKIE = args[i + 1];
    i++;
  }
}

if (!URL || !OUTPUT_DIR) {
  console.log('用法：node download-xiaohongshu-album.js <URL> <输出目录> [--cookie <COOKIE_STRING>]');
  process.exit(1);
}

// 创建输出目录
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

console.log(`开始下载小红书图集：${URL}`);
console.log(`输出目录：${OUTPUT_DIR}`);
console.log('---');

const DEFAULT_COOKIE = COOKIE || process.env.XHS_COOKIE || '';

const options = {
  headers: {
    'Cookie': DEFAULT_COOKIE,
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
  }
};

// 下载文件辅助函数
function downloadFile(url, filepath) {
  return new Promise((resolve, reject) => {
    https.get(url, options, (res) => {
      if (res.statusCode === 302 || res.statusCode === 301) {
        downloadFile(res.headers.location, filepath).then(resolve).catch(reject);
        return;
      }
      
      const chunks = [];
      res.on('data', chunk => chunks.push(chunk));
      res.on('end', () => {
        fs.writeFileSync(filepath, Buffer.concat(chunks));
        const size = (fs.statSync(filepath).size / 1024).toFixed(1);
        resolve(size);
      });
    }).on('error', reject);
  });
}

// 提取 imageList（使用括号匹配）
function extractImageList(html) {
  const startIndex = html.indexOf('"imageList":');
  if (startIndex === -1) return null;
  
  const arrayStart = html.indexOf('[', startIndex);
  if (arrayStart === -1) return null;
  
  let depth = 0;
  let endIndex = arrayStart;
  for (let i = arrayStart; i < html.length; i++) {
    if (html[i] === '[') depth++;
    else if (html[i] === ']') depth--;
    if (depth === 0) {
      endIndex = i;
      break;
    }
  }
  
  const imageListStr = html.substring(arrayStart, endIndex + 1);
  
  try {
    return JSON.parse(imageListStr);
  } catch (e) {
    return null;
  }
}

// 获取页面并下载图片
https.get(URL, options, (res) => {
  let data = '';
  res.on('data', chunk => data += chunk);
  res.on('end', async () => {
    try {
      const imageList = extractImageList(data);
      
      if (!imageList || imageList.length === 0) {
        console.log('❌ 未找到图片数据');
        console.log('');
        console.log('可能原因：');
        console.log('  1. Cookie 已过期或无效，请使用 --cookie 参数提供有效 Cookie');
        console.log('  2. 该帖子是视频类型，请使用 parse-xiaohongshu.js 脚本');
        console.log('  3. 链接已失效或帖子被删除');
        process.exit(1);
      }
      
      console.log(`✅ 找到 ${imageList.length} 张图片`);
      
      // 下载所有图片
      let successCount = 0;
      for (let i = 0; i < imageList.length; i++) {
        const img = imageList[i];
        const infoList = img.infoList || [];
        const imgInfo = infoList.find(info => info.imageScene === 'WB_DFT') || infoList[0];
        
        if (!imgInfo || !imgInfo.url) {
          console.log(`⚠️ 图片 ${i + 1} 无有效 URL`);
          continue;
        }
        
        let imgUrl = imgInfo.url.replace('http://', 'https://');
        const filename = path.join(OUTPUT_DIR, `image-${String(i + 1).padStart(2, '0')}.webp`);
        
        try {
          const size = await downloadFile(imgUrl, filename);
          console.log(`✅ 已保存：image-${String(i + 1).padStart(2, '0')}.webp (${size}KB)`);
          successCount++;
        } catch (err) {
          console.log(`❌ 图片 ${i + 1} 下载失败：${err.message}`);
        }
      }
      
      console.log('---');
      console.log(`完成！共 ${imageList.length} 张图片，成功下载 ${successCount} 张`);
      console.log(`输出目录：${OUTPUT_DIR}`);
      
    } catch (err) {
      console.log('❌ 解析失败:', err.message);
      process.exit(1);
    }
  });
}).on('error', (err) => {
  console.log('❌ 请求失败:', err.message);
  process.exit(1);
});


path: scripts/download-xiaohongshu-video.sh
content: #!/bin/bash
#
# 小红书视频下载脚本
# 功能：使用 yt-dlp 下载小红书视频并抽帧
#
# 用法：
#   ./download-xiaohongshu-video.sh <URL> <输出目录> [--cookie <COOKIE_STRING>]
#
# 示例：
#   ./download-xiaohongshu-video.sh "https://www.xiaohongshu.com/explore/69b8f9360000000021039d96" ./output
#   ./download-xiaohongshu-video.sh "URL" ./output --cookie "unread=xxx; id_token=xxx"
#

set -e

URL="$1"
OUTPUT_DIR="$2"
COOKIE=""

# 解析参数
shift 2
while [[ $# -gt 0 ]]; do
  case $1 in
    --cookie)
      COOKIE="$2"
      shift 2
      ;;
    *)
      echo "未知参数：$1"
      exit 1
      ;;
  esac
done

if [ -z "$URL" ] || [ -z "$OUTPUT_DIR" ]; then
  echo "用法：./download-xiaohongshu-video.sh <URL> <输出目录> [--cookie <COOKIE_STRING>]"
  echo "示例：./download-xiaohongshu-video.sh \"https://www.xiaohongshu.com/explore/xxx\" ./output"
  exit 1
fi

# 创建输出目录
mkdir -p "$OUTPUT_DIR"

echo "开始下载小红书视频：$URL"
echo "输出目录：$OUTPUT_DIR"
echo "---"

# 准备 Cookie 文件
COOKIE_FILE="/tmp/xhs_cookies_$$"
COOKIE_ARG=""

if [ -n "$COOKIE" ]; then
  # 将 Cookie 字符串转换为 Netscape 格式
  echo "# Netscape HTTP Cookie File" > "$COOKIE_FILE"
  
  echo "$COOKIE" | tr ';' '\n' | while read -r cookie; do
    cookie=$(echo "$cookie" | xargs)  # 去除首尾空格
    if [ -n "$cookie" ]; then
      key=$(echo "$cookie" | cut -d'=' -f1)
      value=$(echo "$cookie" | cut -d'=' -f2-)
      if [ -n "$key" ] && [ -n "$value" ]; then
        echo -e ".xiaohongshu.com\tTRUE\t/\tTRUE\t0\t${key}\t${value}" >> "$COOKIE_FILE"
      fi
    fi
  done
  
  COOKIE_ARG="--cookies $COOKIE_FILE"
  echo "✅ Cookie 文件已创建：$COOKIE_FILE"
fi

# 检查 yt-dlp 是否安装
if ! command -v yt-dlp &> /dev/null; then
  echo "❌ yt-dlp 未安装，请先安装：brew install yt-dlp"
  exit 1
fi

# 下载视频
echo ""
echo "正在下载视频..."
VIDEO_PATH="$OUTPUT_DIR/video.mp4"

if [ -n "$COOKIE_ARG" ]; then
  yt-dlp $COOKIE_ARG -f best -o "$VIDEO_PATH" "$URL"
else
  yt-dlp -f best -o "$VIDEO_PATH" "$URL"
fi

# 检查下载结果
if [ -f "$VIDEO_PATH" ]; then
  SIZE=$(ls -lh "$VIDEO_PATH" | awk '{print $5}')
  echo ""
  echo "---"
  echo "✅ 视频已下载：video.mp4 ($SIZE)"
  echo "输出目录：$OUTPUT_DIR"
  
  # 询问是否抽帧
  echo ""
  read -p "是否需要视频抽帧？（每 5 秒一帧，输入 y 确认）: " -n 1 -r
  echo
  
  if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo ""
    echo "正在抽帧..."
    cd "$OUTPUT_DIR"
    ffmpeg -i video.mp4 -vf "fps=1/5" -q:v 2 frame_%03d.jpg 2>&1 | tail -5
    
    FRAME_COUNT=$(ls -1 frame_*.jpg 2>/dev/null | wc -l | xargs)
    echo ""
    echo "✅ 抽帧完成！共 ${FRAME_COUNT} 帧"
    echo "输出文件：frame_001.jpg, frame_002.jpg..."
  fi
  
else
  echo "❌ 视频下载失败，文件不存在"
  exit 1
fi

# 清理临时 Cookie 文件
if [ -n "$COOKIE_ARG" ] && [ -f "$COOKIE_FILE" ]; then
  rm -f "$COOKIE_FILE"
fi


path: scripts/extract-douyin-images.js
content: /**
 * 抖音图集图片提取工具
 * 用法：node extract-douyin-images.js <抖音 URL> <输出目录>
 */

const { mkdirSync, writeFileSync } = require('fs')
const path = require('path')
const { buildAnalysis } = require('./build-content-analysis')

const DEFAULT_HEADERS = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
  Referer: 'https://www.douyin.com/',
}

async function extractImages(url, outputDir) {
  const awemeId = url.match(/video\/([^/?]*)/)?.[1] || url.match(/note\/([^/?]*)/)?.[1]
  if (!awemeId) throw new Error('无法提取视频 ID')
  
  const requestUrl = `https://www.douyin.com/jingxuan?modal_id=${awemeId}`
  console.log(`正在获取：${requestUrl}`)
  
  const response = await fetch(requestUrl, { headers: DEFAULT_HEADERS })
  if (!response.ok) throw new Error(`请求失败：${response.status}`)
  
  const html = await response.text()
  
  // 提取 videoDetail JSON
  const pattern = /"([^"]*?(?:videoDetail)[^"]*?)"/g
  const matches = html.match(pattern)
  
  let images = []
  if (matches) {
    for (const m of matches) {
      try {
        const decoded = decodeURIComponent(m.slice(1, -1))
        const data = JSON.parse(decoded)
        const videoDetail = data?.app?.videoDetail
        if (videoDetail?.images) {
          images = videoDetail.images
          console.log(`找到 ${images.length} 张图片`)
          break
        }
      } catch (e) {}
    }
  }
  
  if (images.length === 0) {
    throw new Error('未找到图片数据')
  }
  
  // 创建输出目录
  mkdirSync(outputDir, { recursive: true })
  
  // 下载图片
  const downloadedImages = []
  for (let i = 0; i < images.length; i++) {
    const img = images[i]
    let imgUrl = img.urlList?.[0] || img.url_list?.[0] || img.download_addr?.url_list?.[0]
    
    if (!imgUrl) {
      console.log(`跳过图片 ${i + 1}：无法获取 URL`)
      continue
    }
    
    console.log(`下载图片 ${i + 1}/${images.length}...`)
    
    try {
      const imgResponse = await fetch(imgUrl, { headers: DEFAULT_HEADERS })
      if (!imgResponse.ok) {
        console.log(`图片 ${i + 1} 下载失败：${imgResponse.status}`)
        continue
      }
      
      const buffer = Buffer.from(await imgResponse.arrayBuffer())
      const ext = imgUrl.match(/\.(\w+)(?:\?|$)/)?.[1] || 'jpg'
      const filename = `image-${String(i + 1).padStart(2, '0')}.${ext}`
      const filepath = path.join(outputDir, filename)
      
      writeFileSync(filepath, buffer)
      downloadedImages.push({ index: i + 1, path: filepath, url: imgUrl })
      console.log(`✅ 已保存：${filename} (${img.width}x${img.height})`)
    } catch (e) {
      console.log(`图片 ${i + 1} 下载错误：${e.message}`)
    }
  }
  
  return {
    total: images.length,
    downloaded: downloadedImages.length,
    images: downloadedImages,
    outputDir,
  }
}

// 主程序
(async () => {
  const url = process.argv[2]
  const outputDir = process.argv[3] || './douyin-images'
  
  if (!url) {
    console.log('用法：node extract-douyin-images.js <抖音 URL> [输出目录]')
    process.exit(1)
  }
  
  console.log(`开始提取抖音图集：${url}`)
  console.log(`输出目录：${outputDir}`)
  console.log('---')
  
  try {
    const result = await extractImages(url, outputDir)
    console.log('---')
    console.log(`完成！共 ${result.total} 张图片，成功下载 ${result.downloaded} 张`)
    console.log(`保存位置：${result.outputDir}`)
    
    if (result.images.length > 0) {
      console.log('\n图片列表:')
      result.images.forEach(img => {
        console.log(`  ${img.index}. ${path.basename(img.path)}`)
      })
    }
    const analysis = buildAnalysis({
      title: '',
      text: '',
      platform: '抖音',
      mediaType: '图片',
      visual: `图集共 ${result.total} 张，成功下载 ${result.downloaded} 张`,
      maxChars: 100,
    })
    console.log('\n---')
    console.log(JSON.stringify({
      type: 'album',
      total_images: result.total,
      downloaded_images: result.downloaded,
      output_dir: result.outputDir,
      content_analysis: analysis.analysis,
      analysis_score: analysis.score,
      analysis_evidence: analysis.evidence,
    }, null, 2))
  } catch (e) {
    console.error(`错误：${e.message}`)
    process.exit(1)
  }
})()


path: scripts/package.sh
content: #!/bin/bash
#
# Skill 打包脚本
# 将 social-media-analysis skill 打包为 zip 文件，方便分享
#

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUTPUT_DIR="${1:-.}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
PACKAGE_NAME="social-media-analysis-skill-${TIMESTAMP}.zip"

echo "📦 打包 Skill: $SKILL_DIR"
echo "输出目录：$OUTPUT_DIR"
echo "文件名：$PACKAGE_NAME"
echo ""

# 进入 skill 目录
cd "$SKILL_DIR"

# 创建 zip 包（排除不必要的文件）
zip -r "$OUTPUT_DIR/$PACKAGE_NAME" \
  SKILL.md \
  scripts/ \
  -x "*.DS_Store" \
  -x "*.pyc" \
  -x "__pycache__/*" \
  -x ".git/*"

if [ $? -eq 0 ]; then
  echo ""
  echo "✅ 打包完成！"
  echo "文件：$OUTPUT_DIR/$PACKAGE_NAME"
  echo ""
  echo "📋 包含内容："
  unzip -l "$OUTPUT_DIR/$PACKAGE_NAME" | tail -20
else
  echo "❌ 打包失败"
  exit 1
fi


path: scripts/parse-bilibili.js
content: #!/usr/bin/env node
/**
 * 哔哩哔哩（B站）视频下载脚本
 * 功能：下载B站视频并提取信息
 * 
 * 用法：
 *   node parse-bilibili.js <视频URL> [输出目录]
 * 
 * 示例：
 *   node parse-bilibili.js "https://www.bilibili.com/video/BV1AUA3zcE2Z/" ./bilibili-output
 * 
 * 说明：
 *   本脚本调用 yt-dlp 下载视频，需要提前安装 yt-dlp
 *   安装方式：brew install yt-dlp
 */

const { execSync } = require('child_process');
const fs = require('fs');
const path = require('path');
const { buildAnalysis } = require('./build-content-analysis');

// 解析命令行参数
const args = process.argv.slice(2);
const URL = args[0];
const OUTPUT_DIR = args[1] || './bilibili-download';

if (!URL) {
  console.log('用法：node parse-bilibili.js <视频URL> [输出目录]');
  console.log('示例：node parse-bilibili.js "https://www.bilibili.com/video/BV1AUA3zcE2Z/" ./output');
  console.log('\n说明：需要提前安装 yt-dlp（brew install yt-dlp）');
  process.exit(1);
}

// 创建输出目录
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

console.log(`开始解析B站视频：${URL}`);
console.log(`输出目录：${OUTPUT_DIR}`);
console.log('---');

// 提取BV号
function extractBvid(url) {
  const match = url.match(/BV([a-zA-Z0-9]+)/);
  return match ? match[1] : null;
}

const bvid = extractBvid(URL);
if (bvid) {
  console.log(`BV号: BV${bvid}`);
}

// 主函数
async function main() {
  try {
    // 获取视频信息
    console.log('\n📊 获取视频信息...');
    const infoJson = execSync(
      `yt-dlp --dump-json "${URL}" 2>/dev/null`,
      { encoding: 'utf8', maxBuffer: 10 * 1024 * 1024 }
    );
    const info = JSON.parse(infoJson);
    
    console.log(`标题: ${info.title}`);
    console.log(`UP主: ${info.uploader}`);
    console.log(`时长: ${info.duration}秒`);
    console.log(`播放量: ${info.view_count}`);
    console.log(`点赞: ${info.like_count}`);
    console.log(`标签: ${info.tags?.join(', ')}`);
    
    // 下载视频（1080P + 最佳音频）
    console.log('\n📥 开始下载视频（1080P）...');
    const outputPath = path.join(OUTPUT_DIR, 'video.mp4');
    
    execSync(
      `yt-dlp -f "30080+30280" -o "${outputPath}" "${URL}" 2>&1`,
      { stdio: 'inherit' }
    );
    
    // 检查文件大小
    const stats = fs.statSync(outputPath);
    const sizeMB = (stats.size / 1024 / 1024).toFixed(2);
    
    console.log('\n✅ 下载完成！');
    console.log(`文件路径: ${outputPath}`);
    console.log(`文件大小: ${sizeMB}MB`);
    
    // 输出结果
    const result = {
      title: info.title,
      uploader: info.uploader,
      bvid: info.id,
      duration: info.duration,
      view_count: info.view_count,
      like_count: info.like_count,
      tags: info.tags,
      thumbnail: info.thumbnail,
      download_path: outputPath,
      file_size: `${sizeMB}MB`,
      webpage_url: info.webpage_url
    };
    const analysis = buildAnalysis({
      title: info.title || '',
      text: (info.description || '').toString(),
      platform: 'B站',
      mediaType: '视频',
      visual: `视频时长 ${info.duration || 0} 秒，UP主 ${info.uploader || '未知'}`,
      maxChars: 100,
    });
    result.content_analysis = analysis.analysis;
    result.analysis_score = analysis.score;
    result.analysis_evidence = analysis.evidence;
    
    console.log('\n---');
    console.log(JSON.stringify(result, null, 2));
    
  } catch (err) {
    console.error('❌ 错误:', err.message);
    console.log('\n提示：');
    console.log('  1. 确保已安装 yt-dlp: brew install yt-dlp');
    console.log('  2. 确保视频链接有效');
    console.log('  3. 如果需要下载1080P高码率或会员视频，需要添加Cookie');
    process.exit(1);
  }
}

main();


path: scripts/parse-douyin-video.js
content: /**
 * 抖音视频解析与下载 - 无外部依赖，仅用 Node 内置模块。
 * 需 Node 18+（fetch、Readable.fromWeb、stream/promises）。
 *
 * 用法:
 *   仅解析: node parse-douyin-video.js <抖音URL>
 *   下载到临时目录: node parse-douyin-video.js <抖音URL> --download
 *   下载到指定路径: node parse-douyin-video.js <抖音URL> <输出文件路径>
 */

const { createWriteStream, statSync } = require('fs')
const { Readable } = require('stream')
const { pipeline } = require('stream/promises')
const { tmpdir } = require('os')
const path = require('path')
const { buildAnalysis } = require('./build-content-analysis')

const DEFAULT_HEADERS = {
  'User-Agent':
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  Accept:
    'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
  'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
  Connection: 'keep-alive',
  Referer: 'https://www.douyin.com/',
}

async function extractAwemeId(url) {
  let match = url.match(/modal_id=([0-9]+)/)
  if (match) return match[1]

  match = url.match(/video\/([^/?]*)/)
  if (match) return match[1]

  match = url.match(/note\/([^/?]*)/)
  if (match) return match[1]

  match = url.match(/(https?:\/\/\S+)/)
  if (match) {
    try {
      const res = await fetch(match[1], { method: 'HEAD', redirect: 'follow' })
      const finalUrl = res.url
      match = finalUrl.match(/video\/([^/?]*)/)
      if (match) return match[1]
      match = finalUrl.match(/note\/([^/?]*)/)
      if (match) return match[1]
    } catch {
      // 重定向失败，继续抛下面的错误
    }
  }

  throw new Error(`无法从抖音URL中提取视频ID: ${url}`)
}

async function parseDouyinVideoUrl(url) {
  const awemeId = await extractAwemeId(url)
  const requestUrl = `https://www.douyin.com/jingxuan?modal_id=${awemeId}`

  const response = await fetch(requestUrl, { method: 'GET', headers: DEFAULT_HEADERS })
  if (!response.ok) {
    throw new Error(`请求失败，状态码: ${response.status}`)
  }

  const htmlContent = await response.text()
  const pattern = /"([^"]*?(?:playAddr|searchProps|app)[^"]*?)"/g
  const matches = htmlContent.match(pattern)

  let targetMatch = null
  if (matches) {
    for (const match of matches) {
      if (
        match.includes('playAddr') &&
        match.includes('searchProps') &&
        match.includes('app')
      ) {
        targetMatch = match.slice(1, -1)
        break
      }
    }
  }

  if (!targetMatch) {
    throw new Error('未找到包含视频数据的JSON字符串')
  }

  const decodedJson = decodeURIComponent(targetMatch)
  const videoDataJson = JSON.parse(decodedJson)

  const videoDetail = videoDataJson?.app?.videoDetail
  if (!videoDetail) {
    throw new Error('未找到videoDetail数据')
  }

  const desc = videoDetail.desc ?? ''
  const videoInfo = videoDetail.video ?? {}
  const playAddr = videoInfo.playAddr ?? []

  let videoUrl = ''
  if (Array.isArray(playAddr) && playAddr.length > 0) {
    if (typeof playAddr[0] === 'object' && playAddr[0]?.src) {
      videoUrl = playAddr[0].src
    } else if (
      playAddr.length > 1 &&
      typeof playAddr[1] === 'object' &&
      playAddr[1]?.src
    ) {
      videoUrl = playAddr[1].src
    }
  }

  if (!videoUrl) {
    throw new Error('无法获取有效的视频播放地址')
  }

  // 从desc中提取标题作为文件名
  let fileName = '视频'
  if (desc) {
    // 移除话题标签，取前30个字符作为文件名
    fileName = desc.replace(/#\S+/g, '').trim().substring(0, 30) || '视频'
    // 移除特殊字符
    fileName = fileName.replace(/[\\/:*?"<>|]/g, '')
  }
  fileName += '.mp4'

  return {
    title: desc,
    desc,
    cover_url: '',
    file_name: fileName,
    video_urls: [
      {
        video_url: videoUrl,
        resolution_type: 720,
        resolution_str: '默认',
      },
    ],
  }
}

async function downloadVideoToFile(videoUrl, outputPath) {
  const response = await fetch(videoUrl, {
    method: 'GET',
    headers: {
      ...DEFAULT_HEADERS,
      Referer: 'https://www.douyin.com/',
    },
  })
  if (!response.ok) {
    throw new Error(`下载失败，状态码: ${response.status}`)
  }
  const body = response.body
  if (!body) {
    throw new Error('响应体为空')
  }
  const nodeStream = Readable.fromWeb(body)
  const dest = createWriteStream(outputPath)
  await pipeline(nodeStream, dest)
}

/**
 * 下载到系统临时目录
 * @returns {string} 临时文件路径
 */
function createTempFilePath() {
  const tmpDir = tmpdir()
  const tempDir = path.join(tmpDir, `douyin-${Date.now()}`)
  // 使用fs模块的mkdirSync
  require('fs').mkdirSync(tempDir, { recursive: true })
  return path.join(tempDir, 'video.mp4')
}



async function parseAndDownload(douyinUrl, outputPathOrFlag) {
  const result = await parseDouyinVideoUrl(douyinUrl)
  
  // 判断是否需要下载
  let downloadPath = null
  if (outputPathOrFlag === '--download' || outputPathOrFlag === '-d') {
    // 下载到系统临时目录
    downloadPath = createTempFilePath()
    await downloadVideoToFile(result.video_urls[0].video_url, downloadPath)
  } else if (outputPathOrFlag) {
    // 下载到指定路径
    downloadPath = outputPathOrFlag
    await downloadVideoToFile(result.video_urls[0].video_url, downloadPath)
  }
  
  if (downloadPath) {
    return { result, download_path: downloadPath }
  }
  return { result }
}

async function main() {
  const args = process.argv.slice(2)
  
  if (args.length < 1) {
    console.error('用法:')
    console.error('  仅解析: node parse-douyin-video.js <抖音URL>')
    console.error('  下载到临时目录: node parse-douyin-video.js <抖音URL> --download')
    console.error('  下载到指定路径: node parse-douyin-video.js <抖音URL> <输出文件路径>')
    process.exit(1)
  }

  const url = args[0]
  
  // 解析参数模式
  let mode = 'parse'  // parse, download
  let outputPath = null
  
  if (args.length >= 2) {
    if (args[1] === '--download') {
      mode = 'download'
    } else {
      // 认为是输出文件路径
      mode = 'download'
      outputPath = args[1]
    }
  }

  try {
    let output
    
    if (mode === 'download') {
      // 下载模式
      const downloadFlag = outputPath || '--download'
      const { result, download_path } = await parseAndDownload(url, downloadFlag)
      output = { ...result }
      if (download_path) {
        output.download_path = download_path
        console.error('已下载到:', download_path)
      }
    } else {
      // 仅解析
      const { result } = await parseAndDownload(url, null)
      output = result
    }
    const analysis = buildAnalysis({
      title: output.title || '',
      text: output.desc || '',
      platform: '抖音',
      mediaType: '视频',
      visual: output.download_path ? `视频已下载到 ${output.download_path}` : '已解析视频链接',
      maxChars: 100,
    })
    output.content_analysis = analysis.analysis
    output.analysis_score = analysis.score
    output.analysis_evidence = analysis.evidence
    
    console.log(JSON.stringify(output, null, 2))
  } catch (err) {
    const msg = err instanceof Error ? err.message : String(err)
    console.error('失败:', msg)
    process.exit(1)
  }
}

const isDirectRun =
  typeof process !== 'undefined' &&
  process.argv[1]?.includes('parse-douyin-video')
if (isDirectRun) {
  main()
}

module.exports = {
  parseDouyinVideoUrl,
  downloadVideoToFile,
  parseAndDownload,
  createTempFilePath,
}


path: scripts/parse-toutiao-playwright.js
content: #!/usr/bin/env node
/**
 * 今日头条文章图片提取脚本（Playwright 渲染版）
 * 功能：使用 Playwright 渲染页面后提取并下载图片
 * 
 * 适用场景：
 * - 静态脚本 (parse-toutiao.js) 无法提取图片时
 * - 页面需要 JavaScript 动态加载内容
 * 
 * 用法：
 *   node scripts/parse-toutiao-playwright.js <文章 URL> [输出目录]
 * 
 * 示例：
 *   node scripts/parse-toutiao-playwright.js "https://www.toutiao.com/w/7620088161695253044" ./output
 * 
 * 前置准备：
 *   npm install playwright
 *   npx playwright install chromium
 */

const { chromium } = require('playwright');
const fs = require('fs');
const path = require('path');
const { buildAnalysis } = require('./build-content-analysis');

// 解析命令行参数
const args = process.argv.slice(2);
const URL = args[0];
const OUTPUT_DIR = args[1] || './toutiao-output';

if (!URL) {
  console.log('用法：node parse-toutiao-playwright.js <文章 URL> [输出目录]');
  console.log('示例：node parse-toutiao-playwright.js "https://www.toutiao.com/w/xxx" ./output');
  process.exit(1);
}

// 创建输出目录
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

console.log(`开始解析今日头条：${URL}`);
console.log(`输出目录：${OUTPUT_DIR}`);
console.log('---');

// 主函数
(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  
  console.log('加载页面...');
  await page.goto(URL, { waitUntil: 'networkidle', timeout: 30000 });
  await page.waitForTimeout(5000); // 等待图片完全加载
  
  // 提取文章信息和图片
  const content = await page.evaluate(() => {
    const article = document.querySelector('article') || 
                    document.querySelector('.article') || 
                    document.body;
    const title = document.querySelector('h1')?.textContent || '未知标题';
    
    // 提取所有文章相关图片（过滤小图）
    const allImages = Array.from(document.querySelectorAll('img'))
      .map(img => ({ 
        src: img.src, 
        width: img.width, 
        height: img.height,
        alt: img.alt 
      }))
      .filter(img => 
        img.src.includes('toutiaoimg.com') && 
        img.width > 100 && 
        img.height > 100
      );
    
    return { title, images: allImages };
  });
  
  console.log(`标题：${content.title}`);
  console.log(`找到图片：${content.images.length}`);
  
  if (content.images.length === 0) {
    console.log('\n⚠️ 未找到符合条件的图片');
    console.log('可能原因：');
    console.log('  1. 文章本身没有图片');
    console.log('  2. 图片 URL 不包含 toutiaoimg.com');
    console.log('  3. 页面加载失败');
    await browser.close();
    process.exit(1);
  }
  
  // 下载前 5 张大图
  const downloadCount = Math.min(content.images.length, 5);
  console.log(`\n开始下载前 ${downloadCount} 张图片...\n`);
  
  const results = [];
  for (let i = 0; i < downloadCount; i++) {
    const img = content.images[i];
    const filename = path.join(OUTPUT_DIR, `image-${String(i+1).padStart(2, '0')}.jpg`);
    
    console.log(`下载 ${i+1}/${downloadCount}...`);
    
    try {
      // 通过浏览器上下文下载图片（绕过防盗链）
      const imgData = await page.evaluate(async (imgUrl) => {
        const res = await fetch(imgUrl);
        const blob = await res.blob();
        const reader = new FileReader();
        return new Promise((resolve, reject) => {
          reader.onloadend = () => resolve(reader.result.split(',')[1]);
          reader.onerror = reject;
          reader.readAsDataURL(blob);
        });
      }, img.src);
      
      fs.writeFileSync(filename, Buffer.from(imgData, 'base64'));
      const size = (fs.statSync(filename).size / 1024).toFixed(1);
      console.log(`✅ image-${String(i+1).padStart(2, '0')}.jpg (${size}KB)`);
      results.push({ index: i+1, filename, size: `${size}KB` });
    } catch (err) {
      console.log(`❌ [${i+1}/${downloadCount}] 下载失败：${err.message}`);
    }
  }
  
  await browser.close();
  
  // 输出结果
  console.log('\n---');
  console.log('下载完成！');
  const visualSummary = results.length > 0
    ? `下载 ${results.length} 张文章配图`
    : '未下载到配图';
  const analysis = buildAnalysis({
    title: content.title || '',
    text: '',
    platform: '今日头条',
    mediaType: '图片',
    visual: visualSummary,
    maxChars: 100,
  });
  console.log(JSON.stringify({
    title: content.title,
    totalImages: content.images.length,
    downloaded: results.length,
    outputDir: OUTPUT_DIR,
    images: results,
    content_analysis: analysis.analysis,
    analysis_score: analysis.score,
    analysis_evidence: analysis.evidence,
  }, null, 2));
  
})().catch(err => {
  console.log('❌ 错误:', err.message);
  process.exit(1);
});


path: scripts/parse-toutiao.js
content: #!/usr/bin/env node
/**
 * 今日头条文章图片提取脚本
 * 功能：提取文章中的图片URL并下载
 * 
 * 用法：
 *   node parse-toutiao.js <文章URL> <输出目录>
 * 
 * 示例：
 *   node parse-toutiao.js "https://www.toutiao.com/w/1860414912331850/" ./toutiao-output
 * 
 * 说明：
 *   今日头条页面需要JavaScript渲染，此脚本通过解析页面HTML提取图片URL
 *   由于今日头条的反爬机制，建议在浏览器中打开页面后使用开发者工具提取图片URL
 *   或使用浏览器截图功能保存内容
 */

const https = require('https');
const fs = require('fs');
const path = require('path');

// 解析命令行参数
const args = process.argv.slice(2);
const URL = args[0];
const OUTPUT_DIR = args[1] || './toutiao-download';

if (!URL) {
  console.log('用法：node parse-toutiao.js <文章URL> [输出目录]');
  console.log('示例：node parse-toutiao.js "https://www.toutiao.com/w/xxx/" ./output');
  console.log('\n说明：今日头条页面需要JavaScript渲染，建议在浏览器中打开后提取图片URL');
  process.exit(1);
}

// 创建输出目录
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

console.log(`开始解析今日头条：${URL}`);
console.log(`输出目录：${OUTPUT_DIR}`);
console.log('---');

// 下载文件辅助函数
function downloadFile(url, filepath) {
  return new Promise((resolve, reject) => {
    const client = url.startsWith('https') ? https : require('http');
    client.get(url, {
      headers: {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
        'Referer': 'https://www.toutiao.com/',
      }
    }, (res) => {
      if (res.statusCode === 302 || res.statusCode === 301) {
        // 跟随重定向
        downloadFile(res.headers.location, filepath).then(resolve).catch(reject);
        return;
      }
      
      if (res.statusCode !== 200) {
        reject(new Error(`HTTP ${res.statusCode}`));
        return;
      }
      
      const chunks = [];
      res.on('data', chunk => chunks.push(chunk));
      res.on('end', () => {
        fs.writeFileSync(filepath, Buffer.concat(chunks));
        const size = (fs.statSync(filepath).size / 1024).toFixed(1);
        resolve(size);
      });
    }).on('error', reject);
  });
}

// 提取文章ID
function extractArticleId(url) {
  const patterns = [
    /\/w\/(\d+)\//,
    /\/article\/(\d+)\//,
    /\/item\/(\d+)/,
  ];
  
  for (const pattern of patterns) {
    const match = url.match(pattern);
    if (match) return match[1];
  }
  return null;
}

// 主函数
async function main() {
  const articleId = extractArticleId(URL);
  if (articleId) {
    console.log(`文章 ID: ${articleId}`);
  }

  // 访问页面获取HTML
  https.get(URL, {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
      'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
      'Referer': 'https://www.toutiao.com/',
    }
  }, (res) => {
    let data = '';
    res.on('data', chunk => data += chunk);
    res.on('end', async () => {
      try {
        // 检查是否需要登录或页面被拦截
        if (data.includes('请登录') || data.includes('验证码')) {
          console.log('⚠️ 页面需要登录或验证，无法直接获取内容');
          console.log('提示：请使用浏览器打开页面，手动提取图片URL');
          process.exit(1);
        }
        
        // 提取文章标题
        const titleMatch = data.match(/<title>([^<]*)<\/title>/);
        const title = titleMatch ? titleMatch[1].trim() : '未知标题';
        console.log(`标题: ${title}`);
        
        // 从HTML中提取图片URL（多种模式）
        const imageUrls = [];
        
        // 模式1: 标准图片URL
        const imgRegex = /https?:\/\/[^\s"'<>]+\.(jpg|jpeg|png|webp|gif)/gi;
        let match;
        while ((match = imgRegex.exec(data)) !== null) {
          imageUrls.push(match[0]);
        }
        
        // 模式2: 今日头条特定图片域名
        const toutiaoImgRegex = /https?:\/\/[a-z0-9-]+\.toutiaoimg\.com\/[^\s"'<>]+/gi;
        while ((match = toutiaoImgRegex.exec(data)) !== null) {
          imageUrls.push(match[0]);
        }
        
        // 去重并过滤
        const uniqueUrls = [...new Set(imageUrls)].filter(url => {
          // 过滤掉非文章图片
          return url.includes('toutiaoimg.com') && 
                 !url.includes('avatar') && 
                 !url.includes('icon') &&
                 !url.includes('logo') &&
                 !url.includes('.svg');
        });
        
        if (uniqueUrls.length === 0) {
          console.log('\n⚠️ 未从HTML中找到图片');
          console.log('说明：今日头条页面需要JavaScript渲染才能加载图片');
          console.log('建议：使用浏览器开发者工具提取图片URL，或使用以下方法：');
          console.log('  1. 在浏览器中打开页面');
          console.log('  2. 按F12打开开发者工具');
          console.log('  3. 在Console中执行：document.querySelectorAll("article img").forEach(img => console.log(img.src))');
          console.log('  4. 复制输出的URL并使用curl下载');
          
          // 输出页面中可能包含图片URL的JSON数据
          const jsonMatch = data.match(/"imageList":\s*(\[[^\]]+\])/);
          if (jsonMatch) {
            console.log('\n✅ 发现图片列表数据，尝试解析...');
            try {
              const imageList = JSON.parse(jsonMatch[1]);
              console.log(`找到 ${imageList.length} 张图片信息`);
            } catch (e) {
              // 解析失败
            }
          }
          
          process.exit(1);
        }
        
        console.log(`\n找到 ${uniqueUrls.length} 张图片`);
        console.log('开始下载...\n');
        
        // 下载图片（最多10张）
        const results = [];
        for (let i = 0; i < Math.min(uniqueUrls.length, 10); i++) {
          const imgUrl = uniqueUrls[i];
          const ext = imgUrl.match(/\.(jpg|jpeg|png|webp|gif)/i)?.[1] || 'jpg';
          const filename = path.join(OUTPUT_DIR, `image-${String(i + 1).padStart(2, '0')}.${ext}`);
          
          try {
            const size = await downloadFile(imgUrl, filename);
            console.log(`✅ [${i + 1}/${uniqueUrls.length}] ${path.basename(filename)} (${size}KB)`);
            results.push({ index: i + 1, filename, size, url: imgUrl });
          } catch (err) {
            console.log(`❌ [${i + 1}/${uniqueUrls.length}] 下载失败: ${err.message}`);
          }
        }
        
        // 输出结果
        console.log('\n---');
        console.log('下载完成！');
        console.log(JSON.stringify({
          title,
          articleId,
          totalImages: uniqueUrls.length,
          downloaded: results.length,
          outputDir: OUTPUT_DIR,
          images: results
        }, null, 2));
        
      } catch (err) {
        console.log('❌ 解析失败:', err.message);
        process.exit(1);
      }
    });
  }).on('error', (err) => {
    console.log('❌ 请求失败:', err.message);
    process.exit(1);
  });
}

main();


path: scripts/parse-weibo.js
content: #!/usr/bin/env node
/**
 * 微博图片/视频下载脚本
 * 支持：视频下载、单图/多图图集下载
 * 
 * 用法：
 *   node parse-weibo.js <微博URL> [输出目录]
 * 
 * 示例：
 *   node parse-weibo.js https://weibo.com/7976115583/QwgTm2OVI ./weibo-download
 */

const https = require('https');
const fs = require('fs');
const path = require('path');
const { buildAnalysis } = require('./build-content-analysis');

// 解析命令行参数
const url = process.argv[2];
const outputDir = process.argv[3] || './weibo-download';

if (!url) {
    console.error('❌ 请提供微博 URL');
    console.error('用法：node parse-weibo.js <微博URL> [输出目录]');
    process.exit(1);
}

// 从 URL 提取微博 ID
function extractWeiboId(url) {
    // 支持格式：
    // https://weibo.com/xxxx/xxxxx
    // https://m.weibo.cn/status/xxxxx
    // https://weibo.com/xxxx/xxxxx?from=...
    
    const patterns = [
        /weibo\.com\/\d+\/([a-zA-Z0-9]+)/,  // PC端
        /m\.weibo\.cn\/status\/([a-zA-Z0-9]+)/,  // 移动端
        /weibo\.com\/\d+\/([a-zA-Z0-9]+)\?/  // 带参数
    ];
    
    for (const pattern of patterns) {
        const match = url.match(pattern);
        if (match) return match[1];
    }
    
    // 如果都不匹配，尝试从最后一段提取
    const parts = url.split('/');
    const lastPart = parts[parts.length - 1];
    if (lastPart && !lastPart.includes('?')) {
        return lastPart;
    }
    
    return null;
}

// 调用微博 API 获取数据
async function fetchWeiboData(weiboId) {
    const apiUrl = `https://m.weibo.cn/statuses/show?id=${weiboId}`;
    
    return new Promise((resolve, reject) => {
        const options = {
            headers: {
                'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15',
                'Referer': 'https://m.weibo.cn/',
                'X-Requested-With': 'XMLHttpRequest'
            }
        };
        
        https.get(apiUrl, options, (res) => {
            let data = '';
            res.on('data', chunk => data += chunk);
            res.on('end', () => {
                try {
                    const json = JSON.parse(data);
                    if (json.ok === 1) {
                        resolve(json.data);
                    } else {
                        reject(new Error(`API 返回错误: ${json.msg || '未知错误'}`));
                    }
                } catch (e) {
                    reject(new Error(`解析 JSON 失败: ${e.message}`));
                }
            });
        }).on('error', reject);
    });
}

// 下载文件（支持重定向）
async function downloadFile(url, filepath) {
    return new Promise((resolve, reject) => {
        const doDownload = (downloadUrl, callback) => {
            const protocol = downloadUrl.startsWith('https') ? https : require('http');
            protocol.get(downloadUrl, { 
                headers: { 
                    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
                    'Referer': 'https://m.weibo.cn/'
                } 
            }, (res) => {
                if (res.statusCode === 301 || res.statusCode === 302) {
                    // 跟随重定向
                    callback(res.headers.location);
                } else if (res.statusCode === 200) {
                    const file = fs.createWriteStream(filepath);
                    res.pipe(file);
                    file.on('finish', () => {
                        file.close();
                        resolve();
                    });
                    file.on('error', reject);
                } else {
                    reject(new Error(`HTTP ${res.statusCode}`));
                }
            }).on('error', reject);
        };
        
        doDownload(url, (redirectUrl) => {
            doDownload(redirectUrl, () => {
                // 已经处理过了
            });
        });
    });
}

// 主函数
async function main() {
    console.log('🔍 解析微博链接...');
    console.log(`   URL: ${url}`);
    
    const weiboId = extractWeiboId(url);
    if (!weiboId) {
        console.error('❌ 无法从 URL 提取微博 ID');
        process.exit(1);
    }
    console.log(`   微博 ID: ${weiboId}`);
    
    // 创建输出目录
    if (!fs.existsSync(outputDir)) {
        fs.mkdirSync(outputDir, { recursive: true });
    }
    
    console.log('\n📡 获取微博数据...');
    let data;
    try {
        data = await fetchWeiboData(weiboId);
    } catch (e) {
        console.error(`❌ 获取数据失败: ${e.message}`);
        process.exit(1);
    }
    
    // 提取基本信息
    const user = data.user?.screen_name || '未知用户';
    const text = data.text?.replace(/<[^>]+>/g, '') || '';
    const picNum = data.pic_num || 0;
    
    console.log(`   用户: ${user}`);
    console.log(`   内容: ${text.substring(0, 50)}${text.length > 50 ? '...' : ''}`);
    console.log(`   图片数量: ${picNum}`);
    
    // 判断类型并下载
    if (picNum > 0) {
        // 图集类型
        console.log('\n📷 下载图集...');
        
        const pics = data.pics || [];
        const results = [];
        
        for (let i = 0; i < pics.length; i++) {
            const pic = pics[i];
            const imgUrl = pic.large?.url || pic.url;
            const filename = path.join(outputDir, `image_${String(i + 1).padStart(2, '0')}.jpg`);
            
            try {
                await downloadFile(imgUrl, filename);
                const stats = fs.statSync(filename);
                console.log(`   ✅ [${i + 1}/${pics.length}] ${path.basename(filename)} (${(stats.size / 1024).toFixed(1)} KB)`);
                results.push({ index: i + 1, filename, size: stats.size, url: imgUrl });
            } catch (e) {
                console.log(`   ❌ [${i + 1}/${pics.length}] 下载失败: ${e.message}`);
            }
        }
        
        // 输出结果
        const analysis = buildAnalysis({
            title: '',
            text,
            platform: '微博',
            mediaType: '图片',
            visual: `共下载 ${results.length} 张图片`,
            maxChars: 100,
        });
        console.log('\n✅ 下载完成！');
        console.log(JSON.stringify({
            type: 'album',
            user,
            text,
            picNum,
            outputDir,
            images: results,
            content_analysis: analysis.analysis,
            analysis_score: analysis.score,
            analysis_evidence: analysis.evidence,
        }, null, 2));
        
    } else {
        // 可能是视频，提示使用 yt-dlp
        console.log('\n⚠️ 未检测到图片，可能是视频类型');
        console.log('   请使用 yt-dlp 下载视频：');
        console.log(`   yt-dlp -f best -o "${outputDir}/video.mp4" "${url}"`);
        
        const analysis = buildAnalysis({
            title: '',
            text,
            platform: '微博',
            mediaType: '视频',
            visual: '',
            maxChars: 100,
        });
        console.log(JSON.stringify({
            type: 'video',
            user,
            text,
            suggestion: '使用 yt-dlp 下载',
            command: `yt-dlp -f best -o "${outputDir}/video.mp4" "${url}"`,
            content_analysis: analysis.analysis,
            analysis_score: analysis.score,
            analysis_evidence: analysis.evidence,
        }, null, 2));
    }
}

main().catch(console.error);


path: scripts/parse-xiaohongshu.js
content: #!/usr/bin/env node
/**
 * 小红书帖子解析脚本
 * 功能：判断类型（图集/视频）+ 下载图片/视频
 * 
 * 用法：
 *   node parse-xiaohongshu.js <URL> <输出目录> [--cookie <COOKIE_STRING>]
 * 
 * 示例：
 *   node parse-xiaohongshu.js "https://www.xiaohongshu.com/explore/69ba4a870000000028009ac0" ./output
 *   node parse-xiaohongshu.js "URL" ./output --cookie "unread=xxx; id_token=xxx"
 */

const https = require('https');
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
const { buildAnalysis } = require('./build-content-analysis');

// 解析命令行参数
const args = process.argv.slice(2);
const URL = args[0];
const OUTPUT_DIR = args[1];
let COOKIE = '';

// 解析 --cookie 参数
for (let i = 2; i < args.length; i++) {
  if (args[i] === '--cookie' && args[i + 1]) {
    COOKIE = args[i + 1];
    i++;
  }
}

if (!URL || !OUTPUT_DIR) {
  console.log('用法：node parse-xiaohongshu.js <URL> <输出目录> [--cookie <COOKIE_STRING>]');
  console.log('示例：node parse-xiaohongshu.js "https://www.xiaohongshu.com/explore/xxx" ./output');
  process.exit(1);
}

// 创建输出目录
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

// 提取笔记 ID
function extractNoteId(url) {
  const patterns = [
    /explore\/([a-f0-9]+)/,
    /discovery\/item\/([a-f0-9]+)/,
    /modal_id=([a-f0-9]+)/,
  ];
  
  for (const pattern of patterns) {
    const match = url.match(pattern);
    if (match) return match[1];
  }
  return null;
}

const NOTE_ID = extractNoteId(URL);
if (!NOTE_ID) {
  console.log('❌ 无法从小红书 URL 中提取笔记 ID');
  process.exit(1);
}

console.log(`开始解析小红书：${URL}`);
console.log(`笔记 ID: ${NOTE_ID}`);
console.log(`输出目录：${OUTPUT_DIR}`);
console.log('---');

// 默认 Cookie（如果用户提供则使用用户的）
const DEFAULT_COOKIE = COOKIE || process.env.XHS_COOKIE || '';

const options = {
  headers: {
    'Cookie': DEFAULT_COOKIE,
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
  }
};

// 下载文件辅助函数
function downloadFile(url, filepath) {
  return new Promise((resolve, reject) => {
    https.get(url, options, (res) => {
      if (res.statusCode === 302 || res.statusCode === 301) {
        // 重定向
        downloadFile(res.headers.location, filepath).then(resolve).catch(reject);
        return;
      }
      
      const chunks = [];
      res.on('data', chunk => chunks.push(chunk));
      res.on('end', () => {
        fs.writeFileSync(filepath, Buffer.concat(chunks));
        const size = (fs.statSync(filepath).size / 1024).toFixed(1);
        resolve(size);
      });
    }).on('error', reject);
  });
}

// 提取 imageList（使用括号匹配）
function extractImageList(html) {
  const startIndex = html.indexOf('"imageList":');
  if (startIndex === -1) {
    return null;
  }
  
  const arrayStart = html.indexOf('[', startIndex);
  if (arrayStart === -1) {
    return null;
  }
  
  // 括号匹配找到数组结束位置
  let depth = 0;
  let endIndex = arrayStart;
  for (let i = arrayStart; i < html.length; i++) {
    if (html[i] === '[') depth++;
    else if (html[i] === ']') depth--;
    if (depth === 0) {
      endIndex = i;
      break;
    }
  }
  
  const imageListStr = html.substring(arrayStart, endIndex + 1);
  
  try {
    return JSON.parse(imageListStr);
  } catch (e) {
    console.log('⚠️ 解析 imageList 失败:', e.message);
    return null;
  }
}

// 判断类型并下载
https.get(URL, options, (res) => {
  let data = '';
  res.on('data', chunk => data += chunk);
  res.on('end', async () => {
    try {
      // 尝试提取 imageList
      const imageList = extractImageList(data);
      
      if (imageList && imageList.length > 0) {
        // 图集类型
        console.log(`✅ 图集类型，找到 ${imageList.length} 张图片`);
        
        const imagesDir = path.join(OUTPUT_DIR, 'images');
        if (!fs.existsSync(imagesDir)) {
          fs.mkdirSync(imagesDir, { recursive: true });
        }
        
        // 下载所有图片
        let successCount = 0;
        for (let i = 0; i < imageList.length; i++) {
          const img = imageList[i];
          const infoList = img.infoList || [];
          const imgInfo = infoList.find(info => info.imageScene === 'WB_DFT') || infoList[0];
          
          if (!imgInfo || !imgInfo.url) {
            console.log(`⚠️ 图片 ${i + 1} 无有效 URL`);
            continue;
          }
          
          // 转换 http 为 https
          let imgUrl = imgInfo.url.replace('http://', 'https://');
          
          // 去除水印参数（可选）
          // imgUrl = imgUrl.split('?')[0];
          
          const filename = path.join(imagesDir, `image-${String(i + 1).padStart(2, '0')}.webp`);
          
          try {
            const size = await downloadFile(imgUrl, filename);
            console.log(`✅ 已保存：image-${String(i + 1).padStart(2, '0')}.webp (${size}KB)`);
            successCount++;
          } catch (err) {
            console.log(`❌ 图片 ${i + 1} 下载失败：${err.message}`);
          }
        }
        
        console.log('---');
        console.log(`完成！共 ${imageList.length} 张图片，成功下载 ${successCount} 张`);
        console.log(`输出目录：${imagesDir}`);
        const analysis = buildAnalysis({
          title: '',
          text: '',
          platform: '小红书',
          mediaType: '图片',
          visual: `图集共 ${imageList.length} 张，成功下载 ${successCount} 张`,
          maxChars: 100,
        });
        console.log(JSON.stringify({
          type: 'album',
          note_id: NOTE_ID,
          total_images: imageList.length,
          downloaded_images: successCount,
          output_dir: imagesDir,
          content_analysis: analysis.analysis,
          analysis_score: analysis.score,
          analysis_evidence: analysis.evidence,
        }, null, 2));
        
      } else {
        // 可能是视频类型，尝试提取视频信息
        console.log('ℹ️ 未找到 imageList，可能是视频类型');
        
        // 尝试使用 yt-dlp 下载视频
        try {
          console.log('尝试使用 yt-dlp 下载视频...');
          
          const videoPath = path.join(OUTPUT_DIR, 'video.mp4');
          const cookieFile = '/tmp/xhs_cookies.txt';
          
          let cookieArg = '';
          if (DEFAULT_COOKIE) {
            // 创建临时 Cookie 文件
            const cookieContent = DEFAULT_COOKIE.split(';').map(c => {
              const [key, value] = c.trim().split('=');
              if (key && value) {
                return `.xiaohongshu.com\tTRUE\t/\tTRUE\t0\t${key.trim()}\t${value.trim()}`;
              }
              return '';
            }).filter(line => line).join('\n');
            
            fs.writeFileSync(cookieFile, cookieContent);
            cookieArg = `--cookies ${cookieFile}`;
          }
          
          const cmd = `yt-dlp ${cookieArg} -f best -o "${videoPath}" "${URL}" 2>&1`;
          const result = execSync(cmd, { encoding: 'utf8' });
          
          if (fs.existsSync(videoPath)) {
            const size = (fs.statSync(videoPath).size / 1024 / 1024).toFixed(1);
            console.log('---');
            console.log(`✅ 视频已下载：video.mp4 (${size}M)`);
            console.log(`输出目录：${OUTPUT_DIR}`);
            console.log('');
            console.log('提示：使用以下命令进行视频抽帧：');
            console.log(`  ffmpeg -i ${videoPath} -vf "fps=1/5" -q:v 2 ${path.join(OUTPUT_DIR, 'frame_%03d.jpg')}`);
            const analysis = buildAnalysis({
              title: '',
              text: '',
              platform: '小红书',
              mediaType: '视频',
              visual: `视频已下载，大小 ${size}MB`,
              maxChars: 100,
            });
            console.log(JSON.stringify({
              type: 'video',
              note_id: NOTE_ID,
              download_path: videoPath,
              file_size_mb: size,
              content_analysis: analysis.analysis,
              analysis_score: analysis.score,
              analysis_evidence: analysis.evidence,
            }, null, 2));
          } else {
            console.log('❌ 视频下载失败，文件不存在');
          }
          
        } catch (err) {
          console.log('❌ yt-dlp 下载失败:', err.message);
          console.log('');
          console.log('提示：');
          console.log('  1. 确保已安装 yt-dlp: brew install yt-dlp');
          console.log('  2. 提供有效的 Cookie: --cookie "unread=xxx; id_token=xxx"');
          console.log('  3. 检查 URL 是否有效');
        }
      }
      
    } catch (err) {
      console.log('❌ 解析失败:', err.message);
      process.exit(1);
    }
  });
}).on('error', (err) => {
  console.log('❌ 请求失败:', err.message);
  process.exit(1);
});


path: scripts/run_backfill.js
content: #!/usr/bin/env node
/**
 * social-media-analysis 的 Clawhub 自动执行入口：
 * 依次运行两个回填脚本：
 * 1) backfill-content-analysis-by-source.js（非新浪微博渠道）
 * 2) backfill-weibo-content-analysis.js（新浪微博渠道）
 */

const { spawn } = require("child_process");
const path = require("path");

function runNode(scriptFile) {
  return new Promise((resolve, reject) => {
    const child = spawn(process.execPath, [scriptFile], {
      stdio: "inherit",
      env: process.env,
    });
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) resolve();
      else reject(new Error(`script_failed=${path.basename(scriptFile)} exit_code=${code}`));
    });
  });
}

async function main() {
  const smoke = ["1", "true", "yes"].includes(
    String(process.env.RUN_SMOKE_TEST || "").trim().toLowerCase()
  );

  if (smoke) {
    const tests = [
      {
        script: "backfill-content-analysis-by-source.js",
        env: { TEST_CHANNEL: "抖音APP", TEST_URL: "https://www.douyin.com/share/video/7621027680988917926" },
      },
      {
        script: "backfill-content-analysis-by-source.js",
        env: { TEST_CHANNEL: "快手APP", TEST_URL: "https://www.kuaishou.com/short-video/3xuyqtb4mxdy4jc" },
      },
      {
        script: "backfill-content-analysis-by-source.js",
        env: { TEST_CHANNEL: "小红书APP", TEST_URL: "https://www.xiaohongshu.com/explore/69c3530f00000000220255d4" },
      },
      {
        script: "backfill-content-analysis-by-source.js",
        env: { TEST_CHANNEL: "酷安APP", TEST_URL: "https://www.coolapk.com/feed/70874324?s=OWNjZGIzYzYwZzY5YzM1NTRlega1603" },
      },
      {
        script: "backfill-content-analysis-by-source.js",
        env: { TEST_CHANNEL: "今日头条", TEST_URL: "http://www.toutiao.com/item/1860602228145289" },
      },
      {
        script: "backfill-weibo-content-analysis.js",
        env: { TEST_SOURCE_CHANNEL: "新浪微博", TEST_URL: "https://weibo.com/3047053367/QxHoSdUEx" },
      },
    ];

    const dir = __dirname;
    for (const t of tests) {
      const scriptFile = path.join(dir, t.script);
      const childEnv = { ...process.env, ...t.env };
      await new Promise((resolve, reject) => {
        const child = require("child_process").spawn(process.execPath, [scriptFile], {
          stdio: "inherit",
          env: childEnv,
        });
        child.on("error", reject);
        child.on("exit", (code) => {
          if (code === 0) resolve();
          else reject(new Error(`smoke_failed=${t.script} exit_code=${code}`));
        });
      });
    }
    return;
  }

  const dir = __dirname;
  const multi = path.join(dir, "backfill-content-analysis-by-source.js");
  const weibo = path.join(dir, "backfill-weibo-content-analysis.js");

  await runNode(multi);
  await runNode(weibo);
}

main().catch((e) => {
  console.error(String(e && e.message ? e.message : e));
  process.exit(1);
});



Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)