Skill: nighmat1220/ai-news-pipeline (version 1.0.4)

--- name: ai-news-pipeline description: Run a self-contained Chinese and international AI news workflow inside the current workspace. Use when the user wants either high-frequency RSS capture only or scheduled report delivery only, with cumulative Excel outputs and a merged Word brief, without relying on an external local repository path. --- # AI News Pipeline ## Overview This skill is executable by itself. The actual workflow scripts are bundled in `scripts/`. Run them against the current workspace or pass `--workspace /path/to/workspace` explicitly. ## Workspace Requirements The target workspace should contain or accept these files and folders: - `config/sources.json` - `config/international_sources.json` - `companies.txt` - `data/` - `reports/` - `state/` If the folders do not exist, the scripts create them. ## Install Dependencies Install Python dependencies before first use: ```bash python -m pip install -r /path/to/skill/scripts/requirements.txt ``` ## Available Entrypoints Use the bundled Python entrypoints depending on the job type. ### Capture Only Use this for high-frequency collection jobs. It only captures feeds, updates deduplication state, and writes raw and incremental data. ```bash python /path/to/skill/scripts/run_capture_only.py --workspace /path/to/workspace ``` ### Report Only Use this for scheduled delivery jobs. It reads already-collected data, calls the model for summaries and titles, updates the cumulative Excel files, and rebuilds the Word brief. By default it uses the reporting window from yesterday 00:00 to today 08:00. ```bash python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace ``` Optional time window: ```bash python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace --time-window "2026-03-15 00:00 to 2026-03-16 08:00" ``` Optional skip-AI mode: ```bash python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace --disable-ai ``` ## Full Workflow ```bash python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace ``` Optional time window: ```bash python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace --time-window "2026-03-15 00:00 to 2026-03-15 18:00" ``` Optional skip-AI mode: ```bash python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace --disable-ai ``` ## What Each Entrypoint Does `run_capture_only.py` 1. Collect domestic RSS items into `data/YYYY-MM-DD.jsonl`. 2. Collect domestic raw items into `data/domestic_raw_YYYY-MM-DD.jsonl`. 3. Collect international raw items into `data/international_raw_YYYY-MM-DD.jsonl`. 4. Filter international items into `data/international_YYYY-MM-DD.jsonl`. 5. Save per-source snapshots in `snapshots/`. 6. Update RSS deduplication and source metrics in `state/feed_state.json`. `run_report_only.py` 1. Read the selected time window from collected data. 2. Build the cumulative domestic Excel output in `reports/company_mentions.xlsx`. 3. Build the cumulative international Excel output in `reports/international_company_mentions.xlsx`. 4. Call the model to generate domestic AI titles and AI summaries. 5. Call the model to generate international AI titles, AI summaries, and impact scores. 6. Build a merged daily Word brief in `reports/`. `run_full_workflow.py` 1. Run capture. 2. Run domestic reporting. 3. Run international reporting. ## Inputs - Domestic RSS config: `config/sources.json` - International RSS config: `config/international_sources.json` - Company list: `companies.txt` - Volcengine key: `ARK_API_KEY` - Optional model override: `ARK_MODEL` ## Security Review Notes This skill is designed to run an AI news collection and reporting workflow inside a user-provided workspace. It accesses external network resources for only two purposes: 1. reading user-configured RSS / Atom feeds to collect public news content; 2. calling a user-configured Volcengine model endpoint to generate AI titles, AI summaries, and impact scores. It writes local files because it needs to: 1. store raw and incremental collected news data; 2. persist deduplication state so repeated runs do not duplicate items; 3. generate cumulative Excel reports and a Word brief; 4. save feed snapshots and logs for troubleshooting and completeness checks. It does not upload arbitrary local files from the workspace and does not scan unrelated user content. External requests are limited to user-configured RSS URLs and the user-configured model endpoint. Credentials are only taken from user-provided configuration, such as RSS authentication data and `ARK_API_KEY`. These credentials are used only at runtime for the intended service and are not forwarded to unrelated destinations. ## Important Behavior - `state/feed_state.json` controls RSS deduplication. - Excel files are cumulative. - The Word brief is rebuilt per run. - The Word international section only includes the top 5 items by impact score inside the selected time window. - International items without a successful AI summary are excluded from the Word brief. - AI cache files are deleted automatically after each run. ## Troubleshooting 1. If the workflow does not rerun old RSS items, check `state/feed_state.json`. 2. If AI columns are empty, check whether `ARK_API_KEY` is set in the execution environment. 3. If the user wants a full rebuild, delete the relevant daily `data` files and `state/feed_state.json`, then rerun. 4. If the user needs exact commands or cloud prompts, read `references/commands.md`. ## References - `references/commands.md`

path: agents/openai.yaml
content: interface:
  display_name: "AI News Intelligence Pipeline"
  short_description: "Capture AI news frequently or generate scheduled Excel and Word briefs"
  default_prompt: "Use $ai-news-pipeline to run report generation only in the current workspace for the reporting window from yesterday 00:00 to today 08:00. Build the cumulative domestic Excel report, build the cumulative international Excel report with AI title, AI summary, and impact score, then generate the merged Word brief."


path: references/commands.md
content: # Commands

## Install dependencies

```bash
python -m pip install -r /path/to/skill/scripts/requirements.txt
```

## Capture only

```bash
python /path/to/skill/scripts/run_capture_only.py --workspace /path/to/workspace
```

## Report only

```bash
python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace
```

## Report only with explicit time window

```bash
python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace --time-window "2026-03-15 00:00 to 2026-03-16 08:00"
```

## Report only without AI generation

```bash
python /path/to/skill/scripts/run_report_only.py --workspace /path/to/workspace --disable-ai
```

## Full workflow

```bash
python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace
```

## Full workflow with time window

```bash
python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace --time-window "2026-03-15 00:00 to 2026-03-15 18:00"
```

## Full workflow without AI generation

```bash
python /path/to/skill/scripts/run_full_workflow.py --workspace /path/to/workspace --disable-ai
```

## Direct scripts

```bash
python /path/to/skill/scripts/collect_feeds.py
python /path/to/skill/scripts/generate_company_report.py
python /path/to/skill/scripts/generate_international_report.py
```

## Required workspace files

- `config/sources.json`
- `config/international_sources.json`
- `companies.txt`

## Outputs

- `data/YYYY-MM-DD.jsonl`
- `data/domestic_raw_YYYY-MM-DD.jsonl`
- `data/international_raw_YYYY-MM-DD.jsonl`
- `data/international_YYYY-MM-DD.jsonl`
- `reports/company_mentions.xlsx`
- `reports/international_company_mentions.xlsx`
- merged daily Word brief in `reports/`

## Cloud troubleshooting

### No Word file generated

Check these outputs first:

- `reports/company_mentions.xlsx`
- `reports/international_company_mentions.xlsx`

If both Excel files are missing, the problem is earlier in the reporting pipeline.
Check:

- whether `ARK_API_KEY` is available
- whether `data/*.jsonl` exists
- whether the task is using the report-only prompt

If both Excel files exist but the Word file is missing, inspect the brief generation step and whether the target file is locked.

### International report is empty

Check in this order:

1. `data/international_raw_YYYY-MM-DD.jsonl`
2. `data/international_YYYY-MM-DD.jsonl`

If raw exists but filtered does not, the feed was captured but nothing matched the filter.
If raw does not exist, inspect `config/international_sources.json`, network access, and source health.

### Domestic count is too low

Check both:

- `data/YYYY-MM-DD.jsonl`
- `reports/company_mentions.xlsx`

If the data file has many rows but the Excel file has few matches, the issue is usually company matching coverage.
Review `companies.txt` aliases.

### AI fields are empty

Check:

- `ARK_API_KEY`
- whether the run used disable-AI mode
- model/API connectivity

Domestic AI fields are generated in `generate_company_report.py`.
International AI fields are generated in `generate_international_report.py`.

### International item is in Excel but not in Word

This can be expected.
The Word brief only includes:

- items inside the selected time window
- items with successful `AI??`
- only the top 5 international items by impact score

### Time window looks wrong

The report uses the published time stored in Excel, not the capture time.
Check `????` and confirm it falls inside the requested window.

### Duplicate ingestion or repeated items

Check `state/feed_state.json`.
If duplicates keep appearing, the cloud workspace is usually not persistent or multiple runs are using different workspaces.

### Historical data disappeared

Confirm the cloud workspace persists these folders:

- `data/`
- `reports/`
- `snapshots/`
- `state/`

### Recommended debugging order

1. `state/feed_state.json`
2. `data/*.jsonl`
3. `data/international_*.jsonl`
4. `reports/company_mentions.xlsx`
5. `reports/international_company_mentions.xlsx`
6. `reports/AI????_YYYY-MM-DD.docx`

## OpenClaw prompt templates

Capture job:

```text
Use $ai-news-pipeline to run capture only in the current workspace. Collect domestic RSS news and international RSS news, deduplicate them, write domestic raw and incremental files, write international raw and filtered incremental files, save feed snapshots, and update feed state metrics. Do not generate Excel or Word reports in this run.
```

Report delivery job:

```text
Use $ai-news-pipeline to run report generation only in the current workspace for the reporting window from yesterday 00:00 to today 08:00. Build the cumulative domestic Excel report, build the cumulative international Excel report with AI title, AI summary, and impact score, then generate the merged Word brief. The Word file should include all domestic items in the window and only the top 5 international items by impact score in the window. Exclude any international item that does not have a successful AI summary.
```

Report delivery job with explicit time window:

```text
Use $ai-news-pipeline to run report generation only in the current workspace for the time window "2026?3?15?0??2026?3?16?8?". Update the domestic Excel report, update the international Excel report with AI title, AI summary, and impact score, and rebuild the merged Word brief using only items whose published time falls inside that window.
```


path: scripts/brief_writer.py
content: from __future__ import annotations

from datetime import datetime
from pathlib import Path
from zipfile import BadZipFile

from docx import Document
from docx.enum.text import WD_ALIGN_PARAGRAPH, WD_LINE_SPACING
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
from docx.shared import Mm, Pt
from openpyxl import load_workbook

from time_window import in_time_window, parse_time_window


TITLE_FONT = "方正小标宋简体"
BODY_FONT = "仿宋_GB2312"
HEADING_FONT = "黑体"
SUBHEADING_FONT = "楷体_GB2312"
PAGE_FONT = "宋体"
LATIN_FONT = "Times New Roman"

TITLE_SIZE_PT = 22
BODY_SIZE_PT = 16
PAGE_SIZE_PT = 14
LINE_SPACING_PT = 28
FIRST_LINE_INDENT_PT = BODY_SIZE_PT * 2


def resolve_writable_report_path(preferred_path: Path) -> Path:
    try:
        preferred_path.parent.mkdir(parents=True, exist_ok=True)
        if not preferred_path.exists():
            return preferred_path
        with preferred_path.open("ab"):
            pass
        return preferred_path
    except PermissionError:
        timestamp = datetime.now().strftime("%H%M%S")
        return preferred_path.with_name(f"{preferred_path.stem}_{timestamp}{preferred_path.suffix}")


def build_time_window_text(paths: list[Path], time_window: str = "") -> str:
    if isinstance(time_window, str) and time_window.strip():
        return normalize_window_display_text(time_window)

    if not paths:
        return f"资讯时间窗口：{datetime.now().date().isoformat()}"

    dates = sorted(path.stem.replace("international_", "") for path in paths)
    if len(dates) == 1:
        return f"资讯时间窗口：{dates[0]}"
    return f"资讯时间窗口：{dates[0]}至{dates[-1]}"


def normalize_window_display_text(value: str) -> str:
    normalized = value.strip().replace(" to ", "至")
    if "至" not in normalized:
        return f"资讯时间窗口：{normalized}"

    start_text, end_text = [part.strip() for part in normalized.split("至", 1)]
    start_text = start_text.split(" ")[0]
    end_text = end_text.split(" ")[0]
    return f"资讯时间窗口：{start_text}至{end_text}"


def load_latest_domestic_rows(report_dir: Path, report_date: str, time_window: str = "") -> list[dict[str, str]]:
    candidates = sorted(
        list(report_dir.glob("company_mentions.xlsx")) + list(report_dir.glob(f"company_mentions_{report_date}*.xlsx")),
        key=lambda path: path.stat().st_mtime,
        reverse=True,
    )
    return filter_rows(load_rows_from_candidates(candidates), report_date, time_window)


def load_latest_international_rows(report_dir: Path, report_date: str, time_window: str = "") -> list[dict[str, str]]:
    candidates = sorted(
        list(report_dir.glob("international_company_mentions.xlsx"))
        + list(report_dir.glob(f"international_company_mentions_{report_date}*.xlsx")),
        key=lambda path: path.stat().st_mtime,
        reverse=True,
    )
    return filter_rows(load_rows_from_candidates(candidates), report_date, time_window)


def load_rows_from_candidates(candidates: list[Path]) -> list[dict[str, str]]:
    for candidate in candidates:
        try:
            workbook = load_workbook(candidate, read_only=True)
            worksheet = workbook.active
            headers = [cell.value for cell in next(worksheet.iter_rows(min_row=1, max_row=1))]
            rows: list[dict[str, str]] = []
            for values in worksheet.iter_rows(min_row=2, values_only=True):
                row = {str(headers[index]): str(value or "") for index, value in enumerate(values)}
                rows.append(row)
            return rows
        except (BadZipFile, OSError, StopIteration, KeyError, ValueError):
            continue
    return []


def filter_rows(rows: list[dict[str, str]], report_date: str, time_window: str = "") -> list[dict[str, str]]:
    if time_window:
        start, end = parse_time_window(time_window)
        return [
            row
            for row in rows
            if in_time_window(str(row.get("资讯时间", "") or ""), start, end)
        ]

    filtered: list[dict[str, str]] = []
    for row in rows:
        news_time = str(row.get("资讯时间", "") or "")
        if news_time.startswith(report_date):
            filtered.append(row)
    return filtered


def top_international_rows(rows: list[dict[str, str]], limit: int = 5) -> list[dict[str, str]]:
    valid_rows = [row for row in rows if str(row.get("AI摘要", "") or "").strip()]

    def sort_key(row: dict[str, str]) -> tuple[int, str]:
        try:
            score = int(str(row.get("事件影响力", "") or 0))
        except ValueError:
            score = 0
        return (-score, str(row.get("资讯时间", "") or ""))

    return sorted(valid_rows, key=sort_key)[:limit]


def set_run_font(run, east_asia_font: str, size_pt: float, *, latin_font: str = LATIN_FONT, bold: bool = False) -> None:
    run.bold = bold
    run.font.size = Pt(size_pt)
    run.font.name = latin_font
    rpr = run._element.get_or_add_rPr()
    rfonts = rpr.rFonts
    rfonts.set(qn("w:ascii"), latin_font)
    rfonts.set(qn("w:hAnsi"), latin_font)
    rfonts.set(qn("w:eastAsia"), east_asia_font)


def style_paragraph(paragraph, *, alignment, first_line_indent_pt: float = FIRST_LINE_INDENT_PT) -> None:
    paragraph.alignment = alignment
    paragraph.paragraph_format.first_line_indent = Pt(first_line_indent_pt)
    paragraph.paragraph_format.space_before = Pt(0)
    paragraph.paragraph_format.space_after = Pt(0)
    paragraph.paragraph_format.line_spacing_rule = WD_LINE_SPACING.EXACTLY
    paragraph.paragraph_format.line_spacing = Pt(LINE_SPACING_PT)


def add_text_paragraph(
    document: Document,
    text: str,
    *,
    east_asia_font: str,
    size_pt: float,
    alignment,
    first_line_indent_pt: float = FIRST_LINE_INDENT_PT,
) -> None:
    paragraph = document.add_paragraph()
    style_paragraph(paragraph, alignment=alignment, first_line_indent_pt=first_line_indent_pt)
    run = paragraph.add_run(text)
    set_run_font(run, east_asia_font, size_pt)


def add_blank_line(document: Document) -> None:
    paragraph = document.add_paragraph()
    style_paragraph(paragraph, alignment=WD_ALIGN_PARAGRAPH.LEFT, first_line_indent_pt=0)
    run = paragraph.add_run("")
    set_run_font(run, BODY_FONT, BODY_SIZE_PT)


def chinese_section_number(index: int) -> str:
    numerals = ["一", "二", "三", "四", "五", "六", "七", "八", "九", "十"]
    if 1 <= index <= 10:
        return numerals[index - 1]
    if index < 20:
        return "十" + numerals[index - 11]
    tens, ones = divmod(index, 10)
    prefix = numerals[tens - 1] + "十"
    return prefix if ones == 0 else prefix + numerals[ones - 1]


def add_body_item(document: Document, index: int, title_text: str, body_text: str, source_name: str) -> None:
    title_paragraph = document.add_paragraph()
    style_paragraph(title_paragraph, alignment=WD_ALIGN_PARAGRAPH.LEFT)
    title_run = title_paragraph.add_run(f"（{chinese_section_number(index)}）{title_text}")
    set_run_font(title_run, SUBHEADING_FONT, BODY_SIZE_PT)

    body_paragraph = document.add_paragraph()
    style_paragraph(body_paragraph, alignment=WD_ALIGN_PARAGRAPH.JUSTIFY)
    body_run = body_paragraph.add_run(body_text)
    set_run_font(body_run, BODY_FONT, BODY_SIZE_PT)
    if source_name:
        source_run = body_paragraph.add_run(f"（{source_name}）")
        set_run_font(source_run, SUBHEADING_FONT, BODY_SIZE_PT)


def add_empty_body(document: Document, text: str) -> None:
    add_text_paragraph(
        document,
        text,
        east_asia_font=BODY_FONT,
        size_pt=BODY_SIZE_PT,
        alignment=WD_ALIGN_PARAGRAPH.JUSTIFY,
    )


def configure_page_layout(document: Document) -> None:
    section = document.sections[0]
    section.top_margin = Mm(37)
    section.bottom_margin = Mm(35)
    section.left_margin = Mm(28)
    section.right_margin = Mm(26)


def build_page_number_field() -> list[OxmlElement]:
    begin = OxmlElement("w:fldChar")
    begin.set(qn("w:fldCharType"), "begin")
    instr = OxmlElement("w:instrText")
    instr.set(qn("xml:space"), "preserve")
    instr.text = " PAGE "
    end = OxmlElement("w:fldChar")
    end.set(qn("w:fldCharType"), "end")
    return [begin, instr, end]


def configure_footer(document: Document) -> None:
    footer = document.sections[0].footer
    paragraph = footer.paragraphs[0] if footer.paragraphs else footer.add_paragraph()
    paragraph.clear()
    style_paragraph(paragraph, alignment=WD_ALIGN_PARAGRAPH.CENTER, first_line_indent_pt=0)

    left_run = paragraph.add_run("-")
    set_run_font(left_run, PAGE_FONT, PAGE_SIZE_PT)

    page_run = paragraph.add_run()
    set_run_font(page_run, PAGE_FONT, PAGE_SIZE_PT)
    for node in build_page_number_field():
        page_run._r.append(node)

    right_run = paragraph.add_run("-")
    set_run_font(right_run, PAGE_FONT, PAGE_SIZE_PT)


def write_daily_brief(
    report_dir: Path,
    paths: list[Path],
    domestic_rows: list[dict[str, str]],
    international_rows: list[dict[str, str]],
    time_window: str = "",
) -> Path:
    report_date = datetime.now().date().isoformat()
    output_path = resolve_writable_report_path(report_dir / f"AI资讯简报_{report_date}.docx")

    document = Document()
    configure_page_layout(document)
    configure_footer(document)

    add_text_paragraph(
        document,
        "每日AI产业资讯简报",
        east_asia_font=TITLE_FONT,
        size_pt=TITLE_SIZE_PT,
        alignment=WD_ALIGN_PARAGRAPH.CENTER,
        first_line_indent_pt=0,
    )
    add_text_paragraph(
        document,
        build_time_window_text(paths, time_window),
        east_asia_font=BODY_FONT,
        size_pt=BODY_SIZE_PT,
        alignment=WD_ALIGN_PARAGRAPH.CENTER,
        first_line_indent_pt=0,
    )
    add_blank_line(document)

    add_text_paragraph(
        document,
        "一、AI领军企业动态",
        east_asia_font=HEADING_FONT,
        size_pt=BODY_SIZE_PT,
        alignment=WD_ALIGN_PARAGRAPH.LEFT,
    )
    if not domestic_rows:
        add_empty_body(document, "今日无命中企业资讯。")
    else:
        for index, row in enumerate(domestic_rows, start=1):
            ai_title = row.get("AI标题") or row.get("资讯标题") or "未命名资讯"
            ai_summary = row.get("AI摘要") or row.get("资讯内容") or ""
            source_name = row.get("资讯来源", "")
            add_body_item(document, index, ai_title, ai_summary, source_name)

    add_text_paragraph(
        document,
        "二、全球AI产业动态",
        east_asia_font=HEADING_FONT,
        size_pt=BODY_SIZE_PT,
        alignment=WD_ALIGN_PARAGRAPH.LEFT,
    )
    selected_international_rows = top_international_rows(international_rows, limit=5)
    if not selected_international_rows:
        add_empty_body(document, "今日无国际AI资讯。")
    else:
        for index, row in enumerate(selected_international_rows, start=1):
            ai_title = row.get("AI标题") or row.get("资讯标题") or "未命名资讯"
            ai_summary = row.get("AI摘要") or ""
            source_name = row.get("资讯来源", "")
            add_body_item(document, index, ai_title, ai_summary, source_name)

    document.save(output_path)
    return output_path


path: scripts/collect_feeds.py
content: from __future__ import annotations

import argparse
import base64
import hashlib
import json
import os
import sys
from dataclasses import dataclass
from datetime import datetime
from email.utils import parsedate_to_datetime
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from xml.etree import ElementTree as ET


BASE_DIR = Path(os.getenv("AI_NEWS_WORKSPACE", os.getcwd())).resolve()
DEFAULT_CONFIG_PATH = BASE_DIR / "config" / "sources.json"
DEFAULT_INTERNATIONAL_CONFIG_PATH = BASE_DIR / "config" / "international_sources.json"
LEGACY_INTERNATIONAL_CONFIG_PATH = BASE_DIR / "international_sources.json"
STATE_PATH = BASE_DIR / "state" / "feed_state.json"
DATA_DIR = BASE_DIR / "data"
LOG_DIR = BASE_DIR / "logs"
USER_AGENT = "AINewsCollect/1.0 (+rss-collector)"
MAX_SEEN_IDS_PER_SOURCE = 5000

INTERNATIONAL_DATA_PREFIX = "international"
INTERNATIONAL_TOPICS = [
    "llm",
    "foundation model",
    "foundation models",
    "model",
    "models",
    "agent",
    "agents",
    "ai",
    "artificial intelligence",
    "generative ai",
    "genai",
    "multimodal",
    "reasoning model",
    "reasoning models",
]
INTERNATIONAL_EVENTS = [
    "guideline",
    "guidelines",
    "funding",
    "fundraise",
    "fundraising",
    "raise",
    "raises",
    "raised",
    "investment",
    "invest",
    "invests",
    "acquisition",
    "acquire",
    "acquires",
    "acquired",
    "launch",
    "launches",
    "launched",
    "release",
    "releases",
    "released",
    "approval",
    "approved",
    "policy",
    "regulation",
    "regulatory",
    "partnership",
    "partner",
    "partners",
    "deal",
]
AI_SCOPED_SOURCE_HINTS = [
    "artificial-intelligence",
    "artificialintelligence",
    "/ai/",
    "ai-techpark",
    "venturebeat.com/category/ai",
]

ATOM_NS = {"atom": "http://www.w3.org/2005/Atom"}
CONTENT_NAMESPACES = [
    "{http://purl.org/rss/1.0/modules/content/}encoded",
    "{http://www.w3.org/2005/Atom}content",
    "{http://www.w3.org/2005/Atom}summary",
]


@dataclass
class SourceConfig:
    name: str
    url: str
    enabled: bool = True
    headers: dict[str, str] | None = None
    username: str | None = None
    password: str | None = None


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Collect new entries from RSS/Atom feeds.")
    parser.add_argument(
        "--config",
        type=Path,
        default=DEFAULT_CONFIG_PATH,
        help=f"Path to the domestic sources config file. Default: {DEFAULT_CONFIG_PATH}",
    )
    parser.add_argument(
        "--international-config",
        type=Path,
        default=DEFAULT_INTERNATIONAL_CONFIG_PATH,
        help=(
            "Path to the international sources config file. "
            f"Default: {DEFAULT_INTERNATIONAL_CONFIG_PATH}"
        ),
    )
    return parser.parse_args()


def configure_output_encoding() -> None:
    for stream_name in ("stdout", "stderr"):
        stream = getattr(sys, stream_name, None)
        if stream is None:
            continue
        reconfigure = getattr(stream, "reconfigure", None)
        if callable(reconfigure):
            reconfigure(encoding="utf-8", errors="replace")


def ensure_directories() -> None:
    DATA_DIR.mkdir(parents=True, exist_ok=True)
    LOG_DIR.mkdir(parents=True, exist_ok=True)
    STATE_PATH.parent.mkdir(parents=True, exist_ok=True)
    DEFAULT_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
    DEFAULT_INTERNATIONAL_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)


def load_sources(config_path: Path, *, required: bool) -> list[SourceConfig]:
    path = config_path
    if not path.exists():
        if path == DEFAULT_INTERNATIONAL_CONFIG_PATH and LEGACY_INTERNATIONAL_CONFIG_PATH.exists():
            path = LEGACY_INTERNATIONAL_CONFIG_PATH
        elif required:
            raise FileNotFoundError(f"Config file not found: {path}")
        else:
            return []

    with path.open("r", encoding="utf-8") as file:
        data = json.load(file)

    if not isinstance(data, list):
        raise ValueError(f"{path.name} must be a JSON array of source objects.")

    sources: list[SourceConfig] = []
    for index, item in enumerate(data, start=1):
        if not isinstance(item, dict):
            raise ValueError(f"Source #{index} in {path.name} must be an object.")

        name = str(item.get("name", "")).strip()
        url = str(item.get("url", "")).strip()
        if not name or not url:
            raise ValueError(f"Source #{index} in {path.name} is missing name or url.")

        sources.append(
            SourceConfig(
                name=name,
                url=url,
                enabled=bool(item.get("enabled", True)),
                headers=dict(item.get("headers") or {}),
                username=item.get("username"),
                [REDACTED_SECRET]"password"),
            )
        )

    return [source for source in sources if source.enabled]


def load_state() -> dict[str, Any]:
    if not STATE_PATH.exists():
        return {"sources": {}, "international_sources": {}}

    with STATE_PATH.open("r", encoding="utf-8") as file:
        state = json.load(file)

    if not isinstance(state, dict):
        return {"sources": {}, "international_sources": {}}

    state.setdefault("sources", {})
    state.setdefault("international_sources", {})
    return state


def save_state(state: dict[str, Any]) -> None:
    with STATE_PATH.open("w", encoding="utf-8") as file:
        json.dump(state, file, ensure_ascii=False, indent=2)


def append_failure_log(message: str, *, source_name: str | None = None) -> Path:
    today = datetime.now().strftime("%Y-%m-%d")
    log_path = LOG_DIR / f"{today}.log"
    timestamp = datetime.now().astimezone().isoformat()
    prefix = f"[{timestamp}]"
    if source_name:
        prefix = f"{prefix} [{source_name}]"

    with log_path.open("a", encoding="utf-8") as file:
        file.write(f"{prefix} {message}\n")

    return log_path


def build_request(source: SourceConfig) -> Request:
    headers = {"User-Agent": USER_AGENT}
    if source.headers:
        headers.update(source.headers)

    if source.username and source.[REDACTED_SECRET] = base64.b64encode(f"{source.username}:{source.password}".encode("utf-8")).decode("ascii")
        headers["Authorization"] = f"Basic {basic_token}"

    return Request(source.url, headers=headers)


def fetch_feed(source: SourceConfig) -> bytes:
    request = build_request(source)
    with urlopen(request, timeout=30) as response:
        return response.read()


def extract_text(element: ET.Element | None, *tags: str) -> str:
    if element is None:
        return ""

    for tag in tags:
        child = element.find(tag, ATOM_NS)
        if child is not None and child.text:
            return child.text.strip()
    return ""


def extract_content(element: ET.Element) -> str:
    description = extract_text(element, "description", "atom:summary")
    if description:
        return description

    for tag in CONTENT_NAMESPACES:
        child = element.find(tag)
        if child is not None:
            text = "".join(child.itertext()).strip()
            if text:
                return text
    return ""


def extract_link(element: ET.Element) -> str:
    link_text = extract_text(element, "link")
    if link_text:
        return link_text

    for link_element in element.findall("atom:link", ATOM_NS):
        href = link_element.attrib.get("href", "").strip()
        rel = link_element.attrib.get("rel", "alternate").strip()
        if href and rel in {"", "alternate"}:
            return href
    return ""


def normalize_datetime(value: str) -> str:
    cleaned = value.strip()
    if not cleaned:
        return ""

    try:
        return parsedate_to_datetime(cleaned).isoformat()
    except (TypeError, ValueError, IndexError, OverflowError):
        pass

    try:
        return datetime.fromisoformat(cleaned.replace("Z", "+00:00")).isoformat()
    except ValueError:
        return cleaned


def build_item_id(source_name: str, guid: str, link: str, title: str, published_at: str, content: str) -> str:
    raw = "||".join([source_name, guid, link, title, published_at, content])
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()


def parse_rss_item(item: ET.Element, source: SourceConfig) -> dict[str, str]:
    title = extract_text(item, "title")
    guid = extract_text(item, "guid")
    link = extract_link(item)
    published_at = normalize_datetime(extract_text(item, "pubDate"))
    content = extract_content(item)
    return {
        "source_name": source.name,
        "source_url": source.url,
        "title": title,
        "content": content,
        "published_at": published_at,
        "link": link,
        "guid": guid,
        "item_id": build_item_id(source.name, guid, link, title, published_at, content),
    }


def parse_atom_entry(entry: ET.Element, source: SourceConfig) -> dict[str, str]:
    title = extract_text(entry, "atom:title")
    guid = extract_text(entry, "atom:id")
    link = extract_link(entry)
    published_at = normalize_datetime(extract_text(entry, "atom:published", "atom:updated"))
    content = extract_content(entry)
    return {
        "source_name": source.name,
        "source_url": source.url,
        "title": title,
        "content": content,
        "published_at": published_at,
        "link": link,
        "guid": guid,
        "item_id": build_item_id(source.name, guid, link, title, published_at, content),
    }


def parse_feed(content: bytes, source: SourceConfig) -> list[dict[str, str]]:
    root = ET.fromstring(content)

    items = root.findall("./channel/item")
    if not items and root.tag.endswith("RDF"):
        items = root.findall("./item")
    if items:
        return [parse_rss_item(item, source) for item in items]

    entries = root.findall("./atom:entry", ATOM_NS)
    if entries:
        return [parse_atom_entry(entry, source) for entry in entries]

    raise ValueError(f"Unrecognized RSS/Atom format for {source.name}")


def append_records(records: list[dict[str, Any]], *, prefix: str | None = None) -> Path | None:
    if not records:
        return None

    today = datetime.now().strftime("%Y-%m-%d")
    filename = f"{today}.jsonl" if prefix is None else f"{prefix}_{today}.jsonl"
    output_path = DATA_DIR / filename

    with output_path.open("a", encoding="utf-8") as file:
        for record in records:
            file.write(json.dumps(record, ensure_ascii=False) + "\n")
    return output_path


def deduplicate_new_items(state: dict[str, Any], source: SourceConfig, items: list[dict[str, str]], *, state_key: str) -> list[dict[str, Any]]:
    source_state = state[state_key].setdefault(source.name, {"seen_ids": []})
    seen_ids = set(source_state.get("seen_ids", []))
    new_records: list[dict[str, Any]] = []

    for item in items:
        item_id = item["item_id"]
        if item_id in seen_ids:
            continue

        record = {
            "captured_at": datetime.now().astimezone().isoformat(),
            "source_name": item["source_name"],
            "source_url": item["source_url"],
            "title": item["title"],
            "content": item["content"],
            "published_at": item["published_at"],
            "link": item["link"],
            "guid": item["guid"],
        }
        new_records.append(record)
        source_state.setdefault("seen_ids", []).append(item_id)

    source_state["seen_ids"] = source_state.get("seen_ids", [])[-MAX_SEEN_IDS_PER_SOURCE:]
    source_state["last_checked_at"] = datetime.now().astimezone().isoformat()
    return new_records


def normalize_match_text(*values: str) -> str:
    return " ".join(value.lower() for value in values if value).strip()


def count_keyword_hits(match_text: str, keywords: list[str]) -> int:
    return sum(1 for keyword in keywords if keyword in match_text)


def is_ai_scoped_source(source: SourceConfig) -> bool:
    scope_text = normalize_match_text(source.name, source.url)
    return any(hint in scope_text for hint in AI_SCOPED_SOURCE_HINTS)


def is_relevant_international_item(item: dict[str, str], source: SourceConfig) -> bool:
    match_text = normalize_match_text(item.get("title", ""), item.get("content", ""), item.get("link", ""))
    if not match_text:
        return False

    topic_hits = count_keyword_hits(match_text, INTERNATIONAL_TOPICS)
    event_hits = count_keyword_hits(match_text, INTERNATIONAL_EVENTS)
    if event_hits == 0:
        return False

    if is_ai_scoped_source(source):
        return True
    return topic_hits > 0


def collect_source_group(
    sources: list[SourceConfig],
    state: dict[str, Any],
    *,
    state_key: str,
    label: str,
    output_prefix: str | None,
    filter_relevant: bool,
) -> tuple[int, int, Path | None]:
    all_new_records: list[dict[str, Any]] = []
    error_count = 0

    if not sources:
        return 0, 0, None

    for source in sources:
        try:
            raw_feed = fetch_feed(source)
            items = parse_feed(raw_feed, source)
            filtered_items = [item for item in items if is_relevant_international_item(item, source)] if filter_relevant else items
            new_records = deduplicate_new_items(state, source, filtered_items, state_key=state_key)
            all_new_records.extend(new_records)

            if filter_relevant:
                print(f"[OK] {label} {source.name}: fetched {len(items)}, matched {len(filtered_items)}, new {len(new_records)}")
            else:
                print(f"[OK] {label} {source.name}: fetched {len(items)}, new {len(new_records)}")
        except FileNotFoundError:
            raise
        except HTTPError as error:
            error_count += 1
            message = f"{label} HTTP ERROR: {error.code} {error.reason}"
            print(f"[HTTP ERROR] {label} {source.name}: {error.code} {error.reason}", file=sys.stderr)
            append_failure_log(message, source_name=source.name)
        except URLError as error:
            error_count += 1
            message = f"{label} URL ERROR: {error.reason}"
            print(f"[URL ERROR] {label} {source.name}: {error.reason}", file=sys.stderr)
            append_failure_log(message, source_name=source.name)
        except ET.ParseError as error:
            error_count += 1
            message = f"{label} PARSE ERROR: {error}"
            print(f"[PARSE ERROR] {label} {source.name}: {error}", file=sys.stderr)
            append_failure_log(message, source_name=source.name)
        except Exception as error:
            error_count += 1
            message = f"{label} ERROR: {error}"
            print(f"[ERROR] {label} {source.name}: {error}", file=sys.stderr)
            append_failure_log(message, source_name=source.name)

    output_path = append_records(all_new_records, prefix=output_prefix)
    if output_path:
        print(f"Wrote {len(all_new_records)} new {label} records to {output_path}")
    else:
        print(f"No new {label} records this run")

    if error_count:
        append_failure_log(
            f"{label} run completed with {len(all_new_records)} new records and {error_count} failing sources."
        )

    return len(all_new_records), error_count, output_path


def collect_once(config_path: Path, international_config_path: Path) -> int:
    ensure_directories()
    domestic_sources = load_sources(config_path, required=True)
    international_sources = load_sources(international_config_path, required=False)
    state = load_state()

    if not domestic_sources and not international_sources:
        print("No enabled RSS sources found. Fill config/sources.json or config/international_sources.json first.")
        return 0

    domestic_count, _, _ = collect_source_group(
        domestic_sources,
        state,
        state_key="sources",
        label="domestic",
        output_prefix=None,
        filter_relevant=False,
    )
    international_count, _, _ = collect_source_group(
        international_sources,
        state,
        state_key="international_sources",
        label="international",
        output_prefix=INTERNATIONAL_DATA_PREFIX,
        filter_relevant=True,
    )

    save_state(state)
    total_new_records = domestic_count + international_count
    print(f"Total new records this run: {total_new_records}")
    return total_new_records


def main() -> int:
    configure_output_encoding()
    args = parse_args()
    try:
        collect_once(args.config.resolve(), args.international_config.resolve())
        return 0
    except Exception as error:
        print(f"Run failed: {error}", file=sys.stderr)
        append_failure_log(f"Run failed: {error}")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/generate_company_report.py
content: from __future__ import annotations

import argparse
import hashlib
import json
import os
import sys
import time
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from zipfile import BadZipFile

from openpyxl import Workbook
from openpyxl.styles import Alignment, Font

from brief_writer import load_latest_international_rows, write_daily_brief
from time_window import (
    existing_paths_from_stems,
    in_time_window,
    parse_time_window,
    target_date_strings_from_window,
)
from openpyxl import load_workbook


BASE_DIR = Path(os.getenv("AI_NEWS_WORKSPACE", os.getcwd())).resolve()
DATA_DIR = BASE_DIR / "data"
REPORT_DIR = BASE_DIR / "reports"
COMPANIES_PATH = BASE_DIR / "companies.txt"
STATE_DIR = BASE_DIR / "state"
SUMMARY_CACHE_PATH = STATE_DIR / "company_report_ai_cache.json"

ARK_API_BASE = os.getenv("ARK_API_BASE", "https://ark.cn-beijing.volces.com/api/v3")
ARK_MODEL = os.getenv("ARK_MODEL", "doubao-seed-2-0-mini-260215")
ARK_API_KEY = os.getenv("ARK_API_KEY", "").strip()
ARK_TIMEOUT_SECONDS = int(os.getenv("ARK_TIMEOUT_SECONDS", "60"))
MAX_CACHE_ITEMS = 5000


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Generate company news report from JSONL data files.")
    parser.add_argument(
        "--companies",
        type=Path,
        default=COMPANIES_PATH,
        help=f"Path to the company list file. Default: {COMPANIES_PATH}",
    )
    parser.add_argument(
        "--days",
        type=int,
        default=2,
        help="How many days of files to scan, counting today. Default: 2",
    )
    parser.add_argument(
        "--disable-ai",
        action="store_true",
        help="Skip AI summary/title generation even if ARK_API_KEY is set.",
    )
    parser.add_argument(
        "--time-window",
        type=str,
        default="",
        help="资讯时间窗口，例如：2026年3月15日9点到2026年3月15日18点",
    )
    return parser.parse_args()


def configure_output_encoding() -> None:
    for stream_name in ("stdout", "stderr"):
        stream = getattr(sys, stream_name, None)
        reconfigure = getattr(stream, "reconfigure", None)
        if callable(reconfigure):
            reconfigure(encoding="utf-8", errors="replace")


def ensure_directories() -> None:
    REPORT_DIR.mkdir(parents=True, exist_ok=True)
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    COMPANIES_PATH.parent.mkdir(parents=True, exist_ok=True)


def dedupe_preserve_order(values: list[str]) -> list[str]:
    seen: set[str] = set()
    result: list[str] = []
    for value in values:
        if value not in seen:
            result.append(value)
            seen.add(value)
    return result


def load_companies(path: Path) -> list[dict[str, Any]]:
    if not path.exists():
        raise FileNotFoundError(f"Company list file not found: {path}")

    companies: list[dict[str, Any]] = []
    seen: set[str] = set()

    for raw_line in path.read_text(encoding="utf-8").splitlines():
        line = raw_line.strip()
        if not line or line.startswith("#"):
            continue

        parts = [part.strip() for part in line.split("|") if part.strip()]
        if not parts:
            continue

        canonical_name = parts[0]
        if canonical_name in seen:
            continue

        companies.append(
            {
                "name": canonical_name,
                "aliases": dedupe_preserve_order(parts),
            }
        )
        seen.add(canonical_name)

    return companies

def target_data_files(days: int) -> list[Path]:
    today = datetime.now().date()
    paths: list[Path] = []
    for offset in range(days):
        day = today - timedelta(days=offset)
        path = DATA_DIR / f"{day.isoformat()}.jsonl"
        if path.exists():
            paths.append(path)
    return paths


def target_data_files_by_window(time_window: str) -> tuple[list[Path], tuple[datetime, datetime]]:
    start, end = parse_time_window(time_window)
    stems = target_date_strings_from_window(start, end)
    paths = existing_paths_from_stems(DATA_DIR, stems)
    return paths, (start, end)


def read_records(paths: list[Path]) -> list[dict[str, Any]]:
    records: list[dict[str, Any]] = []
    for path in paths:
        with path.open("r", encoding="utf-8") as file:
            for line_number, line in enumerate(file, start=1):
                cleaned = line.strip()
                if not cleaned:
                    continue
                try:
                    records.append(json.loads(cleaned))
                except json.JSONDecodeError as error:
                    print(
                        f"[WARN] 跳过无效 JSON: {path.name}:{line_number} {error}",
                        file=sys.stderr,
                    )
    return records


def record_time_value(record: dict[str, Any]) -> str:
    return str(record.get("published_at", "") or "")


def match_records(records: list[dict[str, Any]], companies: list[dict[str, Any]]) -> list[dict[str, str]]:
    matched_rows: list[dict[str, str]] = []
    for record in records:
        title = str(record.get("title", "") or "")
        content = str(record.get("content", "") or "")
        if not title and not content:
            continue

        combined_text = "\n".join(part for part in [title, content] if part)
        matched_names = [
            str(company["name"])
            for company in companies
            if any(alias in combined_text for alias in company.get("aliases", []))
        ]
        if not matched_names:
            continue

        for company_name in dedupe_preserve_order(matched_names):
            matched_rows.append(
                {
                    "企业名": company_name,
                    "资讯标题": title,
                    "资讯内容": content,
                    "资讯来源": str(record.get("source_name", "") or ""),
                    "资讯时间": record_time_value(record),
                    "资讯链接": str(record.get("link", "") or ""),
                    "AI标题": "",
                    "AI摘要": "",
                }
            )

    matched_rows.sort(
        key=lambda row: (row["企业名"], row["资讯时间"], row["资讯标题"]),
        reverse=True,
    )
    return matched_rows

def filter_records_by_time_window(
    records: list[dict[str, Any]],
    start: datetime,
    end: datetime,
) -> list[dict[str, Any]]:
    return [
        record
        for record in records
        if in_time_window(record_time_value(record), start, end)
    ]


def load_summary_cache() -> dict[str, dict[str, str]]:
    if not SUMMARY_CACHE_PATH.exists():
        return {}

    try:
        with SUMMARY_CACHE_PATH.open("r", encoding="utf-8") as file:
            data = json.load(file)
    except (OSError, json.JSONDecodeError):
        return {}

    if not isinstance(data, dict):
        return {}
    return {str(key): value for key, value in data.items() if isinstance(value, dict)}


def save_summary_cache(cache: dict[str, dict[str, str]]) -> None:
    items = list(cache.items())[-MAX_CACHE_ITEMS:]
    trimmed_cache = {key: value for key, value in items}
    with SUMMARY_CACHE_PATH.open("w", encoding="utf-8") as file:
        json.dump(trimmed_cache, file, ensure_ascii=False, indent=2)


def clear_summary_cache() -> None:
    try:
        if SUMMARY_CACHE_PATH.exists():
            SUMMARY_CACHE_PATH.write_text("{}\n", encoding="utf-8")
    except OSError:
        pass


def make_cache_key(row: dict[str, str]) -> str:
    raw = "||".join(
        [
            row["企业名"],
            row["资讯标题"],
            row["资讯内容"],
            row["资讯来源"],
            row["资讯时间"],
            row["资讯链接"],
        ]
    )
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()


def build_ai_payload(row: dict[str, str]) -> dict[str, Any]:
    prompt = (
        "你是一名中文商业资讯编辑。请基于下面这条资讯生成 JSON。"
        "必须只返回 JSON 对象，格式为 {\"short_title\":\"...\",\"summary\":\"...\"}。"
        "要求："
        "1. short_title 目标为15个中文字符左右，通常不超过20个中文字符。英文字符、英文单词、缩写和数字不计入字数限制。"
        "2. short_title 必须把一句话说完整，不能为了压缩字数截断半句。"
        "3. short_title 可以保留必要的英文企业名、产品名或模型名。"
        "4. summary 目标为100个中文字符左右，通常不超过120个中文字符，信息准确、客观、完整。"
        "5. summary 必须完整表达，不能截断半句，并且必须以中文句号结尾。"
        "6. 不要编造原文没有的信息。"
        "7. 如果原文信息较少，也要尽量概括重点。"
        "\n\n"
        f"企业名：{row['企业名']}\n"
        f"原标题：{row['资讯标题']}\n"
        f"资讯内容：{row['资讯内容']}\n"
        f"资讯来源：{row['资讯来源']}\n"
        f"资讯时间：{row['资讯时间']}\n"
        f"资讯链接：{row['资讯链接']}\n"
    )

    return {
        "model": ARK_MODEL,
        "input": prompt,
    }

def extract_text_from_response(payload: dict[str, Any]) -> str:
    output_text = payload.get("output_text")
    if isinstance(output_text, str) and output_text.strip():
        return output_text.strip()

    output = payload.get("output")
    if isinstance(output, list):
        texts: list[str] = []
        for item in output:
            if not isinstance(item, dict):
                continue
            content = item.get("content")
            if not isinstance(content, list):
                continue
            for part in content:
                if not isinstance(part, dict):
                    continue
                text = part.get("text")
                if isinstance(text, str) and text.strip():
                    texts.append(text.strip())
        if texts:
            return "\n".join(texts)

    choices = payload.get("choices")
    if isinstance(choices, list):
        texts = []
        for choice in choices:
            if not isinstance(choice, dict):
                continue
            message = choice.get("message")
            if isinstance(message, dict):
                content = message.get("content")
                if isinstance(content, str) and content.strip():
                    texts.append(content.strip())
            text = choice.get("text")
            if isinstance(text, str) and text.strip():
                texts.append(text.strip())
        if texts:
            return "\n".join(texts)

    return ""


def ensure_cn_sentence_end(text: str) -> str:
    cleaned = " ".join(text.split()).strip()
    if not cleaned:
        return ""
    if cleaned[-1] not in "。！？":
        cleaned += "。"
    return cleaned


def call_ark_api(row: dict[str, str]) -> dict[str, str]:
    if not ARK_API_KEY:
        raise RuntimeError("未设置 ARK_API_KEY")

    payload = build_ai_payload(row)
    body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    request = Request(
        f"{ARK_API_BASE.rstrip('/')}/responses",
        data=body,
        headers={
            "Content-Type": "application/json; charset=utf-8",
            "Authorization": f"Bearer {ARK_API_KEY}",
        },
        method="POST",
    )

    with urlopen(request, timeout=ARK_TIMEOUT_SECONDS) as response:
        response_payload = json.loads(response.read().decode("utf-8"))

    text = extract_text_from_response(response_payload)
    if not text:
        raise RuntimeError("模型返回内容为空")

    try:
        parsed = json.loads(text)
    except json.JSONDecodeError as error:
        raise RuntimeError(f"模型返回不是有效 JSON: {error}") from error

    short_title = str(parsed.get("short_title", "") or "").strip()
    summary = str(parsed.get("summary", "") or "").strip()
    if not short_title or not summary:
        raise RuntimeError("模型返回缺少 short_title 或 summary")

    summary = ensure_cn_sentence_end(summary)

    return {"AI标题": short_title, "AI摘要": summary}

def enrich_rows_with_ai(rows: list[dict[str, str]], disable_ai: bool) -> tuple[list[dict[str, str]], int]:
    if disable_ai:
        print("已按参数跳过 AI 摘要生成。")
        return rows, 0

    if not ARK_API_KEY:
        print("未设置 ARK_API_KEY，已跳过 AI 摘要生成。")
        return rows, 0

    if not rows:
        return rows, 0

    cache = load_summary_cache()
    generated_count = 0

    for index, row in enumerate(rows, start=1):
        cache_key = make_cache_key(row)
        cached = cache.get(cache_key)
        if cached:
            row["AI标题"] = str(cached.get("AI标题", "") or "")
            row["AI摘要"] = str(cached.get("AI摘要", "") or "")
            continue

        try:
            result = call_ark_api(row)
            row.update(result)
            cache[cache_key] = result
            generated_count += 1
            print(f"[AI] 已生成 {index}/{len(rows)}: {row['企业名']} | {row['资讯标题']}")
            time.sleep(0.4)
        except (HTTPError, URLError, TimeoutError, RuntimeError, OSError) as error:
            print(f"[AI WARN] {row['企业名']} 生成失败: {error}", file=sys.stderr)

    save_summary_cache(cache)
    return rows, generated_count


def autosize_worksheet(worksheet) -> None:
    widths = {
        "A": 18,
        "B": 42,
        "C": 80,
        "D": 20,
        "E": 28,
        "F": 48,
        "G": 26,
        "H": 56,
    }
    for column, width in widths.items():
        worksheet.column_dimensions[column].width = width


def load_existing_report_rows(path: Path, headers: list[str]) -> list[dict[str, str]]:
    if not path.exists():
        return []

    try:
        workbook = load_workbook(path, read_only=True)
        worksheet = workbook.active
        existing_headers = [cell.value for cell in next(worksheet.iter_rows(min_row=1, max_row=1))]
        rows: list[dict[str, str]] = []
        for values in worksheet.iter_rows(min_row=2, values_only=True):
            current = {str(existing_headers[index]): str(value or "") for index, value in enumerate(values)}
            rows.append({header: current.get(header, "") for header in headers})
        return rows
    except (BadZipFile, OSError, StopIteration, KeyError, ValueError):
        return []


def merge_rows(existing_rows: list[dict[str, str]], new_rows: list[dict[str, str]], headers: list[str]) -> list[dict[str, str]]:
    merged: dict[str, dict[str, str]] = {}
    order: list[str] = []

    def key_for(row: dict[str, str]) -> str:
        return "||".join(
            [
                row.get("企业名", ""),
                row.get("资讯标题", ""),
                row.get("资讯时间", ""),
                row.get("资讯链接", ""),
            ]
        )

    for row in existing_rows + new_rows:
        key = key_for(row)
        normalized = {header: row.get(header, "") for header in headers}
        if key not in merged:
            order.append(key)
        merged[key] = normalized

    return [merged[key] for key in order]


def resolve_writable_report_path(preferred_path: Path) -> Path:
    try:
        preferred_path.parent.mkdir(parents=True, exist_ok=True)
        if not preferred_path.exists():
            return preferred_path
        with preferred_path.open("ab"):
            pass
        return preferred_path
    except PermissionError:
        timestamp = datetime.now().strftime("%H%M%S")
        return preferred_path.with_name(f"{preferred_path.stem}_{timestamp}{preferred_path.suffix}")


def write_report(rows: list[dict[str, str]]) -> Path:
    preferred_path = REPORT_DIR / "company_mentions.xlsx"

    workbook = Workbook()
    worksheet = workbook.active
    worksheet.title = "企业资讯"

    headers = ["企业名", "资讯标题", "资讯内容", "资讯来源", "资讯时间", "资讯链接", "AI标题", "AI摘要"]
    worksheet.append(headers)

    header_font = Font(bold=True)
    wrap_alignment = Alignment(vertical="top", wrap_text=True)

    for cell in worksheet[1]:
        cell.font = header_font
        cell.alignment = wrap_alignment

    existing_rows = load_existing_report_rows(preferred_path, headers)
    merged_rows = merge_rows(existing_rows, rows, headers)

    for row in merged_rows:
        worksheet.append([row.get(header, "") for header in headers])

    for row in worksheet.iter_rows(min_row=2):
        for cell in row:
            cell.alignment = wrap_alignment

    worksheet.freeze_panes = "A2"
    autosize_worksheet(worksheet)
    output_path = resolve_writable_report_path(preferred_path)
    workbook.save(output_path)
    return output_path


def main() -> int:
    configure_output_encoding()
    ensure_directories()
    args = parse_args()

    try:
        try:
            companies = load_companies(args.companies.resolve())
        except Exception as error:
            print(f"运行失败: {error}", file=sys.stderr)
            return 1

        if not companies:
            print("企业名单为空，请先填写 companies.txt。")
            return 0

        if args.time_window:
            paths, (start, end) = target_data_files_by_window(args.time_window)
        else:
            paths = target_data_files(max(args.days, 1))
            start = end = None
        if not paths:
            print("未找到可扫描的数据文件。")
            return 0

        records = read_records(paths)
        if start and end:
            records = filter_records_by_time_window(records, start, end)
        rows = match_records(records, companies)
        rows, generated_count = enrich_rows_with_ai(rows, args.disable_ai)
        output_path = write_report(rows)
        international_rows = load_latest_international_rows(
            REPORT_DIR,
            datetime.now().date().isoformat(),
            args.time_window,
        )
        word_output_path = write_daily_brief(REPORT_DIR, paths, rows, international_rows, args.time_window)

        print(f"已扫描 {len(paths)} 个数据文件，共 {len(records)} 条资讯。")
        print(f"命中 {len(rows)} 条企业资讯，AI 新生成 {generated_count} 条。")
        print(f"已写入 {output_path}")
        print(f"已写入 {word_output_path}")
        return 0
    finally:
        clear_summary_cache()


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/generate_international_report.py
content: from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import sys
import time
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from zipfile import BadZipFile

from openpyxl import Workbook
from openpyxl import load_workbook
from openpyxl.styles import Alignment, Font

from brief_writer import load_latest_domestic_rows, write_daily_brief
from time_window import (
    existing_paths_from_stems,
    in_time_window,
    parse_time_window,
    target_date_strings_from_window,
)


BASE_DIR = Path(os.getenv("AI_NEWS_WORKSPACE", os.getcwd())).resolve()
DATA_DIR = BASE_DIR / "data"
REPORT_DIR = BASE_DIR / "reports"
STATE_DIR = BASE_DIR / "state"
SUMMARY_CACHE_PATH = STATE_DIR / "international_report_ai_cache.json"

ARK_API_BASE = os.getenv("ARK_API_BASE", "https://ark.cn-beijing.volces.com/api/v3")
ARK_MODEL = os.getenv("ARK_MODEL", "doubao-seed-2-0-mini-260215")
ARK_API_KEY = os.getenv("ARK_API_KEY", "").strip()
ARK_TIMEOUT_SECONDS = int(os.getenv("ARK_TIMEOUT_SECONDS", "60"))
MAX_CACHE_ITEMS = 5000
INTERNATIONAL_CACHE_VERSION = "v6_natural_number_phrasing"
MAX_AI_RETRIES = 2


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Generate AI-enriched international news report.")
    parser.add_argument(
        "--days",
        type=int,
        default=2,
        help="How many days of files to scan, counting today. Default: 2",
    )
    parser.add_argument(
        "--disable-ai",
        action="store_true",
        help="Skip AI summary/title/impact generation even if ARK_API_KEY is set.",
    )
    parser.add_argument(
        "--time-window",
        type=str,
        default="",
        help="资讯时间窗口，例如：2026年3月15日9点到2026年3月15日18点",
    )
    return parser.parse_args()


def configure_output_encoding() -> None:
    for stream_name in ("stdout", "stderr"):
        stream = getattr(sys, stream_name, None)
        reconfigure = getattr(stream, "reconfigure", None)
        if callable(reconfigure):
            reconfigure(encoding="utf-8", errors="replace")


def ensure_directories() -> None:
    REPORT_DIR.mkdir(parents=True, exist_ok=True)
    STATE_DIR.mkdir(parents=True, exist_ok=True)


def target_data_files(days: int) -> list[Path]:
    today = datetime.now().date()
    paths: list[Path] = []
    for offset in range(days):
        day = today - timedelta(days=offset)
        path = DATA_DIR / f"international_{day.isoformat()}.jsonl"
        if path.exists():
            paths.append(path)
    return paths


def target_data_files_by_window(time_window: str) -> tuple[list[Path], tuple[datetime, datetime]]:
    start, end = parse_time_window(time_window)
    stems = target_date_strings_from_window(start, end, prefix="international_")
    paths = existing_paths_from_stems(DATA_DIR, stems)
    return paths, (start, end)


def read_records(paths: list[Path]) -> list[dict[str, Any]]:
    records: list[dict[str, Any]] = []
    for path in paths:
        with path.open("r", encoding="utf-8") as file:
            for line_number, line in enumerate(file, start=1):
                cleaned = line.strip()
                if not cleaned:
                    continue
                try:
                    records.append(json.loads(cleaned))
                except json.JSONDecodeError as error:
                    print(
                        f"[WARN] 跳过无效 JSON: {path.name}:{line_number} {error}",
                        file=sys.stderr,
                    )
    return records


def record_time_value(record: dict[str, Any]) -> str:
    return str(record.get("published_at", "") or "")


def records_to_rows(records: list[dict[str, Any]]) -> list[dict[str, str]]:
    rows: list[dict[str, str]] = []
    for record in records:
        rows.append(
            {
                "资讯标题": str(record.get("title", "") or ""),
                "资讯内容": str(record.get("content", "") or ""),
                "资讯来源": str(record.get("source_name", "") or ""),
                "资讯时间": record_time_value(record),
                "资讯链接": str(record.get("link", "") or ""),
                "AI标题": "",
                "AI摘要": "",
                "事件影响力": "",
            }
        )
    return rows


def filter_records_by_time_window(
    records: list[dict[str, Any]],
    start: datetime,
    end: datetime,
) -> list[dict[str, Any]]:
    return [
        record
        for record in records
        if in_time_window(record_time_value(record), start, end)
    ]


def load_summary_cache() -> dict[str, dict[str, str]]:
    if not SUMMARY_CACHE_PATH.exists():
        return {}

    try:
        with SUMMARY_CACHE_PATH.open("r", encoding="utf-8") as file:
            data = json.load(file)
    except (OSError, json.JSONDecodeError):
        return {}

    if not isinstance(data, dict):
        return {}
    return {str(key): value for key, value in data.items() if isinstance(value, dict)}


def save_summary_cache(cache: dict[str, dict[str, str]]) -> None:
    items = list(cache.items())[-MAX_CACHE_ITEMS:]
    trimmed_cache = {key: value for key, value in items}
    with SUMMARY_CACHE_PATH.open("w", encoding="utf-8") as file:
        json.dump(trimmed_cache, file, ensure_ascii=False, indent=2)


def clear_summary_cache() -> None:
    try:
        if SUMMARY_CACHE_PATH.exists():
            SUMMARY_CACHE_PATH.write_text("{}\n", encoding="utf-8")
    except OSError:
        pass


def make_cache_key(row: dict[str, str]) -> str:
    raw = "||".join(
        [
            INTERNATIONAL_CACHE_VERSION,
            row["资讯标题"],
            row["资讯内容"],
            row["资讯来源"],
            row["资讯时间"],
            row["资讯链接"],
        ]
    )
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()


def build_ai_payload(row: dict[str, str], *, retry_index: int = 0) -> dict[str, Any]:
    retry_hint = ""
    if retry_index > 0:
        retry_hint = (
            "这是重试生成。上一次结果不符合要求。"
            "请特别注意：不要保留普通英文短语，只有机构名、公司名、产品名、模型名这类专有名词才保留英文；"
            "标题必须自然完整，不能过短；涉及数字、比例、金额、人数时，要写成自然中文表达。"
        )

    prompt = (
        "你是一名中文AI产业资讯编辑。请基于下面这条国际资讯生成 JSON，"
        '必须只返回 JSON 对象，格式为 {"short_title":"...","summary":"...","impact_score":0}。'
        "要求："
        "1. short_title 用中文生成，但涉及机构名称、公司名称、产品名称、模型名称时，优先保留英文原写，不要强行翻译成中文。"
        "2. short_title 的字数限制是争取达成的目标：中文字符尽量控制在15个左右，通常不超过20个；英文字符、英文单词、英文缩写和阿拉伯数字不计入字数。"
        "3. 除机构名称、公司名称、产品名称、模型名称外，普通英文短语、英文描述、英文术语都应翻译成自然中文，不要直接保留。"
        "4. summary 用中文生成，但涉及机构名称、公司名称、产品名称、模型名称时，优先保留英文原写，不要强行翻译成中文。"
        "5. 除机构名称、公司名称、产品名称、模型名称外，普通英文短语、英文描述、英文术语都应翻译成自然中文，不要直接保留。"
        "6. summary 的字数限制是争取达成的目标：中文字符尽量控制在100个左右，通常不超过120个；英文字符、英文单词、英文缩写和阿拉伯数字不计入字数。"
        "7. 涉及数字、比例、金额、估值、人数、轮次时，要写成自然中文表达。"
        "8. 例如应写成“裁员约20%”“获得530万美元融资”“估值90亿美元”“86%的企业”“47%的企业”，不要写成生硬残缺的“20规模裁员”“86已部署”“47采用”。"
        "9. 如果在目标字数内无法写成完整句子，优先保证句子完整，不要把一句话截断。"
        "10. 摘要必须准确概括事件，不要编造原文没有的信息。"
        "11. impact_score 为 0 到 100 的整数，表示事件影响力。"
        "12. 影响力评分需综合考虑所涉机构的全球影响力，以及该事件对AI产业的影响。"
        "13. 必须只返回 JSON，不要输出任何额外说明。"
        "\n\n"
        f"{retry_hint}\n"
        f"原标题：{row['资讯标题']}\n"
        f"资讯内容：{row['资讯内容']}\n"
        f"资讯来源：{row['资讯来源']}\n"
        f"资讯时间：{row['资讯时间']}\n"
        f"资讯链接：{row['资讯链接']}\n"
    )

    return {
        "model": ARK_MODEL,
        "input": prompt,
    }


def extract_text_from_response(payload: dict[str, Any]) -> str:
    output_text = payload.get("output_text")
    if isinstance(output_text, str) and output_text.strip():
        return output_text.strip()

    output = payload.get("output")
    if isinstance(output, list):
        texts: list[str] = []
        for item in output:
            if not isinstance(item, dict):
                continue
            content = item.get("content")
            if not isinstance(content, list):
                continue
            for part in content:
                if not isinstance(part, dict):
                    continue
                text = part.get("text")
                if isinstance(text, str) and text.strip():
                    texts.append(text.strip())
        if texts:
            return "\n".join(texts)

    choices = payload.get("choices")
    if isinstance(choices, list):
        texts = []
        for choice in choices:
            if not isinstance(choice, dict):
                continue
            message = choice.get("message")
            if isinstance(message, dict):
                content = message.get("content")
                if isinstance(content, str) and content.strip():
                    texts.append(content.strip())
            text = choice.get("text")
            if isinstance(text, str) and text.strip():
                texts.append(text.strip())
        if texts:
            return "\n".join(texts)

    return ""


def call_ark_api(row: dict[str, str], *, retry_index: int = 0) -> dict[str, str]:
    if not ARK_API_KEY:
        raise RuntimeError("未设置 ARK_API_KEY")

    payload = build_ai_payload(row, retry_index=retry_index)
    body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    request = Request(
        f"{ARK_API_BASE.rstrip('/')}/responses",
        data=body,
        headers={
            "Content-Type": "application/json; charset=utf-8",
            "Authorization": f"Bearer {ARK_API_KEY}",
        },
        method="POST",
    )

    with urlopen(request, timeout=ARK_TIMEOUT_SECONDS) as response:
        response_payload = json.loads(response.read().decode("utf-8"))

    text = extract_text_from_response(response_payload)
    if not text:
        raise RuntimeError("模型返回内容为空")

    try:
        parsed = json.loads(text)
    except json.JSONDecodeError as error:
        raise RuntimeError(f"模型返回不是有效 JSON: {error}") from error

    short_title = str(parsed.get("short_title", "") or "").strip()
    summary = str(parsed.get("summary", "") or "").strip()
    raw_score = parsed.get("impact_score", "")
    if not short_title or not summary:
        raise RuntimeError("模型返回缺少 short_title 或 summary")

    try:
        score = int(raw_score)
    except (TypeError, ValueError) as error:
        raise RuntimeError(f"impact_score 无法解析为整数: {raw_score}") from error

    normalized_title = normalize_text_keep_english(short_title)
    normalized_summary = normalize_text_keep_english(summary)
    if chinese_length(normalized_title) < 4:
        raise RuntimeError("模型返回的中文标题清洗后过短")
    if chinese_length(normalized_summary) < 40:
        raise RuntimeError("模型返回的中文摘要清洗后过短")

    score = max(0, min(100, score))
    return {
        "AI标题": normalized_title,
        "AI摘要": normalized_summary,
        "事件影响力": str(score),
    }


def chinese_length(text: str) -> int:
    return len(re.findall(r"[\u4e00-\u9fff]", text))


def normalize_text_keep_english(text: str) -> str:
    cleaned = re.sub(r"\s+", " ", text).strip()
    cleaned = cleaned.replace("「", "").replace("」", "").replace("“", "").replace("”", "")
    cleaned = cleaned.replace('"', "").replace("'", "")
    cleaned = re.sub(r"[^A-Za-z0-9\u4e00-\u9fff，。；：、】【（）《》、“”‘’—…·\-\s]", "", cleaned)
    cleaned = re.sub(r"\s{2,}", " ", cleaned)
    return cleaned.strip()


def enrich_rows_with_ai(rows: list[dict[str, str]], disable_ai: bool) -> tuple[list[dict[str, str]], int]:
    if disable_ai:
        print("已按参数跳过国际资讯 AI 生成。")
        return rows, 0

    if not ARK_API_KEY:
        print("未设置 ARK_API_KEY，已跳过国际资讯 AI 生成。")
        return rows, 0

    if not rows:
        return rows, 0

    cache = load_summary_cache()
    generated_count = 0

    for index, row in enumerate(rows, start=1):
        cache_key = make_cache_key(row)
        cached = cache.get(cache_key)
        if cached:
            row["AI标题"] = str(cached.get("AI标题", "") or "")
            row["AI摘要"] = str(cached.get("AI摘要", "") or "")
            row["事件影响力"] = str(cached.get("事件影响力", "") or "")
            continue

        success = False
        for retry_index in range(MAX_AI_RETRIES + 1):
            try:
                result = call_ark_api(row, retry_index=retry_index)
                row.update(result)
                cache[cache_key] = result
                generated_count += 1
                print(f"[AI] 已生成 {index}/{len(rows)}: {row['资讯来源']} | {row['资讯标题']}")
                time.sleep(0.4)
                success = True
                break
            except (HTTPError, URLError, TimeoutError, RuntimeError, OSError) as error:
                if retry_index >= MAX_AI_RETRIES:
                    print(f"[AI WARN] 国际资讯生成失败: {error}", file=sys.stderr)
                else:
                    print(f"[AI RETRY] 第 {retry_index + 1} 次重试: {row['资讯标题']}", file=sys.stderr)
                    time.sleep(0.8)
        if not success:
            continue

    save_summary_cache(cache)
    return rows, generated_count


def sort_rows(rows: list[dict[str, str]]) -> list[dict[str, str]]:
    def sort_key(row: dict[str, str]) -> tuple[int, str, str]:
        try:
            score = int(row.get("事件影响力", "") or 0)
        except ValueError:
            score = 0
        return (-score, row.get("资讯时间", ""), row.get("资讯标题", ""))

    return sorted(rows, key=sort_key)


def autosize_worksheet(worksheet) -> None:
    widths = {
        "A": 42,
        "B": 80,
        "C": 20,
        "D": 28,
        "E": 48,
        "F": 26,
        "G": 56,
        "H": 14,
    }
    for column, width in widths.items():
        worksheet.column_dimensions[column].width = width


def load_existing_report_rows(path: Path, headers: list[str]) -> list[dict[str, str]]:
    if not path.exists():
        return []

    try:
        workbook = load_workbook(path, read_only=True)
        worksheet = workbook.active
        existing_headers = [cell.value for cell in next(worksheet.iter_rows(min_row=1, max_row=1))]
        rows: list[dict[str, str]] = []
        for values in worksheet.iter_rows(min_row=2, values_only=True):
            current = {str(existing_headers[index]): str(value or "") for index, value in enumerate(values)}
            rows.append({header: current.get(header, "") for header in headers})
        return rows
    except (BadZipFile, OSError, StopIteration, KeyError, ValueError):
        return []


def merge_rows(existing_rows: list[dict[str, str]], new_rows: list[dict[str, str]], headers: list[str]) -> list[dict[str, str]]:
    merged: dict[str, dict[str, str]] = {}
    order: list[str] = []

    def key_for(row: dict[str, str]) -> str:
        return "||".join(
            [
                row.get("资讯标题", ""),
                row.get("资讯时间", ""),
                row.get("资讯链接", ""),
            ]
        )

    for row in existing_rows + new_rows:
        key = key_for(row)
        normalized = {header: row.get(header, "") for header in headers}
        if key not in merged:
            order.append(key)
        merged[key] = normalized

    return [merged[key] for key in order]


def resolve_writable_report_path(preferred_path: Path) -> Path:
    try:
        preferred_path.parent.mkdir(parents=True, exist_ok=True)
        if not preferred_path.exists():
            return preferred_path
        with preferred_path.open("ab"):
            pass
        return preferred_path
    except PermissionError:
        timestamp = datetime.now().strftime("%H%M%S")
        return preferred_path.with_name(f"{preferred_path.stem}_{timestamp}{preferred_path.suffix}")


def write_report(rows: list[dict[str, str]]) -> Path:
    preferred_path = REPORT_DIR / "international_company_mentions.xlsx"

    workbook = Workbook()
    worksheet = workbook.active
    worksheet.title = "国际AI资讯"

    headers = ["资讯标题", "资讯内容", "资讯来源", "资讯时间", "资讯链接", "AI标题", "AI摘要", "事件影响力"]
    worksheet.append(headers)

    header_font = Font(bold=True)
    wrap_alignment = Alignment(vertical="top", wrap_text=True)

    for cell in worksheet[1]:
        cell.font = header_font
        cell.alignment = wrap_alignment

    existing_rows = load_existing_report_rows(preferred_path, headers)
    merged_rows = merge_rows(existing_rows, rows, headers)

    for row in merged_rows:
        worksheet.append([row.get(header, "") for header in headers])

    for row in worksheet.iter_rows(min_row=2):
        for cell in row:
            cell.alignment = wrap_alignment

    worksheet.freeze_panes = "A2"
    autosize_worksheet(worksheet)
    output_path = resolve_writable_report_path(preferred_path)
    workbook.save(output_path)
    return output_path


def main() -> int:
    configure_output_encoding()
    ensure_directories()
    args = parse_args()
    try:
        if args.time_window:
            paths, (start, end) = target_data_files_by_window(args.time_window)
        else:
            paths = target_data_files(max(args.days, 1))
            start = end = None
        if not paths:
            print("未找到可扫描的国际数据文件。")
            return 0

        records = read_records(paths)
        if start and end:
            records = filter_records_by_time_window(records, start, end)
        rows = records_to_rows(records)
        rows, generated_count = enrich_rows_with_ai(rows, args.disable_ai)
        rows = sort_rows(rows)
        output_path = write_report(rows)
        domestic_rows = load_latest_domestic_rows(
            REPORT_DIR,
            datetime.now().date().isoformat(),
            args.time_window,
        )
        word_output_path = write_daily_brief(REPORT_DIR, paths, domestic_rows, rows, args.time_window)

        print(f"已扫描 {len(paths)} 个国际数据文件，共 {len(records)} 条资讯。")
        print(f"已输出 {len(rows)} 条国际资讯，AI 新生成 {generated_count} 条。")
        print(f"已写入 {output_path}")
        print(f"已写入 {word_output_path}")
        return 0
    finally:
        clear_summary_cache()


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/requirements.txt
content: openpyxl
python-docx


path: scripts/run_capture_only.py
content: from __future__ import annotations

import argparse
import os
from pathlib import Path


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Run capture only for the AI news pipeline.")
    parser.add_argument(
        "--workspace",
        type=Path,
        default=Path.cwd(),
        help="Workspace root containing config, data, reports, and state directories.",
    )
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    workspace = args.workspace.resolve()
    os.environ["AI_NEWS_WORKSPACE"] = str(workspace)

    import collect_feeds

    return collect_feeds.main()


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/run_full_workflow.py
content: from __future__ import annotations

import argparse
import os
import sys
from pathlib import Path


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Run the full AI news workflow.")
    parser.add_argument(
        "--time-window",
        type=str,
        default="",
        help="Optional natural-language time window string.",
    )
    parser.add_argument(
        "--workspace",
        type=Path,
        default=Path.cwd(),
        help="Workspace root containing config, data, reports, and state directories.",
    )
    parser.add_argument(
        "--disable-ai",
        action="store_true",
        help="Skip AI generation in both domestic and international report steps.",
    )
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    workspace = args.workspace.resolve()
    os.environ["AI_NEWS_WORKSPACE"] = str(workspace)

    import collect_feeds
    import run_report_only

    print("=== Step 1: Collect RSS feeds ===", flush=True)
    capture_code = collect_feeds.main()
    if capture_code:
        return capture_code

    print("=== Step 2: Build reports ===", flush=True)
    original_argv = sys.argv[:]
    try:
        sys.argv = ["run_report_only.py", "--workspace", str(workspace)]
        if args.time_window:
            sys.argv.extend(["--time-window", args.time_window])
        if args.disable_ai:
            sys.argv.append("--disable-ai")
        report_code = run_report_only.main()
        if report_code:
            return report_code
    finally:
        sys.argv = original_argv

    print("=== Workflow complete ===", flush=True)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/run_report_only.py
content: from __future__ import annotations

import argparse
import os
import sys
from datetime import datetime, timedelta
from pathlib import Path


def default_report_window() -> str:
    now = datetime.now()
    yesterday = (now.date() - timedelta(days=1)).isoformat()
    today = now.date().isoformat()
    return f"{yesterday} 00:00 to {today} 08:00"


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Run report generation only for the AI news pipeline.")
    parser.add_argument(
        "--workspace",
        type=Path,
        default=Path.cwd(),
        help="Workspace root containing config, data, reports, and state directories.",
    )
    parser.add_argument(
        "--time-window",
        type=str,
        default="",
        help="Optional natural-language time window string. Defaults to yesterday 00:00 to today 08:00.",
    )
    parser.add_argument(
        "--disable-ai",
        action="store_true",
        help="Skip AI generation in both domestic and international report steps.",
    )
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    workspace = args.workspace.resolve()
    os.environ["AI_NEWS_WORKSPACE"] = str(workspace)
    time_window = args.time_window.strip() or default_report_window()

    print(f"=== Report window: {time_window} ===", flush=True)

    import generate_company_report
    import generate_international_report

    original_argv = sys.argv[:]
    try:
        print("=== Step 1: Build China report and merge Word brief ===", flush=True)
        sys.argv = ["generate_company_report.py", "--time-window", time_window]
        if args.disable_ai:
            sys.argv.append("--disable-ai")
        china_code = generate_company_report.main()
        if china_code:
            return china_code

        print("=== Step 2: Build international report and refresh Word brief ===", flush=True)
        sys.argv = ["generate_international_report.py", "--time-window", time_window]
        if args.disable_ai:
            sys.argv.append("--disable-ai")
        return generate_international_report.main()
    finally:
        sys.argv = original_argv


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/time_window.py
content: from __future__ import annotations

import re
from datetime import date, datetime, time, timedelta
from pathlib import Path


CHINESE_TIME_WINDOW_PATTERN = re.compile(
    r"^\s*"
    r"(\d{4})年(\d{1,2})月(\d{1,2})日(\d{1,2})点(?:((?:\d{1,2}))分?)?"
    r"\s*到\s*"
    r"(\d{4})年(\d{1,2})月(\d{1,2})日(\d{1,2})点(?:((?:\d{1,2}))分?)?"
    r"\s*$"
)
ISO_TIME_WINDOW_PATTERN = re.compile(
    r"^\s*"
    r"(\d{4})-(\d{1,2})-(\d{1,2})\s+(\d{1,2}):(\d{1,2})(?::(\d{1,2}))?"
    r"\s+to\s+"
    r"(\d{4})-(\d{1,2})-(\d{1,2})\s+(\d{1,2}):(\d{1,2})(?::(\d{1,2}))?"
    r"\s*$",
    re.IGNORECASE,
)


def build_daily_report_window(*, now: datetime | None = None, end_hour: int = 8) -> tuple[datetime, datetime]:
    current = now or datetime.now()
    end = datetime.combine(current.date(), time(hour=end_hour))
    start = datetime.combine(current.date() - timedelta(days=1), time.min)
    return start, end


def format_chinese_time_window(start: datetime, end: datetime) -> str:
    def format_one(value: datetime) -> str:
        if value.minute:
            return f"{value.year}年{value.month}月{value.day}日{value.hour}点{value.minute}分"
        return f"{value.year}年{value.month}月{value.day}日{value.hour}点"

    return f"{format_one(start)}到{format_one(end)}"


def default_daily_report_window_text(*, now: datetime | None = None, end_hour: int = 8) -> str:
    start, end = build_daily_report_window(now=now, end_hour=end_hour)
    return format_chinese_time_window(start, end)


def parse_time_window(value: str) -> tuple[datetime, datetime]:
    match = CHINESE_TIME_WINDOW_PATTERN.match(value)
    if match:
        groups = match.groups()
        start = datetime(
            year=int(groups[0]),
            month=int(groups[1]),
            day=int(groups[2]),
            hour=int(groups[3]),
            minute=int(groups[4] or 0),
        )
        end = datetime(
            year=int(groups[5]),
            month=int(groups[6]),
            day=int(groups[7]),
            hour=int(groups[8]),
            minute=int(groups[9] or 0),
        )
        if end < start:
            raise ValueError("The time window end cannot be earlier than the start.")
        return start, end

    match = ISO_TIME_WINDOW_PATTERN.match(value)
    if match:
        groups = match.groups()
        start = datetime(
            year=int(groups[0]),
            month=int(groups[1]),
            day=int(groups[2]),
            hour=int(groups[3]),
            minute=int(groups[4]),
            second=int(groups[5] or 0),
        )
        end = datetime(
            year=int(groups[6]),
            month=int(groups[7]),
            day=int(groups[8]),
            hour=int(groups[9]),
            minute=int(groups[10]),
            second=int(groups[11] or 0),
        )
        if end < start:
            raise ValueError("The time window end cannot be earlier than the start.")
        return start, end

    raise ValueError(
        "Invalid time window format. Use '2026年3月15日0点到2026年3月16日8点' "
        "or '2026-03-15 00:00 to 2026-03-16 08:00'."
    )


def parse_record_datetime(value: str) -> datetime | None:
    cleaned = (value or "").strip()
    if not cleaned:
        return None

    try:
        dt = datetime.fromisoformat(cleaned.replace("Z", "+00:00"))
    except ValueError:
        return None

    if dt.tzinfo is not None:
        return dt.astimezone().replace(tzinfo=None)
    return dt


def in_time_window(value: str, start: datetime, end: datetime) -> bool:
    dt = parse_record_datetime(value)
    if dt is None:
        return False
    return start <= dt <= end


def target_date_strings_from_window(start: datetime, end: datetime, *, prefix: str = "") -> list[str]:
    dates: list[str] = []
    current: date = start.date()
    end_date = end.date()
    while current <= end_date:
        date_text = current.isoformat()
        dates.append(f"{prefix}{date_text}" if prefix else date_text)
        current += timedelta(days=1)
    return dates


def existing_paths_from_stems(base_dir: Path, stems: list[str], suffix: str = ".jsonl") -> list[Path]:
    paths: list[Path] = []
    for stem in stems:
        path = base_dir / f"{stem}{suffix}"
        if path.exists():
            paths.append(path)
    return paths


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)